PS1 IPP Czar Logs for the week 2013.11.18 - 2013.11.24

(Up to PS1 IPP Czar Logs)

Monday : 2013.11.18

• 13:00 bill set stdscience to off in preparation for daily restart.
• 13:10 looks like we only have two storage partitions with any space
• 13:15 set distribution and pstamp to stop because I have a big query running on the ippc17 mysql database that might block operations
• 13:44 stdscience restarted with STS.rp.2013 label out. So there is nothing to do.
  • a reminder that the LAP label is also out because of low disk space, so no LAP processing has happened for over a week.
• 14:21 Bill stopped all pantasks to pick up a bug fix in psModules. assertion failure !isfinite(z) caused by zero axes.major in pmMoelSetShape
  • 14:25 the fix is in.
• 14:55 MEH: adding 1x compute3 into cleanup to help get through and make some space

Tuesday : 2013.11.19

Wednesday : 2013.11.20

mark is czar

• 07:15 MEH: nightly finished
  • diffim fault 5 from new tag changes still -- diff_id= 495078 skycell.033 diff_id= 495197 skycell.072
  • summit fault on bias image -- interrupt on TDI plan, set exposure to drop

    pztool -updatepzexp -exp_name c6616g0001b -inst gpc1 -telescope ps1 -set_state drop -summit_id 671838 -dbname gpc1
    
    -- update ippdb01
    update summitExp set exp_type ='broken', imfiles=0, fault=0 where exp_name='c6616g0001b';
    

• 10:00 MEH: deep stack processing is moving to a local deepstack pantasks with the normal 1x compute3 allocation for tag modifications... deepstack pantasks should remain stopped and compute3 in limited use unless rebalanced properly
• 11:00 MEH: even though MD05 was observed last night, not going to be a regular field and diffims not going to be made (nor a new refstack made until a deepstack made)
  • it is there, might as will enable the last season refstack for diffims in near future
• 19:05 MEH: ipp066 may have crashed -- yes, power cycled and back up (ipp066-20131120-crash.log)

Thursday : 2013.11.21

mark is czar

• 15:30 MEH: restarting stdsci to enable all MD refstacks for any random field that pops up and the final return. need to manually remove following hosts from stsci (and all pantasks) while mysql is in heavy use there

  ipp011
  ipp049
  ipp052
  ipp061
  ipp063
  ipp065
  -- more may need to be added if have nightly data tonight and gets stuck
  

Friday : 2013.11.22

Saturday : 2013.11.23

Sunday : 2013.11.24