PS1 IPP Czar Logs for the week 2013.12.02 - 2013.12.08

(Up to PS1 IPP Czar Logs)

Monday : 2013.12.02

Bill is czar today

• 10:47 Set all data with label like ps_ud% to be cleaned. Restarted cleanup pantasks.
• 13:30 Dropped the 6 STD.nightlyscience chipRuns from 11-27 and 11-29 for exposures with filter = 'OPEN' We don't process those because we have no flat detrends
• 13:35 set all pantasks to stop in preparation for full restart. (The only one doing anything currently is cleanup)
• 16:15 set STS.rp.2013 distribution bundles to be cleaned
• 19:05 MEH: a reminder when restarting any pantasks, hosts may need to be manually turned off to account for DVO/PSPS/mysql use on machines until mysql etc can be restarted or jobs/nightly processing is likely to get jammed -- currently looks like ipp048--ipp053, 039, 057 may still be problem childs.
  • ipp053 holding onto registration job >3ks, chip jobs >2ks so far..
  • turning these ipp048--ipp053 off

Tuesday : 2013.12.03

• 08:35 Bill: Serge reports that ipp is stuck. It appears that ipp057 is down and some registration jobs are stuck trying to access files there power cycling it
• 16:10 MEH: in addition to manually taking nodes out of processing if DVO/PSPS overloading, they may also need to be put into neb repair (more of an issue now that there is open space on many of the similar data nodes
• 16:20 MEH: tweak_ssdiff to get the SSdiff out for QUB

Wednesday : 2013.12.04

• 09:38 Bill: dropped chip_id 919320 and 919321 because they will fail to run. The filter for these STD exposures is OPEN and we don't have flats for that case.
• 12:25 MEH: ipp065 needed to be taken out of stack as well.. tweak_ssdiff for whatever MD stacks are available now
• 17:50 MEH: Heather finished restarting mysql on ipp048-ipp053, ipp065 so putting back into pantasks

Thursday : 2013.12.05

Friday : 2013.12.06

Saturday : 2013.12.07

Sunday : 2013.12.08