== PS1 IPP Czar Logs for the week 2013.12.02 - 2013.12.08 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.12.02 ===
Bill is czar today
 * 10:47 Set all data with label like ps_ud% to be cleaned. Restarted cleanup pantasks.
 * 13:30 Dropped the 6 STD.nightlyscience chipRuns from 11-27 and 11-29 for exposures with filter = 'OPEN' We don't process those because we have no flat detrends
 * 13:35 set all pantasks to stop in preparation for full restart. (The only one doing anything currently is cleanup)
 * 16:15 set STS.rp.2013 distribution bundles to be cleaned
 * 19:05 MEH: '''a reminder''' when restarting any pantasks, hosts may need to be manually turned off to account for DVO/PSPS/mysql use on machines until mysql etc can be restarted or jobs/nightly processing is likely to get jammed  -- currently  looks like ipp048--ipp053, 039, 057 may still be problem childs.
  * ipp053 holding onto registration job >3ks, chip jobs >2ks so far..
  * turning these ipp048--ipp053 off
=== Tuesday : 2013.12.03 ===
 * 08:35 Bill: Serge reports that ipp is stuck. It appears that ipp057 is down and some registration jobs are stuck trying to access files there power cycling it
 * 16:10 MEH: in addition to manually taking nodes out of processing if DVO/PSPS overloading, they may also need to be put into neb repair (more of an issue now that there is open space on many of the similar data nodes
 * 16:20 MEH: tweak_ssdiff to get the SSdiff out for QUB
=== Wednesday : 2013.12.04 ===
 * 09:38 Bill: dropped chip_id 919320 and 919321 because they will fail to run. The filter for these STD exposures is OPEN and we don't have flats for that case.
 * 12:25 MEH: ipp065 needed to be taken out of stdci and stack as well..
=== Thursday : 2013.12.05 ===

=== Friday : 2013.12.06 ===

=== Saturday : 2013.12.07 ===

=== Sunday : 2013.12.08 ===