== PS1 IPP Czar Logs for the week 2012.04.09 - 2012.04.15 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.04.09 ===
 * 09:30 Bill set all data with label ps_ud% to be cleaned
 * 09:55 stopped pantasks in order to pick up a bug fix for psphotStack
 * 10:02 set pantasks to run. Restarted deepstack. Reverted faults with fault == 4
 * 11:40 changed state of staticskyRuns in state new.wait and abs(glat > 10) to new
 * 12:01 Started running camera destreak restore in the distribution pantasks. To stop use "destreak.revert.off"
 * 23:10 Mark: looks like ipp056 has become unresponsive, power cycling and back up.
  
=== Tuesday : 2012.04.10 ===

=== Wednesday : 2012.04.11 ===
 * 12:00 Bill stopped processing in order to rebuild psphot to pick up a workaround to the memory explosion issue. If the number of peaks found is > 50,000 we exit psphotStack with fault 1
 * 17:42 distribution pantasks was acting funny so Bill restarted it
=== Thursday : 2012.04.12 ===
 * 11:27 ipp017 is backing up processing because it is in swap heck trying to do 6 STS warps at a time. Set two hosts to off.

=== Friday : 2012.04.13 ===

 * 06:05 EAM : a hung mount point for ipp008 on ipp042 was blocking jobs on ipp042.  I tried to force.umount ipp008 (after killing off the relevant jobs), but failed.  I rebooted ipp042.

 * 06:10 EAM : pantasks have been running since Apr 5, and stdscience is running slow as a result.  I've shutdown everything and restarted it all.
 * 09:00 bill: distribution is backing up turning off camera destreak restore (destreak.revert)
 * 09:30 Bill: restarted deepstack
=== Saturday : 2012.04.14 ===

=== Sunday : 2012.04.15 ===