== PS1 IPP Czar Logs for the week 2012.02.13 - 2012.02.19 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.02.13 ===
 * 10:30 Bill restarted distribution. The rcserver.makefileset.run task was hung up. pantasks_server though the maximum five jobs was running but pcontrol had no jobs it it's queue. 
  
=== Tuesday : 2012.02.14 ===
 * 20:02 Bill restarted stdscience and distribution. Earlier today restarted pstamp and update.

=== Wednesday : 2012.02.15 ===
 * 22:35 Bill set stdscience and distribution to stop in preparation for restart.
 * 22:45 restarted stdscience and distribution

=== Thursday : 2012.02.16 ===
Bill is czar today

 * 09:53 stdscience is a bit bogged down due to backlog of jobs wanting ipp015. Setting LAP and STS labels to inactive to stop their chipRun processing until the situation improves
 * 10:05 ThreePi chips are done. Turning labels back to active and setting chip.off
 * 10:56 ThreePi warps are done. All diffSkyfiles are in the queue. Turning chip.on
 * 16:34 ipp056  and ipp058 load spiked and the system sbecame unresponsive. Power cycled them. There was something on the console that looked like a machine code stack trace but it wasn't recognizable.
 * 16:45 stdscience set to stop in peperation for the daily restart
 * 18:00 restarted stdscience
=== Friday : 2012.02.17 ===
Bill is czar today
 * 10:40 ipp041 went down. It is preventing all jobs in the queue from completing. Setting all pantasks to stop for the time being. Ran "neb-host ipp041 down"
 * 11:04 all pantasks servers shut down. Lingering jobs pkilled. Restarting stdscience with reverts turned off.

=== Saturday : 2012.02.18 ===

=== Sunday : 2012.02.19 ===