PS1 IPP Czar Logs for the week 2012.02.13 - 2012.02.19

(Up to PS1 IPP Czar Logs)

Monday : 2012.02.13

• 10:30 Bill restarted distribution. The rcserver.makefileset.run task was hung up. pantasks_server though the maximum five jobs was running but pcontrol had no jobs it it's queue.

Tuesday : 2012.02.14

• 20:02 Bill restarted stdscience and distribution. Earlier today restarted pstamp and update.

Wednesday : 2012.02.15

• 22:35 Bill set stdscience and distribution to stop in preparation for restart.
• 22:45 restarted stdscience and distribution

Thursday : 2012.02.16

Bill is czar today

• 09:53 stdscience is a bit bogged down due to backlog of jobs wanting ipp015. Setting LAP and STS labels to inactive to stop their chipRun processing until the situation improves
• 10:05 ThreePi chips are done. Turning labels back to active and setting chip.off
• 10:56 ThreePi warps are done. All diffSkyfiles are in the queue. Turning chip.on
• 16:34 ipp056 and ipp058 load spiked and the system sbecame unresponsive. Power cycled them. There was something on the console that looked like a machine code stack trace but it wasn't recognizable.
• 16:45 stdscience set to stop in peperation for the daily restart
• 18:00 restarted stdscience

Friday : 2012.02.17

Bill is czar today

• 10:40 ipp041 went down. It is preventing all jobs in the queue from completing. Setting all pantasks to stop for the time being. Ran "neb-host ipp041 down"
• 11:04 all pantasks servers shut down. Lingering jobs pkilled. Restarting stdscience with reverts turned off.

Saturday : 2012.02.18

Sunday : 2012.02.19