PS1 IPP Czar Logs for the week 2011-02-28 - 2011.03.06

(Up to PS1 IPP Czar Logs)

Monday : 2011.02.28

Roy is czar.

• No observations last night. Churning through magic and magicDS for labels: ThreePi.136 and STS.2010.raw.
• 11:21 bill rebooted ipp005
• 12:10 bill noticed that update pantasks died. Probably triggered by the ipp005 errors. He restarted it. Also changed pstamp_job_run to distinguish between temporarily not available files and permanently gone files.
• sometime later bill noticed that the distribution pantasks server died as well and restarted it. sigh.
• 13:49 bill doubled the number of hosts working on updates and increased the poll limit from 32 to 100.
• 17:54 mr fussy was unhappy with the progress for distribution. Not enough jobs in queue. Restarted and used 3 x default.hosts
• 21:50 it didn't help much
• 21:50 replication pantasks died. Last timestamp on logfile 21:09. Restarted by bills.