PS1 IPP Czar Logs for the week 2012.04.02 - 2012.04.08

(Up to PS1 IPP Czar Logs)

Monday : 2012.04.02

Mark is czar

• 07:00 full night of 600 exposures, data still downloading (slower than normal?)
• 09:20 data finished downloading ~30min ago, 99% processed.
• 18:05 Bill added one compliment of wave3 and one of compute to deepstack (staticsky)
• 18:51 Bill power cycled ipp015 because it died with a hardware problem: "This is not a software problem!"

Tuesday : 2012.04.03

Serge is czar

• 09:00 No observation last night. No LAP queued
• 15:00 heather has some stuff for the cluster (jtrp) - low priority.
• 15:27 ippb02 and ippb03 are now up from the nebulous point of view
• 21:36 Bill: Restarted registration pantasks
• 23:55 Mark: stdscience seems to be under-running, going to try a restart. oops, lost cut/paste of the jtrp label, was it MD03.jtrp.v2?

Wednesday : 2012.04.04

• 06:30 Bill: ipp015 crashed again. Set it to off in nebulous and power cycled it. The nebulous apache servers seem to be running amok. Sst pantasks' to stop to let things calm down.
• 06:50 ipp015 did not come up after 3 power cycles. Leaving it powerd on.
• 07:00 restarted apache servers on ippc01 - ippc10 set pantasks' to run
• 11:35 Serge: rebooted ipp021 since it has a weird behavior (can't 'ls /export/ipp021.0/nebulous/1c/54' for instance). dmesg claims that process 7543/nfsd is "Tainted".

Thursday : 2012.04.05

Friday : 2012.04.06

Saturday : 2012.04.07

Sunday : 2012.04.08