PS1 IPP Czar Logs for the week 2012.04.02 - 2012.04.08

(Up to PS1 IPP Czar Logs)

Monday : 2012.04.02

Mark is czar

• 07:00 full night of 600 exposures, data still downloading (slower than normal?)
• 09:20 data finished downloading ~30min ago, 99% processed.
• 18:05 Bill added one compliment of wave3 and one of compute to deepstack (staticsky)
• 18:51 Bill power cycled ipp015 because it died with a hardware problem: "This is not a software problem!"

Tuesday : 2012.04.03

Serge is czar

• 09:00 No observation last night. No LAP queued
• 15:00 heather has some stuff for the cluster (jtrp) - low priority.
• 15:27 ippb02 and ippb03 are now up from the nebulous point of view
• 21:36 Bill: Restarted registration pantasks
• 23:55 Mark: stdscience seems to be under-running, going to try a restart. oops, lost cut/paste of the jtrp label, was it MD03.jtrp.v2?

Wednesday : 2012.04.04

• 06:30 Bill: ipp015 crashed again. Set it to off in nebulous and power cycled it. The nebulous apache servers seem to be running amok. Sst pantasks' to stop to let things calm down.
• 06:50 ipp015 did not come up after 3 power cycles. Leaving it powerd on.
• 07:00 restarted apache servers on ippc01 - ippc10 set pantasks' to run
• 11:35 Serge: rebooted ipp021 (after 203 days) since it has a weird behavior (can't 'ls /export/ipp021.0/nebulous/1c/54' for instance). dmesg claims that process 7543/nfsd is "Tainted".
• 12:04 Shut down the ipp to switch to ipp-20120404. Started most things up but ran into a config file problem that causes everything to fail. Also one of the new noisemap detrend files is only located on ipp015 so chip processing is stalled. Setting pantasks states to stop until that gets fixed.

Thursday : 2012.04.05

Friday : 2012.04.06

Saturday : 2012.04.07

Sunday : 2012.04.08