PS1 IPP Czar Logs for the week 2011-04-25 - 2011-05-01

(Up to PS1 IPP Czar Logs)

Monday : 2011-04-25

Bill is czar today.

• 09:00 The 576 STS.2010.b exposures queued for reprocessing so far have finished and have been posted to the distribution server. Queued the chip, warp, and diff runs for cleanup
• 11:58 set pantasks' to run. They were stopped to allow Chris to make some changes to nebulous.
• 13:56 CZW: ATRC storage nodes completely added to nebulous. replication pantasks is stopped, as we want to do some replication tests to send data to these disks.

Tuesday : 2011-04-26

• 08:00 Bill burntool was stuck due to a database error that left a rawImfile in an incorrect state. Fixed by running the command that failed by hand.

  regtool -updateprocessedimfile -dbname gpc1 -exp_id 330078 -class_id 'XY46' -burntool_state 0 -set_state pending_burntool
  
  

• 08:05 Bill warp was stuck because the book warpPendingSkyCell was full of entries in pantaskState DONE. Fixed by running warp.reset stdscience.
• 09:32 ipp041 is generating lots of faults because it cannot see /data/ippb02.0. force.umount hung. Running sudo umount -f -v /data/ippb02.2 hung as well. Set ipp041 to off in stdscience.
• 10:56 Heather snuck in some more jt stacks. (MD02.jtrp)
• 11:15 Serge tries to replicate ippc02.gpc1 onto ipp001. There might be unusual activity between them.

Wednesday : 2011-04-27

• 07:20 Bill noticed that warp was stuck. warpPendingSkyCell book full of DONE jobs. warp.off wait for any running jobs to finish ; warp.reset ; warp.on
• 07:30 interestingly the czartool graph for warp did not flatten out even though there were no warp jobs running. Perhaps I caught it just before it would have flat lined.
• 10:27 Bill set label STS.2010.b to inactive to prevent chip processing from slowing down last night's science processing.
• 10:31 Bill set ipp012 and ipp020 nebulous state to up (they were set to repair over the weekend)

Thursday : 2011-04-28

Friday : 2011-04-29

Saturday : 2011-04-30

Sunday : 2011-05-01