PS1 IPP Czar Logs for the week 2011-08-22 to 2011-08-28

(Up to PS1 IPP Czar Logs)

Monday : 2011-08-22

• 9:00 Bill shut down publishing, rcserver, and the postage stamp server to recover from out of disk space issue on ipppc17
• 12:20 Bill fixed a camera run with corrupt files (250162) that was causing some warps to fail
• 12:20 Bill fixed a corrupt skycell
• evening Bill queued diffs for STS.201108 19, 20, 21
• raised priority for STS and M31 nightlyscience to 400 to match the other labels.

Tuesday : 2011-08-23

• 00:00 Mark: made change to ppStack PSF matching rejection before new stacks ready and queued MD01.refstacks
• 7:40 Bill: added each of the stare nodes to stdscience twice
• 15:55 CZW: Restart stdscience to jumpstart processing after nebulous issue (still not fully understood).
• 17:50 CZW: Cycling power on ippc06 for the second time today. Removed from stdscience and stack processing.

Wednesday : 2011-08-24

• 02:15 Bill finds many faults on czartool. ipp014 load over 100. Unresponsive to ssh. Nothing on the console power cycled it.
• 02:57 ipp014 left a mess behind. Did force.umount on several hosts. On some it did not work. Removed stare01 and stare02 from stdscience for this reason
• 03:10 destreak is a mess. Many failed_revert runs. This will have to wait until morning
• 03:14 warpPendingSkyCell full of pages in state DONE. reset it
• 03:21 warp is in a bad way. There are over 200 runs to do yet only 50 skycells are showing up in the book warptool -towarped produces the expected output. I'm going to stop and restart stdscience.
• 03:41 stdscience restarted. killed all lingering ppImage, pswarp, and ppSub jobs on the cluster. (There were many stuck in the SOAP recv call)
• 8:12 found a few more nodes that can't access ipp014. did force.umount
• 8:13 warpPendingSkycell full with DONE again. Leaving alone for Gene to possibly debug.

Thursday : 2011-08-25

Friday : 2011-08-26

Saturday : 2011-08-27

Sunday : 2011-08-28