PS1 IPP Czar Logs for the week 2011-08-22 to 2011-08-28

(Up to PS1 IPP Czar Logs)

Monday : 2011-08-22

• 9:00 Bill shut down publishing, rcserver, and the postage stamp server to recover from out of disk space issue on ipppc17
• 12:20 Bill fixed a camera run with corrupt files (250162) that was causing some warps to fail
• 12:20 Bill fixed a corrupt skycell
• evening Bill queued diffs for STS.201108 19, 20, 21
• raised priority for STS and M31 nightlyscience to 400 to match the other labels.

Tuesday : 2011-08-23

• 00:00 Mark: made change to ppStack PSF matching rejection before new stacks ready and queued MD01.refstacks
• 7:40 Bill: added each of the stare nodes to stdscience twice
• 15:55 CZW: Restart stdscience to jumpstart processing after nebulous issue (still not fully understood).
• 17:50 CZW: Cycling power on ippc06 for the second time today. Removed from stdscience and stack processing.

Wednesday : 2011-08-24

• 02:15 Bill finds many faults on czartool. ipp014 load over 100. Unresponsive to ssh. Nothing on the console power cycled it.
• 02:57 ipp014 left a mess behind. Did force.umount on several hosts. On some it did not work. Removed stare01 and stare02 from stdscience for this reason
• 03:10 destreak is a mess. Many failed_revert runs. This will have to wait until morning
• 03:14 warpPendingSkyCell full of pages in state DONE. reset it
• 03:21 warp is in a bad way. There are over 200 runs to do yet only 50 skycells are showing up in the book warptool -towarped produces the expected output. I'm going to stop and restart stdscience.
• 03:41 stdscience restarted. killed all lingering ppImage, pswarp, and ppSub jobs on the cluster. (There were many stuck in the SOAP recv call)
• 08:12 found a few more nodes that can't access ipp014. did force.umount
• 08:13 warpPendingSkycell full with DONE again. Leaving alone for Gene to possibly debug.
• 09:45 stopped processing. Gavin reset the nfs mounts of ipp014
• 09:55 rebuilt pantasks with a bug fix from Gene that should fix the warp book getting full bug.
• 09:57 started stdscience only. Still getting faults. reverts off
• 10:04 started summit_copy with only gpc1. All files have been copied except for some recent LED exposures some of which are failing. Set pantasks to stop.
• 10:09 started registration. It is registerting and applying burntool successfully. Looks like fixing ipp014 repaired the problem.
• 10:18 getting faults from ipp039 It is not mounting /data/ipp014.0. Set it to off in stdscience.
• 10:30 ipp006 ipp049 ipp053 ippc18, 13, and 14 are also not mounting ipp014. Reported to Gavin. Setting the host to repair
• 11:38 CZW: rebooting offending nodes appears to resolve the problem. I have restarted registration and stdscience, and jobs appear to be completing correctly now.

Thursday : 2011-08-25

Friday : 2011-08-26

Saturday : 2011-08-27

Sunday : 2011-08-28