(Up to PS1 IPP Czar Logs)

Monday : 2012.07.16

Bill is czar today.

• 10:10 stdscience was stuck. There are several thousand chip runs outstanding but no jobs are getting queued. Czartool indicates that chip processing stopped making progress around 20:15 last night while outstanding warps got finished just after midnight. The book chipPendingImfile had lots of entries in init state but nothing was getting run. I restarted stdscience.
• 13:40 Chris discovered that ippc03 was out of space in /tmp which was causing nebulous requests to fail. This undoubtedly contributed to the processing failures. Bill deleted the log files and restarted the apache servers.
• 13:55 chip reverts are off because we have a number of exposures that are faulted repeatedly due to incorrect burntool state. Chris is working on fixing the burntool states.
• 15:18 I see that chip reverts have been turned back on.

Tuesday : 2012.07.17

Bill is czar again today

• 10:18 LAP is done through warp except for 4 exposures that have chips with missing burntool tables and one warp process whose perl script got stuck trying to check files on one of the stsci nodes. Killed that process and the warp completed. Also had to rerun chip 493934 XY55 because the output variance image was corrupt.
• 10:45 Used Chris's do_burntool_single.pl script to recreate the missing burntool tables.
• 11:15 ipp008 has some chip processes that have been stuck. Gene and I tried to fix the problem to no avail. Rebooted the system.
• 11:23 restarted stdscience
• 13:23 restored a couple of "lost" raw imfiles with previously unknown copies that Serge was able to find with his new databases. More info later.

Heather is the afternoon czar

Wednesday : 2012.07.18

Mark is czar

• 02:30 fixed ppStack bug, turning LAP stacks back on for another test sample. about 900/3500 likely had this problem in the earlier run of stack.
• 03:10 the 140 extra stack processing is hitting the stsci machines fairly hard, turning off 1x stare+2x compute+1x compute2 (~35) to start with. still causing some large spike, taking out 1x stare+1x compute2+1x wave4 more (~27).
• 09:40 czar rate plot indicates ~300 stacks/hr with ~82 nodes. stsci nodes still getting spiked (nfs) but have handled for past few hrs.
• 11:00 adding back 1x compute2+1x stare+1x wave4 for 103 nodes, stsci node spikes are broader now (time) -- monitoring
• 11:30 ganglia reports ipp008 all memory swapped but isnt. gmond restart worked for a little bit but back to saying all memory swapped. keep eye on.

Thursday : 2012.07.19

Friday : 2012.07.20

Saturday : 2012.07.21

Sunday : 2012.07.22