(Up to PS1 IPP Czar Logs)

Monday : 2012.07.16

Bill is czar today.

• 10:10 stdscience was stuck. There are several thousand chip runs outstanding but no jobs are getting queued. Czartool indicates that chip processing stopped making progress around 20:15 last night while outstanding warps got finished just after midnight. The book chipPendingImfile had lots of entries in init state but nothing was getting run. I restarted stdscience.
• 13:40 Chris discovered that ippc03 was out of space in /tmp which was causing nebulous requests to fail. This undoubtedly contributed to the processing failures. Bill deleted the log files and restarted the apache servers.
• 13:55 chip reverts are off because we have a number of exposures that are faulted repeatedly due to incorrect burntool state. Chris is working on fixing the burntool states.
• 15:18 I see that chip reverts have been turned back on.

Tuesday : 2012.07.17

• 10:18 LAP is done through warp except for 4 exposures that have chips with missing burntool tables and one warp process whose perl script got stuck trying to check files on one of the stsci nodes. Killed that process and the warp completed. Also had to rerun chip 493934 XY55 because the output variance image was corrupt.

Wednesday : 2012.07.18

Thursday : 2012.07.19

Friday : 2012.07.20

Saturday : 2012.07.21

Sunday : 2012.07.22