(Up to PS1 IPP Czar Logs)

Monday : 2012-09-17

Bill is czar today.

• 09:00 LAP processing is stuck due to a diff cleanup issue Chris will repair.
• 09:56 stopping processing in order to restart all pantasks setting turning ipp037 on as a compute node.
• 10:30 LAP is moving along nicely now
• 10:38 repaired a number of missing files (found with deneb-locate). Two were actually lost for good: o5364g0119o.ota65.fits and o5361g0129o.ota65.fits
• 13:11 ipp020 has lost communication with the virtual /data/ipp037.0 (stsci00) shut it down then power cycled after the usual hang in "unmount nfs file systems" Because I'm lazy today I'm removing the host from processing lists.
• 15:07 setting lap.off for a bit so we can debug the state of the confused state czartool's chip data without the counts changing out from under us.
• 15:40 ipp028 has gone out to lunch. Lots of nfs errors in dmesg. Something about general protection fault as well but that did not show up in the console log. Power cycled it.
• 15:55 turned lap.on but left lap.cleanup.off for a little while longer
• 17:00 ipp028 is not visible in nebulous but it seems to be up. Setting it to repair state for now.
• 17:14 Ok I'm convinced that czartool is just getting confused by the goto_cleaned rate. Turned lap.cleanup.on

Tuesday : 2012-09-18

• 06:40 Processing rate dropped through the night to ~60 per hour from 100 per hour. Restarted stdscience. ipp020 are in host lists but in repair mode for nebulous until the sun is higher in the sky.

Wednesday : 2012-09-19

Thursday : 2012-09-20

Friday : 2012-09-21

Saturday : 2012-09-22

Sunday : 2012-09-23