PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011.10.24

• 08:30 large load spike on ippc18 ~08:20, what caused it? looks like only iqanalysis running. why is it running on ippc18?
• 08:50 restarted distribution, set stare01-04 off manually for Cindy to upgrade disks. removing ippc10 from processing for Cindy to use parts in bringing ipp036 back up. also removed from pantasks_hosts.input in case servers restarted.
• 10:00 adding "keep" as valid state in pxtools.c that pxUpdateRun can use, like "wait" but with more permanence.
• 10:30 working on setting run states to keep/wait for faulted products due to ipp036 problems so not cleaned up.
• 17:00 Cindy got ipp036 running again with 1 cpu, set neb-host to repair for now. working though stalled data products.
• 22:00 mostly cleaned up, LAP is trickling through with nightly science. still some stuck in destreak holding it up though.

Tuesday : 2011-10-25

• 08:30 Corrupted chip file (1491546650.gpc1:ThreePi.nt:2011:10:25:o5859g0375o.414060:o5859g0375o.414060.ch.331016.XY63.ch.mk.fits) preventing camera stage correct execution. Reprocessing perl runchipimfile.pl --chip_id 331016 --class_id XY63
• 09:00 ippdb02 is still ingesting the mysqldump (It's still ingesting the instance table).
• 09:45 warp.revert.off (need some time to fix the flaky cam files). perl runcameraexp.pl --cam_id 307180
• 10:00 warp.revert.on
• 10:30 stare02,03 removed from processing so Cindy can finish+fix disks on those systems

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD