== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.10.24 ===
 * 08:30 large load spike on ippc18 ~08:20, what caused it? looks like only iqanalysis running. why is it running on ippc18?
 * 08:50 restarted distribution, set stare01-04 off manually for Cindy to upgrade disks. removing ippc10 from processing for Cindy to use parts in bringing ipp036 back up. also removed from pantasks_hosts.input in case servers restarted.
 * 10:00 adding "keep" as valid state in pxtools.c that pxUpdateRun can use, like "wait" but with more permanence.
 * 10:30 working on setting run states to keep/wait for faulted products due to ipp036 problems so not cleaned up.
 * 17:00 Cindy got ipp036 running again with 1 cpu, set neb-host to repair for now. working though stalled data products.
 * 22:00 mostly cleaned up, LAP is trickling through with nightly science. still some stuck in destreak holding it up though.
=== Tuesday : 2011-10-25 ===
 * 08:30 Corrupted chip file (1491546650.gpc1:ThreePi.nt:2011:10:25:o5859g0375o.414060:o5859g0375o.414060.ch.331016.XY63.ch.mk.fits) preventing camera stage correct execution. Reprocessing {{{perl runchipimfile.pl --chip_id 331016 --class_id XY63}}}
 * 09:00 ippdb02 is still ingesting the mysqldump (It's still ingesting the instance table).
 * 09:45 warp.revert.off (need some time to fix the flaky cam files). {{{perl runcameraexp.pl --cam_id 307180}}}
 * 10:00 warp.revert.on
 * 10:30 stare02,03 removed from processing so Cindy can finish+fix disks on those systems
 * 11:00 Mark: with ipp036 back up, looked at if any of the summit copy downloads were lost when ipp036 burned (literally..). the files involved look okay, instance and filesizes are correct.
{{{
gpc1/20111021/o5855g0001d/o5855g0001d.ota55.fits
gpc1/20111021/o5855g0002d/o5855g0002d.ota55.fits
gpc1/20111021/o5855g0003d/o5855g0003d.ota55.fits
gpc1/20111021/o5855g0004d/o5855g0004d.ota55.fits
gpc1/20111021/o5855g0005d/o5855g0005d.ota55.fits
gpc1/20111021/o5855g0006o/o5855g0006o.ota55.fits
gpc1/20111021/o5855g0007o/o5855g0007o.ota55.fits
gpc1/20111021/o5855g0008o/o5855g0008o.ota55.fits
gpc1/20111021/o5855g0009o/o5855g0009o.ota55.fits
gpc1/20111021/o5855g0010o/o5855g0010o.ota55.fits
gpc1/20111021/o5855g0011o/o5855g0011o.ota55.fits
gpc1/20111021/o5855g0012o/o5855g0012o.ota55.fits
gpc1/20111021/o5855g0013o/o5855g0013o.ota55.fits
gpc1/20111021/o5855g0014o/o5855g0014o.ota55.fits
gpc1/20111021/o5855g0015o/o5855g0015o.ota55.fits
gpc1/20111021/o5855g0016o/o5855g0016o.ota55.fits
gpc1/20111021/o5855g0017o/o5855g0017o.ota55.fits
}}}
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===