PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011.10.24

• 08:30 large load spike on ippc18 ~08:20, what caused it? looks like only iqanalysis running. why is it running on ippc18?
• 08:50 restarted distribution, set stare01-04 off manually for Cindy to upgrade disks. removing ippc10 from processing for Cindy to use parts in bringing ipp036 back up. also removed from pantasks_hosts.input in case servers restarted.
• 10:00 adding "keep" as valid state in pxtools.c that pxUpdateRun can use, like "wait" but with more permanence.
• 10:30 working on setting run states to keep/wait for faulted products due to ipp036 problems so not cleaned up.
• 17:00 Cindy got ipp036 running again with 1 cpu, set neb-host to repair for now. working though stalled data products and unsetting states back from wait to full.
• 20:00 older distribution set not running and not faulted, so ran disttool -revertrun with the specific dist_id for the OSS(2, dist_id=844903,849986), MD09(26, dist_id=847915+), 3PI(9, dist_id=847993+), LAP(30, dist_id=805442+) and pushed the data out.
• 22:00 mostly cleaned up, LAP is trickling through with nightly science. still some stuck in destreak holding it up though. Any LAP sets running over this time period likely not full stacks or fields with XY76 missing (i.e., LAP_id of 1367-1385).

Tuesday : 2011-10-25

• 08:30 Corrupted chip file (1491546650.gpc1:ThreePi.nt:2011:10:25:o5859g0375o.414060:o5859g0375o.414060.ch.331016.XY63.ch.mk.fits) preventing camera stage correct execution. Reprocessing perl runchipimfile.pl --chip_id 331016 --class_id XY63
• 09:00 ippdb02 is still ingesting the mysqldump (It's still ingesting the instance table).
• 09:45 warp.revert.off (need some time to fix the flaky cam files). perl runcameraexp.pl --cam_id 307180
• 10:00 warp.revert.on
• 10:30 stare02,03 removed from processing so Cindy can finish+fix disks on those systems
• 11:00 Mark: with ipp036 back up, looked at if any of the summit copy downloads were lost when ipp036 burned (literally..) on Oct. 20 HST. the files involved look okay, instance and filesizes are correct.

  gpc1/20111021/o5855g0001d/o5855g0001d.ota55.fits
  gpc1/20111021/o5855g0002d/o5855g0002d.ota55.fits
  gpc1/20111021/o5855g0003d/o5855g0003d.ota55.fits
  gpc1/20111021/o5855g0004d/o5855g0004d.ota55.fits
  gpc1/20111021/o5855g0005d/o5855g0005d.ota55.fits
  gpc1/20111021/o5855g0006o/o5855g0006o.ota55.fits
  gpc1/20111021/o5855g0007o/o5855g0007o.ota55.fits
  gpc1/20111021/o5855g0008o/o5855g0008o.ota55.fits
  gpc1/20111021/o5855g0009o/o5855g0009o.ota55.fits
  gpc1/20111021/o5855g0010o/o5855g0010o.ota55.fits
  gpc1/20111021/o5855g0011o/o5855g0011o.ota55.fits
  gpc1/20111021/o5855g0012o/o5855g0012o.ota55.fits
  gpc1/20111021/o5855g0013o/o5855g0013o.ota55.fits
  gpc1/20111021/o5855g0014o/o5855g0014o.ota55.fits
  gpc1/20111021/o5855g0015o/o5855g0015o.ota55.fits
  gpc1/20111021/o5855g0016o/o5855g0016o.ota55.fits
  gpc1/20111021/o5855g0017o/o5855g0017o.ota55.fits
  

• 15:15 bill noticed that there were a couple of postage stamp requests stuck since the 17th! Not sure why. Restarted the pstamp and update pantasks.

Wednesday : 2011.10.25

• 02:00 did the early morning registration kick.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD