== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : 2011.10.24 === * 08:30 large load spike on ippc18 ~08:20, what caused it? looks like only iqanalysis running. why is it running on ippc18? * 08:50 restarted distribution, set stare01-04 off manually for Cindy to upgrade disks. removing ippc10 from processing for Cindy to use parts in bringing ipp036 back up. also removed from pantasks_hosts.input in case servers restarted. * 10:00 adding "keep" as valid state in pxtools.c that pxUpdateRun can use, like "wait" but with more permanence. * 10:30 working on setting run states to keep/wait for faulted products due to ipp036 problems so not cleaned up. * 17:00 Cindy got ipp036 running again with 1 cpu, set neb-host to repair for now. working though stalled data products and unsetting states back from wait to full. * 20:00 older distribution set not running and not faulted, so ran disttool -revertrun with the specific dist_id for the OSS(2, dist_id=844903,849986), MD09(26, dist_id=847915+), 3PI(9, dist_id=847993+), LAP(30, dist_id=805442+) and pushed the data out. * 22:00 mostly cleaned up, LAP is trickling through with nightly science. still some stuck in destreak holding it up though. Any LAP sets running over this time period likely not full stacks or fields with XY76 missing (i.e., LAP_id of 1367-1385). === Tuesday : 2011-10-25 === * 08:30 Corrupted chip file (1491546650.gpc1:ThreePi.nt:2011:10:25:o5859g0375o.414060:o5859g0375o.414060.ch.331016.XY63.ch.mk.fits) preventing camera stage correct execution. Reprocessing {{{perl runchipimfile.pl --chip_id 331016 --class_id XY63}}} * 09:00 ippdb02 is still ingesting the mysqldump (It's still ingesting the instance table). * 09:45 warp.revert.off (need some time to fix the flaky cam files). {{{perl runcameraexp.pl --cam_id 307180}}} * 10:00 warp.revert.on * 10:30 stare02,03 removed from processing so Cindy can finish+fix disks on those systems * 11:00 Mark: with ipp036 back up, looked at if any of the summit copy downloads were lost when ipp036 burned (literally..) on Oct. 20 HST. the files involved look okay, instance and filesizes are correct. {{{ gpc1/20111021/o5855g0001d/o5855g0001d.ota55.fits gpc1/20111021/o5855g0002d/o5855g0002d.ota55.fits gpc1/20111021/o5855g0003d/o5855g0003d.ota55.fits gpc1/20111021/o5855g0004d/o5855g0004d.ota55.fits gpc1/20111021/o5855g0005d/o5855g0005d.ota55.fits gpc1/20111021/o5855g0006o/o5855g0006o.ota55.fits gpc1/20111021/o5855g0007o/o5855g0007o.ota55.fits gpc1/20111021/o5855g0008o/o5855g0008o.ota55.fits gpc1/20111021/o5855g0009o/o5855g0009o.ota55.fits gpc1/20111021/o5855g0010o/o5855g0010o.ota55.fits gpc1/20111021/o5855g0011o/o5855g0011o.ota55.fits gpc1/20111021/o5855g0012o/o5855g0012o.ota55.fits gpc1/20111021/o5855g0013o/o5855g0013o.ota55.fits gpc1/20111021/o5855g0014o/o5855g0014o.ota55.fits gpc1/20111021/o5855g0015o/o5855g0015o.ota55.fits gpc1/20111021/o5855g0016o/o5855g0016o.ota55.fits gpc1/20111021/o5855g0017o/o5855g0017o.ota55.fits }}} * 15:15 bill noticed that there were a couple of postage stamp requests stuck since the 17th! Not sure why. Restarted the pstamp and update pantasks. === Wednesday : 2011.10.25 === * 00:00 finding many scattered chips failed to update properly into the full state. not sure the best way to repair. * 02:00 Mark: did the early morning registration kick. impatient, didnt want to wait for revert. * 10:00 still working through trying to clean up hangups in LAP. 4 chip failed to update, 3 have both instances on ippb02 and both non-existant. example {{{ whichnode neb://ipp040.0/gpc1/20100517/o5333g0639o/o5333g0639o.ota14.fits ippb02.1 available ippb02.0 available neb-stat --validate neb://ipp040.0/gpc1/20100517/o5333g0639o/o5333g0639o.ota14.fits 0 NON-EXISTANT file:///data/ippb02.1/nebulous/43/f0/963040170.gpc1:20100517:o5333g0639o:o5333g0639o.ota14.fits 0 NON-EXISTANT file:///data/ippb02.0/nebulous/43/f0/963047776.gpc1:20100517:o5333g0639o:o5333g0639o.ota14.fits }}} {{{ chiptool -updateprocessedimfile -set_state full -chip_id 308565 -class_id XY14 -dbname gpc1 chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 308565 -class_id XY14 -dbname gpc1 regtool -updateprocessedimfile -set_state corrupt -class_id XY14 -exp_id 169255 -dbname gpc1 chiptool -updateprocessedimfile -set_state full -chip_id 327433 -class_id XY33 -dbname gpc1 chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 327433 -class_id XY33 -dbname gpc1 regtool -updateprocessedimfile -set_state corrupt -class_id XY33 -exp_id 200311 -dbname gpc1 chiptool -updateprocessedimfile -set_state full -chip_id 309238 -class_id XY26 -dbname gpc1 chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 309238 -class_id XY26 -dbname gpc1 regtool -updateprocessedimfile -set_state corrupt -class_id XY26 -exp_id 250192 -dbname gpc1 }}} * 18:45 restarted standard science * 18:50 dropped exposure in LAP run to finish it up and see if will start a new one cleanly. {{{ laptool -updateexp -lap_id 1367 -exp_id 373259 -set_data_state drop -dbname gpc1 }}} * 20:00 new LAP run lap_id=1407 ran for bit, but updates still being hung up by error_cleaned. * 21:50 registration stalled ~50min, kicking {{{ o5861g0029o XY05 -14 full neb://ipp007.0/gpc1/20111027/o5861g0029o/o5861g0029o.ota05.fits o5861g0029o XY06 -1 check_burntool neb://ipp007.0/gpc1/20111027/o5861g0029o/o5861g0029o.ota06.fits regtool -updateprocessedimfile -exp_id 414834 -class_id XY06 -set_state pending_burntool -dbname gpc1 }}} === Thursday : YYYY.MM.DD === === Friday : YYYY.MM.DD === === Saturday : YYYY.MM.DD === === Sunday : YYYY.MM.DD ===