| Version 45 (modified by , 15 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2011-10-24 - 2011-10-30
(Up to PS1 IPP Czar Logs)
Monday : 2011.10.24
- 08:30 large load spike on ippc18 ~08:20, what caused it? looks like only iqanalysis running. why is it running on ippc18?
- 08:50 restarted distribution, set stare01-04 off manually for Cindy to upgrade disks. removing ippc10 from processing for Cindy to use parts in bringing ipp036 back up. also removed from pantasks_hosts.input in case servers restarted.
- 10:00 adding "keep" as valid state in pxtools.c that pxUpdateRun can use, like "wait" but with more permanence.
- 10:30 working on setting run states to keep/wait for faulted products due to ipp036 problems so not cleaned up.
- 17:00 Cindy got ipp036 running again with 1 cpu, set neb-host to repair for now. working though stalled data products and unsetting states back from wait to full.
- 20:00 older distribution set not running and not faulted, so ran disttool -revertrun with the specific dist_id for the OSS(2, dist_id=844903,849986), MD09(26, dist_id=847915+), 3PI(9, dist_id=847993+), LAP(30, dist_id=805442+) and pushed the data out.
- 22:00 mostly cleaned up, LAP is trickling through with nightly science. still some stuck in destreak holding it up though. Any LAP sets running over this time period likely not full stacks or fields with XY76 missing (i.e., LAP_id of 1367-1385).
Tuesday : 2011-10-25
- 08:30 Corrupted chip file (1491546650.gpc1:ThreePi.nt:2011:10:25:o5859g0375o.414060:o5859g0375o.414060.ch.331016.XY63.ch.mk.fits) preventing camera stage correct execution. Reprocessing
perl runchipimfile.pl --chip_id 331016 --class_id XY63 - 09:00 ippdb02 is still ingesting the mysqldump (It's still ingesting the instance table).
- 09:45 warp.revert.off (need some time to fix the flaky cam files).
perl runcameraexp.pl --cam_id 307180 - 10:00 warp.revert.on
- 10:30 stare02,03 removed from processing so Cindy can finish+fix disks on those systems
- 11:00 Mark: with ipp036 back up, looked at if any of the summit copy downloads were lost when ipp036 burned (literally..) on Oct. 20 HST. the files involved look okay, instance and filesizes are correct.
gpc1/20111021/o5855g0001d/o5855g0001d.ota55.fits gpc1/20111021/o5855g0002d/o5855g0002d.ota55.fits gpc1/20111021/o5855g0003d/o5855g0003d.ota55.fits gpc1/20111021/o5855g0004d/o5855g0004d.ota55.fits gpc1/20111021/o5855g0005d/o5855g0005d.ota55.fits gpc1/20111021/o5855g0006o/o5855g0006o.ota55.fits gpc1/20111021/o5855g0007o/o5855g0007o.ota55.fits gpc1/20111021/o5855g0008o/o5855g0008o.ota55.fits gpc1/20111021/o5855g0009o/o5855g0009o.ota55.fits gpc1/20111021/o5855g0010o/o5855g0010o.ota55.fits gpc1/20111021/o5855g0011o/o5855g0011o.ota55.fits gpc1/20111021/o5855g0012o/o5855g0012o.ota55.fits gpc1/20111021/o5855g0013o/o5855g0013o.ota55.fits gpc1/20111021/o5855g0014o/o5855g0014o.ota55.fits gpc1/20111021/o5855g0015o/o5855g0015o.ota55.fits gpc1/20111021/o5855g0016o/o5855g0016o.ota55.fits gpc1/20111021/o5855g0017o/o5855g0017o.ota55.fits
- 15:15 bill noticed that there were a couple of postage stamp requests stuck since the 17th! Not sure why. Restarted the pstamp and update pantasks.
Wednesday : 2011.10.26
- 00:00 finding many scattered chips failed to update properly into the full state. not sure the best way to repair.
- 02:00 Mark: did the early morning registration kick. impatient, didnt want to wait for revert.
- 10:00 still working through trying to clean up hangups in LAP. 4 chip failed to update, 3 have both instances on ippb02 and both non-existant. example
whichnode neb://ipp040.0/gpc1/20100517/o5333g0639o/o5333g0639o.ota14.fits ippb02.1 available ippb02.0 available neb-stat --validate neb://ipp040.0/gpc1/20100517/o5333g0639o/o5333g0639o.ota14.fits 0 NON-EXISTANT file:///data/ippb02.1/nebulous/43/f0/963040170.gpc1:20100517:o5333g0639o:o5333g0639o.ota14.fits 0 NON-EXISTANT file:///data/ippb02.0/nebulous/43/f0/963047776.gpc1:20100517:o5333g0639o:o5333g0639o.ota14.fitschiptool -updateprocessedimfile -set_state full -chip_id 308565 -class_id XY14 -dbname gpc1 chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 308565 -class_id XY14 -dbname gpc1 regtool -updateprocessedimfile -set_state corrupt -class_id XY14 -exp_id 169255 -dbname gpc1 chiptool -updateprocessedimfile -set_state full -chip_id 327433 -class_id XY33 -dbname gpc1 chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 327433 -class_id XY33 -dbname gpc1 regtool -updateprocessedimfile -set_state corrupt -class_id XY33 -exp_id 200311 -dbname gpc1 chiptool -updateprocessedimfile -set_state full -chip_id 309238 -class_id XY26 -dbname gpc1 chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 309238 -class_id XY26 -dbname gpc1 regtool -updateprocessedimfile -set_state corrupt -class_id XY26 -exp_id 250192 -dbname gpc1
- 18:45 restarted standard science
- 18:50 dropped exposure in LAP run to finish it up and see if will start a new one cleanly.
laptool -updateexp -lap_id 1367 -exp_id 373259 -set_data_state drop -dbname gpc1
- 20:00 new LAP run lap_id=1407 ran for bit, but updates still being hung up by error_cleaned.
- 21:50 registration stalled ~50min, kicking
o5861g0029o XY05 -14 full neb://ipp007.0/gpc1/20111027/o5861g0029o/o5861g0029o.ota05.fits o5861g0029o XY06 -1 check_burntool neb://ipp007.0/gpc1/20111027/o5861g0029o/o5861g0029o.ota06.fits regtool -updateprocessedimfile -exp_id 414834 -class_id XY06 -set_state pending_burntool -dbname gpc1
Thursday : 2011.10.27
- 11:00 Bill track down some of the issues hanging up LAP (need to add more details), warps clearning and >1500 stacks running.. stealing stare nodes for stack pantasks.
- 12:00 Mark: dumping as many nodes into stack as is stable.. down to 1200 stacks now. stopping stdscience so new LAP chips not get in way, yet.
- 13:30 down to ~500 stacks, restarting stdscience, stack, distribution for normal processing again.
- 13:45 Serge: restarted crashed mysql server on ipp001
- 13:50 Serge: ipp026 removed from processing
- 14:30 Mark: while LAP moving on to processing a few new fields, found a few more error_cleaned. cleared with
chiptool -chip_id 327485 -class_id XY57 -updateprocessedimfile -set_state cleaned -dbname gpc1 chiptool -setimfiletoupdate -chip_id 327485 -class_id XY57 -set_label LAP.ThreePi.20110809 -dbname gpc1 chiptool -chip_id 327487 -class_id XY61 -updateprocessedimfile -set_state cleaned -dbname gpc1 chiptool -setimfiletoupdate -chip_id 327487 -class_id XY61 -set_label LAP.ThreePi.20110809 -dbname gpc1 chiptool -chip_id 327493 -class_id XY62 -updateprocessedimfile -set_state cleaned -dbname gpc1 chiptool -setimfiletoupdate -chip_id 327493 -class_id XY62 -set_label LAP.ThreePi.20110809 -dbname gpc1 chiptool -chip_id 327496 -class_id XY46 -updateprocessedimfile -set_state cleaned -dbname gpc1 chiptool -setimfiletoupdate -chip_id 327496 -class_id XY46 -set_label LAP.ThreePi.20110809 -dbname gpc1
- 15:20 Mark: stdscience barely loading jobs. restarting fixed.
- 17:30 all old/stalled LAP runs cleared, moving along again and with >90% as chip/warp updates finishing up the overlap area.
- 22:00 LAP running into missing OTA again with only copies on ippb02 non-existant...
Friday : 2011-10-28
Bill is czar today
- 07:00 Only 50ish exposures taken last night. Nightly processing is complete.
- 11:20 Fixed a number of stuck LAP exposures due to missing raw files or bad instances (the automatic fix it code in chip_imfile.pl didn't always work)
Fixes (set_ignored is regtool -updateprocessedimfile -set_ignored -exp_id <exp_id> -class_id <class_id> set_ignored 207007 XY45 set_ignored 191847 XY57 set_ignored 181607 XY34 neb-cull neb://ipp040.0/gpc1/20100615/o5362g0330o/o5362g0330o.ota14.fits --volume ippb02.2 neb-replicate neb://ipp040.0/gpc1/20100615/o5362g0330o/o5362g0330o.ota14.fits --volume ipp040.0 set_ignored 191842 XY15 neb-cull neb://ipp030.0/gpc1/20100615/o5362g0327o/o5362g0327o.ota37.fits --volume ippb02.1 neb-replicate --volume ipp030.0 neb://ipp030.0/gpc1/20100615/o5362g0327o/o5362g0327o.ota37.fits
- 13:00 ingestion of nebulous on ippdb02: instance table ingestion finished a while ago. Now ingesting storage_object table. For info, the size of instance.ibd on ippdb00 is 354GB on ippdb00 while it's 283 GB on ippdb02).
Saturday : 2011-10-29
- 00:00 ippc06 not responding, processing stalled.. restarting ippc06, nothing on console, rebooted okay.
- 01:00 processing back up
- 01:30 registration needed restarting.
Sunday : 2011-10-30
Note:
See TracWiki
for help on using the wiki.
