PS1 IPP Czar Logs for the week 2011-10-24 - 2011-10-30

(Up to PS1 IPP Czar Logs)

Monday : 2011.10.24

  • 08:30 large load spike on ippc18 ~08:20, what caused it? looks like only iqanalysis running. why is it running on ippc18?
  • 08:50 restarted distribution, set stare01-04 off manually for Cindy to upgrade disks. removing ippc10 from processing for Cindy to use parts in bringing ipp036 back up. also removed from pantasks_hosts.input in case servers restarted.
  • 10:00 adding "keep" as valid state in pxtools.c that pxUpdateRun can use, like "wait" but with more permanence.
  • 10:30 working on setting run states to keep/wait for faulted products due to ipp036 problems so not cleaned up.
  • 17:00 Cindy got ipp036 running again with 1 cpu, set neb-host to repair for now. working though stalled data products and unsetting states back from wait to full.
  • 20:00 older distribution set not running and not faulted, so ran disttool -revertrun with the specific dist_id for the OSS(2, dist_id=844903,849986), MD09(26, dist_id=847915+), 3PI(9, dist_id=847993+), LAP(30, dist_id=805442+) and pushed the data out.
  • 22:00 mostly cleaned up, LAP is trickling through with nightly science. still some stuck in destreak holding it up though. Any LAP sets running over this time period likely not full stacks or fields with XY76 missing (i.e., LAP_id of 1367-1385).

Tuesday : 2011-10-25

  • 08:30 Corrupted chip file (1491546650.gpc1:ThreePi.nt:2011:10:25:o5859g0375o.414060:o5859g0375o.414060.ch.331016.XY63.ch.mk.fits) preventing camera stage correct execution. Reprocessing perl runchipimfile.pl --chip_id 331016 --class_id XY63
  • 09:00 ippdb02 is still ingesting the mysqldump (It's still ingesting the instance table).
  • 09:45 warp.revert.off (need some time to fix the flaky cam files). perl runcameraexp.pl --cam_id 307180
  • 10:00 warp.revert.on
  • 10:30 stare02,03 removed from processing so Cindy can finish+fix disks on those systems
  • 11:00 Mark: with ipp036 back up, looked at if any of the summit copy downloads were lost when ipp036 burned (literally..) on Oct. 20 HST. the files involved look okay, instance and filesizes are correct.
    gpc1/20111021/o5855g0001d/o5855g0001d.ota55.fits
    gpc1/20111021/o5855g0002d/o5855g0002d.ota55.fits
    gpc1/20111021/o5855g0003d/o5855g0003d.ota55.fits
    gpc1/20111021/o5855g0004d/o5855g0004d.ota55.fits
    gpc1/20111021/o5855g0005d/o5855g0005d.ota55.fits
    gpc1/20111021/o5855g0006o/o5855g0006o.ota55.fits
    gpc1/20111021/o5855g0007o/o5855g0007o.ota55.fits
    gpc1/20111021/o5855g0008o/o5855g0008o.ota55.fits
    gpc1/20111021/o5855g0009o/o5855g0009o.ota55.fits
    gpc1/20111021/o5855g0010o/o5855g0010o.ota55.fits
    gpc1/20111021/o5855g0011o/o5855g0011o.ota55.fits
    gpc1/20111021/o5855g0012o/o5855g0012o.ota55.fits
    gpc1/20111021/o5855g0013o/o5855g0013o.ota55.fits
    gpc1/20111021/o5855g0014o/o5855g0014o.ota55.fits
    gpc1/20111021/o5855g0015o/o5855g0015o.ota55.fits
    gpc1/20111021/o5855g0016o/o5855g0016o.ota55.fits
    gpc1/20111021/o5855g0017o/o5855g0017o.ota55.fits
    
  • 15:15 bill noticed that there were a couple of postage stamp requests stuck since the 17th! Not sure why. Restarted the pstamp and update pantasks.

Wednesday : 2011.10.26

  • 00:00 finding many scattered chips failed to update properly into the full state. not sure the best way to repair.
  • 02:00 Mark: did the early morning registration kick. impatient, didnt want to wait for revert.
  • 10:00 still working through trying to clean up hangups in LAP. 4 chip failed to update, 3 have both instances on ippb02 and both non-existant. example
    whichnode neb://ipp040.0/gpc1/20100517/o5333g0639o/o5333g0639o.ota14.fits
    ippb02.1 available
    ippb02.0 available
    
    neb-stat --validate neb://ipp040.0/gpc1/20100517/o5333g0639o/o5333g0639o.ota14.fits
          0                     NON-EXISTANT file:///data/ippb02.1/nebulous/43/f0/963040170.gpc1:20100517:o5333g0639o:o5333g0639o.ota14.fits
          0                     NON-EXISTANT file:///data/ippb02.0/nebulous/43/f0/963047776.gpc1:20100517:o5333g0639o:o5333g0639o.ota14.fits
    
     chiptool -updateprocessedimfile -set_state full -chip_id 308565 -class_id XY14 -dbname gpc1
     chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 308565 -class_id XY14 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY14 -exp_id 169255 -dbname gpc1
    
     chiptool -updateprocessedimfile -set_state full -chip_id 327433 -class_id XY33 -dbname gpc1
     chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 327433 -class_id XY33 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY33 -exp_id 200311 -dbname gpc1
    
     chiptool -updateprocessedimfile -set_state full -chip_id 309238 -class_id XY26 -dbname gpc1
     chiptool -updateprocessedimfile -fault 0 -set_quality 42 -chip_id 309238 -class_id XY26 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY26 -exp_id 250192 -dbname gpc1
    
    
  • 18:45 restarted standard science
  • 18:50 dropped exposure in LAP run to finish it up and see if will start a new one cleanly.
    laptool -updateexp -lap_id 1367 -exp_id 373259 -set_data_state drop -dbname gpc1
    
  • 20:00 new LAP run lap_id=1407 ran for bit, but updates still being hung up by error_cleaned.
  • 21:50 registration stalled ~50min, kicking
    o5861g0029o  XY05 -14 full neb://ipp007.0/gpc1/20111027/o5861g0029o/o5861g0029o.ota05.fits
    o5861g0029o  XY06 -1 check_burntool neb://ipp007.0/gpc1/20111027/o5861g0029o/o5861g0029o.ota06.fits
    
    regtool -updateprocessedimfile -exp_id 414834 -class_id XY06 -set_state pending_burntool -dbname gpc1
    

Thursday : 2011.10.27

  • 11:00 Bill track down some of the issues hanging up LAP (need to add more details), warps clearning and >1500 stacks running.. stealing stare nodes for stack pantasks.
  • 12:00 Mark: dumping as many nodes into stack as is stable.. down to 1200 stacks now. stopping stdscience so new LAP chips not get in way, yet.
  • 13:30 down to ~500 stacks, restarting stdscience, stack, distribution for normal processing again.
  • 13:45 Serge: restarted crashed mysql server on ipp001
  • 13:50 Serge: ipp026 removed from processing
  • 14:30 Mark: while LAP moving on to processing a few new fields, found a few more error_cleaned. cleared with
    chiptool -chip_id 327485 -class_id XY57 -updateprocessedimfile -set_state cleaned -dbname gpc1
    chiptool -setimfiletoupdate -chip_id 327485 -class_id XY57 -set_label LAP.ThreePi.20110809 -dbname gpc1
    
    chiptool -chip_id 327487 -class_id XY61 -updateprocessedimfile -set_state cleaned -dbname gpc1
    chiptool -setimfiletoupdate -chip_id 327487 -class_id XY61 -set_label LAP.ThreePi.20110809 -dbname gpc1
    
    chiptool -chip_id 327493 -class_id XY62 -updateprocessedimfile -set_state cleaned -dbname gpc1
    chiptool -setimfiletoupdate -chip_id 327493 -class_id XY62 -set_label LAP.ThreePi.20110809 -dbname gpc1
    
    chiptool -chip_id 327496 -class_id XY46 -updateprocessedimfile -set_state cleaned -dbname gpc1
    chiptool -setimfiletoupdate -chip_id 327496 -class_id XY46 -set_label LAP.ThreePi.20110809 -dbname gpc1
    
  • 15:20 Mark: stdscience barely loading jobs. restarting fixed.
  • 17:30 all old/stalled LAP runs cleared, moving along again and with >90% as chip/warp updates finishing up the overlap area.
  • 22:00 LAP running into missing OTA again with only copies on ippb02 non-existant...

Friday : 2011-10-28

Bill is czar today

  • 07:00 Only 50ish exposures taken last night. Nightly processing is complete.
  • 11:20 Fixed a number of stuck LAP exposures due to missing raw files or bad instances (the automatic fix it code in chip_imfile.pl didn't always work)
Fixes (set_ignored is regtool -updateprocessedimfile -set_ignored -exp_id <exp_id> -class_id <class_id>
set_ignored 207007 XY45
set_ignored 191847 XY57
set_ignored 181607 XY34
neb-cull neb://ipp040.0/gpc1/20100615/o5362g0330o/o5362g0330o.ota14.fits --volume ippb02.2
neb-replicate neb://ipp040.0/gpc1/20100615/o5362g0330o/o5362g0330o.ota14.fits --volume ipp040.0
set_ignored 191842 XY15
neb-cull neb://ipp030.0/gpc1/20100615/o5362g0327o/o5362g0327o.ota37.fits --volume ippb02.1
neb-replicate --volume ipp030.0 neb://ipp030.0/gpc1/20100615/o5362g0327o/o5362g0327o.ota37.fits

  • 13:00 ingestion of nebulous on ippdb02: instance table ingestion finished a while ago. Now ingesting storage_object table. For info, the size of instance.ibd on ippdb00 is 354GB on ippdb00 while it's 283 GB on ippdb02).

Saturday : 2011-10-29

  • 00:00 ippc06 not responding, processing stalled.. restarting ippc06, nothing on console, rebooted okay.
  • 01:00 processing back up
  • 01:30 registration needed restarting. while stalled also restarted distribution
  • 02:00 registration advanced and stalled several times
    CheckStatus: o5863g0278o neb://ipp021.0/gpc1/20111029/o5863g0278o/o5863g0278o.ota33.fits 2011-10-29T09:34:40.000000 60 0 stop run  full 415521 OBJECT NULL   -14 XY33 ota33 1 1
    
    o5863g0278o  XY03 -1 check_burntool neb://ipp006.0/gpc1/20111029/o5863g0278o/o5863g0278o.ota03.fits
    o5863g0278o  XY06 -1 check_burntool neb://ipp007.0/gpc1/20111029/o5863g0278o/o5863g0278o.ota06.fits
    o5863g0278o  XY10 -1 check_burntool neb://ipp008.0/gpc1/20111029/o5863g0278o/o5863g0278o.ota10.fits
    o5863g0278o  XY12 -1 check_burntool neb://ipp009.0/gpc1/20111029/o5863g0278o/o5863g0278o.ota12.fits
    o5863g0278o  XY54 -1 check_burntool neb://ipp035.0/gpc1/20111029/o5863g0278o/o5863g0278o.ota54.fits
    o5863g0278o  XY67 -1 check_burntool neb://ipp047.0/gpc1/20111029/o5863g0278o/o5863g0278o.ota67.fits
    o5863g0278o  XY76 -1 check_burntool neb://ipp053.0/gpc1/20111029/o5863g0278o/o5863g0278o.ota76.fits
    
    regtool -updateprocessedimfile -exp_id 415521  -class_id XY03 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 415521  -class_id XY06 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 415521  -class_id XY10 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 415521  -class_id XY12 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 415521  -class_id XY54 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 415521  -class_id XY67 -set_state pending_burntool -dbname gpc1
    regtool -updateprocessedimfile -exp_id 415521  -class_id XY76 -set_state pending_burntool -dbname gpc1
    ... and more ...
    
  • 10:00 stdscience not fully loading jobs. restarting.
  • 18:00 running stare_nodes.sh off and adding to hosts_ignore_stare in ~ipp/ippconfig/pantasks_hosts.input to remove stare nodes from processing for the stare night tonight. need to look into what is holding up LAP processing so can be running during the night...
  • 18:30 clean up some of nightly science hanging around
    -- M31 stack 
     failed to read /data/ipp048.0/nebulous/d3/6c/1505104359.gpc1:M31.nt:2011:10:29:o5863g0178o.415421:o5863g0178o.415421.wrp.295025.skycell.064.mask.fits
    neb://ipp048.0/gpc1/M31.nt/2011/10/29//o5863g0178o.415421/o5863g0178o.415421.wrp.295025.skycell.064.mask.fits 
    -- M31 - diff 
     failed to read /data/ipp048.0/nebulous/d3/6c/1505104359.gpc1:M31.nt:2011:10:29:o5863g0178o.415421:o5863g0178o.415421.wrp.295025.skycell.064.mask.fits
    
    perl ~ipp/src/ipp-20110622/tools/runwarpskycell.pl --warp_id 295025 --skycell_id skycell.064 --redirect-output 
    
    -- MD09 destreak 43 faults, ran several times and slowly cleared
    magicdstool -clearstatefaults -dbname gpc1 -label MD09.nightlyscience -set_state new -state failed_revert
    
    -- MD09 magick fault
     Error reading difference image to be processed, /data/ipp014.0/nebulous/06/28/1505037057.gpc1:MD09.nightlyscience:2011:10:28:MD09.V2:skycell.042:MD09.V2.skycell.042.WS.dif.184693.fits
    
    perl ~ipp/src/ipp-20110622/tools/rundiffskycell.pl --redirect-output --diff_id 184693 --skycell_id skycell.042
    
    -- MD04.deeptest.20111026 had fault
     neb://@HOST@.0/gpc1/condor_MD04.V3_01.haf/o5662g0155o.322086/o5662g0155o.322086.wrp.280030.skycell.072.fits does not exist
    
    perl ~ipp/src/ipp-20110622/tools/runwarpskycell.pl --warp_id 280030 --skycell_id skycell.072 --redirect-output 
    
  • 19:00 on to LAP
    --> diff - need to rerun warp
     failed to read /data/ipp031.0/nebulous/46/37/1502802626.gpc1:LAP.ThreePi.20110809:2011:10:28:o5810g0325o.387937:o5810g0325o.387937.wrp.294625.skycell.2642.051.mask.fits
    --> stack
     failed to read /data/ipp031.0/nebulous/46/37/1502802626.gpc1:LAP.ThreePi.20110809:2011:10:28:o5810g0325o.387937:o5810g0325o.387937.wrp.294625.skycell.2642.051.mask.fits
    
    perl ~ipp/src/ipp-20110622/tools/runwarpskycell.pl --warp_id 294625 --skycell_id skycell.2642.051 --redirect-output
    
    --> warps
     failed to read /data/ipp008.0/nebulous/b0/54/1509250604.gpc1:LAP.ThreePi.20110809:2011:10:29:o5791g0421o.378517:o5791g0421o.378517.cm.310935.XY02.mk.fits
    
    perl ~ipp/src/ipp-20110622/tools/runcameraexp.pl --redirect-output --cam_id 310935
    
  • 20:00 LAP chips stuck as non-existant instances both on ippb02
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334157 -class_id XY41 -dbname gpc1
     regtool -updateprocessedimfile -set_ignored -exp_id 209754 -class_id XY41 -dbname gpc1
    --> is no set_ignored option? maybe not compiled for ipp-20110622? -- moves forward w/ previous method
    
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 333820 -class_id XY31 -dbname gpc1
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 333888 -class_id XY33 -dbname gpc1
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 333929 -class_id XY05 -dbname gpc1
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334024 -class_id XY55 -dbname gpc1
    
     regtool -updateprocessedimfile -set_state corrupt -class_id XY41 -exp_id 209754 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY55 -exp_id 188604 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY05 -exp_id 172185 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY33 -exp_id 207416 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY31 -exp_id 209722 -dbname gpc1
    
  • 20:30 more LAP chips
    -- auto-fix fault -- Unable to attempt repair: neb://ipp034.0/gpc1/20100604/o5351g0577o/o5351g0577o.ota57.fits, neb-stat also reports 3 instances but only 2 listed.. looks like newly create file was put onto ippb02 for some reason (fault repair code?)
    -rw-rw-r-- 1 ipp 18284544 Oct 29 11:21 /data/ippb02.0/nebulous/db/32/951205820.gpc1:20100604:o5351g0577o:o5351g0577o.ota57.fits
    -rw-rw-r-- 1 ipp 24096960 Oct  8 04:51 /data/ippb02.2/nebulous/db/32/951208155.gpc1:20100604:o5351g0577o:o5351g0577o.ota57.fits
    
    -- will need to be re-ordered? 
    neb-cull neb://ipp034.0/gpc1/20100604/o5351g0577o/o5351g0577o.ota57.fits --volume ippb02.0
    neb-replicate neb://ipp034.0/gpc1/20100604/o5351g0577o/o5351g0577o.ota57.fits --volume ipp034.0
    
    -- problem neb://ipp025.0/gpc1/20100629/o5376g0429o/o5376g0429o.ota76.fits
          0                     NON-EXISTANT file:///data/ipp053.0/nebulous/b3/f6/344255086.gpc1:20100629:o5376g0429o:o5376g0429o.ota76.fits
          1 c18f618e4eceb914e5f8f5a9d1fe972c file:///data/ippb02.0/nebulous/b3/f6/1157759838.gpc1:20100629:o5376g0429o:o5376g0429o.ota76.fits
    
    cp /data/ippb02.0/nebulous/b3/f6/1157759838.gpc1:20100629:o5376g0429o:o5376g0429o.ota76.fits /data/ipp053.0/nebulous/b3/f6/344255086.gpc1:20100629:o5376g0429o:o5376g0429o.ota76.fits
    
    -- problem neb://ipp013.0/gpc1/20100806/o5414g0236o/o5414g0236o.ota22.fits
          0                     NON-EXISTANT file:///data/ippb02.2/nebulous/5d/e6/977954790.gpc1:20100806:o5414g0236o:o5414g0236o.ota22.fits
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ippb02.0/nebulous/5d/e6/977963164.gpc1:20100806:o5414g0236o:o5414g0236o.ota22.fits
    neb-cull neb://ipp013.0/gpc1/20100806/o5414g0236o/o5414g0236o.ota22.fits --volume ippb02.2
    neb-replicate neb://ipp013.0/gpc1/20100806/o5414g0236o/o5414g0236o.ota22.fits --volume ipp013.0
    --> turned out file was corrupted on ippb02.0 anyways.. 
    chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334302 -class_id XY22 -dbname gpc1
    regtool -updateprocessedimfile -set_state corrupt -class_id XY22 -exp_id 203623 -dbname gpc1
    
    
  • 21:30 more LAP chips
    -- Couldn't find burntool table: neb://ipp036.0/gpc1/20100917/o5456g0083o/o5456g0083o.ota67.burn.tbl
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp012.0/nebulous/9a/90/469255506.gpc1:20100917:o5456g0083o:o5456g0083o.ota67.burn.tbl
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ippb01.1/nebulous/9a/90/914206096.gpc1:20100917:o5456g0083o:o5456g0083o.ota67.burn.tbl
    --> 0 sized files.. setting quality 42 to move on
    
    chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 333311 -class_id XY67 -dbname gpc1
    
    -- Couldn't find burntool table: neb://ipp036.0/gpc1/20100917/o5456g0089o/o5456g0089o.ota67.burn.tbl
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp012.0/nebulous/61/f0/469255979.gpc1:20100917:o5456g0089o:o5456g0089o.ota67.burn.tbl
          1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp014.0/nebulous/61/f0/793789385.gpc1:20100917:o5456g0089o:o5456g0089o.ota67.burn.tbl
    --> 0 sized files -- again
    
    chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 333686 -class_id XY67 -dbname gpc1
    
    -- 
    Culling peaks from footprints using the smoothed image
    big footprint: 4422.000000 612.000000 to 4674.000000 1200.000000 (51415 pix)
    big footprint: 3683.000000 3812.000000 to 4053.000000 4222.000000 (74372 pix)
    Assertion failed in function pmFootprintCullPeaks at pmFootprintCullPeaks.c:120. Error stack:
    upper limit does not include max flux
    Backtrace depth: 12
    Backtrace 0: p_psAssert
    Backtrace 1: pmFootprintCullPeaks
    Backtrace 2: psphotCullPeaks
    Backtrace 3: psphotFindFootprints
    Backtrace 4: psphotFindDetectionsReadout
    Backtrace 5: psphotFindDetections
    Backtrace 6: psphotReadout
    Backtrace 7: (unknown)
    Backtrace 8: (unknown)
    Backtrace 9: (unknown)
    Backtrace 10: __libc_start_main
    Backtrace 11: (unknown)
    --> unknown error - set quality 42 and move on for now
    
    chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334142 -class_id XY67 -dbname gpc1
    
  • 22:00 more LAP chip
    -- problem neb://ipp040.0/gpc1/20100820/o5428g0193o/o5428g0193o.ota15.fits
          0                     NON-EXISTANT file:///data/ippb02.1/nebulous/53/28/963490033.gpc1:20100820:o5428g0193o:o5428g0193o.ota15.fits
          0                     NON-EXISTANT file:///data/ippb02.0/nebulous/53/28/963490647.gpc1:20100820:o5428g0193o:o5428g0193o.ota15.fits
    
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334304 -class_id XY15 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY15 -exp_id 209767 -dbname gpc1
    
  • 22:10 few chips now moving through and resulted in pileup at stack with ~1100, will be a while. misc distribution faulted, ran
    disttool -revertrun -label LAP.ThreePi.20110809 -fault 2 -dbname gpc1
    disttool -revertrun -label MD09.nightlyscience -fault 2 -dbname gpc1
    disttool -revertrun -label ThreePi.nightlyscience -fault 2 -dbname gpc1
    
  • 22:30 with pileup in stack, will do the daily restart of stdscience and distribution.

Sunday : 2011-10-30

  • 09:15 doesn't look like stare nodes in use, putting nodes back into processing for now. summitcopy still downloading stare+3PI data
  • 14:30 not all stare+3PI copied from summit yet. summitcopy pantasks down, restarting.
  • 15:00 while data finishes downloading, fixing some LAP
    -- more with both instances missing on ippb02
    neb://ipp005.0/gpc1/20100629/o5376g0151o/o5376g0151o.ota02.fits
    neb://ipp035.0/gpc1/20100629/o5376g0431o/o5376g0431o.ota62.fits
    neb://ipp028.0/gpc1/20100619/o5366g0182o/o5366g0182o.ota27.fits
    neb://ipp040.0/gpc1/20100726/o5403g0224o/o5403g0224o.ota15.fits
    neb://ipp025.0/gpc1/20100530/o5346g0435o/o5346g0435o.ota76.fits
    
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334360 -class_id XY02 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY02 -exp_id 188606 -dbname gpc1
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334367 -class_id XY62 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY62 -exp_id 188901 -dbname gpc1
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334451 -class_id XY27 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY27 -exp_id 183983 -dbname gpc1
     chiptool -updateprocessedimfile -set_quality 42 -fault 0 -chip_id 334952 -class_id XY15 -dbname gpc1
     regtool -updateprocessedimfile -set_state corrupt -class_id XY15 -exp_id 197273 -dbname gpc1
    
    
    -- trouble with neb://ipp025.0/gpc1/20100530/o5346g0435o/o5346g0435o.ota76.fits
          0                     NON-EXISTANT file:///data/ipp053.0/nebulous/69/61/298845531.gpc1:20100530:o5346g0435o:o5346g0435o.ota76.fits
          1 da6d56624afd62ca903d3be8bc23ba7a file:///data/ippb00.2/nebulous/69/61/1175753790.gpc1:20100530:o5346g0435o:o5346g0435o.ota76.fits
     cp /data/ippb00.2/nebulous/69/61/1175753790.gpc1:20100530:o5346g0435o:o5346g0435o.ota76.fits /data/ipp053.0/nebulous/69/61/298845531.gpc1:20100530:o5346g0435o:o5346g0435o.ota76.fits
    
  • 15:50 all stare and 3PI downloaded, the little bit of 3PI is processing after
    regtool -updateprocessedimfile -exp_id  416736 -class_id XY56 -set_state pending_burntool -dbname gpc1
    

  • 23:00 looks like bad weather, clearing some LAP faults for running over night.
    -- diffim fault, rerun warp --  Reading FITS file /data/ipp027.0/nebulous/52/18/1516144205.gpc1:LAP.ThreePi.20110809:2011:10:31:o5758g0364o.363019:o5758g0364o.363019.wrp.296961.skycell.2642.043.wt.fits failed.
    
    perl ~ipp/src/ipp-20110622/tools/runwarpskycell.pl --warp_id 296961 --skycell_id skycell.2642.043 --redirect-output 
    
    -- magic faults, rerun diffims
    perl ~ipp/src/ipp-20110622/tools/rundiffskycell.pl --redirect-output --diff_id 185303  --skycell_id skycell.2535.092
    
    perl ~ipp/src/ipp-20110622/tools/rundiffskycell.pl --redirect-output --diff_id 185750  --skycell_id skycell.2528.035
    
    perl ~ipp/src/ipp-20110622/tools/rundiffskycell.pl --redirect-output --diff_id 185422  --skycell_id skycell.2633.007
    
    -- camera fault, corrupt chip .wt. file, re-run chip
    perl ~ipp/src/ipp-20110622/tools/runchipimfile.pl --chip_id 334929 --class_id XY53 --redirect-output
    
    
    
  • 23:45 restarted distribution, several older LAP runs finishing up.