IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 1 and Version 2 of PS1_IPP_Czarlog_20190211


Ignore:
Timestamp:
Feb 11, 2019, 5:58:45 PM (7 years ago)
Author:
fairlamb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • PS1_IPP_Czarlog_20190211

    v1 v2  
    66=== Monday : 2019.02.11 ===
    77  * 08:15 EAM : without the ipp-b nodes, and ippb04 & ippb05 in repair for rsyncs, many update jobs failed due to lack of a backup target.  I've put those two machines back to nebulous 'up' and reverted the failed update jobs.
     8
     9JRF: Tidying up the mess I left and failed to rectify over the weekend
     10
     11Some errors in chip stage for ps_ud_WEB
     12{{{
     13Exp Name    Class ID    Exp ID  Chip ID State   Label   data grp    dist grp    Fault   Revert Command
     14o5327g0047o XY26    165316  512230  update  ps_ud_WEB   ecliptic.rp.7hr     4   chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1
     15o5675g0136o XY26    328761  512289  update  ps_ud_WEB   ecliptic.rp.7hr     4   chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1
     16o5675g0136o XY61    328761  512289  update  ps_ud_WEB   ecliptic.rp.7hr     4   chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1
     17o5675g0136o XY74    328761  512289  update  ps_ud_WEB   ecliptic.rp.7hr     4   chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1
     18o5829g0438o XY26    399488  512334  update  ps_ud_WEB   ecliptic.rp.7hr     4   chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1
     19o5829g0438o XY61    399488  512334  update  ps_ud_WEB   ecliptic.rp.7hr     4   chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1
     20o5829g0438o XY62    399488  512334  update  ps_ud_WEB   ecliptic.rp.7hr     4   chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1
     21}}}
     22Query the first exp_id:
     23{{{
     24SELECT chipRun.exp_id,chipRun.chip_id,chipRun.label,chipRun.state, camRun.cam_id,camRun.state, fakeRun.fake_id,fakeRun.state, warpRun.warp_id,warpRun.state FROM chipRun LEFT JOIN camRun USING (chip_id) LEFT JOIN fakeRun USING (cam_id) LEFT JOIN warpRun USING (fake_id) WHERE exp_id=165316;
     25}}}
     26
     27Gives the following:
     28{{{+--------+---------+---------------------+---------+---------+-------+---------+-------+---------+---------+
     29| exp_id | chip_id | label               | state   | cam_id  | state | fake_id | state | warp_id | state   |
     30+--------+---------+---------------------+---------+---------+-------+---------+-------+---------+---------+
     31| 165316 |   92712 | goto_cleaned        | cleaned |   76896 | full  |   63594 | full  |   61845 | cleaned |
     32| 165316 |   92712 | goto_cleaned        | cleaned |   76896 | full  |   63594 | full  |   64803 | cleaned |
     33| 165316 |  113216 | cleanup_testing_old | cleaned |   96018 | full  |   82702 | full  |   81939 | cleaned |
     34| 165316 |  512230 | ps_ud_WEB           | update  |  494867 | full  |  474675 | full  |  476374 | update  |
     35| 165316 | 2128504 | PSNSC.wfix.20180525 | full    | 2094976 | full  | 2063980 | full  | 2070634 | full    |
     36| 165316 | 2129009 | PSNSC.wfix.20180525 | new     |    NULL | NULL  |    NULL | NULL  |    NULL | NULL    |
     37+--------+---------+---------------------+---------+---------+-------+---------+-------+---------+---------+}}}
     38So it looks like there is a complete state for this exposure. There also appears to be an erroneous one 2129009 which has been sent after that is not needed (I'm guessing this did not process as the label was missing).
     39Should cleanup chip_id=512230 and warp_id=476374 to get rid of the errors:
     40{{{
     41chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 512230
     42warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 476374
     43}}}
     44
     45However, notice that there is another entry that was sent for reprocessing. This is likely a mistake on my part not realising that it had been reprocessed successfully. I already set it to cleaned, but really it should have been either 'dropped' from the table as it hasn't been done and doesn't need to be, or it should just be processed as I'm unsure if a later chip_id will cause problems (Mark also said he was unsure if it would cause problems).
     46
     47Looks the same for both of the other exposures, so this time I will set enable the label in pantasks, and get the other exposures to process. Then clean the remaining images:
     48{{{
     49pantasks_client
     50add.label PSNSC.wfix.20180525
     51exit
     52
     53chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 512289
     54chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 512334
     55warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 476431
     56warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 476477
     57}}}
     58
     59
     60Errors in the warp stage for ps_ud_MOPS.2
     61{{{
     62o5327g0047o 165316  512230  494867  474675  476374  skycell.1873.066    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     63o5327g0047o 165316  512230  494867  474675  476374  skycell.1873.076    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     64o5675g0136o 328761  512289  494926  474732  476431  skycell.1874.008    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     65o5675g0143o 328768  512296  494933  474739  476438  skycell.1790.092    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     66o5675g0143o 328768  512296  494933  474739  476438  skycell.1874.008    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     67o5829g0438o 399488  512334  494975  474778  476477  skycell.1873.083    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     68o5829g0438o 399488  512334  494975  474778  476477  skycell.1873.093    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     69o5328g0050o 165913  512261  494898  477328  479027  skycell.1789.088    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     70o5328g0050o 165913  512261  494898  477328  479027  skycell.1789.089    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     71o5328g0050o 165913  512261  494898  477328  479027  skycell.1789.099    update  ps_ud_MOPS.2    2   warp.revert.on and warp.revert.off
     72}}}
     73
     74The three of these exposures have the same ids as the chips we just fixed from ps_ud_WEB, so we know these have been processed now and we just need to tidy things up. Since the wap_ids were sent to clean the errors should disappear. DO THEY??!
     75For the remaining problem, let us query:
     76{{{
     77SELECT chipRun.exp_id,chipRun.chip_id,chipRun.label,chipRun.state, camRun.cam_id,camRun.state, fakeRun.fake_id,fakeRun.state, warpRun.warp_id,warpRun.state FROM chipRun LEFT JOIN camRun USING (chip_id) LEFT JOIN fakeRun USING (cam_id) LEFT JOIN warpRun USING (fake_id) WHERE exp_id=165913;
     78}}}
     79This one has been reprocessed successfully...multiple times (silly John not realising he had already done this)!
     80
     81It's just a case of sending that warp_id to be cleaned up:
     82{{{
     83warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 479027
     84}}}
     85
     86For exp_id=328768 it's the same treatment, just send the old warp_id (476438) to be cleaned.
     87
     88
     89Final note! Most things appear cleaned up, however there is now some stuff that I have allowed to reprocess again but it getting stuck in the burntool step. Neb logs say that it cannot find particular nebulous files -> because they are on the ippb machines that are down? Not sure how to confirm.
    890
    991=== Tuesday : YYYY.MM.DD ===