| Version 4 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.02.11 - 2019.02.17
(Up to PS1 IPP Czar Logs)
Monday : 2019.02.11
- 08:15 EAM : without the ipp-b nodes, and ippb04 & ippb05 in repair for rsyncs, many update jobs failed due to lack of a backup target. I've put those two machines back to nebulous 'up' and reverted the failed update jobs.
JRF: Tidying up the mess I left and failed to rectify over the weekend
Some errors in chip stage for ps_ud_WEB
Exp Name Class ID Exp ID Chip ID State Label data grp dist grp Fault Revert Command o5327g0047o XY26 165316 512230 update ps_ud_WEB ecliptic.rp.7hr 4 chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1 o5675g0136o XY26 328761 512289 update ps_ud_WEB ecliptic.rp.7hr 4 chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1 o5675g0136o XY61 328761 512289 update ps_ud_WEB ecliptic.rp.7hr 4 chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1 o5675g0136o XY74 328761 512289 update ps_ud_WEB ecliptic.rp.7hr 4 chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1 o5829g0438o XY26 399488 512334 update ps_ud_WEB ecliptic.rp.7hr 4 chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1 o5829g0438o XY61 399488 512334 update ps_ud_WEB ecliptic.rp.7hr 4 chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1 o5829g0438o XY62 399488 512334 update ps_ud_WEB ecliptic.rp.7hr 4 chiptool -revertprocessedimfile -fault 4 -label ps_ud_WEB -dbname gpc1
Query the first exp_id:
SELECT chipRun.exp_id,chipRun.chip_id,chipRun.label,chipRun.state, camRun.cam_id,camRun.state, fakeRun.fake_id,fakeRun.state, warpRun.warp_id,warpRun.state FROM chipRun LEFT JOIN camRun USING (chip_id) LEFT JOIN fakeRun USING (cam_id) LEFT JOIN warpRun USING (fake_id) WHERE exp_id=165316;
Gives the following:
+--------+---------+---------------------+---------+---------+-------+---------+-------+---------+---------+ | exp_id | chip_id | label | state | cam_id | state | fake_id | state | warp_id | state | +--------+---------+---------------------+---------+---------+-------+---------+-------+---------+---------+ | 165316 | 92712 | goto_cleaned | cleaned | 76896 | full | 63594 | full | 61845 | cleaned | | 165316 | 92712 | goto_cleaned | cleaned | 76896 | full | 63594 | full | 64803 | cleaned | | 165316 | 113216 | cleanup_testing_old | cleaned | 96018 | full | 82702 | full | 81939 | cleaned | | 165316 | 512230 | ps_ud_WEB | update | 494867 | full | 474675 | full | 476374 | update | | 165316 | 2128504 | PSNSC.wfix.20180525 | full | 2094976 | full | 2063980 | full | 2070634 | full | | 165316 | 2129009 | PSNSC.wfix.20180525 | new | NULL | NULL | NULL | NULL | NULL | NULL | +--------+---------+---------------------+---------+---------+-------+---------+-------+---------+---------+
So it looks like there is a complete state for this exposure. There also appears to be an erroneous one 2129009 which has been sent after that is not needed (I'm guessing this did not process as the label was missing). Should cleanup chip_id=512230 and warp_id=476374 to get rid of the errors:
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 512230 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 476374
However, notice that there is another entry that was sent for reprocessing. This is likely a mistake on my part not realising that it had been reprocessed successfully. I already set it to cleaned, but really it should have been either 'dropped' from the table as it hasn't been done and doesn't need to be, or it should just be processed as I'm unsure if a later chip_id will cause problems (Mark also said he was unsure if it would cause problems).
Looks the same for both of the other exposures, so this time I will set enable the label in pantasks, and get the other exposures to process. Then clean the remaining images:
pantasks_client add.label PSNSC.wfix.20180525 exit chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 512289 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 512334 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 476431 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 476477
Errors in the warp stage for ps_ud_MOPS.2
o5327g0047o 165316 512230 494867 474675 476374 skycell.1873.066 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5327g0047o 165316 512230 494867 474675 476374 skycell.1873.076 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5675g0136o 328761 512289 494926 474732 476431 skycell.1874.008 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5675g0143o 328768 512296 494933 474739 476438 skycell.1790.092 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5675g0143o 328768 512296 494933 474739 476438 skycell.1874.008 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5829g0438o 399488 512334 494975 474778 476477 skycell.1873.083 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5829g0438o 399488 512334 494975 474778 476477 skycell.1873.093 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5328g0050o 165913 512261 494898 477328 479027 skycell.1789.088 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5328g0050o 165913 512261 494898 477328 479027 skycell.1789.089 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off o5328g0050o 165913 512261 494898 477328 479027 skycell.1789.099 update ps_ud_MOPS.2 2 warp.revert.on and warp.revert.off
The three of these exposures have the same ids as the chips we just fixed from ps_ud_WEB, so we know these have been processed now and we just need to tidy things up. Since the wap_ids were sent to clean the errors should disappear. DO THEY??! For the remaining problem, let us query:
SELECT chipRun.exp_id,chipRun.chip_id,chipRun.label,chipRun.state, camRun.cam_id,camRun.state, fakeRun.fake_id,fakeRun.state, warpRun.warp_id,warpRun.state FROM chipRun LEFT JOIN camRun USING (chip_id) LEFT JOIN fakeRun USING (cam_id) LEFT JOIN warpRun USING (fake_id) WHERE exp_id=165913;
This one has been reprocessed successfully...multiple times (silly John not realising he had already done this)!
It's just a case of sending that warp_id to be cleaned up:
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 479027
For exp_id=328768 it's the same treatment, just send the old warp_id (476438) to be cleaned.
Final note! Most things appear cleaned up, however there is now some stuff that I have allowed to reprocess again but it getting stuck in the burntool step. Neb logs say that it cannot find particular nebulous files -> because they are on the ippb machines that are down? Not sure how to confirm.
Tuesday : 2019.02.12
CCL: takes nodes down for Haydn: "Last week we replaced a disk in the RAID, which has completed rebuilding, but it still keeps on sending daily complaint messages. In the past, rebooting once has fix this."
neb-host --host ipp127.0 --state down --volume --note "CCL: set down for restart" neb-host --host ipp127.1 --state down --volume --note "CCL: set down for restart"
