PS1 IPP Czar Logs for the week 2019.03.18 - 2019.03.24
(Up to PS1 IPP Czar Logs)
Czar of the week: John, with MEH as czar-lord (previously: TdB + EAM)
Monday : 2019.03.18
- TdB: In the morning (05:45), I noticed ippb18 was reported as down by nagios. When trying to get more information, I noticed neb-host was reporting too many connections and no machines where showing up in IPP monitor (all greyed out). Following a quick back and forth with Gene I then set all pantasks to stop using:
check_system.sh stop
Furthermore, the compute nodes ippc71-75 were showing huge load on ganglia, so we turned off apache on those machines, using:/etc/init.d/apache2 stop
as root on each of the machines.
- Following this, we (John, Roger, Gene, myself) all came in at 09:00 to diagnose and fix the issue. First, we looked at the dump of processlist on ippc75:
mysql -h localhost -u root -e "show processlist" > neb.proc.20190318
This showed 64 connections from ippc76, which is not on the list of nebservers (see ~ippitc/ippconfig/nebservers.csh) and also showed soem connections from ippc28. Looking at the configuration file /etc/apache2/modules.d/apache2-mod_perl-startup.pl showed 256 MaxClients connections set up for the compute nodes. Machine ippc76 has StartServers = 64, so it will start with normally 64 connections, but could go up to 256 total. Combining all machines, this results in a max of 1349 connections, which is a lot.
Looking at the max connections in MYSQL using:
select @@max_connections;shows that the maximum is set to 1024, which is less than 1349, so that could be a reason for the failure. We are setting that higher now, to avoid this issue:
SET GLOBAL max_connections = 2048 (in mysql)
The full reason for the break is not totally clear, but the huge load spikes on the apache servers roughly coincide with ippb18 going down, so that might well be related.
For now, we are setting the apache servers back to on, using:
/etc/init.d/apache2 startas root on each of the machines.
Finally, the pantasks, are set back to run using:
check_system.sh run
- MEH: bulk updates now continuing again in ~ippmops/stdscience
- MEH: sending more chips to cleanup for warps in full state
- ipp091 excess load from being harassed from cleanup and failed BBU, leaving in neb-host repair now
- MEH: ippb04, b05 causing many i/o faults from the rsyncs -- set neb-host down from repair since many primary copies of files there... -- will need to temporarily set back to repair to clear other faults for the cases where files are only on those machines
Afternoon czar formation (ft. MEH) by JRF
The recipes directory contains a bunch of configurations:
ippitc/src/ipp-20170121/ippconfig/recipes
We'll look at the reductionClasses.mdc config file:
Contains the configuration for how to process something. In this case the CHIP_PPIMAGE stage is set to CHIP_NOMASKTEST instead of the usual CHIP.
# test for MOPS request reprocessing without masking
NOMASKTEST METADATA
CHIP_PPIMAGE STR CHIP_NOMASKTEST
.
.
.
BACKGROUND_PSWARP STR BACKGROUND
PSVIDEOPHOT STR PSVIDEOPHOT
END
There are separate configs for each camera (go up a directory then into the relevant camera dir). There are a bunch of configs here are empty in the main config, but have entries for the separate cameras. This is because data from each camera should be treated separately.
WATCH OUT: If you reprocess something with a different config then the stamp server will use whatever the latest config was for a request! Therefore, it's best to do some crazy reprocessing locally, so as not to pollute the database with entries that could confuse the current stamp request system.
Time to reprocess some stuff for Rob with no masks (as seen on PSNSC_MOPS page and czar log from 2019.02.18 - 2019.02.24):
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0726o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0707o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0688o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0669o
As you can see we're using the NOMASKTEST reduction (which we just had a look at).
Don't forget to add the label to pantasks to make it run!
stuck jobs by JRF
Time to fix some stuck jobs! Some files are needed from ippb05, but it has been misbehaving and was set to down ealier today. We shall quickly put it to repair so that the files can be accessed:
neb-host ippb05 repair --note "JRF: down -> repair: so that some broken updates can clear on ippmonitor"
Then we shall revert the chip for the stuck job so that it can continue:
chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1
You don't need to specify the chip_id, it'll just revert all faults that are stuck.
If you come across a stamp that you suspect maybe be stuck, have a look at it using:
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
remove '-simple' to get some more information.
Later that same evening: Jobs approaching 100k, so restarting stdscience pantasks. A bunch of failed jobs are beginning to accumulate too, but I'll save that fun for the morning.
Tuesday : 2019.03.19
JRF (morning): Jobs approaching 100k, so restarting stdscience pantasks. Flicked ippb04 and ippb05 to repair, from down, so that some files could be accessed to clear the backlog of tasks.
czar formation (ft MEH) by JRF
warp fault stuck in full state by JRF
Weird warp fault where the state is full:
o7071g0124o 875375 1530574 1497865 1471180 1477774 skycell.1618.088 full update.OSS
Query the neb-workdir (note the wild card at the end):
neb-ls -p -l neb://ipp069.0/gpc1/OSS.nt/2015/02/18//o7071g0124o.875375/o7071g0124o.875375.wrp.1477774.skycell.1618.088%
shows us that one of the mask files is in a funny state where is 0 size:
-rw-rw-r-- 1 ippitc users 0 Mar 8 09:27 /data/ipp125.0/nebulous/9f/e9/11203362808.gpc1:OSS.nt:2015:02:18:o7071g0124o.875375:o7071g0124o.875375.wrp.1477774.skycell.1618.088.mask.fits
To solve this, we will have to put the warp to clean:
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1477774
check that is has cleaned succesfully (query the database). Then set it to update:
warptool -dbname gpc1 -setskyfiletoupdate -set_label update.OSS -warp_id 1477774
Solving other warp issues
Move the stuff that is currently broken under the label 'update.OSS' to the 'update.OSS.broke' label. To figure out which ones need to be moved have a look at the state, fault, and skyfile
Will need to add this new label to pantasks so it can get priority and actually work on them.
The following shows the query used, and how to pipe it to 'awks' so that it can run the chiptool/warptool command to update the labels on the chip/warp id pairs.
mysql -uXXXXX -pXXXXX -hippdb08 gpc1 -B --skip-column-names -e
'SELECT chip_id,
warp_id,
warpRun.label,
warpRun.state,
warpSkyfile.fault,
chipRun.label,
chipRun.state
FROM warpRun
JOIN warpSkyfile USING (warp_id)
JOIN fakeRun USING (fake_id)
JOIN camRun USING (cam_id)
JOIN chipRun USING (chip_id)
WHERE warpRun.label LIKE "update.OSS"
AND warpSkyfile.fault > 0
AND warpRun.state LIKE "update"
GROUP BY warp_id;'
| awk '{print"
chiptool -dbname gpc1 -updaterun -chip_id "$1" -state update -set_label update.OSS.broke -pretend;
warptool -dbname gpc1 -updaterun -warp_id "$2" -state update -set_label update.OSS.broke -pretend
"}'
| xargs -0 bash -c
Note: The above its formatted for readability, you'll have to squish it into one line, fill in the XXXXXs, and remove pretend.
Next, add the label to stdscience (pantasks> add.label update.OSS.broke)
Then put the b nodes up to repair to help clear things out.
JRF (late evening): Gave pantasks a restart as it was close to 100k jobs
Wednesday : 2019.03.20
MEH moved across the remaining update.OSS.broke labels.
The SVN server died! EAM has been moving a backup of it to a new server.
Broken/pending MOPS stamp requests
There are some broken stamps that Rob has requested:
count label req_id name timestamp fault exp_id dtime utc_timestamp() 1 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 466358 00:04:05 2019-03-20 22:20:01 2 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 466379 00:04:05 2019-03-20 22:20:01 2 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 705006 00:04:05 2019-03-20 22:20:01 2 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 705027 00:04:05 2019-03-20 22:20:01 1 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 705045 00:04:05 2019-03-20 22:20:01 1 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 705068 00:04:05 2019-03-20 22:20:01
Time to take the exp_id and have a look at them:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (466358,466379,705006,705027,705045,705068); +-------------+--------+---------+---------------------+---------+---------+---------+---------+---------------------+---------+ | exp_name | exp_id | chip_id | label | state | cam_id | state | warp_id | label | state | +-------------+--------+---------+---------------------+---------+---------+---------+---------+---------------------+---------+ | o6002g0321o | 466358 | 1067421 | update.hold20190318 | update | 1036073 | full | 1018412 | update.hold20190318 | update | | o6002g0340o | 466379 | 1067424 | update.hold20190318 | update | 1036086 | full | 1018431 | update.hold20190318 | update | | o6698g0496o | 705045 | 1078654 | update.hold20190318 | update | 1040692 | full | 1022614 | update.hold20190318 | update | | o6698g0517o | 705068 | 1078658 | update.hold20190318 | update | 1040696 | full | 1022879 | update.hold20190318 | update | +-------------+--------+---------+---------------------+---------+---------+---------+---------+---------------------+---------+
There are a bunch of cleaned ones but the most recent ones have the label update.hold20190318
set them to update:
chiptool -dbname gpc1 -updaterun -chip_id XXXXX -state update -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id XXXXX -state update -set_label ps_ud_MOPS
None of them seem to be moving through... Mark suggested trying it will -setimfiletoupdate:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXX
Two of these failed...
o6002g0321o XY37 466358 542901 update ps_ud_MOPS LAP.ThreePi.20120706.20120813 LAP.ThreePi 2 o6002g0340o XY46 466379 542904 update ps_ud_MOPS LAP.ThreePi.20120706.20120813 LAP.ThreePi 2
..., so I'll revert them with the suggested command
chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1
This did not fully succeed, they remain stuck in the update state.. I'm going to try cleaning...
chiptool -dbname gpc1 -updaterun -chip_id 1067421 -set_state cleaned chiptool -dbname gpc1 -updaterun -chip_id 1067424 -set_state cleaned chiptool -dbname gpc1 -updaterun -chip_id 1078654 -set_state cleaned chiptool -dbname gpc1 -updaterun -chip_id 1078658 -set_state cleaned
...then updating:
chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1067421 -set_label ps_ud_MOPS chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1067424 -set_label ps_ud_MOPS chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1078654 -set_label ps_ud_MOPS chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1078658 -set_label ps_ud_MOPS
This sometimes fixes the problem.
Hung stamp requests by JRF
Going back to the stuff that had hung pstamps, on the pstamp server we can see that nothing has moved for about 10 hours. If you come across a stamp that you suspect maybe be stuck, have a look at it using:
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
remove '-simple' to get some more information.
More pstamp investigations: by JRF
2 MOPS 918835 weryk_5C92D8E1 2019-03-21 00:40:58 26 289960 00:14:03 2019-03-21 00:55:01 2 MOPS 918835 weryk_5C92D8E1 2019-03-21 00:40:58 26 289981 00:14:03 2019-03-21 00:55:01
Time to take the exp_id and have a look at them:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (289960, 289981); +-------------+--------+---------+-------------------+---------+--------+-------+---------+-----------------------------------+---------+ | exp_name | exp_id | chip_id | label | state | cam_id | state | warp_id | label | state | +-------------+--------+---------+-------------------+---------+--------+-------+---------+-----------------------------------+---------+ | o5591g0062o | 289960 | 830503 | ps_ud_MOPS | update | 806578 | full | 788140 | ps_ud_MOPS | update | | o5591g0083o | 289981 | 830524 | ps_ud_MOPS | update | 806604 | full | 788180 | ps_ud_MOPS | update | +-------------+--------+---------+-------------------+---------+--------+-------+---------+-----------------------------------+---------+
set them to update:
chiptool -dbname gpc1 -updaterun -chip_id 830503 -state update -set_label ps_ud_MOPS chiptool -dbname gpc1 -updaterun -chip_id 830524 -state update -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 788140 -state update -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 788180 -state update -set_label ps_ud_MOPS
Then run this to force them to update...
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 830503 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 830524
And for the warps:
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id 788140 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id 788180
Without a mask by JRF
Rob has requested some things to be processed without a mask: o7255g0537o 04:22:26.871 +21:06:31.62 K17W15C o7255g0531o 04:22:26.688 +21:06:27.34 K17W15C o7255g0525o 04:22:26.506 +21:06:23.05 K17W15C o7255g0519o 04:22:26.323 +21:06:18.76 K17W15C
So I've queued them up to reprocess without a mask i.e. -set_reduction NOMASKTEST
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0537o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0531o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0525o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0519o
They go through fine
Broken IPP jobs: label changing + more by JRF
There are two warp_ids that have problems: 1640466, 1640460
Find out the sate of these:
SELECT chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) WHERE warpRun.warp_id IN (1640466, 1640460); +---------+------------------+--------+---------+-------+---------+------------------+--------+ | chip_id | label | state | cam_id | state | warp_id | label | state | +---------+------------------+--------+---------+-------+---------+------------------+--------+ | 1693054 | update.X.ThreePi | update | 1660571 | full | 1640460 | update.X.ThreePi | update | | 1693060 | update.X.ThreePi | update | 1660577 | full | 1640466 | update.X.ThreePi | update | +---------+------------------+--------+---------+-------+---------+------------------+--------+
For exercises sake, we're going to change the label for the first line's chip and warp id to ps_ud_MOPS (as it's related to some stamps that MOPS are after):
chiptool -dbname gpc1 -updaterun -chip_id 1693054 -state update -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1640460 -state update -set_label ps_ud_MOPS
More IPP monitor errors (solving missing log) by JRF
Checking out some of the errors, one of them complains about a file that it could not use:
Unable to perform warp.
-> pmConfigFileRead (pmConfig.c:317): Problem in configure files
Unable to read user configuration from /data/ipp069.0/nebulous/2c/d5/7195324599.gpc1:OSS.nt:2013:10:08:o6573g0097o.663490:o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
Having a look at the file it is empty...It's probably not supposed to be, so we will have a look at the nebulous entry:
neb-stat --validate neb://ipp124.1/gpc1/OSS.nt/2013/10/08//o6573g0097o.663490/o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
This shows that the other two copies of this file are not empty, which is good (see the checksum: d41d8cd98f00b204e9800998ecf8427e)
36.pswarp.mdc
object id: 3133998992
key: gpc1/OSS.nt/2013/10/08/o6573g0097o.663490/o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
read lock: 0
write lock:
epoch: 2013-10-08 00:08:21
mtime: 0000-00-00 00:00:00
md5sum count: 2
requested instances: 2
available instances: 3
existing instances: 3
total instances: 3
instance location:
1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp069.0/nebulous/2c/d5/7195324599.gpc1:OSS.nt:2013:10:08:o6573g0097o.663490:o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
1 7e5f777b2a1658fb976957e5c5c9639b file:///data/ipp091.0/nebulous/2c/d5/7195346311.gpc1:OSS.nt:2013:10:08:o6573g0097o.663490:o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
1 7e5f777b2a1658fb976957e5c5c9639b file:///data/ipp072.0/nebulous/2c/d5/7296944772.gpc1:OSS.nt:2013:10:08:o6573g0097o.663490:o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
To solve this there are two options:
1) delete the empty file 2) replace the empty file with one of the correct ones.
We chose option 1. Simply copy one of the good ones over the old one
Stalled IPP monitor jobs
Checking out some stalled update.OSS.broke things that are not faulted, so do not appear in IPP monitor:
SELECT chip_id, chipProcessedImfile.fault, warpRun.warp_id,warpRun.label,warpRun.state,warpSkyfile.fault,chipRun.label chiplabel,chipRun.state FROM warpRun JOIN warpSkyfile USING (warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN chipProcessedImfile USING (chip_id) WHERE warpRun.label LIKE "update.OSS.broke" AND warpRun.state LIKE "update" GROUP BY warpRun.warp_id; +---------+-------+---------+------------------+--------+-------+--------------+--------------+ | chip_id | fault | warp_id | label | state | fault | label | state | +---------+-------+---------+------------------+--------+-------+--------------+--------------+ | 1257054 | 0 | 1200666 | update.OSS.broke | update | 0 | update.OSS | goto_cleaned | | 1268153 | 0 | 1214242 | update.OSS.broke | update | 0 | goto_cleaned | cleaned | | 1280537 | 0 | 1225941 | update.OSS.broke | update | 0 | goto_cleaned | cleaned | . . . | 1730278 | 0 | 1677448 | update.OSS.broke | update | 0 | goto_cleaned | cleaned | | 1749642 | 0 | 1696720 | update.OSS.broke | update | 0 | update.OSS | full | . . . | 1762720 | 0 | 1709795 | update.OSS.broke | update | 0 | update.OSS | full | +---------+-------+---------+------------------+--------+-------+--------------+--------------+
The first one is kinda suspicious, as it's state is goto_cleaned. So let's set it to update:
chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1257054 -set_label update.OSS.broke
That didn't quite work, Sending to clean instead:
chiptool -dbname gpc1 -updaterun -chip_id 1257054 -set_label goto_cleaned
In the meantime, try the second on the list:
chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1268153 -set_label update.OSS.broke
that one was fine...
+---------+-------+---------+------------------+--------+-------+------------------+--------------+ | chip_id | fault | warp_id | label | state | fault | chiplabel | state | +---------+-------+---------+------------------+--------+-------+------------------+--------------+ | 1257054 | 0 | 1200666 | update.OSS.broke | update | 0 | goto_cleaned | goto_cleaned | | 1268153 | 0 | 1214242 | update.OSS.broke | update | 0 | update.OSS.broke | update | | 1280537 | 0 | 1225941 | update.OSS.broke | update | 0 | goto_cleaned | cleaned |
So what happened to the first chip 1257054? Check out the cleanup stdout logs (pantasks logs), and we see some problems: It mostly complains about missing log files that were expected on old stsci machines.
So then we went to have a look at the chipProcssedImfile to see if there is anything weird:
SELECT chip_id, chipProcessedImfile.fault, chipProcessedImfile.class_id,chipProcessedImfile.data_state,chipProcessedImfile.quality FROM chipProcessedImfile WHERE chip_id=1257054;
Running the above query you can see that the chips are in a mix of full/cleaned states...which is not the way it should be
We'll set it all to cleaned: chiptool -dbname gpc1 -updaterun -chip_id 1257054 -set_state cleaned
Now, we can finally try again with chiptool stuff: chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1257054 -set_label update.OSS.broke
And so the chip state updated to clean...but the warp tool is still in update (later in here).
Most other things just seem stuck on the old missing files from ippb05 problem (so I put it into repair for the time being).
chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1268153 -set_label update.OSS.broke chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1280537 -set_label update.OSS.broke chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1530574 -set_label update.OSS.broke chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1591282 -set_label update.OSS.broke chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1599586 -set_label update.OSS.broke chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1617899 -set_label update.OSS.broke chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1685737 -set_label update.OSS.broke chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1726939 -set_label update.OSS.broke chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1730278 -set_label update.OSS.broke
Thursday : 2019.03.21
Broken stamps -> assertion failures -> reprocessing by JRF
Tasks that are pending, as found by:
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple 10818461 new warp 475995 skycell.1432.053 gpc1 /data/ipp113.0/pstamp/work/2019/01/10/914957 ps_ud_MOPS.2 F 0 1 MOPS 918857 /data/ipp113.0/pstamp/work/2019/01/10/914957/checkdep.10818461.log 500 10818468 new warp 475962 skycell.1432.053 gpc1 /data/ipp113.0/pstamp/work/2019/01/10/914957 ps_ud_MOPS.2 F 0 1 MOPS 918857 /data/ipp113.0/pstamp/work/2019/01/10/914957/checkdep.10818468.log 500 10959532 new warp 475995 skycell.1432.054 gpc1 /data/ipp113.0/pstamp/work/2019/03/21/918857 ps_ud_MOPS F 0 0 MOPS 918857 /data/ipp113.0/pstamp/work/2019/03/21/918857/checkdep.10959532.log 500 10959534 new warp 475962 skycell.1432.054 gpc1 /data/ipp113.0/pstamp/work/2019/03/21/918857 ps_ud_MOPS F 0 0 MOPS 918857 /data/ipp113.0/pstamp/work/2019/03/21/918857/checkdep.10959534.log 500 10959535 new warp 475957 skycell.1432.054 gpc1 /data/ipp113.0/pstamp/work/2019/03/21/918857 ps_ud_MOPS F 0 0 MOPS 918857 /data/ipp113.0/pstamp/work/2019/03/21/918857/checkdep.10959535.log 500
Let's query these:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (475995,475962,475995,475962,475957); +---------+----------------------------+--------+---------+----------------------------+--------+ | chip_id | chipLabel | state | warp_id | warpLabel | state | +---------+----------------------------+--------+---------+----------------------------+--------+ | 511806 | update.X.ecliptic.rp.broke | update | 475957 | update.X.ecliptic.rp.broke | update | | 511811 | update.X.ecliptic.rp.broke | update | 475962 | update.X.ecliptic.rp.broke | update | | 511845 | update.X.ecliptic.rp.broke | update | 475995 | update.X.ecliptic.rp.broke | update | +---------+----------------------------+--------+---------+----------------------------+--------+
These ones are in update, and under the broke label. So we'll change their labels to ps_ud_MOPS so that they get processed and can diagnose the errors from there:
chiptool -dbname gpc1 -updaterun -chip_id 511806 -set_label ps_ud_MOPS chiptool -dbname gpc1 -updaterun -chip_id 511811 -set_label ps_ud_MOPS chiptool -dbname gpc1 -updaterun -chip_id 511845 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 475957 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 475962 -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 475995 -set_label ps_ud_MOPS
Some of these jobs now fail. Three of them all fail in the same way, complaining about:
ppImage: psphotSourceSize.c:122: psphotSourceSizeReadout: Assertion `status' failed.
This implies that there is failure in the software, likely because of code changes, so it cannot reproduce previous results.
Get the exposure names for these, then reprocess them (since they are w-band we should use the following, see the PSNSC_MOPS page for details):
chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20190321 -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20190321 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20190321.x.date -exp_name o5535g0266o chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20190321 -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20190321 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20190321.x.date -exp_name o5535g0271o chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20190321 -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20190321 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20190321.x.date -exp_name o5535g0313o
and don't forget to add the label!
Secret email from ippc18 gives the following:
count(*) label req_id name timestamp fault exp_id dtime utc_timestamp() 1 MOPS 918857 weryk_5C941E46 2019-03-21 23:30:40 25 263293 00:15:07 2019-03-21 23:45:01
Let's check out this exposure, 263293:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id=263293; +-------------+--------+---------+----------------------------+------------+---------+----------------------------+------------+ | exp_name | exp_id | chip_id | chip_label | chip_state | warp_id | warp_label | warp_state | +-------------+--------+---------+----------------------------+------------+---------+----------------------------+------------+ | o5535g0308o | 263293 | 167699 | goto_cleaned_redo | cleaned | 135866 | goto_cleaned | cleaned | | o5535g0308o | 263293 | 511840 | update.X.ecliptic.rp.broke | update | 475985 | update.X.ecliptic.rp.broke | update | +-------------+--------+---------+----------------------------+------------+---------+----------------------------+------------+
Might as well move this across too:
chiptool -dbname gpc1 -updaterun -chip_id 511840 -set_label ps_ud_MOPS
warptool -dbname gpc1 -updaterun -warp_id 475985 -set_label ps_ud_MOPS
When ps_ud_MOPS processes this one, it faults. It complains about missing config files...so just reprocess it:
But how should we reprocess it you ask? Well check the commment/filter etc from the rawExp table. This too is w-band, so the following will do:
chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20190321 -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20190321 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20190321.x.date -exp_name o5535g0308o
Now cleanup all of the old stuff:
chiptool -dbname gpc1 -updaterun -chip_id 511806 -set_state goto_cleaned -set_label goto_cleaned chiptool -dbname gpc1 -updaterun -chip_id 511811 -set_state goto_cleaned -set_label goto_cleaned chiptool -dbname gpc1 -updaterun -chip_id 511845 -set_state goto_cleaned -set_label goto_cleaned warptool -dbname gpc1 -updaterun -warp_id 475957 -set_state goto_cleaned -set_label goto_cleaned warptool -dbname gpc1 -updaterun -warp_id 475962 -set_state goto_cleaned -set_label goto_cleaned warptool -dbname gpc1 -updaterun -warp_id 475995 -set_state goto_cleaned -set_label goto_cleaned chiptool -dbname gpc1 -updaterun -chip_id 511840 -set_state goto_cleaned -set_label goto_cleaned warptool -dbname gpc1 -updaterun -warp_id 475985 -set_state goto_cleaned -set_label goto_cleaned
Adding a new label (that persists on restart) by JRF
The LAP.PV3 stuff needs to updated next, and this will take a while. So it makes sense to add a new label that will be loaded automatically when pantasks starts up.
To do this simply add the following line to the 'input' file in the stdscience directory: {{
add.label update.LAP.PV3
}} put it in a sensible place too, like next to all the other labels (best take a copy of input too).
This new label currently has the highest priority, as it defaults to highest if there is no entry... this is not what we want. To add the label to the list of labels, and set the priority, use the label tool! To check if it is present first:
labeltool -dbname gpc1 -listlabel -simple | grep update
Add label and set priority:
labeltool -dbname gpc1 -definelabel -set_label update.LAP.PV3 -set_priority 301
In this case we want it to be a lower priority than MOPS.2 which was 310, so we chose 301 (and 1 higher that update.OSS hehehe)
Give it a few minutes and ippmonitor should reflect these changes by showing it in the correct place.
There are some broken jobs on IPP monitor by JRF
Most had memory problems and only needed reverting.
Friday : 2019.03.22
Broken jobs breakfast saga by JRF
What better way to start the day than a cup of coffee and some red stalled jobs on IPP monitor.
ps_ud_MOPS chip faults by JRF
A ps_ud_MOPS job stuck in the chip stage:
exp_name chip exp_id chip_id state label data_group dist_group o5102g0194o XY65 101412 1280264 update ps_ud_MOPS LAP.PV3.20140730.ipp.20141126 LAP.ThreePi SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_id=101412; +-------------+--------+---------+---------------------+------------+ | exp_name | exp_id | chip_id | chip_label | chip_state | +-------------+--------+---------+---------------------+------------+ | o5102g0194o | 101412 | 1280264 | ps_ud_MOPS | update | +-------------+--------+---------+---------------------+------------+
However, the log appears to be missing for this file...
log file (LOG.IMFILE.UPDATE, 'neb://@HOST@.0/gpc1/LAP.PV3.20140730.ipp/2014/11/26/o5102g0194o.101412/o5102g0194o.101412.ch.1280264') not found
I tried to revert the job:
chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1
Which did not work...
Perhaps it should be reprocessed? It's LAP.PV3 in the r-band, so should use:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190322 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190322 -set_reduction LAP_SCIENCE -exp_name o5102g0194o
I have submitted this:
Query the exposure:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE rawExp.exp_id=101412; +-------------+--------+---------+--------------------+------------+---------+--------------------+------------+ | exp_name | exp_id | chip_id | chip_label | chip_state | warp_id | warp_label | warp_state | +-------------+--------+---------+--------------------+------------+---------+--------------------+------------+ | o5102g0194o | 101412 | 29631 | SVS.20091005 | cleaned | 13660 | SVS.20091005 | cleaned | | o5102g0194o | 101412 | 223654 | goto_cleaned.rerun | cleaned | 191413 | goto_cleaned.rerun | cleaned | | o5102g0194o | 101412 | 693009 | goto_cleaned_redo | cleaned | 650278 | goto_cleaned | cleaned | | o5102g0194o | 101412 | 1280264 | ps_ud_MOPS | update | 1260815 | ps_ud_MOPS | update | | o5102g0194o | 101412 | 2129470 | mopsfix | full | 2071601 | mopsfix | full | +-------------+--------+---------+--------------------+------------+---------+--------------------+------------+
All done, send the old stuff to clean:
chiptool -dbname gpc1 -updaterun -chip_id 1280264 -set_state goto_cleaned -set_label goto_cleaned
warptool -dbname gpc1 -updaterun -warp_id 1260815 -set_state goto_cleaned -set_label goto_cleaned
Repeat for this process for the other failed chip job
ps_ud_MOPS warp faults by JRF
There are two ps_ud_MOPS stuck in the warp stage:
Exp Name Exp ID Chip ID Cam ID Fake ID Warp ID Skycell ID State Label Fault o6055g0196o 484983 1093091 1059583 1035273 1041865 skycell.2257.095 update ps_ud_MOPS 2 o6055g0220o 485006 1093094 1059586 1035276 1041868 skycell.2257.095 update ps_ud_MOPS 2
These both have the same error as the previous chip problem; no log file can be found...so reprocess. They are LAP.PV3 y-band:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190322 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190322 -set_reduction LAP_SCIENCE -exp_name o6055g0196o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190322 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190322 -set_reduction LAP_SCIENCE -exp_name o6055g0220o
I've sent them to reprocess... so check that they are done:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE rawExp.exp_id IN (484983,485006); +-------------+--------+---------+--------------------+------------+---------+--------------------+------------+ | exp_name | exp_id | chip_id | chip_label | chip_state | warp_id | warp_label | warp_state | +-------------+--------+---------+--------------------+------------+---------+--------------------+------------+ | o5102g0194o | 101412 | 29631 | SVS.20091005 | cleaned | 13660 | SVS.20091005 | cleaned | | o5102g0194o | 101412 | 223654 | goto_cleaned.rerun | cleaned | 191413 | goto_cleaned.rerun | cleaned | | o5102g0194o | 101412 | 693009 | goto_cleaned_redo | cleaned | 650278 | goto_cleaned | cleaned | | o5102g0194o | 101412 | 1280264 | ps_ud_MOPS | update | 1260815 | ps_ud_MOPS | update | | o5102g0194o | 101412 | 2129470 | mopsfix | full | 2071601 | mopsfix | full | +-------------+--------+---------+--------------------+------------+---------+--------------------+------------+
They're now full, so clean 'em up
warptool -dbname gpc1 -updaterun -warp_id 1041865 -set_state goto_cleaned -set_label goto_cleaned
warptool -dbname gpc1 -updaterun -warp_id 1041868 -set_state goto_cleaned -set_label goto_cleaned
Lots of update.LAP.PV3 have missing logs
All of this show a warp state of full, but still have a fault of 2, and they complain about missing logs file along the lines of:
neb://stsci05.1/gpc1/LAP.PV3.20140730.ipp/2014/10/04/o5358g0257o.179786/o5358g0257o.179786.wrp.1069329.skycell.0938.038
i.e. they all start neb://stsciXX.X
This looks different to yesterday's issue... they are
SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state warp_state,warpRun.label warp_label,warpRun.data_group,warpSkyfile.fault,warpSkyfile.path_base FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.warp_id = '1302416' AND warpSkyfile.skycell_id like 'skycell.0788.077' LIMIT 80;
Maybe do this from earlier in the week:
Weird warp fault where the state is full:
o5410g0454o 201542 1345623 1321690 1295801 1302416 skycell.0788.077 full update.LAP.PV3Query the neb-workdir (note the wild card at the end):
neb-ls -p -l neb://stsci08.1/gpc1/LAP.PV3.20140730.bigmem/2014/12/19/o5410g0454o.201542/o5410g0454o.201542.wrp.1302416.skycell.0788.077%
We get a warning about the stats file being missing:
2019/03/22 13:53:48 | ipp113 | FATAL | Nebulous::Client::find_instances - unhandled fault - database error: no instances available for key: neb:///gpc1/LAP.PV3.20140730.bigmem/2014/12/19/o5410g0454o.201542/o5410g0454o.201542.wrp.1302416.skycell.0788.077.stats at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 1991, <DATA> line 14. unhandled fault - database error: no instances available for key: neb:///gpc1/LAP.PV3.20140730.bigmem/2014/12/19/o5410g0454o.201542/o5410g0454o.201542.wrp.1302416.skycell.0788.077.stats at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 1991, <DATA> line 14.
To solve this, we will have to put the warp to clean:
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1477774check that is has cleaned succesfully (query the database).
SELECT warpRun.warp_id, warpRun.state FROM warpRun WHERE warpRun.warp_id IN (1069329,1073367,1073369,1073428,1074105,1075041,1075048,1075067,1075098,1075975,1075977,1076071,1078051,1078064,1078068,1078071,1078077,1078082,1078083,1078084,1078085,1078087,1078088,1078090,1078091,1078092,1078093,1078094,1078095,1078096,1078098,1078099,1078110,1078112,1078133,1078136,1078138,1078139,1078143,1078144,1078145,1078147,1078173,1078748,1078753,1078760,1078768,1082284,1082971,1083016,1083027,1083255,1083912,1302416);
Then set it to update:
warptool -dbname gpc1 -setskyfiletoupdate -set_label update.OSS -warp_id 1477774
That solved the majority, 139/141, issues. Two issues remain: 1) is a fault 4 so just revert it 2) the other is a fault 3, but has the same issues as the above, where the state is full but it is failing to find a specific log file. So -> set to clean -> set to update
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1079367 warptool -dbname gpc1 -setskyfiletoupdate -set_label update.OSS -warp_id 1079367
Might require more investigation...
Broken warp in update.OSS by JRF
The broken warp in update.OSS wants ippb04...so will need to put that up into repair and revert the warptool job.
Maybe Rob job not all good by JRF
Have a look at ippitc email about pending stamps:
1 MOPS 918892 weryk_5C95386D 2019-03-22 20:19:52 25 217487 00:10:09 2019-03-22 20:30:01 1 MOPS 918892 weryk_5C95386D 2019-03-22 20:19:52 25 217488 00:10:09 2019-03-22 20:30:01 1 MOPS 918892 weryk_5C95386D 2019-03-22 20:19:52 25 217490 00:10:09 2019-03-22 20:30:01 2 MOPS 918892 weryk_5C95386D 2019-03-22 20:19:52 25 217508 00:10:09 2019-03-22 20:30:01
Check 'em oot:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (217487,217488,217490,217508); +-------------+--------+---------+--------------+------------+---------+-------------------+------------+ | exp_name | exp_id | chip_id | chip_label | chip_state | warp_id | warp_label | warp_state | +-------------+--------+---------+--------------+------------+---------+-------------------+------------+ | o5440g0432o | 217487 | 1267031 | goto_cleaned | cleaned | 1214718 | MD09.PV3.20141104 | full | | o5440g0433o | 217488 | 1267032 | goto_cleaned | cleaned | 1214735 | MD09.PV3.20141104 | full | | o5440g0436o | 217490 | 1267034 | goto_cleaned | cleaned | 1214706 | MD09.PV3.20141104 | full | | o5440g0453o | 217508 | 1262954 | goto_cleaned | cleaned | 1213228 | MD09.PV3.20141104 | full | +-------------+--------+---------+--------------+------------+---------+-------------------+------------+
The warps look good..., so maybe no problem here
Still a bunch of pending things according to this:
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple | grep "new warp" | cut -f4 -d" " | awk '{ print " warptool -dbname gpc1 -listrun -simple -warp_id "$1}' | tcsh
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple | grep "new warp" | cut -f4 -d" " | awk '{ print " warptool -dbname gpc1 -listrun -simple -warp_id "$1}' | tcsh | awk '{ print "chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id "$23" ; warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id "$1 }'
The later updates both runs, aka:
chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id xxxxxx warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id xxxxxx
Saturday : 2019.03.23
- 11:11 EAM : checking that I can edit a page on the wiki
- MEH: ~half ganglia not reporting in -- due to something happening yesterday ~1800 HST -- manually restarting gmond restores reporting, John doing primary nodes in processing, ippx left for Roger and Thomas to try
- ipp134-139, ippb11,b19-b23 were reporting down on ganglia page, but were actually already reporting normally (2 instances...) -- gmetad restart on ippops2 was needed to clear, something glitched the system
ganglia demon failed across the cluster by JRF
JRF: Had to restart it on a whole load of machines... MEH suggested:
sudo /etc/init.d/gmond status sudo /etc/init.d/gmond restart
on the newer ubuntu machines ipp127+:
sudo service ganglia-monitor status sudo service ganglia-monitor restart
Though the above didn't always work on the ubuntu machines
ps_ud_MOPS error by JRF
o6473g0463o XY44 627000 1347469 update ps_ud_MOPS LAP.PV3.20140730.bigmem.20141220 LAP.ThreePi 3 PS1 GPC1 Tried to revert, it failed. Then tried to update, it also failed.
So I'm just going to rerun the exposure:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190323 -set_reduction LAP_SCIENCE -exp_name 627000
Check it's done
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name="o6473g0463o";
Then clean up the old stuff
pstamp problems by JRF
10960392 new warp 1999761 skycell.0751.033 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918963 ps_ud_MOPS F 0 0 MOPS 918963 /data/ipp113.0/pstamp/work/2019/03/24/918963/checkdep.10960392.log 500 10960394 new warp 1999761 skycell.0751.034 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918963 ps_ud_MOPS F 0 0 MOPS 918963 /data/ipp113.0/pstamp/work/2019/03/24/918963/checkdep.10960394.log 500 10960485 new warp 1875836 skycell.1195.025 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918965 ps_ud_MOPS F 0 0 MOPS 918965 /data/ipp113.0/pstamp/work/2019/03/24/918965/checkdep.10960485.log 500 10960487 new warp 1875812 skycell.1195.025 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918965 ps_ud_MOPS F 0 0 MOPS 918965 /data/ipp113.0/pstamp/work/2019/03/24/918965/checkdep.10960487.log 500 10960522 new warp 1712458 skycell.1547.065 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918965 ps_ud_MOPS F 0 0 MOPS 918965 /data/ipp113.0/pstamp/work/2019/03/24/918965/checkdep.10960522.log 500 SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (1999761,1875836,1875812,1712458); +-------------+---------+---------+------------+------------+---------+-----------------+------------+ | exp_name | exp_id | chip_id | chip_label | chip_state | warp_id | warp_label | warp_state | +-------------+---------+---------+------------+------------+---------+-----------------+------------+ | o7487g0126o | 1073157 | 1765399 | ps_ud_MOPS | update | 1712458 | ps_ud_MOPS | update | | o7869g0337o | 1236447 | 1929751 | ps_ud_MOPS | update | 1875812 | ps_ud_MOPS | update | | o7869g0356o | 1236465 | 1929770 | ps_ud_MOPS | update | 1875836 | update_recovery | update | | o8185g0309o | 1379681 | 2057059 | ps_ud_MOPS | update | 1999761 | ps_ud_MOPS | update | +-------------+---------+---------+------------+------------+---------+-----------------+------------+
I'm just gonna try and update them, see what happens.
chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1765399 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1929751 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1929770 chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 2057059 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1712458 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1875812 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1875836 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1999761
It worked!
Sunday : 2019.03.24
chips stuck in full
There are a bunch of chips and warps stuck in the full state...
SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipProcessedImfile.fault != 0 AND chipRun.label like 'update.LAP.PV3' AND chipRun.state LIKE "full" LIMIT 80 OFFSET 0; +-------------+--------+---------+-------+----------------+-------+--------+ | exp_name | exp_id | chip_id | state | label | fault | camera | +-------------+--------+---------+-------+----------------+-------+--------+ | o5252g0508o | 141142 | 1047387 | full | update.LAP.PV3 | 2 | GPC1 | | o5653g0315o | 320147 | 1047394 | full | update.LAP.PV3 | 2 | GPC1 | | o6396g0082o | 599426 | 1047433 | full | update.LAP.PV3 | 2 | GPC1 | | o6412g0186o | 607383 | 1047440 | full | update.LAP.PV3 | 2 | GPC1 | | o6741g0472o | 720298 | 1047444 | full | update.LAP.PV3 | 2 | GPC1 | | o6741g0491o | 720316 | 1047445 | full | update.LAP.PV3 | 2 | GPC1 | | o6782g0167o | 732865 | 1047446 | full | update.LAP.PV3 | 2 | GPC1 | | o6782g0200o | 732898 | 1047451 | full | update.LAP.PV3 | 2 | GPC1 | | o6782g0200o | 732898 | 1047451 | full | update.LAP.PV3 | 2 | GPC1 | | o5297g0292o | 156364 | 1047462 | full | update.LAP.PV3 | 2 | GPC1 | | o5306g0280o | 159686 | 1047465 | full | update.LAP.PV3 | 2 | GPC1 | | o5653g0302o | 320134 | 1047468 | full | update.LAP.PV3 | 2 | GPC1 | | o6001g1268o | 465334 | 1047470 | full | update.LAP.PV3 | 2 | GPC1 | | o6001g1269o | 465335 | 1047471 | full | update.LAP.PV3 | 2 | GPC1 | | o6001g1269o | 465335 | 1047471 | full | update.LAP.PV3 | 2 | GPC1 | | o6001g1286o | 465347 | 1047472 | full | update.LAP.PV3 | 2 | GPC1 | | o6001g1287o | 465350 | 1047473 | full | update.LAP.PV3 | 2 | GPC1 | | o6013g0292o | 469066 | 1047474 | full | update.LAP.PV3 | 2 | GPC1 | | o6013g0294o | 469068 | 1047475 | full | update.LAP.PV3 | 2 | GPC1 | +-------------+--------+---------+-------+----------------+-------+--------+
Attempt to clean...
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047387 . chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047475
then update...
chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047387 . chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047475
HOWEVER, this failed and I'm not sure why...
So there are now some fault 2 entries with goto_cleaned... query this to get them:
SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipRun.chip_id IN (1047387,1047394,1047433,1047440,1047444,1047445,1047446,1047451,1047462,1047465,1047468,1047470,1047471,1047472,1047473,1047474,1047475);
DEFERED UNTIL NEXT WEEK!
Remaining chip issues: by JRF
o5280g0301o XY01 151113 1059611 update update.LAP.PV3 o5280g0317o XY01 151129 1059616 update update.LAP.PV3 o5280g0316o XY01 151128 1059817 update update.LAP.PV3 o6091g0094o XY22 498248 1060186 update update.LAP.PV3
They all complain about a missing mdc config file:
WARNING: Config dump file neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.ppImage.mdc is missing. Using current recipes and file rules.
For the first 3 it also complains about burntool state not matching that in the chip_imfile.pl
Sent them to revert...perhaps they will clear themselves up (update: they did not, tune in to Monday's czar log to find out what happened)
warps:
SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault,rawExp.camera FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.label like 'update.LAP.PV3' LIMIT 80 OFFSET 0; +-------------+--------+---------+---------+------------------+--------+----------------+-------+--------+ | exp_name | exp_id | chip_id | warp_id | skycell_id | state | label | fault | camera | +-------------+--------+---------+---------+------------------+--------+----------------+-------+--------+ | o6013g0286o | 469059 | 1047397 | 1004746 | skycell.0840.097 | full | update.LAP.PV3 | 2 | GPC1 | | o6782g0167o | 732865 | 1047446 | 1004795 | skycell.0841.016 | full | update.LAP.PV3 | 2 | GPC1 | | o6782g0200o | 732898 | 1047451 | 1004800 | skycell.0840.017 | full | update.LAP.PV3 | 2 | GPC1 | | o5297g0307o | 156380 | 1047463 | 1004812 | skycell.0759.058 | full | update.LAP.PV3 | 2 | GPC1 | | o5649g0216o | 318039 | 1048195 | 1003988 | skycell.0839.048 | update | update.LAP.PV3 | 4 | GPC1 | +-------------+--------+---------+---------+------------------+--------+----------------+-------+--------+
To solve this, we will have to put the warps to clean:
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id XXXXX
check that is has cleaned successfully by querying the database e.g.
SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault,rawExp.camera FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.warp_id IN (1004746,1004795,1004800,1004812,1003988);
Then set it to update:
warptool -dbname gpc1 -setskyfiletoupdate -set_label update.OSS -warp_id XXXXX
