| Version 9 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.03.18 - 2019.03.24
(Up to PS1 IPP Czar Logs)
Czar of the week: John, with MEH as czar-lord (previously: TdB)
Monday : 2019.03.18
- TdB: In the morning (05:45), I noticed ippb18 was reported as down by nagios. When trying to get more information, I noticed neb-host was reporting too many connections and no machines where showing up in IPP monitor (all greyed out). Following a quick back and forth with Gene I then set all pantasks to stop using:
check_system.sh stop
Furthermore, the compute nodes ippc71-75 were showing huge load on ganglia, so we turned off apache on those machines, using:/etc/init.d/apache2 stop
as root on each of the machines.
- Following this, we (John, Roger, Gene, myself) all came in at 09:00 to diagnose and fix the issue. First, we looked at the dump of processlist on ippc75:
mysql -h localhost -u root -e "show processlist" > neb.proc.20190318
This showed 64 connections from ippc76, which is not on the list of nebservers (see ~ippitc/ippconfig/nebservers.csh) and also showed soem connections from ippc28. Looking at the configuration file /etc/apache2/modules.d/apache2-mod_perl-startup.pl showed 256 MaxClients connections set up for the compute nodes. Machine ippc76 has StartServers = 64, so it will start with normally 64 connections, but could go up to 256 total. Combining all machines, this results in a max of 1349 connections, which is a lot.
Looking at the max connections in MYSQL using:
select @@max_connections;shows that the maximum is set to 1024, which is less than 1349, so that could be a reason for the failure. We are setting that higher now, to avoid this issue:
SET GLOBAL max_connections = 2048 (in mysql)
The full reason for the break is not totally clear, but the huge load spikes on the apache servers roughly coincide with ippb18 going down, so that might well be related.
For now, we are setting the apache servers back to on, using:
/etc/init.d/apache2 startas root on each of the machines.
Finally, the pantasks, are set back to run using:
check_system.sh run
- MEH: bulk updates now continuing again in ~ippmops/stdscience
- MEH: sending more chips to cleanup for warps in full state
- ipp091 excess load from being harassed from cleanup and failed BBU, leaving in neb-host repair now
- MEH: ippb04, b05 causing many i/o faults from the rsyncs -- set neb-host down from repair since many primary copies of files there... -- will need to temporarily set back to repair to clear other faults for the cases where files are only on those machines
JRF: (afternoon czar formation ft. MEH)
The recipes directory contains a bunch of configurations:
ippitc/src/ipp-20170121/ippconfig/recipes
We'll look at the reductionClasses.mdc config file:
Contains the configuration for how to process something. In this case the CHIP_PPIMAGE stage is set to CHIP_NOMASKTEST instead of the usual CHIP.
# test for MOPS request reprocessing without masking
NOMASKTEST METADATA
CHIP_PPIMAGE STR CHIP_NOMASKTEST
.
.
.
BACKGROUND_PSWARP STR BACKGROUND
PSVIDEOPHOT STR PSVIDEOPHOT
END
There are separate configs for each camera (go up a directory then into the relevant camera dir). There are a bunch of configs here are empty in the main config, but have entries for the separate cameras. This is because data from each camera should be treated separately.
WATCH OUT: If you reprocess something with a different config then the stamp server will use whatever the latest config was for a request! Therefore, it's best to do some crazy reprocessing locally, so as not to pollute the database with entries that could confuse the current stamp request system.
Time to reprocess some stuff for Rob with no masks (as seen on PSNSC_MOPS page and czar log from 2019.02.18 - 2019.02.24):
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0726o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0707o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0688o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7403g0669o
As you can see we're using the NOMASKTEST reduction that we just had a look at the config for.
Don't forget to add the label to pantasks to make it run!
Time to fix some stuck jobs! Some files are needed from ippb05, but it has been misbehaving and was set to down ealier today. We shall quickly put it to repair so that the files can be accessed:
neb-host ippb05 repair --note "JRF: down -> repair: so that some broken updates can clear on ippmonitor"
Then we shall revert the chip for the stuck job so that it can continue:
chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1
You don't need to specify the chip_id, it'll just revert all faults that are stuck.
If you come across a stamp that you suspect maybe be stuck, have a look at it using:
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
remove '-simple' to get some more information.
JRF (evening): Jobs approaching 100k, so restarting stdscience pantasks. A bunch of failed jobs are beginning to accumulate too, but I'll save that fun for the morning.
Tuesday : 2019.03.19
JRF (morning): Jobs approaching 100k, so restarting stdscience pantasks. Flicked ippb04 and ippb05 to repair, from down, so that some files could be accessed to clear the backlog of tasks.
JRF (afternoon) - czar formation ft. MEH:
Maybe tomorrow ecliptic.RP
Weird warp fault where the state is full:
o7071g0124o 875375 1530574 1497865 1471180 1477774 skycell.1618.088 full update.OSS
Query the neb-workdir (note the wild card at the end):
neb-ls -p -l neb://ipp069.0/gpc1/OSS.nt/2015/02/18//o7071g0124o.875375/o7071g0124o.875375.wrp.1477774.skycell.1618.088%
shows us that one of the mask files is in a funny state where is 0 size:
-rw-rw-r-- 1 ippitc users 0 Mar 8 09:27 /data/ipp125.0/nebulous/9f/e9/11203362808.gpc1:OSS.nt:2015:02:18:o7071g0124o.875375:o7071g0124o.875375.wrp.1477774.skycell.1618.088.mask.fits
To solve this, we will have to put the warp to clean:
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1477774
check that is has cleaned succesfully (query the database). Then set it to update:
warptool -dbname gpc1 -setskyfiletoupdate -set_label update.OSS -warp_id 1477774
Solving warp issues Move the stuff that is currently broken under the label 'update.OSS' to the 'update.OSS.broke' label. To figure out which ones need to be moved have a look at the state, fault, and skyfile
Will need to add this new label to pantasks so it can get priority and actually work on them.
The following shows the query used, and how to pipe it to 'awks' so that it can run the chiptool/warptool command to update the labels on the chip/warp id pairs.
mysql -uXXXXX -pXXXXX -hippdb08 gpc1 -B --skip-column-names -e
'SELECT chip_id,
warp_id,
warpRun.label,
warpRun.state,
warpSkyfile.fault,
chipRun.label,
chipRun.state
FROM warpRun
JOIN warpSkyfile USING (warp_id)
JOIN fakeRun USING (fake_id)
JOIN camRun USING (cam_id)
JOIN chipRun USING (chip_id)
WHERE warpRun.label LIKE "update.OSS"
AND warpSkyfile.fault > 0
AND warpRun.state LIKE "update"
GROUP BY warp_id;'
| awk '{print"
chiptool -dbname gpc1 -updaterun -chip_id "$1" -state update -set_label update.OSS.broke -pretend;
warptool -dbname gpc1 -updaterun -warp_id "$2" -state update -set_label update.OSS.broke -pretend
"}'
| xargs -0 bash -c
Note: The above its formatted for readability, you'll have to squish it into one line, fill in the XXXXXs, and remove pretend.
Next, add the label to stdscience (pantasks> add.label update.OSS.broke)
Then put the b nodes up to repair to help clear things out.
JRF (late evening): Gave pantasks a restart as it was close to 100k jobs
Wednesday : 2019.03.20
MEH moved across the remaining update.OSS.broke labels.
The SVN server died!
May want to figure out which exposures might have already been reprocessed too, as these ones do not need to be done. we are going to add the LAP PV3 label
There are some broken stamps that Rob has requested.
count label req_id name timestamp fault exp_id dtime utc_timestamp() 1 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 466358 00:04:05 2019-03-20 22:20:01 2 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 466379 00:04:05 2019-03-20 22:20:01 2 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 705006 00:04:05 2019-03-20 22:20:01 2 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 705027 00:04:05 2019-03-20 22:20:01 1 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 705045 00:04:05 2019-03-20 22:20:01 1 MOPS 918829 weryk_5C92B8D3 2019-03-20 22:15:56 25 705068 00:04:05 2019-03-20 22:20:01
Time to take the exp_id and have a look:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (466358,466379,705006,705027,705045,705068);
There are a bunch of cleaned ones but the most recent ones have the label update.hold20190318
Moved them to the ps_ud_MOPS label e.g.
chiptool -dbname gpc1 -updaterun -chip_id 428036 -state update -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 387740 -state update -set_label ps_ud_MOPS
And then set them to update:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 428036
