| Version 23 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.05.10 - 2019.05.16
(Up to PS1 IPP Czar Logs)
Friday : 2019.05.10
CZAR HANDOVER by CCL
- Haydn mentioned he will check them on 5/13 (remind him maybe)
- ipp128: Device: /dev/bus/0 [megaraid_disk_15], Read SMART Self-Test Log Failed
- ipp139: Device: /dev/bus/0 [megaraid_disk_37], Read SMART Self-Test Log Failed
- ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
- ippc129, ippc135, ippc145, ippc149, ippc150: drive issue
- update.LAP.PV3: 15k warp prcessing jobs on ippmonitor, some of them might have been processed to be "full" state by queue their data_state in the warpSkyfile table. Their data quality is not equal to 0 which may affect the state keep "update".
- QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).
- these huge jobs may cause ipp restart unsuccessfully (stdscience happened twice), make sure all server are running by check_system.sh
- TdB: Taking over as czar. Following the weekly IPP meeting, it was mentioned having too many jobs queued for updates might lead to issues in communication between the servers and the pantasks. Potentially, this is affecting the stdscience restart that Roger mentioned. To assess the situation, I removed 10k warp updates from current processing to a hold label, which will be fed back in as processing continues:
mysql -hX -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.LAP.PV3" and warpRun.state like "update" group by warp_id limit 10000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3.lazy -warp_id "$2 }' | tcshI took the opportunity to restart the stdscience pantask after clearing them as well. Restart went fine, and check_system shows it is running fine as well.
- TdB: Around 18:00 we hit a situation where chips were still updating but warps were not. I did a bit of digging:
mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "update" group by chip_id limit 500; '
There are a lot of exps that have corresponding warps in the hold label, so the ones currently in warp update are waiting on their chip updates to finish. Send some of the warps in the hold label back to the regular label if their corresponding chip state is already in full, they can run through:
mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "full" and warpRun.label LIKE "update.LAP.PV3.lazy" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$2 }' | tcsh
Saturday : 2019.05.11
- TdB: One exposure of the night was ruined, which caused chunk OSSR.R13S1.12.Q.i to be incomplete. Check the status and see if a desperate diff needs to be made:
mysql -hscidbm -uX -pX gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.object;"which returns:chunk object warp1 warp2 warp3 warp4 diff1 diff2 diff3 diff4 workdir label data_group reduction OSSR.R13S1.12.Q.i ps1_28_0298 2107392 2107411 2107431 2107450 1764479 1764479 1764537 1764537 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0299 2107384 2107405 2107423 2107444 1764467 1764467 1764525 1764525 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0303 2107399 2107418 2107438 2107456 1764488 1764488 1764546 1764546 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0304 2107390 2107408 2107428 2107449 1764473 1764473 1764536 1764536 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0305 2107388 2107407 2107426 2107446 1764472 1764472 1764528 1764528 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0310 2107397 2107416 2107436 2107454 1764487 1764487 1764543 1764543 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0391 2107383 2107404 2107425 2107443 1764468 1764468 1764519 1764519 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0399 2107400 2107421 2107441 2107459 1764497 1764497 1764556 1764556 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0400 2107389 2107414 2107430 2107451 1764485 1764485 1764538 1764538 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0401 2107391 2107410 2107429 2107448 1764478 1764478 1764531 1764531 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0402 2107382 2107403 0 2107442 1764462 1765059 0 1765059 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0403 2107385 2107402 2107420 2107440 1764461 1764461 1764520 1764520 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0410 2107398 2107419 2107437 2107458 1764495 1764495 1764554 1764554 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0412 2107395 2107412 2107434 2107453 1764480 1764480 1764544 1764544 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0413 2107393 2107415 2107432 2107452 1764486 1764486 1764539 1764539 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0414 2107387 2107409 2107424 2107445 1764474 1764474 1764527 1764527 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0415 2107386 2107406 2107427 2107447 1764469 1764469 1764532 1764532 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0416 2107381 2107401 2107422 2107439 1764457 1764457 1764516 1764516 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0426 2107396 2107417 2107435 2107457 1764493 1764493 1764547 1764547 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT OSSR.R13S1.12.Q.i ps1_28_0428 2107394 2107413 2107433 2107455 1764484 1764484 1764545 1764545 neb://@HOST@.0/gpc1/20190511 OSS.nightlyscience OSS.20190511 SWEETSPOT
In fact, it has already been diffed, since we see two separate diff_ids for this obejct, diff_id=1764462 (made from 2107382 and 2107403) and diff_id=1765059 (made from 2107403 and 2107442). I am guessing the nightly_science.pl script kicked in here, at the end of the night.
----------------
There are some warps complaining about missing .mk files. First set them to update to see if that fixes the problem:
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292074 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292075 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292079failing that, send to reprocess:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6259g0268o chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6259g0280o chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6315g0144oand clean up their chip and warp ID:
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342029 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342030 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342034 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292074 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292075 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292079
--------------------
There are also some I/O related errors with ippb18, with fault codes 3 and 4. Judging by Ganglia, it is seeing some fairly heavy usage. Send the exps to revert, but put ippb18 into repair for the time being to give it a break.
neb-host --state repair --host ippb18 --note 'TdB: up->repair, due to it getting hammered and producing I/O errors'
--------------------- fixing some errors. send to clean and back
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1407389 chiptool -dbname gpc1 -updaterun -set_label ps_ud_WEB -chip_id 1407389 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1407389
--------------------- night: ippc112 is not behaving well. It appears to be down, and repeated power cycling is not helping. Mark removed it from processing and I notified Haydn about it.
Sunday : 2019.05.12
- TdB: Machine ippb18 is now down on ganglia (though no email from nagios) and the console display log is listing EDAC errors. I put it to neb-host down and powered it off in the console (it was already reported as down anyways). Then, I emailed Haydn about it.
Fixing some missing .mk files again:
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1295348Didn't work. send to reprocess:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5231g0024o chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1344272 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1295348
Also some corrupted memory blocks that need reprocessing:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6235g0387o chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6284g0320o chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1364687 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1413646 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1334220 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1363203
*TdB: 18:00 we have finished going through the chip exposure that Roger queued up, but there are still ~6k warps that are lagging behind and are imbalanced. In principle it would be ok to run with just warps updating to remove the imbalance, but might as well feed in a small batch of chip+warps anyway, since warp run at a higher rate than chips. But unsure which data_group we were doing, since I could find no info on the czarlog from last week about which daa_groups Roger queued up. Looking at the query John provided we should be at data_group = LAP.PV3.20140730.20150% (but please correct me if I am wrong). Therefore, setting a batch of 1000 exp to update, using:
mysql -hscidbm -uX -pX gpc1 -B -e ' select exp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) where warpRun.data_group like "LAP.PV3.20140730.20150%" group by warp_id ; ' | grep -v exp_id | awk '{ print "mysql -hscidbm -u ippuser -pippuser gpc1 -B -e \" select s.exp_name,s.label,s.data_group,s.chip_id,s.warp_id,s.state from ( select chip_id,warp_id,exp_name,camRun.label,camRun.data_group,warpRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_id="$1" order by warp_id desc ) as s limit 1 ;\" " }' | tcsh | grep -v exp_name | grep LAP.PV3.2014 | grep cleaned | awk '{print "chiptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -chip_id "$4" ; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id "$4" ;warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$5"; warptool -dbname gpc1 -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$5 }' | head -n 1000 | tcsh
Monday : 2019.05.13
*TdB: Check on the disk space of some of the machines, following cleanup:
Machines ipp123-126 have filled up their disks. Putting them to repair to prevent more data going there. Also ipp110.0 is full
neb-host --state repair --host ipp123 --note 'TdB: up->repair, disk is getting very full' neb-host --state repair --host ipp124 --note 'TdB: up->repair, disk is getting very full' neb-host --state repair --host ipp125 --note 'TdB: up->repair, disk is getting very full' neb-host --state repair --host ipp126 --note 'TdB: up->repair, disk is getting very full' neb-host --state repair --host ipp110 --note 'TdB: up->repair, disk is getting very full'But the recent cleanup has freed enoguh space on ipp106 and ipp107 to allow them to go back to up.
neb-host --state up --host ipp106 --note 'TdB: repair->up, disk space freed in cleanup' neb-host --state up --host ipp107 --note 'TdB: repair->up, disk space freed in cleanup'
- MEH: cleanup chip updates example
mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) where warpRun.label like "update.%" and warpRun.data_group like "%" and warpRun.state like "full" and chipRun.state like "full" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -state full -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1 }' mysql -hippdb08 -uX -pX gpc1 -B -e ' select count(chip_id) from chipRun where state like "goto_cleaned" and label like "goto_cleaned"; select count(warp_id) from warpRun where state like "goto_cleaned" and label like "goto_cleaned"; select count(diff_id) from diffRun where state like "goto_cleaned" and label like "goto_cleaned"; select count(stack_id) from stackRun where state like "goto_cleaned" and label like "goto_cleaned"; select count(dist_id) from distRun where state like "goto_cleaned" and label like "goto_cleaned"; '; date
- TdB: afternoon:
Some missing .mk files in the warps. Set them to update the skyfile
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1342050 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1344264 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1358970 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1365663 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1368220
Didn't work. send to reprocess from scratch, instead.
Also some memory blocks that need reverting:
warptool -revertwarped -fault 4 -label update.LAP.PV3 -dbname gpc1
-----------------------
Some of the warp update processing got stuck, probably due to their chip exps being in wait state. Query those and send them to cleanup, followed by an update. Pump them to a file first
mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "wait" group by chip_id; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1"; chiptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -chip_id "$1"; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id "$1 }' > LAP.PV3_waitsThere are 1389 of them. Send them to cleanup and then back to update. That solved most of the problems with the remaining update warps
----------------------
IPP meeting: Cleanup:
We should periodically clean up the chip stage pixels for the updates we are running, to preserve disk space. Use the following query, and queue up a 1000 or so once in a while, or a few thousand overnight:
mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) where warpRun.label like "update.%" and warpRun.data_group like "%" and warpRun.state like "full" and chipRun.state like "full" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -state full -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1 }' | tcshupdate 2019-05-14: I have been steadily cleaning, and there are about 2k left as of right now. So, these will be done before too long, but future czars should keep an eye on them and periodically clean to save space. ----------------------
Haydn wants to swap some drives from ippc17 and ippc90. Use check_system.sh to remove them from processing:
./check_system.sh hostoff ippc17 (do this a bunch of times) ./check_system.sh hostoff ippc90After the swap, put them back up
./check_system.sh hoston ippc17 (do this a bunch of times) ./check_system.sh hoston ippc90
----------------------
Chips with wait states: There is still the ~700k chip exps with a state=wait that need to be dealt with. We can query the labels and data_groups that they occupy, using:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait" GROUP BY data_group,label) as s ORDER by waitct ASC;' mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait" GROUP BY label) as s ORDER by waitct ASC;'The vast majority of them are in fact exps which have a label of goto_clean% which are likely previously cleaned chip exps. However, we cannot simply set their state to cleaned, since we have no way of telling whether the cleanup operation went succesfully or failed. Therefore, we need to send the whole lot back to cleanup. Nonetheless, by parsin the xps, we can preferentially send exps that we still need to use in the update processing before doing things previosuly cleaned. Focus on labels like %LAP.PV3% or %OSS% at first and slowly end them to cleanup in between other cleanup jobs.
I made a quick inventory of the predominant labels that have large (>100) numbers of wait state chip exps:
+--------+-----------------------------------+-----------------------------------+ | waitct | data_group | label | +--------+-----------------------------------+-----------------------------------+ | 104 | EUCLID.MD07.PV0.T1 | EUCLID.MD07.PV0.T1 | | 104 | EUCLID.MD08.PV0.T1 | EUCLID.MD08.PV0.T1 | | 106 | NULL | NULL | | 106 | altaz.20080920 | altaz.20080920 | | 109 | LAP.PV3.20140730.20140918 | goto_cleaned.broke190323 | | 114 | dropped | STS.rp.2013.hold | | 117 | scitest.200807 | scitest.200807 | | 122 | QUB.20160517 | QUB.PS16cgx | | 141 | flatcorr.20081109 | flatcorr.20081109 | | 141 | flatcorr.20100124 | flatcorr.20100124 | | 142 | flattest.20091010 | flattest.20091010 | | 142 | MD03.20091228 | MD03.nightlystack | | 142 | flatcorr.20091003 | flatcorr.20091003 | | 180 | PV2.PHOTO.STD.20141028.i | PV2.PHOTO.STD.20141028 | | 180 | flatcorr.20081107 | flatcorr.20081107 | | 187 | Bright3Pi.20190402 | Bright3Pi.nightlyscience | | 203 | flatcorr.20090509 | flatcorr.20090509 | | 205 | MD07.20091123 | MD07.20091123 | | 209 | OSS.20141207 | goto_cleaned.broke190312 | | 214 | opposition.20080922 | cleanup_testing_old | | 217 | ThreePi.Run2.z.v0 | ThreePi.Run2.z.v0 | | 231 | mops.fixbrokenLAP.20180323 | mopsfix | | 248 | SAS.20151208 | SAS.20151208 | | 248 | czw.SAS.201402114 | lanl.SAS.201402114 | | 264 | ThreePi.Run2.i.v0 | ThreePi.Run2.i.v0 | | 282 | ThreePi.yz.20091203 | ThreePi.yz.20091203 | | 304 | m31.ps1.20170627 | m31.ps1.20170627.wait | | 365 | ThreePi_NorthernRegion.090729 | ThreePi_NorthernRegion.090729 | | 573 | altaz.20080920 | goto_purged | | 596 | LAP.PV3.20140730.20141008 | goto_cleaned.broke | | 641 | scitest.20090327 | scitest.20090327 | | 788 | LAP.PV3.20140730.20150117 | update.LAP.PV3 | | 967 | LAP.PV3.20140730.20140927 | goto_cleaned.redoitchold | | 1252 | m31.ps1.20170627 | m31.ps1.20170627 | | 1756 | LAP.PV3.20140730.20140807 | PV3.cleanup | | 2239 | ThreePi.20140126 | goto_cleaned.redoitc | | 3360 | PV3.Pole.Reprocess.20180510 | PV3.Pole.Reprocess.20180510 | | 15431 | STS.rp.2013.20120412 | STS.rp.2013 | | 51293 | ThreePi.20100220 | goto_cleaned.rerun | | 166252 | scitest.200808 | goto_cleaned_redo | | 443284 | altstrip.200806 | goto_cleaned | +--------+-----------------------------------+-----------------------------------+We can send a data_group to clean up, using:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT chip_id,data_group,label FROM chipRun WHERE state like "wait" and data_group like "LAP.PV3.20140730.20140807";' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1}' | tcsh
- TdB: at around 17:30, the IPPmonitor stalled. Turns out, mysql crashed on ippdb05. Let's check things:
Look at the error log under /var/log/mysql (log into ippdb05 as own user)
190513 17:16:46 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=33554432 read_buffer_size=262144 max_used_connections=15 max_connections=2048 threads_connected=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1605632 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x7f664c587e30 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x7f664c587e30, backtrace may not be correct. Bogus stack limit or frame pointer, fp=0x7f664c587e30, stack_bottom=0x44c90000, thread_stack=262144, aborting backtrace. Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at (nil) is invalid pointer thd->thread_id=4 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. pure virtual method called terminate called without an active exception
So, let's try to restart mysql on ippdb05 and see if ti comes back up. As root, do:
sudo /etc/init.d/mysql start * Starting mysql ... * Strange, the socket file already exist in "/var/run/mysqld/mysqld.sock" * it will be removed now and re-created by the MySQL server * BUT please make your checks. * Starting mysql (/etc/mysql/my.cnf) * MySQL NOT started (0)But, it probably did start. Check, using:
ps aux | grep mysql mysql 27881 93.3 7.5 146558796 14950952 ? Ssl 18:25 0:33 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sockIndeed, it started just fine.
Checking the error log file again shows:
190513 18:26:29 InnoDB: Started; log sequence number 5173 4202608365 190513 18:26:29 [Note] Recovering after a crash using /var/lib/mysql/mysqld-bin 190513 18:26:29 [Note] Starting crash recovery... 190513 18:26:29 [Note] Crash recovery finished. 190513 18:26:29 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem. InnoDB: Rolling back of trx id 2 1776754758 completed 190513 18:26:29 InnoDB: Rollback of non-prepared transactions completed 190513 18:26:29 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.77-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Gentoo Linux mysql-community-5.0.77-r1
------------------------------------------------------
Then we noticed the replication server is also down. the sql slave is not quite working right. On ippdb05, log into mysql as root:
mysql -ulocalhost -uroot -pX show slave status\Gthen, inside mysql do:
STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;That set the replication back up, and seconds behind master started to tick down.
------------------------
On ippdb08 (the primary) go into mysql as root as well, and do:
show processlist;which shows:
| 1636903827 | repl_gpc1 | ippdb05.ifa.hawaii.edu:50012 | NULL | Binlog Dump | 197 | Has sent all binlog to slave; waiting for binlog to be updated | NULLSo, it is probably back up to date
-------------------
It seems there is also a problem on ipp116. The replication slave there seems stuck, and does not respond to a stop slave; inside mysql. shut it down externally, using:
mysqladmin -uroot -pxxx shutdown nowThe mysql.err log shows a normal shutdown, which is good. Checking ps aux | grep mysql shows that the mysql process is still running. IPPmonitor is now unstuck, but shows that ipp116 has a DB connection error. Trying to see how to shut down the process without killing it hard.
sudo /etc/init.d/mysql stop * WARNING: mysql has not yet been started. sudo /etc/init.d/mysql zap * Manually resetting mysql to stopped state.It is still there!
Try this:
sudo mysqladmin -uroot -pXXX shutdown now mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)' Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!It has not updated any files in two hours, since the crash. So, it is likely now a zombie process. Mark tried to kill it nicely and then killed it hard after. Then, we started it anew, using:
sudo /etc/init.d/mysql startThat worked, and the error log looks good. Now we need to restart the replication on this machine.
mysql -ulocalhost -uroot -pXXX show slave status\G STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;
Keep an eye on the seconds behind master, using: show slave status\G After a good 20 minutes, both of them had caught up to the master and things were working well again.
- MEH: starting some older diff cleanup now to help free up space on lower data nodes
Tuesday : 2019.05.14
- TdB: Haydn wants to redistribute the x nodes ippx001-044 to get better temperature behaviour inside the cabinet (they are all adjacent at the top of a half-filled cabinet). remove them from processing:
go to ~ippps2/stdscience and in to pantasks_client
controller status There are 6 sets of them controller host off ippx001 controller host off ippx002 controller host off ippx003 controller host off ippx004 controller host off ippx005 controller host off ippx006 controller host off ippx007 controller host off ippx008 controller host off ippx009 controller host off ippx010 controller host off ippx011 controller host off ippx012 controller host off ippx013 controller host off ippx014 controller host off ippx015 controller host off ippx016 controller host off ippx017 controller host off ippx018 controller host off ippx019 controller host off ippx020 controller host off ippx021 controller host off ippx022 controller host off ippx023 controller host off ippx024 controller host off ippx025 controller host off ippx026 controller host off ippx027 controller host off ippx028 controller host off ippx029 controller host off ippx030 controller host off ippx031 controller host off ippx032 controller host off ippx033 controller host off ippx034 controller host off ippx035 controller host off ippx036 controller host off ippx037 controller host off ippx038 controller host off ippx039 controller host off ippx040 controller host off ippx041 controller host off ippx042 controller host off ippx043 controller host off ippx044
Machines ipp128 and ipp139 have been set to repair with warnings "Read SMART Self-Test Log Failed" and "smartd daemon complained about failures". According to an email from Haydn, both of these are spurious warnings, and not critical. Therefore, set them back to up.
neb-host --state up --host ipp128 --note 'TdB: repair->up, after being put in repair with spurious errors' neb-host --state up --host ipp139 --note 'TdB: repair->up, after being put in repair with spurious errors'
Feeding the update beast more I have continued to feed updates to the update beast. However, it was not very clear to me what the current and upcoming order of things is. therefore, I tried to write up a (quick) query for checking the number of exposures that 'need doing' within %LAP.PV3% labels. To be clear, this means the number of exposures with a camRun.label like %LAP.PV3% for which the most recent (re-)processing is in a %cleaned or update state. The query is:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT count(*),gather.data_group,gather.label FROM (SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.state,subb.data_group,subb.label FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.label LIKE "%LAP.PV3%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state,warpRun.data_group,camRun.label FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.label NOT LIKE "%broke%" AND warpRun.label NOT LIKE "%hold%" AND rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id)) as gather WHERE (state LIKE "%cleaned" OR state LIKE "update") GROUP BY data_group;'and the resulting list is:
+----------+--------------------------------------------+------------------------------------+ | count(*) | camRun.data_group | camRun.label | +----------+--------------------------------------------+------------------------------------+ | 4 | czw.thilker.RA8.20130228 | czw.thilker | | 1 | LAP.PV3.20140730.20140814 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.20140908 | LAP.PV3.20140730.final | | 2 | LAP.PV3.20140730.20140912 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.20140914 | LAP.PV3.20140730.final | | 5 | LAP.PV3.20140730.20140929 | LAP.PV3.20140730.final | | 12 | LAP.PV3.20140730.20141008 | LAP.PV3.20140730.final | | 20 | LAP.PV3.20140730.20141014 | LAP.PV3.20140730.final | | 10 | LAP.PV3.20140730.20141018 | LAP.PV3.20140730.final | | 2 | LAP.PV3.20140730.20141021 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.20141102 | LAP.PV3.20140730.final | | 67 | LAP.PV3.20140730.20141111 | LAP.PV3.20140730.final | | 39 | LAP.PV3.20140730.20141112 | LAP.PV3.20140730.final | | 209 | LAP.PV3.20140730.20141115 | LAP.PV3.20140730.final | | 17 | LAP.PV3.20140730.20141117 | LAP.PV3.20140730.final | | 2 | LAP.PV3.20140730.20141118 | LAP.PV3.20140730.final | | 3 | LAP.PV3.20140730.20141216 | LAP.PV3.20140730.final | | 2 | LAP.PV3.20140730.20150107 | LAP.PV3.20140730.final | | 11 | LAP.PV3.20140730.20150108 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.20150113 | LAP.PV3.20140730.final | | 74 | LAP.PV3.20140730.20150114 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.20150115 | LAP.PV3.20140730.final | | 105 | LAP.PV3.20140730.20150117 | LAP.PV3.20140730.final | | 463 | LAP.PV3.20140730.20150118 | LAP.PV3.20140730.final | | 26 | LAP.PV3.20140730.20150119 | LAP.PV3.20140730.final | | 60 | LAP.PV3.20140730.20150120 | LAP.PV3.20140730.final | | 111 | LAP.PV3.20140730.20150121 | LAP.PV3.20140730.final | | 1490 | LAP.PV3.20140730.20150122 | LAP.PV3.20140730.final | | 2876 | LAP.PV3.20140730.20150123 | LAP.PV3.20140730.final | | 343 | LAP.PV3.20140730.20150124 | LAP.PV3.20140730.final | | 647 | LAP.PV3.20140730.20150125 | LAP.PV3.20140730.final | | 501 | LAP.PV3.20140730.20150126 | LAP.PV3.20140730.final | | 375 | LAP.PV3.20140730.20150127 | LAP.PV3.20140730.final | | 309 | LAP.PV3.20140730.20150128 | LAP.PV3.20140730.final | | 680 | LAP.PV3.20140730.20150129 | LAP.PV3.20140730.final | | 4550 | LAP.PV3.20140730.20150130 | LAP.PV3.20140730.final | | 1728 | LAP.PV3.20140730.20150131 | LAP.PV3.20140730.final | | 221 | LAP.PV3.20140730.20150201 | LAP.PV3.20140730.final | | 887 | LAP.PV3.20140730.20150202 | LAP.PV3.20140730.final | | 602 | LAP.PV3.20140730.20150203 | LAP.PV3.20140730.final | | 661 | LAP.PV3.20140730.20150204 | LAP.PV3.20140730.final | | 9 | LAP.PV3.20140730.20150205 | LAP.PV3.20140730.final | | 260 | LAP.PV3.20140730.20150206 | LAP.PV3.20140730.final | | 2745 | LAP.PV3.20140730.20150207 | LAP.PV3.20140730.final | | 1705 | LAP.PV3.20140730.20150208 | LAP.PV3.20140730.final | | 997 | LAP.PV3.20140730.20150209 | LAP.PV3.20140730.final | | 758 | LAP.PV3.20140730.20150210 | LAP.PV3.20140730.final | | 1009 | LAP.PV3.20140730.20150211 | LAP.PV3.20140730.final | | 717 | LAP.PV3.20140730.20150212 | LAP.PV3.20140730.final | | 466 | LAP.PV3.20140730.20150213 | LAP.PV3.20140730.final | | 14 | LAP.PV3.20140730.20150214 | LAP.PV3.20140730.final | | 47 | LAP.PV3.20140730.20150215 | LAP.PV3.20140730.final | | 240 | LAP.PV3.20140730.20150216 | LAP.PV3.20140730.final | | 84 | LAP.PV3.20140730.20150217 | LAP.PV3.20140730.final | | 377 | LAP.PV3.20140730.20150218 | LAP.PV3.20140730.final | | 79 | LAP.PV3.20140730.20150219 | LAP.PV3.20140730.final | | 875 | LAP.PV3.20140730.20150220 | LAP.PV3.20140730.final | | 463 | LAP.PV3.20140730.20150221 | LAP.PV3.20140730.final | | 1455 | LAP.PV3.20140730.20150222 | LAP.PV3.20140730.final | | 1330 | LAP.PV3.20140730.20150223 | LAP.PV3.20140730.final | | 129 | LAP.PV3.20140730.20150224 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.bigmem.20141206 | LAP.PV3.20140730.final | | 113 | LAP.PV3.20140730.bigmem.20150101 | LAP.PV3.20140730.final | | 689 | LAP.PV3.20140730.bigmem.20150102 | LAP.PV3.20140730.final | | 3290 | LAP.PV3.20140730.bigmem.20150105 | LAP.PV3.20140730.final | | 1263 | LAP.PV3.20140730.bigmem.20150107 | LAP.PV3.20140730.final | | 894 | LAP.PV3.20140730.bigmem.20150108 | LAP.PV3.20140730.final | | 108 | LAP.PV3.20140730.bigmem.20150109 | LAP.PV3.20140730.final | | 97 | LAP.PV3.20140730.bigmem.20150110 | LAP.PV3.20140730.final | | 110 | LAP.PV3.20140730.bigmem.20150111 | LAP.PV3.20140730.final | | 719 | LAP.PV3.20140730.bigmem.20150112 | LAP.PV3.20140730.final | | 814 | LAP.PV3.20140730.bigmem.20150113 | LAP.PV3.20140730.final | | 1312 | LAP.PV3.20140730.bigmem.20150114 | LAP.PV3.20140730.final | | 793 | LAP.PV3.20140730.bigmem.20150115 | LAP.PV3.20140730.final | | 668 | LAP.PV3.20140730.bigmem.20150116 | LAP.PV3.20140730.final | | 40 | LAP.PV3.20140730.bigmem.20150117 | LAP.PV3.20140730.final | | 1027 | LAP.PV3.20140730.bigmem.20150118 | LAP.PV3.20140730.final | | 2212 | LAP.PV3.20140730.bigmem.20150119 | LAP.PV3.20140730.final | | 831 | LAP.PV3.20140730.bigmem.20150120 | LAP.PV3.20140730.final | | 104 | LAP.PV3.20140730.bigmem.20150121 | LAP.PV3.20140730.final | | 860 | LAP.PV3.20140730.bigmem.20150122 | LAP.PV3.20140730.final | | 871 | LAP.PV3.20140730.bigmem.20150123 | LAP.PV3.20140730.final | | 355 | LAP.PV3.20140730.bigmem.20150124 | LAP.PV3.20140730.final | | 6383 | LAP.PV3.20140730.bigmem.20150125 | LAP.PV3.20140730.dupe | | 378 | LAP.PV3.20140730.bigmem.20150128 | LAP.PV3.20140730.final | | 2927 | LAP.PV3.20140730.bigmem.20150129 | LAP.PV3.20140730.final | | 552 | LAP.PV3.20140730.bigmem.20150130 | LAP.PV3.20140730.final | | 321 | LAP.PV3.20140730.bigmem.20150131 | LAP.PV3.20140730.final | | 2980 | LAP.PV3.20140730.bigmem.20150202 | LAP.PV3.20140730.final | | 122 | LAP.PV3.20140730.bigmem.20150204 | LAP.PV3.20140730.final | | 1431 | LAP.PV3.20140730.bigmem.20150205 | LAP.PV3.20140730.final | | 74 | LAP.PV3.20140730.bigmem.20150206 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.ipp.20140907 | LAP.PV3.20140730.final | | 6 | LAP.PV3.20140730.ipp.20140908 | LAP.PV3.20140730.final | | 3 | LAP.PV3.20140730.ipp.20140909 | LAP.PV3.20140730.final | | 4 | LAP.PV3.20140730.ipp.20140930 | LAP.PV3.20140730.final | | 2 | LAP.PV3.20140730.ipp.20141106 | LAP.PV3.20140730.final | | 7 | LAP.PV3.20140730.ipp.20141107 | LAP.PV3.20140730.final | | 4 | LAP.PV3.20140730.ipp.20141108 | LAP.PV3.20140730.final | | 2 | LAP.PV3.20140730.ipp.20141117 | LAP.PV3.20140730.dupe | | 1 | LAP.PV3.20140730.ipp.20141118 | LAP.PV3.20140730.final | | 3 | LAP.PV3.20140730.ipp.20141120 | LAP.PV3.20140730.dupe | | 3 | LAP.PV3.20140730.ipp.20141121 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.ipp.20141122 | LAP.PV3.20140730.final | | 2 | LAP.PV3.20140730.ipp.20141126 | LAP.PV3.20140730.final | | 1 | LAP.PV3.20140730.ipp.20150218 | LAP.PV3.20140730.final | | 1 | LAP.PV3W.20150312.20150320 | LAP.PV3W.20150312 | | 19 | LAP.PV3W.20150312.20150329 | LAP.PV3W.20150312 | | 16 | LAP.PV3W.20150312.20150409 | LAP.PV3W.20150312 | | 8 | LAP.PV3W.20150312.20150413 | LAP.PV3W.20150312 | | 810 | LAP.ThreePi.20120706.20120710 | PV2.cleanup | | 796 | LAP.ThreePi.20120706.20120711 | PV2.cleanup | | 120 | LAP.ThreePi.20120706.20120712 | PV2.cleanup | | 30 | LAP.ThreePi.20120706.20121002 | PV2.cleanup | | 2 | LAP.ThreePi.20120706.20121004 | PV2.cleanup | | 3 | LAP.ThreePi.20120706.20121205 | PV2.cleanup | | 4 | LAP.ThreePi.20120706.20130316 | PV2.cleanup | | 7 | LAP.ThreePi.20120706.20130317 | PV2.cleanup | | 81 | LAP.ThreePi.20120706.20130318 | PV2.cleanup | | 11 | LAP.ThreePi.20120706.20130319 | PV2.cleanup | | 33 | LAP.ThreePi.20120706.20130320 | PV2.cleanup | | 6 | MD04.test3pistack.20140217 | MD04.test3pistack.20140217 | | 1 | mops.fixbrokenLAP.20180323 | ws_nightly_update | | 12 | OSS.20140210 | OSS.nightlyscience | | 40 | OSS.20140220 | OSS.nightlyscience | | 32 | OSS.20140222 | OSS.nightlyscience | | 12 | OSS.20140306 | OSS.nightlyscience | | 19 | OSS.20140405 | OSS.nightlyscience | | 15 | OSS.20140417 | OSS.nightlyscience | | 52 | OSS.20140418 | OSS.nightlyscience | | 1 | OSS.20140423 | OSS.nightlyscience | | 30 | OSS.20140504 | OSS.nightlyscience | | 3 | OSS.20140614 | OSS.nightlyscience | | 4 | OSS.20141202 | OSS.nightlyscience | | 8 | OSS.20141206 | OSS.nightlyscience | | 52 | OSS.20141207 | OSS.nightlyscience | | 12 | OSS.20141208 | OSS.nightlyscience | | 24 | OSS.20141215 | OSS.nightlyscience | | 57 | OSS.20141216 | OSS.nightlyscience | | 8 | OSS.20141217 | OSS.nightlyscience | | 45 | OSS.20141230 | OSS.nightlyscience | | 24 | OSS.20150102 | OSS.nightlyscience | | 8 | OSS.20150106 | OSS.nightlyscience | | 20 | OSS.20150112 | OSS.nightlyscience | | 20 | OSS.20150114 | OSS.nightlyscience | | 8 | OSS.20150128 | OSS.nightlyscience | | 4 | OSS.20150130 | OSS.nightlyscience | | 8 | OSS.20150131 | OSS.nightlyscience | | 12 | OSS.20150202 | OSS.nightlyscience | | 273 | PSNSC.qubicref.20171107 | PSNSC.qubicref.20171107 | | 1 | ThreePi.20100314 | ThreePi.nightlyscience | | 16 | ThreePi.20120927 | PV2.cleanup | | 6 | ThreePi.20121103 | PV2.cleanup | | 3 | ThreePi.20121114 | PV2.cleanup | | 14 | ThreePi.20121125 | PV2.cleanup | | 2 | ThreePi.20121126 | PV2.cleanup | | 19 | ThreePi.20121130 | PV2.cleanup | | 12 | ThreePi.20121201 | PV2.cleanup | | 4 | ThreePi.20121209 | PV2.cleanup | | 4 | ThreePi.20121215 | PV2.cleanup | | 22 | ThreePi.20121221 | PV2.cleanup | | 3 | ThreePi.20121222 | PV2.cleanup | | 4 | ThreePi.20121228 | PV2.cleanup | | 6 | ThreePi.20130103 | PV2.cleanup | | 2 | ThreePi.20130116 | PV2.cleanup | | 2 | ThreePi.20130117 | PV2.cleanup | | 2 | ThreePi.20130121 | PV2.cleanup | | 21 | ThreePi.20130122 | PV2.cleanup | | 31 | ThreePi.20130123 | PV2.cleanup | | 4 | ThreePi.20130126 | PV2.cleanup | | 26 | ThreePi.20130203 | PV2.cleanup | | 4 | ThreePi.20130209 | PV2.cleanup | | 18 | ThreePi.20130210 | PV2.cleanup | | 34 | ThreePi.20130221 | PV2.cleanup | | 2 | ThreePi.20130301 | PV2.cleanup | | 2 | ThreePi.20130307 | PV2.cleanup | | 6 | ThreePi.20130319 | PV2.cleanup | | 8 | ThreePi.20130320 | PV2.cleanup | | 3 | ThreePi.20130408 | PV2.cleanup | | 4 | ThreePi.20130409 | PV2.cleanup | | 1 | ThreePi.20130411 | PV2.cleanup | | 14 | ThreePi.20130413 | PV2.cleanup | | 41 | ThreePi.20130417 | PV2.cleanup | | 32 | ThreePi.20130424 | PV2.cleanup | | 15 | ThreePi.20130425 | PV2.cleanup | | 4 | ThreePi.20130503 | PV2.cleanup | | 8 | ThreePi.20130524 | PV2.cleanup | | 1 | ThreePi.20130527 | PV2.cleanup | | 3 | ThreePi.20130612 | PV2.cleanup | | 1 | ThreePi.20131019 | ThreePi.nightlyscience | | 2 | ThreePi.20131024 | ThreePi.nightlyscience | | 2 | ThreePi.20131117 | ThreePi.nightlyscience | | 1 | ThreePi.20131118 | ThreePi.nightlyscience | | 1 | ThreePi.20131119 | ThreePi.nightlyscience | | 12 | ThreePi.20131129 | ThreePi.nightlyscience | | 10 | ThreePi.20131201 | ThreePi.nightlyscience | | 11 | ThreePi.20131202 | ThreePi.nightlyscience | | 11 | ThreePi.20131203 | ThreePi.nightlyscience | | 11 | ThreePi.20131205 | ThreePi.nightlyscience | | 18 | ThreePi.20131206 | ThreePi.nightlyscience | | 3 | ThreePi.20131207 | ThreePi.nightlyscience | | 6 | ThreePi.20131211 | ThreePi.nightlyscience | | 5 | ThreePi.20131212 | ThreePi.nightlyscience | | 4 | ThreePi.20131214 | ThreePi.nightlyscience | | 2 | ThreePi.20131215 | ThreePi.nightlyscience | | 22 | ThreePi.20131218 | ThreePi.nightlyscience | | 16 | ThreePi.20131225 | ThreePi.nightlyscience | | 4 | ThreePi.20131226 | ThreePi.nightlyscience | | 53 | ThreePi.20140104 | ThreePi.nightlyscience | | 4 | ThreePi.20140105 | ThreePi.nightlyscience | | 2 | ThreePi.20140106 | ThreePi.nightlyscience | | 26 | ThreePi.20140111 | ThreePi.nightlyscience | | 6 | ThreePi.20140121 | ThreePi.nightlyscience | | 6 | ThreePi.20140123 | ThreePi.nightlyscience | | 9 | ThreePi.20140124 | ThreePi.nightlyscience | | 6 | ThreePi.20140125 | ThreePi.nightlyscience | | 8 | ThreePi.20140209 | ThreePi.nightlyscience | | 1 | ThreePi.20140225 | ThreePi.nightlyscience | | 12 | ThreePi.20140227 | ThreePi.nightlyscience | | 2 | ThreePi.20140309 | ThreePi.nightlyscience | | 68 | ThreePi.20140317 | ThreePi.nightlyscience | | 4 | ThreePi.20140318 | ThreePi.nightlyscience | | 34 | ThreePi.20140325 | ThreePi.nightlyscience | | 4 | ThreePi.20140405 | ThreePi.nightlyscience | | 54 | ThreePi.20140417 | ThreePi.nightlyscience | | 5 | ThreePi.20140521 | ThreePi.nightlyscience | | 3 | ThreePi.20140616 | ThreePi.nightlyscience | | 1 | ThreePi.20140827 | ThreePi.nightlyscience | | 22 | ThreePi.20141130 | ThreePi.nightlyscience | | 15 | ThreePi.20141201 | ThreePi.nightlyscience | | 8 | ThreePi.20141202 | ThreePi.nightlyscience | | 4 | ThreePi.20141203 | ThreePi.nightlyscience | | 9 | ThreePi.20141207 | ThreePi.nightlyscience | | 7 | ThreePi.20141208 | ThreePi.nightlyscience | | 6 | ThreePi.20141220 | ThreePi.nightlyscience | | 16 | ThreePi.20141221 | ThreePi.nightlyscience | | 8 | ThreePi.20141226 | ThreePi.nightlyscience | | 18 | ThreePi.20141227 | ThreePi.nightlyscience | | 2 | ThreePi.20141229 | ThreePi.nightlyscience | | 2 | ThreePi.DM.20100401 | ThreePi.DM.20100401 | | 3 | TrailFitting.reprocessing.20130731.2013731 | TrailFitting.reprocessing.20130731 | +----------+--------------------------------------------+------------------------------------+
---------------------
Czar meeting at 13:30: We want to test the ippx nodes, now that they have been redistributed within their cabinet. Best way, set them as the only nodes to be used by stdscience for the ~2hours until the daily restart. Look at hosts groups in ippconfig/pantasks_hosts.input to find which host groups are loaded by the regular stdscience pantasks:
macro hosts.stdscience echo hosts.stdscience control run reap hosts.ignore.storage ## 5x, 4x seems to work better, fewer faults MEH 20170316 ## reduce 3x si0, boost 5x ci0+1 while dvo/psps running MEH 20170812 ## adding more power with data nodes s4-8 after large MOPS update testing MEH 180524 for i 0 3 hosts add si0 hosts add s4 hosts add s5 hosts add s6 hosts add s7 hosts add s8 end # for i 0 5 hosts add ci0 hosts add ci1 end # add another ci1 but not ci0 since ippitc+pantasks running there too MEH 180524 hosts add ci1 control run endNow then, to get the x nodes in, we first added the hosts definitions for the ippx nodes to the ippconfig/pantasks_hosts.input file. Then, we restarted the pantasks to load those in, and then use 'hosts off' with the appropriate groups to turn off the current set of nodes. Then, we add hosts groups x0,x1,x2 a total of 10 times, to get their CPUs fully utilised. They are using about 80 percent CPU on ganglia but seem to be behaving alright. Keep an eye on boottime in ganglia, to see if any x nodes randomly reboot, in prep for PS2. Tomorrow morning, add the xnodes back in to test them again, and keep an eye on them.
->The testing made the alarm on one of the PDUs in that cabinet go off, indicating it is drawing too much power. To fight that, we turned machines ippx033-ippx036 back to off in the pantask since that machine is plugged to the offending PDU which has the alarm going off.
Wednesday : 2019.05.15
Thursday : 2019.05.16
Note:
See TracWiki
for help on using the wiki.
