| Version 22 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.05.03 - 2019.05.09
(Up to PS1 IPP Czar Logs)
Friday : 2019.05.03
pstamp requests by JRF
Received some error emails about pstamp requests. checking out the pstamp server we can see that over 800,000 jobs have been submitted by QUB.
The errors come from a script run by the crontab on ippc26 (this is mentioned in the emails). The script checks if pstamp has over 100,000 jobs done as this is were pantasks becomes flaky.
Checking out the status of the pstamp pantasks keep an eye on:
request.finish.run
as we DO NOT want to restart if this is running as it will cause funkiness.
So let's restart pstamp, doing the usual:
pantasks> stop pantasks> shutdown now --check that the jobs have finished first ~ippitc/pstamp> pantasks_server & pantasks_client pantasks> server input input pantasks> setup pantasks> run
I have added ippc76 to the hosts_apache group in the ~ippitc/ippconfig/pantasks_hosts.input file. This is so that when the nodes are allocated that ippc76 will be turned off when the apache group is removed.
Since over 800,000 requests were made the pstamp pantasks will need restarting about every hour or so.
- JRF: ippc72 has died, sent an email to Haydn, left the PDU off. Also commented it out of the
~ippitc/ippconfig/nebservers.cshfile.
CZAR HANDOVER by JRF
- ipp139 has started having issues so there will likely be some liaising with hardware people to fix that.
- ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
- 'wait' state: the past week has seen some chip states set to 'wait' which is not a valid state. They can appear in IPP processing at times, and the quickest way to fix them is: change label + imfile update:
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id XXXXXX chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id XXXXXX
the update second part is needed as the change from wait to update isn't usually recognized. - More updates will need to be queued, details of how to do this are given in the Thursday 2019-05-02 czarlog
- Guide for desperate diffs is in the czarlog for last week. HOWEVER, TdB likely has a better way of doing it by now
- QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).
CCL took over after 2pm
- CCL: need to modify /home/panstarrs/ipp/local/bin/apachedisk_chk.sh as ipp on ippc19 to solve the crontab error message, i.e., remove 72 & add 76 for host list.
ssh: connect to host ippc72 port 22: No route to host /home/panstarrs/ipp/local/bin/apachedisk_chk.sh: line 20: [: : integer expression expected /home/panstarrs/ipp/local/bin/apachedisk_chk.sh: line 30: [: : integer expression expected
- CCL: 20:30 pstamp over 100k jobs --> restart
ipp@ippc19 ==> ippc76 Host key verification failed. remove known_hosts related to ippc19, i.e., remove this line "10.10.20.74 ssh-rsa xxxxxx"
ipp107 disk usage > 97% , set it to be repair to avoid data storing. neb-host --state repair --host ipp107 --note 'CCL: up->repair: disk full > 97%'
o5567g0384o XY64 274464 1505781 update update.LAP.PV3 LAP.PV3.20140730.ipp.20150211 LAP.ThreePi 2 PS1 GPC1 2011-01-06 15:42:26 171.377302 29.393080 ps1_11_0821 y.00000 30.00 1.0360 119.31 12.36 2176 5.38 135.3PI.y.QE-1 ps1_11_0821 visit 1
neb://ipp051.0/gpc1/20110106/o5567g0384o/o5567g0384o.ota64.burn.tbl size 0
ipp_apply_burntool_fix.pl --exp_name o5567g0384o --class_id XY64 --verbose --dbname gpc1
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1505781
chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1505781
Saturday : 2019.05.04
- CCL: login to ipp113 as ippitc, but got this error
/usr/bin/xauth: /data/ippc64.1/ippitc/.Xauthority not writable, changes will be ignored /usr/bin/xauth: /data/ippc64.1/ippitc/.Xauthority not writable, changes ignored can be solved by removing .Xauthority file mv .Xauthority to .Xauthority.20190504
- CCL: 12:00 feed update.LAP.PV3 3000 jobs
Sunday : 2019.05.05
- CCL: fix some "wait" state and 0 size burntool files
o5501g0025o XY32 248673 1260880 wait update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5501g0048o XY32 248696 1260884 wait update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5501g0029o XY65 248677 1260949 wait update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5571g0094o XY32 278832 1286390 wait update.LAP.PV3 LAP.PV3.20140730.20141127 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5453g0489o XY65 225170 1334081 wait update.LAP.PV3 LAP.PV3.20140730.20141216 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5546g0268o XY53 265358 1343253 wait update.LAP.PV3 LAP.PV3.20140730.20141218 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5450g0473o XY64 223534 1344130 wait update.LAP.PV3 LAP.PV3.20140730.20141218 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5450g0473o XY65 223534 1344130 wait update.LAP.PV3 LAP.PV3.20140730.20141218 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o7003g0073o XY45 835825 1354236 wait update.LAP.PV3 LAP.PV3.20140730.20141222 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o7003g0073o XY46 835825 1354236 wait update.LAP.PV3 LAP.PV3.20140730.20141222 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1260880 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1260884 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1260949 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1286390 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1334081 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1343253 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1344130 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1344130 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1354236 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1354236 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1260880 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1260884 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1260949 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1286390 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1334081 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1343253 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1344130 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1344130 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1354236 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1354236 o5378g0584o XY56 190307 1276729 update update.LAP.PV3 LAP.PV3.20140730.20141125 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5378g0584o XY57 190307 1276729 update update.LAP.PV3 LAP.PV3.20140730.20141125 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5478g0182o XY57 236958 1277511 update update.LAP.PV3 LAP.PV3.20140730.20141125 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5478g0216o XY56 236993 1277514 update update.LAP.PV3 LAP.PV3.20140730.20141125 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5478g0208o XY56 236986 1277690 update update.LAP.PV3 LAP.PV3.20140730.20141125 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5478g0213o XY56 236990 1277717 update update.LAP.PV3 LAP.PV3.20140730.20141125 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5480g0418o XY56 238251 1277735 update update.LAP.PV3 LAP.PV3.20140730.20141125 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5491g0418o XY57 243569 1278795 update update.LAP.PV3 LAP.PV3.20140730.20141126 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5487g0313o XY56 242341 1282224 update update.LAP.PV3 LAP.PV3.20140730.20141126 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5566g0202o XY56 273184 1282460 update update.LAP.PV3 LAP.PV3.20140730.20141126 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 Image burntool version does not match current accepted version. at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834 ipp_apply_burntool_fix.pl --exp_name o5378g0584o --class_id XY56 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5378g0584o --class_id XY57 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5478g0182o --class_id XY57 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5478g0216o --class_id XY56 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5478g0208o --class_id XY56 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5478g0213o --class_id XY56 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5480g0418o --class_id XY56 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5491g0418o --class_id XY57 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5487g0313o --class_id XY56 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5566g0202o --class_id XY56 --dbname gpc1 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1276729 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1277511 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1277514 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1277690 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1277717 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1277735 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1278795 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1282224 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1282460 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1276729 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1277511 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1277514 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1277690 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1277717 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1277735 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1278795 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1282224 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1282460
- CCL: 17:30 The stdscience server shows not running when I use check_system.sh to check the status. So I mannually restart it, but the log shows it has been running since 15:30.
Monday : 2019.05.06
- TdB: update regarding desperate diffs and testing:
We want a testcase that does not involve live testing. Therefore, make use of the daily testquad. I ran a separate reduction of the quad under the label tdbdifftest. This results in the following exposures:
Exp Name Exp ID Chip ID Cam ID Fake ID Warp ID state label data grp Date/Time Object Comment o6370g0351o 590994 2158115 2116160 2085063 2091717 full tdbdifftest tdbdifftest.20190503 2013-03-19 11:37:26 ps1_14_2124 162.OSS.O.Q.w ps1_14_2124 visit 1 o6370g0373o 591016 2158118 2116155 2085069 2091722 full tdbdifftest tdbdifftest.20190503 2013-03-19 11:59:29 ps1_14_2124 162.OSS.O.Q.w ps1_14_2124 visit 2 o6370g0395o 591038 2158116 2116151 2085064 2091718 full tdbdifftest tdbdifftest.20190503 2013-03-19 12:21:39 ps1_14_2124 162.OSS.O.Q.w ps1_14_2124 visit 3 o6370g0417o 591060 2158117 2116154 2085067 2091720 full tdbdifftest tdbdifftest.20190503 2013-03-19 12:43:52 ps1_14_2124 162.OSS.O.Q.w ps1_14_2124 visit 4
We can then make the regular diffs with those:difftool -dbname gpc1 -definewarpwarp -warp_id 2091717 -template_warp_id 2091722 -backwards -set_workdir neb://@HOST@.0/gpc1/tdbdifftest.20190503 -set_dist_group NULL -set_label tdbdifftest -set_data_group tdbdifftest.20190503 -set_reduction SWEETSPOT -simple -rerun difftool -dbname gpc1 -definewarpwarp -warp_id 2091718 -template_warp_id 2091720 -backwards -set_workdir neb://@HOST@.0/gpc1/tdbdifftest.20190503 -set_dist_group NULL -set_label tdbdifftest -set_data_group tdbdifftest.20190503 -set_reduction SWEETSPOT -simple -rerun
resulting in the following exposures:diff_ID state label data grp dist grp Magicked tess ID both ways Time registered reduction workdir 1759513 full tdbdifftest tdbdifftest.20190503 NULL 0 RINGS.V3 1 2019-05-04 00:17:41 SWEETSPOT neb://@HOST@.0/gpc1/tdbdifftest.20190503 1759514 full tdbdifftest tdbdifftest.20190503 NULL 0 RINGS.V3 1 2019-05-04 00:17:42 SWEETSPOT neb://@HOST@.0/gpc1/tdbdifftest.20190503
Now then, we can simulate losing a warp exposure by changing its label and doing query only for the tdbdifftest label (in warpRun and diffRun in this case). In this case, lose the visit1 warp and the v1-v2 diff:warptool -dbname gpc1 -updaterun -set_label tdbdifftest1 -warp_id 2091717 difftool -dbname gpc1 -updaterun -set_label tdbdifftest1 -diff_id 1759513
Then, run a query to get the information:mysql -hX -uX -pippuser gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND warpRun.label LIKE 'tdbdifftest' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND diffRun.label LIKE 'tdbdifftest' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND diffRun.label LIKE 'tdbdifftest' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.object;"
which returns:chunk object warp1 warp2 warp3 warp4 diff1 diff2 diff3 diff4 workdir label data_group reduction 162.OSS.O.Q.w ps1_14_2124 0 2091722 2091718 2091720 0 0 1759514 1759514 neb://@HOST@.0/gpc1/20130319 tdbdifftest tdbdifftest.20190503 SWEETSPOT
That shows clearly that 1 warp is missing and two visits have been diffed. Running the desperate diffs creation script with the recipe from the PSNC_MOPS wiki page:mysql -hX -uX -pippuser gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND warpRun.label LIKE 'tdbdifftest' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND diffRun.label LIKE 'tdbdifftest' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND diffRun.label LIKE 'tdbdifftest' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.object;" | grep -v object | awk '{nwarp=0; if($3!=0) nwarp++; if($4!=0) nwarp++; if($5!=0) nwarp++; if($6!=0) nwarp++ ; ndiff=0; if($7!=0) ndiff++; if($8!=0) ndiff++; if($9!=0) ndiff++; if($10!=0) ndiff++ ; if(nwarp == 3 && nwarp !=ndiff) {if($7==0 && $3==0){w1=$4; w2=$5} else if($8==0 && $4==0){w1=$3; w2=$5} else if($9==0 && $5==0){w1=$4; w2=$6} else if($10==0 && $6==0){w1=$4; w2=$5}; print "difftool -dbname gpc1 -definewarpwarp -warp_id "w1" -template_warp_id "w2" -backwards -set_workdir "$11" -set_dist_group NULL -set_label "$12" -set_data_group "$13" -set_reduction "$14" -simple -rerun -set_note desperate diff -pretend";}}'and returns:difftool -dbname gpc1 -definewarpwarp -warp_id 2091722 -template_warp_id 2091718 -backwards -set_workdir neb://@HOST@.0/gpc1/20130319 -set_dist_group NULL -set_label tdbdifftest -set_data_group tdbdifftest.20190503 -set_reduction SWEETSPOT -simple -rerun -set_note desperate diff -pretend
This is the desired results!
----------------------------
Now, we know this is not something that happens in really. The first diffable pair will always be diffed. Simulate that instead by leaving the warpID label as is (first visit missing), but putting both diffRun labels to off. Then, make a v2-v3 diff:
difftool -dbname gpc1 -definewarpwarp -warp_id 2091722 -template_warp_id 2091718 -backwards -set_workdir neb://@HOST@.0/gpc1/tdbdifftest.20190503 -set_dist_group NULL -set_label tdbdifftest -set_data_group tdbdifftest.20190503 -set_reduction SWEETSPOT -simple -rerunThen, we have the following diffs:
diff_ID state label data grp dist grp Magicked tess ID both ways Time registered reduction workdir 1759513 full tdbdifftest1 tdbdifftest.20190503 NULL 0 RINGS.V3 1 2019-05-04 00:17:41 SWEETSPOT neb://@HOST@.0/gpc1/tdbdifftest.20190503 1759514 full tdbdifftest1 tdbdifftest.20190503 NULL 0 RINGS.V3 1 2019-05-04 00:17:42 SWEETSPOT neb://@HOST@.0/gpc1/tdbdifftest.20190503 1760928 full tdbdifftest tdbdifftest.20190503 NULL 0 RINGS.V3 1 2019-05-06 21:16:16 SWEETSPOT neb://@HOST@.0/gpc1/tdbdifftest.20190503Now run the query script, which returns:
chunk object warp1 warp2 warp3 warp4 diff1 diff2 diff3 diff4 workdir label data_group reduction 162.OSS.O.Q.w ps1_14_2124 0 2091722 2091718 2091720 0 1760928 1760928 0 neb://@HOST@.0/gpc1/20130319 tdbdifftest tdbdifftest.20190503 SWEETSPOTThis looks like what we expect. Run the desperate diff creation:
mysql -hX -uX -pippuser gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND warpRun.label LIKE 'tdbdifftest' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND diffRun.label LIKE 'tdbdifftest' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND diffRun.label LIKE 'tdbdifftest' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.object;" | grep -v object | awk '{nwarp=0; if($3!=0) nwarp++; if($4!=0) nwarp++; if($5!=0) nwarp++; if($6!=0) nwarp++ ; ndiff=0; if($7!=0) ndiff++; if($8!=0) ndiff++; if($9!=0) ndiff++; if($10!=0) ndiff++ ; if(nwarp == 3 && nwarp !=ndiff) {if($7==0 && $3==0){w1=$4; w2=$5} else if($8==0 && $4==0){w1=$3; w2=$5} else if($9==0 && $5==0){w1=$4; w2=$6} else if($10==0 && $6==0){w1=$4; w2=$5}; print "difftool -dbname gpc1 -definewarpwarp -warp_id "w1" -template_warp_id "w2" -backwards -set_workdir "$11" -set_dist_group NULL -set_label "$12" -set_data_group "$13" -set_reduction "$14" -simple -rerun -set_note desperate diff -pretend";}}'which returns:
difftool -dbname gpc1 -definewarpwarp -warp_id 2091722 -template_warp_id 2091718 -backwards -set_workdir neb://@HOST@.0/gpc1/20130319 -set_dist_group NULL -set_label tdbdifftest -set_data_group tdbdifftest.20190503 -set_reduction SWEETSPOT -simple -rerun -set_note desperate diff -pretendSo, the 'usual' recipe does not work in all cases. i.e. it works off of a set strategy but does not check against existing (and therefore missing) diffs. For now, we can adjust the fixed recipe based on the fact that we know the first diffable pair will always be diffed. This affects the case of v1 or v2 being missing. v1 missing results in v2-v3 diff being made and v3-v4 being needed; v2 missing results in v1-v3 being made and v3-v4 being needed. Try this instead:
mysql -hX -uX -pippuser gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND warpRun.label LIKE 'tdbdifftest' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND diffRun.label LIKE 'tdbdifftest' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND diffRun.label LIKE 'tdbdifftest' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.object;" | grep -v object | awk '{nwarp=0; if($3!=0) nwarp++; if($4!=0) nwarp++; if($5!=0) nwarp++; if($6!=0) nwarp++ ; ndiff=0; if($7!=0) ndiff++; if($8!=0) ndiff++; if($9!=0) ndiff++; if($10!=0) ndiff++ ; if(nwarp == 3 && nwarp !=ndiff) {if($7==0 && $3==0){w1=$5; w2=$6} else if($8==0 && $4==0){w1=$5; w2=$6} else if($9==0 && $5==0){w1=$4; w2=$6} else if($10==0 && $6==0){w1=$4; w2=$5}; print "difftool -dbname gpc1 -definewarpwarp -warp_id "w1" -template_warp_id "w2" -backwards -set_workdir "$11" -set_dist_group NULL -set_label "$12" -set_data_group "$13" -set_reduction "$14" -simple -rerun -set_note desperate diff -pretend";}}'which returns:
difftool -dbname gpc1 -definewarpwarp -warp_id 2091718 -template_warp_id 2091720 -backwards -set_workdir neb://@HOST@.0/gpc1/20130319 -set_dist_group NULL -set_label tdbdifftest -set_data_group tdbdifftest.20190503 -set_reduction SWEETSPOT -simple -rerun -set_note desperate diff -pretendThat now works. But clearly, want to make an explicit check against the diff_ids available.
In conclusion, to create the correct desperate diffs for a given chunk, run:
mysql -hX -uX -pippuser gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R10S1.14.Q.w%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R10S1.14.Q.w%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R10S1.14.Q.w%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.object;" | grep -v object | awk '{nwarp=0; if($3!=0) nwarp++; if($4!=0) nwarp++; if($5!=0) nwarp++; if($6!=0) nwarp++ ; ndiff=0; if($7!=0) ndiff++; if($8!=0) ndiff++; if($9!=0) ndiff++; if($10!=0) ndiff++ ; if(nwarp == 3 && nwarp !=ndiff) {if($7==0 && $3==0){w1=$5; w2=$6} else if($8==0 && $4==0){w1=$5; w2=$6} else if($9==0 && $5==0){w1=$4; w2=$6} else if($10==0 && $6==0){w1=$4; w2=$5}; print "difftool -dbname gpc1 -definewarpwarp -warp_id "w1" -template_warp_id "w2" -backwards -set_workdir "$11" -set_dist_group NULL -set_label "$12" -set_data_group "$13" -set_reduction "$14" -simple -rerun -set_note desperate diff -pretend";}}' | bashor for a whole night, using:
mysql -hX -uX -pippuser gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.chunk,selchunk.object;" | grep -v object | awk '{nwarp=0; if($3!=0) nwarp++; if($4!=0) nwarp++; if($5!=0) nwarp++; if($6!=0) nwarp++ ; ndiff=0; if($7!=0) ndiff++; if($8!=0) ndiff++; if($9!=0) ndiff++; if($10!=0) ndiff++ ; if(nwarp == 3 && nwarp !=ndiff) {if($7==0 && $3==0){w1=$5; w2=$6} else if($8==0 && $4==0){w1=$5; w2=$6} else if($9==0 && $5==0){w1=$4; w2=$6} else if($10==0 && $6==0){w1=$4; w2=$5}; print "difftool -dbname gpc1 -definewarpwarp -warp_id "w1" -template_warp_id "w2" -backwards -set_workdir "$11" -set_dist_group NULL -set_label "$12" -set_data_group "$13" -set_reduction "$14" -simple -rerun -set_note desperate diff -pretend";}}' | bash
- CCL: ippb04 & ippb05 the nebulous rsyncs and re-inserts are complete for all three partitions on ippb05 as well as the .0 and .1 partitions on ippb04. So we set the xattr value to 5 i.e., machine is retired.
neb-voladm --vname ippb04.0 --xattr 5 --user nebulous --db nebulous --host ippdb09 --pass xxxx neb-voladm --vname ippb04.1 --xattr 5 --user nebulous --db nebulous --host ippdb09 --pass xxxx neb-voladm --vname ippb04.2 --xattr 5 --user nebulous --db nebulous --host ippdb09 --pass xxxx neb-voladm --vname ippb05.0 --xattr 5 --user nebulous --db nebulous --host ippdb09 --pass xxxx neb-voladm --vname ippb05.1 --xattr 5 --user nebulous --db nebulous --host ippdb09 --pass xxxx neb-voladm --vname ippb05.2 --xattr 5 --user nebulous --db nebulous --host ippdb09 --pass xxxx neb-host --state down --host ippb04 --note 'CCL: repair->down rsyncs and re-inserts are complete, set xattr to 5 and down' neb-host --state down --host ippb05 --note 'CCL: repair->down rsyncs and re-inserts are complete, set xattr to 5 and down'
- CCL: " ippc90/Current Load is CRITICAL " Not sure what cause many jobs stick on ippc90, as well as pantasks jobs on stdscience. I tried to restart the machine by "shutdown -r now", but it did not work. So I use the console to power off and on and it gets back to normal.
- CCL: Haydn is fixing OS drive fail on ippb17
neb-host --state down --host ippb17 --note 'CCL: up->down: Haydn is fixing OS drive fail'
- CCL:
o6169g0117o XY52 516240 1334425 update update.LAP.PV3 LAP.PV3.20140730.20141216 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o6624g0245o XY26 680033 1336054 update update.LAP.PV3 LAP.PV3.20140730.bigmem.20141217 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5546g0268o XY53 265358 1343253 update update.LAP.PV3 LAP.PV3.20140730.20141218 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o6336g0059o XY25 580411 1363142 update update.LAP.PV3 LAP.PV3.20140730.20141225 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 error message --------------------- log filename: /data/ipp137.1/nebulous/be/4e/11808459694.gpc1:LAP.PV3.20140730:2014:12:16:o6169g0117o.516240:o6169g0117o.516240.ch.1334425.XY52.log.update WARNING: Config dump file neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/12/16/o6169g0117o.516240/o6169g0117o.516240.ch.1334425.XY52.ppImage.mdc is missing. Using current recipes and file rules. Previous log file moved to neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/12/16/o6169g0117o.516240/o6169g0117o.516240.ch.1334425.XY52.log.update.20190506T081425 Starting script /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl on ippc106 FULL COMMAND: /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl --threads 4 --exp_id 516240 --chip_id 1334425 --chip_imfile_id 79890507 --class_id XY52 --uri neb://ipp033.0/gpc1/20120830/o6169g0117o/o6169g0117o.ota52.fits --camera GPC1 --run-state update --deburned 0 --outroot neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/12/16/o6169g0117o.516240/o6169g0117o.516240.ch.1334425 --redirect-output --reduction LAP_SCIENCE --dbname gpc1 --verbose PSF file neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/12/16/o6169g0117o.516240/o6169g0117o.516240.ch.1334425.XY52.psf is missing at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 653 main::rerun_photometry('neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/12/16/o6169g0117o.5...', 'XY52') called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 233 Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/whichnode neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/12/16/o6169g0117o.516240/o6169g0117o.516240.ch.1334425.XY52.psf]... ipp069.0 available PSPHOT.PSF.SAVE is missing but we cannot regenerate it at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834 main::my_die('PSPHOT.PSF.SAVE is missing but we cannot regenerate it', 516240, 1334425, 'XY52', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 683 main::rerun_photometry('neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/12/16/o6169g0117o.5...', 'XY52') called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 233 --------------------------- try revert chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 not help, try clean chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1334425 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1334425 not help, try burntool fix ipp_apply_burntool_fix.pl --exp_name o6169g0117o --class_id XY52 --dbname gpc1 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1334425 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1334425 not help, try reprocessing chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o6169g0117o then clean chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1363142 PS. MEH suggested to try neb-mv *.psf to *.bad next time. burntool file 0 size ---------- o5507g0209o XY34 251629 1393384 update update.LAP.PV3 LAP.PV3.20140730.20150107 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5507g0209o --class_id XY34 --dbname gpc1 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1393384 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1393384
Tuesday : 2019.05.07
- CCL 12:30 : try to find warp_id and corresponding chip_id to see their chip state
SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,camRun.cam_id,fakeRun.fake_id,warpRun.warp_id,warpRun.state,warpRun.label,warpRun.data_group \ FROM warpRun, fakeRun, camRun, chipRun, rawExp \ WHERE warpRun.fake_id = fakeRun.fake_id \ AND fakeRun.cam_id = camRun.cam_id \ AND camRun.chip_id = chipRun.chip_id \ AND chipRun.exp_id = rawExp.exp_id \ AND warpRun.state like 'update' \ AND warpRun.label like 'update.LAP.PV3' order by exp_name limit 10 mysql xxx -B -e 'SELECT chipRun.chip_id FROM warpRun, fakeRun, camRun, chipRun, rawExp \ WHERE warpRun.fake_id = fakeRun.fake_id \ AND fakeRun.cam_id = camRun.cam_id \ AND camRun.chip_id = chipRun.chip_id \ AND chipRun.exp_id = rawExp.exp_id \ AND warpRun.state like "update" \ AND warpRun.label like "update.LAP.PV3" \ AND chipRun.state not like "update" | \ sed 's/^/chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id /' > wait-update.sh && sed 's/^/chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id /' > setimfiletoupdate.sh ** might not be all at "wait" state with "chipRun.state not like update"
- CCL: errors show again after running the script above.
o6169g0117o XY52 516240 1334425 update update.LAP.PV3 LAP.PV3.20140730.20141216 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o6624g0245o XY26 680033 1336054 update update.LAP.PV3 LAP.PV3.20140730.bigmem.20141217 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5546g0268o XY53 265358 1343253 update update.LAP.PV3 LAP.PV3.20140730.20141218 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1334425 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1336054 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1343253
- CCL: burntool file 0 size
o5456g0424o XY62 226855 1260660 update update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5456g0424o XY63 226855 1260660 update update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5456g0425o XY62 226856 1260661 update update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5456g0425o XY63 226856 1260661 update update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5456g0450o XY62 226881 1260662 update update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5456g0450o XY63 226881 1260662 update update.LAP.PV3 LAP.PV3.20140730.20141120 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 ** burntool 0 size file issue ipp_apply_burntool_fix.pl --exp_name o5456g0424o --class_id XY62 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5456g0424o --class_id XY63 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5456g0425o --class_id XY62 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5456g0425o --class_id XY63 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5456g0450o --class_id XY62 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5456g0450o --class_id XY63 --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5500g0566o --class_id XY34 --dbname gpc1 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1260660 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1260661 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1260662 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1260660 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1260661 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1260662
machine retired
- CCL 14:00: EAM demonstrates how to manually remove ippb05.0, ippb05.1, and ipp054 - ipp066 (excluding ipp064) from mountedvol table.
mysql -h ippdb09 -unebulous -pxxx mountedvol select * from mountedvol where name = 'ippxxx' delete from mountedvol where name = 'ippxxx' mysql -h ippdb08 -uipp -pxxx czardb select * from hosts where host = 'ippxxx' delete from hosts where host = 'ippxxx' also ipp067 works fine now. neb-host --state up --host ipp067 --note 'CCL: repair->up, well behavior after repair'
Wednesday : 2019.05.08
- MEH: running test diffs for MOPS in ippmops/stdsicence using m0,m1,c2 group nodes
- CCL: Some warp_id have "update" state on ippmonitor, however they have been processed to be "full" state by checking their data_state in the warpSkyfile table. Their data quality is not equal to 0 which may affect updating the "state".
select skycell_id,data_state,warp_id,quality,fault from warpSkyfile where warp_id in (1153778,1153779,1339669,1153780,1153781,1154735,1154736,1154793,1210215) group by quality; +------------------+------------+---------+---------+-------+ | skycell_id | data_state | warp_id | quality | fault | +------------------+------------+---------+---------+-------+ | skycell.1415.004 | full | 1210215 | 257 | 0 | | skycell.1325.094 | full | 1210215 | 3006 | 0 | | skycell.2448.080 | full | 1153778 | 8007 | 0 | +------------------+------------+---------+---------+-------+ 3 rows in set (0.00 sec)
Thursday : 2019.05.09
- CCL: recheck repaired hosts
neb-host | grep '1 0 1 0' these keep repairing ipp032 ipp032.0 2019-02-20 15:36:41 used by MOPS for stamps ipp064 ipp064.0 2019-02-19 14:08:46 should always be in repair until further notice ipp083 ipp083.0 2019-02-19 14:23:50 TdB: bad BBU since May 2018 state so slow WT state; up->repair ipp087 ipp087.0 2019-02-19 14:15:06 TdB: bad BBU state on 02-19-2019 so slow WT state; up->repair ipp091 ipp091.0 2019-04-01 15:04:01 down->repair: back to repair after failed BBU and high load neb-host --state repair --host ipp091 --note 'CCL: repair: need to replace the battery 20190508' ipp096 ipp096.0 2019-04-27 07:55:27 JRF: up -> repair: disk is getting quite full ipp112 ipp112.0 2019-02-08 14:09:48 JRF: disk getting full so moving to repair: up->repair ipp114 ipp114.0 2019-02-08 14:10:28 JRF: disk getting full so moving to repair: up->repair ipp121 ipp121.0 2017-12-12 06:48:46 vol .1 log reset ipp121 ipp121.1 2017-12-12 06:48:46 vol .1 log reset ipp139 ipp139.0 2019-05-02 22:31:34 JRF: up -> repair: smartd daemon complained about failures, putting to reapir as precaution ipp139 ipp139.1 2019-05-02 22:31:34 JRF: up -> repair: smartd daemon complained about failures, putting to reapir as precaution these can be set up ipp070 ipp070.0 2019-04-03 09:49:50 TdB: up->repair, to take the load off of many many data requests neb-host --state up --host ipp070 --note 'CCL: repair->up: Battery relearn completed 20190411' ipp073 ipp073.0 2019-03-12 16:40:00 TdB: up->repair: put to repair after a RAID failure. neb-host --state up --host ipp073 --note 'CCL: repair->up: RAID rebuilt 20190312' ipp076 ipp076.0 2019-04-06 09:43:16 TdB: up->repair, LSI alert and RAID degraded neb-host --state up --host ipp076 --note 'CCL: repair->up: it was LSI alert and RAID degraded 20190406' ipp084 ipp084.0 2019-02-08 14:05:52 MEH: bad BBU so slow WT state; up->repair since 2017-08-03 at least neb-host --state up --host ipp084 --note 'CCL: repair->up: Battery relearn completed 20190426' ipp097 ipp097.0 2019-02-26 11:21:39 JRF: controller battery failed so changing from up->repair neb-host --state up --host ipp097 --note 'CCL: repair->up: last battery alert was on 04-15' ipp106 ipp106.1 2019-04-16 07:07:39 CCL: up->repair : disk getting full so moving to repair ipp107 ipp107.0 2019-05-03 20:47:16 CCL: up->repair: disk full > 97% ipp107 ipp107.1 2019-05-03 20:47:16 CCL: up->repair: disk full > 97% neb-host --state up --host ipp106 --note 'CCL: repair->up: full disk alert clear' neb-host --state up --host ipp107 --note 'CCL: repair->up: full disk alert clear' neb-host --state repair --host ipp107 --note 'CCL: up->repair: disk full > 99%' ipp112 ipp112.1 2019-04-10 11:13:54 JRF: up -> repair: disk is getting full so putting in repair neb-host --state up --host ipp112.1 --volume --note 'CCL: repair->up: full disk alert clear' ipp113 ipp113.0 2018-06-18 17:10:49 turn on since shuffle and dvo calibration are complete neb-host --state up --host ipp113.0 --volume --note 'CCL: repair->up: turn on since shuffle and dvo calibration are complete 2018-06-18' ipp114 ipp114.1 2019-02-14 15:10:03 CCL: disk over 97% used so moving to repair: up->repair neb-host --state up --host ipp114.1 --volume --note 'CCL: repair->up: full disk alert clear'
Note:
See TracWiki
for help on using the wiki.
