IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:PS1_IPP_Czarlog_20190510

Version 5 (modified by tdeboer, 7 years ago) ( diff )

--

PS1 IPP Czar Logs for the week 2019.05.10 - 2019.05.16

    1. PS1 IPP Czar Logs for the week 2019.05.10 - 2019.05.16
      1. Friday : 2019.05.10
      2. Saturday : 2019.05.11
      3. Sunday : 2019.05.12
      4. Monday : 2019.05.13
      5. Tuesday : 2019.05.14
      6. Wednesday : 2019.05.15
      7. Thursday : 2019.05.16

(Up to PS1 IPP Czar Logs)

Friday : 2019.05.10

CZAR HANDOVER by CCL

  • Haydn mentioned he will check them on 5/13 (remind him maybe)
  • ipp128: Device: /dev/bus/0 [megaraid_disk_15], Read SMART Self-Test Log Failed
  • ipp139: Device: /dev/bus/0 [megaraid_disk_37], Read SMART Self-Test Log Failed
  • ippc72 is down: details mentioned above, not sure what the next action will be (another hardware issue)
  • ippc129, ippc135, ippc145, ippc149, ippc150: drive issue
  • update.LAP.PV3: 15k warp prcessing jobs on ippmonitor, some of them might have been processed to be "full" state by queue their data_state in the warpSkyfile table. Their data quality is not equal to 0 which may affect the state keep "update".
  • QUB have put in a lot of stamp requests, you may need to restart pstamp a couple more times today (when jobs done >100k).
  • these huge jobs may cause ipp restart unsuccessfully (stdscience happened twice), make sure all server are running by check_system.sh
  • TdB: Taking over as czar. Following the weekly IPP meeting, it was mentioned having too many jobs queued for updates might lead to issues in communication between the servers and the pantasks. Potentially, this is affecting the stdscience restart that Roger mentioned. To assess the situation, I removed 10k warp updates from current processing to a hold label, which will be fed back in as processing continues: 
    mysql -hX -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.LAP.PV3" and warpRun.state like "update" group by warp_id limit 10000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3.lazy -warp_id "$2 }' | tcsh
    I took the opportunity to restart the stdscience pantask after clearing them as well. Restart went fine, and check_system shows it is running fine as well.
  • TdB: Around 18:00 we hit a situation where chips were still updating but warps were not. I did a bit of digging: 
    mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "update" group by chip_id limit 500; '

There are a lot of exps that have corresponding warps in the hold label, so the ones currently in warp update are waiting on their chip updates to finish. Send some of the warps in the hold label back to the regular label if their corresponding chip state is already in full, they can run through: 

mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "full" and warpRun.label LIKE "update.LAP.PV3.lazy" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$2 }' | tcsh

Saturday : 2019.05.11

  • TdB: One exposure of the night was ruined, which caused chunk OSSR.R13S1.12.Q.i to be incomplete. Check the status and see if a desperate diff needs to be made: 
    mysql -hscidbm -uX -pX gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk LEFT JOIN ((SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id) UNION (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp2=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R13S1.12.Q.i%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id)) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.object;"
    which returns: 
    chunk	object	warp1	warp2	warp3	warp4	diff1	diff2	diff3	diff4	workdir	label	data_group	reduction
OSSR.R13S1.12.Q.i	ps1_28_0298	2107392	2107411	2107431	2107450	1764479	1764479	1764537	1764537	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0299	2107384	2107405	2107423	2107444	1764467	1764467	1764525	1764525	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0303	2107399	2107418	2107438	2107456	1764488	1764488	1764546	1764546	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0304	2107390	2107408	2107428	2107449	1764473	1764473	1764536	1764536	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0305	2107388	2107407	2107426	2107446	1764472	1764472	1764528	1764528	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0310	2107397	2107416	2107436	2107454	1764487	1764487	1764543	1764543	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0391	2107383	2107404	2107425	2107443	1764468	1764468	1764519	1764519	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0399	2107400	2107421	2107441	2107459	1764497	1764497	1764556	1764556	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0400	2107389	2107414	2107430	2107451	1764485	1764485	1764538	1764538	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0401	2107391	2107410	2107429	2107448	1764478	1764478	1764531	1764531	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0402	2107382	2107403	0	2107442	1764462	1765059	0	1765059	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0403	2107385	2107402	2107420	2107440	1764461	1764461	1764520	1764520	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0410	2107398	2107419	2107437	2107458	1764495	1764495	1764554	1764554	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0412	2107395	2107412	2107434	2107453	1764480	1764480	1764544	1764544	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0413	2107393	2107415	2107432	2107452	1764486	1764486	1764539	1764539	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0414	2107387	2107409	2107424	2107445	1764474	1764474	1764527	1764527	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0415	2107386	2107406	2107427	2107447	1764469	1764469	1764532	1764532	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0416	2107381	2107401	2107422	2107439	1764457	1764457	1764516	1764516	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0426	2107396	2107417	2107435	2107457	1764493	1764493	1764547	1764547	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
OSSR.R13S1.12.Q.i	ps1_28_0428	2107394	2107413	2107433	2107455	1764484	1764484	1764545	1764545	neb://@HOST@.0/gpc1/20190511	OSS.nightlyscience	OSS.20190511	SWEETSPOT
    In fact, it has already been diffed, since we see two separate diff_ids for this obejct, diff_id=1764462 (made from 2107382 and 2107403) and diff_id=1765059 (made from 2107403 and 2107442). I am guessing the nightly_science.pl script kicked in here, at the end of the night.

----------------

There are some warps complaining about missing .mk files. First set them to update to see if that fixes the problem: 

warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292074
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292075
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1292079

failing that, send to reprocess: 

chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6259g0268o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6259g0280o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6315g0144o

and clean up their chip and warp ID: 

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342029 
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342030 
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1342034 

warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292074
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292075
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1292079

--------------------

There are also some I/O relatd errors with ippb18, with fault codes 3 and 4. Judging by Ganglia, it is seeing some fairly heavy usage.

send the exps to revert, but put ippb18 into repair for the time being to give it a break. 

neb-host --state repair --host ippb18 --note 'TdB: up->repair, due to it getting hammered and producing I/O errors'

Sunday : 2019.05.12

Monday : 2019.05.13

Tuesday : 2019.05.14

Wednesday : 2019.05.15

Thursday : 2019.05.16

Note: See TracWiki for help on using the wiki.