Changes between Version 31 and Version 32 of PS1_IPP_Czarlog_20190510

Timestamp:

May 16, 2019, 1:50:54 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -1206 +1206 @@
   * 
 
-
-
-
+ * TdB: More x nodes testing. No spontaneous reboots overnight, so the nodes are stable in that respect. This time, we will run updates on them for 4 hours and keep nodes ippx029-36 off from the start, and will monitor power draw more often
+{{{
+11:30: turns out Mark just stopped the pantasks to rebuild the ps1 ops tag. So, there was not a good performance measure
+12:30: same thing
+13:00:
+i06pdu2: 
+PDU Current(Normal):  23.8 A    Max Detected: 32.1 A
+Bank(A): Current(Normal):  14.7 A       Max Detected: 17.9 A
+Bank(B): Current(Normal):  9.1 A        Max Detected: 14.7 A
+Temperature-Internal(T1):  49.9°C(121.7°F)      Max Detected: 51.9°C(125.3°F)
+
+i06pdu1:
+PDU Current(Normal):  23.5 A    Max Detected: 28.4 A
+Bank(A): Current(Normal):  13.6 A       Max Detected: 16.1 A
+Bank(B): Current(Normal):  10.1 A       Max Detected: 12.7 A
+Temperature-Internal(T1):  50.9°C(123.5°F)      Max Detected: 53.9°C(128.9°F)
+
+13:30:
+i06pdu2: 
+PDU Current(Normal):  23.3 A    Max Detected: 32.1 A
+Bank(A): Current(Normal):  14.2 A       Max Detected: 17.9 A
+Bank(B): Current(Normal):  9.2 A        Max Detected: 14.7 A
+Temperature-Internal(T1):  50.9°C(123.5°F)      Max Detected: 51.9°C(125.3°F)
+
+i06pdu1:
+PDU Current(Normal):  24.1 A    Max Detected: 28.4 A
+Bank(A): Current(Normal):  14.1 A       Max Detected: 16.1 A
+Bank(B): Current(Normal):  10.0 A       Max Detected: 12.7 A
+Temperature-Internal(T1):  51.9°C(125.3°F)      Max Detected: 53.9°C(128.9°F)
+}}}
+
+ ---------
+
+ Clearing some errors:
+
+ Quite a few burntools:
+{{{
+ipp_apply_burntool_fix.pl --exp_name o5522g0447o --class_id XY65 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5547g0537o --class_id XY65 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5522g0464o --class_id XY65 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5547g0517o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5547g0539o --class_id XY64 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5546g0560o --class_id XY64 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5523g0632o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5548g0515o --class_id XY65 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5548g0537o --class_id XY65 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5548g0535o --class_id XY64 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5548g0319o --class_id XY65 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5548g0346o --class_id XY65 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5502g0531o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5485g0528o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5485g0539o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5500g0685o --class_id XY33 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5547g0139o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5547g0124o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5567g0178o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5557g0484o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5567g0177o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5515g0485o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5547g0533o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5560g0511o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5560g0589o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5560g0549o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5561g0582o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5570g0243o --class_id XY65 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5267g0238o --class_id XY30 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5264g0154o --class_id XY30 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5548g0521o --class_id XY31 --verbose --dbname gpc1
+ipp_apply_burntool_fix.pl --exp_name o5240g0262o --class_id XY65 --verbose --dbname gpc1
+}}}
+ Also, some missing .mk files in the warps. Set them to update the skyfile
+{{{
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1417786
+}}}
+ It didn't work, so reprocess it:
+{{{
+chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5492g0589o 
+
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1464189 
+warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1417786 
+}}}
+ Finally, another PSPHOT.PSF.SAVE is missing errors, which need reprocessing from scratch
+{{{
+chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5500g0685o 
+
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1469468 
+warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1420896 
+}}}
+
+ --------------
+
+ I got fed up with wait states still appearing in the update.LAP.PV3 label, due to the data_group of the chipRun exp that corresponds to the most recent processing of that particular warp being in wait state. We have been cleaning up wait states by data_group, but if there is exps that come from data_groups that have not been cleaned yet, they can still be in wait. In this case, the update batch looks for warp exps in the cleaned state from data_group="LAP.PV3.20140730.20150%" but there was a batch of chip exps that was most recently reprocessed
+from a data_group that was not cleaned yet. To avoid these situations in the future, I decided to run my query that find most recent reprocessings and limit that to the wait state (but expanded it to cover every data_group)
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.maxstate,subb.data_group,subb.label,subb.maxchipstate FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state as maxstate,warpRun.data_group,camRun.label,chipRun.state as maxchipstate FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id) WHERE subb.maxstate LIKE "cleaned" and subb.maxchipstate LIKE "wait" group by subb.data_group;' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id  "$3}' > todo
+}}}
+ It contains 1667 exps, all of which I sent to clean-up, and finished cleaning just fine.
+
+
+