Changes between Version 39 and Version 40 of PS1_IPP_Czarlog_20190510

Timestamp:

May 17, 2019, 10:36:56 AM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -1382 +1382 @@
 * MEH: sending more old diffs to cleanup to help free up space now that the WWdiff update bug is fixed -- also sending old WS distribution to cleanup and removing old .mops links
 
-
+ * TdB: Cleaned out some stuck files in update processing:
+ Looking at mysql, there are more exps in update under update.LAP.PV3 than IPPmonitor is showing. A likely cause is that the data_state of those is not in update. Need to investiagte a bit, since those could be in wait state.
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT chip_id,data_group,label FROM chipRun WHERE state like "update" and label like "update.LAP.PV3";'
+}}}
+Get more info:
+{{{
+mysql -hippdb08 -uX -pX gpc1 -e "SELECT count(*) as cnt,chipRun.chip_id,chipRun.state as chip_state,chipRun.data_group as chip_datgrp,chipRun.label as chip_label,camRun.data_group as cam_datgrp,warpRun.warp_id,warpRun.state as wrp_state,warpRun.label as wrp_label,warpRun.data_group as wrp_datgrp,chipProcessedImfile.data_state as chip_datstate,warpSkyfile.data_state as wrp_datstate FROM warpRun join warpSkyfile using (warp_id) JOIN fakeRun using (fake_id) JOIN camRun using (cam_id) JOIN chipRun  using (chip_id) JOIN chipProcessedImfile using (chip_id) WHERE chipRun.label like 'update.LAP.PV3' AND chipRun.state = 'update' group by chipRun.chip_id,chipRun.state,warpRun.state,chipProcessedImfile.data_state,warpSkyfile.data_state" > LAP.PV3_datastate_new
+}}}
+ There was a case of a warp exp stuck in goto_cleaned with label = update.LAP.PV3 and corresponding chip stuck in cleaned state
+{{{
+chiptool -dbname gpc1 -updaterun  -set_label update.LAP.PV3 -chip_id 1131183
+chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1131183
+
+warptool -dbname gpc1 -updaterun -set_label goto_cleaned -warp_id 1082990  
+warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id 1082990
+warptool -dbname gpc1 -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id 1082990
+}}}
+ There are also 43 warp exps stuck in cleaned state. Sending them to skyfileupdate does nothing. Send them back to cleanup and then back to update
+ Next,there are chip exps in update state with data_state=cleaned. Try to send them to setimfiletoupdate and see if that helps:
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipProcessedImfile.data_state like "cleaned" group by chip_id; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id "$1 }' | tcsh
+}}}
+ There are also chip exps stuck in the full data_state but with general state=update. Try to send them to setimfiletoupdate and see if that helps:
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipProcessedImfile.data_state like "full" and warpRun.state like "full" and chipRun.state like "update" group by chip_id; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id "$1 }' | tcsh
+}}}
+ I will continue tomorrow, since there are more stuck files.