Changes between Version 10 and Version 11 of PS1_IPP_Czarlog_20190510

Timestamp:

May 13, 2019, 5:06:29 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -164 +164 @@
 }}}
 
+ * TdB: afternoon:
+ Some missing .mk files in the warps. Set them to update the skyfile
+{{{
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1342050
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1344264
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1358970
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1365663
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1368220
+}}}
+ Didn't work. send to reprocess from scratch, instead.
+
+ Also some memory blocks that need reverting:
+{{{
+warptool -revertwarped -fault 4 -label update.LAP.PV3 -dbname gpc1 
+}}}
+
+ -----------------------
+ Some of the warp update processing got stuck, probably due to their chip exps being in wait state. Query those and send them to cleanup, followed by an update. Pump them to a file first
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "wait" group by chip_id; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id  "$1"; chiptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -chip_id "$1"; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id "$1 }' > LAP.PV3_waits
+}}}
+ There are 1389 of them. Send them to cleanup and then back to update. That solved most of the problems with the remaining update warps
+
+ ----------------------
+ '''IPP meeting''': 
+ Cleanup:
+
+ We should periodically clean up the chip stage pixels for the updates we are running, to preserve disk space. Use the following query, and queue up a 1000 or so once in a while, or a few thousand overnight:
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id)  where warpRun.label like "update.%" and warpRun.data_group like "%" and warpRun.state like "full" and chipRun.state like "full" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -state full -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1 }' | tcsh
+}}}
+
+ ----------------------
+ Haydn wants to swap some drives from ippc17 and ippc90. Use check_system.sh to remove them from processing:
+{{{
+./check_system.sh hostoff ippc17 (do this a bunch of times)
+./check_system.sh hostoff ippc90
+}}}
+ After the swap, put them back up
+{{{
+./check_system.sh hoston ippc17 (do this a bunch of times)
+./check_system.sh hoston ippc90
+}}}
+
+
 === Tuesday : 2019.05.14 ===