Changes between Version 5 and Version 6 of PS1_IPP_Czarlog_20190426

Timestamp:

Apr 29, 2019, 2:32:40 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190426

@@ -142 +142 @@
 === Monday : 2019.04.29 ===
 
-Still have the dark issue.
-
-
-Still a bunch of broken jobs to work through, so here we go!
+
+'''Prep for Haydn''' by JRF
+
+I've set the following to repair to prepare for Haydn doing some maintenance:
+{{{
+ipp107, ipp108, ipp117
+ipp068
+ipp127-ipp142 (at least all the ones that are tracked by neb-host)
+}}}
+
+There are a bunch of compute nodes that Haydn might be messing with soon and they are currently in use. So these should be removed from current processing (c91, c85):
+{{{
+check_system.sh host off
+}}}
+
+12:20pm : Haydn is now installing the PDUs in M.03, so I've set ipp137-139 to down, along with ipp134-ipp136 as these are affected by the switch in that cabinet (THIS IS NOT DOCUMENTED ANYWHERE).
+
+
+'''Dark issue from yesterday''' by JRF
+
+The two exposures {{{o8602g0001d}}} and {{{o8602g0001d}}} are being reported as not copied to MHPCC.
+
+Had a look at the logs for the summit copy:
+{{{
+logs/201904/pantasks_logs.20190428.153056/summit.copy.log.bz2
+}}}
+nothing about these exposures, starts at the one after them (o8602g0003d).
+
+Having a look at some of the IPPmonitor tables, we can see it in the summitExp table, and there's also some stuff in the pzPending table (this is really the pzDownload table), but it is not seen in the newExp table, and the state in the summitExp table is 'run', even though we have all 60 expected parts of the image.
+
+Check out the status:
+{{{
+~/src/ipp-20170121/tools/checkexp --verbose
+}}}
+It reports that these two exposures should be moved to the next stage of processing.
+
+Going to try and revert the copy, to do it again:
+{{{
+pztool -revertcopied -exp_name o8602g0001d -inst gpc1 -telescope ps1 -dbname gpc1
+}}}
+
+get an error saying it should have affected atleast 1 row, but it did not
+{{{
+mysql> SELECT * FROM pzDownloadExp WHERE exp_name IN ("o8602g0001d","o8602g0002d");
++-----------+-------------+--------+-----------+-------+---------------------+
+| summit_id | exp_name    | camera | telescope | state | epoch               |
++-----------+-------------+--------+-----------+-------+---------------------+
+|   1460891 | o8602g0001d | gpc1   | ps1       | run   | 2019-04-28 15:28:28 | 
+|   1460892 | o8602g0002d | gpc1   | ps1       | run   | 2019-04-28 15:28:57 | 
++-----------+-------------+--------+-----------+-------+---------------------+
+}}}
+
+We updated the first one to the stop state
+{{{
+pztool -updatepzexp -exp_name o8602g0001d -inst gpc1 -telescope ps1 -summit_id 1460891 -set_state stop -dbname gpc1
+}}}
+
+However, it does not appear in the newExp table (SELECT * FROM newExp WHERE summit_id = 1460891;)
+
+Using 'checkexp' again we see that the exp_id is 'null' for o8602g0001d
+
+What we need to do is properly advance it.
+{{{
+pztool -advance -summit_id 1460891 -exp_name o8602g0001d -inst gpc1 -telescope ps1 -end_stage reg -workdir neb://@HOST@.0/gpc1/20190429 -dbname gpc1
+}}}
+Might have done this twice... so there are two copies in the rawExp table.
+
+
+Ok, let's fix the 2nd dark! :
+{{{
+pztool -updatepzexp -exp_name o8602g0002d -inst gpc1 -telescope ps1 -summit_id 1460892 -set_state stop -dbname gpc1
+pztool -advance -summit_id 1460892 -exp_name o8602g0002d -inst gpc1 -telescope ps1 -end_stage reg -workdir neb://@HOST@.0/gpc1/20190429 -dbname gpc1
+}}}
+
+Done!
+
+
+'''Broken update stuff (continued)'''
+
+a broken job in the ps_ud_WEB chip stage:
+{{{
+o6224g0254o XY02  537186  1281637 update  ps_ud_WEB LAP.PV3.20140730.ipp.20141126 LAP.ThreePi 2
+}}}
+complains about:{{{
+neb://@HOST@.0/gpc1/LAP.PV3.20140730.ipp/2014/11/26/o6224g0254o.537186/o6224g0254o.537186.ch.1281637.XY02.ppImage.mdc is missing
+}}}
+try to update:
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label mopsfix -chip_id 1281637
+Note that it is now under a different label.
+It still complains, and the files it is missing are on ippb04, so we turned on the nodes that it complained about. It still fails.
+The main complain now is that the burntool version does not match. So we will attempt a burntool fix:
+{{{
+ipp_apply_burntool_fix.pl --exp_name o6224g0254o --class_id XY02 --verbose --dbname gpc1
+}}}
+This fails in the camera stage now, but this is because the file it is after is on a node that is temporarily down (Haydn is doing work on it). So we should wait until it is back up.
+
+
+
+
+
+To do: 
+ - Once ipp134-139 are back online revert the mopsfix job stuck in the camera stage
+ - send everything in update.LAP.PV3 to clean...
+ - ...once cleaned, send them all to update. This should fix most of the chip issues
+ - For the remaining problems:
+    -- if they complain about missing cm (camera) files then they should be reprocessed.