Changes between Version 9 and Version 10 of PS1_IPP_Czarlog_20180319

Timestamp:

Mar 25, 2018, 9:55:16 PM (8 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20180319

@@ -7 +7 @@
  * MEH: had normal nightly data nodes .0 ipp100,101,102,103,104,118,119,122 in repair in the past due to lack of space, plenty of space for a few weeks now but not put back up... putting neb-host up to help keep them balanced with .1 partitions...
 
-=== Tuesday : 2018.03.24 ===
+=== Tuesday : 2018.03.20 ===
  * MEH: pstamp parsing can get behind w/ large QUB requests, bump parse nodes from 20->50 of 60 available (~ippitc/pstamp/input set.parse.limit 50) leaving a few to do stamps and push the requests through more quickly (estimate ~20min now rather than ~hour typically) until some sort or priority can be added to the parsing for MOPS
  * MEH: ps_ud_% cleanup moved from 1200-1400 HST to 1800 HST to help avoid update cleanups blocking stamps completing for QUB (~ippitc/pstamp/input adding shift.trange.update.cleanup)
+ * MEH: another set of messed up LAP images due to missing files from the stsci shuffle that were not restored/fixed -- updates for MOPS stamps are blocked for warps with state error_cleaned due to some skycells triggering an error in cleanup -- manually set warp to update with label LAP.PV3.wrp_broken.do.not.clean for warp_id 1300413, 1302730, 1189291, 1187717
+  * unique label must be used otherwise gets triggered for normal automatic cleanup later and will break due to missing files setting error_cleaned and block updates again in the future... suspect this has been happening for some other "fixes"
+  * this is not a good method solution for all cases as some warp skycell with data_state full with warp state error_cleaned could have missing .fits images and other products preventing complete updates 
+ * MEH: another blocked update due to chip having state error_cleaned (chip_id 940878) -- looks like the occasional glitch in cleanup, manually setting goto_cleaned and cleared and updated fine again
+
 === Wednesday : 2018.03.21 ===
  * MEH: Haydn shutting down nodes at ATRC @0700 HST -- setting neb-host down on ippb00-03,06-15 
   * ippb00 not powering up, Haydn needs to look into tomorrow
   * ippb01-03,06 back to neb-host repair; ippb07-15 neb-host up like before powered down
- * MEH: rebuild ipp-20170121 ippconfig for retaining warps for 7 days, diffs for 3 days now (likely will be set to keep warps even longer, ipp-20170121 nightly_science.config) and adding misc updates for K2 and config for Bright nightly processing -- test quad run and checked by Serge as required for any ops tag changes
-=== Thursday : YYYY.MM.DD ===
+ * MEH: rebuild ipp-20170121 ippconfig for retaining warps for 7 days, diffs for 3 days now (likely will be set to keep warps even longer, ipp-20170121 nightly_science.config) and adding misc updates for K2 and config for Bright nightly processing -- dailytestset quad run and checked by Serge as required for any ops tag changes
+
+=== Thursday : 2018.03.22 ===
+ * MEH: odd ganglia has been showing up but not reporting anything for ippdb03,ippc20-22,ippc17 for a while... just needed gmond restarted
 
 === Friday : 2018.03.23 ===
- * MEH: misc faults on ippmonitor for ps_ud_MOPS -- a mess of missing LAP files from the earlier shuffle with stsci nodes still not restored or dealt with
+ * MEH: misc faults on ippmonitor for ps_ud_MOPS -- a large mess of missing LAP files from the earlier shuffle with stsci nodes still not restored or dealt with that are blocking cleanup and/or updates on both chips and warps -- doing a reprocessing of the following exposures through warp to make the images available for MOPS under label/data_group mopsfix/mops.fixbrokenLAP.20180323 -- o6043g0500o, o6138g0432o, o6083g0390o, o6149g0175o, o6041g0497o, o6043g0516o
+ * MEH: Hadyn fixed ippc27 IP (was using the same IP as ippc26), unclear how long this was happening
+
 === Saturday : YYYY.MM.DD ===
 
-=== Sunday : YYYY.MM.DD ===
+=== Sunday : 2018.03.25 === 
+ * MEH: another set of broken LAP.PV3 data due to missing files in nebulous -- o6138g0449o, o6149g0158o (LAP.PV3.20140730.bigmem.20150105) chip state error_cleaned as the correct state due to missing files on some of the chips for the exposure (chip_id 1387283, 1387912), was okay earlier in week so appears someone manually set state to cleaned earlier to clear the error and the chips needed were updated okay, however, once they were sent to cleanup normally, they were returned to the error_cleaned state blocking updates again (since they are actually broken) -- while manually setting the error_cleaned state to cleaned clears the error short term to allow the okay chips in the exposure to update, it doesn't really fix the problem... -- setting these two exposures to reprocess as well mopsfix/mops.fixbrokenLAP.20180323
+ * MEH: update Bright.nightlyscience comment in svn nightly_science.config to process without crontab
+