Changes between Version 6 and Version 7 of PS1_IPP_Czarlog_20170313

Timestamp:

Mar 16, 2017, 11:26:56 PM (9 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20170313

@@ -16 +16 @@
  * 14:20 CZW: I've chagned the Nebulous-Server max_used_space parameter to 0.989 on ippc70-ippc75, and restarted their apache servers.  This change will allow the ITC nodes to be used at a higher fill factor than before, allowing processing to shift there.
  * 16:20 CZW: With the shift of processing to ITC, I've disabled the trange limits on the rsync pantasks (/data/stare00.1/watersc1/itc_rsync_lists/rsync_run).  This will allow jobs to run internal to the MRTCB over night, as the target hosts aren't used for nightly processing.  I've also increased the loading slightly (3x instead of 1x, still running 10MB/s rsync jobs), and may move it up a bit more (5x max).
- * 18:20 MEH: shutting down pantasks @MRTC-B and starting summitcopy+registration+stdscience @ITC --
+ * 18:20 MEH: shutting down pantasks @MRTC-B and starting summitcopy+registration+stdscience @ITC with agreement from MOPS -- 
+
 === Thursday : 2017.03.16 ===
+ * MEH: nightly processing @ITC
+  * summitcopy,registration,stdscience,pstamp,distribution,cleanup,stack all running -- check_system.sh setup for all
+   * changed 6x si0, 3x ci0/1 to 5x, 4x for fewer faults, but then seemed to get a pileup in warp 
+  * ippqub/stdscience_ws running normally @MRTC-B, should not be touched other than set to stop -- WS/SSdiffs/ps_ud_QUB running in AM after nightly directly targeting ipp105,117 datanodes only -- ps_ud_QUB 1x ITC loading during night
+  * WWdiffs hacked to only produce diff+inv uncompressed images now
+  * data targeted more to ipp105,117 now -- more may be needed and may help to reduce ipp105,117 to 1x loading only in processing
+  * gpc2 removed from ippMonitor/czarpoll.pl to help speed up refresh (not waste time on those queries and plots) ~20 min now
+  * many WSdiff faults due to missing stacks -- have to just manually set quality 42 to clear for all
+  * many pstamp chip update faults due to missing raw and aux products -- pstamp will skip after a bit, but probably helps to cleanup before nightly so cycles not wasted on reverts to fail again and again
+  * pstamp update cleanup back on (defaults 22-00 UTC daily, maybe better if made 02-04 UTC?) -- will help keep space cleaned up, but may need to block MPIA/MPE/WEB.BIG/PSI.BIG if large jobs show up or possibly just remove label during night
+
 
 === Friday : YYYY.MM.DD ===