Changes between Version 17 and Version 18 of PS1_IPP_Czarlog_20170313

Timestamp:

Mar 17, 2017, 10:27:36 AM (9 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20170313

@@ -24 +24 @@
    * many fault 2 seem to be coming from c70-c75 (apache nodes), -1x loading -- didn't really help, remove all from stdscience with hosts_apache group(other pantasks seem ok)
   * ippqub/stdscience_ws running normally @MRTC-B, should not be touched other than set to stop -- WS/SSdiffs/ps_ud_QUB running (5x si0, 5x ci0/1) in AM after nightly and directly targeting ipp105,117 datanodes only -- ps_ud_QUB 1x ITC loading during night
-  * WWdiffs hacked to only produce diff+inv uncompressed images now
+  * WWdiffs hacked to only produce diff+inv uncompressed images now -- %SS,3PI starting night %.20170317
   * data targeted more to ipp105,117 now -- more may be needed and may help to reduce ipp105,117 to 1x loading only in processing
   * gpc2 removed from ippMonitor/czarpoll.pl to help speed up refresh (not waste time on those queries and plots) ~20 min now
   * many WSdiff faults by skycell due to missing stacks -- have to just manually set quality 42 to clear for all so QUB can get diffs available other skycells
   * many pstamp chip update faults due to missing raw and mdc/psf products -- pstamp will skip after a bit, but probably helps to cleanup before nightly so cycles not wasted on reverts to fail again and again
-   * pstamp update cleanup back on (defaults 22-00 UTC daily, maybe better if made 02-04 UTC?) -- will help keep space cleaned up, but may need to block MPIA/MPE/WEB.BIG/PSI.BIG if large jobs show up or possibly just remove label during night
+   * pstamp update cleanup back on (defaults 22-00 UTC daily, maybe better if made later, 02-04 UTC? maybe conflict w/ normal cleanup) -- will help keep space cleaned up, but may need to block MPIA/MPE/WEB.BIG/PSI.BIG if large jobs show up or possibly just remove label during night
   * summitcopy keeping up well, only ~5-10 exposure lag
   * startup had quite a bit of lag, chip jobs >300-400s, camera>1ks -- mounts to all the datanodes being re-established? logs seem to indicate stalling on gpc1 addprocessed exposure? MRTC-B/ITC link seems very active at start of nightly processing as well --