Changes between Version 14 and Version 15 of PS1_IPP_Czarlog_20170313

Timestamp:

Mar 17, 2017, 2:19:24 AM (9 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20170313

@@ -22 +22 @@
   * summitcopy,registration,stdscience,pstamp,distribution,cleanup,stack all running -- ~ippitc/check_system.sh setup for all
    * changed default 6x si0, 3x ci0/1 to 5x si0, 4x ci0/1 for fewer faults, but then seemed to get a pileup in warp -- may be due to extra targeting on ipp105,117 (many pending for ipp117) so bumping unwant 20->30->40
-   * many fault 2 seem to be coming from c70-c75 (apache nodes), -1x loading
+   * many fault 2 seem to be coming from c70-c75 (apache nodes), -1x loading -- didn't really help, remove all from stdscience with hosts_apache group(other pantasks seem ok)
   * ippqub/stdscience_ws running normally @MRTC-B, should not be touched other than set to stop -- WS/SSdiffs/ps_ud_QUB running in AM after nightly directly targeting ipp105,117 datanodes only -- ps_ud_QUB 1x ITC loading during night
   * WWdiffs hacked to only produce diff+inv uncompressed images now
   * data targeted more to ipp105,117 now -- more may be needed and may help to reduce ipp105,117 to 1x loading only in processing
   * gpc2 removed from ippMonitor/czarpoll.pl to help speed up refresh (not waste time on those queries and plots) ~20 min now
-  * many WSdiff faults due to missing stacks -- have to just manually set quality 42 to clear for all
-  * many pstamp chip update faults due to missing raw and aux products -- pstamp will skip after a bit, but probably helps to cleanup before nightly so cycles not wasted on reverts to fail again and again
-  * pstamp update cleanup back on (defaults 22-00 UTC daily, maybe better if made 02-04 UTC?) -- will help keep space cleaned up, but may need to block MPIA/MPE/WEB.BIG/PSI.BIG if large jobs show up or possibly just remove label during night
+  * many WSdiff faults by skycell due to missing stacks -- have to just manually set quality 42 to clear for all so QUB can get diffs available other skycells
+  * many pstamp chip update faults due to missing raw and mdc/psf products -- pstamp will skip after a bit, but probably helps to cleanup before nightly so cycles not wasted on reverts to fail again and again
+   * pstamp update cleanup back on (defaults 22-00 UTC daily, maybe better if made 02-04 UTC?) -- will help keep space cleaned up, but may need to block MPIA/MPE/WEB.BIG/PSI.BIG if large jobs show up or possibly just remove label during night
   * summitcopy keeping up well, only ~5-10 exposure lag
   * startup had quite a bit of lag, chip jobs >300-400s, camera>1ks -- mounts to all the datanodes being re-established? logs seem to indicate stalling on gpc1 addprocessed exposure? MRTC-B/ITC link seems very active at start of nightly processing as well --