Changes between Version 5 and Version 6 of PS1_IPP_Czarlog_20170911

Timestamp:

Sep 15, 2017, 7:03:26 AM (9 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20170911

@@ -17 +17 @@
  * 15:30 EAM : ran out of space on ippdb08 /export/ippdb08.0 due to a problem with the binlogs.  I cleared out some old files first (a test dump and an old, partial copy of nebulous mysql).  I then discovered that the binlogs had been getting very large for a while: this was caused by an error introduced into gzgetexp on July 15 which resulting it in attempting to re-download every failed exposure, instead of only the past night.  I fixed the code in pzgetexp to only check for the past 100 exposures.  I also cleared the binlogs up to Sept 12.
  * 16:20 EAM : updated ippMonitor to report replication status for ippdb01 for nebulous (instead of ipp115)
- * MEH: ippc61 appears to have been crashed since weekend? cannot ssh into, no console login (or messages) -- power cycle twice without booting -- email Haydn to look at if time -- Haydn thinks its already on his list of ones to work on and hasn't been up for a while and maybe IP address mixup -- conflicts w/ what ganglia was reporting 
+ * MEH: ippc61 appears to have been crashed since weekend? cannot ssh into, no console login (or messages) -- power cycle twice without booting -- email Haydn to look at if time -- Haydn thinks its already on his list of ones to work on and hasn't been up for a while and maybe IP address mixup -- conflicts w/ what ganglia was reporting and that Weryk has been actively using it as well
  * MEH: nightly pantasks stop.block.restart seems to get pstamp into weird/broken state if trying to make stamps/update with faults -- happened a couple times now -- manually clearing stalled MOPS jobs...
 === Friday : YYYY.MM.DD ===