== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : YYYY.MM.DD ===
  
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : YYYY.MM.DD ===

=== Thursday : 2017.09.14 ===
 * MEH: ippc65 down since early AM, Hadyn power cycled
 * MEH: ipp@ippc19 crontab adding /export/ippdbXX check added now since ippdb08 filled up surprisingly today
 * MEH: ippdb06,db08 ganglia plots missing since weekend.. -- critical machines, useful to have plots.. -- /etc/init.d/gmond restart
 * MEH: sending missed and misc nightly processing to cleanup again
 * 15:30 EAM : ran out of space on ippdb08 /export/ippdb08.0 due to a problem with the binlogs.  I cleared out some old files first (a test dump and an old, partial copy of nebulous mysql).  I then discovered that the binlogs had been getting very large for a while: this was caused by an error introduced into gzgetexp on July 15 which resulting it in attempting to re-download every failed exposure, instead of only the past night.  I fixed the code in pzgetexp to only check for the past 100 exposures.  I also cleared the binlogs up to Sept 12.
 * 16:20 EAM : updated ippMonitor to report replication status for ippdb01 for nebulous (instead of ipp115)
 * MEH: ippc61 appears to have been crashed since weekend? cannot ssh into, no console login (or messages) -- power cycle twice without booting -- email Haydn to look at if time
 * MEH: nightly pantasks stop.block.restart seems to get pstamp into weird/broken state if trying to make stamps/update -- happened a couple times now -- manually clearing stalled MOPS jobs...
=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===