[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012-10-29 ===
Serge is czar
 * 06:05: Still copying data from the summit (about 60 images). Nothing to worry about.
 * 06:10: Cleaning gpc1 of old backups preventing mysql slave to run and backups to be successfully performed.
 * 08:50: Nightly processing complete
 * 15:45 MEH: stdscience needs regular restart. will start to feed in MD05 reprocessing
=== Tuesday : 2012.10.30 ===
 * 00:20 MEH: made mistake of looking at processing.. looks like ipp016 has lost it around 22:30, cannot ssh into and jobs (registration/stdscience) running there stalled but disk seems to be available (but in repair). will try to isolate for processing tonight to get going again.. then reboot.  [wiki:Ipp016-crash-20121029]
  * restarted registration and stdscience
  * MD03 also removed from nightly diffs until refstack is made.
 * 11:00 MEH: the failing exposure report from OTIS turned out to be an added whitespace to the TXT page on the datastore when going from 9999->10000 during the night of 10/17 that made the downloaded file not a real FITS file and hence the missing END card. Todd fixed and cleared the Problem Notification email. 
 * 13:59 CZW: Noticed a large number of chips failing with perfectly fine input data.  Tracked down the issue to ippc02 and /tmp/ being filled with the nebulous_server.log.  Moved log to /export/ippc02.0/ipp/nebulous_server_20121030.log, and I'm now bzip2-ing it.  I've restarted apache and reverted all LAP faults, which should resolve the issue.
 * 15:00 MEH: MD05.GR0 chip->warp+NS running along side LAP
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===