(Up to PS1 IPP Czar Logs)

Monday : 2012-10-29

Serge is czar

• 06:05: Still copying data from the summit (about 60 images). Nothing to worry about.
• 06:10: Cleaning gpc1 of old backups preventing mysql slave to run and backups to be successfully performed.
• 08:50: Nightly processing complete
• 15:45 MEH: stdscience needs regular restart. will start to feed in MD05 reprocessing

Tuesday : 2012.10.30

• 00:20 MEH: made mistake of looking at processing.. looks like ipp016 has lost it around 22:30, cannot ssh into and jobs (registration/stdscience) running there stalled but disk seems to be available (but in repair). will try to isolate for processing tonight to get going again.. then reboot. Ipp016-crash-20121029
  • restarted registration and stdscience
  • MD03 also removed from nightly diffs until refstack is made.
• 11:00 MEH: the failing exposure report from OTIS turned out to be an added whitespace to the TXT page on the datastore when going from 9999->10000 during the night of 10/17 that made the downloaded file not a real FITS file and hence the missing END card. Todd fixed and cleared the Problem Notification email.
• 13:59 CZW: Noticed a large number of chips failing with perfectly fine input data. Tracked down the issue to ippc02 and /tmp/ being filled with the nebulous_server.log. Moved log to /export/ippc02.0/ipp/nebulous_server_20121030.log, and I'm now bzip2-ing it. I've restarted apache and reverted all LAP faults, which should resolve the issue.
• 15:00 MEH: MD05.GR0 chip->warp+NS running along side LAP
• 16:05 MEH: stdscience regular restart

Wednesday : YYYY.MM.DD

• 09:35 (Serge): Publishing last night data to IPP-MOPS-DEV datastore

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD