PS1 IPP Czar Logs for the week 2011-03-28 - 2011-04-03

(Up to PS1 IPP Czar Logs)

Monday: 2011-03-28

• (serge) Started gpc1 ingestion for replication on ippc02 (at 09:52:21)
• 14:00 CZW: Stopped all pantasks to allow for upgrade of ISP database.
• 14:39 CZW: Beginning flush for binlog status and initial dump.
• 14:47:31 gpc1 ingestion complete
• 14:57 gpc1 replication started. Seconds_Behind_Master: 327235
• 14:58 CZW: Finished updates to ISP database. Beginning flush for binlog status for final dump.
• 15:04 CZW: Making final dump. Completed while typing. Upgrade complete. See attached logfile for details.
• 22:44 CZW: ippdb01 had load of ~20. Stopped all pantasks, let jobs finish, and then checked mysql processlist. Even with all jobs finished or timedout, a large number of queries were still pending. Cleared them, and restarted summitcopy and registration. Removed old dates from stdscience pantasks, as it might be adding extra queries for completed dates (some dates did not have morning darks taken).
• 22:47 CZW: restarting stdscience.
• 22:50 CZW: removed old dates from registration to ensure burntool backlog can be cleared without waiting for the date to cycle around (seven old dates were still being examined).

Tuesday: 2011-03-29

Wednesday: 2011-03-30

Thursday: 2011-03-31

Friday: 2011-04-01

Saturday: 2011-04-02

Sunday: 2011-04-03