PS1 IPP Czar Logs for the week 2011.03.21 - 2011.03.27

(Up to PS1 IPP Czar Logs)

Monday : 2011.03.21

Tuesday : 2011.03.22

Wednesday : 2011.03.23

Roy is czar.

• 08:00 No data last night....
• 13:15 Bill is doing some processing from a pantasks on ippc25.

Thursday : 2011.03.24

Roy is czar.

• 08:00 No data last night....

• 14:00 CZW Stopping processing for summit_id addition to database.
• 14:41 CZW Cleanup finally finished, beginning dumps of GPC1 and ISP databases.
• 15:28 CZW Waiting for initial GPC1 dump to finish. Rebuilding tag for ippuser.
• 15:33 CZW Dump finished. Making changes to GPC1 database for schema_version 1.1.69.
• 15:45 CZW New IPP user build complete.
• 18:13 CZW Commit transaction, completing upgrade to schema 1.1.69. Beginning second dump.
• 19:35 CZW Second dump finished. Starting on ISP (which should be faster?).
• 19:55 CZW ISP upgrade aborted. I can't get the read lock, and I've been waiting almost a half hour. I have removed the add.database isp entries from summitcopy/input and registration/input.
• 19:57 CZW Restarting servers: ./check_system.sh start.server / start / run
• 20:06 CZW bugfix on pztool due to missing command line options.
• 20:19 CZW commented out offending line in pztool.c (511) because I can't see why it's still complaining.
• 20:28 CZW fixing new bug at the pztool -advance stage. Luckily, this should be the end of where changes happened.

Friday : 2011.03.25

• 05:00 EAM ppStack was failing -- error in the new timers; I updated the code and merged the changes back to the trunk
• 06:00 EAM ppSub is failing for MD06 y-band stack-stack diffs. I've set the run (diff_id = 117893) state to wait until it can be debugged
• 09:50 EAM 1 diff failed due to a corrupt warp. I reverted with runwarpskycell.pl --redirect-output --warp_id 175863 --skycell_id skycell.1805.128
• 09:55 EAM 6 warps failed from I/O, I reverted manually; 2 other diffs had failed due to a negative kron normalization (implies a very bad subtraction) : I merged 2 lines of code that address this as bad quality.

Saturday : 2011.03.26

Sunday : 2011.03.27