PS1 IPP Czar Logs for the week 2011-05-02 - 2011-05-6

(Up to PS1 IPP Czar Logs)

Monday : 2011-05-02

• 11:00 or so Bill set up to publish the CNP data for mops. Changed labels for cleaned up diffRuns from goto_cleaned to CNP.nightlyscience
• 11:10 Bill added CNP to survey.destreak. Apparently we had not added it to the stdscience input file.
• 11:25 Bill queued 14 exposures from STS.20110430 to be re-warped with new tessellation STS.V4
• (pm) Serge is the culprit for all suspect activity involving ippdb02.

Tuesday : 2011-05-03

• 10:45 Bill: One STS.2010.b chip run failed because it has filter = 'not available' set it to drop. Chip 222458 XY75 had a corrupted fits file. Fixed that.

Wednesday : 2011-05-04

• 11:00 Bill restarted the pstamp and update pantasks and set all data updated for MPIA stamps to be cleaned.
• 12:10 Bill postage stamp server has finished all of it's outstanding jobs. Set all data with ps_ud label to be cleaned.

Thursday : 2011-05-05

• 07:20 Bill: IfA has submitted a postage stamp request storm. Unfortunately the update pantasks died at 22:22 last night, so little got done overnight. Restarted update with double the number of hosts.
• 10:30 Serge (ab)uses mysql server*s* on ippdb02. It is recommended not to use this host.
• 19:55 Bill ippc13 the pantasks server for update became unresponsive this afternoon. Power cycled by bills.

Friday : 2011-05-06

• 06:40 Bill ipp024 paniced during the night. See the production cluster page. Power cycled. distribution stopped until it comes back up.
• 06:59 Bill: There are several destreak jobs that are failing and then failing to revert due to corrupt files on ipp024. Since the chip inputs have been destreaked it will take a bit of work to fix this problem.
• 23:00 CZW: Stopped LAP code. There's a bug somewhere that is launching chipRuns that are unassociated with a lapRun. I have a reasonably good idea where it is, so I'll take a look tomorrow.

Saturday : 2011-05-07

• 16:00 CZW: Restarted LAP code after applying bug fixes to diff pairing (bad OBJECT values cause impossible diffs), chip remaking (chiptool needs -pstamp_order to return the correct value and break an infinite loop), and final stack dist_group (wasn't set, is now set correctly).
• 19:35 Bill: I had diff.revert.off in order to debug a problem earlier this morning. I forgot to turn it back on. I just did.

Sunday : 2011-05-08