== PS1 IPP Czar Logs for the week 2011-05-02 - 2011-05-6 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011-05-02 ===
 * 11:00 or so Bill set up to publish the CNP data for mops. Changed labels for cleaned up diffRuns from goto_cleaned to CNP.nightlyscience
 * 11:10 Bill added CNP to survey.destreak. Apparently we had not added it to the stdscience input file.
 * 11:25 Bill queued 14 exposures from STS.20110430 to be re-warped with new tessellation STS.V4
 * (pm) Serge is the culprit for all suspect activity involving ippdb02.

=== Tuesday : 2011-05-03 ===
 * 10:45 Bill: One STS.2010.b chip run failed because it has filter = 'not available' set it to drop. Chip 222458 XY75 had a corrupted fits file. Fixed that.

=== Wednesday : 2011-05-04 ===
 * 11:00 Bill restarted the pstamp and update pantasks and set all data updated for MPIA stamps to be cleaned.
 * 12:10 Bill postage stamp server has finished all of it's outstanding jobs. Set all data with ps_ud label to be cleaned.

=== Thursday : 2011-05-05 ===
 * 07:20 Bill: IfA has submitted a postage stamp request storm. Unfortunately the update pantasks died at 22:22 last night, so little got done overnight. Restarted update with double the number of hosts.
 * 10:30 Serge (ab)uses mysql server*s* on ippdb02. It is recommended not to use this host.
 * 19:55 Bill ippc13 the pantasks server for update became unresponsive this afternoon. Power cycled by bills.
=== Friday : 2011-05-06 ===
 * 06:40 Bill ipp024 paniced during the night. See the production cluster page. Power cycled. distribution stopped until it comes back up.
 * 06:59 Bill: There are several destreak jobs that are failing and then failing to revert due to corrupt files on ipp024. Since the chip inputs have been destreaked it will take a bit of work to fix this problem.
 * 23:00 CZW: Stopped LAP code.  There's a bug somewhere that is launching chipRuns that are unassociated with a lapRun. I have a reasonably good idea where it is, so I'll take a look tomorrow.  

=== Saturday : 2011-05-07 ===
 * 16:00 CZW: Restarted LAP code after applying bug fixes to diff pairing (bad OBJECT values cause impossible diffs), chip remaking (chiptool needs -pstamp_order to return the correct value and break an infinite loop), and final stack dist_group (wasn't set, is now set correctly).
 * 19:35 Bill: I had diff.revert.off in order to debug a problem earlier this morning. I forgot to turn it back on. I just did.

=== Sunday : 2011-05-08 ===