== PS1 IPP Czar Logs for the week 2011-09-05 - 2011-09-11 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011-09-05 ===
  
 * 10:00 heather restarted registration - burntool restarted.
 * 10:52 that didn't work. various incantations of regtool -revertstuff didn't work (nothing to revert). Tried regpeek it complained about XY06 o5809g0511o, did this ( regtool -updateprocessedimfile -exp_id 387484 -class_id XY06 -set_state pending_burntool -dbname gpc1  ) and now burntool is moving. I hate burntool.
 * 11:11 some more that needed some help (discovered using regpeek and gpc1):
   * [heather@ippc18 ~ipp/registration]$ regtool -updateprocessedimfile -exp_id 387562 -class_id XY12 -set_state pending_burntool -dbname gpc1
   * [heather@ippc18 ~ipp/registration]$ regtool -updateprocessedimfile -exp_id 387605 -class_id XY65 -set_state pending_burntool -dbname gpc1

=== Tuesday : 2011-09-06 ===
Bill is czar today
 * 08:00 users report strangeness on the data store. They are correct. Sometime around 1pm yesterday mysql stopped processing fileset registration requests. This has left a few thousand incomplete filesets. mysql log indicates that a table is damaged.
 * 08:30 stopped postage stamp pantasks and rcserver task in distribution.
 * 08:50 stopped publishing
 * 08:56 Gavin repaired the database.
 * 09:20 set publishing to run and restarted postage stamp server pantasks in order to have clear fault counts.
 * 10:14 restarted update pantasks
 * 10:36 one of the instances of neb://ipp036.0/gpc1/20110906/o5810g0282o/o5810g0282o.ota55.fits was corrupt. Saved the bad file to ~bills/o5810g0282o.ota55.fits.corrupt for investigation and copied the good one (as measured by successful funpack) on top of it.
 * 12:30 shut down and restarted distribution pantasks because pcontrol was using a full cpu and progress has been slow.
 * 14:10 stopped processing to prepare to reset the nfs mounts for ipp010 ipp017 and ipp020
 * 14:38 all pantasks set to run

=== Wednesday : 2011-09-07 ===
 * 00:30 Mark: stdscience server down ~23:45 (was experiencing network connection during that time to ippc18, related?). restarted and running again.
 * 00:45 haf: removed ippc10 from pantasks - it's having some raid complaints and it scares me.
 * 07:50 Bill restarted registration pantasks. Removed all dates except for 2011-09-07. Registration failed for exp_id = 388654 and class_id = 'xy21' Set fault for that chip, then reverted. This started things moving again.
=== Thursday : 2011-09-08 ===

  * heather is czar.  reverted fault 4 on warp and stack (see if this clears it up)
=== Friday : 2011-09-09 ===

=== Saturday : 2011-09-10 ===

=== Sunday : 2011-09-11 ===