== PS1 IPP Czar Logs for the week 2014.03.23 - 2014.03.29 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.03.23 ===
  * 03:04 Bill processing seems to be going smoothly. Checked the database and found a 537 long threadlist. There was one thread from ippc09 with a runtime of 5070 seconds. Killled it and the number of threads dropped to normal (~143). The previous troublesome thread (see last czarlog) was also from ippc09.
 * 05:37 Bill cleared repeating warp fault warptool -updateskyfile -set_quality 42 -fault 0 -warp_id 1515631 -skycell_id skycell.1474.065
 * 20:00 EAM purged ippdb00 binlogs to mysqld-bin.005022
 * 20:01 EAM stopping & restarting pv3diff pantasks

=== Tuesday :  2014.03.24 ===
 * 08:24 Bill restarted postage stamp server pantasks. There are a lot of MPE requests waiting to be cleaned. I've removed the MPE label for now to allow that backlog to clear out.
  * 09:47 pstamp cleanup finished MPE label re-enabled
 * 16:17 CZW: Bill emailed that there were nebulous thread issues.  I did a series of {{{mysqladmin -u root kill 568262}}} commands, and this seems to have resolved the issue.
 * 19:55 EAM: stopping and restarting the pv3diff pantasks

=== Wednesday : 2014.03.25 ===

 * 15:30 CZW: stopped processing so Haydn can change a battery on ipp094.  
 * 16:30 CZW: shutdown/restarting ipp and pv3diff pantasks so that when we get the ipp094 all clear, they can start with fresh pcontrols.
 * 23:00 MEH: adding SNIa.WS.nightlyscience to regular processing now as well 
=== Thursday : 2014.03.26 ===
 * 03:50 Bill: 3 mysql threads connected to the nebulous database were stuck. Killed them off.

=== Friday : 2014.03.27 ===

=== Saturday : 2014.03.28 ===

=== Sunday : 2014.03.29 ===