== PS1 IPP Czar Logs for the week 2013-12-30 - 2014-01-05 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013-12-30 ===
Bill is the czar today
 * 07:48 Last night's data has finished chip processing. Turning chip.off for now.
 * 09:54 stopping stdscience to restart
  * cleared two diff faults difftool -updatediffskyfile -set_quality 14006 -fault 0  -skycell_id skycell.2466.010 -diff_id 510366 ; difftool -updatediffskyfile -set_quality 14006 -fault 0 -skycell_id skycell.2466.010 -diff_id 510375
  * 10:05 restarted, chip.off, poll set to 600
 * 10:15 restarted pstamp added a set of compute3 hosts to work on the mops stamps.
  * 11:04 mops stamps are done. pstamp is mostly idle now, the qub jobs need images to be updated.
 * 14:24 The cleanup and distribution processing is proceeding very slowly. I suspect that the rsync jobs are part of the problem, but cleanup's pcontrol is spinning severely. Setting cleanup to stop for a bit.
 * 15:30 changed the interval between gpc1 mysql dumps on ippdb03 from 4 to 6 hours. The previous file had not finished it's copy before the next one started, leading to checksum comparison problems.
 * 15:40 restarted cleanup pantasks. Set sts chip and warp data that has already been bundled for distribtuion to be cleaned.
=== Tuesday : 2013-12-31 ===
 * 06:45 summit copy and registration are stuck. Didn't get very much data last night
  * set all pantasks to stop
  * registration process hung on ipp056. Cannot commincate with nfs server ipp029 killed it
  * restarted registration and summit copy pantasks. Very many task timeouts in the status page before restart. took ipp056 out of the host list
  * 06:55 summit copy and registration are proceeding now.
  * summit copy has several stuck processes on ipp056. Will reboot it.

=== Wednesday : 2014-01-01 ===

=== Thursday : 2014-01-02 ===

=== Friday : 2014-01-03 ===

=== Saturday : 2014-01-04 ===

=== Sunday : 2014-01-05 ===