== PS1 IPP Czar Logs for the week 2015.05.24 - 2015.05.31 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.05.25 ===
 * 15:55 MEH: stdsci still having initday problem after first cycle.. restarting stdsci and manually sending things to cleanup after QUB stamps
 * 16:30 MEH: stsci18 appears down again, power cycle --

=== Tuesday : 2015.05.26 ===
 * 06:00 Bill: stsci12 is down.
  * There are a large number of stdscience and pstamp jobs that are hung, probably due to nfs problems. The postage stamp server is completely stopped as a result of this. I restarted that pantasks. stdscience appears to be limping along.
 * 06:30 EAM: just rebooted stsci12.  it took two tries -- the first time it complained during the bios check of the raid about an invalid sas address.  perhaps I did not leave it off long enough.  it is back up now.
 * 10:26 EAM: several repeatedly failing diffs (suspiciously all on the same skycell).  I set to quality 42:
{{{
  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152360 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152374 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152402 -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152419 -fault 0
}}}
 * 10:30 EAM: ... and a problem warp:
{{{
  warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.1036.028 -warp_id 1577382 -fault 0
}}}
 * 15:10 EAM: restarting pv3ffrt & pv3fflt
 * 17:05 EAM: queuing RA = 19 for pv3ffrt, pv3fflt is still working on RA = 21, remote is working on RA = 15.
 * 19:50 EAM: stopping and restarting ~ipp pantasks

=== Wednesday : 2015.05.27 ===
 * 07:25 EAM: restarting pv3ffrt & pv3fflt
 * 17:10 CZW: restarting pv3ffrt & pv3fflt

=== Thursday : 2015.05.28 ===

 * 06:50 EAM: stsci01 was down, rebooted.
 * 08:00 EAM: stopped pv3fflt and pv3ffrt; put the bulge label in pv3ffrt and assigned only himem xnodes (ippx001 - ippx048)
 * 14:30 EAM: rebooted stsci16
 * 15:00 HAF: CZW installed new fix to initday bug, recompiled ipp.  HAF restarted stdsci + others
 * 15:45 EAM: rebooted stsci18
 * 18:09 CZW: rebooting stsci10
 
 * 20:13 HAF: {{{ regtool -updateprocessedimfile -exp_id 920345 -class_id XY06 -set_state pending_burntool -dbname gpc1
}}}
=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===