PS1 IPP Czar Logs for the week 2015.05.24 - 2015.05.31

(Up to PS1 IPP Czar Logs)

Monday : 2015.05.25

• 15:55 MEH: stdsci still having initday problem after first cycle.. restarting stdsci and manually sending things to cleanup after QUB stamps
• 16:30 MEH: stsci18 appears down again, power cycle --

Tuesday : 2015.05.26

• 06:00 Bill: stsci12 is down.
  • There are a large number of stdscience and pstamp jobs that are hung, probably due to nfs problems. The postage stamp server is completely stopped as a result of this. I restarted that pantasks. stdscience appears to be limping along.
• 06:30 EAM: just rebooted stsci12. it took two tries -- the first time it complained during the bios check of the raid about an invalid sas address. perhaps I did not leave it off long enough. it is back up now.
• 10:26 EAM: several repeatedly failing diffs (suspiciously all on the same skycell). I set to quality 42:

    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152360 -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152374 -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152402 -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152419 -fault 0
  

• 10:30 EAM: ... and a problem warp:

    warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.1036.028 -warp_id 1577382 -fault 0
  

• 15:10 EAM: restarting pv3ffrt & pv3fflt
• 17:05 EAM: queuing RA = 19 for pv3ffrt, pv3fflt is still working on RA = 21, remote is working on RA = 15.
• 19:50 EAM: stopping and restarting ~ipp pantasks

Wednesday : 2015.05.27

• 07:25 EAM: restarting pv3ffrt & pv3fflt
• 17:10 CZW: restarting pv3ffrt & pv3fflt

Thursday : 2015.05.28

• 06:50 EAM: stsci01 was down, rebooted.
• 08:00 EAM: stopped pv3fflt and pv3ffrt; put the bulge label in pv3ffrt and assigned only himem xnodes (ippx001 - ippx048)
• 14:30 EAM: rebooted stsci16
• 15:00 HAF: CZW installed new fix to initday bug, recompiled ipp. HAF restarted stdsci + others
• 15:45 EAM: rebooted stsci18
• 18:09 CZW: rebooting stsci10
• 20:13 HAF:

  regtool -updateprocessedimfile -exp_id 920345 -class_id XY06 -set_state pending_burntool -dbname gpc1 
  

• 20:20 EAM: rebooting stsci17 (seems like an extra bad 24 hours -- 5 machines in 1 day! Did they sense that I talked to Prem??)

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD