PS1 IPP Czar Logs for the week 2015.05.24 - 2015.05.31

(Up to PS1 IPP Czar Logs)

Monday : 2015.05.25

• 15:55 MEH: stdsci still having initday problem after first cycle.. restarting stdsci and manually sending things to cleanup after QUB stamps
• 16:30 MEH: stsci18 appears down again, power cycle --

Tuesday : 2015.05.26

• 06:00 Bill: stsci12 is down.
  • There are a large number of stdscience and pstamp jobs that are hung, probably due to nfs problems. The postage stamp server is completely stopped as a result of this. I restarted that pantasks. stdscience appears to be limping along.
• 06:30 EAM: just rebooted stsci12. it took two tries -- the first time it complained during the bios check of the raid about an invalid sas address. perhaps I did not leave it off long enough. it is back up now.
• 10:26 EAM: several repeatedly failing diffs (suspiciously all on the same skycell). I set to quality 42:

    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152360 -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152374 -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152402 -fault 0
    difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -skycell_id skycell.0685.025 -diff_id 1152419 -fault 0
  

• 10:30 EAM: ... and a problem warp:

    warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.1036.028 -warp_id 1577382 -fault 0
  

• 15:10 EAM: restarting pv3ffrt & pv3fflt
• 17:05 EAM: queuing RA = 19 for pv3ffrt, pv3fflt is still working on RA = 21, remote is working on RA = 15.
• 19:50 EAM: stopping and restarting ~ipp pantasks

Wednesday : 2015.05.27

• 07:25 EAM: restarting pv3ffrt & pv3fflt
• 17:10 CZW: restarting pv3ffrt & pv3fflt

Thursday : 2015.05.28

• 06:50 EAM: stsci01 was down, rebooted.
• 08:00 EAM: stopped pv3fflt and pv3ffrt; put the bulge label in pv3ffrt and assigned only himem xnodes (ippx001 - ippx0480

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD