PS1 IPP Czar Logs for the week 2015.07.06 - 2015.07.12

(Up to PS1 IPP Czar Logs)

Monday : 2015.07.06

• 06:55 Bill: stdscience is in pcontrol spin heck stopping to restart
  • 07:00 stdscience pantasks restarted
• 14:11 CZW: The logs indicated PSF generation failures, so I've set these to bad quality so the images flow on to MOPS.

warptool -updateskyfile -warp_id 1601763 -fault 0 -set_quality 42 -skycell_id skycell.2398.001 warptool -updateskyfile -warp_id 1601775 -fault 0 -set_quality 42 -skycell_id skycell.2217.014

• 16:55 CZW: stopping IPP pantasks for a nightly restart, as I'm czar tomorrow.

Tuesday : 2015.07.07

• 05:33 Bill: ipp017 is down. Set it to down in nebulous.
• 05:54 Bill: dropped two warps with "cannot build curve of growth error" -warp_id 1602637 -skycell_id skycell.0864.074 and -warp_id 1602720 -skycell_id skycell.2211.011
• 06:19 Bill: stdscience's queue is nearly full with jobs that have been running for > 5000 seconds. Some of the observations last night were apparently near M31 so it wouldn't be too shocking if psphot and camera were slow, but given that ipp017 is down they may just be stuck with nfs issues.
• 17:42 CZW: restarting the ipp pantasks.

Wednesday : 2015-07-08

• 12:40 CZW: Starting ~ippsky/staticsky pantasks to do staticsky updates. This will be using 2x loading on the x0,x1,x2,x3 nodes. Queue commands can be found in /data/ippc19.0/home/ippsky/queues/staticsky.updates. -updaterun does not accept the -pretend option, so as written, the commands will fail (although these shouldn't cause much if any problem if too many are active).

Thursday : 2015.07.09

• 05:20 MEH: restarting pstamp

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD