PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2015.05.11

• Haydn looking into ipp029 not booting -- had to replace motherboard twice. back to neb-host repair and leaving out of nightly tonight
• 16:00 MEH: Gene done rsync ipp091,092, neb-host up again
• 19:40 MEH: stsci13 seems to be unresponsive for the past hour -- power cycling
• 20:30 EAM: restarting full-force
• 21:30 MEH: doing some cleanup missed last week from all the system up/down issues

Tuesday : 2015.05.12

• 10:25 MEH: ipp029 still down, needs to have neb-host down
• 15:00 CZW: ipp/replication pantasks running warp fpa.res.fits removal. Currently just CNP, but I'll follow with the MD and 3PI data as the files complete.
• 16:15 CZW: ipp029 back up, back to neb-host repair.
• 16:45 CZW: ganglia reports the load on the cluster has dropped, which seems to be a symptom of the ff pending list dropping. We've been running a bit faster than previously, so this may be part of the reason (we're going faster than the define step is). I'm going to take this chance to restart the local ff pantasks.
• 19:55 EAM: I'm stopping pv3ffrt and pv3fflt for a while so I can run a dvomerge of the tycho data into the pv2 dvo. this will also give the ff define jobs to make some headway.

Wednesday : 2015.05.13

• 04:39 Bill: gpc1 chip processing is about 170 exposures behind with throughput of about 25 exposures. I suspect that part of the problem is that stdscience pantasks has been running for several days. I'm going to restart it.
• 06:47 Bill: cleared a gpc2 diff fault with difftool -updatediffskyfile -dbname gpc2 -diff_id 1389 -skycell_id skycell.2067.089 -set_quality 42 -fault 0

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD