PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011.10.03

• 02:50 odd squelching behavior again on processing when plenty of exposures available. stdscience pantasks looks to go completely empty, then reloads. looks like started ~02:20 based on ganglia past hour load plot.
• 03:10 Mark: restarted stdscience to see if helps. fully reloaded by ~03:50.
• 04:10 looks to be happening again. no condor running so wasn't that. whatever it is also involves ippc11, looks like iqanalysis is running. processing back to constant levels ~04:40.
• 04:50 back again, no iqanalysis on ippc11. ssh into other systems 2-5x longer.
• 05:50 looks like processing moving back to regular levels? (by 06:05 yes) this behavior not clear in the ganglia 1d span, only hour span. seems to easily reduce the chip-warp rate to ~30/hr.
• 12:40 Mark: off'd the chip.revert until the missing instance file script is tested to stop filling up the pantasks log.
• 23:00 ipp029 unresponsive, ran neb-host down ipp029
• 23:47 CZW: cycled power on ipp029 as it was preventing LAP destreak from finishing correctly, holding up further processing.

Tuesday : 2011.10.04

• 03:00 Mark: LAP,MD03/4 fully loading processing queue, most of the oscillating load behavior seems to be gone with the ippc18-ippc19 rsync turned off. Asking Gavin/Cindy to set it to do 6am on Wednesday to watch the behavior again then.
• 06:45: roy: everything down from summit (not much, only 21 science exposures)

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD