PS1_IPP_CzarLog_20111003 – Pan-STARRS IPP

Context Navigation

Version 26 (modified by bills, 15 years ago) ( diff )
--

02:50 odd squelching behavior again on processing when plenty of exposures available. stdscience pantasks looks to go completely empty, then reloads. looks like started ~02:20 based on ganglia past hour load plot.
03:10 Mark: restarted stdscience to see if helps. fully reloaded by ~03:50.
04:10 looks to be happening again. no condor running so wasn't that. whatever it is also involves ippc11, looks like iqanalysis is running. processing back to constant levels ~04:40.
04:50 back again, no iqanalysis on ippc11. ssh into other systems 2-5x longer.
05:50 looks like processing moving back to regular levels? (by 06:05 yes) this behavior not clear in the ganglia 1d span, only hour span. seems to easily reduce the chip-warp rate to ~30/hr.
12:40 Mark: off'd the chip.revert until Chris turns on the missing instance file script in order to stop filling up the pantasks log.
23:00 ipp029 unresponsive, ran neb-host down ipp029
23:47 CZW: cycled power on ipp029 as it was preventing LAP destreak from finishing correctly, holding up further processing.

03:00 Mark: LAP,MD03/4 fully loading processing queue, most of the oscillating load behavior seems to be gone with the ippc18-ippc19 rsync turned off. Asking Gavin/Cindy to set it to do 6am on Wednesday to watch the behavior again then.
06:45: roy: everything down from summit (not much, only 21 science exposures)
11:40 Mark: was running 1 node for stacks last night on ippc11 and seemed okay. How many pantasks stack jobs does it take to crack c11? Looks like 4 or less.

07:20 Mark: Gavin moved the ippc18->ippc19 backup rsync to start at 6am and the oscillating behavior is seen at that time now. Hard to tell how much of a hit on processing rate with the change from nightly science to LAP processing. By 9:40 rate decrease is clear by 40-50%. Will ask Gavin to leave rsync at this time (~6-10am) until a solution can be found .
13:50 Bill set data with label like 'ps_ud%' to be cleaned up. This will free up about 1000 exposures worth of chip data and a couple of hundred warps.
14:04 Bill: The slow rsync problem may be happening because the log files have grown very large and thus take a long time to copy. I changed start_server.sh to move the pantasks_logs directory into logs/YYYYMM. Restarted all pantasks except for stack. It is stopped and I'm letting the queue empty. (Mark's MD deep stacks may take an hour or so)
14:49 Decided to let stack continue. Set pantasks state to run.