== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.10.03 ===
 * 02:50 odd squelching behavior again on processing when plenty of exposures available. stdscience pantasks looks to go completely empty, then reloads. looks like started ~02:20 based on ganglia past hour load plot.
 * 03:10 Mark: restarted stdscience to see if helps. fully reloaded by ~03:50.
 * 04:10 looks to be happening again. no condor running so wasn't that. whatever it is also involves ippc11, looks like iqanalysis is running. processing back to constant levels ~04:40.
 * 04:50 back again, no iqanalysis on ippc11. ssh into other systems 2-5x longer.
 * 05:50 looks like processing moving back to regular levels? (by 06:05 yes) this behavior not clear in the ganglia 1d span, only hour span. seems to easily reduce the chip-warp rate to ~30/hr.
 * 12:40 Mark: off'd the chip.revert until Chris turns on the missing instance file script in order to stop filling up the pantasks log.
 * 23:00 ipp029 unresponsive, ran neb-host down ipp029
 * 23:47 CZW: cycled power on ipp029 as it was preventing LAP destreak from finishing correctly, holding up further processing.
=== Tuesday : 2011.10.04 ===
 * 03:00 Mark: LAP,MD03/4 fully loading processing queue, most of the oscillating load behavior seems to be gone with the ippc18-ippc19 rsync turned off. Asking Gavin/Cindy to set it to do 6am on Wednesday to watch the behavior again then.
 * 06:45: roy: everything down from summit (not much, only 21 science exposures)
 * 11:40 Mark: was running 1 node for stacks last night on ippc11 and seemed okay. How many pantasks stack jobs does it take to crack c11? Looks like 4 or less.
=== Wednesday : 2011.10.05 ===
 * 07:20 Mark: Gavin moved the ippc18->ippc19 backup rsync to start at 6am and the oscillating behavior is seen at that time now. Hard to tell how much of a hit on processing rate with the change from nightly science to LAP processing. By 9:40 rate decrease is clear by 40-50%. Will ask Gavin to leave rsync at this time (~6-10am) until a solution can be found .
 * 13:50 Bill set data with label like 'ps_ud%' to be cleaned up. This will free up about 1000 exposures worth of chip data and a couple of hundred warps.
 * 14:04 Bill: The slow rsync problem may be happening because the log files have grown very large and thus take a long time to copy. I changed start_server.sh to move the pantasks_logs directory into logs/YYYYMM. Restarted all pantasks except for stack. It is stopped and I'm letting the queue empty. (Mark's MD deep stacks may take an hour or so)
 * 14:49 Decided to let stack continue. Set pantasks state to run.

=== Thursday : 2011.10.06 ===
 * 08:00 re-nice'ing rsync ippc18->ippc19 not helping much. Will look at a bandwidth limit along with Bill's log rotation of the large log files Friday morning. 
Bill is czar today
 * 10:00 We have 2 repeat faults. 5 warp skycells faile due to a corrupt camera mask file. Rerunning cam_id 288973. The other is a stack failure due to a corrupt warp. Unfortunately the chipRun has been cleaned already. Set the chip run 308628 to update then will rerun the warp.
 * ~10:05 something was started that heavily loaded ippc18 (NFS?) and tanked processing for ~10min
 * 15:34 Restarted stdscience and distribution
 * 22:00 Mark: ipp029 unresponsive, set neb-host down but don't have access to reboot.
=== Friday : 2011.10.07 ===
 * 07:40 rsync ippc18->ippc19 now quick (~20 mins) with Bill's rotation of the logs. A 10MB/s bandwidth limit was also added to help smooth out any possible large transfers in the future. Leaving the start time at 6am to watch if it becomes a problem again. 
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===