== PS1 IPP Czar Logs for the week 2011.11.21 - 2011.11.27 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.11.21  ===
 * 09:20 ippc12 removed from processing for Cindy to swap out parts with ipp029. ipp029 neb-host set to down. when finished ipp029 neb-host repair again and put into full stdscience and distribution processing test (LAP).
 * 10:10 ipp055 set off/on in processing to reset pcontrol for coredumpsize 0 (just removed from deepstack over weekend to avoid 13GB cores) 
 * 16:00 Sine waves are back again [[Image(ipp_before.png)]]
 * 16:05 ipp009 was overwhelmed by three undead pantasks_client started by czarpoll on 11/19. Roy killed them explicitly and the waves disappeared
[[Image(ipp009.jpg)]]
[[Image(ipp_after.jpg)]]
 * 11:59 Mark: stdscience running slow, restarting in 30 if still low. restart helped.

=== Tuesday : 2011.11.22 ===
Chris is czar
 * 11:30 Mark: ipp-20111110 rebuild with Gene's psphotStack fixes added. processing stopped and restarted.
=== Wednesday : 2011.11.23 ===
 * 03:10 (HST, 510 PDT) Mark: ipp066 seems to have gone down ~10pm. not in .Consolerc so manually added (cab8) to personal version and rebooted. added quick log at [wiki:ipp066_log]. magic/DS may need some attention by others (dangerous by me).
=== Thursday : 2011.11.24 ===
 * 04:25 Bill noticed that distribution pantasks was falling behind and that it wasn't running jobs in a timely manner. pcontrol load was high even when pantasks was stopped. Restarted it.
 * 16:05 Bill ignored 5 lost raw files. 4 for LAP one for a postage stamp request. All had both missing instances on ippb02
{{{
ignore 167920 XY14
ignore 203641 XY02
ignore 203635 XY15
ignore 200143 XY33
ignore 193205 XY15
}}}

=== Friday : 2011.11.25 ===
 * 10:45 Bill restarted stdscience since it was only processing about 25 chips per hour. pcontrol had a high load and the number of jobs in the queues was small. We still haven't finished last night's data.

=== Saturday : 2011.11.26 ===

=== Sunday : 2011.11.27 ===