PS1 IPP Czar Logs for the week 2011.11.21 - 2011.11.27

(Up to PS1 IPP Czar Logs)

Monday : 2011.11.21

• 09:20 ippc12 removed from processing for Cindy to swap out parts with ipp029. ipp029 neb-host set to down. when finished ipp029 neb-host repair again and put into full stdscience and distribution processing test (LAP).
• 10:10 ipp055 set off/on in processing to reset pcontrol for coredumpsize 0 (just removed from deepstack over weekend to avoid 13GB cores)
• 16:00 Sine waves are back again
• 16:05 ipp009 was overwhelmed by three undead pantasks_client started by czarpoll on 11/19. Roy killed them explicitly and the waves disappeared

• 11:59 Mark: stdscience running slow, restarting in 30 if still low. restart helped.

Tuesday : 2011.11.22

Chris is czar

• 11:30 Mark: ipp-20111110 rebuild with Gene's psphotStack fixes added. processing stopped and restarted.

Wednesday : 2011.11.23

• 03:10 (HST, 510 PDT) Mark: ipp066 seems to have gone down ~10pm. not in .Consolerc so manually added (cab8) to personal version and rebooted. added quick log at ipp066_log. magic/DS may need some attention by others (dangerous by me).

Thursday : 2011.11.24

• 04:25 Bill noticed that distribution pantasks was falling behind and that it wasn't running jobs in a timely manner. pcontrol load was high even when pantasks was stopped. Restarted it.
• 16:05 Bill ignored 5 lost raw files. 4 for LAP one for a postage stamp request. All had both missing instances on ippb02

  ignore 167920 XY14
  ignore 203641 XY02
  ignore 203635 XY15
  ignore 200143 XY33
  ignore 193205 XY15
  

Friday : 2011.11.25

• 10:45 Bill restarted stdscience since it was only processing about 25 chips per hour. pcontrol had a high load and the number of jobs in the queues was small. We still haven't finished last night's data.

Saturday : 2011.11.26

Sunday : 2011.11.27