PS1 IPP Czar Logs for the week 2011.11.21 - 2011.11.27

(Up to PS1 IPP Czar Logs)

Monday : 2011.11.21

• 09:20 ippc12 removed from processing for Cindy to swap out parts with ipp029. ipp029 neb-host set to down. when finished ipp029 neb-host repair again and put into full stdscience and distribution processing test (LAP).
• 10:10 ipp055 set off/on in processing to reset pcontrol for coredumpsize 0 (just removed from deepstack over weekend to avoid 13GB cores)
• 16:00 Sine waves are back again
• 16:05 ipp009 was overwhelmed by three undead pantasks_client started by czarpoll on 11/19. Roy killed them explicitly and the waves disappeared

• 11:59 Mark: stdscience running slow, restarting in 30 if still low. restart helped.

Tuesday : 2011.11.22

Chris is czar

• 11:30 Mark: ipp-20111110 rebuild with Gene's psphotStack fixes added. processing stopped and restarted.

Wednesday : 2011.11.23

• 03:10 (HST, 510 PDT) Mark: ipp066 seems to have gone down ~10pm. not in .Consolerc so manually added (cab8) to personal version and rebooted. added quick log at ipp066_log. magic/DS may need some attention by others (dangerous by me).

Thursday : 2011.11.24

• 04:25 Bill noticed that distribution pantasks was falling behind and that it wasn't running jobs in a timely manner. pcontrol load was high even when pantasks was stopped. Restarted it.
• 16:05 Bill ignored 5 lost raw files. 4 for LAP one for a postage stamp request. All had both missing instances on ippb02

  ignore 167920 XY14
  ignore 203641 XY02
  ignore 203635 XY15
  ignore 200143 XY33
  ignore 193205 XY15
  

Friday : 2011.11.25

• 10:45 Bill restarted stdscience since it was only processing about 25 chips per hour. pcontrol had a high load and the number of jobs in the queues was small. We still haven't finished last night's data.
• (that increased the rate by the way)
• 19:10 Bill ipp025 needed to be rebooted. Unfortunately it didn't come back up. This caused ipp017's load to ramp up because hung apache processes began to multiply. Chanegd automount map to soft mount ipp025.0 and rebooted ippc17. Noqw it's getting along all right.
• 19:54 Cindy got ipp025 to boot by adjusting BIOS settings

Saturday : 2011.11.26

Sunday : 2011.11.27