PS1 IPP Czar Logs for the week 2011.09.26 - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011.09.26

• 01:50 Mark: stdscience pantasks down. restarted.

Tuesday : 2011.09.27

• 10:20 Mark: ipp033 down, setting neb-host ipp033 down.
• 11:20 ipp033 seemed to come back on its own without reboot, setting neb-host back to up.
• 15:30 Mark: added a sample iostat gmetric for ipp053 to track both normal IO for normal processing and Magic. Chose poor names and broke updates, fixed names but caused oddity with ganglia and plotting CPU on cluster load average summary plot, memory used on cluster memory summary plot, in on network summary plot. Reset gmetric timeout for all bad names and looks back to normal. Removed badly created RRDs.

Wednesday : 2011.09.28

• 09:00 Mark: registration stuck? similar to earlier in the night, XY05, but earlier case fixed itself?

  regtool -updateprocessedimfile -exp_id 400406 -class_id XY05 -set_state pending_burntool -dbname gpc1
  

• 16:50 Mark: with ippc18 overload, some pantasks_servers seem to be offline. working through restarting them.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD