(Up to PS1 IPP Czar Logs)

Monday : 2012.12.17

Mark is czar

• 06:50 Bill added hosts to deepstack 1 x (wave3, wave4, stare, compute2)
• 07:35 MEH: registration pantasks seems to by busy for no reason on ipp052, restarting. not at 100% but still above 50% CPU, haven't noticed this before
• 08:00 MEH: ippc18 home disk <30GB again, rsync'ing the pantasks logs to the Archive link (to ippc18.1) and compressing them through November
• 09:18 Bill: outstanding staticsky runs have completed except for 27 with faults to be investigated. None of these are in the firs psps region. Stopping deepstack. Will restart with 1 x compute3 to work on the galactic center
• 12:05 Bill: added one set of wave3 and one set of wave4 to deep stack

Bill is taking over since Mark is leaving town.

• 16:44 ipp010's swap space has been ramping up. Set to repair in nebulous and restarted rpc.statd. mysqld is pretty busy there as well.
  • MEH: not really ramping more than past day, suspect depends on the cached mysql memory. interesting restarting rpc.statd twice didnt clear the swap used, maybe wasn't orphan rpc.statd processes?
  • MEH: don't forget to run /etc/init.d/gmond restart to clear the purple memory plot in ganglia
• 16:55 decided to reboot ipp010 then set it back to up in nebulous
• 17:45 or so set wave3 nodes in deepstack to off.
• 19:40 MEH: homdisk ippc18.0 back >150GB, few more Archive directories bzip'ing in registration to free up space on ippc18.1
• 20:00 MEH: looks like wave4 is still in active use in deepstack, turning off in stdscience.
• 20:05 MEH: appears wave3 still in use in deepstack from the long runtimes? no, looks like they are still on in deepstack.. turning off in stdscience as well until Bill clears them.

Tuesday : 2012.12.18

Wednesday : 2012.12.19

Thursday : 2012.12.20

Friday : 2012.12.21

Saturday : 2012.12.22

Sunday : 2012.12.23