(Up to PS1 IPP Czar Logs)

Monday : 2012.12.17

Mark is czar

• 06:50 Bill added hosts to deepstack 1 x (wave3, wave4, stare, compute2)
• 07:35 MEH: registration pantasks seems to by busy for no reason on ipp052, restarting. not at 100% but still above 50% CPU, haven't noticed this before
• 08:00 MEH: ippc18 home disk <30GB again, rsync'ing the pantasks logs to the Archive link (to ippc18.1) and compressing them through November
• 09:18 Bill: outstanding staticsky runs have completed except for 27 with faults to be investigated. None of these are in the firs psps region. Stopping deepstack. Will restart with 1 x compute3 to work on the galactic center
• 12:05 Bill: added one set of wave3 and one set of wave4 to deep stack

Bill is taking over since Mark is leaving town.

• 16:44 ipp010's swap space has been ramping up. Set to repair in nebulous and restarted rpc.statd. mysqld is pretty busy there as well.
  • MEH: not really ramping more than past day, suspect depends on the cached mysql memory. interesting restarting rpc.statd twice didnt clear the swap used, maybe wasn't orphan rpc.statd processes?
  • MEH: don't forget to run /etc/init.d/gmond restart to clear the purple memory plot in ganglia
• 16:55 decided to reboot ipp010 then set it back to up in nebulous
• 17:45 or so set wave3 nodes in deepstack to off.
• 19:40 MEH: homdisk ippc18.0 back >150GB, few more Archive directories bzip'ing in registration to free up space on ippc18.1
• 20:00 MEH: looks like wave4 is still in active use in deepstack, turning off in stdscience.
• 20:05 MEH: appears wave3 still in use in deepstack from the long runtimes? no, looks like they are still on in deepstack.. turning off in stdscience as well until Bill clears them.
• 20:20 MEH: MD02,03.GR0 V2->V3 chip-warp processing wrap-up setup to run when system is idle.

Tuesday : 2012.12.18

Wednesday : 2012.12.19

Thursday : 2012.12.20

Friday : 2012.12.21

Saturday : 2012.12.22

Sunday : 2012.12.23