(Up to PS1 IPP Czar Logs)

Monday : 2012.12.17

Mark is czar

• 06:50 Bill added hosts to deepstack 1 x (wave3, wave4, stare, compute2)
• 07:35 MEH: registration pantasks seems to by busy for no reason on ipp052, restarting. not at 100% but still above 50% CPU, haven't noticed this before
• 08:00 MEH: ippc18 home disk <30GB again, rsync'ing the pantasks logs to the Archive link (to ippc18.1) and compressing them through November
• 09:18 Bill: outstanding staticsky runs have completed except for 27 with faults to be investigated. None of these are in the firs psps region. Stopping deepstack. Will restart with 1 x compute3 to work on the galactic center
• 12:05 Bill: added one set of wave3 and one set of wave4 to deep stack

Bill is taking over since Mark is leaving town.

• 16:44 ipp010's swap space has been ramping up. Set to repair in nebulous and restarted rpc.statd. mysqld is pretty busy there as well.
  • MEH: not really ramping more than past day, suspect depends on the cached mysql memory. interesting restarting rpc.statd twice didnt clear the swap used, maybe wasn't orphan rpc.statd processes?
  • MEH: don't forget to run /etc/init.d/gmond restart to clear the purple memory plot in ganglia
• 16:55 decided to reboot ipp010 then set it back to up in nebulous
• 17:45 or so set wave3 nodes in deepstack to off.
• 19:40 MEH: homdisk ippc18.0 back >150GB, few more Archive directories bzip'ing in registration to free up space on ippc18.1
• 20:00 MEH: looks like wave4 is still in active use in deepstack, turning off in stdscience.
• 20:05 MEH: appears wave3 still in use in deepstack from the long runtimes? no, looks like they are still on in deepstack.. turning off in stdscience as well until Bill clears them.
• 20:20 MEH: MD02,03.GR0 V2->V3 chip-warp processing wrap-up setup to run when system is idle.
• 20:25 MEH: ipp010 isn't doing well with normal processing, lost mounts and stalling things.. putting back to repair
• 20:35 ipp010 caught up, taking one instance out of processing and leaving in repair unless someone else wants to watch it
• 20:49 Bill: set wave3 to off in deepstack since the nodes are being used elsewhere. Killed the swapping psphotStack jobs except for the one on ipp050 which is doing detection efficiency so it is *almost* done so trying to let it finish.

Tuesday : 2012.12.18

• 05:20 MEH: succeeded(?) in glock stalling to ipp062 for good 20 mins from other nodes with MD02 chip (some warp) stage and CZW SAS processing, not sure what pushed it over. seems okay again.
• 06:30 MEH: ipp010 should have mysql restarted after its reboot sudo /etc/init.d/mysql start
• 07:30 MEH: MD02,03.GR0 now also running stacks in stack pantasks when no other stacks to do
• sometime Bill: copied a valid raw imfile on top of a broken instance of one of the MD03 raw images

Wednesday : 2012.12.19

Thursday : 2012.12.20

Friday : 2012.12.21

Saturday : 2012.12.22

Sunday : 2012.12.23