[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.12.17 ===
 Mark is czar
 * 06:50 Bill added hosts to deepstack 1 x (wave3, wave4, stare, compute2)
 * 07:35 MEH: registration pantasks seems to by busy for no reason on ipp052, restarting. not at 100% but still above 50% CPU, haven't noticed this before
 * 08:00 MEH: ippc18 home disk <30GB again, rsync'ing the pantasks logs to the Archive link (to ippc18.1) and compressing them through November
 * 09:18 Bill: outstanding staticsky runs have completed except for 27 with faults to be investigated. None of these are in the firs psps region. Stopping deepstack. Will restart with 1 x compute3 to work on the galactic center
 * 12:05 Bill: added one set of wave3 and one set of wave4 to deep stack
Bill is taking over since Mark is leaving town.
 * 16:44 ipp010's swap space has been ramping up. Set to repair in nebulous and restarted rpc.statd. mysqld is pretty busy there as well.
  * MEH: not really ramping more than past day, suspect depends on the cached mysql memory. interesting restarting rpc.statd twice didnt clear the swap used, maybe wasn't orphan rpc.statd processes?
  * MEH: don't forget to run ''/etc/init.d/gmond restart'' to clear the purple memory plot in ganglia
 * 16:55 decided to reboot ipp010  then set it back to up in nebulous
 * 17:45 or so set wave3 nodes in deepstack to off. 
 * 19:40 MEH: homdisk ippc18.0 back >150GB, few more Archive directories bzip'ing in registration to free up space on ippc18.1
 * 20:00 MEH: looks like wave4 is still in active use in deepstack, turning off in stdscience.
 * 20:05 MEH: appears wave3 still in use in deepstack from the long runtimes? no, looks like they are still on in deepstack.. turning off in stdscience as well until Bill clears them.
 * 20:20 MEH: MD02,03.GR0 V2->V3 chip-warp processing wrap-up setup to run when system is idle.
 * 20:25 MEH: ipp010 isn't doing well, lost mounts and stalling things.. putting back to repair
=== Tuesday : 2012.12.18 ===

=== Wednesday : 2012.12.19 ===

=== Thursday : 2012.12.20 ===

=== Friday : 2012.12.21 ===

=== Saturday : 2012.12.22 ===

=== Sunday : 2012.12.23 ===