PS1 IPP Czar Logs for the week 2015.01.05 - 2015.01.11

(Up to PS1 IPP Czar Logs)

Monday : 2015.01.05

• 09:15 EAM : stdlocal is sluggish, restarting it now.
• 10:20 EAM : ipp071 seems to be responding to logins again, so i've put it in neb repair (not down)
• 19:20 MEH: w/ ipp071 back available, the half faulted deepstacks can hopefully finish -- ~ippmd/deepstack/ptolemy.rc -- found possible problem, deepstack stop now

Tuesday : 2015.01.06

• 20:20 MEH: try md stacks again -- ~ippmd/deepstack/ptolemy.rc
• 20:50 EAM: summit is getting heavy winds so I'm leaving stdlocal at high load levels. I'll check again after the KP3 telecon (11pm).
• 21:15 EAM: stdlocal was at 120k warps so I'm restarting it.

Wednesday : 2015.01.07

• 10:44 HAF unjammed registration

   regtool -updateprocessedimfile -exp_id 848923 -class_id XY55 -set_state pending_burntool -dbname gpc1
  regtool -updateprocessedimfile -exp_id 848925 -class_id XY55 -set_state pending_burntool -dbname gpc1
  

• restarted registration (11:12, haf)

Thursday : 2015.01.08

• 03:45 Bill stdscience not making progress. Jobs in pcontrol queue without associated processes on the target host. No ppImage processes in 'who uses cluster page' Shutting down stdscience for a restart
  • unfortunately the who uses cluster page is not working with all of the hosts. There are some ppImage processes still running See for example ippc26
  • ganglia shows ipp059 is down (should have loooked there first. Unfortunately the console app won't accept my password when I try to connect with it
  • I am hesitant to start up a new stdscience pantasks with things in this state

Friday : 2015.01.09

Saturday : 2015.01.10

Sunday : 2015.01.11