== PS1 IPP Czar Logs for the week 2015.01.05 - 2015.01.11 == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : 2015.01.05 === * 09:15 EAM : stdlocal is sluggish, restarting it now. * 10:20 EAM : ipp071 seems to be responding to logins again, so i've put it in neb repair (not down) * 19:20 MEH: w/ ipp071 back available, the half faulted deepstacks can hopefully finish -- ~ippmd/deepstack/ptolemy.rc -- found possible problem, deepstack stop now === Tuesday : 2015.01.06 === * 20:20 MEH: try md stacks again -- ~ippmd/deepstack/ptolemy.rc * 20:50 EAM: summit is getting heavy winds so I'm leaving stdlocal at high load levels. I'll check again after the KP3 telecon (11pm). * 21:15 EAM: stdlocal was at 120k warps so I'm restarting it. === Wednesday : 2015.01.07 === * 10:44 HAF unjammed registration {{{ regtool -updateprocessedimfile -exp_id 848923 -class_id XY55 -set_state pending_burntool -dbname gpc1 regtool -updateprocessedimfile -exp_id 848925 -class_id XY55 -set_state pending_burntool -dbname gpc1 }}} * restarted registration (11:12, haf) === Thursday : 2015.01.08 === * 03:45 Bill stdscience not making progress. Jobs in pcontrol queue without associated processes on the target host. No ppImage processes in 'who uses cluster page' Shutting down stdscience for a restart * unfortunately the who uses cluster page is not working with all of the hosts. There are some ppImage processes still running See for example ippc26 * ganglia shows ipp059 is down (should have loooked there first. Unfortunately the console app won't accept my password when I try to connect with it * I am hesitant to start up a new stdscience pantasks with things in this state === Friday : 2015.01.09 === === Saturday : 2015.01.10 === === Sunday : 2015.01.11 ===