== PS1 IPP Czar Logs for the week 2011.09.26 - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.09.26 ===
  * 01:50 Mark: stdscience pantasks down. restarted.
=== Tuesday : 2011.09.27 ===
 * 10:20 Mark: ipp033 down, setting neb-host ipp033 down. 
 * 11:20 ipp033 seemed to come back on its own without reboot, setting neb-host back to up.
 * 15:30 Mark: added a sample iostat gmetric for ipp053 to track both normal IO for normal processing and Magic. Chose poor names and broke updates, fixed names but caused oddity with ganglia and plotting CPU on cluster load average summary plot, memory used on cluster memory summary plot, in on network summary plot. Reset gmetric timeout for all bad names and looks back to normal. Removed badly created RRDs.
=== Wednesday : 2011.09.28 ===
 * 09:00 Mark: registration stuck? similar to earlier in the night, XY05, but earlier case fixed itself?
{{{
regtool -updateprocessedimfile -exp_id 400406 -class_id XY05 -set_state pending_burntool -dbname gpc1
}}}
 * 16:50 Mark: with ippc18 overload, some pantasks_servers seem to be offline. working through restarting them.
=== Thursday : YYYY.MM.DD ===

=== Friday : 2011.09.30 ===
 * 02:00 Mark: Ken Smith reported unable to access some files from the datastore. mount to ipp042 from ippc17 failure the problem? Mount was working in morning, files also accessible. Someone fixed? Cindy and Gavin have setup sudo permissions for me.
 * 08:00 Registration got stuck around 0200 again. Ran regtool and restarted registration and things are catching up again
{{{
regtool -updateprocessedimfile -exp_id 401101  -class_id XY45 -set_state pending_burntool -dbname gpc1
}}}
 * 09:00 and a few more
{{{
regtool -updateprocessedimfile -exp_id 401108  -class_id XY45 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 401172 -class_id XY52 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 401192 -class_id XY15 -set_state pending_burntool -dbname gpc1
}}}
 * 09:30 more and done.
{{{
regtool -updateprocessedimfile -exp_id 401227 -class_id XY45 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 401249 -class_id XY45 -set_state pending_burntool -dbname gpc1
regtool -updateprocessedimfile -exp_id 401260 -class_id XY21 -set_state pending_burntool -dbname gpc1
}}}
 * 12:30 ipp026 crash. Serge rebooted.
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===