== PS1 IPP Czar Logs for the week 2013.04.29 - 2013.05.05 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday :2013.04.29 ===
  mark is czar
 * 07:00 MEH: lookslike 63 exposures stuck in registration, auto fixing of check_burntool problem not working now?
 * 14:50 MEH: stsci nodes gone, abducted by aliens? no can reach by console so must be network issue. may want to do neb-host down if gone for a while -- Gavin was working on host config issue, all okay.
 * 17:00 MEH: doing regular restart of stdscience
=== Tuesday : 2013.04.30 ===
 mark is czar
 * 04:00 MEH: manually reverted some 40 stacks (ESS,MD), normal nfs faults seem to be happening more often now. auto-revert needed.. looks like typical random compute nodes having issue, maybe overloaded. last night and part of the SAS test stacks Chris ran were 90% from ippc40 in the stack pantask log (nothing clear in messages why and mounts okay).
 * 11:10 MEH: enabling stack.revert -- removing all non-nightly science labels from ~ipp/stack/input,  turned off in stack.pro task itself but adding stack.revert.on into the input file 
=== Wednesday : 2013.05.01 ===
 * 13:30 Bill: stopped all processing and apache servers on ippc17 (datastore) and ipp049 (pstamp-test) to rebuild the ippRequestServer database on ippc17 using innodb tables and restore replication to ippc19
 * 16:00 started up summit copy and registration
=== Thursday : 2013.05.02 ===

 * 04:00 EAM : reconfigured mysql on ipp005 - ipp009 : set innodb_buffer_pool_size back down to 2G from 16G.  I had bumped these up in an attempt to help ipptopsps / dvopsps go faster (avoid swapping), but have since switched to memory engine for the dvoDetectionFull table causing the problem.  I also killed some rpc.statds on ipp008 and ipp010.

=== Friday : 2013.05.03  ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===