PS1 IPP Czar Logs for the week 2013.04.29 - 2013.05.05

(Up to PS1 IPP Czar Logs)

Monday :2013.04.29

mark is czar

• 07:00 MEH: lookslike 63 exposures stuck in registration, auto fixing of check_burntool problem not working now?
• 14:50 MEH: stsci nodes gone, abducted by aliens? no can reach by console so must be network issue. may want to do neb-host down if gone for a while -- Gavin was working on host config issue, all okay.
• 17:00 MEH: doing regular restart of stdscience

Tuesday : 2013.04.30

mark is czar

• 04:00 MEH: manually reverted some 40 stacks (ESS,MD), normal nfs faults seem to be happening more often now. auto-revert needed.. looks like typical random compute nodes having issue, maybe overloaded. last night and part of the SAS test stacks Chris ran were 90% from ippc40 in the stack pantask log (nothing clear in messages why and mounts okay).
• 11:10 MEH: enabling stack.revert -- removing all non-nightly science labels from ~ipp/stack/input, turned off in stack.pro task itself but adding stack.revert.on into the input file

Wednesday : 2013.05.01

• 13:30 Bill: stopped all processing and apache servers on ippc17 (datastore) and ipp049 (pstamp-test) to rebuild the ippRequestServer database on ippc17 using innodb tables and restore replication to ippc19
• 16:00 started up summit copy and registration

Thursday : 2013.05.02

• 04:00 EAM : reconfigured mysql on ipp005 - ipp009 : set innodb_buffer_pool_size back down to 2G from 16G. I had bumped these up in an attempt to help ipptopsps / dvopsps go faster (avoid swapping), but have since switched to memory engine for the dvoDetectionFull table causing the problem. I also killed some rpc.statds on ipp008 and ipp010.

Friday : 2013.05.03

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD