Changes between Version 35 and Version 36 of PS1_IPP_CzarLog_20120312

Timestamp:

Mar 13, 2012, 3:05:39 PM (14 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_CzarLog_20120312

@@ -38 +38 @@
  Mark is czar
  * 07:00 ipp015 is down, and not rebooting... looks like needs someone to physically go check out.
- * 07:30 setting neb-host down ipp015, stopping and shutting down most pantasks while attempting to untangle ipp015
+ * 07:30 setting neb-host down ipp015, stopping and shutting down most pantasks while attempting to untangle things that needed ipp015. not sure what to do about addstar.
  * 08:00 turning summitcopy back on (while going into office), seems to be downloading somewhat.
  * 09:10 Serge: Nebulous snafu. Restarted the mysql server on ippdb00.
  * 10:05 Serge: All apache/nebulous servers have been restarted.
- * 11:00 still killing off, cleaning things up
- * 11:30 restarting things, putting ipp015 into the hosts_ignore_wave1 list (do we want to comment it out completely in the wave1 list?)
+ * 11:00 still killing off hung processes. 
+ * 11:30 restarting things, putting ipp015 into the hosts_ignore_wave1 list (do we want to comment it out completely in the wave1 list?). Bill taking care of postage stamp server.
  * 12:13 CZW: I've modified a copy of repair_bad_instance, and fed it with an input list generated from Serge's slocate databases of ota26 entries on ipp064.  It's currently running singly threaded, and checking all of these raw fits files and repairing bad instances when possible.  This is copying good copies from the ATRC, but as it should not be doing more than one at a time, I do not expect this to overly tax the ATRC computers/ATRC AC.
  * 12:15 Serge: Killed ippdb03 scheduled mysqldump to perform a manual one that I intend to ingest on the broken ipp001. Replication on ippdb03 has been stopped on purpose.
- * 12:40 - nightly science warp faults mostly due to processed chip data needed on ipp015.
+ * 12:40 nightly science cam/warp/stack faults mostly due to processed chip data needed that lives on ipp015. same goes for LAP so LAP is effectively stalled.
  * 12:50 replication pantasks back running, target.on only.
  * 13:00 Gavin setup access to the power control for us on compute3 nodes
  * 14:10 Serge: Restarted replication on ippdb03. {{{CHANGE MASTER TO MASTER_LOG_FILE='mysqld-bin.020936', MASTER_LOG_POS=73345738;}}}
- * 14:30 registration has been very unhappy. ipp011 often having lockd server not responding for ipp008,ipp014 so set those two to repair.
- * 15:00 all data downloaded and registered from last night finally.
+ * 14:30 registration has been very unhappy. ipp011 often having lockd server not responding for ipp008,ipp014 so set those two to repair and will watch with tonight's data.
+ * 15:00 all data downloaded and registered from last night, finally.
 === Wednesday : YYYY.MM.DD ===