Changes between Version 109 and Version 110 of PS1_IPP_Czarlog_20121203

Timestamp:

Dec 7, 2012, 12:28:42 PM (14 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20121203

@@ -132 +132 @@
 === Friday : 2012.12.07 ===
  * 07:20 MEH: mounts on ipp056 gone inaccessible, stalled registration. 
-  * restarting rpc -- not helping 
+  * restarting rpc.statd -- not helping 
   * removing from processing
   * stopping, restarting registration, stdscience, summitcopy. killing hung jobs from stack (LAPs) failed so restarted as well.
@@ -138 +138 @@
  * 08:05 MEH: now wave4 machines are being overloaded (ipp058,054,066..). taking 1-2 instances out of processing as needed
  * 08:46 EAM : rsyncs to stsci machines were causing too much load.  I've stopped them, and I'll restart with lower bandwidth.
- * 09:00 MEH: rebooting ipp056, [wiki:ipp056_log], neb-host up 
+ * 09:00 MEH: rebooting ipp056, [wiki:ipp056_log], neb-host up after a bit
  * 09:02 Serge: Fixed replication (see [wiki:Replication_Issues])
  * 09:05 MEH: ipp054 now the glock tarpit, has frozen registration. has lost many mounts.
-  * restarting rpc and after a bit, mounts back and registration moving along
-  * then stalled again.  neb-host repair
+  * restarting rpc and after ~5-10 min mounts back and registration moving along
+  * then stalled again. put to neb-host repair
  * 09:28 EAM : my rsync system was allowing multiple jobs to hit the same target output machine.  I reworked the scripts to a single connection per source per target, so there should not be the same overloading.  also re-started with 5MB/sec instead of 8MB/sec per connection
  * 09:35 MEH: neb-host stsci up again, maybe help ease the load on wave4 
- * 09:45 MEH: ipp054 lost mounts to ipp030,023 again and restarting rpc not helping. rebooting [wiki:ipp054_log]
+ * 09:45 MEH: ipp054 lost mounts to ipp030,023 again and restarting rpc.statd not helping. rebooting [wiki:ipp054_log]
   * glocks now flushing, registration moving again
  * 10:40 registration finished, slowly putting the wave4 nodes back up