Changes between Version 16 and Version 17 of PS1_IPP_Czarlog_20140106

Timestamp:

Jan 10, 2014, 2:00:14 PM (13 years ago)

Author:

watersc1

Comment:

--

PS1_IPP_Czarlog_20140106

@@ -28 +28 @@
  * 10:45 CZW: I noticed things were clogged earlier.  I restarted apache on a number of the nebulous server hosts, which has allowed nebulous requests to complete again (neb-touch timed out, and a number of summit copy jobs had similar issues).  This doesn't seem to have solved everything, and the load on stsci02 (250-260?) is very suspicious.  I've asked Gavin to reboot this (although since it is accessible, we can do command line reboots.  I did not realize this before now).
  * 11:45 CZW: That didn't seem to work as well as I'd hoped, as the load on stsci02 immediately shot back up to 258.  I'm doing the umount -f thing on all hosts that are accessing stsci02.
+ * 14:00 CZW: stsci02 is still running with a load of 260, and the force umounting has done nothing to fix things.  The /var/log/messages contain entries like {{{Jan 10 13:45:01 stsci02 [10083.305233] Controller in crit error}}} which google suggests are RAID issues.  I'm beginning to suspect that this is causing all disk accesses to hang, and that's causing the problems.  I have stsci02 set to repair in nebulous, which should prevent things from writing new data there.  Since the remaining nightly data downloaded after I did this, we should be able to complete that without much difficulty.  
 === Saturday : 2014-01-11 ===