Changes between Version 7 and Version 8 of PS1_IPP_Czarlog_20190422

Timestamp:

Apr 22, 2019, 9:32:17 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190422

@@ -80 +80 @@
  * MEH: adding QUB.2 to pstamp server for bulk QUB requests (similar to MOPS.2, large/long request not needed ASAP), ps_ud_QUB.2 label and updates done in ippqub:stdscience_ws pantask -- lots of faults, czar not need to worry about at this time and due to the ongoing issue of multiple copies of needed files only on the ippb nodes and none on the actual IPP-ITC cluster...
 
+ * TdB: Around 21:00 I noticed that data was no longer downloading from the summit. A quick look at the summitcopy pantask stdout log showed that MySQL was down, which was causing the problem. Gene checked the process list for mysql:
+{{{
+ps aux | grep mys
+}}}
+ it was not listed, so it had crashed or had been shutdown. Looking at /var/log/mysql/mysqld.err showed the messages about the crash. Gene restarted MySQL on ippdb09 as follows:
+{{{
+restarting MySQL:
+
+/etc/init.d/mysql start
+
+(it complains about the PID file being left behind, but tries to start anyway.  The start also takes too long, and the startup script gives up and claims the start fails.  NOTE this is not true: you need to check the process list (ps aux | grep mys) and check for mysql.  It will probably be there, it is just taking a long time to start up.  You can check the log file (tail -f /var/log/mysql/mysqld.err) and wait for it to say "ready for connections.").
+}}} 
+ Once mysql was back up, nebulous started working again and downloads proceeded OK.
+
+
+
 === Tuesday : 2019.04.23 ===