Changes between Version 24 and Version 25 of PS1_IPP_Czarlog_20190204

Timestamp:

Feb 11, 2019, 1:39:18 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190204

@@ -328 +328 @@
 === Saturday : 2019.02.09 ===
 
+JRF: Have been checking through the list of cluster nodes using neb-host to find nodes that have been left in repair. So that they can be brought back into production if they are ok 
+
+There are config files located in the modules folder that dictate which nodes to save results do. e.g. processing of files relating to GPC1 XY01 get saved to ipp067.0. The current config is found here:
+{{{
+/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/share/pantasks/modules
+}}}
+Best practice is to make a backup in the folder. MEH handled the changes to this file.
+
+To use the new config need to stop and shutdown stdscience pantasks. Make sure that there are no jobs still running after stopping before you shutdown.
+
+Then to restart with the new config go to the machine that was running pantasks and run:
+{{{
+./start_server.sh stdscience
+}}}
+This will start pantasks back up, AND it archives the log (which does not happen if you start it normally).
+
+With pantasks running we need to actually get the scheduler and controller going by using the following commands:
+{{{
+server input input
+setup
+run
+}}}
+
+
+JRF: Going back to the ippmonitor issues:
+There were a lot of LAP chip errors, which look like an improper state. These just need to have the chip updated e.g.
+{{{
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id XXXXXX
+}}}
+
+
 === Sunday : 2019.02.10 === 
+
  * MEH: power at ATRC down, Haydn powered down ippb06-b23 -- set neb-host down for all machines so any primary copies there will switch to secondary on the system 
-  * Gene has ippb04,b05 in repair to rsync and retire -- there are now no backup/archive nodes for replication -- NO processing can proceed -- this is not an acceptable situation as discussed before and a backup node set and plan is needed for future ATRC down event 
-=== Sunday : 2019.02.10 ===
+  * Gene has ippb04,b05 in repair to rsync and retire -- there are now no backup/archive nodes for replication -- NO processing can proceed -- this is not an acceptable situation as discussed before and a backup node set and plan is needed for future ATRC down event.