Changes between Version 7 and Version 8 of PS1_IPP_Czarlog_20190408

Timestamp:

Apr 9, 2019, 5:14:15 PM (8 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190408

@@ -43 +43 @@
 === Tuesday : 2019.MM.DD ===
 
+''' Diffs for breakfast guvner ''' by JRF
+ Nightly processing went smoothly... for the most part
+ 
+ ippdb10 went down around 1am, no real explanation in the logs. Haydn restarted it, and noticed that there was a heat warning light on, seems fine now...
+
+''' Maintenance ''' by JRF
+
+The rack containing ipp131-133 is offline, and I said it could remain so since the machines do not have any significant drives for data storage (the same case is true for ipp140-142).
+
+Haydn continued with the replacement of ipp081. With Gavin's help they got it working again (the reason for the maintenance was a faulty SSD).
+
+ippc64.1, which contains the homedirs and is where the pantasks are run from, was having some issues. It looks like the issues resolved themselves eventually, but Haydn would like to replace. We have given him the go-ahead.
+
+'''Root partition is critical ippc72''' by JRF
+
+This is one of the apache nodes, so the logs will need to be archived.Check out the czar log from (SOME DATE). And see this page for how to clear them:
+
+
+''' Broken LAP.PV3 jobs''' by JRF
+
+There are a bunch of chips stuck in the 'full' state. They mention that a log file is missing:
+e.g. log file (LOG.IMFILE, 'neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/10/05/o5763g0099o.365804/o5763g0099o.365804.ch.1121712') not found
+
+Send them to clean:
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id XXXXX
+
+Once they have finished cleaning, set the imfile to update:
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXXXX
+
+All good!
+
+
+''' update.LAP.PV3 chips stuck in update ''' by JRF
+
+A number of jobs were failing saying they couldn't get specific files, I check on a bunch and the files are there and they seem normal (at least non-zero size). Just reverted them and they seemed to sovle themselves.
+
+It looks like the CPU wait times are quite high across the computer nodes. Some of the b-nodes appear to be getting hammered too.
+
 === Wednesday : 2019.MM.DD ===