Changes between Version 27 and Version 28 of PS1_IPP_Czarlog_20190510

Timestamp:

May 15, 2019, 12:13:57 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -876 +876 @@
 
  Czar meeting at 13:30:
+ '''Testing the ippx nodes'''
  We want to test the ippx nodes, now that they have been redistributed within their cabinet. Best way, set them as the only nodes to be used by stdscience for the ~2hours until the daily restart. Look at hosts groups in ippconfig/pantasks_hosts.input to find which host groups are loaded by the regular stdscience pantasks:
 {{{
@@ -999 +1000 @@
 }}}
 === Wednesday : 2019.05.15 ===
+ * TdB:  '''Testing the ippx nodes'''
+ Yesterday we had a test of the ippx nodes, after they were redistributed within their cabinet. Today, we want to test more. Haydn has set up access for us to the ACS48 serial console on itc-i06-con so we can keep track of the power draw to the PDUs. During the day and night yesterday, the x nodes have been fairly wel behaved. None of the machines has gone down or rebooted, judging by boottime on ganglia.
+
+ Once again, we remove all the regular nodes from stdscience processing, manually. We want to add groups x0,x1 and x2, but want to manually turn off ippx033-036 since those are plugged in to the PDU that was giving alarms yesterday. We will keep an eye on the power draw and see hwo it goes. 
+
+ First, stop the pantasks and remove the hosts:
+{{{
+hosts off si0 x4
+hosts off s4 x4
+hosts off s5 x4
+hosts off s6 x4
+hosts off s7 x4
+hosts off s8 x4
+hosts off ci0 x6
+hosts off ci1 x7
+
+hosts add x0 x10
+hosts add x1 x10
+hosts add x2 x10
+
+controller host off ippx033 x10
+controller host off ippx034 x10
+controller host off ippx035 x10
+controller host off ippx036 x10
+}}}
+ and set pantasks back to run.
+ 
+ -----------------------------
+
+ We want to keep an eye on the power draw and temperature as well. According to Haydn, each circuit in each PDU should have a maximum of 15A, and 12A is more reasonable. We will keep an eye on the power draw using the ACS48 serial console.
+
+ at 11:30 the power/temperature rates are:
+{{{
+ID: i06pdu2 (serial port 47)            Clear Max Current
+ Model: Cyclades PM20i/30A              Clear Max Power
+Reset Max Env Sensors
+ Number of Outlets: 20
+ Number of banks: 2              Single-Phase   
+ Software Version: 1.9.1 May 2, 2007    
+ PDU Current(Normal):  29.6 A   Max Detected: 32.1 A
+ PDU Power Consumption(estimated):  6156.8 W    Max Detected: 6676.8 W
+ PDU Voltage(estimated):  208 V 
+ PDU Power Factor(estimated):  1.00 
+ Bank(A): 
+     Current(Normal):  17.0 A   Max Detected: 17.9 A
+     Power Consumption(estimated):  3536.0 W    Max Detected: 3723.2 W
+     Voltage(estimated):  208 V 
+     Power Factor(estimated):  1.00 
+ Bank(B): 
+     Current(Normal):  12.5 A   Max Detected: 14.7 A
+     Power Consumption(estimated):  2600.0 W    Max Detected: 3057.6 W
+     Voltage(estimated):  208 V 
+     Power Factor(estimated):  1.00 
+ Temperature-Internal(T1):  48.9°C(119.9°F)     Max Detected: 51.9°C(125.3°F)
+
+ID: i06pdu1 (serial port 48)            Clear Max Current
+ Model: Cyclades PM20i/30A              Clear Max Power
+Reset Max Env Sensors
+ Number of Outlets: 20
+ Number of banks: 2              Single-Phase   
+ Software Version: 1.9.1 May 2, 2007    
+ PDU Current(Normal):  26.6 A   Max Detected: 28.4 A
+ PDU Power Consumption(estimated):  5532.8 W    Max Detected: 5907.2 W
+ PDU Voltage(estimated):  208 V 
+ PDU Power Factor(estimated):  1.00 
+ Bank(A): 
+     Current(Normal):  15.4 A   Max Detected: 16.1 A
+     Power Consumption(estimated):  3203.2 W    Max Detected: 3348.8 W
+     Voltage(estimated):  208 V 
+     Power Factor(estimated):  1.00 
+ Bank(B): 
+     Current(Normal):  11.2 A   Max Detected: 12.7 A
+     Power Consumption(estimated):  2329.6 W    Max Detected: 2641.6 W
+     Voltage(estimated):  208 V 
+     Power Factor(estimated):  1.00 
+ Temperature-Internal(T1):  50.9°C(123.5°F)     Max Detected: 53.9°C(128.9°F)
+}}}
+ While pdu1 is at the limit but still ok, pdu2 is drawing too much power. Let's try turning off the machines ippx029-032 as well, and see how that helps.
+{{{
+controller host off ippx029 x10
+controller host off ippx030 x10
+controller host off ippx031 x10
+controller host off ippx032 x10
+}}}
+ For reference, while waiting for jobs to clear before turning off nodes, the power draw was about 7-9.5A per circuit, which is a good baseline to have.
+ at 12:07:
+{{{
+ID: i06pdu2 (serial port 47)            Clear Max Current
+ Model: Cyclades PM20i/30A              Clear Max Power
+Reset Max Env Sensors
+ Number of Outlets: 20
+ Number of banks: 2              Single-Phase   
+ Software Version: 1.9.1 May 2, 2007    
+ PDU Current(Normal):  18.6 A   Max Detected: 32.1 A
+ PDU Power Consumption(estimated):  3868.8 W    Max Detected: 6676.8 W
+ PDU Voltage(estimated):  208 V 
+ PDU Power Factor(estimated):  1.00 
+ Bank(A): 
+     Current(Normal):  10.9 A   Max Detected: 17.9 A
+     Power Consumption(estimated):  2267.2 W    Max Detected: 3723.2 W
+     Voltage(estimated):  208 V 
+     Power Factor(estimated):  1.00 
+ Bank(B): 
+     Current(Normal):  7.6 A    Max Detected: 14.7 A
+     Power Consumption(estimated):  1580.8 W    Max Detected: 3057.6 W
+     Voltage(estimated):  208 V 
+     Power Factor(estimated):  1.00 
+ Temperature-Internal(T1):  49.9°C(121.7°F)     Max Detected: 51.9°C(125.3°F)
+
+ID: i06pdu1 (serial port 48)            Clear Max Current
+ Model: Cyclades PM20i/30A              Clear Max Power
+Reset Max Env Sensors
+ Number of Outlets: 20
+ Number of banks: 2              Single-Phase   
+ Software Version: 1.9.1 May 2, 2007    
+ PDU Current(Normal):  17.9 A   Max Detected: 28.4 A
+ PDU Power Consumption(estimated):  3723.2 W    Max Detected: 5907.2 W
+ PDU Voltage(estimated):  208 V 
+ PDU Power Factor(estimated):  1.00 
+ Bank(A): 
+     Current(Normal):  10.0 A   Max Detected: 16.1 A
+     Power Consumption(estimated):  2080.0 W    Max Detected: 3348.8 W
+     Voltage(estimated):  208 V 
+     Power Factor(estimated):  1.00 
+ Bank(B): 
+     Current(Normal):  7.9 A    Max Detected: 12.7 A
+     Power Consumption(estimated):  1643.2 W    Max Detected: 2641.6 W
+     Voltage(estimated):  208 V 
+     Power Factor(estimated):  1.00 
+ Temperature-Internal(T1):  50.9°C(123.5°F)     Max Detected: 53.9°C(128.9°F)
+}}}
+ This looks much bettter, and well within range
 
 === Thursday : 2019.05.16 ===