IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of PS1_IPP_Czarlog_20190524


Ignore:
Timestamp:
May 24, 2019, 11:15:24 AM (7 years ago)
Author:
fairlamb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • PS1_IPP_Czarlog_20190524

    v1 v1  
     1
     2== PS1 IPP Czar Logs for the week 2019.05.24 - 2019.05.DD ==
     3[[PageOutline]]
     4
     5(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])
     6
     7
     8=== Friday : 2019.05.24 ===
     9
     10
     11     '''ippx nodes power draw'''by JRF
     12
     13Last week the xnodes were redistributed so that they could handle the heat better (a gap was left between each one). This helped the heat issue but we uncovered an issue with too much power being used by the machines.
     14
     15Go to  the following to check the status of the PDUs in i06, the cabient with most of the xnodes:
     16https://itc-i06-con.ifa.hawaii.edu/
     17Then : Applications -> IPDU Power Mgmt -> View IPDUs Info
     18
     19An example readout
     20{{{
     21ID: i06pdu2 (serial port 47)   Clear Max Current
     22 Model: Cyclades PM20i/30A  Clear Max Power
     23Reset Max Env Sensors
     24 Number of Outlets: 20
     25 Number of banks: 2  Single-Phase
     26 Software Version: 1.9.1 May 2, 2007
     27 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
     28 PDU Power Consumption(estimated):  5345.6 W    Max Detected: 6676.8 W
     29 PDU Voltage(estimated):  208 V
     30 PDU Power Factor(estimated):  1.00
     31 Bank(A):
     32     Current(Normal):  15.9 A   Max Detected: 17.9 A
     33     Power Consumption(estimated):  3307.2 W    Max Detected: 3723.2 W
     34     Voltage(estimated):  208 V
     35     Power Factor(estimated):  1.00
     36 Bank(B):
     37     Current(Normal):  9.6 A    Max Detected: 14.7 A
     38     Power Consumption(estimated):  1996.8 W    Max Detected: 3057.6 W
     39     Voltage(estimated):  208 V
     40     Power Factor(estimated):  1.00
     41 Temperature-Internal(T1):  50.9°C(123.5°F)     Max Detected: 51.9°C(125.3°F)
     42 ID: i06pdu1 (serial port 48)   Clear Max Current
     43 Model: Cyclades PM20i/30A  Clear Max Power
     44Reset Max Env Sensors
     45 Number of Outlets: 20
     46 Number of banks: 2  Single-Phase
     47 Software Version: 1.9.1 May 2, 2007
     48 PDU Current(Normal):  25.7 A   Max Detected: 28.4 A
     49 PDU Power Consumption(estimated):  5345.6 W    Max Detected: 5907.2 W
     50 PDU Voltage(estimated):  208 V
     51 PDU Power Factor(estimated):  1.00
     52 Bank(A):
     53     Current(Normal):  14.6 A   Max Detected: 16.1 A
     54     Power Consumption(estimated):  3036.8 W    Max Detected: 3348.8 W
     55     Voltage(estimated):  208 V
     56     Power Factor(estimated):  1.00
     57 Bank(B):
     58     Current(Normal):  11.1 A   Max Detected: 12.7 A
     59     Power Consumption(estimated):  2308.8 W    Max Detected: 2641.6 W
     60     Voltage(estimated):  208 V
     61     Power Factor(estimated):  1.00
     62 Temperature-Internal(T1):  52.9°C(127.1°F)     Max Detected: 53.9°C(128.9°F)
     63}}}
     64
     65We are only really interested in the power values:
     66{{{
     67 1st measure:
     68 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
     69     Current(Normal):  15.9 A   Max Detected: 17.9 A
     70     Current(Normal):  9.6 A    Max Detected: 14.7 A
     71 PDU Current(Normal):  25.7 A   Max Detected: 28.4 A
     72     Current(Normal):  14.6 A   Max Detected: 16.1 A
     73     Current(Normal):  11.1 A   Max Detected: 12.7 A
     74
     75 2nd measure
     76 PDU Current(Normal):  26.2 A   Max Detected: 32.1 A
     77     Current(Normal):  16.2 A   Max Detected: 17.9 A
     78     Current(Normal):  9.8 A    Max Detected: 14.7 A
     79 PDU Current(Normal):  26.1 A   Max Detected: 28.4 A
     80     Current(Normal):  15.2 A   Max Detected: 16.1 A
     81     Current(Normal):  11.2 A   Max Detected: 12.7 A
     82
     83 3rd measure
     84 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
     85     Current(Normal):  16.0 A   Max Detected: 17.9 A
     86     Current(Normal):  10.0 A   Max Detected: 14.7 A
     87 PDU Current(Normal):  25.8 A   Max Detected: 28.4 A
     88     Current(Normal):  15.0 A   Max Detected: 16.1 A
     89     Current(Normal):  10.8 A   Max Detected: 12.7 A
     90 
     91 4th measure
     92 PDU Current(Normal):  25.9 A   Max Detected: 32.1 A
     93     Current(Normal):  15.9 A   Max Detected: 17.9 A
     94     Current(Normal):  10.0 A   Max Detected: 14.7 A
     95 PDU Current(Normal):  25.3 A   Max Detected: 28.4 A
     96     Current(Normal):  14.5 A   Max Detected: 16.1 A
     97     Current(Normal):  10.8 A   Max Detected: 12.7 A
     98}}}
     99Note that the order is PDU2 total, PDU2 Bank A, PDU2 Bank B, PDU1 total, PDU1 Bank A, PDU1 Bank B
     100
     101
     102To turn off some nodes in processing:
     103{{{
     104pantasks> controller host off ippxXXX
     105}}}
     106You'll need to run it for as many times as the node is used (e.g. if it used 6 times by the controller,  then send the command 6 times)
     107
     108
     109Turned off ippx001-008 in pantasks, and straight away there is a drop:
     110{{{
     111 PDU Current(Normal):  22.1 A   Max Detected: 32.1 A
     112     Current(Normal):  12.2 A   Max Detected: 17.9 A
     113     Current(Normal):  9.5 A    Max Detected: 14.7 A
     114 PDU Current(Normal):  25.0 A   Max Detected: 28.4 A
     115     Current(Normal):  14.2 A   Max Detected: 16.1 A
     116     Current(Normal):  11.1 A   Max Detected: 12.7 A
     117
     118 PDU Current(Normal):  21.3 A   Max Detected: 32.1 A
     119     Current(Normal):  11.7 A   Max Detected: 17.9 A
     120     Current(Normal):  9.7 A    Max Detected: 14.7 A
     121 PDU Current(Normal):  26.0 A   Max Detected: 28.4 A
     122     Current(Normal):  14.9 A   Max Detected: 16.1 A
     123     Current(Normal):  11.1 A   Max Detected: 12.7 A
     124
     125}}}
     126
     127 This is close to an acceptable level in pdu2, but still a little high. pdu1 on the otherhand looks relatively unchanged at around 14-15A.
     128
     129 I'll try turning off a few at the bottom of the cabinet (I'm unsure of the setup).
     130
     131
     132Turned off ippx037-040:
     133
     134{{{
     135 PDU Current(Normal):  21.5 A   Max Detected: 32.1 A
     136     Current(Normal):  12.0 A   Max Detected: 17.9 A
     137     Current(Normal):  9.5 A    Max Detected: 14.7 A
     138 PDU Current(Normal):  23.2 A   Max Detected: 28.4 A
     139     Current(Normal):  14.6 A   Max Detected: 16.1 A
     140     Current(Normal):  8.7 A    Max Detected: 12.7 A
     141}}}
     142
     143 After waiting for those nodes to no longer have any jobs, there is still a large number
     144
     145 Although, it looks like Bank B is now lower. And Haydn's email said:
     146 Instead of having three 4-node ippxNNN boxes per circuit, plus the 1G switch and serial console, we should have a maximum of two 4-node ippxNNN boxes per circuit, which should get it down to 10-12A per circuit.
     147
     148Therefore, there should be 2 pdus * 2 circuits * 2 nodes per circuit = 8 available slots!
     149There are 11 sets in the cabinet...
     150I'm guessing the three bottom ones are connected to somewhere else? And that's why turning off ippx037-040 didn't really affect things.
     151
     152Looking at the 6th and 7th ones down the list, they are ipp021-024 and ipp025-028 (as I think pdu2 is above pdu1 based on the ippx001-008 that I turned off).
     153
     154So I'm turning of ippx021-024
     155
     156And the results are (a few mins apart):
     157{{{
     158 PDU Current(Normal):  21.6 A   Max Detected: 32.1 A
     159     Current(Normal):  12.0 A   Max Detected: 17.9 A
     160     Current(Normal):  9.5 A    Max Detected: 14.7 A
     161 PDU Current(Normal):  21.2 A   Max Detected: 28.4 A
     162     Current(Normal):  12.9 A   Max Detected: 16.1 A
     163     Current(Normal):  8.4 A    Max Detected: 12.7 A
     164
     165 PDU Current(Normal):  21.6 A   Max Detected: 32.1 A
     166     Current(Normal):  12.0 A   Max Detected: 17.9 A
     167     Current(Normal):  10.0 A   Max Detected: 14.7 A
     168 PDU Current(Normal):  21.3 A   Max Detected: 28.4 A
     169     Current(Normal):  12.9 A   Max Detected: 16.1 A
     170     Current(Normal):  8.4 A    Max Detected: 12.7 A
     171}}}
     172
     173
     174 So success! Maybe just a little high still, but much more manageable.
     175
     176Which pdu supplies which node should definitely be documented, the ITC cabinet sheet sounds like a logical place for it.
     177
     178
     179'''Czar Handover''' from JRF -> CCL
     180
     181'''Updates'''
     182
     183'''wait cleaning '''
     184
     185''' general cleaning '''
     186
     187''' ippx node power draw'''
     188
     189''' PS1 pointing '''
     190
     191''' PS2 pointing '''
     192
     193
     194=== Saturday : 2019.05.25 ===
     195
     196=== Sunday : 2019.05.26 ===
     197
     198=== Monday : 2019.05.27 ===
     199 
     200=== Tuesday : 2019.05.28 ===
     201
     202=== Wednesday : 2019.05.29 ===
     203
     204=== Thursday : 2019.05.30 ===