== PS1 IPP Czar Logs for the week 2019.05.24 - 2019.05.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])


=== Friday : 2019.05.24 ===

'''Czar Handover''' from JRF -> CCL

 '''Updates'''
Updates still need doing, so far the last label is: {{{LAP.PV3.20140730.20150%}}}.

 '''wait cleaning '''
See last weeks handover notes. I've mostly been cleaning ones with the data_group {{{ThreePi.201%}}}, I think I started at 2018.

 ''' general update cleaning '''
We've hit around 20k, so now would be a good time to clean up some chips. See last weeks handover notes

 ''' ippx node power draw'''
As mentioned today there are still some power draw issues, choices are:
 * monitor ippx node power usage while nightly processing is going on -> remove nodes from stdscience processing if the load is too high (the nodes appear to correspond to the PDU circuits descending on the page and descending on the ITC cabinet sheet -- I'll get Haydn to clarify).


 ''' PS1 pointing '''
Keep an eye on the pointing using the commands listed in last weeks czarlog (under misaligned visits heading).
If they look bad tell ps-obs and Ken, as they should be better than a week ago due to a new pointing model.

 ''' PS2 pointing '''
Watch out for very poor pointing, worth mentioning to people too and it may cause WSdiff issues in the morning (WSdiffs are currently only processed in the morning now).


 ''' Euclid Bright Twi reprocess '''
There are some Euclid observations that have not been processed as they have a comment string with a different format (the config has now been changed so they should process automatically). The Euclid observations under the new format, and before the config change, will need to be reprocessed. See last weeks czar log for the command, and make sure the date in the chiptool command is the date are of the observation (compare with the previous Euclid bright twi observations that have been done before the comment change).




'''ippx nodes power draw'''by JRF

Last week the xnodes were redistributed so that they could handle the heat better (a gap was left between each one). This helped the heat issue but we uncovered an issue with too much power being used by the machines.

Go to  the following to check the status of the PDUs in i06, the cabient with most of the xnodes:
**SEE EMAIL FROM 20190515 FOR URL**
Then : Applications -> IPDU Power Mgmt -> View IPDUs Info

An example readout
{{{
ID: i06pdu2 (serial port 47)   Clear Max Current
 Model: Cyclades PM20i/30A  Clear Max Power
Reset Max Env Sensors
 Number of Outlets: 20
 Number of banks: 2  Single-Phase 
 Software Version: 1.9.1 May 2, 2007
 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
 PDU Power Consumption(estimated):  5345.6 W    Max Detected: 6676.8 W
 PDU Voltage(estimated):  208 V 
 PDU Power Factor(estimated):  1.00 
 Bank(A): 
     Current(Normal):  15.9 A   Max Detected: 17.9 A
     Power Consumption(estimated):  3307.2 W    Max Detected: 3723.2 W
     Voltage(estimated):  208 V 
     Power Factor(estimated):  1.00 
 Bank(B): 
     Current(Normal):  9.6 A    Max Detected: 14.7 A
     Power Consumption(estimated):  1996.8 W    Max Detected: 3057.6 W
     Voltage(estimated):  208 V 
     Power Factor(estimated):  1.00 
 Temperature-Internal(T1):  50.9°C(123.5°F)     Max Detected: 51.9°C(125.3°F)
 ID: i06pdu1 (serial port 48)   Clear Max Current
 Model: Cyclades PM20i/30A  Clear Max Power
Reset Max Env Sensors
 Number of Outlets: 20
 Number of banks: 2  Single-Phase 
 Software Version: 1.9.1 May 2, 2007
 PDU Current(Normal):  25.7 A   Max Detected: 28.4 A
 PDU Power Consumption(estimated):  5345.6 W    Max Detected: 5907.2 W
 PDU Voltage(estimated):  208 V 
 PDU Power Factor(estimated):  1.00 
 Bank(A): 
     Current(Normal):  14.6 A   Max Detected: 16.1 A
     Power Consumption(estimated):  3036.8 W    Max Detected: 3348.8 W
     Voltage(estimated):  208 V 
     Power Factor(estimated):  1.00 
 Bank(B): 
     Current(Normal):  11.1 A   Max Detected: 12.7 A
     Power Consumption(estimated):  2308.8 W    Max Detected: 2641.6 W
     Voltage(estimated):  208 V 
     Power Factor(estimated):  1.00 
 Temperature-Internal(T1):  52.9°C(127.1°F)     Max Detected: 53.9°C(128.9°F)
}}}

We are only really interested in the power values:
{{{
 1st measure:
 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
     Current(Normal):  15.9 A   Max Detected: 17.9 A
     Current(Normal):  9.6 A    Max Detected: 14.7 A
 PDU Current(Normal):  25.7 A   Max Detected: 28.4 A
     Current(Normal):  14.6 A   Max Detected: 16.1 A
     Current(Normal):  11.1 A   Max Detected: 12.7 A

 2nd measure
 PDU Current(Normal):  26.2 A   Max Detected: 32.1 A
     Current(Normal):  16.2 A   Max Detected: 17.9 A
     Current(Normal):  9.8 A    Max Detected: 14.7 A
 PDU Current(Normal):  26.1 A   Max Detected: 28.4 A
     Current(Normal):  15.2 A   Max Detected: 16.1 A
     Current(Normal):  11.2 A   Max Detected: 12.7 A

 3rd measure
 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
     Current(Normal):  16.0 A   Max Detected: 17.9 A
     Current(Normal):  10.0 A   Max Detected: 14.7 A
 PDU Current(Normal):  25.8 A   Max Detected: 28.4 A
     Current(Normal):  15.0 A   Max Detected: 16.1 A
     Current(Normal):  10.8 A   Max Detected: 12.7 A
 
 4th measure
 PDU Current(Normal):  25.9 A   Max Detected: 32.1 A
     Current(Normal):  15.9 A   Max Detected: 17.9 A
     Current(Normal):  10.0 A   Max Detected: 14.7 A
 PDU Current(Normal):  25.3 A   Max Detected: 28.4 A
     Current(Normal):  14.5 A   Max Detected: 16.1 A
     Current(Normal):  10.8 A   Max Detected: 12.7 A
}}}
Note that the order is PDU2 total, PDU2 Bank A, PDU2 Bank B, PDU1 total, PDU1 Bank A, PDU1 Bank B


To turn off some nodes in processing:
{{{
pantasks> controller host off ippxXXX
}}}
You'll need to run it for as many times as the node is used (e.g. if it used 6 times by the controller,  then send the command 6 times)


Turned off ippx001-008 in pantasks, and straight away there is a drop:
{{{
 PDU Current(Normal):  22.1 A   Max Detected: 32.1 A
     Current(Normal):  12.2 A   Max Detected: 17.9 A
     Current(Normal):  9.5 A    Max Detected: 14.7 A
 PDU Current(Normal):  25.0 A   Max Detected: 28.4 A
     Current(Normal):  14.2 A   Max Detected: 16.1 A
     Current(Normal):  11.1 A   Max Detected: 12.7 A

 PDU Current(Normal):  21.3 A   Max Detected: 32.1 A
     Current(Normal):  11.7 A   Max Detected: 17.9 A
     Current(Normal):  9.7 A    Max Detected: 14.7 A
 PDU Current(Normal):  26.0 A   Max Detected: 28.4 A
     Current(Normal):  14.9 A   Max Detected: 16.1 A
     Current(Normal):  11.1 A   Max Detected: 12.7 A

}}}

 This is close to an acceptable level in pdu2, but still a little high. pdu1 on the otherhand looks relatively unchanged at around 14-15A.

 I'll try turning off a few at the bottom of the cabinet (I'm unsure of the setup).


Turned off ippx037-040:

{{{
 PDU Current(Normal):  21.5 A   Max Detected: 32.1 A
     Current(Normal):  12.0 A   Max Detected: 17.9 A
     Current(Normal):  9.5 A    Max Detected: 14.7 A
 PDU Current(Normal):  23.2 A   Max Detected: 28.4 A
     Current(Normal):  14.6 A   Max Detected: 16.1 A
     Current(Normal):  8.7 A    Max Detected: 12.7 A
}}}

 After waiting for those nodes to no longer have any jobs, there is still a large number

 Although, it looks like Bank B is now lower. And Haydn's email said:
 Instead of having three 4-node ippxNNN boxes per circuit, plus the 1G switch and serial console, we should have a maximum of two 4-node ippxNNN boxes per circuit, which should get it down to 10-12A per circuit.

Therefore, there should be 2 pdus * 2 circuits * 2 nodes per circuit = 8 available slots!
There are 11 sets in the cabinet...
I'm guessing the three bottom ones are connected to somewhere else? And that's why turning off ippx037-040 didn't really affect things.

Looking at the 6th and 7th ones down the list, they are ipp021-024 and ipp025-028 (as I think pdu2 is above pdu1 based on the ippx001-008 that I turned off).

So I'm turning of ippx021-024

And the results are (a few mins apart):
{{{
 PDU Current(Normal):  21.6 A   Max Detected: 32.1 A
     Current(Normal):  12.0 A   Max Detected: 17.9 A
     Current(Normal):  9.5 A    Max Detected: 14.7 A
 PDU Current(Normal):  21.2 A   Max Detected: 28.4 A
     Current(Normal):  12.9 A   Max Detected: 16.1 A
     Current(Normal):  8.4 A    Max Detected: 12.7 A

 PDU Current(Normal):  21.6 A   Max Detected: 32.1 A
     Current(Normal):  12.0 A   Max Detected: 17.9 A
     Current(Normal):  10.0 A   Max Detected: 14.7 A
 PDU Current(Normal):  21.3 A   Max Detected: 28.4 A
     Current(Normal):  12.9 A   Max Detected: 16.1 A
     Current(Normal):  8.4 A    Max Detected: 12.7 A
}}}


 So success! Maybe just a little high still, but much more manageable.

Which pdu supplies which node should definitely be documented, the ITC cabinet sheet sounds like a logical place for it.



=== Saturday : 2019.05.25 ===

=== Sunday : 2019.05.26 ===

=== Monday : 2019.05.27 ===
  
=== Tuesday : 2019.05.28 ===

=== Wednesday : 2019.05.29 ===

=== Thursday : 2019.05.30 ===