| Version 9 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.05.24 - 2019.05.DD
(Up to PS1 IPP Czar Logs)
Friday : 2019.05.24
Czar Handover from JRF -> CCL
Updates
Updates still need doing, so far the last label is: LAP.PV3.20140730.20150%.
wait cleaning
See last weeks handover notes. I've mostly been cleaning ones with the data_group ThreePi.201%, I think I started at 2018.
general update cleaning We've hit around 20k, so now would be a good time to clean up some chips. See last weeks handover notes
ippx node power draw As mentioned today there are still some power draw issues, choices are:
- monitor ippx node power usage while nightly processing is going on -> remove nodes from stdscience processing if the load is too high (the nodes appear to correspond to the PDU circuits descending on the page and descending on the ITC cabinet sheet -- I'll get Haydn to clarify).
PS1 pointing Keep an eye on the pointing using the commands listed in last weeks czarlog (under misaligned visits heading). If they look bad tell ps-obs and Ken, as they should be better than a week ago due to a new pointing model.
PS2 pointing Watch out for very poor pointing, worth mentioning to people too and it may cause WSdiff issues in the morning (WSdiffs are currently only processed in the morning now).
Euclid Bright Twi reprocess There are some Euclid observations that have not been processed as they have a comment string with a different format (the config has now been changed so they should process automatically). The Euclid observations under the new format, and before the config change, will need to be reprocessed. See last weeks czar log for the command, and make sure the date in the chiptool command is the date are of the observation (compare with the previous Euclid bright twi observations that have been done before the comment change).
ippx nodes power draw by JRF
Last week the xnodes were redistributed so that they could handle the heat better (a gap was left between each one). This helped the heat issue but we uncovered an issue with too much power being used by the machines.
Go to the following to check the status of the PDUs in i06, the cabient with most of the xnodes: SEE EMAIL FROM 20190515 FOR URL Then : Applications -> IPDU Power Mgmt -> View IPDUs Info
An example readout
ID: i06pdu2 (serial port 47) Clear Max Current
Model: Cyclades PM20i/30A Clear Max Power
Reset Max Env Sensors
Number of Outlets: 20
Number of banks: 2 Single-Phase
Software Version: 1.9.1 May 2, 2007
PDU Current(Normal): 25.7 A Max Detected: 32.1 A
PDU Power Consumption(estimated): 5345.6 W Max Detected: 6676.8 W
PDU Voltage(estimated): 208 V
PDU Power Factor(estimated): 1.00
Bank(A):
Current(Normal): 15.9 A Max Detected: 17.9 A
Power Consumption(estimated): 3307.2 W Max Detected: 3723.2 W
Voltage(estimated): 208 V
Power Factor(estimated): 1.00
Bank(B):
Current(Normal): 9.6 A Max Detected: 14.7 A
Power Consumption(estimated): 1996.8 W Max Detected: 3057.6 W
Voltage(estimated): 208 V
Power Factor(estimated): 1.00
Temperature-Internal(T1): 50.9°C(123.5°F) Max Detected: 51.9°C(125.3°F)
ID: i06pdu1 (serial port 48) Clear Max Current
Model: Cyclades PM20i/30A Clear Max Power
Reset Max Env Sensors
Number of Outlets: 20
Number of banks: 2 Single-Phase
Software Version: 1.9.1 May 2, 2007
PDU Current(Normal): 25.7 A Max Detected: 28.4 A
PDU Power Consumption(estimated): 5345.6 W Max Detected: 5907.2 W
PDU Voltage(estimated): 208 V
PDU Power Factor(estimated): 1.00
Bank(A):
Current(Normal): 14.6 A Max Detected: 16.1 A
Power Consumption(estimated): 3036.8 W Max Detected: 3348.8 W
Voltage(estimated): 208 V
Power Factor(estimated): 1.00
Bank(B):
Current(Normal): 11.1 A Max Detected: 12.7 A
Power Consumption(estimated): 2308.8 W Max Detected: 2641.6 W
Voltage(estimated): 208 V
Power Factor(estimated): 1.00
Temperature-Internal(T1): 52.9°C(127.1°F) Max Detected: 53.9°C(128.9°F)
We are only really interested in the power values:
1st measure:
PDU Current(Normal): 25.7 A Max Detected: 32.1 A
Current(Normal): 15.9 A Max Detected: 17.9 A
Current(Normal): 9.6 A Max Detected: 14.7 A
PDU Current(Normal): 25.7 A Max Detected: 28.4 A
Current(Normal): 14.6 A Max Detected: 16.1 A
Current(Normal): 11.1 A Max Detected: 12.7 A
2nd measure
PDU Current(Normal): 26.2 A Max Detected: 32.1 A
Current(Normal): 16.2 A Max Detected: 17.9 A
Current(Normal): 9.8 A Max Detected: 14.7 A
PDU Current(Normal): 26.1 A Max Detected: 28.4 A
Current(Normal): 15.2 A Max Detected: 16.1 A
Current(Normal): 11.2 A Max Detected: 12.7 A
3rd measure
PDU Current(Normal): 25.7 A Max Detected: 32.1 A
Current(Normal): 16.0 A Max Detected: 17.9 A
Current(Normal): 10.0 A Max Detected: 14.7 A
PDU Current(Normal): 25.8 A Max Detected: 28.4 A
Current(Normal): 15.0 A Max Detected: 16.1 A
Current(Normal): 10.8 A Max Detected: 12.7 A
4th measure
PDU Current(Normal): 25.9 A Max Detected: 32.1 A
Current(Normal): 15.9 A Max Detected: 17.9 A
Current(Normal): 10.0 A Max Detected: 14.7 A
PDU Current(Normal): 25.3 A Max Detected: 28.4 A
Current(Normal): 14.5 A Max Detected: 16.1 A
Current(Normal): 10.8 A Max Detected: 12.7 A
Note that the order is PDU2 total, PDU2 Bank A, PDU2 Bank B, PDU1 total, PDU1 Bank A, PDU1 Bank B
To turn off some nodes in processing:
pantasks> controller host off ippxXXX
You'll need to run it for as many times as the node is used (e.g. if it used 6 times by the controller, then send the command 6 times)
Turned off ippx001-008 in pantasks, and straight away there is a drop:
PDU Current(Normal): 22.1 A Max Detected: 32.1 A
Current(Normal): 12.2 A Max Detected: 17.9 A
Current(Normal): 9.5 A Max Detected: 14.7 A
PDU Current(Normal): 25.0 A Max Detected: 28.4 A
Current(Normal): 14.2 A Max Detected: 16.1 A
Current(Normal): 11.1 A Max Detected: 12.7 A
PDU Current(Normal): 21.3 A Max Detected: 32.1 A
Current(Normal): 11.7 A Max Detected: 17.9 A
Current(Normal): 9.7 A Max Detected: 14.7 A
PDU Current(Normal): 26.0 A Max Detected: 28.4 A
Current(Normal): 14.9 A Max Detected: 16.1 A
Current(Normal): 11.1 A Max Detected: 12.7 A
This is close to an acceptable level in pdu2, but still a little high. pdu1 on the otherhand looks relatively unchanged at around 14-15A.
I'll try turning off a few at the bottom of the cabinet (I'm unsure of the setup).
Turned off ippx037-040:
PDU Current(Normal): 21.5 A Max Detected: 32.1 A
Current(Normal): 12.0 A Max Detected: 17.9 A
Current(Normal): 9.5 A Max Detected: 14.7 A
PDU Current(Normal): 23.2 A Max Detected: 28.4 A
Current(Normal): 14.6 A Max Detected: 16.1 A
Current(Normal): 8.7 A Max Detected: 12.7 A
After waiting for those nodes to no longer have any jobs, there is still a large number
Although, it looks like Bank B is now lower. And Haydn's email said: Instead of having three 4-node ippxNNN boxes per circuit, plus the 1G switch and serial console, we should have a maximum of two 4-node ippxNNN boxes per circuit, which should get it down to 10-12A per circuit.
Therefore, there should be 2 pdus * 2 circuits * 2 nodes per circuit = 8 available slots! There are 11 sets in the cabinet... I'm guessing the three bottom ones are connected to somewhere else? And that's why turning off ippx037-040 didn't really affect things.
Looking at the 6th and 7th ones down the list, they are ipp021-024 and ipp025-028 (as I think pdu2 is above pdu1 based on the ippx001-008 that I turned off).
So I'm turning of ippx021-024
And the results are (a few mins apart):
PDU Current(Normal): 21.6 A Max Detected: 32.1 A
Current(Normal): 12.0 A Max Detected: 17.9 A
Current(Normal): 9.5 A Max Detected: 14.7 A
PDU Current(Normal): 21.2 A Max Detected: 28.4 A
Current(Normal): 12.9 A Max Detected: 16.1 A
Current(Normal): 8.4 A Max Detected: 12.7 A
PDU Current(Normal): 21.6 A Max Detected: 32.1 A
Current(Normal): 12.0 A Max Detected: 17.9 A
Current(Normal): 10.0 A Max Detected: 14.7 A
PDU Current(Normal): 21.3 A Max Detected: 28.4 A
Current(Normal): 12.9 A Max Detected: 16.1 A
Current(Normal): 8.4 A Max Detected: 12.7 A
So success! Maybe just a little high still, but much more manageable.
Which pdu supplies which node should definitely be documented, the ITC cabinet sheet sounds like a logical place for it.
Saturday : 2019.05.25
- MEH: PS1:OTA03 not behaving well still -- adding script John used the other night to the crontab on ippc30
*/20 * * * * ~/sshkill_ppimage_dtime_host_ps1.csh
- CCL: fix some warp problems with missing *.mk.fits (update.LAP.PV3)
Exp Name Exp ID Chip ID Cam ID Fake ID Warp ID Skycell ID State Label Fault Revert Command o5174g0278o 107982 1444296 1413039 1386519 1391801 skycell.1945.070 update update.LAP.PV3 2 warp.revert.on and warp.revert.off ... 550 Couldn't find input file: neb://any/gpc1/LAP.PV3.20140730.bigmem/2015/01/25/o5174g0278o.107982/o5174g0278o.107982.cm.1413039.XY56.mk.fits at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 563. Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warptool -updateskyfile -warp_id 1391801 -skycell_id skycell.1945.070 -fault 2 -dbname gpc1]... find their chip id, clean and update them. 1444294 1444295 1444296 1444297 1444300 1444303 1444305 1444306 1444307 1444309 1452266 1452267 1452313 1461544 1461545 1462366 1462368 1462370 1462371 1462372 1462373 1462374 1462375 1462377 1462378 1462379 1462380 1462381 1462565 1462567 1462603 1462605 1462606 1462607 1462609 1462615 1462619 1462755 1462790 1462794 1462806 1462807 1462810 1462811 1462813 1463125
Sunday : 2019.05.26
Monday : 2019.05.27
