| Version 19 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.05.24 - 2019.05.DD
(Up to PS1 IPP Czar Logs)
Friday : 2019.05.24
Czar Handover from JRF -> CCL
Updates
Updates still need doing, so far the last label is: LAP.PV3.20140730.20150%.
wait cleaning
See last weeks handover notes. I've mostly been cleaning ones with the data_group ThreePi.201%, I think I started at 2018.
general update cleaning We've hit around 20k, so now would be a good time to clean up some chips. See last weeks handover notes
ippx node power draw As mentioned today there are still some power draw issues, choices are:
- monitor ippx node power usage while nightly processing is going on -> remove nodes from stdscience processing if the load is too high (the nodes appear to correspond to the PDU circuits descending on the page and descending on the ITC cabinet sheet -- I'll get Haydn to clarify).
PS1 pointing Keep an eye on the pointing using the commands listed in last weeks czarlog (under misaligned visits heading). If they look bad tell ps-obs and Ken, as they should be better than a week ago due to a new pointing model.
PS2 pointing Watch out for very poor pointing, worth mentioning to people too and it may cause WSdiff issues in the morning (WSdiffs are currently only processed in the morning now).
Euclid Bright Twi reprocess There are some Euclid observations that have not been processed as they have a comment string with a different format (the config has now been changed so they should process automatically). The Euclid observations under the new format, and before the config change, will need to be reprocessed. See last weeks czar log for the command, and make sure the date in the chiptool command is the date are of the observation (compare with the previous Euclid bright twi observations that have been done before the comment change).
ippx nodes power draw by JRF
Last week the xnodes were redistributed so that they could handle the heat better (a gap was left between each one). This helped the heat issue but we uncovered an issue with too much power being used by the machines.
Go to the following to check the status of the PDUs in i06, the cabient with most of the xnodes: SEE EMAIL FROM 20190515 FOR URL Then : Applications -> IPDU Power Mgmt -> View IPDUs Info
An example readout
ID: i06pdu2 (serial port 47) Clear Max Current
Model: Cyclades PM20i/30A Clear Max Power
Reset Max Env Sensors
Number of Outlets: 20
Number of banks: 2 Single-Phase
Software Version: 1.9.1 May 2, 2007
PDU Current(Normal): 25.7 A Max Detected: 32.1 A
PDU Power Consumption(estimated): 5345.6 W Max Detected: 6676.8 W
PDU Voltage(estimated): 208 V
PDU Power Factor(estimated): 1.00
Bank(A):
Current(Normal): 15.9 A Max Detected: 17.9 A
Power Consumption(estimated): 3307.2 W Max Detected: 3723.2 W
Voltage(estimated): 208 V
Power Factor(estimated): 1.00
Bank(B):
Current(Normal): 9.6 A Max Detected: 14.7 A
Power Consumption(estimated): 1996.8 W Max Detected: 3057.6 W
Voltage(estimated): 208 V
Power Factor(estimated): 1.00
Temperature-Internal(T1): 50.9°C(123.5°F) Max Detected: 51.9°C(125.3°F)
ID: i06pdu1 (serial port 48) Clear Max Current
Model: Cyclades PM20i/30A Clear Max Power
Reset Max Env Sensors
Number of Outlets: 20
Number of banks: 2 Single-Phase
Software Version: 1.9.1 May 2, 2007
PDU Current(Normal): 25.7 A Max Detected: 28.4 A
PDU Power Consumption(estimated): 5345.6 W Max Detected: 5907.2 W
PDU Voltage(estimated): 208 V
PDU Power Factor(estimated): 1.00
Bank(A):
Current(Normal): 14.6 A Max Detected: 16.1 A
Power Consumption(estimated): 3036.8 W Max Detected: 3348.8 W
Voltage(estimated): 208 V
Power Factor(estimated): 1.00
Bank(B):
Current(Normal): 11.1 A Max Detected: 12.7 A
Power Consumption(estimated): 2308.8 W Max Detected: 2641.6 W
Voltage(estimated): 208 V
Power Factor(estimated): 1.00
Temperature-Internal(T1): 52.9°C(127.1°F) Max Detected: 53.9°C(128.9°F)
We are only really interested in the power values:
1st measure:
PDU Current(Normal): 25.7 A Max Detected: 32.1 A
Current(Normal): 15.9 A Max Detected: 17.9 A
Current(Normal): 9.6 A Max Detected: 14.7 A
PDU Current(Normal): 25.7 A Max Detected: 28.4 A
Current(Normal): 14.6 A Max Detected: 16.1 A
Current(Normal): 11.1 A Max Detected: 12.7 A
2nd measure
PDU Current(Normal): 26.2 A Max Detected: 32.1 A
Current(Normal): 16.2 A Max Detected: 17.9 A
Current(Normal): 9.8 A Max Detected: 14.7 A
PDU Current(Normal): 26.1 A Max Detected: 28.4 A
Current(Normal): 15.2 A Max Detected: 16.1 A
Current(Normal): 11.2 A Max Detected: 12.7 A
3rd measure
PDU Current(Normal): 25.7 A Max Detected: 32.1 A
Current(Normal): 16.0 A Max Detected: 17.9 A
Current(Normal): 10.0 A Max Detected: 14.7 A
PDU Current(Normal): 25.8 A Max Detected: 28.4 A
Current(Normal): 15.0 A Max Detected: 16.1 A
Current(Normal): 10.8 A Max Detected: 12.7 A
4th measure
PDU Current(Normal): 25.9 A Max Detected: 32.1 A
Current(Normal): 15.9 A Max Detected: 17.9 A
Current(Normal): 10.0 A Max Detected: 14.7 A
PDU Current(Normal): 25.3 A Max Detected: 28.4 A
Current(Normal): 14.5 A Max Detected: 16.1 A
Current(Normal): 10.8 A Max Detected: 12.7 A
Note that the order is PDU2 total, PDU2 Bank A, PDU2 Bank B, PDU1 total, PDU1 Bank A, PDU1 Bank B
To turn off some nodes in processing:
pantasks> controller host off ippxXXX
You'll need to run it for as many times as the node is used (e.g. if it used 6 times by the controller, then send the command 6 times)
Turned off ippx001-008 in pantasks, and straight away there is a drop:
PDU Current(Normal): 22.1 A Max Detected: 32.1 A
Current(Normal): 12.2 A Max Detected: 17.9 A
Current(Normal): 9.5 A Max Detected: 14.7 A
PDU Current(Normal): 25.0 A Max Detected: 28.4 A
Current(Normal): 14.2 A Max Detected: 16.1 A
Current(Normal): 11.1 A Max Detected: 12.7 A
PDU Current(Normal): 21.3 A Max Detected: 32.1 A
Current(Normal): 11.7 A Max Detected: 17.9 A
Current(Normal): 9.7 A Max Detected: 14.7 A
PDU Current(Normal): 26.0 A Max Detected: 28.4 A
Current(Normal): 14.9 A Max Detected: 16.1 A
Current(Normal): 11.1 A Max Detected: 12.7 A
This is close to an acceptable level in pdu2, but still a little high. pdu1 on the otherhand looks relatively unchanged at around 14-15A.
I'll try turning off a few at the bottom of the cabinet (I'm unsure of the setup).
Turned off ippx037-040:
PDU Current(Normal): 21.5 A Max Detected: 32.1 A
Current(Normal): 12.0 A Max Detected: 17.9 A
Current(Normal): 9.5 A Max Detected: 14.7 A
PDU Current(Normal): 23.2 A Max Detected: 28.4 A
Current(Normal): 14.6 A Max Detected: 16.1 A
Current(Normal): 8.7 A Max Detected: 12.7 A
After waiting for those nodes to no longer have any jobs, there is still a large number
Although, it looks like Bank B is now lower. And Haydn's email said: Instead of having three 4-node ippxNNN boxes per circuit, plus the 1G switch and serial console, we should have a maximum of two 4-node ippxNNN boxes per circuit, which should get it down to 10-12A per circuit.
Therefore, there should be 2 pdus * 2 circuits * 2 nodes per circuit = 8 available slots! There are 11 sets in the cabinet... I'm guessing the three bottom ones are connected to somewhere else? And that's why turning off ippx037-040 didn't really affect things.
Looking at the 6th and 7th ones down the list, they are ipp021-024 and ipp025-028 (as I think pdu2 is above pdu1 based on the ippx001-008 that I turned off).
So I'm turning of ippx021-024
And the results are (a few mins apart):
PDU Current(Normal): 21.6 A Max Detected: 32.1 A
Current(Normal): 12.0 A Max Detected: 17.9 A
Current(Normal): 9.5 A Max Detected: 14.7 A
PDU Current(Normal): 21.2 A Max Detected: 28.4 A
Current(Normal): 12.9 A Max Detected: 16.1 A
Current(Normal): 8.4 A Max Detected: 12.7 A
PDU Current(Normal): 21.6 A Max Detected: 32.1 A
Current(Normal): 12.0 A Max Detected: 17.9 A
Current(Normal): 10.0 A Max Detected: 14.7 A
PDU Current(Normal): 21.3 A Max Detected: 28.4 A
Current(Normal): 12.9 A Max Detected: 16.1 A
Current(Normal): 8.4 A Max Detected: 12.7 A
So success! Maybe just a little high still, but much more manageable.
Which pdu supplies which node should definitely be documented, the ITC cabinet sheet sounds like a logical place for it.
Saturday : 2019.05.25
- MEH: PS1:OTA03 not behaving well still -- adding script John used the other night to the crontab on ippc30
*/20 * * * * ~/sshkill_ppimage_dtime_host_ps1.csh
- CCL: fix some warp problems with missing *.mk.fits (update.LAP.PV3)
Exp Name Exp ID Chip ID Cam ID Fake ID Warp ID Skycell ID State Label Fault Revert Command o5174g0278o 107982 1444296 1413039 1386519 1391801 skycell.1945.070 update update.LAP.PV3 2 warp.revert.on and warp.revert.off ... warp id 1391853 1391817 1391801 1392056 1392017 1391968 1392075 1392217 1391810 1392074 1399947 1399948 1399053 1411587 1411937 1415176 1415201 1415242 1415172 1415228 1415184 1415241 1415197 1415213 1415185 1415179 1415183 1415189 1415665 1415645 1415781 1415759 1415782 1415767 1429271 1415829 1415786 1429210 1416312 1416240 1416301 1416388 1416296 1416269 1416401 1416521 1350397 1350403 1376934 Couldn't find input file: neb://any/gpc1/LAP.PV3.20140730.bigmem/2015/01/25/o5174g0278o.107982/o5174g0278o.107982.cm.1413039.XY56.mk.fits at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 563. Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warptool -updateskyfile -warp_id 1391801 -skycell_id skycell.1945.070 -fault 2 -dbname gpc1]... find their chip id, clean and update them. 1444294 1444295 1444296 1444297 1444300 1444303 1444305 1444306 1444307 1444309 1452266 1452267 1452313 1461544 1461545 1462366 1462368 1462370 1462371 1462372 1462373 1462374 1462375 1462377 1462378 1462379 1462380 1462381 1462565 1462567 1462603 1462605 1462606 1462607 1462609 1462615 1462619 1462755 1462790 1462794 1462806 1462807 1462810 1462811 1462813 1463125 1393807 1393804 1428048 then revert warptool -revertwarped -fault 2 -label update.LAP.PV3 -dbname gpc1 warptool -revertwarped -fault 4 -label update.LAP.PV3 -dbname gpc1
- CCL: some items unable to process on OSS.nightlyscience and change their label
chiptool -dbname gpc2 -updaterun -set_label OSS.nightlyscience.drop -chip_id 8 chiptool -dbname gpc2 -updaterun -set_label OSS.nightlyscience.clear -chip_id o8628h0434o
Sunday : 2019.05.26
- CCL: the chip processing of ota03 seems to be longer tonight so I set "kill" script to run every 10 mins.
check chip.imfile.run time under pantasks by "status -taskstats" ++ survey.diff | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.magic | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.addstar | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.mergedvodb | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.destreak | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.dist | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.chip.bg | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.warp.bg | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.publish | 436 2.00 2.41 16.59 | 436 2.00 2.41 16.59 | 0 NONE 0.00 0.00 | ++ survey.staticskysingle | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.skycal | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.lapgroup | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ survey.relexp | 86 10.00 10.45 23.47 | 86 10.00 10.45 23.47 | 0 NONE 0.00 0.00 | ++ survey.relstack | 88 10.00 10.27 12.65 | 88 10.00 10.27 12.65 | 0 NONE 0.00 0.00 | ++ dqstats.load | 1328 1.00 1.55 39.57 | 1328 1.00 1.55 39.57 | 0 NONE 0.00 0.00 | ++ dqstats.run | 165 1.24 1.84 9.42 | 165 1.24 1.84 9.42 | 0 NONE 0.00 0.00 | -+ dqstats.revert | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | +- ns.initday.load | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | ++ ns.dqstats.load | 863 20.00 29.84 80.57 | 862 20.00 29.84 80.57 | 0 NONE 0.00 0.00 | ++ ns.diffs.run | 1325 1.10 3.14 39.56 | 1324 1.10 3.14 39.56 | 1 5.00 5.00 5.00 | ++ fake.imfile.load | 1329 1.00 1.66 32.93 | 1327 1.00 1.66 32.93 | 0 NONE 0.00 0.00 | ++ fake.imfile.run | 8940 1.55 2.72 96.31 | 8923 1.55 2.73 96.31 | 0 NONE 0.00 0.00 | ++ fake.advanceexp | 883 1.00 1.60 27.31 | 883 1.00 1.60 27.31 | 0 NONE 0.00 0.00 | ++ fake.revert | 12 60.00 60.19 60.79 | 12 60.00 60.19 60.79 | 0 NONE 0.00 0.00 | ++ publish.load | 1323 1.00 1.59 39.73 | 1322 1.00 1.59 39.73 | 0 NONE 0.00 0.00 | ++ publish.run | 53 284.57 394.62 553.07 | 50 284.57 412.62 553.07 | 0 NONE 0.00 0.00 | ++ publish.revert | 433 5.00 5.55 35.29 | 433 5.00 5.55 35.29 | 0 NONE 0.00 0.00 | ++ diff.skycell.load | 1324 1.00 1.61 39.72 | 1324 1.00 1.61 39.72 | 0 NONE 0.00 0.00 | ++ diff.skycell.run | 3123 23.83 102.34 240.11 | 3102 23.83 104.53 240.11 | 0 NONE 0.00 0.00 | ++ diff.advance | 888 1.00 1.56 27.32 | 888 1.00 1.56 27.32 | 0 NONE 0.00 0.00 | ++ diff.revert | 14 60.00 60.14 60.42 | 14 60.00 60.14 60.42 | 0 NONE 0.00 0.00 | ++ warp.exp.load | 1339 1.00 1.52 29.31 | 1339 1.00 1.52 29.31 | 0 NONE 0.00 0.00 | ++ warp.exp.run | 148 63.29 92.11 130.70 | 146 63.29 91.87 130.70 | 0 NONE 0.00 0.00 | ++ warp.skycell.load | 1100 1.00 21.18 61.36 | 1099 1.00 21.18 61.36 | 0 NONE 0.00 0.00 | ++ warp.skycell.run | 89696 2.93 65.09 289.73 | 85028 11.66 68.51 289.73 | 4361 2.93 4.58 14.64 | ++ warp.advancerun | 884 1.00 2.33 39.69 | 884 1.00 2.33 39.69 | 0 NONE 0.00 0.00 | ++ warp.revert.overlap | 14 60.00 60.46 62.14 | 13 60.00 60.46 62.14 | 0 NONE 0.00 0.00 | ++ warp.revert.warped | 15 60.00 63.03 73.09 | 15 60.00 63.03 73.09 | 0 NONE 0.00 0.00 | ++ camera.exp.load | 1329 1.00 1.51 29.90 | 1329 1.00 1.51 29.90 | 0 NONE 0.00 0.00 | ++ camera.exp.run | 162 290.24 427.25 739.02 | 150 290.24 436.44 739.02 | 0 NONE 0.00 0.00 | ++ camera.revert | 14 60.00 60.92 66.61 | 14 60.00 60.92 66.61 | 0 NONE 0.00 0.00 | ++ chip.imfile.run | 81531 22.61 55.05 1941.62 | 81186 22.61 54.80 1637.52 | 27 64.03 1318.24 1941.62 | ++ chip.advanceexp | 887 1.00 1.61 39.68 | 887 1.00 1.61 39.68 | 0 NONE 0.00 0.00 | ++ chip.revert | 14 60.00 60.15 60.50 | 13 60.00 60.15 60.50 | 1 60.06 60.06 60.06 | -+ warp.summary.load | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | -+ warp.summary.run | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | -+ diff.summary.load | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | -+ diff.summary.run | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | -+ survey.warpstack.diff | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | -- survey.stackstack.diff | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | 0 NONE 0.00 0.00 | +- ns.stacks.run | 218 21.24 23.59 39.12 | 218 21.24 23.59 39.12 | 0 NONE 0.00 0.00 | ++ chip.imfile.load | 1326 2.00 3.68 66.79 | 1326 2.00 3.68 66.79 | 0 NONE 0.00 0.00 |
Monday : 2019.05.27
- CCL: the first chunk "OSSR.R12N1.17.Q.w" did not finishe due to one bad qulity image "o8631g0087o"
check first chunk chunk object warp1 warp2 warp3 warp4 diff1 diff2 diff3 diff4 workdir label data_group reduction OSSR.R12N1.17.Q.w ps1_33_3589 2112552 2112547 2112559 2112577 1772058 1772058 1772098 1772098 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3590 2112519 2112542 2112567 2112566 1772044 1772044 1772086 1772086 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3596 2112536 2112534 2112561 2112568 1772037 1772037 1772081 1772081 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3597 2112523 2112529 2112545 2112556 1772028 1772028 1772069 1772069 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3604 2112528 2112546 2112564 2112578 1772054 1772054 1772099 1772099 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3683 2112527 2112544 2112549 2112629 1772052 1772052 1772173 1772173 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3694 2112541 2112538 2112553 2112585 1772046 1772046 1772111 1772111 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3695 2112530 2112555 2112581 2112584 1772067 1772067 1772107 1772107 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3696 2112520 2112531 2112560 2112607 1772031 1772031 1772143 1772143 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3697 2112521 2112565 2112557 2112569 1772080 1772080 1772087 1772087 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3710 2112517 2112543 2112548 2112582 1772045 1772045 1772106 1772106 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3711 2112524 2112532 2112551 2112563 1772032 1772032 1772077 1772077 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3712 2112535 2112539 2112550 2112571 1772041 1772041 1772091 1772091 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3713 2112516 2112525 2112540 2112554 1772022 1772022 1772063 1772063 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3728 0 2112533 2112570 2112562 0 1772092 1772092 0 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT OSSR.R12N1.17.Q.w ps1_33_3730 2112526 2112558 2112537 2112573 1772071 1772071 1772094 1772094 neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 SWEETSPOT manually run difftool difftool -dbname gpc1 -definewarpwarp -warp_id 2112533 -template_warp_id 2112562 -backwards -set_workdir neb://@HOST@.0/gpc1/20190528 -set_dist_group NULL -set_label OSS.nightlyscience -set_data_group OSS.20190528 -set_reduction SWEETSPOT -simple -rerun -set_note "desperate diff" -pretend got 1772289 new neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 NULL SWEETSPOT 2019-05-28T09:52:58.069883 RINGS.V3 T T 0 0.000000 nan nan nan nan 1 desperate_diff need run pubtool?
Tuesday : 2019.05.28
Wednesday : 2019.05.29
- TdB: In an effort to automate the desperate diff generation, a revised nightly_science.pl script has been made. It is called nightly_science_desdifftest.pl and found under ~ippitc for the time being. The main change is that a new subroutine is created, called desperate_diff_singles which is called after the regular queue_diffs subroutine when the configuration of a particular label has desperate diffs turned on. This subroutine is much like the desperate_diff_queue subroutine that gets executed at the end of a night, but instead queues up desperate diffs for a given object on the fly when the following conditions are met:
-it has an uneven number of warp exposures -the time since the last exposure is at least x hours, where x is set as an input parameter called desdiffdt in units of hours (The default is 3 hours)
An example call to the script is:nightly_science_desdifftest.pl --dbname gpc1 --camera GPC1 --verbose --check_diffs --date 2019-05-28 --desdiffdt 3. --debug
For the time being, we should test the workings of the script. A crontab has been added on ippc30 which runs the script every 3 hours and pipes the output of the check_diffs for both gpc1 and gpc2 to the ps-ipp-ops mailing list. Furthermore, for the czars, they should try out desperate diff queuing using the new way by picking an appropriate desdiffdt time when the first chunk is done and incomplete. A word of caution: Do not use a desdiffdt that is too short! The timescale is there by virtue of not having a more sensible way of judging whether a chunk is complete and will not be resumed/picked up. Picking a very low desdiffdt could lead to unforeseen circumstanced like queuing desperate diffs for chunks still in progress or temporarily interrupted.
The script will not execute diffs when using the debug option, so we can check its usage and behaviour. The messages regarding the new subroutine are marked using outputs preceded by the header desp_diff_singles:
Thursday : 2019.05.30
Note:
See TracWiki
for help on using the wiki.
