PS1_IPP_Czarlog_20190524 – Pan-STARRS IPP

wiki:PS1_IPP_Czarlog_20190524

Context Navigation

Version 22 (modified by cclin33, 7 years ago) ( diff )
--

PS1 IPP Czar Logs for the week 2019.05.24 - 2019.05.DD

1. PS1 IPP Czar Logs for the week 2019.05.24 - 2019.05.DD

(Up to PS1 IPP Czar Logs)

Friday : 2019.05.24

Czar Handover from JRF -> CCL

Updates Updates still need doing, so far the last label is: LAP.PV3.20140730.20150%.

wait cleaning

See last weeks handover notes. I've mostly been cleaning ones with the data_group ThreePi.201%, I think I started at 2018.

general update cleaning We've hit around 20k, so now would be a good time to clean up some chips. See last weeks handover notes

ippx node power draw As mentioned today there are still some power draw issues, choices are:

monitor ippx node power usage while nightly processing is going on -> remove nodes from stdscience processing if the load is too high (the nodes appear to correspond to the PDU circuits descending on the page and descending on the ITC cabinet sheet -- I'll get Haydn to clarify).

PS1 pointing Keep an eye on the pointing using the commands listed in last weeks czarlog (under misaligned visits heading). If they look bad tell ps-obs and Ken, as they should be better than a week ago due to a new pointing model.

PS2 pointing Watch out for very poor pointing, worth mentioning to people too and it may cause WSdiff issues in the morning (WSdiffs are currently only processed in the morning now).

Euclid Bright Twi reprocess There are some Euclid observations that have not been processed as they have a comment string with a different format (the config has now been changed so they should process automatically). The Euclid observations under the new format, and before the config change, will need to be reprocessed. See last weeks czar log for the command, and make sure the date in the chiptool command is the date are of the observation (compare with the previous Euclid bright twi observations that have been done before the comment change).

ippx nodes power draw by JRF

Last week the xnodes were redistributed so that they could handle the heat better (a gap was left between each one). This helped the heat issue but we uncovered an issue with too much power being used by the machines.

Go to the following to check the status of the PDUs in i06, the cabient with most of the xnodes: SEE EMAIL FROM 20190515 FOR URL Then : Applications -> IPDU Power Mgmt -> View IPDUs Info

An example readout

ID: i06pdu2 (serial port 47)   Clear Max Current
 Model: Cyclades PM20i/30A  Clear Max Power
Reset Max Env Sensors
 Number of Outlets: 20
 Number of banks: 2  Single-Phase 
 Software Version: 1.9.1 May 2, 2007
 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
 PDU Power Consumption(estimated):  5345.6 W    Max Detected: 6676.8 W
 PDU Voltage(estimated):  208 V 
 PDU Power Factor(estimated):  1.00 
 Bank(A): 
     Current(Normal):  15.9 A   Max Detected: 17.9 A
     Power Consumption(estimated):  3307.2 W    Max Detected: 3723.2 W
     Voltage(estimated):  208 V 
     Power Factor(estimated):  1.00 
 Bank(B): 
     Current(Normal):  9.6 A    Max Detected: 14.7 A
     Power Consumption(estimated):  1996.8 W    Max Detected: 3057.6 W
     Voltage(estimated):  208 V 
     Power Factor(estimated):  1.00 
 Temperature-Internal(T1):  50.9°C(123.5°F)     Max Detected: 51.9°C(125.3°F)
 ID: i06pdu1 (serial port 48)   Clear Max Current
 Model: Cyclades PM20i/30A  Clear Max Power
Reset Max Env Sensors
 Number of Outlets: 20
 Number of banks: 2  Single-Phase 
 Software Version: 1.9.1 May 2, 2007
 PDU Current(Normal):  25.7 A   Max Detected: 28.4 A
 PDU Power Consumption(estimated):  5345.6 W    Max Detected: 5907.2 W
 PDU Voltage(estimated):  208 V 
 PDU Power Factor(estimated):  1.00 
 Bank(A): 
     Current(Normal):  14.6 A   Max Detected: 16.1 A
     Power Consumption(estimated):  3036.8 W    Max Detected: 3348.8 W
     Voltage(estimated):  208 V 
     Power Factor(estimated):  1.00 
 Bank(B): 
     Current(Normal):  11.1 A   Max Detected: 12.7 A
     Power Consumption(estimated):  2308.8 W    Max Detected: 2641.6 W
     Voltage(estimated):  208 V 
     Power Factor(estimated):  1.00 
 Temperature-Internal(T1):  52.9°C(127.1°F)     Max Detected: 53.9°C(128.9°F)

We are only really interested in the power values:

 1st measure:
 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
     Current(Normal):  15.9 A   Max Detected: 17.9 A
     Current(Normal):  9.6 A    Max Detected: 14.7 A
 PDU Current(Normal):  25.7 A   Max Detected: 28.4 A
     Current(Normal):  14.6 A   Max Detected: 16.1 A
     Current(Normal):  11.1 A   Max Detected: 12.7 A

 2nd measure
 PDU Current(Normal):  26.2 A   Max Detected: 32.1 A
     Current(Normal):  16.2 A   Max Detected: 17.9 A
     Current(Normal):  9.8 A    Max Detected: 14.7 A
 PDU Current(Normal):  26.1 A   Max Detected: 28.4 A
     Current(Normal):  15.2 A   Max Detected: 16.1 A
     Current(Normal):  11.2 A   Max Detected: 12.7 A

 3rd measure
 PDU Current(Normal):  25.7 A   Max Detected: 32.1 A
     Current(Normal):  16.0 A   Max Detected: 17.9 A
     Current(Normal):  10.0 A   Max Detected: 14.7 A
 PDU Current(Normal):  25.8 A   Max Detected: 28.4 A
     Current(Normal):  15.0 A   Max Detected: 16.1 A
     Current(Normal):  10.8 A   Max Detected: 12.7 A
 
 4th measure
 PDU Current(Normal):  25.9 A   Max Detected: 32.1 A
     Current(Normal):  15.9 A   Max Detected: 17.9 A
     Current(Normal):  10.0 A   Max Detected: 14.7 A
 PDU Current(Normal):  25.3 A   Max Detected: 28.4 A
     Current(Normal):  14.5 A   Max Detected: 16.1 A
     Current(Normal):  10.8 A   Max Detected: 12.7 A

Note that the order is PDU2 total, PDU2 Bank A, PDU2 Bank B, PDU1 total, PDU1 Bank A, PDU1 Bank B

To turn off some nodes in processing:

pantasks> controller host off ippxXXX

You'll need to run it for as many times as the node is used (e.g. if it used 6 times by the controller, then send the command 6 times)

Turned off ippx001-008 in pantasks, and straight away there is a drop:

 PDU Current(Normal):  22.1 A   Max Detected: 32.1 A
     Current(Normal):  12.2 A   Max Detected: 17.9 A
     Current(Normal):  9.5 A    Max Detected: 14.7 A
 PDU Current(Normal):  25.0 A   Max Detected: 28.4 A
     Current(Normal):  14.2 A   Max Detected: 16.1 A
     Current(Normal):  11.1 A   Max Detected: 12.7 A

 PDU Current(Normal):  21.3 A   Max Detected: 32.1 A
     Current(Normal):  11.7 A   Max Detected: 17.9 A
     Current(Normal):  9.7 A    Max Detected: 14.7 A
 PDU Current(Normal):  26.0 A   Max Detected: 28.4 A
     Current(Normal):  14.9 A   Max Detected: 16.1 A
     Current(Normal):  11.1 A   Max Detected: 12.7 A

This is close to an acceptable level in pdu2, but still a little high. pdu1 on the otherhand looks relatively unchanged at around 14-15A.

I'll try turning off a few at the bottom of the cabinet (I'm unsure of the setup).

Turned off ippx037-040:

 PDU Current(Normal):  21.5 A   Max Detected: 32.1 A
     Current(Normal):  12.0 A   Max Detected: 17.9 A
     Current(Normal):  9.5 A    Max Detected: 14.7 A
 PDU Current(Normal):  23.2 A   Max Detected: 28.4 A
     Current(Normal):  14.6 A   Max Detected: 16.1 A
     Current(Normal):  8.7 A    Max Detected: 12.7 A

After waiting for those nodes to no longer have any jobs, there is still a large number

Although, it looks like Bank B is now lower. And Haydn's email said: Instead of having three 4-node ippxNNN boxes per circuit, plus the 1G switch and serial console, we should have a maximum of two 4-node ippxNNN boxes per circuit, which should get it down to 10-12A per circuit.

Therefore, there should be 2 pdus * 2 circuits * 2 nodes per circuit = 8 available slots! There are 11 sets in the cabinet... I'm guessing the three bottom ones are connected to somewhere else? And that's why turning off ippx037-040 didn't really affect things.

Looking at the 6th and 7th ones down the list, they are ipp021-024 and ipp025-028 (as I think pdu2 is above pdu1 based on the ippx001-008 that I turned off).

So I'm turning of ippx021-024

And the results are (a few mins apart):

 PDU Current(Normal):  21.6 A   Max Detected: 32.1 A
     Current(Normal):  12.0 A   Max Detected: 17.9 A
     Current(Normal):  9.5 A    Max Detected: 14.7 A
 PDU Current(Normal):  21.2 A   Max Detected: 28.4 A
     Current(Normal):  12.9 A   Max Detected: 16.1 A
     Current(Normal):  8.4 A    Max Detected: 12.7 A

 PDU Current(Normal):  21.6 A   Max Detected: 32.1 A
     Current(Normal):  12.0 A   Max Detected: 17.9 A
     Current(Normal):  10.0 A   Max Detected: 14.7 A
 PDU Current(Normal):  21.3 A   Max Detected: 28.4 A
     Current(Normal):  12.9 A   Max Detected: 16.1 A
     Current(Normal):  8.4 A    Max Detected: 12.7 A

So success! Maybe just a little high still, but much more manageable.

Which pdu supplies which node should definitely be documented, the ITC cabinet sheet sounds like a logical place for it.

Saturday : 2019.05.25

MEH: PS1:OTA03 not behaving well still -- adding script John used the other night to the crontab on ippc30
```
*/20    *      *       *       *      ~/sshkill_ppimage_dtime_host_ps1.csh 
```

CCL: fix some warp problems with missing *.mk.fits (update.LAP.PV3)

Exp Name	Exp ID	Chip ID	Cam ID	Fake ID	Warp ID	Skycell ID	State	Label	Fault	Revert Command
o5174g0278o	107982	1444296	1413039	1386519	1391801	skycell.1945.070	update	update.LAP.PV3	2	warp.revert.on and warp.revert.off
...
warp id
1391853 1391817 1391801 1392056 1392017 1391968 1392075 1392217 1391810 1392074 1399947 1399948 
1399053 1411587 1411937 1415176 1415201 1415242 1415172 1415228 1415184 1415241 1415197 1415213 1415185 
1415179 1415183 1415189 1415665 1415645 1415781 1415759 1415782 1415767 1429271 1415829 1415786 1429210 
1416312 1416240 1416301 1416388 1416296 1416269 1416401 1416521 1350397 1350403 1376934


Couldn't find input file: neb://any/gpc1/LAP.PV3.20140730.bigmem/2015/01/25/o5174g0278o.107982/o5174g0278o.107982.cm.1413039.XY56.mk.fits at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 563.
Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warptool -updateskyfile -warp_id 1391801 -skycell_id skycell.1945.070 -fault 2 -dbname gpc1]...


find their chip id, clean and update them.
1444294 1444295 1444296 1444297 1444300 1444303 1444305 1444306 1444307 1444309 1452266 1452267 1452313 1461544 1461545 
1462366 1462368 1462370 1462371 1462372 1462373 1462374 1462375 1462377 1462378 1462379 1462380 1462381 1462565 1462567 
1462603 1462605 1462606 1462607 1462609 1462615 1462619 1462755 1462790 1462794 1462806 1462807 1462810 1462811 1462813 1463125
1393807 1393804 1428048

then revert
warptool -revertwarped -fault 2 -label update.LAP.PV3 -dbname gpc1 
warptool -revertwarped -fault 4 -label update.LAP.PV3 -dbname gpc1

CCL: some items unable to process on OSS.nightlyscience and change their label

chiptool -dbname gpc2 -updaterun -set_label OSS.nightlyscience.drop -chip_id 8
chiptool -dbname gpc2 -updaterun -set_label OSS.nightlyscience.clear -chip_id o8628h0434o

Sunday : 2019.05.26

CCL: the chip processing of ota03 seems to be longer tonight so I set "kill" script to run every 10 mins.

check chip.imfile.run time under pantasks by "status -taskstats"
  ++ survey.diff            |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.magic           |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.addstar         |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.mergedvodb      |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.destreak        |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.dist            |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.chip.bg         |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.warp.bg         |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.publish         |   436   2.00   2.41  16.59 |   436   2.00   2.41  16.59 |     0   NONE   0.00   0.00 |
  ++ survey.staticskysingle |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.skycal          |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.lapgroup        |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ survey.relexp          |    86  10.00  10.45  23.47 |    86  10.00  10.45  23.47 |     0   NONE   0.00   0.00 |
  ++ survey.relstack        |    88  10.00  10.27  12.65 |    88  10.00  10.27  12.65 |     0   NONE   0.00   0.00 |
  ++ dqstats.load           |  1328   1.00   1.55  39.57 |  1328   1.00   1.55  39.57 |     0   NONE   0.00   0.00 |
  ++ dqstats.run            |   165   1.24   1.84   9.42 |   165   1.24   1.84   9.42 |     0   NONE   0.00   0.00 |
  -+ dqstats.revert         |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  +- ns.initday.load        |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  ++ ns.dqstats.load        |   863  20.00  29.84  80.57 |   862  20.00  29.84  80.57 |     0   NONE   0.00   0.00 |
  ++ ns.diffs.run           |  1325   1.10   3.14  39.56 |  1324   1.10   3.14  39.56 |     1   5.00   5.00   5.00 |
  ++ fake.imfile.load       |  1329   1.00   1.66  32.93 |  1327   1.00   1.66  32.93 |     0   NONE   0.00   0.00 |
  ++ fake.imfile.run        |  8940   1.55   2.72  96.31 |  8923   1.55   2.73  96.31 |     0   NONE   0.00   0.00 |
  ++ fake.advanceexp        |   883   1.00   1.60  27.31 |   883   1.00   1.60  27.31 |     0   NONE   0.00   0.00 |
  ++ fake.revert            |    12  60.00  60.19  60.79 |    12  60.00  60.19  60.79 |     0   NONE   0.00   0.00 |
  ++ publish.load           |  1323   1.00   1.59  39.73 |  1322   1.00   1.59  39.73 |     0   NONE   0.00   0.00 |
  ++ publish.run            |    53 284.57 394.62 553.07 |    50 284.57 412.62 553.07 |     0   NONE   0.00   0.00 |
  ++ publish.revert         |   433   5.00   5.55  35.29 |   433   5.00   5.55  35.29 |     0   NONE   0.00   0.00 |
  ++ diff.skycell.load      |  1324   1.00   1.61  39.72 |  1324   1.00   1.61  39.72 |     0   NONE   0.00   0.00 |
  ++ diff.skycell.run       |  3123  23.83 102.34 240.11 |  3102  23.83 104.53 240.11 |     0   NONE   0.00   0.00 |
  ++ diff.advance           |   888   1.00   1.56  27.32 |   888   1.00   1.56  27.32 |     0   NONE   0.00   0.00 |
  ++ diff.revert            |    14  60.00  60.14  60.42 |    14  60.00  60.14  60.42 |     0   NONE   0.00   0.00 |
  ++ warp.exp.load          |  1339   1.00   1.52  29.31 |  1339   1.00   1.52  29.31 |     0   NONE   0.00   0.00 |
  ++ warp.exp.run           |   148  63.29  92.11 130.70 |   146  63.29  91.87 130.70 |     0   NONE   0.00   0.00 |
  ++ warp.skycell.load      |  1100   1.00  21.18  61.36 |  1099   1.00  21.18  61.36 |     0   NONE   0.00   0.00 |
  ++ warp.skycell.run       | 89696   2.93  65.09 289.73 | 85028  11.66  68.51 289.73 |  4361   2.93   4.58  14.64 |
  ++ warp.advancerun        |   884   1.00   2.33  39.69 |   884   1.00   2.33  39.69 |     0   NONE   0.00   0.00 |
  ++ warp.revert.overlap    |    14  60.00  60.46  62.14 |    13  60.00  60.46  62.14 |     0   NONE   0.00   0.00 |
  ++ warp.revert.warped     |    15  60.00  63.03  73.09 |    15  60.00  63.03  73.09 |     0   NONE   0.00   0.00 |
  ++ camera.exp.load        |  1329   1.00   1.51  29.90 |  1329   1.00   1.51  29.90 |     0   NONE   0.00   0.00 |
  ++ camera.exp.run         |   162 290.24 427.25 739.02 |   150 290.24 436.44 739.02 |     0   NONE   0.00   0.00 |
  ++ camera.revert          |    14  60.00  60.92  66.61 |    14  60.00  60.92  66.61 |     0   NONE   0.00   0.00 |
  ++ chip.imfile.run        | 81531  22.61  55.05 1941.62 | 81186  22.61  54.80 1637.52 |    27  64.03 1318.24 1941.62 |
  ++ chip.advanceexp        |   887   1.00   1.61  39.68 |   887   1.00   1.61  39.68 |     0   NONE   0.00   0.00 |
  ++ chip.revert            |    14  60.00  60.15  60.50 |    13  60.00  60.15  60.50 |     1  60.06  60.06  60.06 |
  -+ warp.summary.load      |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  -+ warp.summary.run       |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  -+ diff.summary.load      |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  -+ diff.summary.run       |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  -+ survey.warpstack.diff  |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  -- survey.stackstack.diff |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |     0   NONE   0.00   0.00 |
  +- ns.stacks.run          |   218  21.24  23.59  39.12 |   218  21.24  23.59  39.12 |     0   NONE   0.00   0.00 |
  ++ chip.imfile.load       |  1326   2.00   3.68  66.79 |  1326   2.00   3.68  66.79 |     0   NONE   0.00   0.00 |

Monday : 2019.05.27

CCL: the first chunk "OSSR.R12N1.17.Q.w" did not finishe due to one bad qulity image "o8631g0087o"

check first chunk
chunk             	object	        warp1	warp2	warp3	warp4	diff1	diff2	diff3	diff4	workdir	label	data_group	reduction
OSSR.R12N1.17.Q.w	ps1_33_3589	2112552	2112547	2112559	2112577	1772058	1772058	1772098	1772098	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3590	2112519	2112542	2112567	2112566	1772044	1772044	1772086	1772086	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3596	2112536	2112534	2112561	2112568	1772037	1772037	1772081	1772081	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3597	2112523	2112529	2112545	2112556	1772028	1772028	1772069	1772069	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3604	2112528	2112546	2112564	2112578	1772054	1772054	1772099	1772099	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3683	2112527	2112544	2112549	2112629	1772052	1772052	1772173	1772173	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3694	2112541	2112538	2112553	2112585	1772046	1772046	1772111	1772111	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3695	2112530	2112555	2112581	2112584	1772067	1772067	1772107	1772107	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3696	2112520	2112531	2112560	2112607	1772031	1772031	1772143	1772143	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3697	2112521	2112565	2112557	2112569	1772080	1772080	1772087	1772087	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3710	2112517	2112543	2112548	2112582	1772045	1772045	1772106	1772106	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3711	2112524	2112532	2112551	2112563	1772032	1772032	1772077	1772077	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3712	2112535	2112539	2112550	2112571	1772041	1772041	1772091	1772091	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3713	2112516	2112525	2112540	2112554	1772022	1772022	1772063	1772063	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3728	0	2112533	2112570	2112562	0	1772092	1772092	0	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT
OSSR.R12N1.17.Q.w	ps1_33_3730	2112526	2112558	2112537	2112573	1772071	1772071	1772094	1772094	neb://@HOST@.0/gpc1/20190528	OSS.nightlyscience	OSS.20190528	SWEETSPOT

manually run difftool

difftool -dbname gpc1 -definewarpwarp -warp_id 2112533 -template_warp_id 2112562 -backwards -set_workdir neb://@HOST@.0/gpc1/20190528 -set_dist_group NULL -set_label OSS.nightlyscience -set_data_group OSS.20190528 -set_reduction SWEETSPOT -simple -rerun -set_note "desperate diff" -pretend

got 
1772289 new neb://@HOST@.0/gpc1/20190528 OSS.nightlyscience OSS.20190528 NULL SWEETSPOT  2019-05-28T09:52:58.069883 RINGS.V3 T T 0  0.000000 nan nan nan nan 1 desperate_diff

need run pubtool?

Tuesday : 2019.05.28

CCL: fix some chip/warp errors on update.LAP.PV3

SELECT exp_name,chipRun.chip_id,chipRun.state,warpRun.warp_id,warpRun.state,warpRun.label,warpSkyfile.fault FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.label like 'update.LAP.PV3' group by warp_id;
+-------------+---------+--------+---------+--------+----------------+-------+
| exp_name    | chip_id | state  | warp_id | state  | label          | fault |
+-------------+---------+--------+---------+--------+----------------+-------+
| o5476g0043o | 1385555 | full   | 1330070 | update | update.LAP.PV3 |     2 |
| o5477g0043o | 1385562 | full   | 1330072 | update | update.LAP.PV3 |     2 |
| o5492g0068o | 1386613 | full   | 1331100 | update | update.LAP.PV3 |     2 |
| o5492g0047o | 1386606 | full   | 1331103 | update | update.LAP.PV3 |     2 |
| o5492g0071o | 1386614 | full   | 1331116 | update | update.LAP.PV3 |     2 |
| o5492g0050o | 1386607 | full   | 1331119 | update | update.LAP.PV3 |     2 |
| o5545g0178o | 1444319 | full   | 1391886 | update | update.LAP.PV3 |     2 |
| o5545g0373o | 1444321 | full   | 1391954 | update | update.LAP.PV3 |     2 |
| o5545g0160o | 1444314 | full   | 1391960 | update | update.LAP.PV3 |     2 |
| o5511g0558o | 1444386 | full   | 1392003 | update | update.LAP.PV3 |     2 |
| o5545g0159o | 1444313 | full   | 1392018 | update | update.LAP.PV3 |     2 |
| o5545g0355o | 1444228 | full   | 1392057 | update | update.LAP.PV3 |     2 |
| o5545g0175o | 1444317 | full   | 1392116 | update | update.LAP.PV3 |     2 |
| o5511g0576o | 1444389 | full   | 1392209 | update | update.LAP.PV3 |     2 |
| o5545g0177o | 1444318 | full   | 1392220 | update | update.LAP.PV3 |     2 |
| o5545g0174o | 1444316 | full   | 1392402 | update | update.LAP.PV3 |     2 |
| o5561g0366o | 1446275 | full   | 1396664 | update | update.LAP.PV3 |     2 |
| o5545g0198o | 1446526 | full   | 1397154 | update | update.LAP.PV3 |     2 |
| o7047g0144o | 1452313 | full   | 1399053 | update | update.LAP.PV3 |     2 |
| o7047g0165o | 1452266 | full   | 1399947 | update | update.LAP.PV3 |     2 |
| o7047g0166o | 1452267 | full   | 1399948 | update | update.LAP.PV3 |     2 |
| o5567g0255o | 1458509 | full   | 1408829 | update | update.LAP.PV3 |     2 |
| o5593g0155o | 1461891 | update | 1413545 | update | update.LAP.PV3 |     2 |
| o5593g0133o | 1461887 | update | 1413562 | update | update.LAP.PV3 |     2 |
| o5545g0559o | 1462623 | update | 1415795 | update | update.LAP.PV3 |     2 |
| o5566g0474o | 1461739 | update | 1429470 | update | update.LAP.PV3 |     2 |
+-------------+---------+--------+---------+--------+----------------+-------+

If their error messages are related to the missing "mk.fits, ch.fits, etc.", to reprocess them is the ways I can solve the issue.

chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name xxx

then send the old chip and warp to clean 
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxx
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxx

If their error messages are related to missing the "psf, mdc", try copy and overwrite the 0 size file with validated one.
then revert them 
warptool -revertwarped -fault 2 -label update.LAP.PV3 -dbname gpc1

Wednesday : 2019.05.29

TdB: In an effort to automate the desperate diff generation, a revised nightly_science.pl script has been made. It is called nightly_science_desdifftest.pl and found under ~ippitc for the time being. The main change is that a new subroutine is created, called desperate_diff_singles which is called after the regular queue_diffs subroutine when the configuration of a particular label has desperate diffs turned on. This subroutine is much like the desperate_diff_queue subroutine that gets executed at the end of a night, but instead queues up desperate diffs for a given object on the fly when the following conditions are met:
```
-it has an uneven number of warp exposures
-the time since the last exposure is at least x hours, where x is set as an input parameter called desdiffdt in units of hours (The default is 3 hours)
```
An example call to the script is:
```
nightly_science_desdifftest.pl --dbname gpc1 --camera GPC1 --verbose --check_diffs --date 2019-05-28 --desdiffdt 3. --debug
```
For the time being, we should test the workings of the script. A crontab has been added on ippc30 which runs the script every 3 hours and pipes the output of the check_diffs for both gpc1 and gpc2 to the ps-ipp-ops mailing list. Furthermore, for the czars, they should try out desperate diff queuing using the new way by picking an appropriate desdiffdt time when the first chunk is done and incomplete. A word of caution: Do not use a desdiffdt that is too short! The timescale is there by virtue of not having a more sensible way of judging whether a chunk is complete and will not be resumed/picked up. Picking a very low desdiffdt could lead to unforeseen circumstanced like queuing desperate diffs for chunks still in progress or temporarily interrupted.

The script will not execute diffs when using the debug option, so we can check its usage and behaviour. The messages regarding the new subroutine are marked using outputs preceded by the header desp_diff_singles:

CCL: PS1:OTA03 still not get back, I set it run the script from 9pm-5am every 20 mins.

## OTA03 behaving badly, kill long running chip jobs and set quality fault -- only turn on when really needed, very unrestricted...
*/20    21-23,0-5      *       *       *      ~/sshkill_ppimage_dtime_host_ps1.csh

update progress The latest results from TdB's mysql code,

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT count(*),gather.data_group,gather.label FROM (SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.state,subb.data_group,subb.label FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.label LIKE "%LAP.PV3%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state,warpRun.data_group,camRun.label FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.label NOT LIKE "%broke%" AND warpRun.label NOT LIKE "%hold%" AND rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id)) as gather WHERE (state LIKE "%cleaned" OR state LIKE "update") GROUP BY data_group;'

count(*)	data_group	label
1	LAP.PV3.20140730.20140914	LAP.PV3.20140730.final
1	LAP.PV3.20140730.20141031	LAP.PV3.20140730.final
1	LAP.PV3.20140730.20141102	LAP.PV3.20140730.final
1	LAP.PV3.20140730.20141108	LAP.PV3.20140730.final
3	LAP.PV3.20140730.20150125	LAP.PV3.20140730.dupe
24	LAP.PV3.20140730.20150213	LAP.PV3.20140730.final
8	LAP.PV3.20140730.20150216	LAP.PV3.20140730.final
8	LAP.PV3.20140730.20150218	LAP.PV3.20140730.final
42	LAP.PV3.20140730.20150221	LAP.PV3.20140730.final
117	LAP.PV3.20140730.20150222	LAP.PV3.20140730.final
32	LAP.PV3.20140730.20150223	LAP.PV3.20140730.final
1	LAP.PV3.20140730.20150224	LAP.PV3.20140730.final
2	LAP.PV3.20140730.bigmem.20150102	LAP.PV3.20140730.final
1	LAP.PV3.20140730.bigmem.20150105	LAP.PV3.20140730.final
1	LAP.PV3.20140730.bigmem.20150108	LAP.PV3.20140730.final

Above has been update, but some of the exposure chip/warp are keeping on update state, not gone through full state.

mysql -hscidbm -ux -px gpc1 -B -e 'SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.maxstate,subb.data_group,subb.label,subb.maxchipstate FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.data_group LIKE "LAP.PV3.20140730.bigmem.xxxxx" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state as maxstate,warpRun.data_group,camRun.label,chipRun.state as maxchipstate FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id) WHERE subb.maxstate LIKE "cleaned" limit 1000;' | grep -v exp_id | awk '{print "chiptool -dbname gpc1 -updaterun  -set_label update.LAP.PV3 -chip_id  "$3" ; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id  "$3" ; warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$2"; warptool -dbname gpc1  -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$2 }' | tcsh

447	LAP.PV3.20140730.bigmem.20150112	LAP.PV3.20140730.final  
576	LAP.PV3.20140730.bigmem.20150113	LAP.PV3.20140730.final
961	LAP.PV3.20140730.bigmem.20150114	LAP.PV3.20140730.final
581	LAP.PV3.20140730.bigmem.20150115	LAP.PV3.20140730.final
487	LAP.PV3.20140730.bigmem.20150116	LAP.PV3.20140730.final
28	LAP.PV3.20140730.bigmem.20150117	LAP.PV3.20140730.final
753	LAP.PV3.20140730.bigmem.20150118	LAP.PV3.20140730.final
1519	LAP.PV3.20140730.bigmem.20150119	LAP.PV3.20140730.final
592	LAP.PV3.20140730.bigmem.20150120	LAP.PV3.20140730.final
78	LAP.PV3.20140730.bigmem.20150121	LAP.PV3.20140730.final
576	LAP.PV3.20140730.bigmem.20150122	LAP.PV3.20140730.final
580	LAP.PV3.20140730.bigmem.20150123	LAP.PV3.20140730.final
250	LAP.PV3.20140730.bigmem.20150124	LAP.PV3.20140730.final
4884	LAP.PV3.20140730.bigmem.20150125	LAP.PV3.20140730.final
310	LAP.PV3.20140730.bigmem.20150128	LAP.PV3.20140730.final
2286	LAP.PV3.20140730.bigmem.20150129	LAP.PV3.20140730.final
373	LAP.PV3.20140730.bigmem.20150130	LAP.PV3.20140730.final
231	LAP.PV3.20140730.bigmem.20150131	LAP.PV3.20140730.final
1997	LAP.PV3.20140730.bigmem.20150202	LAP.PV3.20140730.final
62	LAP.PV3.20140730.bigmem.20150204	LAP.PV3.20140730.final
1228	LAP.PV3.20140730.bigmem.20150205	LAP.PV3.20140730.final
68	LAP.PV3.20140730.bigmem.20150206	LAP.PV3.20140730.final
2	LAP.PV3.20140730.ipp.20141106	LAP.PV3.20140730.final
7	LAP.PV3.20140730.ipp.20141107	LAP.PV3.20140730.final
4	LAP.PV3.20140730.ipp.20141108	LAP.PV3.20140730.final
2	LAP.PV3.20140730.ipp.20141117	LAP.PV3.20140730.dupe
1	LAP.PV3.20140730.ipp.20141118	LAP.PV3.20140730.final
3	LAP.PV3.20140730.ipp.20141120	LAP.PV3.20140730.dupe
3	LAP.PV3.20140730.ipp.20141121	LAP.PV3.20140730.final
1	LAP.PV3.20140730.ipp.20141122	LAP.PV3.20140730.final
1	LAP.PV3.20140730.ipp.20150218	LAP.PV3.20140730.final
1	LAP.PV3W.20150312.20150320	LAP.PV3W.20150312
19	LAP.PV3W.20150312.20150329	LAP.PV3W.20150312
16	LAP.PV3W.20150312.20150409	LAP.PV3W.20150312
8	LAP.PV3W.20150312.20150413	LAP.PV3W.20150312
30	LAP.ThreePi.20120706.20121002	PV2.cleanup
2	LAP.ThreePi.20120706.20121004	PV2.cleanup
3	LAP.ThreePi.20120706.20121205	PV2.cleanup
1	mops.fixbrokenLAP.20180323	ws_nightly_update
1	OSS.20140423	OSS.nightlyscience
16	ThreePi.20120927	PV2.cleanup

cleanup progress Actually, when I check their data_state from chipProcessedImfile table, most of them have been marked as "cleaned". I also check if there are other states not like 'cleaned' and then send them "goto_cleaned" state.

--> to check if there are any non-cleaned data_state.
SELECT chipProcessedImfile.data_state,chipProcessedImfile.chip_id,chipRun.chip_id,chipProcessedImfile.quality,chipProcessedImfile.fault FROM chipProcessedImfile, chipRun \
WHERE chipProcessedImfile.data_state not like 'cleaned' and chipRun.state LIKE 'wait' AND chipRun.data_group LIKE 'MD07.20%' AND chipRun.chip_id = chipProcessedImfile.chip_id GROUP BY chipProcessedImfile.chip_id,chipProcessedImfile.data_state;

+------------+---------+---------+---------+-------+
| data_state | chip_id | chip_id | quality | fault |
+------------+---------+---------+---------+-------+
| purged     |   19573 |   19573 |       0 |     0 |
| purged     |   19574 |   19574 |       0 |     0 |
| purged     |   19575 |   19575 |       0 |     0 |
| purged     |   19576 |   19576 |       0 |     0 |
...
| full       |  827266 |  827266 |       0 |     0 |
| full       |  848347 |  848347 |       0 |     0 |
+------------+---------+---------+---------+-------+
299 rows in set (0.27 sec)

--> this script send "wait" state, but "non-cleaned" date_state to be "goto_cleaned"
mysql -hscidbm -uippuser -pippuser gpc1 -B -e '\
SELECT chipProcessedImfile.chip_id,chipProcessedImfile.data_state,chipRun.chip_id FROM chipProcessedImfile, chipRun \
WHERE chipProcessedImfile.data_state not like "cleaned" \
  AND chipRun.state LIKE "wait" \
  AND chipRun.data_group LIKE "MD07.20%" \
  AND chipRun.chip_id = chipProcessedImfile.chip_id GROUP BY chipProcessedImfile.chip_id,chipProcessedImfile.data_state;' |grep -v data_state | \
awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id  "$1}'  |tcsh

--> most of "wait" states are in fact with date_state "cleaned", I double check some of them with "neb-stat" and can't find them anymore.
SELECT chipProcessedImfile.data_state,chipProcessedImfile.chip_id,chipRun.chip_id,chipProcessedImfile.quality,chipProcessedImfile.fault FROM chipProcessedImfile, chipRun \
WHERE chipRun.state LIKE 'wait' AND chipRun.data_group LIKE 'LAP.ThreePi.20120706.2012%' AND chipRun.chip_id = chipProcessedImfile.chip_id GROUP BY chipProcessedImfile.chip_id,chipProcessedImfile.data_state;

+------------+---------+---------+---------+-------+
| data_state | chip_id | chip_id | quality | fault |
+------------+---------+---------+---------+-------+
| cleaned    |  657694 |  657694 |       0 |     0 |
| cleaned    |  657695 |  657695 |       0 |     0 |
| cleaned    |  657696 |  657696 |       0 |     0 |
....
| cleaned    |  723556 |  723556 |       0 |     0 |
| cleaned    |  723557 |  723557 |       0 |     0 |
+------------+---------+---------+---------+-------+
18378 rows in set (1.75 sec)

--> this script send "wait" state, but in fact "cleaned" date_state to be "cleaned" state, which avoid executing cleanup script again.
mysql -hscidbm -uippuser -pippuser gpc1 -B -e '\
SELECT chip_id,data_group FROM chipRun \
WHERE  state like "wait" AND data_group LIKE "MD07.20%";' | grep -v data_group | \
awk '{print "chiptool -dbname gpc1 -updaterun -set_state cleaned -set_label goto_cleaned -chip_id  "$1}' | tcsh

From above method, I've checked all "LAP" related chip with "wait" state and set them cleaned. Do we want to set all of the "wait" state to "cleaned" or just "LAP"?
mysql -hscidbm -ux -px gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait"  GROUP BY data_group) as s ORDER by waitct ASC;' | grep LAP

Thursday : 2019.05.30

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text