PS1 IPP Czar Logs for the week 2019.05.17 - 2019.05.23

(Up to PS1 IPP Czar Logs)

Friday : 2019.05.17

• TdB: 11:00 A set of 620 warps were stuck on IPPmonitor and not processing. Sending them to cleanup and back did not help. Their corresponding chip exps have now been sent to cleanup and are being updated again. Hopefully this will dislodge them.

• TdB: 13:46 The night_report.pl script has been updated to properly parse exposures in the BRIGHT obs_mode, which was causing problems in the report from last night. Testing shows it works well now.

CZAR HANDOVER by TdB -> JFL

updates:

The update beast needs to be fed regularly. The current state of %LAP.PV3% updates still needed is:

  42    LAP.PV3.20140730.2014% ->left over exps from wait state cleaning
  23595  LAP.PV3.20140730.20150% 
  32335  LAP.PV3.20140730.bigmem.20150%

A more detailed breakdown can be obtained using:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT count(*),gather.data_group,gather.label FROM (SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.state,subb.data_group,subb.label FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.label LIKE "%LAP.PV3%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state,warpRun.data_group,camRun.label FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.label NOT LIKE "%broke%" AND warpRun.label NOT LIKE "%hold%" AND rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id)) as gather WHERE (state LIKE "%cleaned" OR state LIKE "update") GROUP BY data_group;'

For queuing updates, I was using ym own query besides Mark's one, if there is a need to do selections on e.g. the chip state of the exp corresponding to the maximum warp id exp:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.maxstate,subb.data_group,subb.label,subb.maxchipstate FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.data_group LIKE "LAP.PV3.20140730.20150%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state as maxstate,warpRun.data_group,camRun.label,chipRun.state as maxchipstate FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id) WHERE subb.maxstate LIKE "cleaned" limit 1000;' | grep -v exp_id | awk '{print "chiptool -dbname gpc1 -updaterun  -set_label update.LAP.PV3 -chip_id  "$3" ; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id  "$3" ;warptool -dbname gpc1  -updaterun -set_label update.LAP.PV3 -warp_id "$2"; warptool -dbname gpc1  -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$2 }' | tcsh

One extra thing to think about: We are only updating warps with state=cleaned. Maybe there are warps with state=goto_cleaned which need doing? or state=update that somehow got stuck in a non-updating label?

wait state cleaning:

There are still plenty of wait state chip exps that need to be cleaned. We need to continue doing this until they are all done, in between other tasks. Get a sense of which data_groups need to be done, using:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait"  GROUP BY data_group) as s ORDER by waitct ASC;'

I have cleaned everything that will be used by the future updates, hopefully. But, you never know....

We can send a data_group to clean up, using:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT chip_id,data_group,label FROM chipRun WHERE state like "wait" and data_group like "LAP.PV3.20140730.20150%" and label not like "update.LAP.PV3";' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id  "$1}' | tcsh 

And to check if the cleanup is working ok, we can query the current states (and numbers) for a data_group:

mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label,state from (SELECT count(*) as waitct,data_group,label,state FROM chipRun where data_group like "LAP.PV3.20140730.20150%" GROUP BY data_group,state) as s;' 

general cleaning:

Once in a while the chip pixels that have been processed need to eb cleaning. Typically when a good 10k are done. Use the following query:

mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id)  where warpRun.label like "update.%" and warpRun.data_group like "%" and warpRun.state like "full" and chipRun.state like "full" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -state full -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1 }' | tcsh

PS2 data incoming:

PS2 should be up soon, and the x nodes are set up for their data processing. When they are in use, you might want to take a periodic look at the power draw in cabinet i.06. There was an email from Haydn about how to do this, but see me if you need more help.

General:

-The pstamp and cleanup pantasks need rebooting once their Njobs reach 100k.

-With all this cleaning going on, some machines might be put back to up once enough disk space is freed. Keep an eye on the red on IPPmonitor and see if you think enough space is freed.

-keep an eye on the night_report script when doing bright Euclid observing. See email from this morning. The script was reporting there were 0 diffs expected, which is clearly not right. update: the script has been adjusted, but keep an eye on it.

Saturday : 2019.05.18

getting warm at the ATRC

Temperatures rose pretty high over on Maui at the ATRC cluster (see this website for details: ​https://ps1wiki.ifa.hawaii.edu/cgi-bin/ps1opsDashboard.py)

I set the following machines to repair, just in case they need to be moved to down to reduce heat: ippb06,08,16-23

Setting these to repair, and it started moving into the evening, meant that the temperature dropped to safer levels

Sunday : 2019.05.19

Monday : 2019.05.20

JRF: First night with PS2 for a long time, but processing progressed pretty smoothly for the most part. However, MEH saw that the pointing was poor so suggested removing the OSS.WS.nightlyscience label so that their diff processing doesn't clog up the system. This is because for PS2 the WWdiffs and the WSdiffs share the same pantasks (for PS1 they are separate).

Tuesday : 2019.05.21

WSdiff processing by JRF

Added the OSS.WS.nightlyscience label back into PS2 processing early in the morning, after everything else had finished. The majority of diffs cleared no problem. There was one that I thought was stuck:

0   ippx021    BUSY   9195.68 0.0.1.8bd5  0 diff_skycell.pl --threads @MAX_THREADS@ --diff_id 50495 --skycell_id skycell.1370.020 --diff_skyfile_id 2720978 --outroot neb://ipp137.1/gpc2/OSS.WS.nightlyscience/2019/05/20/RINGS.V3/skycell.1370.020/RINGS.V3.skycell.1370.020.WS.dif.50495 --redirect-output --run-state new --reduction SWEETSPOT_WS --dbname gpc2 --verbose 

It's been there for a good 2 and half hours.

I found some info about WSdiffs being stuck in an older log:

past log : http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20180618
-- turn diff.revert.off, kill ppSub job, wait to fault and manually set quality 42 
difftool -dbname gpc2 -updatediffskyfile -fault 0 -set_quality 42 -diff_id 50495 -skycell_id skycell.1370.020

The job actually finished while I was looking this up... so nevermind.

To ensure that we don't have to manually intervene during the night, by removing the label again, I've changed the ~ippps2/stdscience/input file so that all WS labels are not automatically added on restart. This means that the label can be re-addedto pantasks at our leisure in the morning.

plots by JRF

czarplot.pl can be run manually to generate some plots, for example:

cd ~ippitc/src/ippMonitor/czartool ; ./czarplot.pl -rt -l OSS.WS.nightlyscience -p '12 hour'

This one gives you a rate plot and a cumulative time of the past 12 hours for the specific label. This is useful as at present automatic generation of these plots is disabled (they were barely ever used and greatly slowed down IPPmonitor).

Checking out diffs

To view the quality of the resulting diffs in ds9 try the following query (from MEH):

mysql -hscidbm -uX -pX gpc2 -B -e ' select path_base,diffSkyfile.skycell_id,sources from diffRun join diffSkyfile using (diff_id) join diffInputSkyfile using (diff_id) join warpRun on (diffInputSkyfile.warp1=warpRun.warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where  exp_name="o8507h0371o" and quality=0 and diff_mode=1 group by diffSkyfile.skycell_id limit 5; ' | grep -v path_base | awk '{print "neb-ls -p "$1".fits"}' | bash | xargs ~/bin/ds9 -zscale -prefs nancolor orange -geometry 900x800  -frame last -zoom 0.3 -match frame -tile grid gap 3

Make sure to change the exposure name to the one you desire, and the limit to how many you want to view at once (also note the ds9 path)

For last night, one of the first visits from the first chunk: o8624h0111o one of the second vists of the second chunk: o8624h0181o

The first is a very poor diff where there is a huge overlap in the diffs -> suggesting that the pointing is off. This is also evident when looking at the number of sources detected by the diffs, where we see >10000 deteections; normally less than 1000 are detected.

An example query that will show you how many sources were detected and the fraction of good pixels in the warp and diff stage. Looking at a percentage difference in the good_frac gives an indication on the quality: larger -ve frac -> a worse diff

SELECT 
    diffSkyfile.skycell_id, 
    warpSkyfile.skycell_id, 
    sources, 
    diffSkyfile.good_frac difffrac, 
    warpSkyfile.good_frac warpfrac, 
    diffSkyfile.good_frac - warpSkyfile.good_frac / warpSkyfile.good_frac as good_stuff 
FROM
    diffRun 
    JOIN diffSkyfile USING (diff_id) 
    JOIN diffInputSkyfile USING (diff_id) 
    JOIN warpRun on (diffInputSkyfile.warp1 = warpRun.warp_id) 
    JOIN fakeRun USING (fake_id) 
    JOIN camRun USING (cam_id) 
    JOIN chipRun USING (chip_id) 
    JOIN rawExp USING (exp_id) 
    JOIN warpSkyfile USING (warp_id) 
WHERE 
    exp_name="o8507h0371o" and diff_mode=1 
    AND 
    warpSkyfile.skycell_id = diffSkyfile.skycell_id 
    AND 
    warpSkyfile.good_frac > 0.5
GROUP BY 
    diffSkyfile.skycell_id 
ORDER BY 
    good_stuff DESC
LIMIT 20;

As a on liner:

SELECT diffSkyfile.skycell_id, warpSkyfile.skycell_id, sources, diffSkyfile.good_frac difffrac, warpSkyfile.good_frac warpfrac, diffSkyfile.good_frac - warpSkyfile.good_frac / warpSkyfile.good_frac as good_stuff FROM diffRun JOIN diffSkyfile USING (diff_id) JOIN diffInputSkyfile USING (diff_id) JOIN warpRun on (diffInputSkyfile.warp1 = warpRun.warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) JOIN warpSkyfile USING (warp_id) WHERE exp_name="o8507h0371o" and diff_mode=1 AND warpSkyfile.skycell_id = diffSkyfile.skycell_id AND warpSkyfile.good_frac > 0.5 GROUP BY diffSkyfile.skycell_id ORDER BY good_stuff DESC LIMIT 20;

Wednesday : 2019.05.22

• MEH: manually adding OSS.WS.nightlyscience to ippps2/stdscience now that nightly finished

Thursday : 2019.05.23