PS1 IPP Czar Logs for the week 2019.05.17 - 2019.05.23
(Up to PS1 IPP Czar Logs)
Friday : 2019.05.17
- TdB: 11:00 A set of 620 warps were stuck on IPPmonitor and not processing. Sending them to cleanup and back did not help. Their corresponding chip exps have now been sent to cleanup and are being updated again. Hopefully this will dislodge them.
- TdB: 13:46 The night_report.pl script has been updated to properly parse exposures in the BRIGHT obs_mode, which was causing problems in the report from last night. Testing shows it works well now.
CZAR HANDOVER by TdB -> JFL
updates:
The update beast needs to be fed regularly. The current state of %LAP.PV3% updates still needed is:
42 LAP.PV3.20140730.2014% ->left over exps from wait state cleaning 23595 LAP.PV3.20140730.20150% 32335 LAP.PV3.20140730.bigmem.20150%A more detailed breakdown can be obtained using:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT count(*),gather.data_group,gather.label FROM (SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.state,subb.data_group,subb.label FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.label LIKE "%LAP.PV3%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state,warpRun.data_group,camRun.label FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.label NOT LIKE "%broke%" AND warpRun.label NOT LIKE "%hold%" AND rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id)) as gather WHERE (state LIKE "%cleaned" OR state LIKE "update") GROUP BY data_group;'For queuing updates, I was using ym own query besides Mark's one, if there is a need to do selections on e.g. the chip state of the exp corresponding to the maximum warp id exp:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT suba.exp_id,subb.warp_id,subb.chip_id,subb.maxstate,subb.data_group,subb.label,subb.maxchipstate FROM (SELECT exp_id FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warpRun.state NOT LIKE "%full%" AND rawExp.camera LIKE "%GPC1%" AND camRun.data_group LIKE "LAP.PV3.20140730.20150%" GROUP BY exp_id) as suba INNER JOIN (SELECT * FROM (SELECT warp_id,chipRun.exp_id,chipRun.chip_id,warpRun.state as maxstate,warpRun.data_group,camRun.label,chipRun.state as maxchipstate FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.camera LIKE "%GPC1%" ORDER BY warp_id DESC) as od GROUP BY od.exp_id) AS subb ON (subb.exp_id = suba.exp_id) WHERE subb.maxstate LIKE "cleaned" limit 1000;' | grep -v exp_id | awk '{print "chiptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -chip_id "$3" ; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id "$3" ;warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$2"; warptool -dbname gpc1 -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$2 }' | tcshOne extra thing to think about: We are only updating warps with state=cleaned. Maybe there are warps with state=goto_cleaned which need doing? or state=update that somehow got stuck in a non-updating label?
wait state cleaning:
There are still plenty of wait state chip exps that need to be cleaned. We need to continue doing this until they are all done, in between other tasks. Get a sense of which data_groups need to be done, using:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait" GROUP BY data_group) as s ORDER by waitct ASC;'I have cleaned everything that will be used by the future updates, hopefully. But, you never know....
We can send a data_group to clean up, using:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT chip_id,data_group,label FROM chipRun WHERE state like "wait" and data_group like "LAP.PV3.20140730.20150%" and label not like "update.LAP.PV3";' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1}' | tcshAnd to check if the cleanup is working ok, we can query the current states (and numbers) for a data_group:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label,state from (SELECT count(*) as waitct,data_group,label,state FROM chipRun where data_group like "LAP.PV3.20140730.20150%" GROUP BY data_group,state) as s;'
general cleaning:
Once in a while the chip pixels that have been processed need to eb cleaning. Typically when a good 10k are done. Use the following query:
mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) where warpRun.label like "update.%" and warpRun.data_group like "%" and warpRun.state like "full" and chipRun.state like "full" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -state full -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1 }' | tcsh
PS2 data incoming:
PS2 should be up soon, and the x nodes are set up for their data processing. When they are in use, you might want to take a periodic look at the power draw in cabinet i.06. There was an email from Haydn about how to do this, but see me if you need more help.
General:
-The pstamp and cleanup pantasks need rebooting once their Njobs reach 100k.
-With all this cleaning going on, some machines might be put back to up once enough disk space is freed. Keep an eye on the red on IPPmonitor and see if you think enough space is freed.
-keep an eye on the night_report script when doing bright Euclid observing. See email from this morning. The script was reporting there were 0 diffs expected, which is clearly not right. update: the script has been adjusted, but keep an eye on it.
Saturday : 2019.05.18
getting warm at the ATRC
Temperatures rose pretty high over on Maui at the ATRC cluster (see this website for details: https://ps1wiki.ifa.hawaii.edu/cgi-bin/ps1opsDashboard.py)
I set the following machines to repair, just in case they need to be moved to down to reduce heat: ippb06,08,16-23
Setting these to repair, and it started moving into the evening, meant that the temperature dropped to safer levels
Sunday : 2019.05.19
Monday : 2019.05.20
JRF: First night with PS2 for a long time, but processing progressed pretty smoothly for the most part. However, MEH saw that the pointing was poor so suggested removing the OSS.WS.nightlyscience label so that their diff processing doesn't clog up the system. This is because for PS2 the WWdiffs and the WSdiffs share the same pantasks (for PS1 they are separate).
Tuesday : 2019.05.21
WSdiff processing by JRF
Added the OSS.WS.nightlyscience label back into PS2 processing early in the morning, after everything else had finished. The majority of diffs cleared no problem. There was one that I thought was stuck:
0 ippx021 BUSY 9195.68 0.0.1.8bd5 0 diff_skycell.pl --threads @MAX_THREADS@ --diff_id 50495 --skycell_id skycell.1370.020 --diff_skyfile_id 2720978 --outroot neb://ipp137.1/gpc2/OSS.WS.nightlyscience/2019/05/20/RINGS.V3/skycell.1370.020/RINGS.V3.skycell.1370.020.WS.dif.50495 --redirect-output --run-state new --reduction SWEETSPOT_WS --dbname gpc2 --verbose
It's been there for a good 2 and half hours.
I found some info about WSdiffs being stuck in an older log:
past log : http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20180618 -- turn diff.revert.off, kill ppSub job, wait to fault and manually set quality 42 difftool -dbname gpc2 -updatediffskyfile -fault 0 -set_quality 42 -diff_id 50495 -skycell_id skycell.1370.020
The job actually finished while I was looking this up... so nevermind.
To ensure that we don't have to manually intervene during the night, by removing the label again, I've changed the ~ippps2/stdscience/input file so that all WS labels are not automatically added on restart. This means that the label can be re-addedto pantasks at our leisure in the morning.
plots by JRF
czarplot.pl can be run manually to generate some plots, for example:
cd ~ippitc/src/ippMonitor/czartool ; ./czarplot.pl -rt -l OSS.WS.nightlyscience -p '12 hour'
This one gives you a rate plot and a cumulative time of the past 12 hours for the specific label. This is useful as at present automatic generation of these plots is disabled (they were barely ever used and greatly slowed down IPPmonitor).
Checking out diffs
To view the quality of the resulting diffs in ds9 try the following query (from MEH):
mysql -hscidbm -uX -pX gpc2 -B -e ' select path_base,diffSkyfile.skycell_id,sources from diffRun join diffSkyfile using (diff_id) join diffInputSkyfile using (diff_id) join warpRun on (diffInputSkyfile.warp1=warpRun.warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_name="o8507h0371o" and quality=0 and diff_mode=1 group by diffSkyfile.skycell_id limit 5; ' | grep -v path_base | awk '{print "neb-ls -p "$1".fits"}' | bash | xargs ~/bin/ds9 -zscale -prefs nancolor orange -geometry 900x800 -frame last -zoom 0.3 -match frame -tile grid gap 3
Make sure to change the exposure name to the one you desire, and the limit to how many you want to view at once (also note the ds9 path)
For last night, one of the first visits from the first chunk: o8624h0111o one of the second vists of the second chunk: o8624h0181o
The first is a very poor diff where there is a huge overlap in the diffs -> suggesting that the pointing is off. This is also evident when looking at the number of sources detected by the diffs, where we see >10000 deteections; normally less than 1000 are detected.
An example query that will show you how many sources were detected and the fraction of good pixels in the warp and diff stage. Looking at a percentage difference in the good_frac gives an indication on the quality: larger -ve frac -> a worse diff
SELECT
diffSkyfile.skycell_id,
warpSkyfile.skycell_id,
sources,
diffSkyfile.good_frac difffrac,
warpSkyfile.good_frac warpfrac,
diffSkyfile.good_frac - warpSkyfile.good_frac / warpSkyfile.good_frac as good_stuff
FROM
diffRun
JOIN diffSkyfile USING (diff_id)
JOIN diffInputSkyfile USING (diff_id)
JOIN warpRun on (diffInputSkyfile.warp1 = warpRun.warp_id)
JOIN fakeRun USING (fake_id)
JOIN camRun USING (cam_id)
JOIN chipRun USING (chip_id)
JOIN rawExp USING (exp_id)
JOIN warpSkyfile USING (warp_id)
WHERE
exp_name="o8507h0371o" and diff_mode=1
AND
warpSkyfile.skycell_id = diffSkyfile.skycell_id
AND
warpSkyfile.good_frac > 0.5
GROUP BY
diffSkyfile.skycell_id
ORDER BY
good_stuff DESC
LIMIT 20;
As a on liner:
SELECT diffSkyfile.skycell_id, warpSkyfile.skycell_id, sources, diffSkyfile.good_frac difffrac, warpSkyfile.good_frac warpfrac, diffSkyfile.good_frac - warpSkyfile.good_frac / warpSkyfile.good_frac as good_stuff FROM diffRun JOIN diffSkyfile USING (diff_id) JOIN diffInputSkyfile USING (diff_id) JOIN warpRun on (diffInputSkyfile.warp1 = warpRun.warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) JOIN warpSkyfile USING (warp_id) WHERE exp_name="o8507h0371o" and diff_mode=1 AND warpSkyfile.skycell_id = diffSkyfile.skycell_id AND warpSkyfile.good_frac > 0.5 GROUP BY diffSkyfile.skycell_id ORDER BY good_stuff DESC LIMIT 20;
Wednesday : 2019.05.22
- MEH: manually adding OSS.WS.nightlyscience to ippps2/stdscience now that nightly finished
misaligned visits by JRF
The first chunk on PS2 has similar problems to the previous evening, where there is a large offset between different visits. This can be seen in the diff images themselves:
mysql -hscidbm -uX -pX gpc2 -B -e ' select path_base,diffSkyfile.skycell_id,sources from diffRun join diffSkyfile using (diff_id) join diffInputSkyfile using (diff_id) join warpRun on (diffInputSkyfile.warp1=warpRun.warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_name="o8625h0167o" and quality=0 and diff_mode=1 group by diffSkyfile.skycell_id limit 5; ' | grep -v path_base | awk '{print "neb-ls -p "$1".fits"}' | bash | xargs ~/bin/ds9 -zscale -prefs nancolor orange -geometry 900x800 -frame last -zoom 0.3 -match frame -tile grid gap 3
or by querying the reported RA and Dec positions in the in the smf files:
mysql -hippdb08 -uX -pX gpc2 -B -e ' select path_base,exp_name,object,comment from camRun join camProcessedExp using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where dateobs like "2019-05-22%" and ( comment like "%visit 1" or comment like "%visit 2" ) order by object; ' | grep -v path_base | awk '{print "neb-ls -p "$1".smf"}' | tcsh | fields OBJECT AST_R0 AST_D0
The above gets the path_base for visit pairs (objects) and then finds the smf file to get the position data.
We want to investigate the boresite info to assess how poor the PS2 pointing is. Things we should consider are:
- alt az position
- time of observation
The query:
mysql -hippdb08 -uX -pX gpc2 -B -e ' select path_base,exp_name,object,comment from camRun join camProcessedExp using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where dateobs like "2019-05-22%" and ( comment like "%visit 1" or comment like "%visit 2" ) order by object; ' | grep -v path_base | awk '{print "neb-ls -p "$1".smf"}' | tcsh | fields OBJECT AST_R0 AST_D0
This provides us with pairs of visits (one on each line), so that we can compare the RA and DEC positions.
TODO: Just grab all those pointings, and plot them to see how they change across the night, for each night (just an MJD would be simple too)
warp_stack_diff macro by JRF
At the moment, the warpstack diffs processed by PS2 are not the best quality thanks to the poor masking and general unknown qualities of PS2. Coupled with this is the fact that the WWdiff and WSdiff processing is all done by stdscience for PS2 (unlike PS1 where it is separate). This means that stdscience could get clogged up with WSdiffs that keep failing and not process the WWdiffs as fast (the latter are critical for MOPS).
What we plan to do is limit the time when warpstacks are run. This is determined by a macro living in file: ~ippps2/stdscience/input
Looking at the ippitc version, there are the following macros:
## to change set the time UT
macro warpstack.trange.all
task survey.warpstack.diff
trange -reset
end
end
macro warpstack.trange.day
task survey.warpstack.diff
trange -reset
trange 17:00:00 23:59:00
trange 00:00:00 05:00:00
end
end
One for setting WS to be active all day, the other to only be during the daytime (note that the times are UTC time).
These can be called during the setup, under the 'macro setup' section at the top of the input file, to automatically set when they should be used:
#blah blah stuff ## modify the trange for wsdiff: ## trange.all -- run day and night; trange.day -- run only in the day warpstack.trange.all #warpstack.trange.day
You can only use one at a time (using more will overwrite the other). The above example shows the case where we are observing all the time.
The macros are already in the input file for ippps2, but if they were not we could just copy the macros into the file and add the calling of them to the setup macro.
The most important part is that we commented out 'warpstack.trange.all', and uncommented 'warpstack.trange.day' in the setup. Then WSdiffs will only be queued during the day.
Euclid twilight observations by JRF
Angie has fixed the Euclid comment strings to be more consistent with other observations, for example the before and after:
| BRIGHT | 120 | Euclid twilight M8592 y; pos 10h offset from ps1_19_4388 Day 17 | | BRIGHT | 120 | Euclid.twilight.E8621.10a.z ps1_19_5863 Day 0 |
essentially the chunk names are now similar to the quad names used by MOPS and other parts of our processing.
However, this means that they are no longer being picked up by the nightly_science processing. So we shall modify the nighty_science.config to fix this.
nightly_science.config - contains the config for nightly_science.pro - loads it nightly_science.pl - runs it all
The config for PS1 lives here:
~ippitc/src/ipp-20170121/ippconfig/recipes/nightly_science.config
The config for handling the Euclid, bright, z-band observations during full moon has 'Euclid.r%' in their comments, while the ones in our case belong to different settings where it is looking for 'Euclid ' - note the space.
The config looks like this:
TARGETS METADATA NAME STR Bright DISTRIBUTION STR NODIST TESS STR RINGS.V3 OBSMODE STR BRIGHT STACKABLE BOOL FALSE DIFFABLE BOOL TRUE OFFNIGHT_DIFFS BOOL FALSE DESPERATE_DIFFS BOOL TRUE REDUCTION STR EUCLID_DEFAULT COMMENT STR Euclid.r% # DIFF S16 7 END TARGETS METADATA NAME STR BrightTwi DISTRIBUTION STR NODIST TESS STR RINGS.V3 OBSMODE STR BRIGHT STACKABLE BOOL FALSE DIFFABLE BOOL FALSE OFFNIGHT_DIFFS BOOL FALSE DESPERATE_DIFFS BOOL FALSE REDUCTION STR EUCLID_DEFAULT COMMENT STR Euclid END
So, now we can change the config to handle the new comment format:
TARGETS METADATA NAME STR BrightTwi DISTRIBUTION STR NODIST TESS STR RINGS.V3 OBSMODE STR BRIGHT STACKABLE BOOL FALSE DIFFABLE BOOL FALSE OFFNIGHT_DIFFS BOOL FALSE DESPERATE_DIFFS BOOL FALSE REDUCTION STR EUCLID_DEFAULT COMMENT STR Euclid.twi% END
Save this, and let's check that the built version of the config file is similar (bar this change). The reason to check is that we don't want to erase any genuine changes in the build version -- remember this is a little cheeky just putting the file there, but it's because of how tied the config is to the build.
The build file is here:
~ippitc/psconfig/ipp-20170121.lin64/share/ippconfig/recipes/nightly_science.config
Diff them:
diff nightly_science.config ~/src/ipp-20170121/ippconfig/recipes/nightly_science.config
If the only change is they one you're making great. In this case it was, so copy the src version across to the build version.
queuing Euclid twilight observations by JRF
We may have changed the config to recognise the new Euclid format, but there have already been observations taken under the new format that have not been processed. Therefore, we need to send them to processing.
To do this we are going to do something similar to running the test quad:
An example of the test quad (as a reminder):
chiptool -dbname gpc2 -definebyquery -set_label MOPS.ps2testquad3.20190516ops -set_workdir neb://@HOST@.0/gpc2/MOPS.ps2testquad3.20190516ops -set_data_group MOPS.ps2testquad3.20190516ops -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -simple -exp_name o8507h0371o -pretend
For processing these exposures the label, work_dir, data_group, dist_group, and reduction need to be changed (and expname of course).
chiptool -dbname gpc1 -definebyquery -set_label BrightTwi.nightlyscience -set_workdir neb://@HOST@.0/gpc1/BrightTwi.nt/2019/05/17/ -set_data_group BrightTwi.20190518 -set_dist_group NODIST -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction EUCLID_DEFAULT -simple -exp_name -pretend
So previously, if we look at something that has been done:
select * from chipRun WHERE data_group LIKE "BrightTwi.201905%";
TODO: script this up
wait state cleaning by JRF
Have a look to see what is still waiting:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait" GROUP BY data_group) as s ORDER by waitct ASC;'
Choose a data group, and chuck it into the grinder:
mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT chip_id,data_group,label FROM chipRun WHERE state like "wait" and data_group like "LAP.ThreePi.2014%" and label not like "update.LAP.PV3" LIMIT 5000;' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id "$1}' | tcsh
Just replace the data_group with whichever you want to clean. Note that this one is limited to 5000 so as not to overload the system and that the label is not "update.LAP.PV3" so that we do not mess with the updates we're doing too.
Have a check that they're doing ok:
SELECT count(*), label FROM chipRun where state LIKE "goto_cleaned" GROUP BY label;
OTA03 issues by JRF
OTA03 decided it had had enough and went crazy, causing some major stalling during the night in the chip stage (jobs stalling in pantasks for well over 2000 seconds).
To fix this the jobs need to be killed and moved to quality 42 (since this is manual change).
Fortunately, there is handy script that MEH has that does this. Check out:
~mhuber/bin/sshkill_ppimage_dtime_host_ps1.csh
The gist of it is that is looks at the long running chip jobs in pantasks, then ssh's to each machine for the jobs and runs another kill script to stop ppimage.bat. After this, it waits 20 seconds for them to clear and goes and sets the jobs -- which are all in state 'new' and fault 4 -- to fault 0 and quality 42.
I've temporarily added this to the crontab on ippc30 (since this is where a bunch of other ippitc crontabs are). I'm waiting to see if it works
