Changes between Version 11 and Version 12 of PS1_IPP_Czarlog_20190527

Timestamp:

May 21, 2019, 12:46:02 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190527

@@ -75 +75 @@
 
 === Monday : 2019.05.20 ===
+
+
+JRF: First night with PS2 for a long time, but processing progressed pretty smoothly for the most part. However, MEH saw that the pointing was poor so suggested removing the OSS.WS.nightlyscience label so that their diff processing doesn't clog up the system. This is because for PS2 the WWdiffs and the WSdiffs share the same pantasks (for PS1 they are separate).
+
   
 === Tuesday : 2019.05.21 ===
+
+''' WSdiff processing ''' by JRF
+
+Added the OSS.WS.nightlyscience label back into PS2 processing early in the morning, after everything else had finished. The majority of diffs cleared no problem. There was one that I thought was stuck:
+{{{
+0   ippx021    BUSY   9195.68 0.0.1.8bd5  0 diff_skycell.pl --threads @MAX_THREADS@ --diff_id 50495 --skycell_id skycell.1370.020 --diff_skyfile_id 2720978 --outroot neb://ipp137.1/gpc2/OSS.WS.nightlyscience/2019/05/20/RINGS.V3/skycell.1370.020/RINGS.V3.skycell.1370.020.WS.dif.50495 --redirect-output --run-state new --reduction SWEETSPOT_WS --dbname gpc2 --verbose 
+}}}
+It's been there for a good 2 and half hours.
+
+I found some info about WSdiffs being stuck in an older log:
+{{{
+past log : http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20180618
+-- turn diff.revert.off, kill ppSub job, wait to fault and manually set quality 42 
+difftool -dbname gpc2 -updatediffskyfile -fault 0 -set_quality 42 -diff_id 50495 -skycell_id skycell.1370.020
+}}}
+The job actually finished while I was looking this up... so nevermind.
+
+To ensure that we don't have to manually intervene during the night, by removing the label again, I've changed the {{{~ippps2/stdscience/input}}} file so that all WS labels are not automatically added on restart. This means that the label can be re-addedto pantasks at our leisure in the morning.
+
+
+''' plots ''' by JRF
+
+czarplot.pl can be run manually to generate some plots, for example:
+{{{
+cd ~ippitc/src/ippMonitor/czartool ; ./czarplot.pl -rt -l OSS.WS.nightlyscience -p '12 hour'
+}}}
+This one gives you a **r**ate plot and a cumulative **t**ime of the past 12 hours for the specific label. This is useful as at present automatic generation of these plots is disabled (they were barely ever used and greatly slowed down IPPmonitor).
+
+
+''' Checking out diffs '''
+
+To view the quality of the resulting diffs in ds9 try the following query (from MEH):
+{{{
+mysql -hscidbm -uippuser -pippuser gpc2 -B -e ' select path_base,diffSkyfile.skycell_id,sources from diffRun join diffSkyfile using (diff_id) join diffInputSkyfile using (diff_id) join warpRun on (diffInputSkyfile.warp1=warpRun.warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where  exp_name="o8507h0371o" and quality=0 and diff_mode=1 group by diffSkyfile.skycell_id limit 5; ' | grep -v path_base | awk '{print "neb-ls -p "$1".fits"}' | bash | xargs ~/bin/ds9 -zscale -prefs nancolor orange -geometry 900x800  -frame last -zoom 0.3 -match frame -tile grid gap 3
+}}}
+Make sure to change the exposure name to the one you desire, and the limit to how many you want to view at once (also note the ds9 path)
+
+For last night, one of the first visits from the first chunk:
+o8624h0111o
+one of the second vists of the second chunk:
+o8624h0181o
+
+The first is a very poor diff where there is a huge overlap in the diffs -> suggesting that the pointing is off. This is also evident when looking at the number of sources detected by the diffs, where we see >10000 deteections; normally less than 1000 are detected.
+
+An example query that will show you how many sources were detected and the fraction of good pixels in the warp and diff stage. Looking at a percentage difference in the good_frac gives an indication on the quality: larger -ve frac -> a worse diff
+{{{
+SELECT 
+    diffSkyfile.skycell_id, 
+    warpSkyfile.skycell_id, 
+    sources, 
+    diffSkyfile.good_frac difffrac, 
+    warpSkyfile.good_frac warpfrac, 
+    diffSkyfile.good_frac - warpSkyfile.good_frac / warpSkyfile.good_frac as good_stuff 
+FROM
+    diffRun 
+    JOIN diffSkyfile USING (diff_id) 
+    JOIN diffInputSkyfile USING (diff_id) 
+    JOIN warpRun on (diffInputSkyfile.warp1 = warpRun.warp_id) 
+    JOIN fakeRun USING (fake_id) 
+    JOIN camRun USING (cam_id) 
+    JOIN chipRun USING (chip_id) 
+    JOIN rawExp USING (exp_id) 
+    JOIN warpSkyfile USING (warp_id) 
+WHERE 
+    exp_name="o8507h0371o" and diff_mode=1 
+    AND 
+    warpSkyfile.skycell_id = diffSkyfile.skycell_id 
+    AND 
+    warpSkyfile.good_frac > 0.5
+GROUP BY 
+    diffSkyfile.skycell_id 
+ORDER BY 
+    good_stuff DESC
+LIMIT 20;
+}}}
+
+As a on liner:
+{{{
+SELECT diffSkyfile.skycell_id, warpSkyfile.skycell_id, sources, diffSkyfile.good_frac difffrac, warpSkyfile.good_frac warpfrac, diffSkyfile.good_frac - warpSkyfile.good_frac / warpSkyfile.good_frac as good_stuff FROM diffRun JOIN diffSkyfile USING (diff_id) JOIN diffInputSkyfile USING (diff_id) JOIN warpRun on (diffInputSkyfile.warp1 = warpRun.warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) JOIN warpSkyfile USING (warp_id) WHERE exp_name="o8507h0371o" and diff_mode=1 AND warpSkyfile.skycell_id = diffSkyfile.skycell_id AND warpSkyfile.good_frac > 0.5 GROUP BY diffSkyfile.skycell_id ORDER BY good_stuff DESC LIMIT 20;
+}}}
 
 === Wednesday : 2019.05.22 ===