Changes between Version 7 and Version 8 of SAS9DataIntegrity

Timestamp:

Oct 5, 2012, 3:03:23 PM (14 years ago)

Author:

heather

Comment:

--

SAS9DataIntegrity

@@ -149 +149 @@
 '''This integrity fails on all levels.  ng/nr/ni/nz/ny are all less than what we get from the sums of stacks and detections for those filters, and ndetection != ng + nr + ni + nz + ny.'''  These are set in the dvo. Heather suspects this is a dvo problem. 
 
-
+=== dvo integrity checks === 
+
+This is all Gene's checks, using /data/ipp005.0/gpc1/catdirs/SAS/20120901
+
+{{{
+dvo: skyregion 333 335 -1 1
+dvo: avextract g:nphot g:ncode g:flags -v
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.00.cpt (0 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.01.cpt (1 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.02.cpt (2 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.03.cpt (3 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.04.cpt (4 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.05.cpt (5 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.06.cpt (6 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.07.cpt (7 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.08.cpt (8 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.09.cpt (9 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.10.cpt (10 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.11.cpt (11 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.12.cpt (12 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.13.cpt (13 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.14.cpt (14 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.15.cpt (15 of 16)
+dvo: mextract nmeas photcode -v dbflags
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.00.cpt (0 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.01.cpt (1 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.02.cpt (2 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.03.cpt (3 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.04.cpt (4 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.05.cpt (5 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.06.cpt (6 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/n0000/0559.07.cpt (7 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.08.cpt (8 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.09.cpt (9 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.10.cpt (10 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.11.cpt (11 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.12.cpt (12 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.13.cpt (13 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.14.cpt (14 of 16)
+trying /data/ipp005.0/gpc1/catdirs/SAS/20120901/s0000/5225.15.cpt (15 of 16)
+dvo: vstat g:ncode
+mean: 0.898337, stdev: 2.45667, min: 0, max: 26, median: 0, mode: 26, Npts: 764449
+dvo: echo $TOTAL
+686733
+dvo: echo {$TOTAL - 686619}
+114
+dvo: subset photcode_g_stk = photcode where (photcode == 11000)
+dvo: echo photcode_g_stk[]
+177372
+dvo: echo {photcode_g_stk[] - 176180}
+1192
+dvo: echo {(photcode_g_stk[] - 176180) / photcode_g_stk[]}
+0.00672033917417
+dvo: subset photcode_g_exp = photcode where ((photcode > 10000) && (photcode < 10100))
+dvo: echo photcode_g_exp[]
+521635
+dvo: echo {(photcode_g_exp[] - 454940)/photcode_g_exp[] }
+0.12785760158
+}}}
+
+
+==== summary ====
+
+for same area of sky:
+
+|| test   || psps  || dvo  || comment  ||
+|| sum(ng) in object table  || 686819  ||  686733 || diff = 114   not too crazy, may be due to rounding errors ||
+|| sum( g detections ) in stack ||  176180 || 177372 || diff = 1192  ||
+|| sum( g detections ) in p2     || 454940  || 521635  || diff = 521635  --- what? ||
+
+integrity is not happy...  need to investigate if there are filters in ipptopsps to block things. what are they?
+
+ahh!  does integrity hold up in dvo for n detections in objects  = n detections in stacks + n detections in p2
+
+NO:   686733 != 177372 + 521635   => 686733 != 699007
+
+however we don't expect this because of overlapping skycells: dvo counts all of them and the flag to pick the primary skycell detection is not used yet (sa9 just missed it by a few days). 
+
+