| | 1 | Serge populated a database consisting of all the columns in the false detection fits files that were sent to us. |
| | 2 | |
| | 3 | I made some cuts and found some things: |
| | 4 | |
| | 5 | There are 397521 false detections. Of these, there are only 45 different 'flags' that are used: |
| | 6 | |
| | 7 | mysql> select count(*), flags, flags2 from DetectionFeatures group by flags; |
| | 8 | +----------+-----------+--------+ |
| | 9 | | count(*) | flags | flags2 | |
| | 10 | +----------+-----------+--------+ |
| | 11 | | 128 | 2107393 | 128 | |
| | 12 | | 791 | 35653633 | 128 | |
| | 13 | | 1 | 35653645 | 160 | |
| | 14 | | 1 | 35653889 | 224 | |
| | 15 | | 454 | 35661825 | 128 | |
| | 16 | | 24 | 102762501 | 192 | |
| | 17 | | 108851 | 169869313 | 208 | |
| | 18 | | 191 | 169869325 | 208 | |
| | 19 | | 11 | 169869333 | 192 | |
| | 20 | | 1 | 169869389 | 128 | |
| | 21 | | 26004 | 169869441 | 192 | |
| | 22 | | 51 | 169869453 | 192 | |
| | 23 | | 61 | 169869569 | 192 | |
| | 24 | | 53 | 169869577 | 192 | |
| | 25 | | 1 | 169869641 | 128 | |
| | 26 | | 69 | 169869697 | 192 | |
| | 27 | | 23 | 169869705 | 192 | |
| | 28 | | 224897 | 169885697 | 240 | |
| | 29 | | 315 | 169885701 | 224 | |
| | 30 | | 1 | 169885761 | 192 | |
| | 31 | | 444 | 169885953 | 192 | |
| | 32 | | 56 | 169885961 | 192 | |
| | 33 | | 2 | 169885969 | 192 | |
| | 34 | | 8 | 170917889 | 192 | |
| | 35 | | 5 | 170917901 | 240 | |
| | 36 | | 8 | 170918017 | 208 | |
| | 37 | | 1 | 170918029 | 224 | |
| | 38 | | 6 | 203423749 | 128 | |
| | 39 | | 4 | 203423877 | 128 | |
| | 40 | | 2865 | 236978181 | 128 | |
| | 41 | | 315 | 236978245 | 128 | |
| | 42 | | 119 | 236978309 | 128 | |
| | 43 | | 3780 | 236994567 | 192 | |
| | 44 | | 294 | 236994597 | 128 | |
| | 45 | | 4 | 236994631 | 192 | |
| | 46 | | 26 | 238026757 | 144 | |
| | 47 | | 19 | 238026885 | 224 | |
| | 48 | | 2 | 238043143 | 240 | |
| | 49 | | 1 | 241172485 | 192 | |
| | 50 | | 618 | 268437504 | 128 | |
| | 51 | | 3 | 268437760 | 128 | |
| | 52 | | 1 | 268437768 | 128 | |
| | 53 | | 21738 | 402685952 | 128 | |
| | 54 | | 5273 | 402686080 | 128 | |
| | 55 | | 1 | 403734656 | 128 | |
| | 56 | +----------+-----------+--------+ |
| | 57 | 45 rows in set (1.40 sec) |
| | 58 | |
| | 59 | Most lump into only a handful of flags. |
| | 60 | |
| | 61 | I did investigations of flag2 first, because there are fewer of those. I did cuts to remove 64,32,16,8, since those all had to do with mask, cosmicray, burntool, starspikes (ON_SPIKE, ON_STARCORE, ON_BURNTOOL, ON_CONVPOOR), which are not necessary objects we want to count. |
| | 62 | |
| | 63 | I then cut on DEFECT, SATURATED, CR_LIMIT, EXT_LIMIT, and SIZE_SKIPPED on flag. These seemed like reasonable candidates for flags that maybe should be used to filter out bad detections. |
| | 64 | |
| | 65 | I also did cuts to remove NULL mags (is there any mag which should ALWAYS be calculated? I can cut on that instead of the mag I cut) |
| | 66 | |
| | 67 | |
| | 68 | Here's what we have left: |
| | 69 | mysql> select count(*), flags, flags2 from DetectionFeatures where (!(flags2 & 64) and !(flags2 & 32) and !(flags2 & 16) and !(flags2&8) and !(flags & 2048) and !(flags & 4096) and !(flags&8192) and !(flags&16384) and !(flags&268435456)) and !(cal_psf_mag is null) group by flags,flags2; |
| | 70 | +----------+-----------+--------+ |
| | 71 | | count(*) | flags | flags2 | |
| | 72 | +----------+-----------+--------+ |
| | 73 | | 55035 | 169869313 | 128 | |
| | 74 | | 12 | 169869325 | 128 | |
| | 75 | | 1 | 169869389 | 128 | |
| | 76 | | 191 | 169869441 | 128 | |
| | 77 | | 2 | 169869569 | 128 | |
| | 78 | | 8 | 169869577 | 128 | |
| | 79 | | 1 | 169869641 | 128 | |
| | 80 | | 3 | 169869697 | 128 | |
| | 81 | | 6 | 203423749 | 128 | |
| | 82 | | 4 | 203423877 | 128 | |
| | 83 | | 959 | 236978181 | 128 | |
| | 84 | | 252 | 236978245 | 128 | |
| | 85 | | 75 | 236978309 | 128 | |
| | 86 | | 8 | 238026757 | 128 | |
| | 87 | | 1 | 238026885 | 128 | |
| | 88 | +----------+-----------+--------+ |
| | 89 | 15 rows in set (0.24 sec) |
| | 90 | |
| | 91 | for a total of 56558 false detections. |
| | 92 | |
| | 93 | I investigated flag 169869313, which contains most of the false detections. Unfortunately, the flag is not interesting (it did a fit, it calculated a mag, etc), there are no further things to cut out. So, I created region files for the remaining false detections (see /data/ipp009.0/heather/heather), and looked at a few of the ones that had the largest number of detections. I see nothing that stands out: |
| | 94 | |
| | 95 | [[Image(ds9.jpg)]] |
| | 96 | |
| | 97 | The numbers represent the flag setting, and the circles are the remaining false detections. |
| | 98 | |
| | 99 | |
| | 100 | |
| | 101 | |
| | 102 | |
