Changes between Initial Version and Version 1 of DetectionsToMeasurements

Timestamp:

Sep 16, 2015, 2:44:35 PM (11 years ago)

Author:

bills

Comment:

--

DetectionsToMeasurements

@@ +1 @@
+= Tracing PSPS Detections back to the source data =
+Bill Sweeney
+
+TODO: put up link here
+
+While triaging SAS 39 it became clear that connecting the data in the PSPS back to the source in the IPP and DVO was non-trivial. We need to do
+this to insure the completeness and integrity of the PSPS database. 
+
+The first thing Bill attacked was comparing the data in the Detection Table  (single frame detections) with the original psphot detections which
+are located in a camera stage '.smf' file.
+
+The smf file contains 2 extensions for each OTA in the exposure. The extension that we are interested in is the '.psf' extension. The other
+contains the results of the detection efficience measurements.
+
+== Creating a concatenated file containing all detections in one table ==
+The multi-extension structure is kind of hard to work with for our purposes. So I wrote a program that concatenates all of the results into
+a single table. In order to make the detections from the different OTAs distinguishable a column called IMAGE_ID is added. This column is
+set to the value (exp_id * 100 + IPP_IDET) which matches the format of the column Detection.imageID in PSPS. 
+
+Consider the SAS2 exposure 356801. The smf file may be found with the following command.
+{{{
+% camtool -processedexp -exp_id 356801 -label SAS.20141118 | grep path_base | awk '{print $3".smf"}'
+neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf
+}}}
+And then the concatsmf may be used to extractx the detections
+{{{
+% concatsmf neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf o5745g0427o.356801.cm.1218577.smf.tcat
+/data/ipp079.0/nebulous/8f/82/5477976254.SAS2:SAS.20141118:o5745g0427o.356801:o5745g0427o.356801.cm.1218577.smf
+
+% ftlist o5745g0427o.356801.cm.1218577.smf.tcat h
+
+        Name               Type       Dimensions
+        ----               ----       ----------
+HDU 1   Primary Array      Image      Int1(3627)                    
+HDU 2   o5745g0427o.356801.cm.1218577.smf BinTable    58 cols x 58059 rows    
+}}}
+
+So we have 58059 detections from this exposure.
+
+Next we go to PSPS and find Detections from this exposure with the query
+