Tracing PSPS Detections back to the source data

Bill Sweeney

TODO: put up link here

While triaging SAS 39 it became clear that connecting the data in the PSPS back to the source in the IPP and DVO was non-trivial. We need to do this to insure the completeness and integrity of the PSPS database.

The first thing Bill attacked was comparing the data in the Detection Table (single frame detections) with the original psphot detections which are located in a camera stage '.smf' file.

The smf file contains 2 extensions for each OTA in the exposure. The extension that we are interested in is the '.psf' extension. The other contains the results of the detection efficience measurements.

Creating a concatenated file containing all detections in one table

The multi-extension structure is kind of hard to work with for our purposes. So I wrote a program that concatenates all of the results into a single table. In order to make the detections from the different OTAs distinguishable a column called IMAGE_ID is added. This column is set to the value (exp_id * 100 + IPP_IDET) which matches the format of the column Detection.imageID in PSPS.

Consider the SAS2 exposure 356801. The smf file may be found with the following command.

% camtool -processedexp -exp_id 356801 -label SAS.20141118 | grep path_base | awk '{print $3".smf"}'
neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf

And then the concatsmf may be used to extractx the detections

% concatsmf neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf o5745g0427o.356801.cm.1218577.smf.tcat
/data/ipp079.0/nebulous/8f/82/5477976254.SAS2:SAS.20141118:o5745g0427o.356801:o5745g0427o.356801.cm.1218577.smf

% ftlist o5745g0427o.356801.cm.1218577.smf.tcat h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3627)                    
HDU 2   o5745g0427o.356801.cm.1218577.smf BinTable    58 cols x 58059 rows    

So we have 58059 detections from this exposure.

Next we go to PSPS and find Detections from this exposure with the query