Tracing PSPS Detections back to the source data

Bill Sweeney

TODO: put up link here

While triaging SAS 39 it became clear that connecting the data in the PSPS back to the source in the IPP and DVO was non-trivial. We need to do this to insure the completeness and integrity of the PSPS database.

The first thing Bill attacked was comparing the data in the Detection Table (single frame detections) with the original psphot detections which are located in a camera stage '.smf' file.

The smf file contains 2 extensions for each OTA in the exposure. The extension that we are interested in is the '.psf' extension. The other contains the results of the detection efficiency measurements.

Creating a concatenated file containing all detections in one table

The multi-extension structure is kind of hard to work with for our purposes. So I wrote a program that concatenates all of the results into a single table. In order to make the detections from the different OTAs distinguishable a column called IMAGE_ID is added. This column is set to the value (exp_id * 100 + IPP_IDET) which matches the format of the column Detection.imageID in PSPS.

The program is a perl scrip that uses standard IPP programs and the ​STILTS library.

Consider the SAS2 exposure 356801. The smf file may be found with the following command.

% camtool -processedexp -exp_id 356801 -label SAS.20141118 | grep path_base | awk '{print $3".smf"}'
neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf

And then the concatsmf may be used to extract the detections

% concatsmf neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf o5745g0427o.356801.cm.1218577.smf.tcat
/data/ipp079.0/nebulous/8f/82/5477976254.SAS2:SAS.20141118:o5745g0427o.356801:o5745g0427o.356801.cm.1218577.smf

% ftlist o5745g0427o.356801.cm.1218577.smf.tcat h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3627)                    
HDU 2   o5745g0427o.356801.cm.1218577.smf BinTable    58 cols x 58059 rows    

NOTE: concatsmf assumes that the basename of the smf file has begins format $exp_name.$exp_id.cm.$cam_id (but only exp_id is currently used)

So we have 58059 detections from this exposure.

Finding the PSPS Detections for an exposure

Next we go to PSPS and find Detections from this exposure with the query

select imageID/100 frameID, Detection.*  INTO mydb.[o5745g0427o_356801] FROM Detection where imageID/100=356801

The frameID is not necessary, I added it because I thought that it might be useful.

After downloading the results of this query we find that there are only 58014 detections. 45 apparently did not make their way into PSPS.

Did they get lost loading into DVO or in the ippToPsps process?

Let's figure out what's missing.

The next step is to trace the detections in PSPS back to the DVO catalog.