Tracing PSPS Detections back to the source data

Bill Sweeney

TODO: put up link here

While triaging SAS 39 it became clear that connecting the data in the PSPS back to the source in the IPP and DVO was non-trivial. We need to do this to insure the completeness and integrity of the PSPS database.

The first thing Bill attacked was comparing the data in the Detection Table (single frame detections) with the original psphot detections which are located in a camera stage '.smf' file.

The smf file contains 2 extensions for each OTA in the exposure. The extension that we are interested in is the '.psf' extension. The other contains the results of the detection efficiency measurements.

Creating a concatenated file containing all detections in one table

The multi-extension structure is kind of hard to work with for our purposes. So I wrote a program that concatenates all of the results into a single table. In order to make the detections from the different OTAs distinguishable a column called IMAGE_ID is added. This column is set to the value (exp_id * 100 + IPP_IDET) which matches the format of the column Detection.imageID in PSPS.

The program is a perl scrip that uses standard IPP programs and the ​STILTS library. Several of the examples here also use programs from ​FTOOLS in particular ftlist.

Consider the SAS2 exposure 356801. The smf file may be found with the following command.

% camtool -processedexp -exp_id 356801 -label SAS.20141118 | grep path_base | awk '{print $3".smf"}'
neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf

And then the concatsmf may be used to extract the detections

% concatsmf neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf o5745g0427o.356801.cm.1218577.smf.tcat
/data/ipp079.0/nebulous/8f/82/5477976254.SAS2:SAS.20141118:o5745g0427o.356801:o5745g0427o.356801.cm.1218577.smf

% ftlist o5745g0427o.356801.cm.1218577.smf.tcat h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3627)                    
HDU 2   o5745g0427o.356801.cm.1218577.smf BinTable    58 cols x 58059 rows    

NOTE: concatsmf assumes that the basename of the smf file has begins with the string $exp_name.$exp_id.cm.$cam_id (but only exp_id is currently used).

So we have 58059 detections from this exposure.

Finding the PSPS Detections for an exposure

Next we go to PSPS and find Detections from this exposure with the query

select imageID/100 frameID, Detection.*  INTO mydb.[o5745g0427o_356801] FROM Detection where imageID/100=356801

The frameID is not necessary, I added it because I thought that it might be useful.

After downloading the results of this query we find that there are only 58014 detections. 45 apparently did not make their way into PSPS.

Did they get lost loading into DVO or in the ippToPsps process?

Let's try and figure out what's missing.

Matching PSPS Detections to SMF Detections

matchIppToPsps is another STILTS based perl script. It matches detections in two files. The first is the psps output. The second the concatenated smf detections like we made above.

To find the matches use

% matchIppToPsps o5745g0427o_356801.fit o5745g0427o.356801.cm.1218577.smf.tcat o5745g0427o.356801.joined.fit

And to find the missing detections use
% matchIppToPsps o5745g0427o_356801.fit o5745g0427o.356801.cm.1218577.smf.tcat o5745g0427o.356801.missing.fit --join=2not1

% ftlist o5745g0427o.356801.missing.fit h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3597)                    
HDU 2   Joined             BinTable    58 cols x 45 rows            

The next step is to trace the detections in PSPS back to the DVO catalog.