Tracing PSPS Detections back to the source data

Bill Sweeney

TODO: put up link here

While triaging SAS 39 it became clear that connecting the data in the PSPS back to the source in the IPP and DVO was non-trivial. We need to do this to insure the completeness and integrity of the PSPS database.

The first thing Bill attacked was comparing the data in the Detection Table (single frame detections) with the original psphot detections which are located in a camera stage '.smf' file.

The smf file contains 2 extensions for each OTA in the exposure. The extension that we are interested in is the '.psf' extension. The other contains the results of the detection efficiency measurements.

Creating a concatenated file containing all detections in one table

The multi-extension structure is kind of hard to work with for our purposes. So I wrote a program that concatenates all of the results into a single table. In order to make the detections from the different OTAs distinguishable a column called IMAGE_ID is added. This column is set to the value (exp_id * 100 + IPP_IDET) which matches the format of the column Detection.imageID in PSPS.

The program is a perl scrip that uses standard IPP programs and the ​STILTS library. Several of the examples here also use programs from ​FTOOLS in particular ftlist.

Consider the SAS2 exposure 356801. The smf file may be found with the following command.

% camtool -processedexp -exp_id 356801 -label SAS.20141118 | grep path_base | awk '{print $3".smf"}'
neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf

And then the concatsmf may be used to extract the detections

% concatsmf neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf o5745g0427o.356801.cm.1218577.smf.tcat
/data/ipp079.0/nebulous/8f/82/5477976254.SAS2:SAS.20141118:o5745g0427o.356801:o5745g0427o.356801.cm.1218577.smf

% ftlist o5745g0427o.356801.cm.1218577.smf.tcat h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3627)                    
HDU 2   o5745g0427o.356801.cm.1218577.smf BinTable    58 cols x 58059 rows    

NOTE: concatsmf assumes that the basename of the smf file has begins with the string $exp_name.$exp_id.cm.$cam_id (but only exp_id is currently used).

So we have 58059 detections from this exposure.

Finding the PSPS Detections for an exposure

Next we go to PSPS and find Detections from this exposure with the query

select imageID/100 frameID, Detection.*  INTO mydb.[o5745g0427o_356801] FROM Detection where imageID/100=356801

The frameID is not necessary, I added it because I thought that it might be useful.

After downloading the results of this query we find that there are only 58014 detections. 45 apparently did not make their way into PSPS.

Did they get lost loading into DVO or in the ippToPsps process?

Let's try and figure out what's missing.

Matching PSPS Detections to SMF Detections

matchIppToPsps is another STILTS based perl script. It matches detections in two files. The first is the psps output. The second the concatenated smf detections like we made above.

To find the matches use

% matchIppToPsps o5745g0427o_356801.fit o5745g0427o.356801.cm.1218577.smf.tcat o5745g0427o.356801.joined.fit

And to find the missing detections use
% matchIppToPsps o5745g0427o_356801.fit o5745g0427o.356801.cm.1218577.smf.tcat o5745g0427o.356801.missing.fit --join=2not1

% ftlist o5745g0427o.356801.missing.fit h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3597)                    
HDU 2   Joined             BinTable    58 cols x 45 rows            

The next step is to trace the detections in PSPS back to the DVO catalog.

Finding particular detections in the DVO catalog files

Let me say up front that there are undoubtedly ways to use DVO commands to accomplish this, but I found it quicker and more flexible to use STILTS.

The file o5745g0427o.356801.missing.fit contains a list of the psphot detections that were not matched to the PSPS detections. Perusing these detections I choose one not near the edge of the frame: IMAGE_ID 35680124 (from OTA XY24), DET_ID 140.

The program detectToMeasure may be used to look for the missing detections in the DVO catalog files.

If we just want to find which catalog and dvo "image" contains this detection we can use { (ipp079:~/work/psps) bills% detectToMeasure --ra 333.719989076316 --dec -4.47253338047446 --exp_id 356801 --class_id XY24 RA DEC NAME CAT_ID EXTERN_ID IMAGE_ID catalogPrefix 333.719989076316 -4.47253338047446 s0000/5228.01 NULL 75160686 4638 /data/ipp079.0/gpc1/catdirs/SAS/SAS39.V1/s0000/5228.01 } To find the missing detection add --det_id 140 { detectToMeasure --ra 333.719989076316 --dec -4.47253338047446 --exp_id 356801 --class_id XY24 --det_id 140 --output det140.fit

Sure enough it's in there. % ftlist det140.fit h

Name Type Dimensions ---- ---- ----------

HDU 1 Primary Array Image Int1(3659) HDU 2 DVO_MEASURE_PS1_V5 BinTable 63 cols x 1 rows }

If the --list-others option is used it will extract all detections for the dvo object for the detection in question. if --list-others --this-image is used it finds detections from this image which are associated with the same object.

% detectToMeasure --ra 333.719989076316 --dec -4.47253338047446 --exp_id 356801 --class_id XY24 --det_id 140 --output thisobject.fit --this-image --list-others
RA            DEC      NAME     CAT_ID EXTERN_ID IMAGE_ID catalogPrefix
333.719989076316 -4.47253338047446 s0000/5228.01 NULL 75160686 4638 /data/ipp079.0/gpc1/catdirs/SAS/SAS39.V1/s0000/5228.01

% ftlist thisobject.fit h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3659)                    
HDU 2   DVO_MEASURE_PS1_V5 BinTable    63 cols x 2 rows  

Looks like ippToPsps doesn't handle objects with multiple detections in the same image correctly.