Tracing PSPS Detections back to the source data

Bill Sweeney

back up to ippToPspsSAS39BillsNotes

While triaging SAS 39 it became clear that connecting the data in the PSPS back to the source in the IPP and DVO was non-trivial. We need to do this when debugging the completeness and integrity of the ippToPsps process.

The first thing I attacked was comparing the data in the Detection Table (single frame detections) with the original psphot detections which are located in a camera stage '.smf' file.

The smf file contains 2 extensions for each OTA in the exposure. The extension that we are interested in is the '.psf' extension. The other contains the results of the detection efficiency measurements.

All of the programs mentioned here have been added to the directory tools/bills in the ipp trunk.

Creating a concatenated file containing all detections in one table

The multi-extension structure is kind of hard to work with for our purposes. So I wrote a program that concatenates all of the results into a single table. In order to make the detections from the different OTAs distinguishable a column called IMAGE_ID is added. This column is set to the value (exp_id * 100 + IPP_IDET) which matches the format of the column Detection.imageID in PSPS.

The program is a perl scrip that uses standard IPP programs and the ​STILTS library. Several of the examples here also use programs from ​FTOOLS in particular ftlist but the scripts do not use anything from FTOOLS.

Consider the SAS2 exposure 356801. The smf file may be found with the following command.

% camtool -processedexp -exp_id 356801 -label SAS.20141118 | grep path_base | awk '{print $3".smf"}'
neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf

The progarm concatsmf may be used to extract the detections into a file containing a single table.

% concatsmf neb://any/SAS2/SAS.20141118/o5745g0427o.356801/o5745g0427o.356801.cm.1218577.smf o5745g0427o.356801.cm.1218577.smf.tcat
/data/ipp079.0/nebulous/8f/82/5477976254.SAS2:SAS.20141118:o5745g0427o.356801:o5745g0427o.356801.cm.1218577.smf

% ftlist o5745g0427o.356801.cm.1218577.smf.tcat h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3627)                    
HDU 2   o5745g0427o.356801.cm.1218577.smf BinTable    58 cols x 58059 rows    

NOTE: concatsmf assumes that the basename of the smf file has begins with the string $exp_name.$exp_id.cm.$cam_id (but only exp_id is currently used).

So we have 58059 detections from this exposure.

Finding the PSPS Detections for an exposure

Next we go to PSPS and find Detections from this exposure with the query

select imageID/100 frameID, Detection.*  INTO mydb.[o5745g0427o_356801] FROM Detection where imageID/100=356801

The frameID is not necessary, I added it because I thought that it might be useful.

After downloading the results of this query we find that there are only 58014 detections. 45 apparently did not make their way into PSPS.

Did they get lost loading into DVO or in the ippToPsps process?

Let's try and figure out what's missing.

Matching PSPS Detections to SMF Detections

matchIppToPsps is another STILTS based perl script. It matches detections in two files. The first is the output of our psps query. The second the concatenated smf detections file.

To find the matches use

% matchIppToPsps o5745g0427o_356801.fit o5745g0427o.356801.cm.1218577.smf.tcat o5745g0427o.356801.joined.fit

And to find the missing detections use

% matchIppToPsps o5745g0427o_356801.fit o5745g0427o.356801.cm.1218577.smf.tcat o5745g0427o.356801.missing.fit --join=2not1

% ftlist o5745g0427o.356801.missing.fit h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3597)                    
HDU 2   Joined             BinTable    58 cols x 45 rows            

45 missing detections. The next step is to trace the detections in PSPS back to the DVO catalog.

Using a P2 Batch file instead of the output of a psps query

The new tool concatp2batch may be used to take a P2 ippToPsps batch and combine all of the Detections extensions into a single file. The resulting file may be used as input to matchIppToPSPS

EXAMPLE
concatp2batch 00356775.FITS 00356775.detections.fits
matchIppToPsps 00356775.detections.fits o5745g0401o.356775.cm.1218547.fits matched.fits
matchIppToPsps 00356775.detections.fits o5745g0401o.356775.cm.1218547.fits unmatched.fits --join 2not1

Finding particular detections in the DVO catalog files

Let me say up front that there are undoubtedly ways to use DVO commands to accomplish this, but I found it quicker and more flexible to use STILTS.

The file o5745g0427o.356801.missing.fit contains a list of the psphot detections that were not matched to the PSPS detections. Perusing these detections I choose one not near the edge of the frame: IMAGE_ID 35680124 (from OTA XY24), DET_ID 140.

The program detectToMeasure may be used to look for the missing detections in the DVO catalog files.

If we just want to find which catalog and dvo "image" contains this detection we can use

(ipp079:~/work/psps) bills% detectToMeasure --ra 333.719989076316 --dec -4.47253338047446 --exp_id 356801 --class_id XY24
RA            DEC      NAME     CAT_ID EXTERN_ID IMAGE_ID catalogPrefix
333.719989076316 -4.47253338047446 s0000/5228.01 NULL 75160686 4638 /data/ipp079.0/gpc1/catdirs/SAS/SAS39.V1/s0000/5228.01

To find the missing detection add --det_id 140

detectToMeasure --ra 333.719989076316 --dec -4.47253338047446 --exp_id 356801 --class_id XY24 --det_id 140 --output det140.fit

Sure enough it's in there.
% ftlist det140.fit h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3659)                    
HDU 2   DVO_MEASURE_PS1_V5 BinTable    63 cols x 1 rows

If the --list-others option is used it will extract all detections for the dvo object for the detection in question. if --list-others --this-image is used it finds detections from this image which are associated with the same object.

% detectToMeasure --ra 333.719989076316 --dec -4.47253338047446 --exp_id 356801 --class_id XY24 --det_id 140 --output thisobject.fit --this-image --list-others
RA            DEC      NAME     CAT_ID EXTERN_ID IMAGE_ID catalogPrefix
333.719989076316 -4.47253338047446 s0000/5228.01 NULL 75160686 4638 /data/ipp079.0/gpc1/catdirs/SAS/SAS39.V1/s0000/5228.01

% ftlist thisobject.fit h

        Name               Type       Dimensions
        ----               ----       ----------
HDU 1   Primary Array      Image      Int1(3659)                    
HDU 2   DVO_MEASURE_PS1_V5 BinTable    63 cols x 2 rows  

We have two detections from the same DVO object and image. Looks like ippToPsps doesn't handle objects with multiple detections in the same image correctly.

It turns out that there are also missing detections where there are not Detections from the associated dvo object in PSPS. It may be that these are single detection objects near the edge of the field of view that were skipped for some reason. More analysis to come.

In all cases though the missing detections are within 2 arc seconds of a detection that is in PSPS.

Review of the missing detections

7 of the 45 have double detections for the same object. The other 38 are single detections. I queried a couple looking for an Object with ippObjectID = (CATDIR<<32 | OBJ_ID) in psps. No luck. Looks like this style of missing detections had their objects dropped for some reason.

Comparison of values in the Detection table with the smf

The values appear to be transferred properly per the current schema. Fluxes in Detection are cmfFlux/expTime as defined.