IPP to PSPS interface: ippToPsps

Links

​Datastore test area for PSPS

Introduction

ippToPsps is the interface between IPP and PSPS. In short, ippToPsps creates FITS files from data found in IPP smf and cmf files, as well as relevant DVO databases, then publishes them to a datastore in the form of jobs. On the PSPS side, the DXLayer polls the datastore, collects jobs when they become available, then converts the contents to csv files before sending them on to special MSSQL loader software, which inserts them into the PSPS database. Ultimately there will be feedback from PSPS regarding errors in the received data, to which ippToPsps will need to respond.

It is intended that the binary tables in the FITS files generated by ippToPsps will match the PSPS database schemas perfectly, the consequence being that any alterations to the PSPS database schema will only affect ippToPsps code, and not the DXLayer. A certain amount of data validation will be performed by ippToPsps before publication.

Architecture

ippToPsps

ippToPsps is a C program within the IPP build. When given the correct arguments it will generate a single FITS for the specified product (above). The program is run from a perl script, which itself generates a list of exposure IDs based on arguments provided by the user (label etc). An instance of ippToPsps is run per exposure ID.

DXLayer

...

Loader

...

Products

ippToPsps creates three products.

• Detections
• Diffs
• Stacks

Detections

Input

Input is one {{smf}} file for a given exposure and an associated DVO database from which to retrieve object, and other, IDs.

Output

A FITS output file is generated with the following extensions:

1 Primary extension

1 FrameMeta extension

1 ImageMeta extension per chip

1 Detection extension per chip

So, 122 extensions in all, included the obligatory primary header.

Validation

Some validation is performed to aid the data's insertion into the PSPS database.

Diffs

Input

A set of cmf files, one for each skycell covered by a particular exposure.

Output

A FITS output file is generated with the following extensions:

...

Validation

Unresolved fields

Below are tables detailing which fields in the PSPS FITS files are still not populated by ippToPsps.

Unresolved fields for camera stage ('detections')

FrameMeta

PSPS field	PSPS type	PSPS Description	IPP source	Comments
surveyID	TINYINT	survey flag identifier		e.g. 3PI, M31, MD, STS - need common enum
filterID	TINYINT	filter identifier		IPP uses string (g.00000). Need enum.
cameraID	SMALLINT	camera identifier		set to 1?
cameraConfigID	SMALLINT	camera configuration identifier		set to 0?
telescopeID	SMALLINT	telescope identifier		set to 1?
analysisVer	SMALLINT	version number of analysis
p1Recip	SMALLINT	IPP phase 1 recipe ID
p2Recip	SMALLINT	IPP phase 2 recipe ID
p3Recip	SMALLINT	IPP phase 3 recipe ID
nP2Images	SMALLINT	number of OTA/CCD images in this P2 exposure		always 60?
astroScat	REAL	global astrometric scatter		have values per chip, but global?
photoScat	REAL	global photometric scatter
nAstRef	INT	number of astrometric reference sources		have values per chip, but global?
nPhoRef	INT	number of photometric reference sources		have values per chip, but global?
polyOrder	TINYINT	polynomial order of astrometry fit	smf.phu.NPLYTERM?
calibModNum	SMALLINT	calibration modification number

ImageMeta

PSPS field	PSPS type	PSPS Description	IPP source	Comments
filterID	TINYINT	filter ID		IPP uses string (g.00000). Need enum.
bias	REAL	detector bias level		missing from smf
biasScat	REAL	scatter in bias level		missing from smf
photoScat	REAL	photometric scatter for chip	smf.phu.ZPT_OBS?
nPhoRef	INT	number of photometric reference sources	smf.chip.NPSFSTAR? smf.chip.NSTARS?
psfFwhm	REAL	PSF full width at half maximum	smf.chip.FWHM_MAJ?
psfModelID	INT	PSF model identifier		missing from smf
psfExtra1	REAL	PSF parameters		missing from smf
psfExtra2	REAL	PSF parameters		missing from smf
detectorID	SMALLINT	identifier for actual CCD chip		IPP uses string (CCID58-1-02a2). Need enum.
qaFlags	INT	Q/A flags		PSPS needs two flag fields
detrend1	BIGINT	identifier of detrend image 1		IPP uses strings, PSPS wants INTs
detrend2	BIGINT	identifier of detrend image 2		IPP uses strings, PSPS wants INTs
detrend3	BIGINT	identifier of detrend image 3		IPP uses strings, PSPS wants INTs
detrend4	BIGINT	identifier of detrend image 4		IPP uses strings, PSPS wants INTs
detrend5	BIGINT	identifier of detrend image 5		IPP uses strings, PSPS wants INTs
detrend6	BIGINT	identifier of detrend image 6		IPP uses strings, PSPS wants INTs
detrend7	BIGINT	identifier of detrend image 7		IPP uses strings, PSPS wants INTs
detrend8	BIGINT	identifier of detrend image 8		IPP uses strings, PSPS wants INTs
photoZero	REAL	local derived photometric zero point		smf.phu.ZPT_OBS?
polyOrder	TINYINT	polynomial order of astrometry fit	smf.chip.NPLYTERM?
calibModNum	SMALLINT	calibration modification number		set to 0?

Detection

PSPS field	PSPS type	PSPS Description	IPP source	Comments
filterID	TINYINT	filter ID: g=1, r=2, i=3, z=4, y=5, w=6, ...		IPP uses string (g.00000). Need enum.
surveyID	TINYINT	survey flag identifier		e.g. 3PI, M31, MD, STS - need common enum
psfLikelihood	REAL	PSF likelihood	smf.psf.PSF_QF?
psfCf	REAL	PSF coverage factor	smf.psf.PSF_QF?
infoFlag	INT	bitmask indicating provenance information
activeFlag	TINYINT	indicates whether this detection/orphan is still a detection/orphan
historyModNum	SMALLINT	modification number in the O-D association history		set to 0?

Unresolved fields for diff stage ('objects')

StackMeta

PSPS field	PSPS type	PSPS Description	IPP source	Comments
stackMetaID	BIGINT	stack identifier
skycellID	INT	skycell region identifier		missing from cmf files
surveyID	TINYINT	survey flag identifier		e.g. 3PI, M31, MD, STS - need common enum
photoCalID	INT	photometry code numerical id
filterID	TINYINT	filter identifier		IPP uses string (g.00000). Need enum.
stackTypeID	TINYINT	stack type identifier
refImageID	BIGINT	identifier of image used as reference for analysis
subtrImageID	BIGINT	identifier of image subtracted to generate difference image
analVer	SMALLINT	analysis version index
nP2Images	SMALLINT	number of P2 images contributing to this cell
astroScat	FLOAT	0.astrometric scatter for chip
photoScat	FLOAT	0.photometric scatter for chip
nAstroRef	INT	number of astrometric reference sources
nPhoRef	INT	number of photometric reference sources
psfFwhm	FLOAT	PSF full width at half maximum
psfmodelID	SMALLINT	PSF model identifier
psfExtra1	FLOAT	PSF parameters
psfExtra2	FLOAT	PSF parameters
photoZero	FLOAT	local derived photometric zero point
photoColor	FLOAT	local derived photometric color term
polyOrder	TINYINT	polynomial order of astrometry fit
pca1x3y0	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x2y1	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x1y2	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x0y3	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x2y0	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x1y1	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x0y2	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x3y0	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x2y1	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x1y2	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x0y3	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x2y0	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x1y1	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x0y2	FLOAT	polynomial coefficients for the astrometric fit		not available
calibModNum	SMALLINT	calibration modification number
dataRelease	TINYINT	Data release

StackToImage

PSPS field	PSPS type	PSPS Description	IPP source	Comments
stackMetaID	BIGINT	stack identifier
imageID	BIGINT	hashed exposure-ccdID identifier

StackLowSigDelta/StackHighSigDelta

PSPS field	PSPS type	PSPS Description	IPP source	Comments
partitionKey	BIGINT	partition key
stackDiffID	BIGINT	ODM detection identifier
ippDetectID	BIGINT	detection id generated by IPP
filterID	TINYINT	filter ID: g=1, r=2, i=3, z=4, y=5, w=6,..
psfLikelihood	FLOAT	PSF likelihood
infoFlag	INT	indicator of strange propeties
nFrames	INT	number of frames contributing to source
htmID	BIGINT	HTM index for the detection
zoneID	INT	Zone index
raErr	FLOAT	estimated error in ra
decErr	FLOAT	estimated error in dec
cx	FLOAT	cartesian x
cy	FLOAT	cartesian y
cz	FLOAT	cartesian z
calMag	FLOAT	calibrated magnitude
calMagErr	FLOAT	estimated error in calMag
dataRelease	TINYINT	Data release when this detection was taken

Unresolved fields for stack stage

Not yet...