= IPP to PSPS interface: {{{ippToPsps}}} =

[[PageOutline]]

= Links =

[http://datastore.ipp.ifa.hawaii.edu/PSPS_test Datastore test area for PSPS]

= Introduction =

{{{ippToPsps}}} is the interface between IPP and PSPS. In short, {{{ippToPsps}}} creates FITS files from data found in IPP {{{smf}}} and {{{cmf}}} files, as well as relevant DVO databases, then publishes them to a datastore in the form of ''jobs''. On the PSPS side, the DXLayer polls the datastore, collects jobs when they become available, then converts the contents to {{{csv}}} files before sending them on to special MSSQL loader software, which inserts them into the PSPS database. Ultimately there will be feedback from PSPS regarding errors in the received data, to which {{{ippToPsps}}} will need to respond.

It is intended that the binary tables in the FITS files generated by {{{ippToPsps}}} will match the PSPS database schemas perfectly, the consequence being that any alterations to the PSPS database schema will only affect {{{ippToPsps}}} code, and not the DXLayer. A certain amount of data validation will be performed by {{{ippToPsps}}} before publication.

= Architecture =

== ippToPsps ==

{{{ippToPsps}}} is a C program within the IPP build. When given the correct arguments it will generate a single FITS for the specified product (above). The program is run from a perl script, which itself generates a list of exposure IDs based on arguments provided by the user (label etc). An instance of {{{ippToPsps}}} is run per exposure ID.

== DXLayer ==

...

== Loader ==

...

= Products =

{{{ippToPsps}}} creates three ''products''.

 * '''Detections'''
 * '''Diffs'''
 * '''Stacks'''

== Detections ==

'''Input'''

Input is one IPP camera-stage {{{smf}}} file for a given exposure and an associated DVO database from which to retrieve object, and other, IDs.

'''Output'''

 A FITS output file is generated with the following extensions:

1 '''{{{Primary}}}''' extension[[BR]]

1 '''{{{FrameMeta}}}''' extension[[BR]]

1 '''{{{ImageMeta}}}''' extension per chip[[BR]]

1 '''{{{Detection}}}''' extension per chip[[BR]]

So, 122 extensions in all, included the obligatory primary header.

'''Validation'''

Some validation is performed to aid the data's insertion into the PSPS database.

== Diffs ==

'''Input'''

A set of {{{cmf}}} files, one for each skycell covered by a particular exposure.

'''Output'''

A FITS output file is generated with the following extensions:

...

'''Validation'''

= Unresolved fields =

Below are tables detailing which fields in the PSPS FITS files are still not populated by {{{ippToPsps}}}.

== Unresolved fields for camera stage ('detections') ==

'''{{{FrameMeta}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||    analysisVer ||            SMALLINT ||  version number of analysis  || || pending ||
||    p1Recip     ||           SMALLINT ||  IPP phase 1 recipe ID  || || pending ||
||    p2Recip     ||          SMALLINT ||  IPP phase 2 recipe ID  || || pending ||
||    p3Recip     ||         SMALLINT ||  IPP phase 3 recipe ID  || || pending ||
||    nAstRef    ||             INT      ||  number of astrometric reference sources  || || have values per chip, but global? ||
||    nPhoRef    ||             INT      ||  number of photometric reference sources   || || have values per chip, but global? ||

'''{{{ImageMeta}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
|| bias      ||  REAL      || detector bias level || || pending ||
|| biasScat  || REAL      || scatter in bias level || || pending||
|| photoScat  ||  REAL      || photometric scatter for chip || -999 for now || ||
|| nPhoRef    ||   INT     || number of photometric reference sources || -999 for now || ||
|| psfModelID ||  INT       || PSF model identifier||  || pending ||
|| qaFlags     || INT       || Q/A flags || || Get from DVO... ||
|| detrend1    || BIGINT    || identifier of detrend image 1 || || IPP uses strings, PSPS wants INTs ||
|| detrend2    || BIGINT    || identifier of detrend image 2 || || IPP uses strings, PSPS wants INTs||
|| detrend3    || BIGINT    || identifier of detrend image 3 || ||IPP uses strings, PSPS wants INTs ||
|| detrend4     || BIGINT    || identifier of detrend image 4 || ||IPP uses strings, PSPS wants INTs ||
|| detrend5    || BIGINT    || identifier of detrend image 5 || ||IPP uses strings, PSPS wants INTs ||
|| detrend6    || BIGINT    || identifier of detrend image 6 || ||IPP uses strings, PSPS wants INTs ||
|| detrend7    || BIGINT    || identifier of detrend image 7 || || IPP uses strings, PSPS wants INTs||
|| detrend8    || BIGINT    || identifier of detrend image 8 || ||IPP uses strings, PSPS wants INTs ||
|| polyOrder   || TINYINT   || polynomial order of astrometry fit || smf.chip.NPLYTERM? || ||



'''{{{Detection}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||

||    psfLikelihood       ||        REAL || PSF likelihood  || ??? || ||
||    infoFlag||    INT   || bitmask indicating provenance information      || || ||
||    activeFlag||  TINYINT  || indicates whether this detection/orphan is still a detection/orphan || || ||


== Unresolved fields for diff stage ('objects') ==

'''{{{StackMeta}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||stackMetaID||BIGINT||stack identifier|| || ||
||skycellID||INT||skycell region identifier || || missing from cmf files ||
||surveyID||TINYINT||survey flag identifier || || e.g. 3PI, M31, MD, STS - need common enum ||
||photoCalID||INT||photometry code numerical id || || ||
||filterID||TINYINT||filter identifier || ||IPP uses string (g.00000). Need enum. ||
||stackTypeID||TINYINT||stack type identifier || || ||
||refImageID||BIGINT||identifier of image used as reference for analysis || || ||
||subtrImageID||BIGINT||identifier of image subtracted to generate difference image || || ||
||analVer||SMALLINT||analysis version index || || ||
||nP2Images||SMALLINT||number of P2 images contributing to this cell || || ||
||astroScat||FLOAT||0.astrometric scatter for chip || || ||
||photoScat||FLOAT||0.photometric scatter for chip || || ||
||nAstroRef||INT||number of astrometric reference sources || || ||
||nPhoRef||INT||number of photometric reference sources || || ||
||psfFwhm||FLOAT||PSF full width at half maximum || || ||
||psfmodelID||SMALLINT||PSF model identifier || || ||
||psfExtra1||FLOAT||PSF parameters || || ||
||psfExtra2||FLOAT||PSF parameters || || ||
||photoZero||FLOAT||local derived photometric zero point || || ||
||photoColor||FLOAT||local derived photometric color term || || ||
||polyOrder||TINYINT||polynomial order of astrometry fit || || ||
||pca1x3y0||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x2y1||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x1y2||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x0y3||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x2y0||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x1y1||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x0y2||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x3y0||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x2y1||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x1y2||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x0y3||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x2y0||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x1y1||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x0y2||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||calibModNum||SMALLINT||calibration modification number || || ||
||dataRelease||TINYINT||Data release || || ||

'''{{{StackToImage}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||stackMetaID||BIGINT||stack identifier  || || ||
||imageID||BIGINT||hashed exposure-ccdID identifier || || ||

'''{{{StackLowSigDelta/StackHighSigDelta}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||partitionKey||BIGINT||partition key || || ||
||stackDiffID||BIGINT||ODM detection identifier || || ||
||ippDetectID||BIGINT||detection id generated by IPP || || ||
||filterID||TINYINT||filter ID: g=1, r=2, i=3, z=4, y=5, w=6,.. || || ||
||psfLikelihood||FLOAT||PSF likelihood || || ||
||infoFlag||INT||indicator of strange propeties || || ||
||nFrames||INT||number of frames contributing to source || || ||
||htmID||BIGINT||HTM index for the detection || || ||
||zoneID||INT||Zone index || || ||
||raErr||FLOAT||estimated error in ra || || ||
||decErr||FLOAT||estimated error in dec || || ||
||cx||FLOAT||cartesian x || || ||
||cy||FLOAT||cartesian y || || ||
||cz||FLOAT||cartesian z || || ||
||calMag||FLOAT||calibrated magnitude || || ||
||calMagErr||FLOAT||estimated error in calMag || || ||
||dataRelease||TINYINT||Data release when this detection was taken || || ||

== Unresolved fields for stack stage ==

Not yet...