= IPP to PSPS interface: {{{ippToPsps}}} =

= Links =

[http://datastore.ipp.ifa.hawaii.edu/PSPS_test Datastore test area for PSPS]

= Introduction =

{{{ippToPsps}}} is the interface between IPP and PSPS. In short, {{{ippToPsps}}} creates FITS files from data found in IPP {{{smf}}} and {{{cmf}}} files, as well as relevant DVO databases, then publishes them to a datastore in the form of ''jobs''. On the PSPS side, the DXLayer polls the datastore, collects jobs when they become available, converts the contents to {{csv}} files then sends them on to a special MSSQL loader software, which inserts them into the PSPS database. Ultimately there will be feedback from PSPS regarding errors in the received data, to which {{{ippToPsps}}} will need to respond.

It is intended that the binary tables in the FITS files generated by {{{ippToPsps}}} will match the PSPS database schemas perfectly, the consequence being that any alterations to the PSPS database schema will only require changes to {{{ippToPsps}}} code, and not the DXLayer. A certain amount of data validation will be performed by {{ippToPsps}} before publication

= {{{ippToPsps}}} products =

{{{ippToPsps}}} creates three ''products''.

 * '''Detections'''
 * '''Diffs'''
 * '''Stacks'''

= Architecture =

{{ippToPsps}} is a C program within the IPP build. When given the correct arguments it will generate a single FITS for the specified product (above). The program is run from a perl script, which itself is provided ... in order to generate the arguments necessary to run {{ippToPsps}}.

= Input and output =

== Detections ==

'''Input'''

Input is one {{smf}} file for a given exposure and an associated DVO database from which to retrieve object IDs.

'''Output'''

 A FITS output file is generated with the following extensions:

1 {{{'''FrameMeta'''}}} extension
1 {{{'''ImageMeta'''}}} extension per chip
1 {{{'''Detection'''}}} extension per chip

'''Validation'''

Some validation is performed to aid the data's insertion into the PSPS database.

== Diffs ==

'''Input'''

A set of {{cmf}} files, one for each skycell covered by a particular exposure.

'''Output'''

A FITS output file is generated with the following extensions:

...

'''Validation'''

= Unresolved fields =

Below are tables detailing which fields in the PSPS FITS files are still not populated by {{{ippToPsps}}}.

== Unresolved fields for camera stage ('detections') ==

'''{{{FrameMeta}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||    surveyID    ||            TINYINT                ||  survey flag identifier  || || e.g. 3PI, M31, MD, STS - need common enum ||
||    filterID     ||           TINYINT                ||  filter identifier      || || IPP uses string (g.00000). Need enum. ||
||    cameraID      ||          SMALLINT               ||  camera identifier  || || set to 1? ||
||    cameraConfigID||          SMALLINT               ||  camera configuration identifier  || || set to 0? ||
||    telescopeID  ||         SMALLINT               ||  telescope identifier  || || set to 1? ||
||    analysisVer ||            SMALLINT ||  version number of analysis  || || ||
||    p1Recip     ||           SMALLINT ||  IPP phase 1 recipe ID  || || ||
||    p2Recip     ||          SMALLINT ||  IPP phase 2 recipe ID  || || ||
||    p3Recip     ||         SMALLINT ||  IPP phase 3 recipe ID  || || ||
||    nP2Images  ||         SMALLINT ||  number of OTA/CCD images in this P2 exposure  || || always 60? ||
||    astroScat  ||             REAL     || global astrometric scatter  || || have values per chip, but global? ||
||    photoScat  ||             REAL     || global photometric scatter  || || ||
||    nAstRef    ||             INT      ||  number of astrometric reference sources  || || have values per chip, but global? ||
||    nPhoRef    ||             INT      ||  number of photometric reference sources   || || have values per chip, but global? ||
||    polyOrder  ||             TINYINT   ||  polynomial order of astrometry fit  || smf.phu.NPLYTERM? || ||
||    calibModNum||             SMALLINT               ||  calibration modification number  || || ||


'''{{{ImageMeta}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
|| filterID  || TINYINT     || filter ID || || IPP uses string (g.00000). Need enum. ||
|| bias      ||  REAL      || detector bias level || || missing from smf ||
|| biasScat  || REAL      || scatter in bias level || ||  missing from smf||
|| photoScat  ||  REAL      || photometric scatter for chip || smf.phu.ZPT_OBS? || ||
|| nPhoRef    ||   INT     || number of photometric reference sources || smf.chip.NPSFSTAR? smf.chip.NSTARS? || ||
|| psfFwhm   ||  REAL      || PSF full width at half maximum || smf.chip.FWHM_MAJ? || ||
|| psfModelID ||  INT       || PSF model identifier||  ||  missing from smf ||
|| psfExtra1   || REAL      || PSF parameters || || missing from smf  ||
|| psfExtra2   || REAL      || PSF parameters || || missing from smf ||
|| detectorID  || SMALLINT  || identifier for actual CCD chip || || IPP uses string (CCID58-1-02a2). Need enum.||
|| qaFlags     || INT       || Q/A flags || || PSPS needs two flag fields ||
|| detrend1    || BIGINT    || identifier of detrend image 1 || || IPP uses strings, PSPS wants INTs ||
|| detrend2    || BIGINT    || identifier of detrend image 2 || || IPP uses strings, PSPS wants INTs||
|| detrend3    || BIGINT    || identifier of detrend image 3 || ||IPP uses strings, PSPS wants INTs ||
|| detrend4     || BIGINT    || identifier of detrend image 4 || ||IPP uses strings, PSPS wants INTs ||
|| detrend5    || BIGINT    || identifier of detrend image 5 || ||IPP uses strings, PSPS wants INTs ||
|| detrend6    || BIGINT    || identifier of detrend image 6 || ||IPP uses strings, PSPS wants INTs ||
|| detrend7    || BIGINT    || identifier of detrend image 7 || || IPP uses strings, PSPS wants INTs||
|| detrend8    || BIGINT    || identifier of detrend image 8 || ||IPP uses strings, PSPS wants INTs ||
|| photoZero   || REAL      || local derived photometric zero point || || smf.phu.ZPT_OBS? ||
|| polyOrder   || TINYINT   || polynomial order of astrometry fit || smf.chip.NPLYTERM? || ||
|| calibModNum || SMALLINT  || calibration modification number ||  || set to 0? ||



'''{{{Detection}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||filterID||    TINYINT  || filter ID: g=1, r=2, i=3, z=4, y=5, w=6, ...  || || IPP uses string (g.00000). Need enum. ||
||surveyID||    TINYINT || survey flag identifier  || || e.g. 3PI, M31, MD, STS - need common enum ||
||    psfLikelihood       ||        REAL || PSF likelihood  || smf.psf.PSF_QF? || ||
||    psfCf||       REAL || PSF coverage factor  || smf.psf.PSF_QF? || ||
||    infoFlag||    INT   || bitmask indicating provenance information      || || ||
||    activeFlag||  TINYINT  || indicates whether this detection/orphan is still a detection/orphan || || ||
||    historyModNum     ||     SMALLINT || modification number in the O-D association history  || || set to 0? ||

== Unresolved fields for diff stage ('objects) ==

'''{{{StackMeta}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||stackMetaID||BIGINT||stack identifier|| || ||
||skycellID||INT||skycell region identifier || || missing from cmf files ||
||surveyID||TINYINT||survey flag identifier || || e.g. 3PI, M31, MD, STS - need common enum ||
||photoCalID||INT||photometry code numerical id || || ||
||filterID||TINYINT||filter identifier || ||IPP uses string (g.00000). Need enum. ||
||stackTypeID||TINYINT||stack type identifier || || ||
||refImageID||BIGINT||identifier of image used as reference for analysis || || ||
||subtrImageID||BIGINT||identifier of image subtracted to generate difference image || || ||
||analVer||SMALLINT||analysis version index || || ||
||nP2Images||SMALLINT||number of P2 images contributing to this cell || || ||
||astroScat||FLOAT||0.astrometric scatter for chip || || ||
||photoScat||FLOAT||0.photometric scatter for chip || || ||
||nAstroRef||INT||number of astrometric reference sources || || ||
||nPhoRef||INT||number of photometric reference sources || || ||
||psfFwhm||FLOAT||PSF full width at half maximum || || ||
||psfmodelID||SMALLINT||PSF model identifier || || ||
||psfExtra1||FLOAT||PSF parameters || || ||
||psfExtra2||FLOAT||PSF parameters || || ||
||photoZero||FLOAT||local derived photometric zero point || || ||
||photoColor||FLOAT||local derived photometric color term || || ||
||polyOrder||TINYINT||polynomial order of astrometry fit || || ||
||pca1x3y0||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x2y1||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x1y2||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x0y3||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x2y0||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x1y1||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca1x0y2||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x3y0||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x2y1||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x1y2||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x0y3||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x2y0||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x1y1||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||pca2x0y2||FLOAT||polynomial coefficients for the astrometric fit || || not available ||
||calibModNum||SMALLINT||calibration modification number || || ||
||dataRelease||TINYINT||Data release || || ||

'''{{{StackToImage}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||stackMetaID||BIGINT||stack identifier  || || ||
||imageID||BIGINT||hashed exposure-ccdID identifier || || ||

'''{{{StackLowSigDelta/StackHighSigDelta}}}'''

|| '''PSPS field''' || '''PSPS type''' || '''PSPS Description''' || '''IPP source''' || '''Comments''' ||
||partitionKey||BIGINT||partition key || || ||
||stackDiffID||BIGINT||ODM detection identifier || || ||
||ippDetectID||BIGINT||detection id generated by IPP || || ||
||filterID||TINYINT||filter ID: g=1, r=2, i=3, z=4, y=5, w=6,.. || || ||
||psfLikelihood||FLOAT||PSF likelihood || || ||
||infoFlag||INT||indicator of strange propeties || || ||
||nFrames||INT||number of frames contributing to source || || ||
||htmID||BIGINT||HTM index for the detection || || ||
||zoneID||INT||Zone index || || ||
||raErr||FLOAT||estimated error in ra || || ||
||decErr||FLOAT||estimated error in dec || || ||
||cx||FLOAT||cartesian x || || ||
||cy||FLOAT||cartesian y || || ||
||cz||FLOAT||cartesian z || || ||
||calMag||FLOAT||calibrated magnitude || || ||
||calMagErr||FLOAT||estimated error in calMag || || ||
||dataRelease||TINYINT||Data release when this detection was taken || || ||

== Unresolved fields for stack stage ==

Not yet...