Back to ippToPsps main page

Outline

The input for the detection batch is one IPP camera-stage smf file for a given exposure, as well as an associated DVO database from which to retrieve object, and other, IDs. One FITS file is generated for each exposure.

PSPS tables

1 Primary extension
1 FrameMeta extension
1 ImageMeta extension per chip
1 Detection extension per chip
1 SkinnyObject extension per chip
1 ObjectCalColor extension per chip

The full table definitions can be seen here

The 'object ID' is featured in the last three tables, and must remain unique across the exposure (it is generated within DVO). In the merged PSPS database, the primary key on the detections table is both the object ID and detection ID, meaning the the same object can appear in multiple, overlapping exposures as they will have different detection IDs.

Current IPP to PSPS mappings

The following table details the current mapping between IPP and PSPS. The source of smf files queued by ippToPsps is a list of cam_ids that are loaded into the relevant DVO database. These cam_ids are then used to obtain metadata from the IPP GPC1 database, as well as the location of the smf itself.

FrameMeta
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
frameID	unknown	INT	0	unique exposure/frame identifier.	GPC1 database	SELECT exp_id, exp_name, camRun.dist_group FROM camRun JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.cam_id = cam_id
frameName	unknown	VARCHAR(32)		frame name provided by camera software
surveyID	unknown	TINYINT	0	survey flag identifier
filterID	unknown	TINYINT	0	filter identifier
cameraID	unknown	SMALLINT	0	camera identifier
cameraConfigID	unknown	SMALLINT	0	camera configuration identifier
telescopeID	unknown	SMALLINT	0	telescope identifier
analysisVer	unknown	VARCHAR(MAX)		IPP software analysis release
p1Recip	unknown	VARCHAR(MAX)		IPP phase 1 MD5 Checksum
p2Recip	unknown	VARCHAR(MAX)		IPP phase 2 MD5 Checksum
p3Recip	unknown	VARCHAR(MAX)		IPP phase 3 MD5 Checksum
nOTA	unknown	SMALLINT	-999	number of valid OTA/CCD images in this exposure
photoScat	mag	REAL	-999	global photometric scatter
numPhotoRef	unknown	INT	-999	number of photometric reference sources
expStart	day	FLOAT	-999	exposure start time in MJD
expTime	s	REAL	-999	exposure time
airmass	unknown	REAL	-999	airmass at mid-exposure
raBore	deg	FLOAT	-999	RA of telescope boresight
decBore	deg	FLOAT	-999	DEC of telescope boresight
ctype1	unknown	VARCHAR(MAX)		name of astrometric projection in RA
ctype2	unknown	VARCHAR(MAX)		name of astrometric projection in DEC
crval1	deg	FLOAT	-999	RA corresponding to reference pixel
crval2	deg	FLOAT	-999	DEC corresponding to reference pixel
crpix1	unknown	FLOAT	-999	reference pixel value for RA
crpix2	unknown	FLOAT	-999	reference pixel value for DEC
cdelt1	unknown	FLOAT	-999	scale factor for RA
cdelt2	unknown	FLOAT	-999	scale factor for DEC
pc001001	unknown	FLOAT	-999	elements of rotation/Dcale matrix
pc001002	unknown	FLOAT	-999	elements of rotation/Dcale matrix
pc002001	unknown	FLOAT	-999	elements of rotation/Dcale matrix
pc002002	unknown	FLOAT	-999	elements of rotation/Dcale matrix
polyOrder	unknown	TINYINT	255	polynomial order of astrometry fit
pca1x3y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x2y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x1y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x0y3	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x2y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x1y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x0y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x3y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x2y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x1y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x0y3	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x2y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x1y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x0y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
calibModNum	unknown	SMALLINT	0	calibration modification number
dataRelease	unknown	TINYINT	0	Data release
ImageMeta
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
imageID	unknown	BIGINT	0	unique ID for each image, hashed from frameID and ccdID (ALEX)
frameID	unknown	INT	0	frame identifier, also known as exposure number
ccdID	unknown	SMALLINT	0	ccd cell identifier in the focal plane
photoCalID	unknown	INT	0	photometry reduction code identifier
filterID	unknown	TINYINT	0	filter ID
bias	ADU	REAL	-999	detector bias level
biasScat	ADU	REAL	-999	scatter in bias level
sky	ADU	REAL	-999	mean sky level
skyScat	ADU	REAL	-999	scatter in mean sky level
nDetect	unknown	INT	-999	number of detections on CCD
magSat	mag	REAL	-999	saturation magnitude level
completMag	mag	REAL	-999	95% completion level in mag
astroScat	mag	REAL	-999	astrometric scatter for chip
photoScat	unknown	REAL	-999	photometric scatter for chip
numAstroRef	unknown	INT	-999	number of astrometric reference sources
numPhotoRef	unknown	INT	-999	number of photometric reference sources
nx	unknown	SMALLINT	-999	chip dimension in x
ny	unknown	SMALLINT	-999	chip dimension in y
psfModelID	unknown	INT	-999	PSF model identifier
psfFwhm	arcsec	REAL	-999	model psf full width at half maximum at chip center
psfWidMajor	arcsec	REAL	-999	model PSF parameters at chip center
psfWidMinor	arcsec	REAL	-999	model PSF parameters at chip center
psfTheta	deg	REAL	-999	model PSF parameters at chip center
momentFwhm	arcsec	REAL	-999	average PSF full width at half maximum from moments
momentWidMajor	arcsec	REAL	-999	average PSF parameters from moments
momentWidMinor	arcsec	REAL	-999	average PSF parameters from moments
apResid	unknown	REAL	-999	corrected aperture residual
dapResid	unknown	REAL	-999	scatter of aperture corrections
detectorID	unknown	VARCHAR(32)		identifier for actual CCD chip
qaFlags	unknown	BIGINT	-999	Q/A flags for this OTA
detrend1	unknown	VARCHAR(MAX)		identifier of detrend image 1
detrend2	unknown	VARCHAR(MAX)		identifier of detrend image 2
detrend3	unknown	VARCHAR(MAX)		identifier of detrend image 3
detrend4	unknown	VARCHAR(MAX)		identifier of detrend image 4
detrend5	unknown	VARCHAR(MAX)		identifier of detrend image 5
detrend6	unknown	VARCHAR(MAX)		identifier of detrend image 6
detrend7	unknown	VARCHAR(MAX)		identifier of detrend image 7
detrend8	unknown	VARCHAR(MAX)		identifier of detrend image 8
photoZero	mag	REAL	-999	local derived photometric zero point
ctype1	unknown	VARCHAR(MAX)		name of astrometric projection in RA
ctype2	unknown	VARCHAR(MAX)		name of astrometric projection in DEC
crval1	deg	FLOAT	-999	RA corresponding to reference pixel
crval2	deg	FLOAT	-999	DEC corresponding to reference pixel
crpix1	pix	FLOAT	-999	reference pixel value for RA
crpix2	pix	FLOAT	-999	reference pixel value for DEC
cdelt1	unknown	FLOAT	-999	scale factor for RA
cdelt2	unknown	FLOAT	-999	scale factor for DEC
pc001001	unknown	FLOAT	-999	elements of rotation/Dcale matrix
pc001002	unknown	FLOAT	-999	elements of rotation/Dcale matrix
pc002001	unknown	FLOAT	-999	elements of rotation/Dcale matrix
pc002002	unknown	FLOAT	-999	elements of rotation/Dcale matrix
polyOrder	unknown	TINYINT	255	polynomial order of astrometry fit
pca1x3y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x2y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x1y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x0y3	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x2y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x1y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca1x0y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x3y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x2y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x1y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x0y3	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x2y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x1y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
pca2x0y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit
calibModNum	unknown	SMALLINT	0	calibration modification number
dataRelease	unknown	TINYINT	0	Data release
Detection
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
objID	unknown	BIGINT	0	ODM object identifier
detectID	unknown	BIGINT	0	ODM detection identifier
ippObjID	unknown	BIGINT	0	object id generated by IPP
ippDetectID	unknown	BIGINT	0	detection id generated by IPP
filterID	unknown	TINYINT	0	filter ID: g=1, r=2, i=3, z=4, y=5, w=6, ...
surveyID	unknown	TINYINT	0	survey flag identifier
imageID	unknown	BIGINT	0	image id hash of frame identifier & OTA/CCD number
obsTime	day	FLOAT	-999	Time of mid observation
xPos	pix	REAL	-999	measured x on CCD from PSF fit
yPos	pix	REAL	-999	measured y on CCD from PSF fit
xPosErr	pix	REAL	-999	estimated error in x
yPosErr	pix	REAL	-999	estimated error in y
instFlux	adu/s	REAL	-999	PSF instrumental flux
instFluxErr	adu/s	REAL	-999	estimated error in flux
peakADU	adu	REAL	-999	peak count in source pixel
psfWidMajor	arcsec	REAL	-999	model PSF width in major axis
psfWidMinor	arcsec	REAL	-999	model PSF width in minor axis
psfTheta	deg	REAL	-999	model PSF orientation angle
psfLikelihood	unknown	REAL	-999	PSF likelihood
psfCf	unknown	REAL	-999	PSF coverage factor
momentXX	arcsec	REAL	-999	moment XX
momentXY	arcsec	REAL	-999	moment XY
momentYY	arcsec	REAL	-999	momeny YY
apMag	magnitude	REAL	-999	Aperture magnitude
apMagErr	magnitude	REAL	-999	Aperture magnitude error
kronFlux	magnitude	REAL	-999	Kron flux
kronFluxErr	magnitude	REAL	-999	Kron flux error
kronRad	arcsecs	REAL	-999	Kron radius
kronRadErr	arcsecs	REAL	-999	Kron radius error
crLikelihood	unknown	REAL	-999	Likelihood the source is a cosmic ray
extendedLikelihood	unknown	REAL	-999	Likelihood the source is extended
infoFlag	unknown	BIGINT	-999	flag indicating provenance information
sky	adu	REAL	-999	PSF sky level at source
skyErr	adu	REAL	-999	estimated error in sky
sgSep	unknown	REAL	-999	star/galaxy separator
activeFlag	unknown	TINYINT	-999	indicates whether this detection/orphan is still a detection/orphan
assocDate	unknown	DATE	28881231	date object association assigned
historyModNum	unknown	SMALLINT	0	modification number in the O-D association history
dataRelease	unknown	TINYINT	0	Data release when this detection was originally taken. Recalibrations do not affect this value.
SkinnyObject
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
objID	unknown	BIGINT	0	ODM object identifier index
ippObjID	unknown	BIGINT	0	IPP object number
projectionCellID	unknown	INT	-999	projection cell identifier at discovery time
dataRelease	unknown	TINYINT	0	Data release to propagate to the object
surveyID	unknown	TINYINT	255	surveyID </coluumn>
ObjectCalColor
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
objID	unknown	BIGINT	0	ODM object identifier
ippObjID	unknown	BIGINT	0	ipp object identifier
filterID	unknown	TINYINT	0	filter identifier
calColor	mag	REAL	0	color adopted for magnitude calculation
calColorErr	mag	REAL	0	error in calibrating color
calibModNum	unknown	SMALLINT	0	calibration modification number
dataRelease	unknown	TINYINT	0	Data release when this color calibration was established

Running

To create and publish detection batches, do the following from the IPP trunk:

cd ippToPsps/perl
./makeDetections.pl

This will list all options. Essentials are the following:

* REQUIRED: need to provide an output path  -o <path>
* REQUIRED: need to provide a DVO Db        -d <dvoDb>
* REQUIRED: need to provide a DVO location  -l <dvoLocation>

Running with just these arguments will create batches for all unprocessed exposures available in the provided DVO database. It will store the batches in the provided output directory. To publish the batches to the datastore, the -p argument (datastore product) must be provided.

The script connects with the ippToPsps database, so is aware of what has and has not already been processed and published. Only using the -f option can you force it to re-process a batch. Note that publishing the same exposure under two different batch names will cause havoc for the ODM, so be careful with the -f option.

Example usage would be:

./makeDetections.pl -o . -d ThreePi.V1 -l /data/ipp004.0/gpc1/catdirs/ThreePi.V1 -p PSPS_test

This will work through all available exposures in the DVO database 'ThreePi.V1' at the location specified, and publishing to the 'PSPS_test' datastore product, updating the ippToPsps database as it proceeds. Querying this database allows you to see what has been processed, the failure rate, what has been published to the datastore etc.