Back to ippToPsps main page

Outline

The input for the detection batch is one IPP camera-stage smf file for a given exposure, as well as an associated DVO database from which to retrieve object, and other, IDs. One FITS file is generated for each exposure.

PSPS tables

1 Primary extension
1 FrameMeta extension
1 ImageMeta extension per chip
1 Detection extension per chip
1 SkinnyObject extension per chip
1 ObjectCalColor extension per chip

The full table definitions can be seen here

The 'object ID' is featured in the last three tables, and must remain unique across the exposure (it is generated within DVO). In the merged PSPS database, the primary key on the detections table is both the object ID and detection ID, meaning the the same object can appear in multiple, overlapping exposures as they will have different detection IDs.

Current IPP to PSPS mappings

The following table details the current mapping between IPP and PSPS. The source of smf files queued by ippToPsps is a list of cam_ids that are loaded into the relevant DVO database. These cam_ids are then used to obtain metadata from the IPP GPC1 database, as well as the location of the smf itself.

FrameMeta
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
frameID	unknown	INT	0	unique exposure/frame identifier.	GPC1 database	SELECT exp_id FROM camRun JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.cam_id = cam_id
frameName	unknown	VARCHAR(32)		frame name provided by camera software	GPC1 database	SELECT exp_name FROM camRun JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.cam_id = cam_id
surveyID	unknown	TINYINT	0	survey flag identifier	GPC1 database	SELECT camRun.dist_group FROM camRun JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.cam_id = cam_id. The IPP dist_group is then used to get the surveyID from the PSPS 'Survey' table here
filterID	unknown	TINYINT	0	filter identifier	smf primary header	Single character filter (grizy) is used to get PSPS ID from Filter table here
cameraID	unknown	SMALLINT	0	camera identifier	NA	hardcoded to 1
cameraConfigID	unknown	SMALLINT	0	camera configuration identifier	NA	hardcoded to 1
telescopeID	unknown	SMALLINT	0	telescope identifier	NA	hardcoded to 1
analysisVer	unknown	VARCHAR(MAX)		IPP software analysis release	NA	empty CZW: available in database as software_ver/header as PHOT_V
p1Recip	unknown	VARCHAR(MAX)		IPP phase 1 MD5 Checksum	NA	empty
p2Recip	unknown	VARCHAR(MAX)		IPP phase 2 MD5 Checksum	NA	empty
p3Recip	unknown	VARCHAR(MAX)		IPP phase 3 MD5 Checksum	NA	empty
nOTA	unknown	SMALLINT	-999	number of valid OTA/CCD images in this exposure	smf file	count of psf extensions successfully read from the smf file
photoScat	mag	REAL	-999	global photometric scatter	smf primary header	ZPT_ERR
numPhotoRef	unknown	INT	-999	number of photometric reference sources	smf XYnn.hdr extensions	sum of NASTRO over all OTAs
expStart	day	FLOAT	-999	exposure start time in MJD	smf primary header	MJD-OBS
expTime	s	REAL	-999	exposure time	smf primary header	EXPREQ
airmass	unknown	REAL	-999	airmass at mid-exposure	smf primary header	AIRMASS
raBore	deg	FLOAT	-999	RA of telescope boresight	smf primary header	RA
decBore	deg	FLOAT	-999	DEC of telescope boresight	smf primary header	DEC
ctype1	unknown	VARCHAR(MAX)		name of astrometric projection in RA	smf primary header	CTYPE1
ctype2	unknown	VARCHAR(MAX)		name of astrometric projection in DEC	smf primary header	CTYPE2
crval1	deg	FLOAT	-999	RA corresponding to reference pixel	smf primary header	CRVAL1
crval2	deg	FLOAT	-999	DEC corresponding to reference pixel	smf primary header	CRVAL2
crpix1	unknown	FLOAT	-999	reference pixel value for RA	smf primary header	CRPIX1
crpix2	unknown	FLOAT	-999	reference pixel value for DEC	smf primary header	CRPIX2
cdelt1	unknown	FLOAT	-999	scale factor for RA	smf primary header	CDELT1
cdelt2	unknown	FLOAT	-999	scale factor for DEC	smf primary header	CDELT2
pc001001	unknown	FLOAT	-999	elements of rotation/Dcale matrix	smf primary header	PC001001
pc001002	unknown	FLOAT	-999	elements of rotation/Dcale matrix	smf primary header	PC001002
pc002001	unknown	FLOAT	-999	elements of rotation/Dcale matrix	smf primary header	PC002001
pc002002	unknown	FLOAT	-999	elements of rotation/Dcale matrix	smf primary header	PC002002
polyOrder	unknown	TINYINT	255	polynomial order of astrometry fit	smf primary header	NPLYTERM
pca1x3y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA1X3Y0
pca1x2y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA1X2Y1
pca1x1y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA1X1Y2
pca1x0y3	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA1X0Y3
pca1x2y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA1X2Y0
pca1x1y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA1X1Y1
pca1x0y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA1X0Y2
pca2x3y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA2X3Y0
pca2x2y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA2X2Y1
pca2x1y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA2X1Y2
pca2x0y3	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA2X0Y3
pca2x2y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA2X2Y0
pca2x1y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA2X1Y1
pca2x0y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf primary header	PCA2X0Y2
calibModNum	unknown	SMALLINT	0	calibration modification number	NA	hardcoded to 1
dataRelease	unknown	TINYINT	0	Data release	NA	hardcoded to 1
ImageMeta
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
imageID	unknown	BIGINT	0	unique ID for each image, hashed from frameID and ccdID (ALEX)	GPC1 and smf extension name	frameID (below) mashed together with OTA number
frameID	unknown	INT	0	frame identifier, also known as exposure number	GPC1 database	SELECT exp_id FROM camRun JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.cam_id = cam_id
ccdID	unknown	SMALLINT	0	ccd cell identifier in the focal plane	smf	OTA number
photoCalID	unknown	INT	0	photometry reduction code identifier	DVO	image->photcode from DVO Image structure
filterID	unknown	TINYINT	0	filter ID	smf primary header	same as for FrameMeta (see above)
bias	ADU	REAL	-999	detector bias level		empty CZW: probably best to use gpc1.rawExp.bg
biasScat	ADU	REAL	-999	scatter in bias level		empty CZW: probably best to use gpc1.rawExp.bg_stdev
sky	ADU	REAL	-999	mean sky level	smf XYnn.hdr extension	MSKY_MN
skyScat	ADU	REAL	-999	scatter in mean sky level	smf XYnn.hdr extension	MSKY_SIG
nDetect	unknown	INT	-999	number of detections on CCD	smf XYnn.psf extension	count of detections retrieved from the psf extension
magSat	mag	REAL	-999	saturation magnitude level	smf XYnn.hdr extension	FSATUR
completMag	mag	REAL	-999	95% completion level in mag	smf XYnn.hdr extension	FLIMIT
astroScat	mag	REAL	-999	astrometric scatter for chip	smf XYnn.hdr extension	CERROR
photoScat	unknown	REAL	-999	photometric scatter for chip	smf primary header	ZPT_OBS
numAstroRef	unknown	INT	-999	number of astrometric reference sources	smf XYnn.hdr extension	NASTRO
numPhotoRef	unknown	INT	-999	number of photometric reference sources	smf XYnn.hdr extension	NASTRO
nx	unknown	SMALLINT	-999	chip dimension in x	smf XYnn.hdr extension	CNAXIS1
ny	unknown	SMALLINT	-999	chip dimension in y	smf XYnn.hdr extension	CNAXIS2
psfModelID	unknown	INT	-999	PSF model identifier	NA	empty CZW: PS_MODEL_PS1_V1
psfFwhm	arcsec	REAL	-999	model psf full width at half maximum at chip center	smf XYnn.hdr extension	(FWHM_MAJ + FWHM_MIN) / 2.0
psfWidMajor	arcsec	REAL	-999	model PSF parameters at chip center	smf XYnn.hdr extension	FWHM_MAJ
psfWidMinor	arcsec	REAL	-999	model PSF parameters at chip center	smf XYnn.hdr extension	FWHM_MIN
psfTheta	deg	REAL	-999	model PSF parameters at chip center	smf XYnn.hdr extension	ANGLE
momentFwhm	arcsec	REAL	-999	average PSF full width at half maximum from moments	smf XYnn.hdr extension	(IQ_FW1 + IQ_FW2) / 2.0
momentWidMajor	arcsec	REAL	-999	average PSF parameters from moments	smf XYnn.hdr extension	IQ_FW1
momentWidMinor	arcsec	REAL	-999	average PSF parameters from moments	smf XYnn.hdr extension	IQ_FW2
apResid	unknown	REAL	-999	corrected aperture residual	smf XYnn.hdr extension	APMIFIT
dapResid	unknown	REAL	-999	scatter of aperture corrections	smf XYnn.hdr extension	DAPMIFIT
detectorID	unknown	VARCHAR(32)		identifier for actual CCD chip	smf XYnn.hdr extension	DETECTOR
qaFlags	unknown	BIGINT	-999	Q/A flags for this OTA	DVO	image->flags from DVO Image structure
detrend1	unknown	VARCHAR(MAX)		identifier of detrend image 1	smf XYnn.hdr extension	DETREND.MASK
detrend2	unknown	VARCHAR(MAX)		identifier of detrend image 2	smf XYnn.hdr extension	DETREND.DARK
detrend3	unknown	VARCHAR(MAX)		identifier of detrend image 3	smf XYnn.hdr extension	DETREND.FLAT
detrend4	unknown	VARCHAR(MAX)		identifier of detrend image 4	NA	empty CZW: can we add DETREND.FRINGE
detrend5	unknown	VARCHAR(MAX)		identifier of detrend image 5	NA	empty CZW: can we add DETREND.LINEARITY
detrend6	unknown	VARCHAR(MAX)		identifier of detrend image 6	NA	empty
detrend7	unknown	VARCHAR(MAX)		identifier of detrend image 7	NA	empty
detrend8	unknown	VARCHAR(MAX)		identifier of detrend image 8	NA	empty
photoZero	mag	REAL	-999	local derived photometric zero point	smf primary header	ZPT_OBS
ctype1	unknown	VARCHAR(MAX)		name of astrometric projection in RA	smf XYnn.hdr extension	CTYPE1
ctype2	unknown	VARCHAR(MAX)		name of astrometric projection in DEC	smf XYnn.hdr extension	CTYPE2
crval1	deg	FLOAT	-999	RA corresponding to reference pixel	smf XYnn.hdr extension	CRVAL1
crval2	deg	FLOAT	-999	DEC corresponding to reference pixel	smf XYnn.hdr extension	CRVAL2
crpix1	pix	FLOAT	-999	reference pixel value for RA	smf XYnn.hdr extension	CRPIX1
crpix2	pix	FLOAT	-999	reference pixel value for DEC	smf XYnn.hdr extension	CRPIX2
cdelt1	unknown	FLOAT	-999	scale factor for RA	smf XYnn.hdr extension	CDELT1
cdelt2	unknown	FLOAT	-999	scale factor for DEC	smf XYnn.hdr extension	CDELT2
pc001001	unknown	FLOAT	-999	elements of rotation/Dcale matrix	smf XYnn.hdr extension	PC001001
pc001002	unknown	FLOAT	-999	elements of rotation/Dcale matrix	smf XYnn.hdr extension	PC001002
pc002001	unknown	FLOAT	-999	elements of rotation/Dcale matrix	smf XYnn.hdr extension	PC002001
pc002002	unknown	FLOAT	-999	elements of rotation/Dcale matrix	smf XYnn.hdr extension	PC002002
polyOrder	unknown	TINYINT	255	polynomial order of astrometry fit	smf XYnn.hdr extension	NPLYTERM
pca1x3y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA1X3Y0
pca1x2y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA1X2Y1
pca1x1y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA1X1Y2
pca1x0y3	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA1X0Y3
pca1x2y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA1X2Y0
pca1x1y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA1X1Y1
pca1x0y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA1X0Y2
pca2x3y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA2X3Y0
pca2x2y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA2X2Y1
pca2x1y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA2X1Y2
pca2x0y3	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA2X0Y3
pca2x2y0	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA2X2Y0
pca2x1y1	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA2X1Y1
pca2x0y2	unknown	FLOAT	-999	polynomial coefficients for the astrometric fit	smf XYnn.hdr extension	PCA2X0Y2
calibModNum	unknown	SMALLINT	0	calibration modification number	NA	hardcoded to 1
dataRelease	unknown	TINYINT	0	Data release	NA	hardcoded to 1
Detection
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
objID	unknown	BIGINT	0	ODM object identifier	DVO	ave.extID from dvoDetection structure
detectID	unknown	BIGINT	0	ODM detection identifier	DVO	meas.extID from dvoDetection structure
ippObjID	unknown	BIGINT	0	object id generated by IPP	DVO	ave.catID*1000000000 + ave.objID from dvoDetection structure
ippDetectID	unknown	BIGINT	0	detection id generated by IPP	smf XYnn.psf extension	IPP_IDET
filterID	unknown	TINYINT	0	filter ID: g=1, r=2, i=3, z=4, y=5, w=6, ...	smf primary header	same as for FrameMeta (see above)
surveyID	unknown	TINYINT	0	survey flag identifier	GPC1	same as for FrameMeta (see above)
imageID	unknown	BIGINT	0	image id hash of frame identifier & OTA/CCD number	GPC1 and smf extension name	same as for ImageMeta (see above)
obsTime	day	FLOAT	-999	Time of mid observation	smf primary header	(MJD-OBS + EXPTIME) / 172800.0
xPos	pix	REAL	-999	measured x on CCD from PSF fit	smf XYnn.psf extension	X_PSF
yPos	pix	REAL	-999	measured y on CCD from PSF fit	smf XYnn.psf extension	Y_PSF
xPosErr	pix	REAL	-999	estimated error in x	smf XYnn.psf extension	X_PSF_SIG
yPosErr	pix	REAL	-999	estimated error in y	smf XYnn.psf extension	Y_PSF_SIG
instFlux	adu/s	REAL	-999	PSF instrumental flux	smf primary header (EXPTIME) and XYnn.psf extension	POW(10.0, (-0.4*PSF_INST_MAG)) / EXPTIME
instFluxErr	adu/s	REAL	-999	estimated error in flux	smf primary header (EXPTIME) and XYnn.psf extension	ABS((PSF_INST_MAG_SIG*(POW(10.0, (-0.4*PSF_INST_MAG)) / EXPTIME)) / 1.085736)
peakADU	adu	REAL	-999	peak count in source pixel	smf primary header (EXPTIME) and XYnn.psf extension	POW(10.0, (-0.4*PEAK_FLUX_AS_MAG)) / EXPTIME
psfWidMajor	arcsec	REAL	-999	model PSF width in major axis	smf XYnn.psf extension	PSF_MAJOR
psfWidMinor	arcsec	REAL	-999	model PSF width in minor axis	smf XYnn.psf extension	PSF_MINOR
psfTheta	deg	REAL	-999	model PSF orientation angle	smf XYnn.psf extension	PSF_THETA
psfLikelihood	unknown	REAL	-999	PSF likelihood	smf XYnn.psf extension	erfc(abs(EXT_NSIGMA) / sqrt(2))
psfCf	unknown	REAL	-999	PSF coverage factor	smf XYnn.psf extension	PSF_QF
momentXX	arcsec	REAL	-999	moment XX	smf XYnn.psf extension	MOMENTS_XX
momentXY	arcsec	REAL	-999	moment XY	smf XYnn.psf extension	MOMENTS_XY
momentYY	arcsec	REAL	-999	momeny YY	smf XYnn.psf extension	MOMENTS_YY
apMag	magnitude	REAL	-999	Aperture magnitude	smf XYnn.psf extension	AP_MAG
apMagErr	magnitude	REAL	-999	Aperture magnitude error	NA	empty
kronFlux	magnitude	REAL	-999	Kron flux	NA	empty CZW: SMF/XY.psf/KRON_FLUX
kronFluxErr	magnitude	REAL	-999	Kron flux error	NA	empty CZW: SMF/XY.psf/KRON_FLUX_ERR
kronRad	arcsecs	REAL	-999	Kron radius	NA	empty
kronRadErr	arcsecs	REAL	-999	Kron radius error	NA	empty
crLikelihood	unknown	REAL	-999	Likelihood the source is a cosmic ray	smf XYnn.psf extension	(EXT_NSIGMA > 0) ? 0 : 1 - psfLikelihood
extendedLikelihood	unknown	REAL	-999	Likelihood the source is extended	smf XYnn.psf extension	(EXT_NSIGMA > 0) ? 1 - psfLikelihood : 0
infoFlag	unknown	BIGINT	-999	flag indicating provenance information	smf XYnn.psf extension and DVO	dvoDetection.meas.dbFlags << 32 | FLAGS
sky	adu	REAL	-999	PSF sky level at source	smf XYnn.psf extension	SKY
skyErr	adu	REAL	-999	estimated error in sky	smf XYnn.psf extension	SKY_SIGMA
sgSep	unknown	REAL	-999	star/galaxy separator	smf XYnn.psf extension	EXT_NSIGMA
activeFlag	unknown	TINYINT	-999	indicates whether this detection/orphan is still a detection/orphan	NA	hardcoded to 0
assocDate	unknown	DATE	28881231	date object association assigned	NA	date that batch has been prepared
historyModNum	unknown	SMALLINT	0	modification number in the O-D association history	NA	hardcoded to 0
dataRelease	unknown	TINYINT	0	Data release when this detection was originally taken. Recalibrations do not affect this value.	NA	hardcoded to 0
SkinnyObject
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
objID	unknown	BIGINT	0	ODM object identifier index	DVO	same as Detection table - see above
ippObjID	unknown	BIGINT	0	IPP object number	DVO	same as Detection table - see above
projectionCellID	unknown	INT	-999	projection cell identifier at discovery time	NA	empty CZW: the skycell_id contains this
dataRelease	unknown	TINYINT	0	Data release to propagate to the object	NA	hardcoded to 0
surveyID	unknown	TINYINT	255	surveyID	GPC1	same as for FrameMeta (see above)
ObjectCalColor
ODM attribute	units	Data type	Default	Description	IPP source	IPP variable/notes
objID	unknown	BIGINT	0	ODM object identifier	DVO	same as Detection table - see above
ippObjID	unknown	BIGINT	0	ipp object identifier	DVO	same as Detection table - see above
filterID	unknown	TINYINT	0	filter identifier	smf primary header	same as for FrameMeta (see above)
calColor	mag	REAL	0	color adopted for magnitude calculation	NA	empty
calColorErr	mag	REAL	0	error in calibrating color	NA	empty
calibModNum	unknown	SMALLINT	0	calibration modification number	NA	hardcoded to 1
dataRelease	unknown	TINYINT	0	Data release when this color calibration was established	NA	hardcoded to 0

Running

To create and publish detection batches, do the following from the IPP trunk:

cd ippToPsps/perl
./makeDetections.pl

This will list all options. Essentials are the following:

* REQUIRED: need to provide an output path  -o <path>
* REQUIRED: need to provide a DVO Db        -d <dvoDb>
* REQUIRED: need to provide a DVO location  -l <dvoLocation>

Running with just these arguments will create batches for all unprocessed exposures available in the provided DVO database. It will store the batches in the provided output directory. To publish the batches to the datastore, the -p argument (datastore product) must be provided.

The script connects with the ippToPsps database, so is aware of what has and has not already been processed and published. Only using the -f option can you force it to re-process a batch. Note that publishing the same exposure under two different batch names will cause havoc for the ODM, so be careful with the -f option.

Example usage would be:

./makeDetections.pl -o . -d ThreePi.V1 -l /data/ipp004.0/gpc1/catdirs/ThreePi.V1 -p PSPS_test

This will work through all available exposures in the DVO database 'ThreePi.V1' at the location specified, and publishing to the 'PSPS_test' datastore product, updating the ippToPsps database as it proceeds. Querying this database allows you to see what has been processed, the failure rate, what has been published to the datastore etc.