Index: /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/Detection.txt =================================================================== --- /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/Detection.txt (revision 37233) +++ /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/Detection.txt (revision 37234) @@ -46,5 +46,5 @@ apFlux + smf:AP_FLUX apFluxErr + smf:AP_FLUX_SIG -apFillF - smf:AP_NPIX / (pi * (AP_MAG_RADIUS - 0.5)^2) +apFillF + smf:AP_NPIX / (pi * (AP_MAG_RADIUS - 0.5)^2) kronFlux + smf:KRON_FLUX kronFluxErr + smf:KRON_FLUX_ERR Index: /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/MeanObject =================================================================== --- /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/MeanObject (revision 37233) +++ /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/MeanObject (revision 37234) @@ -7,18 +7,18 @@ gMeanPSFMag AB_magnitude REAL 4 -999 mean g psf AB magnitude from best available measurements (best available as current defined by ipp "waterfall logic" gMeanPSFMagErr AB_magnitude REAL 4 -999 estimated error of best availalbe g-band psf AB magnitudes (individual errors added in quadrature) -gMeanPSFMagNct dimensionless SMALLINT 2 -999 number of best available detections included in mean psfMag gMeanPSFMagStd dimensioless REAL 4 -999 standard deviation of best available measurements - compared with MeanErr this is one measure of true variability gMeanPSFMagMin AB_magnitude REAL 4 -999 minimum best available PSFMag - min and max are given for PSFMag as they may indicate variability. gMeanPSFMagMax AB_magnitude REAL 4 -999 maximum best available PSFMag +gMeanPSFMagNpt dimensionless SMALLINT 2 -999 number of best available detections included in mean psfMag gMeanKronMag AB_magnitude REAL 4 -999 mean g kron AB magnitude from best available measurements gMeanKronMagErr AB_magnitude REAL 4 -999 estimated error of mean g kron AB magnitude -gMeanKronMagNct dimensionless SMALLINT 2 -999 number of best available detections included mean Kron mag -gMeanKronMagStd dimensioless REAL 4 -999 standard deviation of best available measurements +gMeanKronMagStd dimensionless REAL 4 -999 standard deviation of best available measurements +gMeanKronMagNpt dimensionless SMALLINT 2 -999 number of best available detections included mean Kron mag gMeanApMag AB_magnitude REAL 4 -999 mean g kron AB magnitude from best available measurements gMeanApMagErr AB_magnitude REAL 4 -999 estimated error of g kron AB magnitude mean -gMeanApMagNct dimensionless SMALLINT 2 -999 number of detections included in best available mean of Kron mag in g filter gMeanApMagStd dimensioless REAL 4 -999 standard deviation of best available measurements +gMeanApMagNpt dimensionless SMALLINT 2 -999 number of detections included in best available mean of Kron mag in g filter gFlags dimensionless INT 4 0 flags for g mean [XXX secfilt.flags is 4byte] Index: /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/schema18.notes.txt =================================================================== --- /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/schema18.notes.txt (revision 37233) +++ /branches/eam_branches/ipp-20140717/ippToPsps/pspsschema/SchemaNew18.0/PSPS_TABLES/Fundamental_IPP_Data_Product_Tables/schema18.notes.txt (revision 37234) @@ -2,102 +2,158 @@ This document describes the new PSPS schema (18.0) and identifies existing problems. +ObjectThin + + Changes since Schema 16.0: + + * DROPPED gcObjID (concept rejected by DRAVG) + * replaced skycellID (REAL) with combination of tessID, projectionID, skycellID (1,2,1 byte INTs) + * replaced raMeanStd, decMeanStd (not calculated) with posMeanChisq + + Concerns: + + * objName is generated by SQL in ippToPsps and is probably fragile: move to dvopsps? + * fields calculated by PSPS and not tested (htmID, zoneID, cx,cy,cz, lambda,beta, l,b) + * nStackObjectRows is not calculated + * should nStackDetections be split per filter? + * quality flag is currently the upper byte of dvo:average.flags, is this reasonable? + * consistency flag is hard-wired as 0 + * raStack, etc are not yet calculated by relastro (set to match raMean, etc) + +MeanObject + + Changes since Schema 16.0: + + * DROPPED gcObjID (concept rejected by DRAVG) + * CHANGED gFlags to be 4 byte not 8 byte (secfilt.flags is 4 byte) + * renamed fields to have consistent naming structure: + * gMeanTYPEMagVALUE, where TYPE = (PSF, Kron, Ap) and VALUE = (Err, Std, Min Max, Npt) + + Concerns: + + * NONE + +Detections + + Changes since Schema 16.0: + + * ADDED apRadius + * DROPPED gcObjID (concept rejected by DRAVG) + * DROPPED kronRadErr (not measured by psphot, probably not needed?) + + Concerns: + + * imageID : this is currently generated from expID + ccd X,Y. + shouldn't we use gpc1.chipImfile.chip_imfile_id? (IMAGE_ID in + header) + * double-check reported value for zero point (needs to include airmass term, etc) + + Implementation Changes: + + * raErr,decErr are now taken from the SMF file not from dvo (x,y errors truncated in dvo) + * dvoRegion now uses CAT_ID from dvopsps + * psfMajor,MinorFWHM use PSF_FWHM_MJA,MIN + * added PSF_CORE to smf + * apFlux, kronFlux use FLUX values directly from smf + Stacks -Measurements related to the stacks are now split into 11 separate -tables depending on the measurements and, in some cases, the Galactic -latitude. The granularity of the split is partly mandated by the -inability of PSPS or its ingest system to handle excessively wide -tables. Note that in all of the tables below, a single row represents -the set of measurements from all 5 filters (grizy) for the aparition -of an astronomical object in a single skycell. The primaryDetection -flag is set to identify the row which comes from the primary skycell -for that location on the sky. + Measurements related to the stacks are now split into 11 separate + tables depending on the measurements and, in some cases, the Galactic + latitude. The granularity of the split is partly mandated by the + inability of PSPS or its ingest system to handle excessively wide + tables. Note that in all of the tables below, a single row represents + the set of measurements from all 5 filters (grizy) for the aparition + of an astronomical object in a single skycell. The primaryDetection + flag is set to identify the row which comes from the primary skycell + for that location on the sky. + + Several of the tables are meant to be filled only for the Galactic cap + regions. As of 2014.08.09, ippToPsps is not setting this limit. + These tables are: + + StackModelFitDeV + StackModelFitExp + StackModelFitSer + StackModelFitExtra + StackApFlxFull + StackApFlxFullC1 + StackApFlxFullC2 + StackPetrosian -Several of the tables are meant to be filled only for the Galactic cap -regions. As of 2014.08.09, ippToPsps is not setting this limit. -These tables are: + Other general concerns: - StackModelFitDeV - StackModelFitExp - StackModelFitSer - StackModelFitExtra - StackApFlxFull - StackApFlxFullC1 - StackApFlxFullC2 - StackPetrosian + * various size-related fields are reported in pixels not arcseconds. + + * I need to double-check the reported zero point and telluric + extinction values. + + * are we defining too many radial apertures? -Other general concerns: + The tables are divided as follows: + + StackObjectThin : basic properties of the stack object detection. + * stackDetectRowID : this field will be set by PSPS and cannot be + tested in the ippToPsps code -* various size-related fields are reported in pixels not arcseconds. + * bestDetection : this field cannot be set until the entire PSPS is + populated. this will be deferred until the end of + the ingest. -* I need to double-check the reported zero point and telluric - extinction values. - -* almost no - -The tables are divided as follows: - -StackObjectThin : basic properties of the stack object detection. - * stackDetectRowID : this field will be set by PSPS and cannot be - tested in the ippToPsps code - * bestDetection : this field cannot be set until the entire PSPS is - populated. this will be deferred until the end of - the ingest. - * primaryDetection : NOT YET TESTED - * xinfoFlag2 : currently only includes bits from - dvo:measure.dbflags but not dvo:secfilt.flags - -* NOTE : stackDetectRowID, bestDetection, primaryDetection are - inherited by the following tables from StackObjectThin, so the issue - above are present in all Stack tables - -StackObjectRaw : raw measurement values from the PSF fits - * xpsfLikelihood : some values seem to be negative, which should - not be allowed - -StackModelFitExtra : extra properties derived from extended source - fits. NOTE: this table is not being generated - -StackModelFitExp : Exponential model fit parameters - * xExpRadiusErr : EXT_WIDTH_MAJ_SIG is not being set in the CMF file - * xExpAbErr : EXT_WIDTH_MIN_SIG is not being set in the CMF file - * xExpPhiErr : not set in CMF - * xExpRa,Dec : this position is currently an approximate based on - X_EXT, X_PSF and the plate scale - * xExpRa,DecErr : circularly symmetric approximation based on S/N - * xExpCf : not calculated in psphot - * xExpChisq : the fit chisq is provided, not the likelihood - -StackModelFitDeV : DeVaucouleur model fit parameters - * NOTE same issues as for Exp - -StackModelFitSer : Sersic model fit parameters - * NOTE same issues as for Exp, also: - * xSerNuErr : value is hard-wired at 0.1, and not well-measured - in CMF - -StackApFlx : flux measured in radial apertures (only 3 radii; raw and - both convolutions) - - * standard deviation fields (eg, gflxR3Std) are measured in psphot as - the standard deviation across the full object. Should this perhaps - be a measurement of the standard deviation after the radial profile - is subtracted? - -StackApFlxFull : flux measured in radial apertures (raw, all radii) - - * see note above on Standard Deviation - * radius 10 is not measured or reported in the CMF - -StackApFlxFullC1 : flux measured in radial apertures (first convolution, all radii) - - * see note above on Standard Deviation - * radius 10 is not measured or reported in the CMF - -StackApFlxFullC2 : flux measured in radial apertures (second convolution, all radii) - - * see note above on Standard Deviation - * radius 10 is not measured or reported in the CMF - -StackPetrosian : petrosian analysis data - * xpetCf : fill factor is mis-calculated in psphot (consistently > 1.0) + * primaryDetection : NOT YET TESTED + * xinfoFlag2 : currently only includes bits from + dvo:measure.dbflags but not dvo:secfilt.flags + + * NOTE : stackDetectRowID, bestDetection, primaryDetection are + inherited by the following tables from StackObjectThin, so the issue + above are present in all Stack tables + + StackObjectRaw : raw measurement values from the PSF fits + * xpsfLikelihood : some values seem to be negative, which should + not be allowed + + StackModelFitExtra : extra properties derived from extended source + fits. NOTE: this table is not being generated + + StackModelFitExp : Exponential model fit parameters + * xExpRadiusErr : EXT_WIDTH_MAJ_SIG is not being set in the CMF file + * xExpAbErr : EXT_WIDTH_MIN_SIG is not being set in the CMF file + * xExpPhiErr : not set in CMF + * xExpRa,Dec : this position is currently an approximation based on + X_EXT, X_PSF and the plate scale + * xExpRa,DecErr : circularly symmetric approximation based on S/N + * xExpCf : not calculated in psphot + * xExpChisq : the fit chisq is provided, not the likelihood + + StackModelFitDeV : DeVaucouleur model fit parameters + * NOTE same issues as for Exp + + StackModelFitSer : Sersic model fit parameters + * NOTE same issues as for Exp, also: + * xSerNuErr : value is hard-wired at 0.1, and not well-measured + in CMF + + StackApFlx : flux measured in radial apertures (only 3 radii; raw and + both convolutions) + + * standard deviation fields (eg, gflxR3Std) are measured in psphot as + the standard deviation across the full object. Should this perhaps + be a measurement of the standard deviation after the radial profile + is subtracted? + + StackApFlxFull : flux measured in radial apertures (raw, all radii) + + * see note above on Standard Deviation + * radius 10 is not measured or reported in the CMF + + StackApFlxFullC1 : flux measured in radial apertures (first convolution, all radii) + + * see note above on Standard Deviation + * radius 10 is not measured or reported in the CMF + + StackApFlxFullC2 : flux measured in radial apertures (second convolution, all radii) + + * see note above on Standard Deviation + * radius 10 is not measured or reported in the CMF + + StackPetrosian : petrosian analysis data + * xpetCf : fill factor is mis-calculated in psphot (consistently > 1.0)