Bill's Notes on the SAS 39 IPP to psps batches
back up to ippToPspsSAS39BillsNotes
Bill looked at a sample of the IPP to PSPS batches posted on the data store. Below are some notes on some things that I noticed.
Comments on the various batch samples examined. I looked for -999 valued columns, zero valued columns that look suspicous and things like that.
Note: In the comments below a leading x in column name means that comment applies to columns for all filters in grizy. For example xQFPerfect.
PSPS_SAS39_OB "Object Batch"
Looks like these batches contain the data for the table MeanObject.
xQFPerfect is always -999 Do these columns make sense for an average object? Seems like this column is a detection level thing. Maybe an average would be useful?
HAF/EAM: we want the biggest value : need to check if xQFperfect is set in dvo
PSPS_SAS39_P2 "P2 Batch"
This contains the data for an individual exposure.
Extensions FrameMeta, 60 * (ImageMeta, Detection)
FrameMeta: 1 row
-999 columns:
~p1Recip, p2Recip, p3Recip~ -- replace those three with "md5sum" from smf
varchar not 8000 characters long
ImageMeta: 1 row -999 columns:
psfModelID
In IPP speak this whould be "3" which is the value for PS_MODEL_PS1_V1 which is listed in the cmf header. Here I suppose it should be in psps space. There is no corresponding value in the FitModel table. Is the intent that it should be PS_MODEL_PSF (1) but. That doesn't allow for the possibility for using a different PSF model type in future releases.
HAF/EAM/KCC: need a table of the models we have used ( fill in FitModel in Metadata Tables) -- fill in fit model, make the cmf have the correct value. SMF header has PSFMODEL keyword
detrend4-8
I think that there are a couple of other detrend types that are used in the processing could be saved here if those would be interesting.
HAF/EAM/KCC: add a few more detrends: NOISEMAP NONLIN VIDEODARK 7/8 would be ignored
0 valued columns
should DataRelease be 0? Definitely not in PV3 I think. calibModNum. Maybe we should start with 1 instead of 0 for first calibration?
HAF: that's my fault (should be 39 for most sas39 batches) - we should call it 3 for PV3 EAM/KCC dataRelease -> processingVersion (all tables)
Detection: many rows of course -999 columns: None 0 valued columns: dataRelease
PSPS_SAS39_DF "Diff Batch"
Extensions: Primary Array, NSkycells * (DiffMeta, DiffToImage, DiffDetection)
These 2 items are more of a comment on the schema. Or perhaps I am missing something important. Where is the ipp diff_id ?
Looks like the diffs for SAS 39 were type 2: warp-stack
DiffMeta It looks like the diffMeta has ImageIDs for the positive and negative images. But PSPS ImageIDs are for chips. Diff inputs are for skycells so there is a many to one problem.
What are posImageID and negImageID ?
HAF/EAM/KCC needs more documentation on posImageID/negImageID warp_id (add postage stamp in there) - need WarpToImage table
NEEDS REVIEW: Ok it looks like posImageID is the warp_id for the diffRun and negImageID looks like it's trying to be the stack_id of the template.
However the skycell_ids aren't SAS. They seem to pv2 stacks (which were recently set to be cleaned). Or maybe I'm lost.
HAF/EAM/KCC not lost, that was done on purpose for broader time ranges on SAS39. (sas39 warp - 3pipv2 stack)
DiffToImage Perhaps this table resolves the issue. It looks like all of these DiffToImage tables are the same. It gives a list of the input ImageMetas.
DiffDetection -999 columns. There are several. Perhaps they are things that aren't measured for diffs. Needs more study.
HAF/EAM/KCC need a list Bill: Here it is.
psfMajorFWHM psfMinorFWHM psfCore momentM3C momentM3S momentM4C momentM4S appFillF infoFlag3 (dataRelease = 38)
PSPS_SAS39_DO "Diff Object Batch"
Extensions: Primary Array, DiffDetObject
I don't know what a diff object is? Diff detections matched to each other from different diffRuns? Looks like it. Must be created in DVO. No attempted match to PSPS objects? Sounds compute intensive I guess.
EAM/HAF/KCC: the association between diff detection and the big object table is of subjective use (depends on the science case)
-999 columns: htmID zoneID cx,cy,cz lamda, beta, l, b, HAF/EAM/KCC : computed by PSPS
-999 column: xQfPerfect HAF/EAM/KCC: should be max qfperfect, look at dvo
PSPS_SAS39_ST "Stack Batch" one per skycell
Extensions 5 StackMeta (5 filters) StackObjectThin StackObjectAttributes And "the various other stack tables: model fits, aperture fluxes, etc" StackToImage: list of images that went into the stack. (Bill, recall ImageID == exp_id * 100 + (int) class_id)
StackMeta Description of each stack (stackMetaID == stackRun.stack_id) -999 columns: magSat, competMag, astroScat, photoScat, nAstro, nPhoRef,
HAF/KCC/EAM
- magsat - we don't calculate it, drop it (stack / detection/ diff)
- completMag - detectionThreshold --> DETEFF.MAGREF (instrumental mag, needs zp / exp correction for stack and detection and diff)
- astroScat - AST_CDX, AST_CDY calc= sqrt(a2+b2) - this is not the astrometric error - a measurement of the calibration but not definitive
- photoScat - ZPT_ERR - describe it better: the scatter of the calibration stars
- nAstro - NASTRO : change to nRefStars
- nPhoRef - (delete)
- add set of similar names (same information, from relexp/relstack) recalAstroScat, recalPhotoScat, recalNAstroStars, recalNPhotoStars
- calibModNum - HAF to remove from schema
StackObjectThin Magnitudes only. No fluxes -999 columns:
stackDetectRowID: which of 5 possible stack detections this row refers to. Up to 3 secondary, 1 best. I think I heard that there were problems setting the primary flag in SAS 39.
0 valued columns:
primary bestDetection xInfoFlag2 !!!!!!!!!!! Stack detections very hard to use without this flag! (HAF to investigate, should be value from cmf) Bill: in some of the files the values are -999 not zero.
StackObjectAttributes Fluxes only (in janskys) no magnitudes.
Has the detection attributes, but not the flags.
EAM/HAF/KCC counts per seconds, we have zp as well. schema description change - forced mean objct is in janskys (can't be in adus) everything that is a flux - > reported in a jansky instead of adu - need conversion from adu to jansky from EAM
galaxy parameters
-999 columns xlogC xlogA stackDetectedRowID
StackModelFitExp, DeV, Ser'
model fit data
-999 columns stackDetectRowID xExpCf, xDevCf, xSerCf
petrosian extension stuff
-999 columns stackDetectRowID 0 valued primaryDetection bestDetection
results for aperture flux measurements for radius 5, 6, and 7 for the 3 images Looks like a choice of the "best" stack for each object.
-999 columns: stackDetectRowID 0 columns: primaryDetection BestDetection
StackApFlxExGalUnc,Con6, Con8
This has all of the app flux radii measurements. Other radii. Non best stacks. (what did I mean here?)
PSPS_SAS_39_FO "Forced object"
ForcedMeanObject -999 columns: xGamma - to be dropped (EAM)
Forced MeanLensing xpsfE1 xpsfE2 are -999 EAM thinks it's being set. There is no columns gpsfE1 or gpsfE2 Is this intentional? Not intentional HAF to investigate EAM to investigate
PSPS_SAS_39_FW "Forced object" Batches are by skycell/filter and thus by fullForceRun
Extensions: ForcedWarpMeta
skycellID has format 1405.07. For consistency with the rest of the schema it should probably be 2 columns projectionCell and skycellID HAF: OK i'll fix it
-999 columns photoCalID analVer astroScat photoScat nAstro nPoRef
photoCalID ??? analVer should be svn number astroScat -- drop photoScat -- same as in stack nAstro -- drop nPhoRef -- same as in stack
0 valued columns magSat -- drop completMag -- as done for stack/p2 (and grab from relexp (fullforce) info for photoscat/ nphoref)
-999 columns psfLikelihood ( repeat as done for Detections)
ForcedWarpExtended No comments
ForcedWarpLensing No comments
ForcedWarpToImage what is forcedWarpID ? It doesn't seem to be an ff_id (HAF: yes, need to add more information) imageID is a pointer to the input's ImageMetas (IPP's chipProcessedImfile) My comments above on the diff's tables may apply although here I did not see anything that specifies the warp.
Here psModelID is a string PS_MODEL_PS1_V1 ( HAF: same as other stuff)
