Bill's Notes on the SAS 39 IPP to psps batches

Bill looked at a sample of the IPP to PSPS batches posted on the data store. Below are some notes on some things that I noticed.

Comments on the various batch samples examined. I looked for -999 valued columns, zero valued columns that look suspicous and things like that.

Note: In the comments below a leading x in column name means that comment applies to columns for all filters in grizy. For example xQFPerfect.

---

PSPS_SAS39_OB "Object Batch"

Looks like these batches contain the data for the table MeanObject.

xQFPerfect is always -999 Do these columns make sense for an average object? Seems like this column is a detection level thing. Maybe an average would be useful?

HAF/EAM: we want the biggest value : need to check if xQFperfect is set in dvo

---

PSPS_SAS39_P2 "P2 Batch"

This contains the data for an individual exposure.

Extensions FrameMeta, 60 * (ImageMeta, Detection)

FrameMeta: 1 row -999 columns: ~p1Recip, p2Recip, p3Recip~ -- replace those three with "md5sum" from smf varchar not 8000 characters long

ImageMeta: 1 row -999 columns:

psfModelID

In IPP speak this whould be "3" which is the value for PS_MODEL_PS1_V1 which is listed in the cmf header. Here I suppose it should be in psps space. There is no corresponding value in the FitModel table. Is the intent that it should be PS_MODEL_PSF (1) but. That doesn't allow for the possibility for using a different PSF model type in future releases.

HAF/EAM/KCC: need a table of the models we have used ( fill in FitModel in Metadata Tables) -- fill in fit model, make the cmf have the correct value. SMF header has PSFMODEL keyword

detrend4-8

I think that there are a couple of other detrend types that are used in the processing could be saved here if those would be interesting.

HAF/EAM/KCC: add a few more detrends: NOISEMAP NONLIN VIDEODARK 7/8 would be ignored

0 valued columns

should DataRelease be 0? Definitely not in PV3 I think. calibModNum. Maybe we should start with 1 instead of 0 for first calibration?

HAF: that's my fault (should be 39 for most sas39 batches) - we should call it 3 for PV3 EAM/KCC dataRelease -> processingVersion (all tables)

Detection: many rows of course -999 columns: None 0 valued columns: dataRelease

---

PSPS_SAS39_DF "Diff Batch"

Extensions: Primary Array, NSkycells * (DiffMeta, DiffToImage, DiffDetection)

These 2 items are more of a comment on the schema. Or perhaps I am missing something important. Where is the ipp diff_id ?

Looks like the diffs for SAS 39 were type 2: warp-stack

DiffMeta It looks like the diffMeta has ImageIDs for the positive and negative images. But PSPS ImageIDs are for chips. Diff inputs are for skycells so there is a many to one problem.

What are posImageID and negImageID ?

HAF/EAM/KCC needs more documentation on posImageID/negImageID warp_id (add postage stamp in there) - need WarpToImage table

NEEDS REVIEW: Ok it looks like posImageID is the warp_id for the diffRun and negImageID looks like it's trying to be the stack_id of the template.

However the skycell_ids aren't SAS. They seem to pv2 stacks (which were recently set to be cleaned). Or maybe I'm lost.

HAF/EAM/KCC not lost, that was done on purpose for broader time ranges on SAS39. (sas39 warp - 3pipv2 stack)

DiffToImage Perhaps this table resolves the issue. It looks like all of these DiffToImage tables are the same. It gives a list of the input ImageMetas.

DiffDetection -999 columns. There are several. Perhaps they are things that aren't measured for diffs. Needs more study.

HAF/EAM/KCC need a list

---

PSPS_SAS39_DO "Diff Object Batch"

Extensions: Primary Array, DiffDetObject

I don't know what a diff object is? Diff detections matched to each other from different diffRuns? Looks like it. Must be created in DVO. No attempted match to PSPS objects? Sounds compute intensive I guess.

EAM/HAF/KCC: the association between diff detection and the big object table is of subjective use (depends on the science case)

-999 columns: htmID zoneID cx,cy,cz lamda, beta, l, b, HAF/EAM/KCC : computed by PSPS

-999 column: xQfPerfect HAF/EAM/KCC: should be max qfperfect, look at dvo

---

PSPS_SAS39_ST "Stack Batch" one per skycell

Extensions 5 StackMeta (5 filters) StackObjectThin StackObjectAttributes And "the various other stack tables: model fits, aperture fluxes, etc" StackToImage: list of images that went into the stack. (Bill, recall ImageID == exp_id * 100 + (int) class_id)

StackMeta Description of each stack (stackMetaID == stackRun.stack_id) -999 columns: magSat, competMag, astroScat, photoScat, nAstro, nPhoRef,

HAF/KCC/EAM

• magsat - we don't calculate it, drop it (stack / detection/ diff)
• completMag - detectionThreshold --> DETEFF.MAGREF (instrumental mag, needs zp / exp correction for stack and detection and diff)
• astroScat - AST_CDX, AST_CDY calc= sqrt(a^2+b2) - this is not the astrometric error - a measurement of the calibration but not definitive
• photoScat - ZPT_ERR - describe it better: the scatter of the calibration stars
• nAstro - NASTRO : change to nRefStars
• nPhoRef - (delete)
• add set of similar names (same information, from relexp/relstack) recalAstroScat, recalPhotoScat, recalNAstroStars, recalNPhotoStars
• calibModNum - HAF to remove from schema

StackObjectThin Magnitudes only. No fluxes -999 columns:

stackDetectRowID: which of 5 possible stack detections this row refers to. Up to 3 secondary, 1 best. I think I heard that there were problems setting the primary flag in SAS 39.

0 valued columns:

primary bestDetection xInfoFlag2 !!!!!!!!!!! Stack detections very hard to use without this flag! (HAF to investigate, should be value from cmf)

StackObjectAttributes Fluxes only (in janskys) no magnitudes.

Has the detection attributes, but not the flags.

EAM/HAF/KCC counts per seconds, we have zp as well. schema description change - forced mean objct is in janskys (can't be in adus) everything that is a flux - > reported in a jansky instead of adu - need conversion from adu to jansky from EAM

StackModelFitExtra

galaxy parameters

StackModelFitExp, DeV, Ser

model fit data

StackPetrosian

petrosian extension stuff

StackApFlux

results for aperture flux measurements for radius 5, 6, and 7 for the 3 images Looks like a choice of the "best" stack for each object.

StackApFlxExGalUnc,Con6, Con8

This has all of the app flux radii measurements. Other radii. Non best stacks. (what did I mean here?)

---

PSPS_SAS_39_FO "Forced object"

ForcedMeanObject -999 columns: xGamma - to be dropped (EAM)

Forced MeanLensing xpsfE1 xpsfE2 are -999 EAM thinks it's being set. There is no columns gpsfE1 or gpsfE2 Is this intentional? Not intentional HAF to investigate EAM to investigate

---

PSPS_SAS_39_FW "Forced object" Batches are by skycell/filter and thus by fullForceRun

Extensions: ForcedWarpMeta

skycellID has format 1405.07. For consistency with the rest of the schema it should probably be 2 columns projectionCell and skycellID HAF: OK i'll fix it

-999 columns photoCalID analVer astroScat photoScat nAstro nPoRef

photoCalID ??? analVer should be svn number astroScat -- drop photoScat -- same as in stack nAstro -- drop nPhoRef -- same as in stack

0 valued columns magSat -- drop completMag -- as done for stack/p2 (and grab from relexp (fullforce) info for photoscat/ nphoref)

ForcedWarpMeasurement

-999 columns psfLikelihood ( repeat as done for Detections)

ForcedWarpExtended No comments

ForcedWarpLensing No comments

ForcedWarpToImage what is forcedWarpID ? It doesn't seem to be an ff_id (HAF: yes, need to add more information) imageID is a pointer to the input's ImageMetas (IPP's chipProcessedImfile) My comments above on the diff's tables may apply although here I did not see anything that specifies the warp.

Here psModelID is a string PS_MODEL_PS1_V1 ( HAF: same as other stuff)