Changes between Initial Version and Version 1 of BillsSas39BatchesNotes

Timestamp:

Aug 14, 2015, 12:08:05 PM (11 years ago)

Author:

bills

Comment:

--

BillsSas39BatchesNotes

@@ +1 @@
+Bill looked at a sample of the IPP to PSPS batches posted on the data store. Below are some notes on some things that I noticed.
+{{{
+Comments on the various batch samples examined. I looked for -999 valued columns, zero valued columns that
+look suspicous and things like that.
+
+Note: In the comments below a leading x in column name means that comment applies to columns for all
+filters in grizy.  For example xQFPerfect.
+
+PSPS_SAS39_OB "Object Batch"
+-----------------------------
+
+Looks like these batches contain the data for the table MeanObject.
+
+xQFPerfect is always -999  Do these columns make sense for an average object? 
+Seems like this column is a detection level thing. Maybe an average would be useful?
+
+
+PSPS_SAS39_P2 "P2 Batch"
+------------------------------
+This contains the data for an individual exposure.
+
+Extensions
+FrameMeta, 60 * (ImageMeta, Detection)
+
+FrameMeta: 1 row
+-999 columns:
+p1Recip, p2Recip, p3Recip
+
+ImageMeta: 1 row 
+-999 columns:
+
+  psfModelID 
+    In IPP speak this whould be "3" which is the value for PS_MODEL_PS1_V1 which is listed in the cmf header. 
+    Here I suppose it should be in psps space.  There is no corresponding value in the FitModel table.
+    Is the intent that it should be PS_MODEL_PSF (1) but.  That doesn't allow for the possibility 
+    for using a different PSF model type in future releases. 
+  detrend4-8
+    I think that there are a couple of other detrend types that are used in the processing could be saved 
+    here if those would be interesting. 
+
+0 valued columns
+  should DataRelease be 0? Definitely not in PV3 I think.
+  calibModNum. Maybe we should start with 1 instead of 0 for first calibration?
+
+Detection: many rows of course
+-999 columns: None
+0 valued columns: dataRelease
+
+
+PSPS_SAS39_DF "Diff Batch"
+------------------------------
+Extensions: Primary Array, NSkycells * (DiffMeta, DiffToImage, DiffDetection)
+
+These 2 items are more of a comment on the schema. Or perhaps I am missing something important.
+Where is the ipp diff_id ?
+
+Looks like the diffs for SAS 39 were type 2: warp-stack
+
+DiffMeta
+It looks like the diffMeta has ImageIDs for the positive and negative images. But PSPS ImageIDs are for chips.
+Diff inputs are for skycells so there is a many to one problem. 
+
+What are posImageID and negImageID ?
+
+NEEDS REVIEW:
+Ok it looks like posImageID is the warp_id for the diffRun and negImageID looks like it's trying to be
+the stack_id of the template.
+
+However the skycell_ids aren't SAS. They seem to pv2 stacks (which were recently set to be cleaned).
+Or maybe I'm lost.
+
+DiffToImage
+Perhaps this table resolves the issue. It looks like all of these DiffToImage tables are the same. It gives a list
+of the input ImageMetas. 
+
+Detection
+-999 columns. There are several. Perhaps they are things that aren't measured for diffs. Needs more study.
+
+
+PSPS_SAS39_DO "Diff Object Batch"
+---------------------------------
+Extensions: Primary Array, DiffDetObject
+
+I don't know what a diff object is? 
+Diff detections matched to each other from different diffRuns? Looks like it. Must be created in DVO.
+No attempted match to PSPS objects? Sounds compute intensive I guess.
+
+-999 columns:
+htmID zoneID cx,cy,cz lamda, beta, l, b, xQfPerfect
+
+
+PSPS_SAS39_ST "Stack Batch" one per skycell
+-------------------------------------------
+Extensions
+----------
+5 StackMeta (5 filters)
+StackObjectThin
+StackObjectAttributes
+And "the various other stack tables: model fits, aperture fluxes, etc"
+StackToImage: list of images that went into the stack. (Bill, recall ImageID == exp_id * 100 + (int) class_id)
+
+StackMeta
+Description of each stack (stackMetaID == stackRun.stack_id)
+-999 columns: magSat, competMag, astroScat, photoScat, nAstro, nPhoRef, calibModNum
+
+StackObjectThin Magnitudes only. No fluxes
+-999 columns: 
+  stackDetectRowID: which of 5 possible stack detections this row refers to. Up to 3 secondary, 1 best.
+  I think I heard that there were problems setting the primary flag in SAS 39.
+
+0 valued columns:
+    primary
+    bestDetection
+    xInfoFlag2 !!!!!!!!!!! Stack detections very hard to use without this flag!
+
+StackObjectAttributes  Fluxes only (in janskys) no magnitudes.
+    Has the detection attributes, but not the flags.
+
+StackModelFitExtra
+    galaxy parameters
+
+StackModelFitExp, DeV, Ser
+    model fit data
+
+StackPetrosian
+    petrosian extension stuff
+
+StackApFlux
+    results for aperture flux measurements for radius 5, 6, and 7 for the 3 images
+    Looks like a choice of the "best" stack for each object.
+
+StackApFlxExGalUnc,Con6, Con8
+    This has all of the app flux radii measurements. Other radii. Non best stacks. (what did I mean here?)
+--------------------------------------------------------------------------------------------------
+PSPS_SAS_39_FO "Forced object"
+
+ForcedMeanObject
+-999 columns:
+xGamma
+
+Forced MeanLensing
+xpsfE1 xpsfE2   are -999 
+There is no columns gpsfE1 or gpsfE2 Is this intentional?
+
+--------------------------------------------------------------------------------------------------
+PSPS_SAS_39_FW "Forced object"
+Batches are by skycell/filter and thus by fullForceRun
+
+Extensions:
+ForcedWarpMeta
+--------------
+skycellID has format 1405.07. For consistency with the rest of the schema it should probably be 2 columns
+projectionCell and skycellID
+
+
+-999 columns
+photoCalID analVer astroScat photoScat nAstro nPoRef
+
+0 valued columns
+magSat
+completMag
+
+
+ForcedWarpMeasurement
+---------------------
+
+-999 columns
+psfLikelihood
+
+ForcedWarpExtended
+---------------------
+No comments
+ForcedWarpLensing
+---------------------
+No comments
+
+ForcedWarpToImage
+-----------------
+what is forcedWarpID ? It doesn't seem to be an ff_id
+imageID is a pointer to the input's ImageMetas (IPP's chipProcessedImfile)
+My comments above on the diff's tables may apply although here I did not see anything that specifies the warp.
+
+Here psModelID is a string PS_MODEL_PS1_V1
+
+
+
+
+
+}}}