This is information for ippToPsps for ingesting the PV2 version of the 3pi database. Substantial changes have been made.

Batch types

There are now 8 types of batches, currently based off of SchemaNew16.0.tar - there are only minor corrections of typos / datatypes / addition of gcobjID

2 digit	type	schema
IN	init batch	ippToPspsPV2_IN
P2	single detections	ippToPspsPV2_P2
ST	stack	ippToPspsPV2_ST
OB	object	ippToPspsPV2_OB
FW	forced warp measurement	ippToPspsPV2_FW
FO	forced object	ippToPspsPV2_FO
DF	diff detection	ippToPspsPV2_DF
DO	diff object	ippToPspsPV2_DO

Schema Manipulation

There are multiple versions / types of the schema

• (schemaNewXX.0.tar ) - These are not complete ( the filters are not expanded), there is extra text, units/datatypes may not be consistent. This is what Heather works from when editing the VO Tables. The column names and orders are definitely correct, use caution with definitions/datatypes/etc.

• VO Tables - What ipptopsps uses - Heather has edited them to contain all the new types of batches, edited the descriptions to make them more accurate, this is the definitive list of everything.

• wiki /csv file - generated from a parsing script using VO Tables as input. These are what conrad/thomas work from

Conversion of datatypes

The parsing script takes the datatypes in the votables (which is what ipptopsps uses in jython) and converts them to microsoft. This is the conversion table:

jython datatype	jython array size	Microsofted
float	1	REAL 4
double	1	FLOAT 8
int	1	INT 4
short	1	SMALLINT 2
unsignedByte	1	TINYINT 1
long	1	BIGINT 8
char	num	VARCHAR(num)

First iteration of new schema and loading

First iteration is loading, these are the following datastores:

• PSPS_SAS_NEW_IN
• PSPS_SAS_NEW_OB
• PSPS_SAS_NEW_P2
• PSPS_SAS_NEW_ST

The idea is to make it easy for Conrad/Thomas to work on one type of batch at a time, and also to be able to easily remove/replace the batches as they are improved. The current SAS we are loading is the dvo for SAS32 - because of this, we are unable to add the new tables related to diffs and forcedwarps.

Init Batch

• should be unchanged ( I think?)

Object Batches

• currently loaded on the datastore with these missing:

• ObjectThin
  • new and unfilled objName, dvoRegionID
  • new and unfillable gcobjid, raStack, decStack, raStackErr, decStackErr, raMeanStd, decMeanStd, nStackDetections, nStackObjectRows
  • remove? skycellID
• ObjectMean
  • new and unfillable gcobjid, ()qfPerfect, ()nIncKronMag, MeanPSFMagStd, MinPSFMag,MaxPSFMag,MeanKronMag,nIncKronMag,MeanKronMagStd,MeanApMag,MeanApMagErr,nIncApMag,MeanApMagStd

P2 Batches

• FrameMeta
  • no changes from previous schema, all columns filled
• ImageMeta
  • new (unfilled) columns - momentXX, momentXY, momentYY, momentM3C, momentM3S, momentM4C, momentM4S, momentR1, momentRH
• Detection
  • new and unfillable - gcobjID (requires new dvo), dvoRegionID (impossible, mark for deletion?)
  • new and unfilled - momentM3C, momentM3S, momentM4C, momentM4S, momentR1, momentRH, telluricExt, pltScale, posAngle (all from smf file?)
  • needs to be filled - randomDetID
  • unfilled on both new and old - kronRad, kronRadErr, apFluxErr

Stack Batches

• StackMeta
  • filterID needs to be filled
  • unfilled on both new and old - magSat, analVer, completMag, astroScat, photoScat, nAstroRef, nPhref, photoColor, calibModNum
• StackToImage
  • no changes from previous schema, all columns filled
• StackObject (formerly StackDetection)
  • ippObjID - needs to be dropped, exploded to 5 filters, and filled.
  • unfilled on both new and old - ()apFluxErr
  • needs to be filled - skyCellID, randomStackObjID,
  • needs to be filled but needs verification from Gene: ()ApMagErr, ()KronMag, ()KronMagErr
  • new and unfillable (needs new dvo or something in cmf file): gcobjID, stackDetectRowID, primaryDetection, bestDetection, ()psfCore, ()ApFillFac
  • unfillable (dvoRegionID) - mark for deletion?
• StackApFlx
  • ippObjID - needs to be dropped, exploded to 5 filters, and filled.
  • needs to be filled - skyCellID, randomStackObjID, surveyID
  • new and unfillable (needs new dvo or something in cmf file): gcobjID, stackDetectRowID, primaryDetection, bestDetection
  • unfillable (dvoRegionID) - mark for deletion?
• StackModelFit
  • ippObjID - needs to be dropped, exploded to 5 filters, and filled.
  • needs to be filled - skyCellID, randomStackObjID, surveyID
  • new and unfillable (needs new dvo or something in cmf file): gcobjID, stackDetectRowID, primaryDetection, bestDetection
  • unfillable (dvoRegionID) - mark for deletion?
• StackModelFit
  • null and unfillable - gcobjid, stackdetectRowID
  • unfilled () logC, ()logA, ()clump
  • unfilled filter (grizy), model (deV,ser,exp)
    • (filter)(model)RadiusErr, (filter)(model)AbErr, (filter)(model)PhiErr, (filter)ra(model)Off, (filter)dec(model)Off, (filter)ra(model)OffErr, (filter)dec(model)OffErr,(filter)(model)Cf, (filter)(model)likelihood
  • unfilled (filter)serNuErr
  • unfillable (dvoRegionID) - mark for deletion?