This is information for ippToPsps for ingesting the PV2 version of the 3pi database.  Substantial changes have been made.


== Batch types ==

There are now 8 types of batches, currently based off of SchemaNew16.0.tar  - there are only minor corrections of typos / datatypes / addition of gcobjID 
 
|| 2 digit || type  || schema ||
||IN || init batch ||  [wiki:ippToPspsPV2_IN]  ||
|| P2 ||single detections|| [wiki:ippToPspsPV2_P2]  ||
||ST ||stack|| [wiki:ippToPspsPV2_ST]   ||
||OB ||object||  [wiki:ippToPspsPV2_OB]  ||
||FW || forced warp measurement|| [wiki:ippToPspsPV2_FW]  ||
||FO || forced object|| [wiki:ippToPspsPV2_FO]  ||
||DF ||diff detection||  [wiki:ippToPspsPV2_DF]  ||
||DO || diff object|| [wiki:ippToPspsPV2_DO]   ||


== Schema Manipulation ==

There are multiple versions / types of the schema

 * (schemaNewXX.0.tar )  -  These are not complete ( the filters are not expanded), there is extra text, units/datatypes may not be consistent.  This is what Heather works from when editing the VO Tables. The column names and orders are definitely correct, use caution with definitions/datatypes/etc.

 * VO Tables  - What ipptopsps uses - Heather has edited them to contain all the new types of batches, edited the descriptions to make them more accurate, this is the definitive list of everything.

 * wiki /csv file - generated from a parsing script using VO Tables as input.  These are what conrad/thomas work from

=== Conversion of datatypes ===

The parsing script takes the datatypes in the votables (which is what ipptopsps uses in jython) and converts them to microsoft. This is the conversion table:

|| jython datatype   || jython array size  ||     Microsofted  ||
|| float       ||  1 || REAL 4 ||
|| double   || 1 || FLOAT 8 ||
||int ||1|| INT 4||
||short ||1|| SMALLINT 2||
||unsignedByte || 1 || TINYINT 1||
||long || 1 || BIGINT 8||
||char || num || VARCHAR(num) ||

== First iteration of new schema and loading ==

First iteration is loading, these are the following datastores:

* PSPS_SAS_NEW_IN
* PSPS_SAS_NEW_OB
* PSPS_SAS_NEW_P2
* PSPS_SAS_NEW_ST

The idea is to make it easy for Conrad/Thomas to work on one type of batch at a time, and also to be able to easily remove/replace the batches as they are improved.  The current SAS we are loading is the dvo for SAS32 - because of this, we are unable to add the new tables related to diffs and forcedwarps.

=== Init Batch ===

 * should be unchanged ( I think?)

=== Object Batches ===

 * currently loaded on the datastore with these missing:

 * ObjectThin
   * new and unfilled objName, dvoRegionID
   * new and unfillable gcobjid, raStack, decStack, raStackErr, decStackErr, raMeanStd, decMeanStd, nStackDetections, nStackObjectRows
   * remove? skycellID
 * ObjectMean
   * new and unfillable gcobjid, ()qfPerfect, ()nIncKronMag, MeanPSFMagStd, MinPSFMag,MaxPSFMag,MeanKronMag,nIncKronMag,MeanKronMagStd,MeanApMag,MeanApMagErr,nIncApMag,MeanApMagStd   

=== P2 Batches ===

 * FrameMeta 
   * no changes from previous schema, all columns filled
 * ImageMeta
   * new (unfilled) columns - momentXX, momentXY, momentYY, momentM3C, momentM3S, momentM4C, momentM4S, momentR1, momentRH
 * Detection
   * new and unfillable - gcobjID (requires new dvo), dvoRegionID (impossible, mark for deletion?)
   * new and unfilled - momentM3C, momentM3S, momentM4C, momentM4S, momentR1, momentRH, telluricExt, pltScale, posAngle (all from smf file?)
   * needs to be filled - randomDetID
   * unfilled on both new and old - kronRad, kronRadErr, apFluxErr

=== Stack Batches ===

 * StackMeta
  * filterID needs to be filled
  * unfilled on both new and old - magSat, analVer, completMag, astroScat, photoScat, nAstroRef, nPhref, photoColor, calibModNum
 * StackToImage
  * no changes from previous schema, all columns filled
 * StackObject (formerly StackDetection)
  * ippObjID - needs to be dropped, exploded to 5 filters, and filled.
  * unfilled on both new and old - ()apFluxErr
  * needs to be filled - skyCellID, randomStackObjID, 
  * needs to be filled but needs verification from Gene: ()ApMagErr, ()KronMag, ()KronMagErr 
  * new and unfillable (needs new dvo or something in cmf file): gcobjID, stackDetectRowID, primaryDetection, bestDetection, ()psfCore, ()ApFillFac
  * unfillable (dvoRegionID) - mark for deletion?
 * StackApFlx
  * ippObjID - needs to be dropped, exploded to 5 filters, and filled.
  * needs to be filled - skyCellID, randomStackObjID, surveyID
  * new and unfillable (needs new dvo or something in cmf file): gcobjID, stackDetectRowID, primaryDetection, bestDetection
  * unfillable (dvoRegionID) - mark for deletion?
 * StackModelFit
  * ippObjID - needs to be dropped, exploded to 5 filters, and filled.
  * needs to be filled - skyCellID, randomStackObjID, surveyID
  * new and unfillable (needs new dvo or something in cmf file): gcobjID, stackDetectRowID, primaryDetection, bestDetection
  * unfillable (dvoRegionID) - mark for deletion?
 * StackModelFit
  * null and unfillable - gcobjid, stackdetectRowID
  * unfilled () logC, ()logA, ()clump
  * unfilled filter (grizy), model (deV,ser,exp) 
    * (filter)(model)RadiusErr, (filter)(model)AbErr, (filter)(model)PhiErr, (filter)ra(model)Off, (filter)dec(model)Off, (filter)ra(model)OffErr, (filter)dec(model)OffErr,(filter)(model)Cf, (filter)(model)likelihood
  * unfilled (filter)serNuErr 
  * unfillable (dvoRegionID) - mark for deletion?