| Version 8 (modified by , 14 years ago) ( diff ) |
|---|
IPP Data Release Management
The IPP does not have a good way of keeping track of what data is intended for or has been released to the PS1SC community. Up until now this has not been a big problem, but several issues have recently led us to believe that some new functionality is necessary to make it easy for users to get access to IPP data products that meet their expectations.
The specific goals are
- To record which versions of the data products have been (and should be) "released" to the community through the various distribution channels.
- To allow users to get the "right" versions of the data in their requests to the postage stamp server
- To record within the IPP domain the results of various re-calibrations and re-processings that may occur
These goals will be met with some enhancements to the IPP data base schema and through some lightweight changes to the procedures for processing and releasing data.
IPP Release Channels
Data is released through several channels.
- The IPP Distribution data store
- The Published Science Products Subsystem (PSPS)
- The IPP Postage Stamp Server
- DVO catalogs
The data is the results of runs through the various stages of the IPP pipeline. For the most part the IPP does not have a very good record of which data went into the various releases. For example, given a single frame detection in the IPP database, there is no direct method to determine which version of the processed pixels the detections were measured on. Making the answer to this and other similar questions easy to obtain is the primary purpose of this project.
IPP Data Products
There are three categories of data products to be covered by the IPP release mechanism
- Single Frame Data Products
- Stacked Image Data Products
- Difference Image Data Products
Single Frame Results
Single exposures move through the IPP pipeline in a number of stages. The data products of interest to end users are the results of the chip and camera stage (focal plane coordinates) and the warp stage. The end results are two sets of images and catalogs of detected sources. The catalogs are then imported into the DVO system and eventually into a PSPS data base. Exposures are also queued for difference processing which will be discussed separately.
Nightly Science
On the night that the exposures are acquired they are run through the IPP "Nightly Science" processing. The resulting catalogs are posted to file sets on the IPP Data Store web server. For certain exposures warped images are placed on the data store as well. This is the first "release" for a given exposure.
The catalog files (smf and cmf) currently remain on the data store indefinitely. For various reasons this needs to be changed. A new system for on demand access to these files is under development. The image files posted to the data store are removed after some time to conserve space.
DVO
Occasionally the DVO catalogs are made available to the community. These consist of calibrated measurements and object associations of individual exposures. We currently have no method of directly tracking in the IPP database whether or not the measurements for a particular exposure were "released" in a given DVO catalog or for remembering the results of the calibrations performed by the DVO programs. This information can be figured out from the DVO tables, but since they are subject to change this information can become "stale".
"Grand" Reprocessing
As time goes on and the algorithms of the IPP improve and the exposures are reprocessed from scratch. This is commonly known as the grand reprocessing. Internally the IPP team refers to this process for the 3Pi survey as "Large Area Processing". For the medium deep fields, the terminology has changed over time but the concepts are somewhat similar.
The results of this processing are added to a new DVO database and from there into PSPS. We currently do not have an efficient method of tracking which sets of processed pixels went into a given PSPS database.
Various Test runs
We currently have no way of tracking which of the various processing runs are intended for end user use and those that are the results of testing. The postage stamp server currently attempts to work around this problem but the results have been mixed. By specifically identifying the data that should be released it will be easy to indentify those runs that should not be released as well.
Stacked Images
The warped images are combined to form stacked images. While for single frames the logical unit is the exposure, stacks are performed on areas of the sky we call skycells. A skycell has a tessellation ID and a skycell number. For example (RINGS.V3, skycell.1406.016) or (MD09.V3, skycell.042)
Three types of stacks have been identified
- Nightly Stacks - constructed from exposures on a single night
- Reference Stacks - constructed from exposures with good image quality that provide good coverage of a region of the sky
- Deep Stacks - constructed from virtually all "good" exposures for a given area of the sky
For the medium deep surveys all three stack types are used. Deep Stacks provide the greatest sensitivity, while reference stacks are used as templates for difference imaging and science that requires the highest resolution. Nightly stacks are used for transient detection.
For the 3PI survey the stacks are produced in the LAP processes. For the purposes of this discussion they are considered "Deep Stacks" even though they are made from significantly fewer input exposures.
Right now the various "releases" of stacks are managed in an ad hoc manner. It is very difficult for a user without access to the IPP database and detailed knowlege of the schema to track what data corresponds to a particular release.
Difference Images
Management of difference images is a subject for further research and will be discussed later.
Definition of an IPP release
A collection of data "released" to the PS1SC community will be represented in the IPP database by a row in 2 new tables called "ippRelease" and "survey". Here are the proposed definitions
mysql> describe survey; +-------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------+-------+ | surveyID | tinyint(8) | NO | PRI | 0 | | | name | varchar(16) | YES | MUL | NULL | | | description | varchar(255) | YES | | NULL | | +-------------+--------------+------+-----+---------+-------+ 3 rows in set (0.00 sec) mysql> describe ippRelease; +-------------+-------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------+-------------+------+-----+---------+----------------+ | rel_id | smallint(6) | NO | PRI | NULL | auto_increment | | surveyID | tinyint(8) | YES | MUL | NULL | | | releaseName | varchar(16) | YES | MUL | NULL | | | state | varchar(16) | YES | MUL | NULL | | | dataRelease | tinyint(8) | YES | | NULL | | +-------------+-------------+------+-----+---------+----------------+ 5 rows in set (0.00 sec)
The survey table is a copy of the PSPS table Survey. Example entries are
+----------+------+------------------------------------+ | surveyID | name | description | +----------+------+------------------------------------+ | 0 | 3PI | PS1 3PI Survey | | 1 | MD01 | PS1 MD01 XMM-LSS-DXS 022224-043000 | +----------+------+------------------------------------+
The columns in ippRelease are
- rel_id primary key for the row
- surveyID for to the survey that the data in a particular release contains
- releaseName - human friendly identifier for the release
- state - current status of the release. Several values have been identified
- active - results for the release are currently available to the community
- pending - release identified, construction may have begun, but the results not yet ready to be available by default
- archive - the release contains the results of a previous processing of the data that has been superseded.
- drop - the results of the release have been abandoned and should be ignored
- dataRelease - If not null, the dataRelease value in the corresponding PSPS tables.
Some examples
+--------+----------+-------------+---------+-------------+ | rel_id | surveyID | releaseName | state | dataRelease | +--------+----------+-------------+---------+-------------+ | 1 | 0 | 3PI.nightly | active | NULL | | 2 | 0 | 3PI.GR0 | active | 0 | | 3 | 0 | 3PI.GR1 | pending | 1 | | 4 | 0 | SAS.V8 | archive | 8 | | 5 | 0 | SAS.V9 | active | 9 | +--------+----------+-------------+---------+-------------+
The 3PI.nightly release is active. Data is being currently added to it and is available to users who request data from that processing. Since these data are no longer being ingested into PSPS the dataRelease field is null.
3PI.GR0 refers to the version of LAP processing that is currently in the PSPS database PS1_3PI. Data corresponding to that release is identified by dataRelease 0, while 3PI.GR1 refers to the future LAP release which will begin construction soon. Once that release gets under way it's state will be set to active while 3PI.GR0 will be set to "archive". XXX: insure that PSPS uses a different dataRelease ...
The two SAS entries refer to PS1_SAS_08 and PS1_SAS_09.
Now we turn our attention to tables that record the actual contents of a release. Single frame results are represented by the table relExp (released exposure) while stacks are represented by relSkycell (released skycell). (We expect that in the future a similar table will be defined for difference stage data products).
Released Exposure
The relExp table encapsulates the data products that were used to construct the released data for a given exposure and the results of the calibration of the exposure. The proposed table definition is as follows
mysql> describe relExp; +------------+------------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +------------+------------------+------+-----+---------+----------------+ | relexp_id | bigint(20) | NO | PRI | NULL | auto_increment | Primary key for the table | rel_id | smallint(6) | YES | MUL | NULL | | The corresponding release | exp_id | bigint(20) | YES | | NULL | | The IPP exposure identifier | chip_id | bigint(20) | YES | | NULL | | id for chipRun | cam_id | bigint(20) | YES | | NULL | | id for camRun | warp_id | bigint(20) | YES | | NULL | | id for warpRun | add_id | bigint(20) | YES | | NULL | | id for addRun (addstar injest into DVO) | state | varchar(16) | YES | MUL | NULL | | state for this row see below | data_state | varchar(16) | YES | MUL | NULL | | see below | flags | int(10) unsigned | YES | | NULL | | | zpt_obs | float | YES | | NULL | | calibrated zero point and error when released | zpt_stdev | float | YES | | NULL | | | path_base | varchar(255) | YES | | NULL | | path to any supporting files | registered | datetime | YES | | NULL | | creation time for row | time_stamp | datetime | YES | | NULL | | time of last update to row +------------+------------------+------+-----+---------+----------------+ Note: this is a prototype. The actual table will use foreign key constraints where appropriate.
This table will allow us to easily identify for a given exposure and release the derived data products that were used to construct the results for the exposure.
Each time an exposure is released a new row will be added to the table.
Here are a couple of examples showing the releases for one of the SAS exposures:
mysql> mysql> select ippRelease.releaseName, relExp.* from ippRelease join relExp using(rel_id) where exp_id = 356773; +-------------+-----------+--------+--------+---------+--------+---------+--------+----------+------------+-------+---------+-----------+-----------+---------------------+---------------------+ | releaseName | relexp_id | rel_id | exp_id | chip_id | cam_id | warp_id | add_id | state | data_state | flags | zpt_obs | zpt_stdev | path_base | registered | time_stamp | +-------------+-----------+--------+--------+---------+--------+---------+--------+----------+------------+-------+---------+-----------+-----------+---------------------+---------------------+ | SAS.V8 | 1 | 4 | 356773 | 496671 | 473031 | 453844 | 883989 | archive | inpsps | 0 | 24.4298 | 0.042454 | NULL | 2012-09-19 15:16:36 | 2012-09-19 15:16:36 | | SAS.V9 | 621 | 5 | 356773 | 496671 | 473031 | 453844 | 888750 | released | inpsps | 0 | 24.4298 | 0.042454 | NULL | 2012-09-19 15:18:51 | 2012-09-19 15:18:51 | +-------------+-----------+--------+--------+---------+--------+---------+--------+----------+------------+-------+---------+-----------+-----------+---------------------+---------------------+ 2 rows in set (0.00 sec)
Note that the list of ids are the same except for the add_id. This is because SAS_v9 was constructed using the SAS_v8 pixels. For single frame exposures only the DVO injest and calibration was redone. This is indicated by the change in add_id.
state / data_state / flags
There are three columns related to the state of the released exposure. We haven't yet settled on which of these data are required to manage the releases. Information that we might want to record include
- state - active, drop, archive (note that the release state
- has the released exposure been imported into PSPS
- did the calibration apply ubercal. Perhaps: if so which version?
- was the data the result of a recalibration, relphot, relastro, something else ...
Life cycle of a relExp
Nightly science
We envision the following potential events in the history of a released exposure for the nightly science releases
- created during nightly science processing once the warp has been completed before the addRun is processed. State active.
- updated to include the add_id of the addRun
Grand Reprocessing
For the grand reprocessing, the relExp goes through two states
- row is created when addstar is performed. State set to pending.
- row is updated when ipptoPSPS is run. At that time the calibration data is updated with the resulting values and the state columns are changed to reflect the PSPS export. state => active
Separating the state into two steps allows the results of the re-processing to be made available prior to PSPS injest if the user requests data from the corresponding release. The data will not be the "default" for the given exposure until the calibration is complete at the time of PSPS ingest.
DVO Release
This schema allows (but does not require) the contents of DVO releases to be recorded. This would allow users of the DVO catalog to refer explicitly to a given set of pixels and calibration.
Released Skycell
The concepts for sky cells are similar to individual exposures. A row in the table refers to a particular release and (tess_id, skycell_id).
Here is the proposed table definition
REATE TABLE relSkycell (
relstack_id BIGINT AUTO_INCREMENT,
rel_id BIGINT,
skycell_id varchar(64)
tess_id varchar(64)
state VARCHAR(16), -- current, pending, archive, drop
data_state VARCHAR(16), -- nightly, relphot, relastro, psps
skycal_id BIGINT, -- link to the run that supplied the data
-- for this release.
-- NOTE: it is the skycal_id that we save,
-- not the stack_id
add_id BIGINT,
zpt_obs FLOAT, -- calibrated zero point for this release of
-- this skycell
zpt_stdev FLOAT,
path_base VARCHAR(255), -- path_base of any supporting files,
-- updated WCS for example,
-- we may just build a new cmf
flags UNSIGNED INT,
registered DATETIME,
time_stamp DATETIME,
PRIMARY KEY (rel_id),
KEY (rel_id, tess_id, skycell_id), -- NOTE: no foreign key constraint
-- becase we don't reference the
-- stack directly. Insert query should
-- enforce match.
FOREIGN KEY(rel_id) REFERENCES ippRelease(rel_id),
FOREIGN KEY(skycal_id) REFERENCES skycalRun(skycal_id),
FOREIGN KEY(add_id) REFERENCES addRun(add_id)
) ENGINE=innodb DEFAULT CHARASET=latin1;
Note that the skycell is defined by the id of the sky calibration run, not the stack_id. The idea is that processing for the stack is not complete until stack photometry and sky calibration is performed.
XXX: What about nightly science stacks? We don't do photometry and calibration on them currently and it's not clear if the results of those stages are necessary. Now those data *are* easily findable in the current system because the label is not changed. I.E. MD09 nightly stack from 2012-09-06 can be found from data_group MD09.20120906 or by selection by MJD-OBS.
