| Version 16 (modified by , 13 years ago) ( diff ) |
|---|
IPP Data Release Management
The IPP does not have a good way of keeping track of what data is intended for or has been released to the PS1SC community. Up until now this has not been a big problem, but several issues have recently led us to believe that some new functionality is necessary to make it easy for users to get access to IPP data products that meet their expectations.
The specific goals are
- To record which versions of the data products have been (and should be) "released" to the community through the various distribution channels.
- To allow users to get the "right" versions of the data in their requests to the postage stamp server
- To record within the IPP domain the results of various re-calibrations and re-processings that may occur
These goals will be met with some enhancements to the IPP data base schema and through some lightweight changes to the procedures for processing and releasing data.
IPP Release Channels
Data is released through several channels.
- The IPP Distribution data store
- The Published Science Products Subsystem (PSPS)
- The IPP Postage Stamp Server
- DVO catalogs
The data is the results of runs through the various stages of the IPP pipeline. For the most part the IPP does not have a very good record of which data went into the various releases. For example, given a single frame detection in the IPP database, there is no direct method to determine which version of the processed pixels the detections were measured on. Making the answer to this and other similar questions easy to obtain is the primary purpose of this project.
IPP Data Products
There are three categories of data products to be covered by the IPP release mechanism
- Single Frame Data Products
- Stacked Image Data Products
- Difference Image Data Products
Single Frame Results
Single exposures move through the IPP pipeline in a number of stages. The data products of interest to end users are the results of the chip and camera stage (focal plane coordinates) and the warp stage. The end results are two sets of images and catalogs of detected sources. The catalogs are then imported into the DVO system and eventually into a PSPS data base. Exposures are also queued for difference processing which will be discussed separately.
Nightly Science
On the night that the exposures are acquired they are run through the IPP "Nightly Science" processing. The resulting catalogs are posted to file sets on the IPP Data Store web server. For certain exposures warped images are placed on the data store as well. This is the first "release" for a given exposure.
The catalog files (smf and cmf) currently remain on the data store indefinitely. For various reasons this needs to be changed. A new system for on demand access to these files is under development. The image files posted to the data store are removed after some time to conserve space.
DVO
Occasionally the DVO catalogs are made available to the community. These consist of calibrated measurements and object associations of individual exposures. We currently have no method of directly tracking in the IPP database whether or not the measurements for a particular exposure were "released" in a given DVO catalog or for remembering the results of the calibrations performed by the DVO programs. This information can be figured out from the DVO tables, but since the calibration results are subject to change this information can become "stale".
"Grand" Reprocessing
As time goes on and the algorithms of the IPP improve and the exposures are reprocessed from scratch. This is commonly known as the grand reprocessing. Internally the IPP team refers to this process for the 3Pi survey as "Large Area Processing". For the medium deep fields, the terminology has changed over time but the concepts are somewhat similar.
The results of this processing are added to a new DVO database and from there into PSPS. We currently do not have an efficient method of tracking which sets of processed pixels went into a given PSPS database.
Various Test runs
We currently have no way of tracking which of the various processing runs are intended for end user use and those that are the results of testing. The postage stamp server currently attempts to work around this problem but the results have been mixed. By specifically identifying the data that should be released it will be easy to indentify those runs that should not be released as well.
Stacked Images
The warped images are combined to form stacked images. While for single frames the logical unit is the exposure, stacks are performed on areas of the sky we call skycells. A skycell has a tessellation ID and a skycell number. For example (RINGS.V3, skycell.1406.016) or (MD09.V3, skycell.042)
Three types of stacks have been identified
- Nightly Stacks - constructed from exposures on a single night
- Reference Stacks - constructed from exposures with good image quality that provide good coverage of a region of the sky
- Deep Stacks - constructed from virtually all "good" exposures for a given area of the sky
For the medium deep surveys all three stack types are used. Deep Stacks provide the greatest sensitivity, while reference stacks are used as templates for difference imaging and science that requires the highest resolution. Nightly stacks are used for transient detection.
For the 3PI survey the stacks are produced in the LAP processes. For the purposes of this discussion they are considered "Deep Stacks" even though they are made from significantly fewer input exposures.
Right now the various "releases" of stacks are managed in an ad hoc manner. It is very difficult for a user without access to the IPP database and detailed knowlege of the schema to track what data corresponds to a particular release.
Difference Images
Management of difference images is a subject for further research and will be discussed later.
Definition of an IPP release
A collection of data "released" to the PS1SC community will be represented in the IPP database by a row in 2 new tables called "ippRelease" and "survey". Here are the proposed definitions
survey
mysql> describe survey; +-------------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------+-------+ | surveyID | tinyint(8) | NO | PRI | 0 | | | name | varchar(16) | YES | MUL | NULL | | | description | varchar(255) | YES | | NULL | | +-------------+--------------+------+-----+---------+-------+ 3 rows in set (0.00 sec)
The survey table is a copy of the PSPS table Survey.
+----------+------+------------------------------------+ | surveyID | surveyName | description | +----------+------------+--------------------------------------------------------------+ | 0 | 3PI | PS1 3PI Survey | | 1 | MD01 | PS1 MD01 XMM-LSS-DXS 022224-043000 | | 2 | MD02 | PS1 MD02 CDFS/GOODS/GEMS 033224-274800 | | 3 | MD03 | PS1 MD03 IFA/Lynx 084300+444000 | | 4 | MD04 | PS1 MD04 COSMOS 100000+021200 | | 5 | MD05 | PS1 MD05 Lockman-DXS 110000+570000 | | 6 | MD06 | PS1 MD06 NGC 4258 121857+471814 | | 7 | MD07 | PS1 MD07 VISTA-Video1 140000+050000 | | 8 | MD08 | PS1 MD08 EliasN1-DXS 160000+570000 | | 9 | MD09 | PS1 MD09 Vimos4-DXS-SSA 220000+003000 | | 10 | MD10 | PS1 MD10 Deep2 233000+000000 | | 11 | M31 | PS1 M31 Andromeda 004242+411600 | | 12 | STS1 | PS1 STS1 Outskirts of Bulge 195048+170339 | | 13 | CAL01 | PS1 Cal North Pole 000000+900000 | | 14 | CAL02 | PS1 Cal SA92 Dense Landolt Std Field 005808+004232 | | 15 | CAL03 | PS1 Cal Perseus Cluster / Abell 426 031900+413300 | | 16 | CAL04 | PS1 Cal SA95 Dense Landolt Std Field 065207+001710 | | 17 | CAL05 | PS1 Cal Taurus dark cloud (diverse extinction) 043000+250000 | | 18 | CAL06 | PS1 Cal SA98 Densle Landolt Std Field 065207+001710 | | 19 | CAL07 | PS1 Cal M81-M82-HolmIX (Local Group) 095640+692200 | | 20 | CAL08 | PS1 Cal Virgo Cluster (Center) 123000+130000 | | 21 | CAL09 | PS1 Cal Hercules Cluster (Center) 160508+1745+28 | | 22 | CAL10 | PS1 Cal M17-M24 Region 181400+000544 | | 23 | CAL11 | PS1 Cal SA110 Dense Landolt Std Field 184207+000544 | | 24 | CAL12 | PS1 Cal Kepler Mission Astrometric field 192240+443000 | | 25 | CAL13 | PS1 Cal 3C454.1 (Northern field) 225033+712919 | | 26 | STS2 | PS1 STS2 Hyades 004000+150000 | | 27 | STS3 | PS1 STS3 Praesepe 083000+200000 | | 28 | SSS | PS1 Solar System Sweet Spot Survey | +----------+------------+--------------------------------------------------------------+
We may omit obsolete surveys.
ippRelease
mysql> describe ippRelease; +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | rel_id | int(11) | NO | PRI | NULL | auto_increment | | surveyID | int(11) | YES | MUL | NULL | | | release_name | varchar(64) | YES | MUL | NULL | | | release_state | varchar(16) | YES | MUL | NULL | | | dataRelease | int(11) | YES | | NULL | | | priority | int(11) | YES | MUL | NULL | | | dvodb | varchar(255) | YES | | NULL | | | ubercal_file | varchar(255) | YES | | NULL | | +---------------+--------------+------+-----+---------+----------------+ 8 rows in set (0.00 sec)
The columns in ippRelease are
- rel_id - the primary key for the row
- surveyID of the corresponding survey
- releaseName - a human friendly identifier for the release
- state - current status of the release. Several values have been identified
- active - results for the release are currently available to the community
- pending - release identified, construction may have begun, but the results not yet ready to be available
- archive - the release contains the results of a previous processing of the data that has been superseded.
- drop - the results of the release have been abandoned and should be ignored
- dataRelease - If not negative, the dataRelease value in the PSPS tables corresponding to this ippRelease.
- priority - An integer value giving the priority of the various releases for the survey.
- dvodb - name of dvo database used for calibration
- ubercal_file - file name of fits table that provided the ubercal calibrations
Here is an example of how this system might work for the 3pi survey. Assume that we have finished the current 2012 grand reprocessing and are ready to start the next one.
mysql> select * from ippRelease where release_name like '3PI%'; +--------+----------+--------------+---------------+-------------+----------+-------------------------+-------------------+ | rel_id | surveyID | release_name | release_state | dataRelease | priority | dvodb | ubercal_file | +--------+----------+--------------+---------------+-------------+----------+-------------------------+-------------------+ | 1 | 0 | 3PI.nightly | active | -1 | 10 | NULL | NULL | | 2 | 0 | 3PI.PV1 | active | 1 | 20 | LAP.ThreePi.20120706.v2 | ucalqw_noref.fits | | 37 | 0 | 3PI.PV2 | active | 2 | 30 | NULL | NULL | +--------+----------+--------------+---------------+-------------+----------+-------------------------+-------------------+ 3 rows in set (0.00 sec)
There are 4 releases shown here.
The 3PI.nightly release is active. Data is being currently added to it by the IPP nightly science processing. Since these data are not being ingested into PSPS the dataRelease column is -1.
3PI.PV1 refers to the 2012 3PI "grand reprocessing". This release is active. dataRelease == 1
3PI.PV2 is the next grand reprocessing dataRelease == 2. The state of this release is active.
The priority column is used to select the release of the data that should returned by the IPP when users request a given exposure or skycell. The query above demonstrates the idea. It orders the releases by descending priority. The postage stamp server and other release mechanisms will use queries ordered this way.
Data will be returned from the highest priority release. In our example this is 3PI.PV2. If the given exposure or skycell has not been (successfully) processed in the highest priority release data will be returned from a lower priority release.
A typical situation is that of a recent exposure. Since it isn't found in the current release the nightly science version is selected.
Note: This scheme will be used for request where the user has not explicitly listed the release of interest. All IPP data will be still be available if the user can identify it.
Once we are ready to make data from the next release available, the release_state and priorities will be adjusted accordingly.
Released Exposure
Now we turn our attention to tables that record the actual contents of a release. Single frame results are represented by the table relExp (released exposure) while stacks are represented by relStack (released stack). We expect that in the future a similar table will be defined for difference stage data.
The relExp table encapsulates the data products that were used to construct the released data for a given exposure and the results of the calibration(s) of the exposure. The proposed table definition is as follows
mysql> describe relExp; +--------------+------------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------+------------------+------+-----+---------+----------------+ | relexp_id | bigint(20) | NO | PRI | NULL | auto_increment | | rel_id | int(11) | YES | MUL | NULL | | | exp_id | bigint(20) | YES | MUL | NULL | | | chip_id | bigint(20) | YES | MUL | NULL | | | cam_id | bigint(20) | YES | MUL | NULL | | | group_id | int(11) | YES | MUL | NULL | | | state | varchar(16) | YES | MUL | NULL | | | flags | int(10) unsigned | YES | | NULL | | | zpt_obs | float | YES | | NULL | | | zpt_stdev | float | YES | | NULL | | | mcal | float | YES | | NULL | | | ubercal_dist | int(11) | YES | | 0 | | | path_base | varchar(255) | YES | | NULL | | | fault | smallint(6) | NO | MUL | NULL | | | registered | datetime | YES | | NULL | | | time_stamp | datetime | YES | | NULL | | +--------------+------------------+------+-----+---------+----------------+ 16 rows in set (0.00 sec)
This table will allow us to easily identify for a given exposure and release the derived data products that were used to construct the results for the exposure.
Each time an exposure is added to a release a new row will be added to this table.
state
A released exposure may go through a sequence of states
- processed - The initial results of the camera stage for the exposure.
- poor_quality - The camRun failed to find a solution for the exposure.
- calibrated - The calibration information (zero point and astrometry) has been updated by a DVO calibration operation
- psps - The information from the exposure has been added to the PSPS database corresponding to the release.
- drop - This relExp is to be dropped for some reason.
Released Stack
The concepts for sky cells are similar to individual exposures. A row in the table refers to a particular release, tess_id, and skycell_id.
mysql> describe relStack; +-------------+------------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------+------------------+------+-----+---------+----------------+ | relstack_id | bigint(20) | NO | PRI | NULL | auto_increment | | rel_id | int(11) | YES | MUL | NULL | | | stack_id | bigint(20) | YES | MUL | NULL | | | skycal_id | bigint(20) | YES | | NULL | | | skycell_id | varchar(64) | YES | | NULL | | | tess_id | varchar(64) | YES | MUL | NULL | | | filter | varchar(16) | YES | | NULL | | | state | varchar(16) | YES | MUL | NULL | | | flags | int(10) unsigned | YES | | NULL | | | stack_type | varchar(16) | YES | MUL | NULL | | | zpt_obs | float | YES | | NULL | | | zpt_stdev | float | YES | | NULL | | | mjd_obs | int(10) unsigned | YES | MUL | NULL | | | path_base | varchar(255) | YES | | NULL | | | fault | smallint(6) | NO | MUL | NULL | | | registered | datetime | YES | | NULL | | | time_stamp | datetime | YES | | NULL | | +-------------+------------------+------+-----+---------+----------------+ 17 rows in set (0.00 sec)
There are differences between relStack and the single frame table.
- stack_type will have one of three values "deep", "reference", or "nightly"
Data Access
The postage stamp server request table specification will be enhanced to include three new columns
- SURVEY_NAME - select data from the corresponding survey
- IPP_RELEASE - select data from corresponding ipp release
- RUN_TYPE - specify stack type (nightly, deep, reference) or difference type (warp-warp, warp-stack, stack-stack)
This will allow users to more efficiently select the data that they want.
If a particular release name is not supplied an appropriate release will be chosen by the system using the priority system described above.
