IPP Data Release Management

The IPP does not have a good way of keeping track of what data is intended for or has been released to the PS1SC community. Up until now this has not been a big problem, but several issues have recently led us to believe that some new functionality is necessary to make it easy for users to get access to IPP data products that meet their expectations.

The specific goals are

• To record which versions of the data products have been (and should be) "released" to the community through the various distribution channels.
• To allow users to get the "right" versions of the data in their requests to the postage stamp server
• To record within the IPP domain the results of various re-calibrations and re-processings that may occur

These goals will be met with some enhancements to the IPP data base schema and through some lightweight changes to the procedures for processing and releasing data.

IPP Release Channels

Data is released through several channels.

• The IPP Distribution data store
• The Published Science Products Subsystem (PSPS)
• The IPP Postage Stamp Server
• DVO catalogs

The data is the results of runs through the various stages of the IPP pipeline. For the most part the IPP does not have a very good record of which data went into the various releases. For example, given a single frame detection in the IPP database, there is no direct method to determine which version of the processed pixels the detections were measured on. Making the answer to this and other similar questions easy to obtain is the primary purpose of this project.

IPP Data Products

There are three categories of data products to be covered by the IPP release mechanism

• Single Frame Data Products
• Stacked Image Data Products
• Difference Image Data Products

Single Frame Results

Single exposures move through the IPP pipeline in a number of stages. The data products of interest to end users are the results of the chip and camera stage (focal plane coordinates) and the warp stage. The end results are two sets of images and catalogs of detected sources. The catalogs are then imported into the DVO system and eventually into a PSPS data base. Exposures are also queued for difference processing which will be discussed separately.

Nightly Science

On the night that the exposures are acquired they are run through the IPP "Nightly Science" processing. The resulting catalogs are posted to file sets on the IPP Data Store web server. For certain exposures warped images are placed on the data store as well. This is the first "release" for a given exposure.

The catalog files (smf and cmf) currently remain on the data store indefinitely. For various reasons this needs to be changed. A new system for on demand access to these files is under development. The image files posted to the data store are removed after some time to conserve space.

DVO

Occasionally the DVO catalogs are made available to the community. These consist of calibrated measurements and object associations of individual exposures. We currently have no method of directly tracking in the IPP database whether or not the measurements for a particular exposure were "released" in a given DVO catalog or for remembering the results of the calibrations performed by the DVO programs. This information can be figured out from the DVO tables, but since they are subject to change this information can become "stale".

"Grand" Reprocessing

As time goes on and the algorithms of the IPP improve and the exposures are reprocessed from scratch. This is commonly known as the grand reprocessing. Internally the IPP team refers to this process for the 3Pi survey as "Large Area Processing". For the medium deep fields, the terminology has changed over time but the concepts are somewhat similar.

The results of this processing are added to a new DVO database and from there into PSPS. We currently do not have an efficient method of tracking which sets of processed pixels went into a given PSPS database.

Various Test runs

We currently have no way of tracking which of the various processing runs are intended for end user use and those that are the results of testing. The postage stamp server currently attempts to work around this problem but the results have been mixed. By specifically identifying the data that should be released it will be easy to indentify those runs that should not be released as well.

Stacked Images

The warped images are combined to form stacked images. While for single frames the logical unit is the exposure, stacks are performed on areas of the sky we call skycells. A skycell has a tessellation ID and a skycell number. For example (RINGS.V3, skycell.1406.016) or (MD09.V3, skycell.042)

Three types of stacks have been identified

• Nightly Stacks - constructed from exposures on a single night
• Reference Stacks - constructed from exposures with good image quality that provide good coverage of a region of the sky
• Deep Stacks - constructed from virtually all "good" exposures for a given area of the sky

For the medium deep surveys all three stack types are used. Deep Stacks provide the greatest sensitivity, while reference stacks are used as templates for difference imaging and science that requires the highest resolution. Nightly stacks are used for transient detection.

For the 3PI survey the stacks are produced in the LAP processes. For the purposes of this discussion they are considered "Deep Stacks" even though they are made from significantly fewer input exposures.

Right now the various "releases" of stacks are managed in an ad hoc manner. It is very difficult for a user without access to the IPP database and detailed knowlege of the schema to track what data corresponds to a particular release.

Difference Images

Management of difference images is a subject for further research and will be discussed later.

Definition of an IPP release

A collection of data "released" to the PS1SC community will be represented in the IPP database by a row in 2 new tables called "ippRelease" and "survey". Here are the proposed definitions

mysql> describe survey;
+-------------+--------------+------+-----+---------+-------+
| Field       | Type         | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| surveyID    | tinyint(8)   | NO   | PRI | 0       |       | 
| name        | varchar(16)  | YES  | MUL | NULL    |       | 
| description | varchar(255) | YES  |     | NULL    |       | 
+-------------+--------------+------+-----+---------+-------+
3 rows in set (0.00 sec)

mysql> describe ippRelease;
+-------------+-------------+------+-----+---------+----------------+
| Field       | Type        | Null | Key | Default | Extra          |
+-------------+-------------+------+-----+---------+----------------+
| rel_id      | smallint(6) | NO   | PRI | NULL    | auto_increment | 
| surveyID    | tinyint(8)  | YES  | MUL | NULL    |                | 
| releaseName | varchar(16) | YES  | MUL | NULL    |                | 
| state       | varchar(16) | YES  | MUL | NULL    |                | 
| dataRelease | tinyint(8)  | YES  |     | NULL    |                | 
+-------------+-------------+------+-----+---------+----------------+
5 rows in set (0.00 sec)

The survey table is a copy of the PSPS table Survey. Example entries are

+----------+------+------------------------------------+
| surveyID | name | description                        |
+----------+------+------------------------------------+
|        0 | 3PI  | PS1 3PI Survey                     | 
|        1 | MD01 | PS1 MD01 XMM-LSS-DXS 022224-043000 | 
+----------+------+------------------------------------+

The columns in ippRelease are

• rel_id primary key for the row
• surveyID for to the survey that the data in a particular release contains
• releaseName - human friendly identifier for the release
• state - current status of the release. Several values have been identified
  • active - results for the release are currently available to the community
  • pending - release identified, construction may have begun, but the data is not yet ready to be available by default
  • archive - the release contains the results of a previous processing of the data that has been superseded.
• dataRelease - If not null, the dataRelease value in the corresponding PSPS tables.

Some examples

+--------+----------+-------------+---------+-------------+
| rel_id | surveyID | releaseName | state   | dataRelease |
+--------+----------+-------------+---------+-------------+
|      1 |        0 | 3PI.nightly | active  |        NULL | 
|      2 |        0 | 3PI.GR0     | active  |           0 | 
|      3 |        0 | 3PI.GR1     | pending |           1 | 
|      4 |        0 | SAS.V8      | archive |           8 | 
|      5 |        0 | SAS.V9      | active  |           9 | 
+--------+----------+-------------+---------+-------------+

The 3PI.nightly release is active. Data is being currently added to it and is available to users who request data from that processing. Since these data are no longer being ingested into PSPS the dataRelease field is null.

3PI.GR0 refers to the version of LAP processing that is currently in the PSPS database PS1_3PI. Data corresponding to that release is identified by dataRelease 0, while 3PI.GR1 refers to the future LAP release which will begin construction soon. Once that release gets under way it's state will be set to active while 3PI.GR0 will be set to "archive"

The two SAS entries refer to PS1_SAS_08 and PS1_SAS_09.

Now we turn our attention to tables that record the actual contents of a release.