Changes between Initial Version and Version 1 of ippReleaseManagement

Timestamp:

Sep 20, 2012, 2:22:36 PM (14 years ago)

Author:

bills

Comment:

--

ippReleaseManagement

@@ +1 @@
+
+== IPP Data Release Management ==
+
+The IPP does not have a good way of keeping track of what data is intended for or has been released
+to the PS1SC community. Up until now this has not been a big problem, but several issues have recently
+led us to believe that some new functionality is necessary to make it easy for users to get access to
+IPP data products that meet their expectations.
+
+The specific goals are
+
+ * To record which versions of the data products have been (and should be) "released" to the community through the various distribution channels.
+ * To allow users to get the "right" versions of the data in their requests to the postage stamp server
+ * To record within the IPP domain the results of various re-calibrations and re-processings that may occur
+
+These goals will be met with some enhancements to the IPP data base schema and through some lightweight changes
+to the procedures for processing and releasing data.
+
+
+== IPP Release Channels ==
+Data is released through several channels.
+
+ * The IPP Distribution data store
+ * The Published Science Products Subsystem (PSPS)
+ * The IPP Postage Stamp Server
+ * DVO catalogs
+
+The data is the results of runs through the various stages of the IPP pipeline. For the most part the IPP does not have a very good record of which data went into the various releases. For example given a single frame detection in the IPP database, there is no direct method to determine which version of the processed pixels the detections were measured on. Making the answer to this and other similar questions easy to obtain is the primary purpose of this
+project.
+
+== IPP Data Products ==
+There are three categories of data products to be covered by the IPP release mechanism
+ * Single Frame Data Products
+ * Stacked Image Data Products
+ * Difference Image Data Products
+
+=== Single Frame Results ===
+Single exposures move through the IPP pipeline in a number of stages. The data products of interest to end users
+are the results of the chip and camera stage (focal plane coordinates) and the warp stage. The end results are
+two sets of images and catalogs of detected sources. The catalogs are then imported into the DVO system and
+eventually into a PSPS data base. Exposures are also queued for difference processing which will be discussed
+separately.
+
+==== Nightly Science ====
+On the night that the exposures are acquired they are run through the IPP "Nightly Science" processing. 
+The resulting catalogs are posted to file sets on the IPP Data Store web server. For certain exposures
+warped images are placed on the data store as well. This is the first "release" for a given exposure. 
+
+The catalog files (smf and cmf) currently remain on the data store indefinitely. For various 
+reasons this needs to be changed. A new system for on demand access to these files is under development.
+The image files posted to the data store are removed after some time to conserve space.
+
+
+
+