Changes between Version 8 and Version 9 of ps1_public_release_pstamp_issues

Timestamp:

May 4, 2015, 2:46:57 PM (11 years ago)

Author:

bills

Comment:

--

ps1_public_release_pstamp_issues

@@ -1 +1 @@
 = Notes on Implementation of the Public Release of PS1 image data products =
 
-Author: Bill Sweeney 2015-05-04
+Note started by: Bill Sweeney 2015-05-04
 
-The plan of record is to migrate the PS1 images to Space Telescope Institute by shipping machines owned by STSCI which have
-been operating as file servers on the IPP cluster on Maui. These machines will contain file systems with the desired data products.
+The plan of record is to migrate the PS1 images to Space Telescope Institute by shipping machines owned by STSCI that have
+been operating as file servers in the IPP cluster on Maui. These machines will contain file systems with the desired data products.
 STSCI will then provide some method for accessing the data.
 
@@ -14 +14 @@
 on what IPP software might be adapted to meet the requirements and what might need to be developed.
 
-The contents here in the first draft come from my understanding of what is needed and what is planned. I'm sure others have thought
-through many issues in detail.
+The contents here in the first draft come from my understanding of what is needed and what is planned. I know that others have already thought
+through many of these issues in some detail.
 
 
@@ -21 +21 @@
 
 The primary way for PS1SC users to access image data products has been through the IPP data store interface. ''Files'' are contained in ''File Sets'' (sub-directories)
-in particular ''Products'' (directories) on the data store HTTP server. This server is part of the IPP cluster on Maui. Access external to the
-IfA is allowed through a proxy server in Manoa on Maui. Access to this server is controlled by restricting access to IP addresses
-that have been identified as belonging to members of the PS1 science consortium. Access to certain products is more
+in particular ''Products'' (directories) on the data store HTTP server. This server is part of the IPP cluster on Maui. Access by users external to the
+IfA is provided through a proxy server in Manoa on Maui. Access is restricted to authorized users by restricting HTTP access to hosts with IP addresses
+that belong to members of the PS1 science consortium. Access to certain products is more
 restricted. Listings of the various levels are given by cgi scripts run by the HTTP server.
 
@@ -34 +34 @@
 This is of great use to users who need timely access to recently acquired science data as it is processed. 
 
-The data store is of less utility to users of the postage stamp server. Those users simply need to know that a request has completed processing and where the results have been posted.
-In practice users who submit requests through the "upload request file method" know in advance what the name of their results file set will be and repeatedly
-ask for it getting 403 file not found errors until they exist.
+The data store is of less utility to users of the postage stamp server. Those users simply need to know that a request has completed processing and where the results may be found.
+In practice, users who submit requests through the "upload request file method" know in advance what the name of their results file set will be and repeatedly
+ask for it getting 403 file not found errors until they exist. PSI provides users with a link to the results filesets.
 
 The Postage Stamp Server is a system which allows users to access images '''from the IPP'''. Several different mechanisms exist for submitting
@@ -42 +42 @@
 control document (ICD) (add link.) 
 
-The server is controlled by a set of pantasks tasks. Work to do is managed through a mysql database. One of the requirements of the server is that it
-support multiple ''projects'' simultaneously so the postage stamp tables are stored in database separate from the project (gpc1) database. Currently the postage stamp
+The server is controlled by a set of pantasks tasks. Work to do is managed through a mysql database. One of the requirements of the server is for it to
+support multiple ''projects'' simultaneously. For this reason, the postage stamp tables are stored in database separate from the project (gpc1) database. Currently the postage stamp
 database is shared with the database that contains the data store tables (ippRequestServer).
 
@@ -66 +66 @@
 The following is my understanding (guess) of the image data to be provided with the public release.
 
- * Postage stamp subsets (cut outs) and complete single frame warp skycell images
+ * Postage stamp subsets (cut outs) and complete single frame warp skycell images, masks, and variance images
   * These are the data products from IPP warp processing (full state) which are being preserved from processing version 3 (PV3). 
   * Are results from the entire run needed, psfs, cmfs, log files etc. I suspect no.
- * Postage stamps and complete stack skycell images
+ * Postage stamps and complete stack skycell images, masks, and variance images
   * These are the data products from the IPP stack processing
   * To provide photometry information for headers and detections equivalent to what the IPP server odes today the skycal cmfs for each skycell are needed (or at least the FITS headers).
  * Stack Summary images for the sky. (MD or just 3PI?)
+ * Selection methods: by coordinate, by skycell, by exposure (single frame only). 
+  * No bydiff. No byid ?
 
 Note: The IPP postage stamp server delivers images based on a single IPP image. That is we do not mosaic images together although is is
-an often requested feature.
+an often requested feature that should be considered.
 
 === Data products that will not be included in the public release ===
@@ -83 +85 @@
  * catalogs (except staticsky)
 
-The chip and diff stage images were not preserved by the IPP for PV3 because of insufficient space.
+The chip and diff stage images were not preserved by the IPP for PV3 because of insufficient space.  The lack of difference images will somewhat
+reduce the utility of the difference detections.
 
 === Metadata Required to Support the implementation of an Image Server ===
 The plan is to ship the images pre installed on the servers being shipped back to STSCI. The IPP locates images using the Nebulous system. 
-Nebulous is essentially a sophisticated lookup tables that maps abstract paths to NFS disk locations in the IPP cluster. 
+Nebulous is essentially a sophisticated lookup table that maps abstract paths to NFS disk locations in the IPP cluster. 
 For example: The file referenced by neb://path_to_bills_file/myfile.fits might resolve to an actual file on disk
 with a name like  /data/stsci19.2/nebulous/a8/a7/6829803927.afile.fits
@@ -95 +98 @@
 
 There are several issues that need to be resolved before these systems are delivered.
- * Data that must stay in the IPP cluster will need to be shuffled off to other systems or replicated.
+ * Data that must stay in the IPP cluster will need to be shuffled off to other systems or replicated to machines that are staying on Maui.
  * Some system that implements the Nebulous interfaces or a subset thereof will need to be delivered. Of course that could be nebulous but:
   * The full nebulous database currently takes up 3.7 TB and contains information on files that are not part of the delivery
-  * Nebulous contains many features that might not be relevant to the new owners
-  * It introduces depencies on NFS, apache, and mysql. These might not be issues but should be noted.
-  * Management requires expertise that does not currently exist outside of the IPP team.
+  * Nebulous contains many features that might not be relevant to the new implementation
+  * It introduces depencies on NFS, apache, and mysql. (These might not be issues but should be noted.)
+  * Management of the Nebulous system requires expertise that does not currently exist outside of the IPP team.
  * Information describing the composition of the delivered IPP data products is required. 
   * This could be a modified copy of the GPC1 mysql database  (594 GB) with the states of various data is modified to match the data being shipped.
-  * Alternatively a database with a simplfied structure (single frame data represented by warp tables augmented with exposure metadata from rawExp) could be loaded
+  * Alternatively a database with a simplfied structure (single frame data represented by warp tables augmented with exposure metadata from rawExp) could be created
  * Description of the skycell tessellations. (Tessellation directories).
 
-In addition we should prepare for the inevitable and come up with a mechanism for the IPP to regenerate lost images and deliver them to STSCI.
+In addition we should prepare for the inevitable and come up with a mechanism for the IPP to regenerate images that get lost due to system failures and getting back to STSCI.
 
 === Postage Stamp Server Requirements ===
 Given that the IPP postage stamp server is so tightly coupled with the IPP and has several complexities to support requirements that the
 public server does not have, the implementors at STSCI may want to build a different implementation.
+
+The requirements include
  * A graphical user interface similar to what PSI (or SDSS) provides
- * A mechanism for users to programmaticaly submit requests. (The IPP's upload mechanism is basically a prototype and not a good model).
- * A simplified request specification bycoord and byexp
+ * A mechanism for advanced users to programmaticaly submit requests. (The IPP's upload mechanism is basically a prototype and not a good model).
+ * System for locating images of interest that correspond to the users' requests
+ * Pixel extractor
 
+It is possible to use the IPP software but would require a nearly full IPP build.