Changes between Version 4 and Version 5 of ps1_public_release_pstamp_issues

Timestamp:

May 4, 2015, 12:43:00 PM (11 years ago)

Author:

bills

Comment:

--

ps1_public_release_pstamp_issues

@@ -63 +63 @@
  * results are stored on the data store
 
-== Data Requirements for Image Server for PS1 Public Release ==
+== Image Data to be provided by a PS1 Public Release Image Server ==
+The following is my understanding (guess) of the image data to be provided with the public release.
 
-* Postage stamp (cut outs) subsets and complete single frame warp skycell images
- * These are the data products from IPP warp processing (full state) which are being preserved for PV3. 
- * Are results from the entire run needed, psfs cmfs etc. I suspect no.
+* Postage stamp subsets (cut outs) and complete single frame warp skycell images
+ * These are the data products from IPP warp processing (full state) which are being preserved from processing version 3 (PV3). 
+ * Are results from the entire run needed, psfs, cmfs, log files etc. I suspect no.
 * Postage stamps and complete stack skycell images
  * These are the data products from the IPP stack processing
- * To provide photometry information for headers and detections equivalent to what the IPP server odes today the skycal cmfs for each skycell are needed.
-* Stack Summary images for a given region of the sky
+ * To provide photometry information for headers and detections equivalent to what the IPP server odes today the skycal cmfs for each skycell are needed (or at least the FITS headers).
+* Stack Summary images for the sky. (MD or just 3PI?)
 
 Note: The IPP postage stamp server delivers images based on a single IPP image. That is we do not mosaic images together although is is
@@ -79 +80 @@
 * detrended chip stage images
 * difference images
+* catalogs (except staticsky)
+These were not preserved by the IPP for PV3 because of insufficient space.
+
+=== Metadata Required to Support the implementation of an Image Server ===
+The plan is to ship the images pre installed on the servers being shipped back to STSCI. The IPP locates images using the Nebulous system. 
+Nebulous is essentially a sophisticated lookup tables that maps abstract paths to NFS disk locations in the IPP cluster. 
+For example: The file referenced by neb://path_to_bills_file/myfile.fits might resolve to an actual file on disk
+with a name like  /data/stsci19.2/nebulous/a8/a7/6829803927.afile.fits
+
+Nebulous is implemented in the IPP through various application programming interfaces which use SOAP to talk to a set of HTTP servers. 
+The HTTP servers communicate with the nebulous mysql database. 
+
+There are several issues that need to be resolved before these systems are delivered.
+* Data that must stay in the IPP cluster will need to be shuffled off to other systems or replicated.
+* Some system that implements the Nebulous interfaces or a subset thereof will need to be delivered. Of course that could be nebulous but:
+  * The full nebulous database currently takes up 3.7 TB and contains information on files that are not part of the delivery
+  * Nebulous contains many features that might not be relevant to the new owners
+  * It introduces depencies on NFS, apache, and mysql. These might not be issues but should be noted.
+  * Management requires expertise that does not currently exist outside of the IPP team.
+* Information describing the composition of the delivered IPP data products is required. 
+  * This could be a modified copy of the GPC1 mysql database  (594 GB) with the states of various data is modified to match the data being shipped.
+  * Alternatively a database with a simplfied structure (single frame data represented by warp tables augmented with exposure metadata from rawExp) could be loaded
 
 
+