Changes between Version 1 and Version 2 of ps1_public_release_pstamp_issues

Timestamp:

May 4, 2015, 11:38:27 AM (11 years ago)

Author:

bills

Comment:

--

ps1_public_release_pstamp_issues

@@ -1 +1 @@
 = Notes on Implementation of the Public Release of PS1 image data products =
 
-Author: Bill Sweeney 2015-05-01
+Author: Bill Sweeney 2015-05-04
 
 The plan of record is to migrate the PS1 images to Space Telescope Institute by shipping machines owned by STSCI which have
@@ -33 +33 @@
 ask for it getting 403 file not found errors until they exist.
 
+The Postage Stamp Server is a system which allows users to access images '''from the IPP'''. Several different mechanisms exist for submitting
+requests to the system. The format for requests is ultimately a fits binary table that meets the requirements listed in the Postage Stamp server interface
+control document (ICD) (add link.) 
 
+The server is controlled by a set of pantasks tasks. Work to do is managed through a mysql database. One of the requirements of the server is that it
+support multiple ''projects'' simultaneously so the postage stamp tables are stored in database separate from the project (gpc1) database. Currently the postage stamp
+database is shared with the database that contains the data store tables (ippRequestServer).
+
+A request is "submitted" by adding a row to a table in the postage stamp server database. As they are processed they transition between 3 states. A submitted request
+starts in new state. The first step is parsing the request file creating one or more jobs for each row in the request table. Parsing is a single threaded process and is a significant bottleneck
+for large request files. Once parsing is complete the request is set to state "run". At this point any jobs that were queued are processed in parallel. Finally when all jobs are complete the
+finish processing is performed on the request. This is also a single threaded process. The primary functions of the finish process are to build the results table and register
+the results fileset on the data store. When these operations are successfully completed the request is set to state 'stop'. Since nearly all of the IPP's single frame
+results are not saved (they are cleaned up) the postage stamp server is able to queue images to be regenerated (updated). This step is called dependency processing
+and is performed after parsing.
+
+It is important to understand that the postage stamp system is intimately tied with the IPP. There are a number of characteristics of the system which are tied
+to the way the IPP operates. Many of these things will not be relevant in the public release environment yet they have added significant complexity to the current system.
+ * The request specification is quite rich allowing essentially any image ever processed by the IPP to be selected.
+ * There are several different selection methods. Many of these methods are rarely used outside of the IfA.
+ * Images often must be regenerated and this process must be managed.
+ * The input image files are stored in the nebulous file system. 
+