Changes between Initial Version and Version 1 of ps1_public_release_pstamp_issues

Timestamp:

May 1, 2015, 9:22:34 AM (11 years ago)

Author:

bills

Comment:

--

ps1_public_release_pstamp_issues

@@ +1 @@
+= Notes on Implementation of the Public Release of PS1 image data products =
+
+Author: Bill Sweeney 2015-05-01
+
+The plan of record is to migrate the PS1 images to Space Telescope Institute by shipping machines owned by STSCI which have
+been operating as file servers on the IPP cluster on Maui. These machines will contain file systems with the desired data products.
+STSCI will then provide some method for accessing the data.
+
+This plan raises a number of outstanding issues and technical work that will be required by the IPP team and the group at STSCI.
+
+This note is intended to be a starting point for fleshing out the requirements and plan for implentation of this work.
+
+
+== Background ==
+
+The primary way for PS1SC users to access image data products has been through the IPP data store interface. ''Files'' are contained in ''File Sets'' (sub-directories)
+in particular ''Products'' (directories) on the data store HTTP server which is part of the IPP cluster on Maui. Access external to the
+IfA is allowed through a proxy server in Manoa on Maui. Access to this server is controlled by restricting access to IP addresses
+that have been identified as belonging to members of the PS1 science consortium. Access to certain products is more
+restricted. Listings of the various levels are given by cgi scripts run by the HTTP server.
+
+Data is placed in the data store (file sets are registered) by the IPP distribution tasks and by the IPP postage stamp server. The distribution jobs
+are queued by the IPP when processing various data sets and reductions. Postage stamp results contain the results of requests submitted
+by individual users.
+
+Use of the data store interface (invented by the PS1 camera group for summit data access) was chosen primarily for historical reasons.
+
+It is most useful for distributing data in a serial fashion. There is an interface which allows clients to ask the server "what's new since this fileset". 
+This is of great use to users waiting for timely distribution of recently acquired science data in near real time. 
+
+The data store is of less utility to users of the postage stamp server. Those users simply need to know that a request has completed processing and where the results have been posted.
+In practice users who submit requests through the "upload request file method" know in advance what the name of their results file set will be and repeatedly
+ask for it getting 403 file not found errors until they exist.
+
+