Notes on Implementation of the Public Release of PS1 image data products

Author: Bill Sweeney 2015-05-01

The plan of record is to migrate the PS1 images to Space Telescope Institute by shipping machines owned by STSCI which have been operating as file servers on the IPP cluster on Maui. These machines will contain file systems with the desired data products. STSCI will then provide some method for accessing the data.

This plan raises a number of outstanding issues and technical work that will be required by the IPP team and the group at STSCI.

This note is intended to be a starting point for fleshing out the requirements and plan for implentation of this work.

Background

The primary way for PS1SC users to access image data products has been through the IPP data store interface. Files are contained in File Sets (sub-directories) in particular Products (directories) on the data store HTTP server which is part of the IPP cluster on Maui. Access external to the IfA is allowed through a proxy server in Manoa on Maui. Access to this server is controlled by restricting access to IP addresses that have been identified as belonging to members of the PS1 science consortium. Access to certain products is more restricted. Listings of the various levels are given by cgi scripts run by the HTTP server.

Data is placed in the data store (file sets are registered) by the IPP distribution tasks and by the IPP postage stamp server. The distribution jobs are queued by the IPP when processing various data sets and reductions. Postage stamp results contain the results of requests submitted by individual users.

Use of the data store interface (invented by the PS1 camera group for summit data access) was chosen primarily for historical reasons.

It is most useful for distributing data in a serial fashion. There is an interface which allows clients to ask the server "what's new since this fileset". This is of great use to users waiting for timely distribution of recently acquired science data in near real time.

The data store is of less utility to users of the postage stamp server. Those users simply need to know that a request has completed processing and where the results have been posted. In practice users who submit requests through the "upload request file method" know in advance what the name of their results file set will be and repeatedly ask for it getting 403 file not found errors until they exist.