Notes on Implementation of the Public Release of PS1 image data products

Author: Bill Sweeney 2015-05-04

The plan of record is to migrate the PS1 images to Space Telescope Institute by shipping machines owned by STSCI which have been operating as file servers on the IPP cluster on Maui. These machines will contain file systems with the desired data products. STSCI will then provide some method for accessing the data.

This plan raises a number of outstanding issues and technical work that will be required by the IPP team and the group at STSCI.

This note is intended to be a starting point for fleshing out the requirements and plan for implentation of this work. First we discuss the current system for delivering images to the consortium. Next we talk about what the requirements for an image server for public release might be. Then we discuss what data must be delivered by the IPP to meet these requirements. Finally we have some thoughts on what IPP software might be adapted to meet the requirements and what might need to be developed.

The contents here in the first draft come from my understanding of what is needed and what is planned. I'm sure others have thought through many issues in detail.

Background

The primary way for PS1SC users to access image data products has been through the IPP data store interface. Files are contained in File Sets (sub-directories) in particular Products (directories) on the data store HTTP server. This server is part of the IPP cluster on Maui. Access external to the IfA is allowed through a proxy server in Manoa on Maui. Access to this server is controlled by restricting access to IP addresses that have been identified as belonging to members of the PS1 science consortium. Access to certain products is more restricted. Listings of the various levels are given by cgi scripts run by the HTTP server.

Data is placed in the data store (file sets are registered) by the IPP distribution tasks and by the IPP postage stamp server. The distribution jobs are queued by the IPP when processing various data sets and reductions. Postage stamp results contain the results of requests submitted by individual users.

Use of the data store interface (invented by the PS1 camera group for summit data access) was chosen primarily for historical reasons. It is most useful for distributing data in a serial fashion. There is an interface which allows clients to ask the server "what's new since this fileset". This is of great use to users who need timely access to recently acquired science data as it is processed.

The data store is of less utility to users of the postage stamp server. Those users simply need to know that a request has completed processing and where the results have been posted. In practice users who submit requests through the "upload request file method" know in advance what the name of their results file set will be and repeatedly ask for it getting 403 file not found errors until they exist.

The Postage Stamp Server is a system which allows users to access images from the IPP. Several different mechanisms exist for submitting requests to the system. The format for requests is ultimately a fits binary table that meets the requirements listed in the Postage Stamp server interface control document (ICD) (add link.)

The server is controlled by a set of pantasks tasks. Work to do is managed through a mysql database. One of the requirements of the server is that it support multiple projects simultaneously so the postage stamp tables are stored in database separate from the project (gpc1) database. Currently the postage stamp database is shared with the database that contains the data store tables (ippRequestServer).

A request is "submitted" by adding a row to a table in the postage stamp server database. As they are processed they transition between 3 states. A submitted request starts in new state. The first step is parsing the request file creating one or more jobs for each row in the request table. Parsing is a single threaded process and is a significant bottleneck for large request files. Once parsing is complete the request is set to state "run". At this point any jobs that were queued are processed in parallel. Finally when all jobs are complete the finish processing is performed on the request. This is also a single threaded process. The primary functions of the finish process are to build the results table and register the results fileset on the data store. When these operations are successfully completed the request is set to state 'stop'. Since nearly all of the IPP's single frame results are not saved (they are cleaned up) the postage stamp server is able to queue images to be regenerated (updated). This step is called dependency processing and is performed after parsing.

It is important to understand that the postage stamp system is intimately tied with the IPP. There are a number of characteristics of the system which are tied to the way the IPP operates. Many of these things have added significant complexity to the current system but will not be relevant in the public release environment.

• The request specification is quite rich allowing essentially any image ever processed by the IPP to be selected.
• There are several different selection methods. Many of these methods are rarely used outside of the IfA.
• Images often must be regenerated and this process must be managed.
• The set of available images is stored in the gpc1 database which is quite large.
• The input image files are stored in the nebulous file system.
• results are stored on the data store

Image Data to be provided by a PS1 Public Release Image Server

The following is my understanding (guess) of the image data to be provided with the public release.

• Postage stamp subsets (cut outs) and complete single frame warp skycell images
  • These are the data products from IPP warp processing (full state) which are being preserved from processing version 3 (PV3).
  • Are results from the entire run needed, psfs, cmfs, log files etc. I suspect no.
• Postage stamps and complete stack skycell images
  • These are the data products from the IPP stack processing
  • To provide photometry information for headers and detections equivalent to what the IPP server odes today the skycal cmfs for each skycell are needed (or at least the FITS headers).
• Stack Summary images for the sky. (MD or just 3PI?)

Note: The IPP postage stamp server delivers images based on a single IPP image. That is we do not mosaic images together although is is an often requested feature.

Data products that will not be included in the public release

• raw gpc1 images
• detrended chip stage images
• difference images
• catalogs (except staticsky)

The chip and diff stage images were not preserved by the IPP for PV3 because of insufficient space.

Metadata Required to Support the implementation of an Image Server

The plan is to ship the images pre installed on the servers being shipped back to STSCI. The IPP locates images using the Nebulous system. Nebulous is essentially a sophisticated lookup tables that maps abstract paths to NFS disk locations in the IPP cluster. For example: The file referenced by neb://path_to_bills_file/myfile.fits might resolve to an actual file on disk with a name like /data/stsci19.2/nebulous/a8/a7/6829803927.afile.fits

Nebulous is implemented in the IPP through various application programming interfaces which use SOAP to talk to a set of HTTP servers. The HTTP servers communicate with the nebulous mysql database.

There are several issues that need to be resolved before these systems are delivered.

• Data that must stay in the IPP cluster will need to be shuffled off to other systems or replicated.
• Some system that implements the Nebulous interfaces or a subset thereof will need to be delivered. Of course that could be nebulous but:
  • The full nebulous database currently takes up 3.7 TB and contains information on files that are not part of the delivery
  • Nebulous contains many features that might not be relevant to the new owners
  • It introduces depencies on NFS, apache, and mysql. These might not be issues but should be noted.
  • Management requires expertise that does not currently exist outside of the IPP team.
• Information describing the composition of the delivered IPP data products is required.
  • This could be a modified copy of the GPC1 mysql database (594 GB) with the states of various data is modified to match the data being shipped.
  • Alternatively a database with a simplfied structure (single frame data represented by warp tables augmented with exposure metadata from rawExp) could be loaded