Notes on Implementation of the Public Release of PS1 image data products

Note started by: Bill Sweeney 2015-05-04

The plan of record is to migrate the PS1 images to Space Telescope Institute by shipping machines owned by STSCI that have been operating as file servers in the IPP cluster on Maui. These machines will contain file systems with the desired data products. STSCI will then provide some method for accessing the data.

This plan raises a number of outstanding issues and technical work that will be required by the IPP team and the group at STSCI.

This note is intended to be a starting point for fleshing out the requirements and plan for implentation of this work. First we discuss the current system for delivering images to the consortium. Next we talk about what the requirements for an image server for public release might be. Then we discuss what data must be delivered by the IPP to meet these requirements. Finally we have some thoughts on what IPP software might be adapted to meet the requirements and what might need to be developed.

The contents here in the first draft come from my understanding of what is needed and what is planned. I know that others have already thought through many of these issues in some detail.

Background

The primary way for PS1SC users to access image data products has been through the IPP data store interface. Files are contained in File Sets (sub-directories) in particular Products (directories) on the data store HTTP server. This server is part of the IPP cluster on Maui. Access by users external to the IfA is provided through a proxy server in Manoa on Maui. Access is restricted to authorized users by restricting HTTP access to hosts with IP addresses that belong to members of the PS1 science consortium. Access to certain products is more restricted. Listings of the various levels are given by cgi scripts run by the HTTP server.

Data is placed in the data store (file sets are registered) by the IPP distribution tasks and by the IPP postage stamp server. The distribution jobs are queued by the IPP when processing various data sets and reductions. Postage stamp results contain the results of requests submitted by individual users.

Use of the data store interface (invented by the PS1 camera group for summit data access) was chosen primarily for historical reasons. It is most useful for distributing data in a serial fashion. There is an interface which allows clients to ask the server "what's new since this fileset". This is of great use to users who need timely access to recently acquired science data as it is processed.

The data store is of less utility to users of the postage stamp server. Those users simply need to know that a request has completed processing and where the results may be found. In practice, users who submit requests through the "upload request file method" know in advance what the name of their results file set will be and repeatedly ask for it getting 403 file not found errors until they exist. PSI provides users with a link to the results filesets.

The Postage Stamp Server is a system which allows users to access images from the IPP. Several different mechanisms exist for submitting requests to the system. The format for requests is ultimately a fits binary table that meets the requirements listed in the Postage Stamp server interface control document (ICD) (add link.)

The server is controlled by a set of pantasks tasks. Work to do is managed through a mysql database. One of the requirements of the server is for it to support multiple projects simultaneously. For this reason, the postage stamp tables are stored in database separate from the project (gpc1) database. Currently the postage stamp database is shared with the database that contains the data store tables (ippRequestServer).

A request is "submitted" by adding a row to a table in the postage stamp server database. As they are processed they transition between 3 states. A submitted request starts in new state. The first step is parsing the request file creating one or more jobs for each row in the request table. Parsing is a single threaded process and is a significant bottleneck for large request files. Once parsing is complete the request is set to state "run". At this point any jobs that were queued are processed in parallel. Finally when all jobs are complete the finish processing is performed on the request. This is also a single threaded process. The primary functions of the finish process are to build the results table and register the results fileset on the data store. When these operations are successfully completed the request is set to state 'stop'. Since nearly all of the IPP's single frame results are not saved (they are cleaned up) the postage stamp server is able to queue images to be regenerated (updated). This step is called dependency processing and is performed after parsing.

It is important to understand that the postage stamp system is intimately tied with the IPP. There are a number of characteristics of the system which are tied to the way the IPP operates. Many of these things have added significant complexity to the current system but will not be relevant in the public release environment.

• The request specification is quite rich allowing essentially any image ever processed by the IPP to be selected.
• There are several different selection methods. Many of these methods are rarely used outside of the IfA.
• Images often must be regenerated and this process must be managed.
• The set of available images is stored in the gpc1 database which is quite large.
• The input image files are stored in the nebulous file system.
• results are stored on the data store

Image Data to be provided by a PS1 Public Release Image Server

The following is my understanding (guess) of the image data to be provided with the public release.

• Postage stamp subsets (cut outs) and complete single frame warp skycell images, masks, and variance images
  • These are the data products from IPP warp processing (full state) which are being preserved from processing version 3 (PV3).
  • Are results from the entire run needed, psfs, cmfs, log files etc. I suspect no.
• Postage stamps and complete stack skycell images, masks, and variance images
  • These are the data products from the IPP stack processing
  • To provide photometry information for headers and detections equivalent to what the IPP server odes today the skycal cmfs for each skycell are needed (or at least the FITS headers).
• Stack Summary images for the sky. (MD or just 3PI?)
• Selection methods: by coordinate, by skycell, by exposure (single frame only).
  • No bydiff. No byid ?

Note: The IPP postage stamp server delivers images based on a single IPP image. That is we do not mosaic images together although is is an often requested feature that should be considered.

Data products that will not be included in the public release

• raw gpc1 images
• detrended chip stage images
• difference images
• catalogs (except staticsky)

The chip and diff stage images were not preserved by the IPP for PV3 because of insufficient space. The lack of difference images will somewhat reduce the utility of the difference detections.

Metadata Required to Support the implementation of an Image Server

The plan is to ship the images pre installed on the servers being shipped back to STSCI. The IPP locates images using the Nebulous system. Nebulous is essentially a sophisticated lookup table that maps abstract paths to NFS disk locations in the IPP cluster. For example: The file referenced by neb://path_to_bills_file/myfile.fits might resolve to an actual file on disk with a name like /data/stsci19.2/nebulous/a8/a7/6829803927.afile.fits

Nebulous is implemented in the IPP through various application programming interfaces which use SOAP to talk to a set of HTTP servers. The HTTP servers communicate with the nebulous mysql database.

There are several issues that need to be resolved before these systems are delivered.

• Data that must stay in the IPP cluster will need to be shuffled off to other systems or replicated to machines that are staying on Maui.
• Some system that implements the Nebulous interfaces or a subset thereof will need to be delivered. Of course that could be nebulous but:
  • The full nebulous database currently takes up 3.7 TB and contains information on files that are not part of the delivery
  • Nebulous contains many features that might not be relevant to the new implementation
  • It introduces depencies on NFS, apache, and mysql. (These might not be issues but should be noted.)
  • Management of the Nebulous system requires expertise that does not currently exist outside of the IPP team.
• Information describing the composition of the delivered IPP data products is required.
  • This could be a modified copy of the GPC1 mysql database (594 GB) with the states of various data is modified to match the data being shipped.
  • Alternatively a database with a simplfied structure (single frame data represented by warp tables augmented with exposure metadata from rawExp) could be created
• Description of the skycell tessellations. (Tessellation directories).

In addition we should prepare for the inevitable and come up with a mechanism for the IPP to regenerate images that get lost due to system failures and getting back to STSCI.

Postage Stamp Server Requirements

Given that the IPP postage stamp server is so tightly coupled with the IPP and has several complexities to support requirements that the public server does not have, the implementors at STSCI may want to build a different implementation.

The requirements include

• A graphical user interface similar to what PSI (or SDSS) provides
• A mechanism for advanced users to programmaticaly submit requests. (The IPP's upload mechanism is basically a prototype and not a good model).
• System for locating images of interest that correspond to the users' requests
• Pixel extractor

It is possible to use the IPP software but would require a nearly full IPP build.

Data products to be available on STSCI nodes

These are the data products (and copy counts) for all of the image data that will be provided to STSCI.

Stage	Filerule Type	Description	IPP local copies	STSCI disk copies	File extension
warp	PSWARP.OUTPUT	warp data image	0	1	.fits
	PSWARP.OUTPUT.MASK	warp mask image	0	1	.mask.fits
	PSWARP.OUTPUT.VARIANCE	warp variance image	0	1	.wt.fits
	PSWARP.OUTPUT.SOURCES	warp catalog data	2	1	.cmf
	PSWARP.CONFIG	warp config file	2	1	.pswarp.mdc
stack	PPSTACK.UNCONV	stack data image	2	1	.unconv.fits
	PPSTACK.UNCONV.MASK	stack mask image	2	1	.unconv.mask.fits
	PPSTACK.UNCONV.VARIANCE	stack variance image	2	1	.unconv.wt.fits
	PPSTACK.UNCONV.EXP	stack exposure time map	2	1	.unconv.exp.fits
	PPSTACK.UNCONV.EXPNUM	stack input number map	2	1	.unconv.num.fits
	PPSTACK.UNCONV.EXPWT	stack weighted exposure time map	2	1	.unconv.expwt.fits
	PSPHOT.PSF.SKY.SAVE	stack PSF	2	1	.psf
	PPSTACK.CONFIG	stack config file	2	1	.ppStack.mdc
skycal	PSPHOT.STACK.OUTPUT	skycal catalog	2	1	.stk.{FILE_ID}.cmf
diff	PPSUB.OUTPUT.KERNELS	difference image convolution kernel	2	1	.subkernel
	PPSUB.OUTPUT.SOURCES	difference catalog	2	1	.cmf
	PPSUB.CONFIG	difference config file	2	1	.ppSub.mdc

In addition, the shuffle to the stsci cluster is producing manifest files containing information about the files located there. The files are found in /data/stsci$XX.$Z/MANIFEST/$stage/$stage.$stage_id.MANIFEST. Each line of the manifest file connects the nebulous ext_id, the disk location, and the md5sum for that file. For example, /data/stsci19.2/MANIFEST/warp/warp.1502611.MANIFEST contains:

gpc1/LAP.PV3.20140730.ipp/2015/02/27/o5861g0254o.415059/o5861g0254o.415059.wrp.1502611.skycell.1351.090.mask.fits /data/stsci02.0/nebulous/43/f8/7095027312.gpc1:LAP.PV3.20140730.ipp:2015:02:27:o5861g0254o.415059:o5861g0254o.415059.wrp.1502611.skycell.1351.090.mask.fits 01b29c0ce6386d9ff4679b25eaaa1343
gpc1/LAP.PV3.20140730.ipp/2015/02/27/o5861g0254o.415059/o5861g0254o.415059.wrp.1502611.skycell.1351.090.fits /data/stsci02.0/nebulous/e8/d7/7095027362.gpc1:LAP.PV3.20140730.ipp:2015:02:27:o5861g0254o.415059:o5861g0254o.415059.wrp.1502611.skycell.1351.090.fits 1ecb496fc97a007b3138e39ebcd939b8
gpc1/LAP.PV3.20140730.ipp/2015/02/27/o5861g0254o.415059/o5861g0254o.415059.wrp.1502611.skycell.1351.090.wt.fits /data/stsci02.0/nebulous/8d/9c/7095027257.gpc1:LAP.PV3.20140730.ipp:2015:02:27:o5861g0254o.415059:o5861g0254o.415059.wrp.1502611.skycell.1351.090.wt.fits 1e24f553c19e04a349b389404e95fc12
gpc1/LAP.PV3.20140730.ipp/2015/02/27/o5861g0254o.415059/o5861g0254o.415059.wrp.1502611.skycell.1351.090.pswarp.mdc /data/stsci02.0/nebulous/cd/c7/7095027329.gpc1:LAP.PV3.20140730.ipp:2015:02:27:o5861g0254o.415059:o5861g0254o.415059.wrp.1502611.skycell.1351.090.pswarp.mdc a9273f69466fd4e13086951189482064
gpc1/LAP.PV3.20140730.ipp/2015/02/27/o5861g0254o.415059/o5861g0254o.415059.wrp.1502611.skycell.1351.090.cmf /data/stsci02.0/nebulous/90/33/7095021595.gpc1:LAP.PV3.20140730.ipp:2015:02:27:o5861g0254o.415059:o5861g0254o.415059.wrp.1502611.skycell.1351.090.cmf 3419907a957c00d24f8e65df75977521

GPC1 database to Nebulous key translation

Each component product in the GPC1 database (warpSkyfile, stackSumSkyfile, diffSkyfile, etc.) contains a path_base field. This contains the initial nebulous key for the output files for that product. Appending the extension listed in the table above to that path_base produces the full nebulous key that points to the disk file. The {FILE_ID} field for the skycal catalog is substituted with the stack_id associated with that skycal run.

The nebulous keys produced are all of the form neb://OPTIONAL_DISK_VOLUME/arbitrary/directory/path/file.extension. The IPP source code contains the parsing code in the parse_neb_key function of Nebulous-Server/lib/Nebulous/Key.pm. The quick version is that everything up to the third / can be removed, yielding the nebulous ext_id, which is simply the directory structure and filename. For the prototype, the ext_id would be arbitrary/directory/path/file.extension. The GPC1 database path_base entries generally have an explicit disk volume listed (such as stsci02.0), as those were the attempted targets for the files. The actual disk location listed in the manifest file will be correct, and may disagree with the path_base location.

Example

For the example, warp_id = 644431: select path_base from warpSkyfile WHERE warp_id = 644431 AND quality = 0 LIMIT 1;

path_base

neb://stsci07.2/gpc1/LAP.ThreePi.20120706/2012/11/09/o5449g0532o.222926/o5449g0532o.222926.wrp.644431.skycell.1233.020

For the warp image for this product, the extension is just .fits, and after removing the volume, the ext_id is gpc1/LAP.ThreePi.20120706/2012/11/09/o5449g0532o.222926/o5449g0532o.222926.wrp.644431.skycell.1233.020.fits

This is listed in the manifest as having real disk location /data/stsci00.1/nebulous/b0/69/7208475404.gpc1:LAP.ThreePi.20120706:2012:11:09:o5449g0532o.222926:o5449g0532o.222926.wrp.644431.skycell.1233.020.fits

Notes on Files Missing from Manfest

ManifestMissingFiles

Note on transferring files back from STScI

The files on the STScI system are mounted in directories with paths like /data/ps1/node03/stps03.1/. This example case maps to what was /data/stsci02.1/ on the IPP cluster; the files are listed by a node number between 1 and 20 (corresponding to stsci00 through stsci19), and then into volumes with the same incremented counter relative to the IPP naming system. The node name is also changed from stsci to stps.