Proposed Postage Stamp User Interface 2012-09-04 {{{ The PS1 software team is finally applying resources to produce a good interactive interface to the IPP postage stamp server. This will be included as part of the PS1 SCI (terminology) interface along side PSPS. The original web interface prototype has many significant limitations. The most important is that it is quite IPP centric. The second is that the user experience is not friendly. Finally, it supports neither the the complete feature set of the existing postage stamp server interface or various other features that users require. The purpose of this note is to outline a more user oriented interface and certain enhancements to the IPP system that are required to implement them. There are three major functions of the current postage stamp request interface 1. Select the images of interest 2. Identify the region of interest - the set of pixels to be extracted 3. Select the output data products to be returned. Some of these functions are really "database operations" yet the postage stamp request specification does not provide very complete set of selectors. In this document we propose splitting many of those functions off from request specification. In particular request creation and submission can optionally be done in two steps. 1. Ask the system what images are available that satisfy selection parameters 2. Select which images from the set returned are to have stamps extracted and submit In the end the experience changes from a "fire and hope" system, to a more controlled user managed system. Review of the Postage Stamp Interface ------------------------------------- The postage stamp server interface is described in the interface control document (LINK). Requests are submitted to the server as fits tables. Each row in the table specifies one set of coordinates and parameters to select the input images and output data products. When processed by the server each row generates zero or more postage stamp jobs. When all of the jobs queued for a given request the results are bundled into a fileset on the IPP data store. A fits file containing a table is also created. Each row in this results file lists the files included in the output and various parameters of the source images. Images of Interest - Stage -------------------------- The postage stamp server interface is based on the various stages of the IPP. Images are selected from one of the stages of the IPP pipeline raw chip warp stack diff Raw stage --------- The images acquired by the GPC1 camera. 60 fits files per exposure on for each OTA (chip), 64 or more extensions per file 1 for each cell. Raw images are of limited use to end users. Chip stage ---------- Outputs from the IPP chip and camera stage. At the chip stage images have been detrended. These operations include dark, flatfield, burntool, and various other chip (and cell) specific corrections. These images have had a fit to the sky background subtracted. Photometry is performed at this stage. At the camera stage the chip stage outputs for the 60 OTAs are analyzed together to arrive at the astrometric solution and the initial zero point is measured. The postage stamp server uses the camera stage astrometric solution to select the pixels and includes the solution as the WCS in the headers of the output files. The zero point measured at the camera stage is not currently included. That must be fixed. Warp stage ---------- After the camera stage the images are transformed (warped) to a given skycell tessellation. The resulting images have right ascension and declination as the X and Y axes. Each skycell gets contributions from chip stage images from one or more of the OTAs. The photometry outputs from the various input chips are combined. Photometry is *not* repeated at the warp stage. The sources files contain the sources detected at the chip stage with coordinates transformed to skycell coordinates. Stack Stage ----------- At the stack stage multiple warps are combined together. More details on this process will be discussed below in the section on data products. Diff Stage ---------- At the Diff stage various combinations of images are subtracted from each other to find differences. These are used to find transient events and moving objects. The major difference modes are Warp-Warp Two individual frames are compared Warp-Stack A single frame is compared to a template stack image Stack-Stack Stack created from a single night's exposures are compared to a template stack There are some other stages of the IPP pipline which are currently ignored by the current postage stamp interface. Many of these are critical to a complete understanding of the data DVO - Associates source detections from single frames and stacks into objects. Zero point and astrometric measurements are refined. The results are exported to PSPS but are not fed back into the images returned by the postage stamp server. Static Sky and Sky Calibration - Performs photometry and astrometric correction to stacked images including extended source measurements. These results are imported into PSPS but for more rapid turn around and easy association with images these results should be available through the pstamp server. Chip Background Stage - Certain images undero further processing to remove the background correction that was applied at the chip stage. (Currently only used for M31.) It has been recommended to add a feature to the postage stamp server to optionally perform this operation on an arbitrary image. Work has been done to implement this but haa not yet been deployed. Warp Background Stage chip background outputs warped to sky coordinates. Background restored stacks - For analysis of large extended objects the background correction is problematic. It is a requirement to the IPP to make avalable through the postage stamp server stack images with the background restored. A project has been started to implement the background restoration. Suggested New user interface ---------------------------- The selection by stage makes perfect sense for the implementation but requires some knowledge of the workings of the IPP. The IPP terminology is somewhat different than that used by PSPS and many users are not familiar with it. We assert here that they shouldn't need to be. The interface should be presented as a series of choices which more reflect the nature of the underlying data than the architecture of the IPP The image type choices should be presented as follows 1. Single Frame Images chip coordinate pixels warped coordinate pixels 2. Stacked Images 3. Difference Images 4. Raw images The UI should present options 1 and 2 prominently and make 3 and especially 4 advanced options. Selection of source images -------------------------- Once the stage is identified, the actual input images need to be selected. The server supports several image selection methods. Note that currently postage stamps are extracted from a *single* IPP image product. No combination between chips or skycells is performed. bycoord Images are selected based on supplied RA and DEC byexp Single frame images selected by name (IPP rawExp.exp_name PSPS FrameMeta.name) byid Specific images are selected by IPP database ID (exp_id, chip_id, warp_id, stack_id, diff_id) byskycell warp, stack, and diff images selected based on tessellation id (tess_id) and skycell_id bydiff specialized method used by MOPS to select images from a particular difference detection Image of Interest filtering --------------------------- A number of parameters can be supplied with the request to narrow down the set of target images. Two obvious ones are filter and observation date. The IPP data_group can be used to narrow down the data set. In practice this is only useful with advice from the IPP team or access to a copy of the IPP's gpc1 mysql data base. Problems with this interface for end users ------------------------------------------ Other than bycoord these methods are not friendly to end users. Unless the user has access to a copy of the gpc1 database or software to find out which skycell contains a given point int the sky it is difficult to narrow down images to specific data sets. Now that PSPS is being loaded with more PSPS data this problem is relieved somewhat. However given inevitable the time lag between the exposures and frames information that PSPS does not currently have is needed to select recent data efficiently. Given the vast data set of PAN Starrs by coordinates has its problems. Asking for all chip images for a spot in one of the medium deep fields is easy to do, will return thousands of images, and is probably not what a user really wants. Also frames have been processed multiple times. The IPP has no notion of what the right set of runs should be "released" to end users for a given exposure. A solution to this problem has been proposed (document in press LINK). In this document we will assume that those features will be available. Suggested Features for the New UI --------------------------------- It is desirable to be able to preview what the results of a given set of requests will include. In the current interface this is problematic because this information is obtained by parsing the request file which is an extremely inefficient operation. Since the new UI is inside a "database environment" we have an opportunity to improve this significantly. While finding all of the images for a given stage that correspond to a set of coordinates is somewhat expensive finding the individual frames is more straightforward. We suggest moving the selection of frames for processing from the postage stamp server to the web interface. The Web interface will thus only do byexp or byid requests. Choices in the UI - Single Frame Images --------------------------------------- Coordinates Sky coordinates RA DEC decimal degrees and sexagesimal standard formats pixel coordinates (advanced) Note that it is desirable to accept a list of coordinates and process them together. Users should be able to upload lists with certain parameters to the UI for example RA DEC FILTER dateobs_min dateobs_max. The user should also be able to provide a list of object or detection ids obtained from a query. Choices in the UI - All image types ----------------------------------- Stamp Size pixels arc seconds entire image Image Selection by coordinates by frame id by skycell id (advanced) Data Release Best (default) PSPS data / release version Original Nightly Processing (not sure if we need this but it's easy) This is a new concept which requires some work on the IPP side. It should probably eventually default to the latest version in PSPS database for a given survey but at the current time this is too restrictive as it hides much of the data. If we are not selecting by frame id a few more selectors should be made available Survey 3PI MD (includes M31, CNP, and STS) SAS SS To use this information requires a new interface with the postage stamp server. Note the individual frames can be identified by rawExp.obs_mode.... but this is buggy for old data. Perhaps something should be added to the new ippRelease table for the given frame. Choices in the UI - Single Frame Images --------------------------------------- Filter hints should perhaps be provided. For example w shouldn't be a choice in the 3pi or MD surveys. STS is only i. Observation Date min Default to the beginning of the mission to avoid buggy data. User can change to accept earlier data if desired max default no limit Data Group This exists in the current interface and should be included but not displayed prominently. Preview Function ---------------- Once the user has selected the parameters he can command the system to perform the image lookup for preview and selection. Ideally this would be a PSPS operation but since we want to make all PS1 data available we propose using the gpc1 database (or a replicated copy or subset). The query would use the selection parameters described above to query the new table ipp release table. This table will have entries for each exposure that indicate which images in the IPP have been "released" by the IPP. Paremeters available exp_id chip_id warp_id diff_id survey_id release (or data version) Other parameters (ra, dec, filter, dateobs) will be included by joining to the rawExp table. When the query returns the relevant data will be presented to the users in a format that allows them to select the target frames for the postage stamp requests. frameID survey filter "released version" comment By released version I mean something that tells whether the data is from one of the grand reprocessed data sets or is the original nightly processing. If a list of coordinates was provided the presentation of the results to the user in friendly way may be tricky. Image Selection from results of PSPS queries -------------------------------------------- One obvious use case is to request postage stamps from the objects or detections in results of a specific PSPS query. Object coordinates could be used to create a bycoord request to the server. If individual detections are available then the frameID may be obtained and byexp requests could be used. A user might also be interested in looking at stamps from frames where the object was not detected so that should be supported as well. Stacked images -------------- The IPP will need a table to manage releases for Stacked images as well. The fundamental unit in this case is the Skycell. stack_id skycell_id survey_id release Other parameters, ra, dec, filter, tess_id will be available through join to the tables stackRun and skycell NOTE: upon further reflection, this funcion could be handled simply by enhancing the table skycalResults which gets updated when psastro gets run on the skycell and could be updated when the data is exported to PSPS. QUESTION: One feature that might be interesting is to find the stacks that a particular single frame is included in. Currently this information is not availalble from the PSPS or postage stamp server. It can be derived from the IPP database. Is this useful? Data Products Available Current Interface ----------------------------------------- Image Mask Variance Sources (CMF) Chip stage psf model Stack Unconvolved images Interface Between Web Interface and postage stamp server -------------------------------------------------------- The user interface will be implemented as part of SCI which from the user point of view is part of PSPS. (Need name for this component) Image listing, stamp extraction, and request status reporting will be performed by the IPP. An set of communication interfaces will be developed that allow these systems to communicate. On the IPP side this will be a simple set of "commands" that are triggered by http requests from the PSPS side. The following sections briefly outline the commands List PSTAMP target images ------------------------- A set of coordinates and other parameters are submitted. The postage stamp server performs *efficient* lookups into the IPP database tables and returns a set of images that meet the selection criteria. The information returned will be sufficient to provide an informative presentation to the user and the means to submit an efficient postage stamp request (byid). One informative piece of information could be flags that indicate whether the target is available on disk for immediate stamp extraction or will require regeneration (update processing) Submit Request -------------- Once the user has selected the target images the UI (wrong term) will submit a request table (or equivalent) to the server for processing. Request Status -------------- Will return the status for a particular request Job Status ---------- Will return the status of particular job(s) Request Control --------------- Cancel job Cancel request etc When complete the results will be placed on a particular IPP data store product where the UI can present to the users as appropriate }}}