Proposed Postage Stamp User Interface 2012-09-04

The PS1 software team is finally applying resources to produce a good interactive
interface to the IPP postage stamp server. This will be included as part of the PS1 SCI (terminology)
interface along side PSPS. 

The original web interface prototype has many significant limitations.  The most important is that
it is quite IPP centric.  The second is that the user experience is not friendly. Finally, it 
supports neither the the complete feature set of the existing postage stamp server interface or various other
features that users require.

The purpose of this note is to outline a more user oriented interface and certain enhancements
to the IPP system that are required to implement them.

There are three major functions of the current postage stamp request interface

    1. Select the images of interest
    2. Identify the region of interest - the set of pixels to be extracted
    3. Select the output data products to be returned.

Some of these functions are really "database operations" yet the postage stamp request
specification does not provide very complete set of selectors.

In this document we propose splitting many of those functions off from request specification.
In particular request creation and submission can optionally be done in two steps. 
 1. Ask the system what images are available that satisfy selection parameters
 2. Select which images from the set returned are to have stamps extracted and submit
 
In the end the experience changes from a "fire and hope" system, to a more controlled user managed system.

Review of the Postage Stamp Interface
-------------------------------------
The postage stamp server interface is described in the interface control document (LINK).

Requests are submitted to the server as fits tables. Each row in the table specifies one set
of coordinates and parameters to select the input images and output data products. When processed
by the server each row generates zero or more postage stamp jobs.  When all of the jobs 
queued for a given request the results are bundled into a fileset on the IPP data store.
A fits file containing a table is also created. Each row in this results file lists the files
included in the output and various parameters of the source images.

Images of Interest - Stage
--------------------------
The postage stamp server interface is based on the various stages of the IPP. Images are
selected from one of the stages of the IPP pipeline

    raw     
    chip   
    warp  
    stack
    diff

Raw stage
---------
The images acquired by the GPC1 camera. 60 fits files per exposure on for each OTA (chip), 
64 or more extensions per file 1 for each cell. Raw images are of limited use to end users.

Chip stage
----------
Outputs from the IPP chip and camera stage.  At the chip stage images have been detrended. These
operations include dark, flatfield, burntool, and various other chip (and cell) specific corrections.  
These images have had a fit to the sky background subtracted. Photometry is performed at this
stage. At the camera stage the chip stage outputs for the 60 OTAs are analyzed together to arrive
at the astrometric solution and the initial zero point is measured.

The postage stamp server uses the camera stage astrometric solution to select the pixels
and includes the solution as the WCS in the headers of the output files. The zero point measured
at the camera stage is not currently included. That must be fixed.

Warp stage
----------
After the camera stage the images are transformed (warped) to a given skycell tessellation.
The resulting images have right ascension and declination as the X and Y axes. Each skycell gets
contributions from chip stage images from one or more of the OTAs. The photometry outputs from
the various input chips are combined.  Photometry is *not* repeated at the warp stage. The sources
files contain the sources detected at the chip stage with coordinates transformed to skycell
coordinates.

Stack Stage
-----------
At the stack stage multiple warps are combined together. More details on this process will be
discussed below in the section on data products.

Diff Stage
----------
At the Diff stage various combinations of images are subtracted from each other to find differences.
These are used to find transient events and moving objects. The major difference modes are
    Warp-Warp   Two individual frames are compared
    Warp-Stack  A single frame is compared to a template stack image
    Stack-Stack Stack created from a single night's exposures are compared to a template stack

There are some other stages of the IPP pipline which are currently ignored by the current postage stamp
interface. Many of these are critical to a complete understanding of the data
    
    DVO - Associates source detections from single frames and stacks into objects. Zero point and
          astrometric measurements are refined. The results are exported to PSPS but  are not fed
          back into the images returned by the postage stamp server.

    Static Sky and Sky Calibration - Performs photometry and astrometric correction to stacked images
        including extended source measurements. These results are imported into PSPS but for more rapid
        turn around and easy association with images these results should be available through the pstamp server.

    Chip Background Stage - Certain images undero further processing to remove the background correction 
    that was applied at the chip stage. (Currently only used for M31.) It has been recommended to add a
    feature to the postage stamp server to optionally perform this operation on an arbitrary image.
    Work has been done to implement this but haa not yet been deployed.

    Warp Background Stage  chip background outputs warped to sky coordinates.

    Background restored stacks - For analysis of large extended objects the background correction is
    problematic. It is a requirement to the IPP to make avalable through the postage stamp server
    stack images with the background restored. A project has been started to implement the 
    background restoration.

Suggested New user interface
----------------------------
The selection by stage makes perfect sense for the implementation but requires some knowledge of the
workings of the IPP. The IPP terminology is somewhat different than that used by PSPS and many users
are not familiar with it. We assert here that they shouldn't need to be.

The interface should be presented as a series of choices which more reflect the nature of the 
underlying data than the architecture of the IPP

The image type choices should be presented as follows

    1. Single Frame Images
        chip coordinate pixels
        warped coordinate pixels

    2. Stacked Images

    3. Difference Images

    4. Raw images

The UI should present options 1 and 2 prominently and make 3 and especially 4 advanced options.

    
Selection of source images
--------------------------
Once the stage is identified, the actual input images need to be selected. The server supports several
image selection methods. Note that currently postage stamps are extracted from a *single* IPP
image product. No combination between chips or skycells is performed.

    bycoord     Images are selected based on supplied RA and DEC 
    byexp       Single frame images selected by name (IPP rawExp.exp_name PSPS FrameMeta.name)
    byid        Specific images are selected by IPP database ID (exp_id, chip_id, warp_id, stack_id, diff_id)
    byskycell   warp, stack, and diff images selected based on tessellation id (tess_id) and skycell_id
    bydiff      specialized method used by MOPS to select images from a particular difference detection

Image of Interest filtering
---------------------------
A number of parameters can be supplied with the request to narrow down the set of target images. Two
obvious ones are filter and observation date. The IPP data_group can be used to narrow down the
data set. In practice this is only useful with advice from the IPP team or access to a copy of the
IPP's gpc1 mysql data base.

Problems with this interface for end users
------------------------------------------
Other than bycoord these methods are not friendly to end users. Unless the user has
access to a copy of the gpc1 database or software to find out which skycell contains
a given point int the sky it is difficult to narrow down images to specific
data sets. 

Now that PSPS is being loaded with more PSPS data this problem is relieved somewhat. However
given inevitable the time lag between the exposures and frames information that PSPS does not currently
have is needed to select recent data efficiently.

Given the vast data set of PAN Starrs by coordinates has its problems. Asking for all chip
images for a spot in one of the medium deep fields is easy to do, will return thousands of
images, and is probably not what a user really wants.

Also frames have been processed multiple times.  The IPP has no notion of what the right set of
runs should be "released" to end users for a given exposure. A solution to this problem has been proposed
(document in press LINK). In this
document we will assume that those features will be available.

Suggested Features for the New UI
---------------------------------
It is desirable to be able to preview what the results of a given set of requests will include.
In the current interface this is problematic because this information is obtained by parsing
the request file which is an extremely inefficient operation.

Since the new UI is inside a "database environment"  we have an opportunity to improve this
significantly.

While finding all of the images for a given stage that correspond to a set of coordinates is
somewhat expensive finding the individual frames is more straightforward. 

We suggest moving the selection of frames for processing from the postage stamp server to
the web interface. The Web interface will thus only do byexp or byid requests.


Choices in the UI - Single Frame Images
---------------------------------------

Coordinates
    Sky coordinates RA DEC decimal degrees and sexagesimal standard formats
    pixel coordinates (advanced)

Note that it is desirable to accept a list of coordinates and process them together. Users
should be able to upload lists with certain parameters to the UI for example
RA DEC FILTER dateobs_min dateobs_max.

The user should also be able to provide a list of object or detection ids obtained from a query.

Choices in the UI - All image types
-----------------------------------

Stamp Size
    pixels
    arc seconds
    entire image

Image Selection
    by coordinates
    by frame id
    by skycell id (advanced)

Data Release
    Best    (default)
    PSPS data / release version
    Original Nightly Processing (not sure if we need this but it's easy)

This is a new concept which requires some work on the IPP side. It should probably eventually
default to the latest version in PSPS database for a given survey but at the current time this is too 
restrictive as it hides much of the data.
    
If we are not selecting by frame id a few more selectors should be made available

Survey
    3PI
    MD (includes M31, CNP, and STS)
    SAS
    SS

To use this information requires a new interface with the postage stamp server. Note the individual
frames can be identified by rawExp.obs_mode.... but this is buggy for old data. Perhaps something
should be added to the new ippRelease table for the given frame.

Choices in the UI - Single Frame Images
---------------------------------------
Filter
    hints should perhaps be provided. For example w shouldn't be a choice in the 3pi or MD surveys.
    STS is only i.

Observation Date
    min Default to the beginning of the mission to avoid buggy data. User can change to accept
        earlier data if desired
    max default no limit


Data Group  This exists in the current interface and should be included but not displayed prominently.

Preview Function
----------------
Once the user has selected the parameters he can command the system to perform the image lookup for
preview and selection.

Ideally this would be a PSPS operation but since we want to make all PS1 data available we propose
using the gpc1 database (or a replicated copy or subset).

The query would use the selection parameters described above to query the new table ipp release table.
This table will have entries for each exposure that indicate which images in the IPP have been 
"released" by the IPP. Paremeters available
    exp_id
    chip_id
    warp_id
    diff_id 
    survey_id
    release (or data version)

Other parameters (ra, dec, filter, dateobs) will be included by joining to the rawExp table.

When the query returns the relevant data will be presented to the users in a format that allows
them to select the target frames for the postage stamp requests. 

    frameID survey filter "released version" comment

By released version I mean something that tells whether the data is from one of the grand reprocessed data
sets or is the original nightly processing.

If a list of coordinates was provided the presentation of the results to the user in friendly
way may be tricky.


Image Selection from results of PSPS queries
--------------------------------------------
One obvious use case is to request postage stamps from the objects or detections in
results of a specific PSPS query.

Object coordinates could be used to create a bycoord request to the server. If individual detections are
available then the frameID may be obtained and byexp requests could be used. A user might also
be interested in looking at stamps from frames where the object was not detected so that should
be supported as well.

Stacked images
--------------
The IPP will need a table to manage releases for Stacked images as well. The fundamental unit in this
case is the Skycell.
    stack_id
    skycell_id
    survey_id
    release

Other parameters, ra, dec, filter, tess_id will be available through join to the tables stackRun 
and skycell

NOTE: upon further reflection, this funcion could be handled simply by enhancing the table skycalResults which gets updated
when psastro gets run on the skycell and could be updated when the data is exported to PSPS.

QUESTION: One feature that might be interesting is to find the stacks that a particular single frame is included
in. Currently this information is not availalble from the PSPS or postage stamp server. It can be
derived from the IPP database. Is this useful?


Data Products Available Current Interface
-----------------------------------------
    Image
    Mask
    Variance
    Sources (CMF)

Chip stage
    psf
    model

Stack
    Unconvolved images

Interface Between Web Interface and postage stamp server
--------------------------------------------------------

The user interface will be implemented as part of SCI which from the user point of view is part of PSPS.
(Need name for this component)

Image listing, stamp extraction, and request status reporting will be performed by the IPP.

An set of communication interfaces will be developed that allow these systems to communicate. On the IPP
side this will be a simple set of "commands" that are triggered by http requests from the PSPS side.

The following sections briefly outline the commands

List PSTAMP target images
-------------------------
A set of coordinates and other parameters are submitted. The postage stamp server performs *efficient*
lookups into the IPP database tables and returns a set of images that meet the selection criteria.

The information returned will be sufficient to provide an informative presentation to the user and
the means to submit an efficient postage stamp request (byid). One informative piece of information
could be flags that indicate whether the target is available on disk for immediate stamp extraction
or will require regeneration (update processing)


Submit Request
--------------
Once the user has selected the target images the UI (wrong term) will submit a request table (or equivalent)
to the server for processing.

Request Status
--------------
Will return the status for a particular request

Job Status
----------
Will return the status of particular job(s)

Request Control
---------------
Cancel job
Cancel request
etc

When complete the results will be placed on a particular IPP data store product where the UI can present
to the users as appropriate