\documentclass[panstarrs,spec]{panstarrs}

\title{IPP Postage Stamp Server} % put in your title
\subtitle{Design}
\author{Bill Sweeney}
\audience{IPP}
\shorttitle{Postage Stamp Design}
\group{Pan-STARRS IPP}
\project{Pan-STARRS IPP}
\organization{Institute for Astronomy}
\version{DR}
\docnumber{PSDC-430-xxx}

\newcommand\ugriz{$u^\prime g^\prime r^\prime i^\prime z^\prime$}
\newcommand\grizy{$g r i z y$}


\begin{document}

\maketitle
\tableofcontents

\pagebreak
\pagenumbering{arabic}

\section{Overview}

The basic functionality of the postage stamp server is to accept lists
of postage stamp requests, determine the corresponding images, extract
the needed pixels from the images, and return the postage stamps to
the user.  The nominal mode of interaction as an interface to MOPS and
the other Science Clients is for the postage lists to be FITS tables
which are placed as incoming data on the IPP Data Store (or the data
store of the requesting system) and for the results to be placed back
on the IPP Data Store, along with a FITS table description of the
resulting output image files.  For the generic user, the requests and
the result may be sent directly via a web server.  

The basic information needed to by the postage stamp server (PSS) to honor a
request consists of: 1) the image of interest and 2) the region on the
image desired (ROI).  This information may be represented in a number of ways
which are discussed in section 2.1. % How do I include a reference by tag?

The implementation of the PSS is a set of \code{pantasks} tasks, standalone programs,
database tables, web server content, and associated programs.

\section{Postage Stamp Requests}


\subsection{Request Table Contents}

Postage stamp requests are packaged for submission to the PSS as a
FITS table. The contents of the table specify the ROI and parameters that 
enable the PSS to choose the images of interest. The columns contained in table are
shown in Table~\ref{RTableDef}. The \code{EXTNAME} for the FITS table is \code{PS1_PS_REQUEST}.

\begin{table}[h]
\begin{center}
\caption{Postage Stamp Request Table Contents}\label{RTableDef}
\begin{tabular}{lllc}
\hline
\hline
{\bf Column Name} & {\bf Datatype} & {\bf Values} &             {\bf Required } \\
\hline
\code{VERSION}  & string     & N.N               &                      yes \\
\code{PROJECT}  & string     & any               &                      yes \\
\code{USER_TAG} & string     & must be unique    &                      yes \\
\code{REQ_TYPE} & string     & bycoord byexp byid &                     yes \\
\code{IMG_TYPE} & string     & raw chip warp diff stack &               yes \\
\code{EXPOSURE} &    string  & any               &                      byexp \\
\code{ID}       &    string  & any               &                      byid \\
\code{CLASS_ID} &    string  & any               &                      byid raw \& byexp raw \\
\code{CHIP_ID}  &    string  & any               &                      byid chip \& byexp chip \\
\\
\code{FILTER}   &    string  & any               &                      no \\
\code{DATE_MIN} &    string  & YYYYMMDD-HH:MM:SS &                      no \\
\code{DATE_MAX} &    string  & YYYYMMDD-HH:MM:SS &                      no \\
\\
\code{RA}       &    string  & sexagesimal or decimal deg &             bycoord, null if pixcenter \\
\code{DEC}      &    string  & segagesimal or decimal deg &             bycoord, null if pixcenter \\
\code{CENTER_X} &    S32     & any               &                      null if not pixcenter \\
\code{CENTER_Y} &    S32     & any               &                      null if not pixcenter \\
\code{D_RA}     &    F32     & \code{arcsec > 0} &                      0 if pixrange \\
\code{D_DEC}    &    F32     & \code{arcsec > 0} &                      0 if pixrange \\
\code{WIDTH}    &    U32     & \code{> 0}        &                      0 if not pixrange \\
\code{HEIGHT}   &    U32     & \code{> 0}        &                      0 if not pixrange \\
\hline
\end{tabular}
\end{center}
\end{table}

\subsubsection{Image Types}
All requests need IMG\_TYPE. This parameter selects the stage in the
image processing pipeline from which the the postage stamp will be extracted.
The allowed values for IMG\_TYPE are raw, chip, warp, diff, and stack.

\subsubsection{Request Types}
The REQ\_TYPE determines how the images of interest are selected. There are three allowed values. 

\begin{itemize}
\item{\textbf{bycoord}} Images are selected by querying the IPP DVO database for images that
contain the celestial coordinates of the ROI center. 
The set of images of interest may be limited by specifying certain criteria as described
in Section \ref{ImageFilters}.

\item{\textbf{byexp}}  The image of interest is the image from a particular exposure
with type IMG\_TYPE.
If IMG\_TYPE is raw the CLASS\_ID must be provided. If IMG\_TYPE is chip 
and the ROI center is specified in pixel coordinates then CHIP\_ID must be specified.

\item{\textbf{byid}}  The image with given IMG\_TYPE and ID is the image of interest.
The ID parameter refers to IPP image database key that corresponds to the IMG\_TYPE. 
e.g: either exp\_id, warp\_id, diff\_id, or stack\_id.

\end{itemize}



\subsubsection{Image Selection Filters}\label{ImageFilters}

A request for given celestial coordinates may refer to many images. The scope of the search can be
limited to certain dates with the parameters DATE\_MIN and DATE\_MAX.

The FILTER parameter may be used to select only images with a given filter name. XXX: What is the
namespace for filter names.  For example Megacam has filter names like U.9301 which IPP maps to u.
Which name do we allow? Either?




\subsection{Creating Postage Stamp Requests}
A postage stamp request table may be created one of three ways.
\begin{itemize}
\item \code{pstamprequest} is a simple command line program provided by the IPP
which builds a request file from user supplied command line arguments.

\item The PSS will include a web server providing a user interface for creating
and submitting postage stamp requests.
This sub-project has not been defined yet.
It is expected that there will be web pages 
containing forms with fields similar to \code{pstamprequest}'s command line arguments.
The web server will be submit the request directly to the PSS and optimize
a step or two in the processing.

\item A client may build the request tables inside it's own software following the table format
described above.

\end{itemize}



\subsubsection{pstamprequest}

\code{pstamprequest} is a simple program takes its command line arguments and
creates a postage stamp request table.  Here are a couple of examples showing how it is used.

\begin{verbatim}
    pstamprequest -project gpc1 -byexp o4435g0537o chip xy33 -pixcenter 2700 2700 -pixrange 100 100  \
                  -utag stampreq1 stampreq1.fits 
\end{verbatim}

\begin{verbatim}
    pstamprequest -project gpc1 -bycoord warp 0:41:49.2 41.2 -pixrange 100 100 -utag stampreq2 stampreq2.fits
\end{verbatim}


The following is a brief description of the command line arguments used in these examples.
The complete set of command line arguments is given in Appendix~\ref{pstamprequestArgs}.


\begin{itemize}

\item{\code{-project gpc1 }} tells the PSS to use the gpc1 project's database. 

\item{\code{-byexp  o4435g0537o chip xy33 }} requests a stamp from the chip processed image 
from chip xy33 from the given exposure. XXX: perhaps we should extend this to allow chip+cell to be
specfied and interpret pixel coordinates as cell coordinates?

\item{\code{-pixcenter 2700 2700 -pixrange 100 100 }} 100 pixel x 100 pixel region of interest 
centered at x = 2700, y = 2700 in chip coordinates.

\item{\code{-bycoord warp}} request stamps from warp images containing the given coordinates.

\item{\code{-skycenter }} specifies the Right Ascension and Declination for the center of the
postage stamp ROI.
Note that in this example postage stamps will be extracted from \emph{all} warp images that contain the ROI.

\item stampreq1.fits and stampreq2.fits are the names of the resulting request table files. 


\item{code{-utag }} The user tag for the request. This tag may be used to refer to the postage stamp
request during processing and is used to generate the names for the resulting postage stamp files.
It is the user's responsibility to assign a unique tag.
The web server interface will automatically select a unique user tag.

\end{itemize}

XXX can project be defaulted using the site file?

\subsubsection{Creating Postage Stamp Requests through the Web Interface}
TBD

\section{Queuing Postage Stamp Requests for processing}

A postage stamp request is submitted to the PSS for processing by placing the request file
on a Data Store.  XXX insert Ref to DataStore spec.
The \code{pantasks} task \code{pstampmonitor} monitors a collection of Data Stores for
new postage stamp request files.
For example the PSS is required to handle postage stamp requests from the MOPS science client,
so \code{pstampmonitor} will be configured to periodically check for the presence of new request
files in the MOPS data store.

The following psuedo-code outlines the operation of pstampmonitor.

\begin{verbatim}
    task pstampmonitor {
        for each Data Store being monitored {
            lookup lastFileDownloaded               // TODO: lookup where? DB, a file?

            newFileLlist = results of query to Data Store: list of files newer than lastFileDownloaded

            for each file in newFileList {
                download request file to PSS local storage

                // add entry to pstampRequests table
                INSERT INTO pstampRequests (status, uri) VALUES ('new', file);

                save lastFileDownloaded = file      // in DB?
            }
        }
    }
\end{verbatim}

When \code{pstampmonitor} finds a new request file it queues the request for processing by adding
a row to the database table \code{pstampRequests}. 
The columns in this table are described in Table~\ref{pstampRequestsTab}


\begin{table}[h]
\begin{center}
\caption{pstampRequests Table Contents}\label{pstampRequestsTab}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
\hline
\code{ps_id}      & integer        &  request id key assigned by database \\
\code{status}     & string         &  status of the request new, run, stop \\
\code{uri}        & string         &  uri for the request table's file    \\
\hline
\end{tabular}
\end{center}
\end{table}

As an optimization, the PSS web interface for creating postage stamp requests will bypass the
request monitor and add the request entry to the database directly.



\section{Launching Postage Stamp Extractions}

The \code{pstamprequestparser} is a pantask tasks which watches for new entries in the DB table
\code{pstampRequests}.
When a new entry is found, it runs the postage stamp request parser \code{pstampparse}
which is a C program that reads the request file and turns it into one or more postage
stamp extraction operations.

\begin{verbatim}
    task pstampRequestParser {
        newRequests = SELECT ps_id, uri FROM pstampRequests WHERE status = 'new';
        foreach req in newRequests {
            execute pstampparse req->ps_id req->uri
        }
    }

    program pstampparse(id, uri) {
        request = metadata read by psFitsReadTable(uri);

        if (request->REQ_TYPE == bycoord) {
            imageList = results of query DVO database for images containing the ROI center
            // XXX: can DVO select by filter or do we have to do it here?

            for each image in imageList {
                if (request->FILTER && (request->FILTER != requestimage->filter))
                    continue;
                if (date range specified) {
                    if (image->DATE is not in range)
                        continue;
                }
                if (image needs to be regenerated)
                    and entry to pstampJob with state regen
                else 
                    add entry to pstampJob with state run
            }
        } else if (request->REQ_TYPE == byexp) {
            image = lookup image of IMG_TYPE for the given exposure in IPP database
            if (image needs to be regenerated)
                and entry to pstampJob with state regen
            else 
                add entry to pstampJob with state run
        } else if (request->REQ_TYPE = byid) {
            image is determined by parameters in request
            if (image needs to be regenerated)
                and entry to pstampJob with state regen
            else 
                add entry to pstampJob with state run
        }
    }
\end{verbatim}

If the request specifies the image directly the request parser's job is quite easy. 
The request parser simply insures that the input images exists and
queues an extraction operation for execution by adding one or more rows describing the commands
to run to the database table \code{pstampJob}.

If the input image does not exist, but the system knows how to generated it the status is
set to regen.



\begin{table}[h]
\begin{center}
\caption{pstampJob Table Contents}\label{pstampJobTab}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
\hline
\code{ps_jobid}   & integer        &  job id key assigned by database \\
\code{ps_id}      & integer        &  request which this job is from             \\
\code{state}      & string         &  status of the job regen, run, stop \\
\code{status}     & integer        &  result code 0 for success, other for error \\
\code{command}    & string         &  command string to execute    \\
\code{output uri} & string         &  uri for the output postage stamp request file\\
\hline
\end{tabular}
\end{center}
\end{table}

XXX: is putting the command strings in the database just adding clutter? Is there a better way to do this?
How about if we create a script the appropriate commands and have pstampregen and pstampextract
exectute that.

XXX: maybe fault should be an error code instead of a boolean so we can tell the user what happened.

\section{Image Regeneration}
\code{pstampregen} is a pantasks task that selects entries in psJob with the state \code{regen}.
Using magic to be specified later this task regenerates the image from the raw data. For example,
regenerating a chip processed image by running the detrend operation. 

When the regeneration process is complete, the state of the jobs entry in pstampJob is set to run.

\section{Extracting the Postage Stamps}

\code{pstampextract} is a pantasks task which looks for entries in pstampJob with state run
and executes the extraction tool program \code{ppstamp}. When ppstamp completes the state
of the job is set to stop and a result code is stored in status.

The postage stamp files are placed within the PSS Data Store.

\code{ppstamp} is the program which takes input fits images and creates the actual postage stamps.
Appendix \ref{ppstampArgs} shows the arguments accepted by ppstamp.

The WCS in the stamp is calculated from the WCS in the input image.
The resulting postage stamp image is a SKYCELL file that corresponds to the image's camera. 
XXX: Using a skycell isn't the right thing to do. I'd like to have a simple image file, but the
existing code that writes WCS to the header requires a pmFPA. I need to write a function that
takes the calculated transforms and gets them to the header without an pmFPA.

\section{Creating the request status file}

Another pantasks task is needed to handle request that generate multiple postage stamp
extractions. This task will figure out when all of jobs are done and build a fits table that
gives the results of the postage stamp request.

From the user's perspective the request is done when the results file with the supplied
user tag becomes available on the PSS Data Store.

\section{Issues}

\begin{itemize}

\item What about Magic? Do we need to do magic processing on un-magic'd images before generating
stamps?

\item Do we want a dry-run mode which will show the user what will be done? It will be easy to generate
requests that result in very many postage stamps being created.

\end{itemize}

\pagebreak

\appendix

\section{pstamprequest Command Line Arguments}\label{pstamprequestArgs}
\begin{verbatim}
    The description of pstamprequestArgs will go here.

    It's pretty much what you'd expect. There are arguments that correspond to each of
    the parameters in the request FITS table.

    ROI specification is just like ppstamp below.
\end{verbatim}

\section{ppstamp Command Line Arguments}\label{ppstampArgs}
\begin{verbatim}

  ppstamp -skycenter RA DEC -pixrange dx dy    [-file INPUT.fits] [-list INPUT.txt] OUTPUT
  ppstamp -skycenter RA DEC -arcrange dRA dDEC [-file INPUT.fits] [-list INPUT.txt] OUTPUT
  ppstamp -pixcenter x y    -pixrange dx dy    [-chip chipname] [-file INPUT.fits] [-list INPUT.txt] OUTPUT
  ppstamp -pixcenter x y    -arcrange dRA dDEC [-chip chipname] [-file INPUT.fits] [-list INPUT.txt] OUTPUT

-skycenter is specified as HH:MM:SS DD:MM:SS or decimal degrees
-arcrange values are seconds of arc

Optional arguments:
         -chip chipName    selects chip (only used with -pixcenter)



\end{verbatim}
\end{document}
