Index: /trunk/doc/stamps/.cvsignore =================================================================== --- /trunk/doc/stamps/.cvsignore (revision 15888) +++ /trunk/doc/stamps/.cvsignore (revision 15888) @@ -0,0 +1,1 @@ +*.log *.dvi *.aux *.toc *.log *.out *.lof *.tbr *.tbd *.pdf *.ps *.tmp Index: /trunk/doc/stamps/Makefile =================================================================== --- /trunk/doc/stamps/Makefile (revision 15887) +++ /trunk/doc/stamps/Makefile (revision 15888) @@ -1,3 +1,3 @@ -# $Id: Makefile,v 1.1 2007-09-13 23:51:19 eugene Exp $ +# $Id: Makefile,v 1.2 2007-12-20 23:37:47 bills Exp $ PDFLATEX = env TEXINPUTS=.:LaTeX:$(TEXINPUTS): pdflatex @@ -6,12 +6,18 @@ help: @echo "USAGE: make (target)" - @echo " targets: all" + @echo " targets: all cd design" -cd: stampsCD.pdf -all : cd +cd: stampsCD.pdf + +design: stampsDesign.pdf + +all : cd design CD = stampsCD.tex +DESIGN = stampsDesign.tex stampsCD.pdf: $(SDRS) + +stampsDesign.pdf: $(DESIGN) add-tag: Index: /trunk/doc/stamps/stampsDesign.tex =================================================================== --- /trunk/doc/stamps/stampsDesign.tex (revision 15888) +++ /trunk/doc/stamps/stampsDesign.tex (revision 15888) @@ -0,0 +1,422 @@ +\documentclass[panstarrs,spec]{panstarrs} + +\title{IPP Postage Stamp Server} % put in your title +\subtitle{Design} +\author{Bill Sweeney} +\audience{IPP} +\shorttitle{Postage Stamp Design} +\group{Pan-STARRS IPP} +\project{Pan-STARRS IPP} +\organization{Institute for Astronomy} +\version{DR} +\docnumber{PSDC-430-xxx} + +\newcommand\ugriz{$u^\prime g^\prime r^\prime i^\prime z^\prime$} +\newcommand\grizy{$g r i z y$} + + +\begin{document} + +\maketitle +\tableofcontents + +\pagebreak +\pagenumbering{arabic} + +\section{Overview} + +The basic functionality of the postage stamp server is to accept lists +of postage stamp requests, determine the corresponding images, extract +the needed pixels from the images, and return the postage stamps to +the user. The nominal mode of interaction as an interface to MOPS and +the other Science Clients is for the postage lists to be FITS tables +which are placed as incoming data on the IPP Data Store (or the data +store of the requesting system) and for the results to be placed back +on the IPP Data Store, along with a FITS table description of the +resulting output image files. For the generic user, the requests and +the result may be sent directly via a web server. + +The basic information needed to by the postage stamp server (PSS) to honor a +request consists of: 1) the image of interest and 2) the region on the +image desired (ROI). This information may be represented in a number of ways +which are discussed in section 2.1. % How do I include a reference by tag? + +The implementation of the PSS is a set of \code{pantasks} tasks, standalone programs, +database tables, web server content, and associated programs. + +\section{Postage Stamp Requests} + + +\subsection{Request Table Contents} + +Postage stamp requests are packaged for submission to the PSS as a +FITS table. The contents of the table specify the ROI and parameters that +enable the PSS to choose the images of interest. The columns contained in table are +shown in Table~\ref{RTableDef}. The \code{EXTNAME} for the FITS table is \code{PS1_PS_REQUEST}. + +\begin{table}[h] +\begin{center} +\caption{Postage Stamp Request Table Contents}\label{RTableDef} +\begin{tabular}{lllc} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Values} & {\bf Required } \\ +\hline +\code{VERSION} & string & N.N & yes \\ +\code{PROJECT} & string & any & yes \\ +\code{USER_TAG} & string & must be unique & yes \\ +\code{REQ_TYPE} & string & bycoord byexp byid & yes \\ +\code{IMG_TYPE} & string & raw chip warp diff stack & yes \\ +\code{EXPOSURE} & string & any & byexp \\ +\code{ID} & string & any & byid \\ +\code{CLASS_ID} & string & any & byid raw \& byexp raw \\ +\code{CHIP_ID} & string & any & byid chip \& byexp chip \\ +\\ +\code{FILTER} & string & any & no \\ +\code{DATE_MIN} & string & YYYYMMDD-HH:MM:SS & no \\ +\code{DATE_MAX} & string & YYYYMMDD-HH:MM:SS & no \\ +\\ +\code{RA} & string & sexagesimal or decimal deg & bycoord, null if pixcenter \\ +\code{DEC} & string & segagesimal or decimal deg & bycoord, null if pixcenter \\ +\code{CENTER_X} & S32 & any & null if not pixcenter \\ +\code{CENTER_Y} & S32 & any & null if not pixcenter \\ +\code{D_RA} & F32 & \code{arcsec > 0} & 0 if pixrange \\ +\code{D_DEC} & F32 & \code{arcsec > 0} & 0 if pixrange \\ +\code{WIDTH} & U32 & \code{> 0} & 0 if not pixrange \\ +\code{HEIGHT} & U32 & \code{> 0} & 0 if not pixrange \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Image Types} +All requests need IMG\_TYPE. This parameter selects the stage in the +image processing pipeline from which the the postage stamp will be extracted. +The allowed values for IMG\_TYPE are raw, chip, warp, diff, and stack. + +\subsubsection{Request Types} +The REQ\_TYPE determines how the images of interest are selected. There are three allowed values. + +\begin{itemize} +\item{\textbf{bycoord}} Images are selected by querying the IPP DVO database for images that +contain the celestial coordinates of the ROI center. +The set of images of interest may be limited by specifying certain criteria as described +in Section \ref{ImageFilters}. + +\item{\textbf{byexp}} The image of interest is the image from a particular exposure +with type IMG\_TYPE. +If IMG\_TYPE is raw the CLASS\_ID must be provided. If IMG\_TYPE is chip +and the ROI center is specified in pixel coordinates then CHIP\_ID must be specified. + +\item{\textbf{byid}} The image with given IMG\_TYPE and ID is the image of interest. +The ID parameter refers to IPP image database key that corresponds to the IMG\_TYPE. +e.g: either exp\_id, warp\_id, diff\_id, or stack\_id. + +\end{itemize} + + + +\subsubsection{Image Selection Filters}\label{ImageFilters} + +A request for given celestial coordinates may refer to many images. The scope of the search can be +limited to certain dates with the parameters DATE\_MIN and DATE\_MAX. + +The FILTER parameter may be used to select only images with a given filter name. XXX: What is the +namespace for filter names. For example Megacam has filter names like U.9301 which IPP maps to u. +Which name do we allow? Either? + + + + +\subsection{Creating Postage Stamp Requests} +A postage stamp request table may be created one of three ways. +\begin{itemize} +\item \code{pstamprequest} is a simple command line program provided by the IPP +which builds a request file from user supplied command line arguments. + +\item The PSS will include a web server providing a user interface for creating +and submitting postage stamp requests. +This sub-project has not been defined yet. +It is expected that there will be web pages +containing forms with fields similar to \code{pstamprequest}'s command line arguments. +The web server will be submit the request directly to the PSS and optimize +a step or two in the processing. + +\item A client may build the request tables inside it's own software following the table format +described above. + +\end{itemize} + + + +\subsubsection{pstamprequest} + +\code{pstamprequest} is a simple program takes its command line arguments and +creates a postage stamp request table. Here are a couple of examples showing how it is used. + +\begin{verbatim} + pstamprequest -project gpc1 -byexp o4435g0537o chip xy33 -pixcenter 2700 2700 -pixrange 100 100 \ + -utag stampreq1 stampreq1.fits +\end{verbatim} + +\begin{verbatim} + pstamprequest -project gpc1 -bycoord warp 0:41:49.2 41.2 -pixrange 100 100 -utag stampreq2 stampreq2.fits +\end{verbatim} + + +The following is a brief description of the command line arguments used in these examples. +The complete set of command line arguments is given in Appendix~\ref{pstamprequestArgs}. + + +\begin{itemize} + +\item{\code{-project gpc1 }} tells the PSS to use the gpc1 project's database. + +\item{\code{-byexp o4435g0537o chip xy33 }} requests a stamp from the chip processed image +from chip xy33 from the given exposure. XXX: perhaps we should extend this to allow chip+cell to be +specfied and interpret pixel coordinates as cell coordinates? + +\item{\code{-pixcenter 2700 2700 -pixrange 100 100 }} 100 pixel x 100 pixel region of interest +centered at x = 2700, y = 2700 in chip coordinates. + +\item{\code{-bycoord warp}} request stamps from warp images containing the given coordinates. + +\item{\code{-skycenter }} specifies the Right Ascension and Declination for the center of the +postage stamp ROI. +Note that in this example postage stamps will be extracted from \emph{all} warp images that contain the ROI. + +\item stampreq1.fits and stampreq2.fits are the names of the resulting request table files. + + +\item{code{-utag }} The user tag for the request. This tag may be used to refer to the postage stamp +request during processing and is used to generate the names for the resulting postage stamp files. +It is the user's responsibility to assign a unique tag. +The web server interface will automatically select a unique user tag. + +\end{itemize} + +XXX can project be defaulted using the site file? + +\subsubsection{Creating Postage Stamp Requests through the Web Interface} +TBD + +\section{Queuing Postage Stamp Requests for processing} + +A postage stamp request is submitted to the PSS for processing by placing the request file +on a Data Store. XXX insert Ref to DataStore spec. +The \code{pantasks} task \code{pstampmonitor} monitors a collection of Data Stores for +new postage stamp request files. +For example the PSS is required to handle postage stamp requests from the MOPS science client, +so \code{pstampmonitor} will be configured to periodically check for the presence of new request +files in the MOPS data store. + +The following psuedo-code outlines the operation of pstampmonitor. + +\begin{verbatim} + task pstampmonitor { + for each Data Store being monitored { + lookup lastFileDownloaded // TODO: lookup where? DB, a file? + + newFileLlist = results of query to Data Store: list of files newer than lastFileDownloaded + + for each file in newFileList { + download request file to PSS local storage + + // add entry to pstampRequests table + INSERT INTO pstampRequests (status, uri) VALUES ('new', file); + + save lastFileDownloaded = file // in DB? + } + } + } +\end{verbatim} + +When \code{pstampmonitor} finds a new request file it queues the request for processing by adding +a row to the database table \code{pstampRequests}. +The columns in this table are described in Table~\ref{pstampRequestsTab} + + +\begin{table}[h] +\begin{center} +\caption{pstampRequests Table Contents}\label{pstampRequestsTab} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{ps_id} & integer & request id key assigned by database \\ +\code{status} & string & status of the request new, run, stop \\ +\code{uri} & string & uri for the request table's file \\ +\hline +\end{tabular} +\end{center} +\end{table} + +As an optimization, the PSS web interface for creating postage stamp requests will bypass the +request monitor and add the request entry to the database directly. + + + +\section{Launching Postage Stamp Extractions} + +The \code{pstamprequestparser} is a pantask tasks which watches for new entries in the DB table +\code{pstampRequests}. +When a new entry is found, it runs the postage stamp request parser \code{pstampparse} +which is a C program that reads the request file and turns it into one or more postage +stamp extraction operations. + +\begin{verbatim} + task pstampRequestParser { + newRequests = SELECT ps_id, uri FROM pstampRequests WHERE status = 'new'; + foreach req in newRequests { + execute pstampparse req->ps_id req->uri + } + } + + program pstampparse(id, uri) { + request = metadata read by psFitsReadTable(uri); + + if (request->REQ_TYPE == bycoord) { + imageList = results of query DVO database for images containing the ROI center + // XXX: can DVO select by filter or do we have to do it here? + + for each image in imageList { + if (request->FILTER && (request->FILTER != requestimage->filter)) + continue; + if (date range specified) { + if (image->DATE is not in range) + continue; + } + if (image needs to be regenerated) + and entry to pstampJob with state regen + else + add entry to pstampJob with state run + } + } else if (request->REQ_TYPE == byexp) { + image = lookup image of IMG_TYPE for the given exposure in IPP database + if (image needs to be regenerated) + and entry to pstampJob with state regen + else + add entry to pstampJob with state run + } else if (request->REQ_TYPE = byid) { + image is determined by parameters in request + if (image needs to be regenerated) + and entry to pstampJob with state regen + else + add entry to pstampJob with state run + } + } +\end{verbatim} + +If the request specifies the image directly the request parser's job is quite easy. +The request parser simply insures that the input images exists and +queues an extraction operation for execution by adding one or more rows describing the commands +to run to the database table \code{pstampJob}. + +If the input image does not exist, but the system knows how to generated it the status is +set to regen. + + + +\begin{table}[h] +\begin{center} +\caption{pstampJob Table Contents}\label{pstampJobTab} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{ps_jobid} & integer & job id key assigned by database \\ +\code{ps_id} & integer & request which this job is from \\ +\code{state} & string & status of the job regen, run, stop \\ +\code{status} & integer & result code 0 for success, other for error \\ +\code{command} & string & command string to execute \\ +\code{output uri} & string & uri for the output postage stamp request file\\ +\hline +\end{tabular} +\end{center} +\end{table} + +XXX: is putting the command strings in the database just adding clutter? Is there a better way to do this? +How about if we create a script the appropriate commands and have pstampregen and pstampextract +exectute that. + +XXX: maybe fault should be an error code instead of a boolean so we can tell the user what happened. + +\section{Image Regeneration} +\code{pstampregen} is a pantasks task that selects entries in psJob with the state \code{regen}. +Using magic to be specified later this task regenerates the image from the raw data. For example, +regenerating a chip processed image by running the detrend operation. + +When the regeneration process is complete, the state of the jobs entry in pstampJob is set to run. + +\section{Extracting the Postage Stamps} + +\code{pstampextract} is a pantasks task which looks for entries in pstampJob with state run +and executes the extraction tool program \code{ppstamp}. When ppstamp completes the state +of the job is set to stop and a result code is stored in status. + +The postage stamp files are placed within the PSS Data Store. + +\code{ppstamp} is the program which takes input fits images and creates the actual postage stamps. +Appendix \ref{ppstampArgs} shows the arguments accepted by ppstamp. + +The WCS in the stamp is calculated from the WCS in the input image. +The resulting postage stamp image is a SKYCELL file that corresponds to the image's camera. +XXX: Using a skycell isn't the right thing to do. I'd like to have a simple image file, but the +existing code that writes WCS to the header requires a pmFPA. I need to write a function that +takes the calculated transforms and gets them to the header without an pmFPA. + +\section{Creating the request status file} + +Another pantasks task is needed to handle request that generate multiple postage stamp +extractions. This task will figure out when all of jobs are done and build a fits table that +gives the results of the postage stamp request. + +From the user's perspective the request is done when the results file with the supplied +user tag becomes available on the PSS Data Store. + +\section{Issues} + +\begin{itemize} + +\item What about Magic? Do we need to do magic processing on un-magic'd images before generating +stamps? + +\item Do we want a dry-run mode which will show the user what will be done? It will be easy to generate +requests that result in very many postage stamps being created. + +\end{itemize} + +\pagebreak + +\appendix + +\section{pstamprequest Command Line Arguments}\label{pstamprequestArgs} +\begin{verbatim} + The description of pstamprequestArgs will go here. + + It's pretty much what you'd expect. There are arguments that correspond to each of + the parameters in the request FITS table. + + ROI specification is just like ppstamp below. +\end{verbatim} + +\section{ppstamp Command Line Arguments}\label{ppstampArgs} +\begin{verbatim} + + ppstamp -skycenter RA DEC -pixrange dx dy [-file INPUT.fits] [-list INPUT.txt] OUTPUT + ppstamp -skycenter RA DEC -arcrange dRA dDEC [-file INPUT.fits] [-list INPUT.txt] OUTPUT + ppstamp -pixcenter x y -pixrange dx dy [-chip chipname] [-file INPUT.fits] [-list INPUT.txt] OUTPUT + ppstamp -pixcenter x y -arcrange dRA dDEC [-chip chipname] [-file INPUT.fits] [-list INPUT.txt] OUTPUT + +-skycenter is specified as HH:MM:SS DD:MM:SS or decimal degrees +-arcrange values are seconds of arc + +Optional arguments: + -chip chipName selects chip (only used with -pixcenter) + + + +\end{verbatim} +\end{document}