Index: /trunk/doc/ipptools/ipptools.tex
===================================================================
--- /trunk/doc/ipptools/ipptools.tex	(revision 6014)
+++ /trunk/doc/ipptools/ipptools.tex	(revision 6014)
@@ -0,0 +1,1322 @@
+\documentclass[panstarrs,spec]{panstarrs}
+
+\title{PStask \& Metadata DB interactions} % put in your title
+\subtitle{Job Flow in IPP}
+\author{Eugene Magnier}
+\audience{IPP}
+%\shorttitle{PSPhot}
+\group{Pan-STARRS IPP}
+\project{Pan-STARRS IPP}
+\organization{Institute for Astronomy}
+\version{DR}
+\docnumber{PSDC-xxx-xxx}
+
+\newcommand\ugriz{$u^\prime g^\prime r^\prime i^\prime z^\prime$}
+\newcommand\grizy{$g r i z y$}
+
+\begin{document}
+\maketitle
+
+\tableofcontents
+\pagebreak 
+\pagenumbering{arabic}
+
+\section{Overview}
+
+This document illustrates the interrelationships between IPP tasks and
+the Metadata Database (MDDB).  The Metadata Database is used to store
+the current IPP analysis state, as well as result processing data
+points.  The tasks needed to define each of the IPP analysis stages
+(Phase 1-4, detrend creation, etc) are illustrated in this document
+and the relevant MDDB tables are listed.  The collection of diagrams
+shows the IPP tasks and the Metadata Database tables needed to manage
+the flow of data through the system.
+
+\note{The flat-field correction analysis stage is not yet defined in
+  this document}
+
+\note{AP DB interfaces are mostly ignored}
+
+\section{PStask queues and metadata tables : persistent vs ephemeral state}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pstasks.01.ps}
+\caption{\label{queues} PStask queues and MDDB tables}
+\end{center}
+\end{figure}
+
+There are some subtleties in the interaction between IPP task
+scheduler, PStasks, the metadata tables which store the system state,
+and the jobs which are immediately being performed.  There is a choice
+to be made between rigorously maintaining the system state in the
+Metadata DB at all times or in keeping an intermediate set of state
+tables.  Keeping the exact system state in the Metadata DB tables
+would require more queries to/from the database and may introduce
+additional latencies which are undesirable.  This is because any
+attempt by the scheduler to initiate a new job would require the
+scheduler to mark the corresponding data item in the Metadata DB (the
+item which acts as the trigger) with a `pending' state, and then mark
+it again as `done' when the job actually completes.  This also has the
+drawback that, if the scheduler crashes, some initial process would be
+required on start up to find all Metadata DB items which are in the
+`pending' state (examining all possible items which can be in such a
+state) and reset them to the `new' state.
+
+We implement an alternative in which the scheduler maintains an
+internal, ephemeral stack of the pending jobs, and only updates the
+system state entries in the Metadata DB when jobs are actually
+completed.  In this scenario, as far as the Metadata DB tables are
+concerned, data items transition only between a `new' and a `done'
+state.  Any jobs which are pending when the system crashes or the
+power is lost are simply dropped, and will be automatically
+re-constructed when the system restarts.  In this paradigm, no
+intermediate operation state is saved, and no partially completed job
+can be recovered.  Since the IPP is defined in terms of a fine
+granularity, with jobs lasting no more than 30 - 120 seconds, this
+model will not have a large impact.
+
+Figure~\ref{queues} illustrates this ephemeral vs persistent state
+information and the interation between the metadata tables and
+PStasks.  The left-hand portion of the diagram illustrates the
+recommended interaction between the metadata database tables and the
+scheduler internal queues.  Some table in the metadata database
+defines a list of data items which are to be processed by some
+analysis job.  The scheduler uses a two-step approach to define the
+analysis jobs based on this list.  First, one scheduler task queries
+the MDDB for a list of pending items, adds the returned items to an
+internal scheduler queue.  The process of adding the elements to the
+queue is defined so that only unique items are added: already existing
+items are skipped.  The entries in the queue consist of the data items
+of interest and an internal temporary state.  At first, this would be
+`pending'.  A second scheduler tasks pops `pending' entries one-by-one
+from this internal queue, submits a job based on the entry, and sets
+the temporary state in the internal queue to `running'.  The internal
+state is needed to prevent the scheduler from re-submitting a job for
+the same data item before the first job is done or assessed.  Since
+the job make take an arbitrary amount of time, the scheduler requires
+a mechanism to remember which data items it has already submitted.
+When the job eventually completes, the metadata database table is
+updated noting the completion.  This may be done either by the job
+itself or by the scheduler.  In addition, the state of the entry in
+the queue can be set to either `done' or the entry can be simply
+removed from the queue.
+
+The purpose of this interaction is to maintain the temporary state
+information within non-persistent elements of the scheduler rather
+than using the metadata database tables to store this information.
+This concept has two advantages.  First, the scheduler internal queues
+are in memory and relatively small, thus interfacing with them is
+quite fast for the scheduler -- this should reduce the system latency.
+Second, by keeping this information non-persistent, the system
+responds correctly to stopping and restarting the scheduler.  Any jobs
+which have not been completed will not be marked in the database, and
+will be restarted naturally by the scheduler.  The alternative, of
+writing a temporary state marker in the database would require a
+restarted scheduler to initially clean all database tables of these
+temporary state markers.
+
+The right-hand portion of the diagram illustrates this process using
+the process of copying the images from the summit as an example.  The
+metadata database table of interest in this case is the list of
+pending images, with entries supplied by a job which queries the
+summit data systems.  The job which is actually performed is a remote
+copy of the image file from the location specified by the summit data
+system to the appropriate location within the IPP Image Server
+(Nebulous).  (As an alternative to the above, the `pending images'
+table may be part of the summit database system, and the `get images'
+command may query the summit directly.  In this scenario, the `copy
+image' command reports to the summit data system that an individual
+image file has been copied.)
+
+In the rest of this document, the use of PStask internal queues to
+manage the temporary data states is glossed over and assumed part of
+the tasks defined in the process.
+
+\section{Summit Copy Tasks}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pstasks.02.ps}
+\caption{\label{pcopy} Summit Copy Tasks}
+\end{center}
+\end{figure}
+
+Figure~\ref{pcopy} illustrates the MDDB tables used to copy data
+(images and metadata tables) from the summit.  The left-hand portion
+of the diagram shows the tables involved in copying images from the
+summit system.  The table of pending image files lists the URLs of the
+individual image files available for transfer, along with their
+associated exposure ID and the camera which generated the image.  Two
+other entries assist in interpreting the file: the class and the class
+ID.  The final entry in this table is the current copy state of the
+file, can have the value of `ready' or `copied'.
+
+The class defines the data grouping represented by this image file and
+may have values of: FPA, Chip, Cell.  This value indicates that the
+provided image file represents the specified portion of the camera
+FPA.  If the value is FPA, the file represents data from a complete
+FPA, though the file may contain pixel data in multiple extensions or
+other groupings to be identified later.  If the value is chip, the
+file contains only data for a single chip, presumably of multiple
+chips available, and equivalently for Cell.  Further discussion of the
+FPA image hierarchy is given in the IPP documents (eg, Modules SDRS).
+The class ID gives the identifier used to name the class level
+corresponding to this file.  This value is necessary to make decisions
+on how to copy the data based on the chip / cell before the data is
+available to IPP components.  Table~\ref{classes} lists likely values
+for the class and class ID for some common cameras.  The system
+described is sufficiently flexible to allow us to transfer the GPC
+images by cell if we eventually decide that is more efficient.
+
+The copy process copies the file from the given URL to the appropriate
+IPP node and adds an entry to the table of new image files, consisting
+of the same information as the pending image file table, though with a
+new value for the URL.  This URL may be an explicit filename, a
+reference to an entry in the image server, or a web address, or
+located on the image server (marked with file:, neb:, and http:,
+respectively).  (TBD: other possible file storage types?  perhaps the
+path could be abstracted without going to the level of the image
+server?  eg: ref:DIR0001/file0001.fits might be in a directory which
+is defined in a table of directories.) After an image file is
+successfully copied, the corresponding state in the `pending chip'
+table is updated from `ready' to `copied'.
+
+\begin{table}
+\begin{center}
+\caption{Camera and Data Classes\label{classes}}
+\begin{tabular}{llll}
+\hline
+\hline
+camera   & class  & classID \\
+\hline
+GPC	 & chip   & chip02 \\
+skyprobe & fpa 	  & sp01 \\
+Megacam  & fpa	  & MegacamSpliced \\
+Suprime	 & chip	  & chip0 \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+The right hand portion of this diagram illustrates the process of
+copying a metadata table.  The table of pending tables lists the URLs
+for the tables which are ready, a unique table ID for each table, and
+the table type.  The copy function copies the listed table and uploads
+the data to the IPP version of the same metadata database.  Two
+examples of metadata tables needed by the IPP for the basic image
+processing system are illustrated: the table of new exposures and the
+table of pending matches.  The first lists the exposures which are
+avilable from the summit system, and all represent entries which are
+available from the Image server.  the second represents the matches
+between exposure IDs and chips
+
+\section{Phase 0}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pstasks.03.ps}
+\caption{\label{phase0} Phase 0 Tasks}
+\end{center}
+\end{figure}
+
+Figure~\ref{phase0} illustrates phase 0, in which the image files are
+categorised, examined for summary information and basic statistics,
+and moved to the tables used to trigger further analysis.  The process
+first examines the `new image files' table.  It selects images from
+this table which have not yet been examined (state is `new').  The
+file header is examined and relevant metadata is extracted (eg, RA,
+DEC, times, and so forth to be defined later).  The process may also
+select a portion of the image pixel data to determine a rough bias and
+background level.  These statistics, whether derived from the header
+or the pixel values, are placed along with image summary information
+in the `raw image files' table, and the state field of the `new image
+files' table is set to `ready'.
+
+The process is also responsible for moving the exposures to the tables
+used for triggering the analysis process.  If the image class is FPA,
+the image can be advanced without waiting for any other image files.
+If the class is Chip or Cell, the process must also examine the `new
+exposure' table for this exposure ID.  The number of class files
+available for this exposure is listed in this table.  The process must
+the select all image files matching the exposure ID with state of
+`ready' and compare the number avalable to the number expected.  If
+the two match, then a new exposure is ready.  Based on the image type
+(from the most recently examined image file header or new exp table?),
+the exposure is added to the `raw exposure' table for images of that
+type.  The allowed types are `detrend', (all bias, dark, flat images),
+`object', `focus'(??), etc.  (** The different tables represent
+different analysis modes.  This process also adds an entry to the exp
+ID / image file match **).  This process also adds all science
+(OBJECT) exposures to the P1 exposure table (for mosaic data) or the
+P2 chip table (for single detector data).  These tables are used to
+trigger the Phase 1 and Phase 2 analysis stages.
+
+\section{Phase 1}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pstasks.04.ps}
+\caption{\label{phase1} Phase 1 Tasks}
+\end{center}
+\end{figure}
+
+Figure~\ref{phase1} shows the tables involved in running the P1
+analysis stage.  There are paths for exposures to enter the analysis
+automatically from the P0 analysis (arrow on left) or to be added
+manually based on a selection from the raw exposure table.  Exposures
+to be analysed by Phase 1 are added to the P1 exposure table with the
+state `new'.  Exposures may be added multiple times for processing and
+reprocessing. The P1 exp table keeps a record of the old attempts for
+debugging and analysis.  Each time an exposure is added to the P1 exp
+table, it is given a new, unique version number, allowing the system
+as a whole to track different analysis attempts.  This method is used
+in all of the image analysis stages (and extrapolated to iterations in
+the detrend analysis steps below).  The top portion of the diagram
+represents the user-space tool which may be used to re-submit an
+exposure or a group of exposures, potentially selected on the basis of
+a query from the raw SCIENCE exposure table.
+
+The P1 exposure table is examined to select the new exposures, these
+are then used to generate the P1 analysis jobs.  Within the analysis
+job, the chips (image files) associated with the exposure are select
+from the raw image file table.  The analysis examines the contents of
+these files, either extract the guide star information from the image
+files (GS table extension) or searches for and centroids the pixels on
+appropriate bright stars.  The analysis results in astrometric
+calibration terms which are written to the astrometric calibration
+file for this exposure.  The location of the astrometric calibration
+file and the statistics of the measurement are written back to the P1
+exposure table.  The images associated with exposures which are
+successfully processed by P1 are then added to the P2 image table,
+which is used to trigger the Phase 2 analysis.
+
+\section{Phase 2}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pstasks.05.ps}
+\caption{\label{phase2} Phase 2 Tasks}
+\end{center}
+\end{figure}
+
+Figure~\ref{phase2} shows the tables involved in running the P2
+analysis stage.  There are paths for images to enter the analysis
+automatically from the P1 analysis (arrow on left) or to be added
+manually based on a selection from the raw exposure and raw image file
+tables.  Images to be analysed by Phase 2 are added to the P2 image
+table with the state `new'.  When images are added to this table, a
+single entry is also added to the P2 exposure table listing the P1 and
+P2 versions for this exposure.  These version numbers must be integers
+starting with 1.  If this image did not have a P1 analysis, the P1
+version is set to 0.  Exposures may be added multiple times for
+processing and reprocessing. The P2 image table keeps a record of the
+old attempts for debugging and analysis.  As with P1, each time a
+collection of associated images from an exposure is added to the P2
+image table, it is given a new, unique version number, allowing the
+system as a whole to track different analysis attempts.  The top
+portion of the diagram represents the user-space tool which may be
+used to re-submit the images for an exposure or a group of exposures,
+potentially selected on the basis of a query from the raw SCIENCE
+exposure and raw image file tables.
+
+The P2 image table is examined to select the `new' images.  These
+images are used to generate P2 analysis jobs.  The P2 analysis uses
+the input url to find and load the image file.  The url may be a file
+on disk, an entry in the image server, Nebulous, etc.  The master
+detrend images matching the specific science image and the conditions
+are selected by examining the table of master detrend frames.  The
+specific detrend image files are selected by using the master detrend
+ID to select the matching the entries in the table of master detrend
+files.  After the analysis, the output image, mask, and FITS table of
+objects, including the astrometry calibration, are written to the P2
+image table, along with summary statistics from the P2 analysis.  The
+state is also updated (to `done').  
+
+Whenever the exposure is completed, the value of Ndone in the P2
+exposure table is incremented.  If all P2 images matching the P2
+exposure version have been completed, the value of Ndone will match
+Nclass, and in this case, the process adds an entry to the P3 exposure
+table.
+
+\section{Phase 3}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pstasks.06.ps}
+\caption{\label{phase3} Phase 3 Tasks}
+\end{center}
+\end{figure}
+
+Figure~\ref{phase3} illustrates the tables involved in the Phase 3
+analysis.  The P3 exposure table lists the exposure ID, the P3
+analysis version, the P2 analysis version to be used as input to this
+P3 analysis, and the recipe to be used.  The P2 exposure and image
+tables are used, in conjunction with the P2 version information, to
+select the P2 output measured objects and the astrometric calibrations
+from P2 and P1.  These measured objects are matched with the reference
+catalog objects, and calibrated astrometry and photometry is produced
+for the full exposure.  The location of the resulting astometry
+calibration table is stored back in the P3 exposure table.  If the
+recipe file specifies, the 2-D photometric and background / fringe
+corrections may also be performed at this stage.  Since these analyses
+require reference data, the recipe may also be used to skip these
+analysis if such reference data is unavailable or unreliable.  At the
+end of Phase 3, the objects from the exposure are inserted into the
+photometry database (this is not shown).
+
+The astrometric calibration portion of Phase 3 is principally needed
+for a mosaic camera.  For single-chip cameras, Phase 3 may be used to
+perform the photometric calibration and simply pass the astrometric
+results along to the output file to be listed in the P3 exposure
+table.  In this way, later stages of the analysis (ie, Phase 4) can
+use the P3 exposure table as input for all cameras, even if all the
+funcionality of Phase 3 is not necessary for that camera.  This would
+be the case for the skyprobe camera, for example.
+
+\section{Phase 4}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pstasks.07.ps}
+\caption{\label{phase4} Phase 4 Tasks}
+\end{center}
+\end{figure}
+
+At the end of Phase 3, the images are ready for Phase 4.  The Phase 3
+diagram shows the output line adding the exposures to be processed by
+Phase 4 to a Phase 4 table.  However, this line is just for
+illustration purposes.  The rules for initiating a Phase 4 run are
+somewhat more complicated than those for running Phases 1-3.  Groups
+of exposures which have an appropriate overlap should be chosen for
+the Phase 4 analysis.  In the steady-state period of PS-4, it may be
+straightforward to choose the exposure groups: they would simply be
+the exposures obtained nearly simultaneously by the four separate
+cameras.  The circumstance for PS-1 will be much more complicated (and
+even PS-4 will probably be more complex than it seems at first
+glance).  For example, in PS-1, we will not have a static sky for most
+of the AP Survery.  In this circumstance, we cannot run P4, at least
+until after the complete AP Reference catalog is built, and
+potentially all exposures re-run through Phase 3.  It may be useful
+for the AP Survey data to split the Phase 4 analysis into two stages:
+image combination and image differencing.  It may even be the case
+that only the combination portion of Phase 4 is performed on the AP
+Survey data.
+
+More generally, the image groups selected for Phase 4 analysis may be
+chosen on the basis of a query of the AP Database (DVO) with some
+rules.  
+
+\note{Phase 4 run can be defined by selecting an observation group, a
+  set of exposures, or a set of rules related to a spatial region (eg,
+  region, time range, and filter}.
+
+\note{Phase 4 discussion (and diagram) needs more work}
+
+\section{Analysis Version and Recipes}
+
+Note that each of the stages P1-P4 refer to the processing version
+from the previous stage.  This allows the processing stage to request
+the correct version of the results from the previous stage, and makes
+it possible to run and re-run the analysis at any stage without
+deleting the earlier results.  As different analysis attempts are
+performed for a given image, the versions branch out.
+
+Also note that at every stage, the entries include a recipe
+identifier.  This is used to select the analysis recipe which should
+be used for this version.  By default, the recipe should be set to the
+current best recipe (use a default name for this?).  This feature
+allows the user to run test analyses with variations on the recipe
+without altering the analysis system.  For example, it is possible to
+use a different flat-field set by specifying alternate rules for the
+flat-field selection in a recipe file.  If it is necessary to run the
+P1-P3 analysis with the raw master flats, for example, the user simply
+defines that selection in the recipe file and submits the images of
+interest to P1 (or P2, etc), with the corresponding entry for the
+recipe.
+
+The recipe file may also be used to specify alternative analysis paths
+and desitinations.  For example, it is not necessary that all analysis
+stops with P4: the recipe file may be used to halt the analysis at P2
+or P3.  In addition, the recipe file may be used to specify an
+alternative destination for the output results.  For example, to
+generate the photometric flat-field correction frame from a collection
+of dithered images, the user may not want the photometry results in
+the main DVO database.  By using the recipe to set an alternative DVO
+database target, and by specifying the use of the raw master flat
+rather than the corrected one, the analysis of the dithered images is
+kept isolated from the other photometry database results.  The
+resulting photometry may be used to construct the new, corrected
+flat-field images, and the processing of the same images using the new
+flat-field images may be sent to the master DVO database.  
+
+\section{Basic Detrend Creation}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pstasks.08.ps}
+\caption{\label{detrend} Detrend Creation Tasks}
+\end{center}
+\end{figure}
+
+Figure~\ref{detrend} illustrates the tables needed for the generic
+detrend construction process, using the flat-field construction as an
+example.  This diagram is somewhat more complex than the preceeding
+versions.  In this diagram, both single jobs and multiple jobs are
+represented by the process elements (the blue ellipses).  In some
+cases, more that one task will be needed to perform the function
+illustrated by a single process task.  The complexity of this diagram
+is enhanced by the need for multiple iterations and both single chip
+and full mosaic processing.  At the moment, the distinction between
+mosaic and single chip cameras is not specifically discussed.
+Finally, the triggers which initiate a specific detrend analysis are
+glossed over.
+
+The detrend analysis is initiated by choosing a type of detrend image
+to be constructed and by specifying the criteria which will be used to
+select the input raw detrend frames for the construction.  For
+example, these criteria could specify that all twilight flat images
+over a certain period of days, perhaps with restrictions on the flux
+levels or the time-from-sunset of the images.  The detrend analysis
+run is given an ID (det ID) which will also be used to identify the
+resulting master detrend frame.  
+
+Given the definition of a master detrend run, the input exposures are
+selected from the raw detrend exposure table, and written to the input
+detrend exposure table.  In the next step, the corresponding image
+files are selected from the table of raw image files.  Since there
+will be a different set of input raw images for each attempt at
+creating a master detrend image, and since any given attempt may use
+some of the same input images as any other attempt, a separate table
+of input raw images is constructed.  
+
+Each of the input raw images may be pre-processed before it may be
+used to construct the detrend frame.  For example, the input
+flat-field images should (probably) be dark- and bias-corrected before
+they are stacked.  The information about these input processed images
+is written to the input images table.  If no processing is needed,
+this step simply copies the appropriate information to the table, and
+points back to the raw image, rather than a processed version.  
+
+The input processed images are combined (stacked) to create a master
+detrend image for the particular data element defined by the image
+class (chip/cell/fpa).  At this stage, not all input images should
+necessarily be included in the stack.  If residual statistics have
+been measured for the input images (say, using a prior stack), then
+some of the input image may be excluded.  The table of residual images
+is used to guide this process.  The information describing the
+resulting master image is written to the master images table.  
+
+The statistics of the master detrend images must examined so that any
+necessary renormalizations may be performed.  For example, after
+stacking the individual flat images, the resulting stacks must be
+renomalized to account for the different ranges of input image fluxes.
+This analysis is least-squares solution in which an appropriate scale
+is determined for each input exposure and a separate gain is
+determined for each of the chips or cells in the camera.  This
+analysis can only performed after all image stacks (ie, for all chips)
+have been constructed.  The resulting information is written to the
+table of master detrend frames.  
+
+Once the master detrend is constructed, the master detrend images may
+be used to construct residual images for each of the input images.
+These residual statistics, as well as the locations of the residual
+images and other related data products (jpeg thumbnails?) are written
+to the residual image table.  Note the red arrow which by-passes the
+stack construction and merge steps and skips directly to the residual
+analysis.  In some cases, it may be useful to have the input images
+confronted with an existing detrend image, and the resulting residual
+values used to guide the rest of the process.  For example, in the
+flat-field analysis, applying an earlier flat can result in a very
+good first-pass rejection of poor input images.  The logic to make
+this leap must be part of the scheduler, since each of the individual
+blocks represent complete processing jobs.
+
+Finally, the residual statistics from the complete mosaic (all input
+images, all chips) are used to assess the quality of the newly
+constructed master detrend image, and to potentially modify the
+selection of input images.  This latter process is performed by
+marking the state of the residual images from this iteration.  The
+stacking process always examines the state information for the
+residual images from the previous iteration, if it exists, when
+constructing the master stack.  Once a master detrend frame has been
+judged of high enough quality, the state of the entry for the frame in
+the master detrend frames table is set to an appropriate value to tell
+the other routines that this image should be used as a master detrend.
+The exact choice of which master detrend frame is used for a given
+science image depends on the recipe along with information such as the
+time period used or the conditions used.
+
+Note that, although this discussion focuses on the construction of
+flat-field images, the same structure should be capable of
+constructing the biases, dark, fringes, etc.  In some cases, as noted
+above, the `process' stage is a null operation.
+
+\pagebreak
+
+\appendix
+\section{IPP top-level commands}
+
+In this section, we define all the necessary top-level commands needed
+by the IPP to implement the data flow discussed in the previous
+sections.  These commands are user commands, and are visible in the
+UNIX command space.  The commands are discussed in the context of both
+the automatic processing and processing of individual entries.  It is
+an important goal that the user commands should provide a
+simple-enough interface that they can be used without needing to have
+the entire infrastructure of the IPP in place to function.  In
+practice, this means that data items which can be acquired from the
+Metadata DB tables can also be provided on the command line and/or in
+the recipe file.  In some cases, user commands are provided to allow a
+manual intervention beyond the automatic processing loops.  This is
+particularly true of the \code{submit.Px} type of commands.  
+
+\note{the command names are for illustration purposes only.  any
+  suggestions for better / fancier names are welcome...}
+
+\begin{verbatim}
+
+copy.image
+  input: url, exp ID, camera, class, class ID
+
+  This program copies the image file from the given URL, updates the
+  new image file table with the descriptive metadata (exp ID, camera,
+  class, class ID), and notifies the external subsystem that the copy
+  has succeeded.  The destination host for the image file is
+  determined by selecting the class ID from the host-for-class table.
+
+copy.table
+  input: url, table ID, table type
+
+  This program copies the metadata table file from the given URL,
+  determines which IPP metadata table it corresponds to, adds the
+  table rows to that metadata table, and notifies the external
+  subsystem that the table copy has succeeded.
+
+classify.image
+  input: (url) or (exp ID + class ID)
+
+  This program examines the header of the specified image file,
+  determines the image type (detrend, science, etc), extracts a
+  specific set of information from the header, and adds a new entry to
+  the raw image files table.  It also checks the new exposure table
+  for the corresponding exp ID, counts the files with the same exp ID
+  that are in the raw image files table, and if they match, adds an
+  entry to the raw exposure table for the appropriate exposure type.
+  Depending on the camera format (mosaic / single chip), the process
+  also adds an entry to either the P1 exposure table or the P2 image table.
+  
+Phase.1
+  input: exp ID
+
+  This program determines the chips which correspond the exposure,
+  loads the guide stars (if they exist) or reads the pixels around
+  bright stars (if the guide stars do not exit.  It determines the
+  centroids for these stars, reads in the astrometric reference stars
+  in the field, matches observed stars to reference stars, determines
+  an astrometric solution, using the default telescope / camera model
+  as the starting point, and writes out the result to the P1 output
+  file, along with the FITS table of observed star measurements (X, Y,
+  inst mag).  It writes the summary statistics back to the P1 exposure
+  table, an new entry in the P2 exposure table, and entries for each
+  of the image files in the P2 images table.
+
+Submit.P2
+  input: exposure selection criteria
+
+  This program uses the provided selection criteria (eg, exp ID, time
+  range + filter, etc) and selects the corresponding exposures and
+  image files.  If the output is specified to the database, it creates
+  new entries in the P2 images and P2 exposure tables, incrementing
+  the version fields if these exposure already exist.  If the output
+  is specified to stdout / file, it writes the corresponding
+  information in a human (and/or machine) readable format.  Note: it
+  is not necessary that a P2 exposure defined by this tool include all
+  possible or available image files.  Note also: this program must
+  validate that the selected exposures have completed Phase 1, or are
+  not required to complete Phase 1.
+
+Phase.2
+  input: (url) or (exp ID + class ID)
+
+  This program reads in the selected image file, determines the
+  matching detrend images, detrends the image data, performs object
+  detection and analysis, loads the corresponding
+  astrometric/photometric reference data, determines the astrometric
+  and photometric calibrations.  It writes out the detrended image,
+  the corresponding mask, and a FITS table of the measured objects,
+  with the astrometric parameters in the FITS table header.  It
+  updates the P2 images table with the statistics of the analysis, and
+  updates the P2 exposure table status.  If this analysis is the last
+  of the Nclass P2 image analyses to be done on this P2 exp ID and
+  version, then the program makes a new entry in the P3 exposure
+  table.  
+
+Submit.P3
+  input: exposure selection criteria
+
+  This program uses the provided selection criteria (eg, exp ID, time
+  range + filter, etc) and selects the corresponding exposures which
+  have succeeded in Phase 2 (a specific P2 version may be specified).
+  If the output is specified to the database, it creates new entries
+  in the P3 exposure table, incrementing the version field if this
+  exposure already exists.  If the output is specified to stdout /
+  file, it writes the corresponding information in a human (and/or
+  machine) readable format.
+
+Phase.3
+  input: (exp ID) or (list of P2 object files)
+
+  This program finds the collection of P2 object files for the
+  specified exposure, reads in the object data and astrometric
+  calibration terms, loads the corresponding astrometric/photometric
+  reference data, determines the improved astrometric and photometric
+  calibrations.  It writes out a single FITS table of the measured
+  objects, with the astrometric parameters in the FITS table header,
+  and astrometric calibration data in a new astrometry file for this
+  exposure.  It updates the P3 exposure table with the statistics of
+  the analysis
+
+Submit.P4
+  input: selection criteria
+
+  This program uses the provided selection criteria (eg, exp ID list,
+  observation group ID, sky region + time range + filter, etc) and
+  selects the corresponding exposures which have succeeded in Phase 3
+  (or Phase 2 if Phase 3 is not needed for this camera / recipe).  It
+  determines the matching sky cells which are overlapped by the
+  selected exposures.  If the output is specified to the database, it
+  creates new entries in the P4 run table and the P4 run input table.
+  If the output is specified to stdout / file, it writes the
+  corresponding information in a human (and/or machine) readable
+  format.
+
+Phase.4
+  input: (run ID) or (sky cell + list of P2 / P3 image files)
+
+  This program uses the skycell and the list of input exposures to
+  find the collection of image files which overlap the given skycell.
+  It then reads the image pixels, warps them to match the skycell
+  geometry, and stacks the image pixels.  It reads the pixels from the
+  skycell and performs the image difference analysis, it photometers
+  the difference image, cleans the input summed image on the basis of
+  the detections, and photometers the input summed image.  It writes
+  out the statistics of the analysis to the P4 run table, the P4 sum
+  table, and the P4 delta table.  If requested, it improves the
+  signal-to-noise in the skycell image, and updates the skycell table.
+
+Detrend.init
+  input: (detrend creation criteria)
+
+  This program adds a new entry to the master detrend run table based
+  on the given criteria.  The criteria may define a time range for the
+  input images, a detrend type, flux ranges, a filter, an observation
+  group ID.  The new master detrend run is created with the iteration
+  set to 00 and the state set to `new'.  
+
+Detrend.get.input.list
+  input: det ID, version
+
+  This program uses the selection criteria defined by the master
+  detrend run (eg, exp ID list, observation group ID, time range, type
+  + filter, etc) to select the corresponding detrend exposures.  It
+  creates a new entry in the master detrend frames table, with state
+  `new' and iteration of 00.  It also writes the exposures to the
+  input detrend exposure table.
+
+Detrend.get.images
+  input: exp ID
+
+  This program identifies the detrend images which correspond to the
+  selected detrend exposure ID and adds an entries for the image files
+  in the input detrend images table, with state set to `raw'.
+
+Detrend.process
+  input: url / detrend type
+
+  This program performs the pre-processing needed by the selected
+  detrend image in order to prepare it for combination with other
+  detrend images.  When completed, the entry for this file in the
+  input detrend images table is update to `proc'.  The processing may
+  be as simple as a null operation (eg, for a bias) or as complex as
+  bias, dark, flat-field, renormalize (eg, for a fringe).
+
+Detrend.stack
+  input: det ID, version, iteration, class ID?
+
+  This program performs the image stacking for a single image class
+  ID.  The stacking process may depend on the detrend type (different
+  for bias from flat from fringe).  The resulting statitics are
+  written to the master detrend image table.  If this is the last of
+  Nclass entries for the given master detrend frame, then the master
+  detrend frames state is updated.
+
+Detrend.merge
+  input: det ID, version, iteration
+
+  This program examines the results of the Nclass master detrend
+  images and performs any necessary re-normalizations.  It also
+  examines the statistics of the individual stacks and summarizes them
+  in the master detrend frame table..
+
+Detrend.residuals
+  input:  det ID, version, iteration, class ID, url
+
+  This program performs the detrending on the given processed input
+  image using the corresponding detrend frame.  The residual image is
+  saved, and an entry is written to the residual images table giving
+  the image location and the residual statistics for this image.
+
+Detrend.assess
+  input: det ID, version, iteration
+
+  This program examines the collection of residual image statistics
+  and creates ensemble statistics for each residual exposure (NOTE: do
+  we need a residual exposure table??).  It examines the ensemble of
+  exposure statistics and determines if 1) the complete stack meets
+  the success criteria and 2) which residual exposure do / do not meet
+  the success criteria.  It updates the master detrend run table, the
+  master detrend frame table, and the residual image / exposure table
+  to note images which should be included / excluded in a future
+  iteration.  
+\end{verbatim}
+
+
+\section{Metadata Database Tables used for IPP Job Flow}
+\label{sec:MetadataTableContents}
+
+The tables presented below define in greater detail the contents of
+the Metadata tables shown in the figures above.  In some cases, the
+quantities (eg, the analysis result statistics) are illustrative, not
+definitive.  In certain tables, data is provided which is redundant
+(non-normal) for ease of use.  In some cases, we may decide not to
+keep this redundant information.  In cases where the choice to
+replicate the redundant information is uncertain, the field names are
+written in {\it italics}.
+
+\begin{table}[bh]
+\begin{center}
+\caption{Pending Image Files\label{tab:PendingImageFiles}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+URL               & string          & file location	       & http://data/file001.fits  \\
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
+state		  & string	    & state of transfer?       & ready / copied		   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{New Image Files\label{tab:NewImageFiles}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+URL               & string          & file location	       & neb://file001.fits        \\
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+camera		  & string	    & camera name	       & MegaPrime / GPC           \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Host for Class\label{tab:HostForClass}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+camera		  & string	    & camera name	       & MegaPrime / GPC           \\
+class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
+hostname	  & string	    & preferred host computer  & po01 / alala.ifa.hawaii.edu \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Raw Image Files\label{tab:RawImageFiles}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+URL               & string          & file location	       & neb://file001.fits        \\
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+camera		  & string	    & camera name	       & MegaPrime / GPC           \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
+type		  & string	    & exposure type	       & bias / flat / science	   \\
+Nreadout	  & int		    & number of readouts       & 1 / 5 / 100		   \\
+Treadout	  & float	    & readout exposure time    & 1.0 (sec)		   \\
+ccdtemp		  & float	    & detector temperature     & 90 (K)			   \\
+\hline
+background	  & float	    & data area median	       & 5.0 (sec)		   \\
+FHWM		  & float	    & average image quality    & 2.5 (arcsec)		   \\
+\hline
+\multicolumn{4}{l}{note: stats below the line are measured, perhaps they go elsewhere?}	   \\
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Pending Metadata Tables\label{tab:PendingTables}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+URL               & string          & file location	       & neb://file001.fits        \\
+table ID	  & string	    & unique table identifier  & table-654321 		   \\
+table type	  & string	    & table content type       & temps / skyprobe data     \\
+state    	  & string	    & copied?                  & ready / copied		   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Copied Metadata Tables\label{tab:CopiedTables}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+URL               & string          & file location	       & neb://file001.fits        \\
+table ID	  & string	    & unique table identifier  & table-654321 		   \\
+table type	  & string	    & table content type       & temps / skyprobe data     \\
+state    	  & string	    & copied?                  & ready / copied		   \\
+time		  & int		    & table arrival time       & 13243413 (s)		   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{New Exposures\label{tab:NewExp}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+camera		  & string	    & camera name	       & MegaPrime / GPC           \\
+telescope	  & string	    & telescope name	       & CFHT / PS-1 / SP-1        \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+Nfiles		  & int		    & number image files       & 64			   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Raw SCIENCE Exposure\label{tab:RawScienceExp}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+camera		  & string	    & camera name	       & MegaPrime / GPC           \\
+telescope	  & string	    & telescope name	       & CFHT / PS-1 / SP-1        \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+Nfiles		  & int		    & number of image files    & 1 / 64			   \\
+\hline
+exptime		  & float	    & exposure time	       & 10.0 (seconds)		   \\
+filter.ID	  & string	    & filter ID (glass)	       & g.PS1.01		   \\
+filter.Band	  & string	    & filter bandpass name     & g			   \\
+airmass		  & float	    & sec(zenith angle)	       & 1.35			   \\
+RA		  & float	    & boresite RA	       & 125.01 (degrees)	   \\
+DEC		  & float	    & boresite DEC	       & 35.01 (degrees)	   \\
+PA		  & float	    & FPA position angle       & 10.1 (degrees)		   \\
+obstime.sec	  & float	    & observation start	       & 123423422 (TAI)	   \\
+obstime.nsec	  & float	    & observation start (nsec) & 1234			   \\
+telfocus	  & float	    & focus distance	       & 1.50 (mm)		   \\
+FPAtemp		  & float	    & FPA temperature	       & 90 (K)			   \\
+dometemp	  & float	    & dome air temperature     & 290 (K)		   \\
+airtemp		  & float	    & outside air temperature  & 285 (K)		   \\
+mirrortemp	  & float	    & primary mirror temp      & 286 (K)		   \\
+teltemp		  & float	    & telescope structure temp & 283 (K)		   \\
+group ID	  & string	    & observation group name   & target-seq-01		   \\
+group seq	  & int		    & sequence number in group & 3			   \\
+\hline
+background	  & float	    & average of file skies    & 5.0			   \\
+dbackground	  & float	    & stdev of file skies      & 1.0			   \\
+FHWM		  & float	    & \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Raw Detrend Exposures\label{tab:RawDetrendExp}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+camera		  & string	    & camera name	       & MegaPrime / GPC           \\
+telescope	  & string	    & telescope name	       & CFHT / PS-1 / SP-1        \\
+type		  & string	    & exposure type	       & bias / domeflat / dark    \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+Nfiles		  & int		    & number of image files    & 1 / 64			   \\
+\hline
+exptime		  & float	    & exposure time	       & 10.0 (seconds)		   \\
+filter.ID	  & string	    & filter ID (glass)	       & g.PS1.01		   \\
+filter.Band	  & string	    & filter bandpass name     & g			   \\
+Alt		  & float	    & boresite Alt	       & 125.01 (degrees)	   \\
+Az		  & float	    & boresite Az	       & 35.01 (degrees)	   \\
+PA		  & float	    & FPA position angle       & 10.1 (degrees)		   \\
+obstime.sec	  & float	    & observation start	       & 123423422 (TAI)	   \\
+obstime.nsec	  & float	    & observation start (nsec) & 1234			   \\
+telfocus	  & float	    & focus distance	       & 1.50 (mm)		   \\
+FPAtemp		  & float	    & FPA temperature	       & 90 (K)			   \\
+dometemp	  & float	    & dome air temperature     & 290 (K)		   \\
+airtemp		  & float	    & outside air temperature  & 285 (K)		   \\
+mirrortemp	  & float	    & primary mirror temp      & 286 (K)		   \\
+teltemp		  & float	    & telescope structure temp & 283 (K)		   \\
+group ID	  & string	    & observation group name   & target-seq-01		   \\
+group seq	  & int		    & sequence number in group & 3			   \\
+callamp.state     & string          & calibration lamp state   & on / off		   \\
+callamp.mode      & string          & calibration lamp mode    & continuum / line / ??	   \\
+\hline
+background	  & float	    & average of file back    & 5.0			   \\
+dbackground	  & float	    & stdev of file back      & 1.0			   \\
+FHWM		  & float	    & \\
+hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\clearpage
+
+\begin{table}[bh]
+\begin{center}
+\caption{P1 Exposures\label{tab:P1-Exp}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+camera		  & string	    & camera name	       & MegaPrime / GPC           \\
+version		  & int		    & P1 version number	       & 01			   \\
+recipe		  & string	    & analysis recipe name     & basic / flattest	   \\
+state		  & string	    & P1 analysis state	       & new / done		   \\
+\hline
+output		  & string	    & location of output file  & rootname.P1.01.smf        \\
+P1-log		  & string	    & location of P1 logfile   & rootname.P1.01.log        \\
+Nstars		  & int		    & number of astrom stars   & 50			   \\
+sigma.X		  & float	    & astrom scatter in X      & 1.0 (arcsec)		   \\
+sigma.Y		  & float	    & astrom scatter in Y      & 1.2 (arcsec)		   \\
+Mcal		  & float	    & nominal zeropoint        & 25.2 (mag)		   \\
+Moff		  & float	    & measure ZP offset        & 0.5 (mag)		   \\
+dMoff		  & float	    & scatter in Moff	       & 0.1 (mag)		   \\
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{P2 Exposures\label{tab:P2-Exp}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+Nfiles	          & int 	    & number of P2 image files & 3			   \\
+Ndone	          & int 	    & number completed	       & 2			   \\
+P1 version        & int		    & P1 version number	       & 01			   \\
+P2 version	  & int		    & P2 version number	       & 01			   \\
+state		  & string	    & P2 analysis state	       & new / done / fail?	   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{P2 Images\label{tab:P2-Images}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
+recipe		  & string	    & analysis recipe name     & basic / flattest	   \\
+state		  & string	    & P2 analysis state	       & new / done / fail?	   \\
+input.url         & string          & file location	       & rootname.fits		   \\
+output.image.url  & string          & file location	       & rootname.P2.01.fits       \\
+output.obj.url    & string          & file location	       & rootname.P2.01.smf        \\
+output.log.url    & string          & file location	       & rootname.P2.01.log	   \\
+\hline
+bias		  & float	    & measured bias value      & 5.0			   \\
+dbias		  & float	    & bias residual scatter    & 5.0			   \\
+fringe            & float           & fringe amplitude         & 10.0 			   \\
+dfringe           & float           & fringe scatter           & 10.0 			   \\
+sky		  & float	    & reduced background       & 5.0			   \\
+dsky		  & float	    & background scatter       & 5.0			   \\
+Nstars		  & int		    & number of astrom stars   & 50			   \\
+sigma.X		  & float	    & astrom scatter in X      & 1.0 (arcsec)		   \\
+sigma.Y		  & float	    & astrom scatter in Y      & 1.2 (arcsec)		   \\
+Mcal		  & float	    & nominal zeropoint        & 25.2 (mag)		   \\
+Moff		  & float	    & measure ZP offset        & 0.5 (mag)		   \\
+dMoff		  & float	    & scatter in Moff	       & 0.1 (mag)		   \\
+runtime           & float           & processing time          & 20.2 (sec)                \\
+\hline
+\multicolumn{4}{l}{note: the stats below the line are examples to be extended}	   \\
+\end{tabular}
+\end{center}
+\end{table}
+
+The P2 analysis produces a reduced (P2) output image, a table of (P2)
+measured objects, and astrometric calibration terms.  The output
+objects are stored as a FITS table.  The astrometric solution for the
+image is stored in the header of the output object file.  Interesting
+statistics from the analysis are written to the MDDB table.
+Additional information describing the analysis is written to the log
+file in a machine-readable format.  Additional entries can be added to
+the MDDB table if we find they are useful for understanding the
+processing results.
+
+\begin{table}[bh]
+\begin{center}
+\caption{P3 Exposures\label{tab:P3-Exp}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+Nfiles	          & int 	    & number of P2 image files & 3			   \\
+Ndone	          & int 	    & number completed	       & 2			   \\
+P1 version        & int		    & P1 version number	       & 01			   \\
+P2 version	  & int		    & P2 version number	       & 01			   \\
+P3 version	  & int		    & P3 version number	       & 01			   \\
+state		  & string	    & P3 analysis state	       & new / done / fail?	   \\
+recipe		  & string	    & analysis recipe name     & basic / flattest	   \\
+output		  & string	    & location of output file  & rootname.P3.01.smf        \\
+P3-log		  & string	    & location of P3 logfile   & rootname.P3.01.log        \\
+\hline
+Nstars		  & int		    & number of astrom stars   & 50			   \\
+sigma.X		  & float	    & astrom scatter in X      & 1.0 (arcsec)		   \\
+sigma.Y		  & float	    & astrom scatter in Y      & 1.2 (arcsec)		   \\
+Mcal		  & float	    & nominal zeropoint        & 25.2 (mag)		   \\
+Moff		  & float	    & measure ZP offset        & 0.5 (mag)		   \\
+dMoff		  & float	    & scatter in Moff	       & 0.1 (mag)		   \\
+runtime           & float           & processing time          & 20.2 (sec)                \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\clearpage
+
+\begin{table}[bh]
+\begin{center}
+\caption{Master Dark Frames\label{tab:DarkFrames}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+det ID		  & string	    & detrend frame ID	       & 654321o		   \\
+version		  & int		    & version of det frame     & 02			   \\
+label		  & string          & descriptive name         & basic			   \\
+recipe		  & string          & creation recipe          & basic			   \\
+type              & string	    & detrend type	       & bias / dark / flat	   \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+Nfiles		  & int	            & number of files          & 2			   \\
+\hline
+exptime		  & float	    & exposure time            & 5.0			   \\
+Ninput		  & int		    & number of input frames   & 5			   \\
+sigma		  & float	    & stack residual stats     & 2.0			   \\ 
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Master Flat Frames\label{tab:FlatFrames}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+det ID		  & string	    & detrend frame ID	       & 654321o		   \\
+version		  & int		    & version of det frame     & 02			   \\
+label		  & string          & descriptive name         & basic			   \\
+type              & string	    & detrend type	       & bias / dark / flat	   \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+Nfiles		  & int	            & number of files          & 2			   \\
+\hline
+filter.ID	  & string	    & filter ID (glass)	       & g.PS1.01 / none	   \\
+filter.Band	  & string	    & filter bandpass name     & g / none		   \\
+Ninput		  & int		    & number of input frames   & 5			   \\
+sigma		  & float	    & stack residual stats     & 2.0			   \\ 
+Ntotal		  & float	    & ave.total input counts   & 1e6			   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Master Fringe Frames\label{tab:FringeFrames}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+det ID		  & string	    & detrend frame ID	       & 654321o		   \\
+version		  & int		    & version of det frame     & 02			   \\
+label		  & string          & descriptive name         & basic			   \\
+type              & string	    & detrend type	       & bias / dark / flat	   \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+Nfiles		  & int	            & number of files          & 2			   \\
+\hline
+filter.ID	  & string	    & filter ID (glass)	       & g.PS1.01 / none	   \\
+filter.Band	  & string	    & filter bandpass name     & g / none		   \\
+Ninput		  & int		    & number of input frames   & 5			   \\
+sigma		  & float	    & stack residual stats     & 2.0			   \\ 
+Ntotal		  & float	    & ave.total input counts   & 1e6			   \\
+fringe		  & float	    & fringe amplitude	       & 52.0			   \\
+dfringe		  & float	    & fringe stdev	       & 5.0			   \\
+airmass.min	  & float	    & minimum airmass	       & 1.00			   \\
+airmass.max	  & float	    & maximum airmass	       & 1.20			   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Master Detrend Image Files\label{tab:DetImageFiles}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+URL               & string          & file location	       & neb://file001.fits        \\
+det ID		  & string	    & detrend frame ID	       & 654321o		   \\
+{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
+{\it type}	  & string	    & exposure type	       & bias / flat / science	   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\clearpage
+
+\begin{table}[bh]
+\begin{center}
+\caption{Master Detrend Run\label{tab:DetRun}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+det ID		  & string	    & detrend frame ID	       & 654321o		   \\
+camera		  & string	    & camera name	       & MegaPrime / GPC           \\
+version		  & int		    & version of det run       & 02			   \\
+type              & string	    & detrend type	       & bias / dark / flat	   \\
+criteria	  & string	    & image selection criteria & filter flags		   \\
+state		  & string	    & analysis state	       & new / done		   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Input Detrend Exp\label{tab:InputDetExp}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+det ID		  & string	    & detrend frame ID	       & 654321o		   \\
+{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
+{\it version}	  & int		    & version of det run       & 02			   \\
+exp ID		  & string	    & exposure ID	       & 654321o		   \\
+state		  & string	    & analysis state	       & new / raw / proc	   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Input Detrend Image Files\label{tab:InputDetImageFiles}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+raw.URL           & string          & location of raw file     & neb://file001.fits        \\
+proc.URL          & string          & location of proc file    & neb://file001.fits        \\
+det ID		  & string	    & detrend frame ID	       & 654321o		   \\
+{\it version}	  & int		    & version of det run       & 02			   \\
+{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
+{\it type}	  & string	    & exposure type	       & bias / flat / science	   \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
+state		  & string	    & analysis state	       & new / raw / proc	   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\begin{table}[bh]
+\begin{center}
+\caption{Input Detrend Resid Image Files\label{tab:InputDetResidImageFiles}}
+\begin{tabular}{llll}
+\hline
+\hline
+{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
+\hline
+image.URL         & string          & location of resid image  & neb://file001.fits        \\
+thumbnail.URL     & string          & location of resid jpeg   & neb://file001.fits        \\
+det ID		  & string	    & detrend frame ID	       & 654321o		   \\
+version		  & int		    & version of det run       & 02			   \\
+iteration	  & int		    & det creation iteration   & 03			   \\
+{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
+{\it type}	  & string	    & exposure type	       & bias / flat / science	   \\
+class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
+class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
+state		  & string	    & analysis state	       & new / raw / proc	   \\
+scatter		  & float	    & residual scatter	       & 1.0			   \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\end{document}
Index: unk/doc/ipptools/pstask.tex
===================================================================
--- /trunk/doc/ipptools/pstask.tex	(revision 6013)
+++ 	(revision )
@@ -1,1322 +1,0 @@
-\documentclass[panstarrs,spec]{panstarrs}
-
-\title{PStask \& Metadata DB interactions} % put in your title
-\subtitle{Job Flow in IPP}
-\author{Eugene Magnier}
-\audience{IPP}
-%\shorttitle{PSPhot}
-\group{Pan-STARRS IPP}
-\project{Pan-STARRS IPP}
-\organization{Institute for Astronomy}
-\version{DR}
-\docnumber{PSDC-xxx-xxx}
-
-\newcommand\ugriz{$u^\prime g^\prime r^\prime i^\prime z^\prime$}
-\newcommand\grizy{$g r i z y$}
-
-\begin{document}
-\maketitle
-
-\tableofcontents
-\pagebreak 
-\pagenumbering{arabic}
-
-\section{Overview}
-
-This document illustrates the interrelationships between IPP tasks and
-the Metadata Database (MDDB).  The Metadata Database is used to store
-the current IPP analysis state, as well as result processing data
-points.  The tasks needed to define each of the IPP analysis stages
-(Phase 1-4, detrend creation, etc) are illustrated in this document
-and the relevant MDDB tables are listed.  The collection of diagrams
-shows the IPP tasks and the Metadata Database tables needed to manage
-the flow of data through the system.
-
-\note{The flat-field correction analysis stage is not yet defined in
-  this document}
-
-\note{AP DB interfaces are mostly ignored}
-
-\section{PStask queues and metadata tables : persistent vs ephemeral state}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pstasks.01.ps}
-\caption{\label{queues} PStask queues and MDDB tables}
-\end{center}
-\end{figure}
-
-There are some subtleties in the interaction between IPP task
-scheduler, PStasks, the metadata tables which store the system state,
-and the jobs which are immediately being performed.  There is a choice
-to be made between rigorously maintaining the system state in the
-Metadata DB at all times or in keeping an intermediate set of state
-tables.  Keeping the exact system state in the Metadata DB tables
-would require more queries to/from the database and may introduce
-additional latencies which are undesirable.  This is because any
-attempt by the scheduler to initiate a new job would require the
-scheduler to mark the corresponding data item in the Metadata DB (the
-item which acts as the trigger) with a `pending' state, and then mark
-it again as `done' when the job actually completes.  This also has the
-drawback that, if the scheduler crashes, some initial process would be
-required on start up to find all Metadata DB items which are in the
-`pending' state (examining all possible items which can be in such a
-state) and reset them to the `new' state.
-
-We implement an alternative in which the scheduler maintains an
-internal, ephemeral stack of the pending jobs, and only updates the
-system state entries in the Metadata DB when jobs are actually
-completed.  In this scenario, as far as the Metadata DB tables are
-concerned, data items transition only between a `new' and a `done'
-state.  Any jobs which are pending when the system crashes or the
-power is lost are simply dropped, and will be automatically
-re-constructed when the system restarts.  In this paradigm, no
-intermediate operation state is saved, and no partially completed job
-can be recovered.  Since the IPP is defined in terms of a fine
-granularity, with jobs lasting no more than 30 - 120 seconds, this
-model will not have a large impact.
-
-Figure~\ref{queues} illustrates this ephemeral vs persistent state
-information and the interation between the metadata tables and
-PStasks.  The left-hand portion of the diagram illustrates the
-recommended interaction between the metadata database tables and the
-scheduler internal queues.  Some table in the metadata database
-defines a list of data items which are to be processed by some
-analysis job.  The scheduler uses a two-step approach to define the
-analysis jobs based on this list.  First, one scheduler task queries
-the MDDB for a list of pending items, adds the returned items to an
-internal scheduler queue.  The process of adding the elements to the
-queue is defined so that only unique items are added: already existing
-items are skipped.  The entries in the queue consist of the data items
-of interest and an internal temporary state.  At first, this would be
-`pending'.  A second scheduler tasks pops `pending' entries one-by-one
-from this internal queue, submits a job based on the entry, and sets
-the temporary state in the internal queue to `running'.  The internal
-state is needed to prevent the scheduler from re-submitting a job for
-the same data item before the first job is done or assessed.  Since
-the job make take an arbitrary amount of time, the scheduler requires
-a mechanism to remember which data items it has already submitted.
-When the job eventually completes, the metadata database table is
-updated noting the completion.  This may be done either by the job
-itself or by the scheduler.  In addition, the state of the entry in
-the queue can be set to either `done' or the entry can be simply
-removed from the queue.
-
-The purpose of this interaction is to maintain the temporary state
-information within non-persistent elements of the scheduler rather
-than using the metadata database tables to store this information.
-This concept has two advantages.  First, the scheduler internal queues
-are in memory and relatively small, thus interfacing with them is
-quite fast for the scheduler -- this should reduce the system latency.
-Second, by keeping this information non-persistent, the system
-responds correctly to stopping and restarting the scheduler.  Any jobs
-which have not been completed will not be marked in the database, and
-will be restarted naturally by the scheduler.  The alternative, of
-writing a temporary state marker in the database would require a
-restarted scheduler to initially clean all database tables of these
-temporary state markers.
-
-The right-hand portion of the diagram illustrates this process using
-the process of copying the images from the summit as an example.  The
-metadata database table of interest in this case is the list of
-pending images, with entries supplied by a job which queries the
-summit data systems.  The job which is actually performed is a remote
-copy of the image file from the location specified by the summit data
-system to the appropriate location within the IPP Image Server
-(Nebulous).  (As an alternative to the above, the `pending images'
-table may be part of the summit database system, and the `get images'
-command may query the summit directly.  In this scenario, the `copy
-image' command reports to the summit data system that an individual
-image file has been copied.)
-
-In the rest of this document, the use of PStask internal queues to
-manage the temporary data states is glossed over and assumed part of
-the tasks defined in the process.
-
-\section{Summit Copy Tasks}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pstasks.02.ps}
-\caption{\label{pcopy} Summit Copy Tasks}
-\end{center}
-\end{figure}
-
-Figure~\ref{pcopy} illustrates the MDDB tables used to copy data
-(images and metadata tables) from the summit.  The left-hand portion
-of the diagram shows the tables involved in copying images from the
-summit system.  The table of pending image files lists the URLs of the
-individual image files available for transfer, along with their
-associated exposure ID and the camera which generated the image.  Two
-other entries assist in interpreting the file: the class and the class
-ID.  The final entry in this table is the current copy state of the
-file, can have the value of `ready' or `copied'.
-
-The class defines the data grouping represented by this image file and
-may have values of: FPA, Chip, Cell.  This value indicates that the
-provided image file represents the specified portion of the camera
-FPA.  If the value is FPA, the file represents data from a complete
-FPA, though the file may contain pixel data in multiple extensions or
-other groupings to be identified later.  If the value is chip, the
-file contains only data for a single chip, presumably of multiple
-chips available, and equivalently for Cell.  Further discussion of the
-FPA image hierarchy is given in the IPP documents (eg, Modules SDRS).
-The class ID gives the identifier used to name the class level
-corresponding to this file.  This value is necessary to make decisions
-on how to copy the data based on the chip / cell before the data is
-available to IPP components.  Table~\ref{classes} lists likely values
-for the class and class ID for some common cameras.  The system
-described is sufficiently flexible to allow us to transfer the GPC
-images by cell if we eventually decide that is more efficient.
-
-The copy process copies the file from the given URL to the appropriate
-IPP node and adds an entry to the table of new image files, consisting
-of the same information as the pending image file table, though with a
-new value for the URL.  This URL may be an explicit filename, a
-reference to an entry in the image server, or a web address, or
-located on the image server (marked with file:, neb:, and http:,
-respectively).  (TBD: other possible file storage types?  perhaps the
-path could be abstracted without going to the level of the image
-server?  eg: ref:DIR0001/file0001.fits might be in a directory which
-is defined in a table of directories.) After an image file is
-successfully copied, the corresponding state in the `pending chip'
-table is updated from `ready' to `copied'.
-
-\begin{table}
-\begin{center}
-\caption{Camera and Data Classes\label{classes}}
-\begin{tabular}{llll}
-\hline
-\hline
-camera   & class  & classID \\
-\hline
-GPC	 & chip   & chip02 \\
-skyprobe & fpa 	  & sp01 \\
-Megacam  & fpa	  & MegacamSpliced \\
-Suprime	 & chip	  & chip0 \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-The right hand portion of this diagram illustrates the process of
-copying a metadata table.  The table of pending tables lists the URLs
-for the tables which are ready, a unique table ID for each table, and
-the table type.  The copy function copies the listed table and uploads
-the data to the IPP version of the same metadata database.  Two
-examples of metadata tables needed by the IPP for the basic image
-processing system are illustrated: the table of new exposures and the
-table of pending matches.  The first lists the exposures which are
-avilable from the summit system, and all represent entries which are
-available from the Image server.  the second represents the matches
-between exposure IDs and chips
-
-\section{Phase 0}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pstasks.03.ps}
-\caption{\label{phase0} Phase 0 Tasks}
-\end{center}
-\end{figure}
-
-Figure~\ref{phase0} illustrates phase 0, in which the image files are
-categorised, examined for summary information and basic statistics,
-and moved to the tables used to trigger further analysis.  The process
-first examines the `new image files' table.  It selects images from
-this table which have not yet been examined (state is `new').  The
-file header is examined and relevant metadata is extracted (eg, RA,
-DEC, times, and so forth to be defined later).  The process may also
-select a portion of the image pixel data to determine a rough bias and
-background level.  These statistics, whether derived from the header
-or the pixel values, are placed along with image summary information
-in the `raw image files' table, and the state field of the `new image
-files' table is set to `ready'.
-
-The process is also responsible for moving the exposures to the tables
-used for triggering the analysis process.  If the image class is FPA,
-the image can be advanced without waiting for any other image files.
-If the class is Chip or Cell, the process must also examine the `new
-exposure' table for this exposure ID.  The number of class files
-available for this exposure is listed in this table.  The process must
-the select all image files matching the exposure ID with state of
-`ready' and compare the number avalable to the number expected.  If
-the two match, then a new exposure is ready.  Based on the image type
-(from the most recently examined image file header or new exp table?),
-the exposure is added to the `raw exposure' table for images of that
-type.  The allowed types are `detrend', (all bias, dark, flat images),
-`object', `focus'(??), etc.  (** The different tables represent
-different analysis modes.  This process also adds an entry to the exp
-ID / image file match **).  This process also adds all science
-(OBJECT) exposures to the P1 exposure table (for mosaic data) or the
-P2 chip table (for single detector data).  These tables are used to
-trigger the Phase 1 and Phase 2 analysis stages.
-
-\section{Phase 1}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pstasks.04.ps}
-\caption{\label{phase1} Phase 1 Tasks}
-\end{center}
-\end{figure}
-
-Figure~\ref{phase1} shows the tables involved in running the P1
-analysis stage.  There are paths for exposures to enter the analysis
-automatically from the P0 analysis (arrow on left) or to be added
-manually based on a selection from the raw exposure table.  Exposures
-to be analysed by Phase 1 are added to the P1 exposure table with the
-state `new'.  Exposures may be added multiple times for processing and
-reprocessing. The P1 exp table keeps a record of the old attempts for
-debugging and analysis.  Each time an exposure is added to the P1 exp
-table, it is given a new, unique version number, allowing the system
-as a whole to track different analysis attempts.  This method is used
-in all of the image analysis stages (and extrapolated to iterations in
-the detrend analysis steps below).  The top portion of the diagram
-represents the user-space tool which may be used to re-submit an
-exposure or a group of exposures, potentially selected on the basis of
-a query from the raw SCIENCE exposure table.
-
-The P1 exposure table is examined to select the new exposures, these
-are then used to generate the P1 analysis jobs.  Within the analysis
-job, the chips (image files) associated with the exposure are select
-from the raw image file table.  The analysis examines the contents of
-these files, either extract the guide star information from the image
-files (GS table extension) or searches for and centroids the pixels on
-appropriate bright stars.  The analysis results in astrometric
-calibration terms which are written to the astrometric calibration
-file for this exposure.  The location of the astrometric calibration
-file and the statistics of the measurement are written back to the P1
-exposure table.  The images associated with exposures which are
-successfully processed by P1 are then added to the P2 image table,
-which is used to trigger the Phase 2 analysis.
-
-\section{Phase 2}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pstasks.05.ps}
-\caption{\label{phase2} Phase 2 Tasks}
-\end{center}
-\end{figure}
-
-Figure~\ref{phase2} shows the tables involved in running the P2
-analysis stage.  There are paths for images to enter the analysis
-automatically from the P1 analysis (arrow on left) or to be added
-manually based on a selection from the raw exposure and raw image file
-tables.  Images to be analysed by Phase 2 are added to the P2 image
-table with the state `new'.  When images are added to this table, a
-single entry is also added to the P2 exposure table listing the P1 and
-P2 versions for this exposure.  These version numbers must be integers
-starting with 1.  If this image did not have a P1 analysis, the P1
-version is set to 0.  Exposures may be added multiple times for
-processing and reprocessing. The P2 image table keeps a record of the
-old attempts for debugging and analysis.  As with P1, each time a
-collection of associated images from an exposure is added to the P2
-image table, it is given a new, unique version number, allowing the
-system as a whole to track different analysis attempts.  The top
-portion of the diagram represents the user-space tool which may be
-used to re-submit the images for an exposure or a group of exposures,
-potentially selected on the basis of a query from the raw SCIENCE
-exposure and raw image file tables.
-
-The P2 image table is examined to select the `new' images.  These
-images are used to generate P2 analysis jobs.  The P2 analysis uses
-the input url to find and load the image file.  The url may be a file
-on disk, an entry in the image server, Nebulous, etc.  The master
-detrend images matching the specific science image and the conditions
-are selected by examining the table of master detrend frames.  The
-specific detrend image files are selected by using the master detrend
-ID to select the matching the entries in the table of master detrend
-files.  After the analysis, the output image, mask, and FITS table of
-objects, including the astrometry calibration, are written to the P2
-image table, along with summary statistics from the P2 analysis.  The
-state is also updated (to `done').  
-
-Whenever the exposure is completed, the value of Ndone in the P2
-exposure table is incremented.  If all P2 images matching the P2
-exposure version have been completed, the value of Ndone will match
-Nclass, and in this case, the process adds an entry to the P3 exposure
-table.
-
-\section{Phase 3}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pstasks.06.ps}
-\caption{\label{phase3} Phase 3 Tasks}
-\end{center}
-\end{figure}
-
-Figure~\ref{phase3} illustrates the tables involved in the Phase 3
-analysis.  The P3 exposure table lists the exposure ID, the P3
-analysis version, the P2 analysis version to be used as input to this
-P3 analysis, and the recipe to be used.  The P2 exposure and image
-tables are used, in conjunction with the P2 version information, to
-select the P2 output measured objects and the astrometric calibrations
-from P2 and P1.  These measured objects are matched with the reference
-catalog objects, and calibrated astrometry and photometry is produced
-for the full exposure.  The location of the resulting astometry
-calibration table is stored back in the P3 exposure table.  If the
-recipe file specifies, the 2-D photometric and background / fringe
-corrections may also be performed at this stage.  Since these analyses
-require reference data, the recipe may also be used to skip these
-analysis if such reference data is unavailable or unreliable.  At the
-end of Phase 3, the objects from the exposure are inserted into the
-photometry database (this is not shown).
-
-The astrometric calibration portion of Phase 3 is principally needed
-for a mosaic camera.  For single-chip cameras, Phase 3 may be used to
-perform the photometric calibration and simply pass the astrometric
-results along to the output file to be listed in the P3 exposure
-table.  In this way, later stages of the analysis (ie, Phase 4) can
-use the P3 exposure table as input for all cameras, even if all the
-funcionality of Phase 3 is not necessary for that camera.  This would
-be the case for the skyprobe camera, for example.
-
-\section{Phase 4}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pstasks.07.ps}
-\caption{\label{phase4} Phase 4 Tasks}
-\end{center}
-\end{figure}
-
-At the end of Phase 3, the images are ready for Phase 4.  The Phase 3
-diagram shows the output line adding the exposures to be processed by
-Phase 4 to a Phase 4 table.  However, this line is just for
-illustration purposes.  The rules for initiating a Phase 4 run are
-somewhat more complicated than those for running Phases 1-3.  Groups
-of exposures which have an appropriate overlap should be chosen for
-the Phase 4 analysis.  In the steady-state period of PS-4, it may be
-straightforward to choose the exposure groups: they would simply be
-the exposures obtained nearly simultaneously by the four separate
-cameras.  The circumstance for PS-1 will be much more complicated (and
-even PS-4 will probably be more complex than it seems at first
-glance).  For example, in PS-1, we will not have a static sky for most
-of the AP Survery.  In this circumstance, we cannot run P4, at least
-until after the complete AP Reference catalog is built, and
-potentially all exposures re-run through Phase 3.  It may be useful
-for the AP Survey data to split the Phase 4 analysis into two stages:
-image combination and image differencing.  It may even be the case
-that only the combination portion of Phase 4 is performed on the AP
-Survey data.
-
-More generally, the image groups selected for Phase 4 analysis may be
-chosen on the basis of a query of the AP Database (DVO) with some
-rules.  
-
-\note{Phase 4 run can be defined by selecting an observation group, a
-  set of exposures, or a set of rules related to a spatial region (eg,
-  region, time range, and filter}.
-
-\note{Phase 4 discussion (and diagram) needs more work}
-
-\section{Analysis Version and Recipes}
-
-Note that each of the stages P1-P4 refer to the processing version
-from the previous stage.  This allows the processing stage to request
-the correct version of the results from the previous stage, and makes
-it possible to run and re-run the analysis at any stage without
-deleting the earlier results.  As different analysis attempts are
-performed for a given image, the versions branch out.
-
-Also note that at every stage, the entries include a recipe
-identifier.  This is used to select the analysis recipe which should
-be used for this version.  By default, the recipe should be set to the
-current best recipe (use a default name for this?).  This feature
-allows the user to run test analyses with variations on the recipe
-without altering the analysis system.  For example, it is possible to
-use a different flat-field set by specifying alternate rules for the
-flat-field selection in a recipe file.  If it is necessary to run the
-P1-P3 analysis with the raw master flats, for example, the user simply
-defines that selection in the recipe file and submits the images of
-interest to P1 (or P2, etc), with the corresponding entry for the
-recipe.
-
-The recipe file may also be used to specify alternative analysis paths
-and desitinations.  For example, it is not necessary that all analysis
-stops with P4: the recipe file may be used to halt the analysis at P2
-or P3.  In addition, the recipe file may be used to specify an
-alternative destination for the output results.  For example, to
-generate the photometric flat-field correction frame from a collection
-of dithered images, the user may not want the photometry results in
-the main DVO database.  By using the recipe to set an alternative DVO
-database target, and by specifying the use of the raw master flat
-rather than the corrected one, the analysis of the dithered images is
-kept isolated from the other photometry database results.  The
-resulting photometry may be used to construct the new, corrected
-flat-field images, and the processing of the same images using the new
-flat-field images may be sent to the master DVO database.  
-
-\section{Basic Detrend Creation}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pstasks.08.ps}
-\caption{\label{detrend} Detrend Creation Tasks}
-\end{center}
-\end{figure}
-
-Figure~\ref{detrend} illustrates the tables needed for the generic
-detrend construction process, using the flat-field construction as an
-example.  This diagram is somewhat more complex than the preceeding
-versions.  In this diagram, both single jobs and multiple jobs are
-represented by the process elements (the blue ellipses).  In some
-cases, more that one task will be needed to perform the function
-illustrated by a single process task.  The complexity of this diagram
-is enhanced by the need for multiple iterations and both single chip
-and full mosaic processing.  At the moment, the distinction between
-mosaic and single chip cameras is not specifically discussed.
-Finally, the triggers which initiate a specific detrend analysis are
-glossed over.
-
-The detrend analysis is initiated by choosing a type of detrend image
-to be constructed and by specifying the criteria which will be used to
-select the input raw detrend frames for the construction.  For
-example, these criteria could specify that all twilight flat images
-over a certain period of days, perhaps with restrictions on the flux
-levels or the time-from-sunset of the images.  The detrend analysis
-run is given an ID (det ID) which will also be used to identify the
-resulting master detrend frame.  
-
-Given the definition of a master detrend run, the input exposures are
-selected from the raw detrend exposure table, and written to the input
-detrend exposure table.  In the next step, the corresponding image
-files are selected from the table of raw image files.  Since there
-will be a different set of input raw images for each attempt at
-creating a master detrend image, and since any given attempt may use
-some of the same input images as any other attempt, a separate table
-of input raw images is constructed.  
-
-Each of the input raw images may be pre-processed before it may be
-used to construct the detrend frame.  For example, the input
-flat-field images should (probably) be dark- and bias-corrected before
-they are stacked.  The information about these input processed images
-is written to the input images table.  If no processing is needed,
-this step simply copies the appropriate information to the table, and
-points back to the raw image, rather than a processed version.  
-
-The input processed images are combined (stacked) to create a master
-detrend image for the particular data element defined by the image
-class (chip/cell/fpa).  At this stage, not all input images should
-necessarily be included in the stack.  If residual statistics have
-been measured for the input images (say, using a prior stack), then
-some of the input image may be excluded.  The table of residual images
-is used to guide this process.  The information describing the
-resulting master image is written to the master images table.  
-
-The statistics of the master detrend images must examined so that any
-necessary renormalizations may be performed.  For example, after
-stacking the individual flat images, the resulting stacks must be
-renomalized to account for the different ranges of input image fluxes.
-This analysis is least-squares solution in which an appropriate scale
-is determined for each input exposure and a separate gain is
-determined for each of the chips or cells in the camera.  This
-analysis can only performed after all image stacks (ie, for all chips)
-have been constructed.  The resulting information is written to the
-table of master detrend frames.  
-
-Once the master detrend is constructed, the master detrend images may
-be used to construct residual images for each of the input images.
-These residual statistics, as well as the locations of the residual
-images and other related data products (jpeg thumbnails?) are written
-to the residual image table.  Note the red arrow which by-passes the
-stack construction and merge steps and skips directly to the residual
-analysis.  In some cases, it may be useful to have the input images
-confronted with an existing detrend image, and the resulting residual
-values used to guide the rest of the process.  For example, in the
-flat-field analysis, applying an earlier flat can result in a very
-good first-pass rejection of poor input images.  The logic to make
-this leap must be part of the scheduler, since each of the individual
-blocks represent complete processing jobs.
-
-Finally, the residual statistics from the complete mosaic (all input
-images, all chips) are used to assess the quality of the newly
-constructed master detrend image, and to potentially modify the
-selection of input images.  This latter process is performed by
-marking the state of the residual images from this iteration.  The
-stacking process always examines the state information for the
-residual images from the previous iteration, if it exists, when
-constructing the master stack.  Once a master detrend frame has been
-judged of high enough quality, the state of the entry for the frame in
-the master detrend frames table is set to an appropriate value to tell
-the other routines that this image should be used as a master detrend.
-The exact choice of which master detrend frame is used for a given
-science image depends on the recipe along with information such as the
-time period used or the conditions used.
-
-Note that, although this discussion focuses on the construction of
-flat-field images, the same structure should be capable of
-constructing the biases, dark, fringes, etc.  In some cases, as noted
-above, the `process' stage is a null operation.
-
-\pagebreak
-
-\appendix
-\section{IPP top-level commands}
-
-In this section, we define all the necessary top-level commands needed
-by the IPP to implement the data flow discussed in the previous
-sections.  These commands are user commands, and are visible in the
-UNIX command space.  The commands are discussed in the context of both
-the automatic processing and processing of individual entries.  It is
-an important goal that the user commands should provide a
-simple-enough interface that they can be used without needing to have
-the entire infrastructure of the IPP in place to function.  In
-practice, this means that data items which can be acquired from the
-Metadata DB tables can also be provided on the command line and/or in
-the recipe file.  In some cases, user commands are provided to allow a
-manual intervention beyond the automatic processing loops.  This is
-particularly true of the \code{submit.Px} type of commands.  
-
-\note{the command names are for illustration purposes only.  any
-  suggestions for better / fancier names are welcome...}
-
-\begin{verbatim}
-
-copy.image
-  input: url, exp ID, camera, class, class ID
-
-  This program copies the image file from the given URL, updates the
-  new image file table with the descriptive metadata (exp ID, camera,
-  class, class ID), and notifies the external subsystem that the copy
-  has succeeded.  The destination host for the image file is
-  determined by selecting the class ID from the host-for-class table.
-
-copy.table
-  input: url, table ID, table type
-
-  This program copies the metadata table file from the given URL,
-  determines which IPP metadata table it corresponds to, adds the
-  table rows to that metadata table, and notifies the external
-  subsystem that the table copy has succeeded.
-
-classify.image
-  input: (url) or (exp ID + class ID)
-
-  This program examines the header of the specified image file,
-  determines the image type (detrend, science, etc), extracts a
-  specific set of information from the header, and adds a new entry to
-  the raw image files table.  It also checks the new exposure table
-  for the corresponding exp ID, counts the files with the same exp ID
-  that are in the raw image files table, and if they match, adds an
-  entry to the raw exposure table for the appropriate exposure type.
-  Depending on the camera format (mosaic / single chip), the process
-  also adds an entry to either the P1 exposure table or the P2 image table.
-  
-Phase.1
-  input: exp ID
-
-  This program determines the chips which correspond the exposure,
-  loads the guide stars (if they exist) or reads the pixels around
-  bright stars (if the guide stars do not exit.  It determines the
-  centroids for these stars, reads in the astrometric reference stars
-  in the field, matches observed stars to reference stars, determines
-  an astrometric solution, using the default telescope / camera model
-  as the starting point, and writes out the result to the P1 output
-  file, along with the FITS table of observed star measurements (X, Y,
-  inst mag).  It writes the summary statistics back to the P1 exposure
-  table, an new entry in the P2 exposure table, and entries for each
-  of the image files in the P2 images table.
-
-Submit.P2
-  input: exposure selection criteria
-
-  This program uses the provided selection criteria (eg, exp ID, time
-  range + filter, etc) and selects the corresponding exposures and
-  image files.  If the output is specified to the database, it creates
-  new entries in the P2 images and P2 exposure tables, incrementing
-  the version fields if these exposure already exist.  If the output
-  is specified to stdout / file, it writes the corresponding
-  information in a human (and/or machine) readable format.  Note: it
-  is not necessary that a P2 exposure defined by this tool include all
-  possible or available image files.  Note also: this program must
-  validate that the selected exposures have completed Phase 1, or are
-  not required to complete Phase 1.
-
-Phase.2
-  input: (url) or (exp ID + class ID)
-
-  This program reads in the selected image file, determines the
-  matching detrend images, detrends the image data, performs object
-  detection and analysis, loads the corresponding
-  astrometric/photometric reference data, determines the astrometric
-  and photometric calibrations.  It writes out the detrended image,
-  the corresponding mask, and a FITS table of the measured objects,
-  with the astrometric parameters in the FITS table header.  It
-  updates the P2 images table with the statistics of the analysis, and
-  updates the P2 exposure table status.  If this analysis is the last
-  of the Nclass P2 image analyses to be done on this P2 exp ID and
-  version, then the program makes a new entry in the P3 exposure
-  table.  
-
-Submit.P3
-  input: exposure selection criteria
-
-  This program uses the provided selection criteria (eg, exp ID, time
-  range + filter, etc) and selects the corresponding exposures which
-  have succeeded in Phase 2 (a specific P2 version may be specified).
-  If the output is specified to the database, it creates new entries
-  in the P3 exposure table, incrementing the version field if this
-  exposure already exists.  If the output is specified to stdout /
-  file, it writes the corresponding information in a human (and/or
-  machine) readable format.
-
-Phase.3
-  input: (exp ID) or (list of P2 object files)
-
-  This program finds the collection of P2 object files for the
-  specified exposure, reads in the object data and astrometric
-  calibration terms, loads the corresponding astrometric/photometric
-  reference data, determines the improved astrometric and photometric
-  calibrations.  It writes out a single FITS table of the measured
-  objects, with the astrometric parameters in the FITS table header,
-  and astrometric calibration data in a new astrometry file for this
-  exposure.  It updates the P3 exposure table with the statistics of
-  the analysis
-
-Submit.P4
-  input: selection criteria
-
-  This program uses the provided selection criteria (eg, exp ID list,
-  observation group ID, sky region + time range + filter, etc) and
-  selects the corresponding exposures which have succeeded in Phase 3
-  (or Phase 2 if Phase 3 is not needed for this camera / recipe).  It
-  determines the matching sky cells which are overlapped by the
-  selected exposures.  If the output is specified to the database, it
-  creates new entries in the P4 run table and the P4 run input table.
-  If the output is specified to stdout / file, it writes the
-  corresponding information in a human (and/or machine) readable
-  format.
-
-Phase.4
-  input: (run ID) or (sky cell + list of P2 / P3 image files)
-
-  This program uses the skycell and the list of input exposures to
-  find the collection of image files which overlap the given skycell.
-  It then reads the image pixels, warps them to match the skycell
-  geometry, and stacks the image pixels.  It reads the pixels from the
-  skycell and performs the image difference analysis, it photometers
-  the difference image, cleans the input summed image on the basis of
-  the detections, and photometers the input summed image.  It writes
-  out the statistics of the analysis to the P4 run table, the P4 sum
-  table, and the P4 delta table.  If requested, it improves the
-  signal-to-noise in the skycell image, and updates the skycell table.
-
-Detrend.init
-  input: (detrend creation criteria)
-
-  This program adds a new entry to the master detrend run table based
-  on the given criteria.  The criteria may define a time range for the
-  input images, a detrend type, flux ranges, a filter, an observation
-  group ID.  The new master detrend run is created with the iteration
-  set to 00 and the state set to `new'.  
-
-Detrend.get.input.list
-  input: det ID, version
-
-  This program uses the selection criteria defined by the master
-  detrend run (eg, exp ID list, observation group ID, time range, type
-  + filter, etc) to select the corresponding detrend exposures.  It
-  creates a new entry in the master detrend frames table, with state
-  `new' and iteration of 00.  It also writes the exposures to the
-  input detrend exposure table.
-
-Detrend.get.images
-  input: exp ID
-
-  This program identifies the detrend images which correspond to the
-  selected detrend exposure ID and adds an entries for the image files
-  in the input detrend images table, with state set to `raw'.
-
-Detrend.process
-  input: url / detrend type
-
-  This program performs the pre-processing needed by the selected
-  detrend image in order to prepare it for combination with other
-  detrend images.  When completed, the entry for this file in the
-  input detrend images table is update to `proc'.  The processing may
-  be as simple as a null operation (eg, for a bias) or as complex as
-  bias, dark, flat-field, renormalize (eg, for a fringe).
-
-Detrend.stack
-  input: det ID, version, iteration, class ID?
-
-  This program performs the image stacking for a single image class
-  ID.  The stacking process may depend on the detrend type (different
-  for bias from flat from fringe).  The resulting statitics are
-  written to the master detrend image table.  If this is the last of
-  Nclass entries for the given master detrend frame, then the master
-  detrend frames state is updated.
-
-Detrend.merge
-  input: det ID, version, iteration
-
-  This program examines the results of the Nclass master detrend
-  images and performs any necessary re-normalizations.  It also
-  examines the statistics of the individual stacks and summarizes them
-  in the master detrend frame table..
-
-Detrend.residuals
-  input:  det ID, version, iteration, class ID, url
-
-  This program performs the detrending on the given processed input
-  image using the corresponding detrend frame.  The residual image is
-  saved, and an entry is written to the residual images table giving
-  the image location and the residual statistics for this image.
-
-Detrend.assess
-  input: det ID, version, iteration
-
-  This program examines the collection of residual image statistics
-  and creates ensemble statistics for each residual exposure (NOTE: do
-  we need a residual exposure table??).  It examines the ensemble of
-  exposure statistics and determines if 1) the complete stack meets
-  the success criteria and 2) which residual exposure do / do not meet
-  the success criteria.  It updates the master detrend run table, the
-  master detrend frame table, and the residual image / exposure table
-  to note images which should be included / excluded in a future
-  iteration.  
-\end{verbatim}
-
-
-\section{Metadata Database Tables used for IPP Job Flow}
-\label{sec:MetadataTableContents}
-
-The tables presented below define in greater detail the contents of
-the Metadata tables shown in the figures above.  In some cases, the
-quantities (eg, the analysis result statistics) are illustrative, not
-definitive.  In certain tables, data is provided which is redundant
-(non-normal) for ease of use.  In some cases, we may decide not to
-keep this redundant information.  In cases where the choice to
-replicate the redundant information is uncertain, the field names are
-written in {\it italics}.
-
-\begin{table}[bh]
-\begin{center}
-\caption{Pending Image Files\label{tab:PendingImageFiles}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-URL               & string          & file location	       & http://data/file001.fits  \\
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
-state		  & string	    & state of transfer?       & ready / copied		   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{New Image Files\label{tab:NewImageFiles}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-URL               & string          & file location	       & neb://file001.fits        \\
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-camera		  & string	    & camera name	       & MegaPrime / GPC           \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Host for Class\label{tab:HostForClass}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-camera		  & string	    & camera name	       & MegaPrime / GPC           \\
-class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
-hostname	  & string	    & preferred host computer  & po01 / alala.ifa.hawaii.edu \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Raw Image Files\label{tab:RawImageFiles}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-URL               & string          & file location	       & neb://file001.fits        \\
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-camera		  & string	    & camera name	       & MegaPrime / GPC           \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
-type		  & string	    & exposure type	       & bias / flat / science	   \\
-Nreadout	  & int		    & number of readouts       & 1 / 5 / 100		   \\
-Treadout	  & float	    & readout exposure time    & 1.0 (sec)		   \\
-ccdtemp		  & float	    & detector temperature     & 90 (K)			   \\
-\hline
-background	  & float	    & data area median	       & 5.0 (sec)		   \\
-FHWM		  & float	    & average image quality    & 2.5 (arcsec)		   \\
-\hline
-\multicolumn{4}{l}{note: stats below the line are measured, perhaps they go elsewhere?}	   \\
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Pending Metadata Tables\label{tab:PendingTables}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-URL               & string          & file location	       & neb://file001.fits        \\
-table ID	  & string	    & unique table identifier  & table-654321 		   \\
-table type	  & string	    & table content type       & temps / skyprobe data     \\
-state    	  & string	    & copied?                  & ready / copied		   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Copied Metadata Tables\label{tab:CopiedTables}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-URL               & string          & file location	       & neb://file001.fits        \\
-table ID	  & string	    & unique table identifier  & table-654321 		   \\
-table type	  & string	    & table content type       & temps / skyprobe data     \\
-state    	  & string	    & copied?                  & ready / copied		   \\
-time		  & int		    & table arrival time       & 13243413 (s)		   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{New Exposures\label{tab:NewExp}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-camera		  & string	    & camera name	       & MegaPrime / GPC           \\
-telescope	  & string	    & telescope name	       & CFHT / PS-1 / SP-1        \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-Nfiles		  & int		    & number image files       & 64			   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Raw SCIENCE Exposure\label{tab:RawScienceExp}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-camera		  & string	    & camera name	       & MegaPrime / GPC           \\
-telescope	  & string	    & telescope name	       & CFHT / PS-1 / SP-1        \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-Nfiles		  & int		    & number of image files    & 1 / 64			   \\
-\hline
-exptime		  & float	    & exposure time	       & 10.0 (seconds)		   \\
-filter.ID	  & string	    & filter ID (glass)	       & g.PS1.01		   \\
-filter.Band	  & string	    & filter bandpass name     & g			   \\
-airmass		  & float	    & sec(zenith angle)	       & 1.35			   \\
-RA		  & float	    & boresite RA	       & 125.01 (degrees)	   \\
-DEC		  & float	    & boresite DEC	       & 35.01 (degrees)	   \\
-PA		  & float	    & FPA position angle       & 10.1 (degrees)		   \\
-obstime.sec	  & float	    & observation start	       & 123423422 (TAI)	   \\
-obstime.nsec	  & float	    & observation start (nsec) & 1234			   \\
-telfocus	  & float	    & focus distance	       & 1.50 (mm)		   \\
-FPAtemp		  & float	    & FPA temperature	       & 90 (K)			   \\
-dometemp	  & float	    & dome air temperature     & 290 (K)		   \\
-airtemp		  & float	    & outside air temperature  & 285 (K)		   \\
-mirrortemp	  & float	    & primary mirror temp      & 286 (K)		   \\
-teltemp		  & float	    & telescope structure temp & 283 (K)		   \\
-group ID	  & string	    & observation group name   & target-seq-01		   \\
-group seq	  & int		    & sequence number in group & 3			   \\
-\hline
-background	  & float	    & average of file skies    & 5.0			   \\
-dbackground	  & float	    & stdev of file skies      & 1.0			   \\
-FHWM		  & float	    & \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Raw Detrend Exposures\label{tab:RawDetrendExp}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-camera		  & string	    & camera name	       & MegaPrime / GPC           \\
-telescope	  & string	    & telescope name	       & CFHT / PS-1 / SP-1        \\
-type		  & string	    & exposure type	       & bias / domeflat / dark    \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-Nfiles		  & int		    & number of image files    & 1 / 64			   \\
-\hline
-exptime		  & float	    & exposure time	       & 10.0 (seconds)		   \\
-filter.ID	  & string	    & filter ID (glass)	       & g.PS1.01		   \\
-filter.Band	  & string	    & filter bandpass name     & g			   \\
-Alt		  & float	    & boresite Alt	       & 125.01 (degrees)	   \\
-Az		  & float	    & boresite Az	       & 35.01 (degrees)	   \\
-PA		  & float	    & FPA position angle       & 10.1 (degrees)		   \\
-obstime.sec	  & float	    & observation start	       & 123423422 (TAI)	   \\
-obstime.nsec	  & float	    & observation start (nsec) & 1234			   \\
-telfocus	  & float	    & focus distance	       & 1.50 (mm)		   \\
-FPAtemp		  & float	    & FPA temperature	       & 90 (K)			   \\
-dometemp	  & float	    & dome air temperature     & 290 (K)		   \\
-airtemp		  & float	    & outside air temperature  & 285 (K)		   \\
-mirrortemp	  & float	    & primary mirror temp      & 286 (K)		   \\
-teltemp		  & float	    & telescope structure temp & 283 (K)		   \\
-group ID	  & string	    & observation group name   & target-seq-01		   \\
-group seq	  & int		    & sequence number in group & 3			   \\
-callamp.state     & string          & calibration lamp state   & on / off		   \\
-callamp.mode      & string          & calibration lamp mode    & continuum / line / ??	   \\
-\hline
-background	  & float	    & average of file back    & 5.0			   \\
-dbackground	  & float	    & stdev of file back      & 1.0			   \\
-FHWM		  & float	    & \\
-hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\clearpage
-
-\begin{table}[bh]
-\begin{center}
-\caption{P1 Exposures\label{tab:P1-Exp}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-camera		  & string	    & camera name	       & MegaPrime / GPC           \\
-version		  & int		    & P1 version number	       & 01			   \\
-recipe		  & string	    & analysis recipe name     & basic / flattest	   \\
-state		  & string	    & P1 analysis state	       & new / done		   \\
-\hline
-output		  & string	    & location of output file  & rootname.P1.01.smf        \\
-P1-log		  & string	    & location of P1 logfile   & rootname.P1.01.log        \\
-Nstars		  & int		    & number of astrom stars   & 50			   \\
-sigma.X		  & float	    & astrom scatter in X      & 1.0 (arcsec)		   \\
-sigma.Y		  & float	    & astrom scatter in Y      & 1.2 (arcsec)		   \\
-Mcal		  & float	    & nominal zeropoint        & 25.2 (mag)		   \\
-Moff		  & float	    & measure ZP offset        & 0.5 (mag)		   \\
-dMoff		  & float	    & scatter in Moff	       & 0.1 (mag)		   \\
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{P2 Exposures\label{tab:P2-Exp}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-Nfiles	          & int 	    & number of P2 image files & 3			   \\
-Ndone	          & int 	    & number completed	       & 2			   \\
-P1 version        & int		    & P1 version number	       & 01			   \\
-P2 version	  & int		    & P2 version number	       & 01			   \\
-state		  & string	    & P2 analysis state	       & new / done / fail?	   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{P2 Images\label{tab:P2-Images}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
-recipe		  & string	    & analysis recipe name     & basic / flattest	   \\
-state		  & string	    & P2 analysis state	       & new / done / fail?	   \\
-input.url         & string          & file location	       & rootname.fits		   \\
-output.image.url  & string          & file location	       & rootname.P2.01.fits       \\
-output.obj.url    & string          & file location	       & rootname.P2.01.smf        \\
-output.log.url    & string          & file location	       & rootname.P2.01.log	   \\
-\hline
-bias		  & float	    & measured bias value      & 5.0			   \\
-dbias		  & float	    & bias residual scatter    & 5.0			   \\
-fringe            & float           & fringe amplitude         & 10.0 			   \\
-dfringe           & float           & fringe scatter           & 10.0 			   \\
-sky		  & float	    & reduced background       & 5.0			   \\
-dsky		  & float	    & background scatter       & 5.0			   \\
-Nstars		  & int		    & number of astrom stars   & 50			   \\
-sigma.X		  & float	    & astrom scatter in X      & 1.0 (arcsec)		   \\
-sigma.Y		  & float	    & astrom scatter in Y      & 1.2 (arcsec)		   \\
-Mcal		  & float	    & nominal zeropoint        & 25.2 (mag)		   \\
-Moff		  & float	    & measure ZP offset        & 0.5 (mag)		   \\
-dMoff		  & float	    & scatter in Moff	       & 0.1 (mag)		   \\
-runtime           & float           & processing time          & 20.2 (sec)                \\
-\hline
-\multicolumn{4}{l}{note: the stats below the line are examples to be extended}	   \\
-\end{tabular}
-\end{center}
-\end{table}
-
-The P2 analysis produces a reduced (P2) output image, a table of (P2)
-measured objects, and astrometric calibration terms.  The output
-objects are stored as a FITS table.  The astrometric solution for the
-image is stored in the header of the output object file.  Interesting
-statistics from the analysis are written to the MDDB table.
-Additional information describing the analysis is written to the log
-file in a machine-readable format.  Additional entries can be added to
-the MDDB table if we find they are useful for understanding the
-processing results.
-
-\begin{table}[bh]
-\begin{center}
-\caption{P3 Exposures\label{tab:P3-Exp}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-Nfiles	          & int 	    & number of P2 image files & 3			   \\
-Ndone	          & int 	    & number completed	       & 2			   \\
-P1 version        & int		    & P1 version number	       & 01			   \\
-P2 version	  & int		    & P2 version number	       & 01			   \\
-P3 version	  & int		    & P3 version number	       & 01			   \\
-state		  & string	    & P3 analysis state	       & new / done / fail?	   \\
-recipe		  & string	    & analysis recipe name     & basic / flattest	   \\
-output		  & string	    & location of output file  & rootname.P3.01.smf        \\
-P3-log		  & string	    & location of P3 logfile   & rootname.P3.01.log        \\
-\hline
-Nstars		  & int		    & number of astrom stars   & 50			   \\
-sigma.X		  & float	    & astrom scatter in X      & 1.0 (arcsec)		   \\
-sigma.Y		  & float	    & astrom scatter in Y      & 1.2 (arcsec)		   \\
-Mcal		  & float	    & nominal zeropoint        & 25.2 (mag)		   \\
-Moff		  & float	    & measure ZP offset        & 0.5 (mag)		   \\
-dMoff		  & float	    & scatter in Moff	       & 0.1 (mag)		   \\
-runtime           & float           & processing time          & 20.2 (sec)                \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\clearpage
-
-\begin{table}[bh]
-\begin{center}
-\caption{Master Dark Frames\label{tab:DarkFrames}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-det ID		  & string	    & detrend frame ID	       & 654321o		   \\
-version		  & int		    & version of det frame     & 02			   \\
-label		  & string          & descriptive name         & basic			   \\
-recipe		  & string          & creation recipe          & basic			   \\
-type              & string	    & detrend type	       & bias / dark / flat	   \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-Nfiles		  & int	            & number of files          & 2			   \\
-\hline
-exptime		  & float	    & exposure time            & 5.0			   \\
-Ninput		  & int		    & number of input frames   & 5			   \\
-sigma		  & float	    & stack residual stats     & 2.0			   \\ 
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Master Flat Frames\label{tab:FlatFrames}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-det ID		  & string	    & detrend frame ID	       & 654321o		   \\
-version		  & int		    & version of det frame     & 02			   \\
-label		  & string          & descriptive name         & basic			   \\
-type              & string	    & detrend type	       & bias / dark / flat	   \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-Nfiles		  & int	            & number of files          & 2			   \\
-\hline
-filter.ID	  & string	    & filter ID (glass)	       & g.PS1.01 / none	   \\
-filter.Band	  & string	    & filter bandpass name     & g / none		   \\
-Ninput		  & int		    & number of input frames   & 5			   \\
-sigma		  & float	    & stack residual stats     & 2.0			   \\ 
-Ntotal		  & float	    & ave.total input counts   & 1e6			   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Master Fringe Frames\label{tab:FringeFrames}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-det ID		  & string	    & detrend frame ID	       & 654321o		   \\
-version		  & int		    & version of det frame     & 02			   \\
-label		  & string          & descriptive name         & basic			   \\
-type              & string	    & detrend type	       & bias / dark / flat	   \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-Nfiles		  & int	            & number of files          & 2			   \\
-\hline
-filter.ID	  & string	    & filter ID (glass)	       & g.PS1.01 / none	   \\
-filter.Band	  & string	    & filter bandpass name     & g / none		   \\
-Ninput		  & int		    & number of input frames   & 5			   \\
-sigma		  & float	    & stack residual stats     & 2.0			   \\ 
-Ntotal		  & float	    & ave.total input counts   & 1e6			   \\
-fringe		  & float	    & fringe amplitude	       & 52.0			   \\
-dfringe		  & float	    & fringe stdev	       & 5.0			   \\
-airmass.min	  & float	    & minimum airmass	       & 1.00			   \\
-airmass.max	  & float	    & maximum airmass	       & 1.20			   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Master Detrend Image Files\label{tab:DetImageFiles}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-URL               & string          & file location	       & neb://file001.fits        \\
-det ID		  & string	    & detrend frame ID	       & 654321o		   \\
-{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
-{\it type}	  & string	    & exposure type	       & bias / flat / science	   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\clearpage
-
-\begin{table}[bh]
-\begin{center}
-\caption{Master Detrend Run\label{tab:DetRun}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-det ID		  & string	    & detrend frame ID	       & 654321o		   \\
-camera		  & string	    & camera name	       & MegaPrime / GPC           \\
-version		  & int		    & version of det run       & 02			   \\
-type              & string	    & detrend type	       & bias / dark / flat	   \\
-criteria	  & string	    & image selection criteria & filter flags		   \\
-state		  & string	    & analysis state	       & new / done		   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Input Detrend Exp\label{tab:InputDetExp}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-det ID		  & string	    & detrend frame ID	       & 654321o		   \\
-{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
-{\it version}	  & int		    & version of det run       & 02			   \\
-exp ID		  & string	    & exposure ID	       & 654321o		   \\
-state		  & string	    & analysis state	       & new / raw / proc	   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Input Detrend Image Files\label{tab:InputDetImageFiles}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-raw.URL           & string          & location of raw file     & neb://file001.fits        \\
-proc.URL          & string          & location of proc file    & neb://file001.fits        \\
-det ID		  & string	    & detrend frame ID	       & 654321o		   \\
-{\it version}	  & int		    & version of det run       & 02			   \\
-{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
-{\it type}	  & string	    & exposure type	       & bias / flat / science	   \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
-state		  & string	    & analysis state	       & new / raw / proc	   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\begin{table}[bh]
-\begin{center}
-\caption{Input Detrend Resid Image Files\label{tab:InputDetResidImageFiles}}
-\begin{tabular}{llll}
-\hline
-\hline
-{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
-\hline
-image.URL         & string          & location of resid image  & neb://file001.fits        \\
-thumbnail.URL     & string          & location of resid jpeg   & neb://file001.fits        \\
-det ID		  & string	    & detrend frame ID	       & 654321o		   \\
-version		  & int		    & version of det run       & 02			   \\
-iteration	  & int		    & det creation iteration   & 03			   \\
-{\it camera}	  & string	    & camera name	       & MegaPrime / GPC           \\
-{\it type}	  & string	    & exposure type	       & bias / flat / science	   \\
-class		  & string	    & file data class	       & Cell / Chip / FPA	   \\ 
-class ID	  & string	    & identify for class       & chip00 / cell0102	   \\
-state		  & string	    & analysis state	       & new / raw / proc	   \\
-scatter		  & float	    & residual scatter	       & 1.0			   \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-
-\end{document}