Changeset 6023

Timestamp:

Jan 16, 2006, 10:04:01 PM (21 years ago)

Author:

eugene

Message:

updated the text and figures

Location:

trunk/doc/ipptools

Files:

• ipptools.tex (modified) (12 diffs)
• pics/pantasks.03.ps (modified) ( previous)
• pics/pantasks.04.ps (modified) ( previous)
• pics/pantasks.05.ps (modified) ( previous)
• pics/pantasks.06.ps (modified) ( previous)
• pics/pantasks.08.ps (modified) ( previous)
• pics/pantasks.09.ps (added)
• pics/pantasks.10.ps (added)
• pics/pantasks.11.ps (added)
• states.sxc (added)
• tasks.sxd (modified) ( previous)

trunk/doc/ipptools/ipptools.tex

@@ -1 +1 @@
 \documentclass[panstarrs,spec]{panstarrs}
 
-\title{PanTasks \& the IPP Analysis Stages}
+\title{IPPTools, PanTasks \& the IPP Analysis Stages}
 \subtitle{Job Relationships and Data Flow}
 \author{Eugene Magnier}
@@ -117 +117 @@
 processing cluster.
 
-\section{Persistent vs Ephemeral State in Pantasks}
+\section{Persistent vs Ephemeral State in PanTasks}
 
 \begin{figure}
 \begin{center}
 \includegraphics[scale=0.85]{pics/pantasks.01.ps}
-\caption{\label{queues} Pantask queues and MDDB tables}
+\caption{\label{queues} PanTasks queues and MDDB tables}
 \end{center}
 \end{figure}
@@ -213 +213 @@
 image file has been copied.)
 
-In the rest of this document, the use of Pantask internal queues to
+In the rest of this document, the use of PanTasks internal queues to
 manage the temporary data states is glossed over and assumed part of
 the tasks defined in the process.
+
+\section{IPP Pipelines Overview}
+
+The IPP as a whole performs all of the image analysis functions
+required by the Pan-STARRS telescopes, including images from the full
+Gigapixel camera (or cameras), the test camera TC-3, and the SkyProbe
+camera.  The IPP is designed to be very flexible, with instrument
+specific details isolated in configuration files associated with the
+different cameras known to the system.  As a result, the organization
+of the top level analysis infrastructure must be sufficiently general
+that a wide range of cameras can be accomodated.  We have a few
+general principles regarding constraints on the data to be processed
+which are used to guide the IPP design and developement:
+
+\begin{itemize}
+\item {\bf Camera Focal Plane Hierarchy} The IPP analysis programs
+  assume that the images to be processed are obtained by a camera
+  which can be represented by our Camera Focal-Plane Hierarchy of data
+  structures.  This hierarchy is discussed in detail in the Modules
+  SDRS, and defines a top-level {\em Focal-Plane Array (FPA)}, which
+  may contain 1 or more {\em Chips}, each of which may contain one or
+  more {\em Cells}.  An {\em FPA} is identified as having a single
+  optical system feeding photons to the detectors.  A {\em Chip} is
+  identified as a unit of data all deriving from a single detector
+  (piece of silicon), while a {\em Cell} is identified as a collection
+  of pixels read out as a continuous cartesian grid.  Finally, a
+  single collection of data from an {\em FPA} may include multiple
+  {\em Readouts} from any or all of the {\em Cells}.  
+
+\item {\bf Exposures vs Groups} The processing presumes that the data
+  is organized into {\em exposures} and exposure {\em groups}.  An
+  exposure represents the data from a single FPA, with the possible
+  subdivision of the exposure into multiple readouts for some or all
+  of the cells.  Exposure {\em Groups} are any group of exposures
+  which are related together in some way; the definition of the {\em
+  Groups} may be provided by the observers, or they may be derived
+  from the characteristics of the exposures.  The use of a particular
+  {\em group} depends on the context of that group.  A few examples of
+  exposure groups:
+
+  \begin{itemize}
+  \item a dithered sequence of exposures to be stacked for cosmetics
+  and improved signal-to-noise.
+  \item a twilight flat-field sequence.
+  \item all images of the same filter within a 10 degree region to be
+  used to construct an sample astrometric reference.  
+  \end{itemize}
+
+\item {\bf Image Files (imfiles) vs Exposures}  Any single exposure
+  may consist of a number of different data files.  The number of {\em
+  imfiles} for a given exposure will depend on the camera, as will the
+  data organization within those image files.  Also, a particular
+  camera will supply files corresponding to one of the particular
+  Focal-Plane Hierarchy elements.  The IPP analysis must be able to
+  interpret the incoming data correctly.
+\end{itemize}
+
+As discussed elsewhere, there are several major types of analysis
+performed by the IPP.  For the purposes of data organization and
+parallel processing efficiencies, we have identified the following
+divisions of the analysis tasks.  These will be discuss in much more
+detail below.
+
+\begin{itemize}
+\item {\bf Science Image Analysis} : This represents the analysis
+  performed on the images obtained by the telescope, and generally
+  performed in real-time, night-by-night.  The science image analysis
+  tasks are further subdivided as follows:
+
+  \begin{itemize}
+  \item {\bf Phase 1} : The full focal-plane array is examined quickly
+  to determine an initial astrometric calibration.  In this step, the
+  OTA guide stars may be used as the astrometric reference; if none
+  are available, predicted bright star positions are examined.  This
+  step is only used for mosaic images, and may be skipped if no guide
+  stars are available {\em and} the astrometric calibration for the
+  telescope / camera is reliable (better than 10 arcseconds).
+
+  \item {\bf Phase 2} : Each image file is analysed independently: the
+  image is detrended (bias, dark, flat, fringe, etc), sources are then
+  detected to a modest level, improved astrometric calibration is
+  performed.
+
+  \item {\bf Phase 3} : The collection of sources measured from all of
+  the image files for the camera are used to determine a global
+  astrometric, and possibly photometric, solution for the exposure.
+  This step is only required for mosaic cameras.
+
+  \item {\bf Phase 4.1} : An exposure group consisting of images
+  obtained in a specific region of the sky are merged together.  In
+  this step, the images are first warped to a common pixel grid, defined by
+  the static sky images.  The collection of images are then used to
+  construct a single, cleaned image by rejecting the outliers from the
+  source images in the stack.  The corresponding static sky pixels are
+  then used to construct a difference image from the resulting stack.
+
+  \item {\bf Magic} : In this step, the difference images are examined
+  to find the trailed images introduced by artificial satelites.
+  These so-called {\em streaks} are excised from the difference
+  images, as well as all of the source images which were used to
+  generate the difference images; the public data sources are updated
+  with the precise, correct time.  Note that this step requires that
+  separate difference images be generated for each of the input
+  images, a step which would be skipped if {\em magic} were avoided.
+  Also note that, until {\em magic} is performed, the publically
+  available time has a limited precision (probably $\sim 1$ minute
+  errors).  This step is only necessary in the operational IPP system
+  given the restrictions from the Air Force.
+
+  \item {\bf Phase 4.2} : After {\em magic} the final difference and
+  the final cleaned stacked image are produced and objects in both
+  images are detected.  The difference sources are used to mask the
+  extreme outliers in the cleaned stack, which is then used to update
+  the Static Sky images. 
+  \end{itemize}
+
+ \item {\bf Static Sky Image Analysis} : While the science image
+ analysis is performed as images are availablef, the static sky image
+ analysi occurs on a very different timescale.  In steady state, the
+ full static sky analysis will take place over the course of a full
+ year.  At any given time, the portion of the sky corresponding to the
+ location of the sun will be under-going the analysis.  In practice,
+ for PS-1, the static sky is produced in a somewhat different fashion
+ than in the steady-state model.  In PS-1, the different survey
+ strategies introduce very different update rates for the static sky.
+ At one extreme, the AP Survey will not have enough data for a
+ complete static sky analysis until nearly 22 months after the survey
+ begins.  At the other extreme, the deep survey, which observes a much
+ smaller portion of the sky, may best be analysed quite frequently.
+ These details are part of the science guidelines of the PS-1 surveys,
+ and are beyond the scope of this document.  Rather, the IPP Static
+ Sky Image Analysis must provide the capability of defining the static
+ sky analysis in a flexible and dynamic fashion.
+
+\item {\bf Basic Detrend Creation Analysis} : The analysis of most of
+  the detrend data is grouped together in a common analysis stage.
+  The differences between the analysis of the bias, dark, flat, and
+  fringe images is primarily one of how the input images are
+  pre-processed, what statistic is used to characterize a given input
+  image, how the input images are scaled before being combined, and
+  what normalization is applied to the resulting image.  All of these
+  types of detrend images can thus be processed with a single analysis
+  pipeline which is made aware of these minor differences.  This stage
+  is never the less fairly complex, and as a result is subdivided into
+  several compenents, as discussed below.
+
+\item {\bf Other analyses} There are a number of other tasks which the
+  IPP must perform that are not well-defined by the different analysis
+  types discussed above.  Some analysis tasks are not automatically
+  triggered, and are thus outside the scope of this document; these
+  are the tasks which are more properly considered as research
+  projects than analysis systems.  The other important automatic tasks
+  are:
+  \begin{itemize}
+    \item {\bf Summit Copy} : In this stage, the data source or data
+    sources are queried for new exposures and image files, which are
+    then copied to the IPP data area.  This stage also includes the
+    copying of other metadata which are not included in the image
+    files.
+    
+    \item {\bf Image Classification} : new images which are introduced
+    to the IPP are examined by this analysis stage and placed in the
+    appropriate table for processing.  This step includes a small
+    amount of accumulating statistics about the images.
+
+    \item {\bf Data File management} : a few tasks are necessary to
+    monitor and maintain the clustered storage system.  These tasks
+    include the automatic duplication and deletion of different types
+    of files from Nebulous, the file storage archive.  This also
+    includes automatic redistribution of machine assignments as
+    hardware is added or removed from the system.  This collection of
+    tasks also includes monitoring of system parameters to alert
+    people in case of dangerous hardware situations.
+
+    \item {\bf Irregular Calibration Data} certain types of
+    calibration information is extracted on different intervals from
+    the more regular detrend images.  These types of calibration data
+    include improved telescope pointing models, astrometric
+    calibrations, photometric calibrations, flat-field correction
+    frames.
+  \end{itemize}
+\end{itemize}
+
+\section{Tables, Tasks and Tools}
+
+The following sections discuss the database tables, the tasks within
+PanTasks, and the collection of programs used by PanTasks to examine
+and manipulate the state tables.  These later programs do not, in
+general, perform any in depth analysis; instead they perform actions
+such as selecting from one table images ready for analysis in a
+following processing step.  This collection of tools is grouped under
+the name of the {\tt ippTools}, and consists of a separate tool for
+each of the different major analysis steps.
+
+The {\tt ippTools} make use of {\em glueforge} to simplify the
+management of the database table schema.  Glueforge provides a single
+mechanism to generate a collection of C data structures, database
+tables, database access APIs, and I/O routines from a simple table
+description configuration file.  All APIs generated by {\em glueforge}
+for the same type of interaction have common naming schemes.  This
+technique has several important advantages.  It makes the writing of C
+database interactions very quick and easy.  It also makes it easy to
+modify the database schema without disrupting the software
+development.  Finally, it provides a simple, self-documenting source
+for data structure of multiple types which can be shared between
+programs or platforms.
+
+Within the following diagrams, we illustrate the database tables used
+to track the state of the IPP.  We also show the commands provided by
+{\tt ippTools} to connect the tables.  Finally, we show the IPP tasks
+which initiate the different analysis steps.  The following set of
+diagrams uses several consistent features.  The blue-and-grey
+rectangles define the metadata database tables.  The blue section
+contains the table name, while the grey section lists a minimal subset
+of the table columns.  The ellipses represent programs (or program
+portions in some cases) executed by PanTasks.  The blue filled
+ellipses represent the {\tt ippTools} commands which are executed
+locally on the computer hosting PanTasks.  The grey-blue ellipses
+represent the commands executed on the parallel cluster, monitored by
+{\tt pcontrol}.  The green ellipses represent commands executed by
+hand for testing and manual intervention.
+
+In most of the analysis tasks, we use a two-table approach to the data
+in order to avoid excessive latencies.  One table is used to track
+quantities which are still pending for a particular stage.  When the
+analysis is completed, these items are moved from the 'pending' tables
+to corresponding 'done' tables.  Although this introduces a somewhat
+higher number of tables and complexity, it will avoid the system from
+slowing down as the number of data items grows with time.  The pending
+tables are searched repeatedly by the {\tt ippTools} programs as they
+attempt to select new data of interest.  In contrast, the done tables
+are searched much less frequently.  
 
 \section{Summit Copy Tasks}
@@ -306 +538 @@
 Figure~\ref{phase0} illustrates phase 0, in which the image files are
 categorised, examined for summary information and basic statistics,
-and moved to the tables used to trigger further analysis.  The process
-first examines the `new image files' table.  It selects images from
-this table which have not yet been examined (state is `new').  The
+and moved to the later phase 'pending' tables to trigger further
+analysis.  The command {\tt p0search -pending} examines the `new
+imfiles' and 'new exposure' tables.  It selects images from this table
+which have not yet been examined (state is `new').  These are returned
+to PanTasks, which sends each image file to a separate analysis node
+running the {\tt p0search -update} command.  With this command, the
 file header is examined and relevant metadata is extracted (eg, RA,
 DEC, times, and so forth to be defined later).  The process may also
@@ -317 +552 @@
 files' table is set to `ready'.
 
-The process is also responsible for moving the exposures to the tables
-used for triggering the analysis process.  If the image class is FPA,
-the image can be advanced without waiting for any other image files.
-If the class is Chip or Cell, the process must also examine the `new
-exposure' table for this exposure ID.  The number of class files
-available for this exposure is listed in this table.  The process must
-the select all image files matching the exposure ID with state of
-`ready' and compare the number avalable to the number expected.  If
-the two match, then a new exposure is ready.  Based on the image type
-(from the most recently examined image file header or new exp table?),
-the exposure is added to the `raw exposure' table for images of that
-type.  The allowed types are `detrend', (all bias, dark, flat images),
-`object', `focus'(??), etc.  (** The different tables represent
-different analysis modes.  This process also adds an entry to the exp
-ID / image file match **).  This process also adds all science
-(OBJECT) exposures to the P1 exposure table (for mosaic data) or the
-P2 chip table (for single detector data).  These tables are used to
-trigger the Phase 1 and Phase 2 analysis stages.
+The {\tt p0search -update} command is also responsible for moving the
+exposures to the tables used for triggering the analysis process.  If
+the image class is FPA, the image can be advanced without waiting for
+any other image files.  If the class is Chip or Cell, the process must
+also examine the `new exposure' table for this exposure ID.  The
+number of class files available for this exposure is listed in this
+table.  The process must the select all image files matching the
+exposure ID with state of `ready' and compare the number avalable to
+the number expected.  If the two match, then a new exposure is ready.
+Based on the image type (from the most recently examined image file
+header or new exp table?), the exposure is added to the `raw exposure'
+table for images of that type.  The allowed types are `detrend', (all
+bias, dark, flat images), `object', `focus'(??), etc.  (** The
+different tables represent different analysis modes.  This process
+also adds an entry to the exp ID / image file match **).  This process
+also adds all science (OBJECT) exposures to the P1 exposure table (for
+mosaic data) or the P2 chip table (for single detector data).  These
+tables are used to trigger the Phase 1 and Phase 2 analysis stages.
 
 \section{Phase 1}
@@ -345 +580 @@
 \end{figure}
 
-Figure~\ref{phase1} shows the tables involved in running the P1
+Figure~\ref{phase1} shows the tables involved in running the Phase 1
 analysis stage.  There are paths for exposures to enter the analysis
-automatically from the P0 analysis (arrow on left) or to be added
+automatically from the Phase 0 analysis (arrow on left) or to be added
 manually based on a selection from the raw exposure table.  Exposures
 to be analysed by Phase 1 are added to the P1 exposure table with the
 state `new'.  Exposures may be added multiple times for processing and
-reprocessing. The P1 exp table keeps a record of the old attempts for
-debugging and analysis.  Each time an exposure is added to the P1 exp
-table, it is given a new, unique version number, allowing the system
-as a whole to track different analysis attempts.  This method is used
-in all of the image analysis stages (and extrapolated to iterations in
-the detrend analysis steps below).  The top portion of the diagram
-represents the user-space tool which may be used to re-submit an
-exposure or a group of exposures, potentially selected on the basis of
-a query from the raw SCIENCE exposure table.
-
-The P1 exposure table is examined to select the new exposures, these
-are then used to generate the P1 analysis jobs.  Within the analysis
-job, the chips (image files) associated with the exposure are select
-from the raw image file table.  The analysis examines the contents of
-these files, either extract the guide star information from the image
-files (GS table extension) or searches for and centroids the pixels on
+reprocessing. The P1 done exposure table keeps a record of the old
+attempts for debugging and analysis.  Each time an exposure is added
+to the P1 exp table, it is given a new, unique version number,
+allowing the system as a whole to track different analysis attempts.
+This method is used in all of the image analysis stages (and
+extrapolated to iterations in the detrend analysis steps below).  The
+top portion of the diagram shows the use of the command {\tt p1search
+-define} to select and submit an exposure or a group of exposures,
+potentially selected on the basis of a query from the raw science
+exposure table.
+
+The P1 pending exposure table is examined by {\tt p1search -pending}
+to select the new exposures, which are sent to PanTasks.  PanTasks
+initiates a separate analysis job (p1astro) for each exposure, which
+are sent to the parallel processing nodes.  Within the analysis job,
+the chips (image files) associated with the exposure are select from
+the raw image file table.  The analysis examines the contents of these
+files, either extract the guide star information from the image files
+(GS table extension) or searches for and centroids the pixels on
 appropriate bright stars.  The analysis results in astrometric
 calibration terms which are written to the astrometric calibration
@@ -373 +611 @@
 exposure table.  The images associated with exposures which are
 successfully processed by P1 are then added to the P2 image table,
-which is used to trigger the Phase 2 analysis.
+which is used to trigger the Phase 2 analysis.  This last step is
+performed by the command {\tt p1search -done}, which is executed
+regularly to search for completed Phase 1 jobs.
 
 \section{Phase 2}
@@ -388 +628 @@
 automatically from the P1 analysis (arrow on left) or to be added
 manually based on a selection from the raw exposure and raw image file
-tables.  Images to be analysed by Phase 2 are added to the P2 image
-table with the state `new'.  When images are added to this table, a
-single entry is also added to the P2 exposure table listing the P1 and
-P2 versions for this exposure.  These version numbers must be integers
-starting with 1.  If this image did not have a P1 analysis, the P1
-version is set to 0.  Exposures may be added multiple times for
-processing and reprocessing. The P2 image table keeps a record of the
-old attempts for debugging and analysis.  As with P1, each time a
-collection of associated images from an exposure is added to the P2
-image table, it is given a new, unique version number, allowing the
-system as a whole to track different analysis attempts.  The top
-portion of the diagram represents the user-space tool which may be
-used to re-submit the images for an exposure or a group of exposures,
-potentially selected on the basis of a query from the raw SCIENCE
-exposure and raw image file tables.
-
-The P2 image table is examined to select the `new' images.  These
-images are used to generate P2 analysis jobs.  The P2 analysis uses
-the input url to find and load the image file.  The url may be a file
-on disk, an entry in the image server, Nebulous, etc.  The master
-detrend images matching the specific science image and the conditions
-are selected by examining the table of master detrend frames.  The
-specific detrend image files are selected by using the master detrend
-ID to select the matching the entries in the table of master detrend
-files.  After the analysis, the output image, mask, and FITS table of
-objects, including the astrometry calibration, are written to the P2
-image table, along with summary statistics from the P2 analysis.  The
-state is also updated (to `done').  
-
-Whenever the exposure is completed, the value of Ndone in the P2
-exposure table is incremented.  If all P2 images matching the P2
-exposure version have been completed, the value of Ndone will match
-Nclass, and in this case, the process adds an entry to the P3 exposure
-table.
+tables.  Image files to be analysed by Phase 2 are added to the P2
+pending imfiles table with the state `new'.  When images are added to
+this table, a single entry is also added to the P2 exposure table
+listing the P1 and P2 versions for this exposure.  These version
+numbers must be integers starting with 1.  If this image did not have
+a P1 analysis, the P1 version is set to 0.  Exposures may be added
+multiple times for processing and reprocessing. The P2 image table
+keeps a record of the old attempts for debugging and analysis.  As
+with P1, each time a collection of associated images from an exposure
+is added to the P2 image table, it is given a new, unique version
+number, allowing the system as a whole to track different analysis
+attempts.  Note that these version numbers are unique for each {\em
+exposure} processed by Phase 2, not just for any image file.  The top
+portion of the diagram illustrates the behavior of the commands {\tt
+p2search -define} and {\tt p2search -quick}.  The first may be used to
+re-submit the images for an exposure or a group of exposures,
+potentially selected on the basis of a query from the raw science
+exposure and raw image file tables.  The second version sends images
+files directly to PanTasks for processing; these entries will not be
+included in the processing tables, and is used only for testing
+purposes.
+
+The P2 pending image table is examined with the command {\tt p2search
+  -pending} to select the `new' images.  These images are used by
+PanTasks to generate P2 analysis jobs, running the analysis command
+{\tt ppImage}.  The P2 analysis uses the input url to find and load
+the image file.  The url may be a file on disk, an entry in the image
+server, Nebulous, etc.  The master detrend images matching the
+specific science image and the conditions are selected by examining
+the table of master detrend frames.  The specific detrend image files
+are selected by using the master detrend ID to select the matching the
+entries in the table of master detrend files.  After the analysis, the
+output image, mask, and FITS table of objects, including the
+astrometry calibration, are written back to the P2 image table, along
+with summary statistics from the P2 analysis.  The state is also
+updated (to `done').
+
+The completed images are examined by the command {\tt p2search -done},
+and when all image files for a single exposure are completed, this
+command migrates them to the P2 done table.  This process is also
+responsible for populating the P3 pending tables so exposures may be
+processing by Phase 3.
 
 \section{Phase 3}
@@ -432 +680 @@
 \end{figure}
 
-Figure~\ref{phase3} illustrates the tables involved in the Phase 3
-analysis.  The P3 exposure table lists the exposure ID, the P3
-analysis version, the P2 analysis version to be used as input to this
-P3 analysis, and the recipe to be used.  The P2 exposure and image
-tables are used, in conjunction with the P2 version information, to
-select the P2 output measured objects and the astrometric calibrations
-from P2 and P1.  These measured objects are matched with the reference
-catalog objects, and calibrated astrometry and photometry is produced
-for the full exposure.  The location of the resulting astometry
+Figure~\ref{phase3} illustrates the tables and commands involved in
+the Phase 3 analysis.  The P3 pending exposure table lists the
+exposure ID, the P3 analysis version, the P2 analysis version to be
+used as input to this P3 analysis, and the recipe to be used.  The
+command {\tt p3search -pending} extracts exposures from this table and
+provides them to PanTasks for processing.  PanTasks launches a Phase 3
+analysis (the command {\tt psastro}?) for each exposure.  In this
+analysis, the P2 exposure and image tables are used, in conjunction
+with the P2 version information, to select the P2 output measured
+objects and the astrometric calibrations from P2 and P1.  These
+measured objects are matched with the reference catalog objects, and
+calibrated astrometry {\em and eventually photometry} is produced for
+the full exposure.  The location of the resulting astometry
 calibration table is stored back in the P3 exposure table.  If the
 recipe file specifies, the 2-D photometric and background / fringe
@@ -636 +888 @@
 above, the `process' stage is a null operation.
 
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pantasks.09.ps}
+\caption{\label{detprocess} Detrend Creation : Process Tasks}
+\end{center}
+\end{figure}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pantasks.10.ps}
+\caption{\label{detresid} Detrend Creation : Residual Tasks}
+\end{center}
+\end{figure}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85]{pics/pantasks.11.ps}
+\caption{\label{detstack} Detrend Creation : Stack and Norm}
+\end{center}
+\end{figure}
+
 \pagebreak
 
@@ -655 +928 @@
 particularly true of the \code{submit.Px} type of commands.  
 
-\note{the command names are for illustration purposes only.  any
-  suggestions for better / fancier names are welcome...}
-
 \begin{verbatim}
 
@@ -690 +960 @@
   also adds an entry to either the P1 exposure table or the P2 image table.
   
+Phase 0 commands:
+
+p0search -pending :
+  * examine the new.imfiles,new.exposures tables and select exposures ready for analysis
+  * output is: (expID) (camera)
+
+p0search -update (expID) (camera):
+  * select a the corresponding images from the new.imfiles/new.exposures table
+  * extract the specified header information
+  * search the summit metadata db tables
+  * write an entry to the raw.imfiles and raw.exposure tables
+  * set the state on the new.imfiles,new.exposure tables
+  * based on the camera config information;
+    * add an entry to the p1.pending table (mosaic) 
+    - or
+    * add an entry to the p2.pending table (single)
+
+p0search -stats (expID) (camera):
+  * select a specified image in the new.imfiles/new.exposures table
+  * extract the specified header information
+  * search the summit metadata db tables
+  * report the image stats
+  [-update without output to MDDB]
+
+p0search -mkraw (expID) (camera):
+  * select a specified image in the new.imfiles/new.exposures table
+  * extract the specified header information
+  * search the summit metadata db tables
+  * write an entry to the raw.imfiles and raw.exposure tables
+  * set the state on the new.imfiles,new.exposure tables
+
+p0search -cleanup:
+  * remove completed entries from the new.imfiles,new.exposure tables
+
+** note : the division of -pending and -update allows separate processes
+   to be examining the image headers and measuring some stats.  these
+   jobs can be run via pcontrol to reduce the load on the PanTasks
+   machines
+
+Phase 1 pipeline tools:
+
+p1search -define [constraints]:
+  * examine the raw.exposures tables and select exposures matching the given criteria
+  * add entries which are allowed (mosaic) to the p1.pending table
+
+p1search -pending :
+  * examine the p1.pending table and select exposures waiting for p1
+  * output: lines consisting of:
+    (expID) (p1version) (camera)
+
+p1search -done :
+  * select completed entries in the p1.pending table
+  * move to the p1.done table
+  * add new entry to the p2.pending tables
+
+Phase 2 pipeline tools:
+
+p2search -quick 
+  * search for images which match in raw.exp,raw.imfiles
+  * output in format which can be used by ppImage pantasks script
+
+p2search -define [options]
+  * input: searches mddb:raw_exposures,raw_images
+  * output: updates mddb:P2_exposures_pending,P2_images_pending
+  * alternative output: identical to p2pending
+ 
+p2search -pending 
+  * input: searches mddb:P2_exposures_pending,P2_images_pending
+  * output: Nlines consisting of:
+    (URL) (expID) (class)
+  * options: ?
+
+p2search -update
+  * examine the imfiles and identify any completed exposures
+
+p2search -done
+  * add completed exposures to the p2.done tables
+  * remove corresponding entries from the p2.pending table
+  * send new entry to the pending p3 table
+
+ppImage file://path/filename file://path/outroot -recipe (recipe) 
+ppImage neb://nebname neb://outroot -recipe (recipe)
+
+restriction options:
+  -time (start) (stop)
+  -camera (camera) 
+  -region (ra,dec) (ra,dec)
+
+
+Phase 3 pipeline tools:
+
+p3search -define :
+  * examine the raw.exposures tables and select exposures matching the given criteria
+  * add entries which are allowed (mosaic) to the p3.pending table
+
+p3search -quick :
+  * examine the raw.exposures tables and select exposures matching the given criteria
+  * return list of entries for p3 processing
+  * output: lines consisting of:
+    (expID) (p3version) (camera)
+
+p3search -pending :
+  * examine the p3.pending exposures table and select exposures waiting for p3
+  * return list of entries for p3 processing
+  * output: lines consisting of:
+    (expID) (p3version) (camera)
+
+p3search -done :
+  * select completed entries in the p3.pending table
+  * move to the p3.done table
+
+
+mkdetrend tools
+
+dettools -define [options]
+ * define a new detRun, specifying the constraints, and adding it to
+ the run table.  if the detRun ID already exists, creates a new
+ version.  the initial state is set to START.  the iteration is set to
+ 0. also creates a new master detrend frame entry
+ (detID.version.iteration define this frame uniquely).
+ * select the input matching a given detRun.  selects the input
+ exposures and the input files.
+
+options : 
+  -ID ID : a free-form string; if not specified, a unique string is constructed
+  -type type : bias dark (mask?) flat fringe (other?)
+  -camera camera 
+  -filter filter
+  -time start stop
+  -exptime min max
+  -airmass min max
+  -expgroup groupID
+
+  (-type and -camera are mandatory)
+  (-filter is mandatory for 'light' types)
+  (-exptime is mandatory for dark)
+
+dettools -pending raw [-state state] [-outmode mode]
+ * select the unprocessed input infiles
+ - output of this program is used by pantasks to schedule the ppImage
+ run on each image
+
+dettools -pending resid [-state state] [-outmode mode]
+ * select the residual images to be processed
+ - output of this program is used by pantasks to schedule the ppImage
+ run on each image
+
+dettools -pending stack [-state state] [-outmode mode]
+ * select the files to be stacked for a given detRun, if exposures are ready
+ - output of this program is used by ppMerge to build a master stack
+
+dettools -pending norm
+ * select the master detrend frames & imfiles to be normalized
+
+dettools -update raw
+ * select the raw input exposures, count processed imfiles, update if done
+
+dettools -update resid
+ * select the resid exposures, count processed resid imfiles, update if done
+ - for a given resid exposure, compile the results from the stacks for
+ each chip and renormalize the output files as needed.
+
+dettools -assess
+ * for a given master detrend frame, compile the results from the
+ residual exposures and assess the validity of the input exposures and of
+ the complete stack.  if too many input images are rejected, the
+ affect the master detrend frame state.  if input images are rejected
+ and a new set of master stacks should be made, create a new master
+ detrend image, incrementing the iteration by one.
+
+** NOTE the sequence is: 
+
+   process, stack, merge, residual, assess
+              ^--------------------------<
+
+   IF we have no relevant detrend image for comparison.  otherwise,
+   the sequence should skip from process to residual on the first
+   pass, with a lower rejection threshold for the first assess pass.
+   the tools above allow either option; it is the choice of state
+   after process that determines which happens next.
+
+
+
+States:
+ input detrend exposures:
+  RAW
+  PROCESSED
+
+ input detrend imfiles:
+  RAW
+  PROCESSED
+  
+ master detrend frames
+  RAW
+  NORMALIZED
+  MKRESID
+  SUCCESS
+  FAILURE
+  RETRY
+  (also has NEW/PRIOR flag)
+
+ master detrend imfiles
+  NEW (not yet created)
+  RAW (created, but not normalized)
+  NORMALIZED
+
+ resid exposure
+  RAW
+  PROCESSED
+  ASSESSED
+
+ resid imfiles:
+  RAW
+  PROCESSED
+
+ master detrend run:
+  NEW
+  DONE
+
+
+
+older descriptions:
+
 Phase.1
   input: exp ID