Index: /trunk/doc/design/psched-tasks.txt
===================================================================
--- /trunk/doc/design/psched-tasks.txt	(revision 4895)
+++ /trunk/doc/design/psched-tasks.txt	(revision 4895)
@@ -0,0 +1,430 @@
+
+
+
+This collection of diagrams shows the IPP tasks and the MDDB tables
+needed to manage the flow of data through the system.  
+
+---
+
+The first diagram illustrates the recommended interaction between the
+metadata database tables and the scheduler internal queues.  Some
+table in the metadata database defines a list of data items which are
+to be processed by some analysis job.  The scheduler uses a two-step
+approach to define the analysis jobs based on this list.  First, one
+scheduler task queries the MDDB for a list of pending items, adds the
+returned items to an internal scheduler queue.  The process of adding
+the elements to the queue is defined so that only unique items are
+added: already existing items are skipped.  The entries in the queue
+consist of the data items of interest and an internal temporary state.
+At first, this would be 'pending'.  A second scheduler tasks pops
+'pending' entries one-by-one from this internal queue, submits a job
+based on the entry, and sets the temporary state in the internal queue
+to 'running'.  The internal state is needed to prevent the scheduler
+from re-submitting a job for the same data item before the first job
+is done or assessed.  Since the job make take an arbitrary amount of
+time, the scheduler requires a mechanism to remember which data items
+it has already submitted.  When the job eventually completes, the
+metadata database table is updated noting the completion.  This may be
+done either by the job itself or by the scheduler.  In addition, the
+state of the entry in the queue can be set to either 'done' or the
+entry can be simply removed from the queue.
+
+The purpose of this interaction is to maintain the temporary state
+information within non-persistent elements of the scheduler rather
+than using the metadata database tables to store this information.
+This concept has two advantages.  First, the scheduler internal queues
+are in memory and relatively small, thus interfacing with them is
+quite fast for the scheduler -- this should reduce the system latency.
+Second, by keeping this information non-persistent, the system
+responds correctly to stopping and restarting the scheduler.  Any jobs
+which have not been completed will not be marked in the database, and
+will be restarted naturally by the scheduler.  The alternative, of
+writing a temporary state marker in the database would require a
+restarted scheduler to initially clean all database tables of these
+temporary state markers.
+
+The second diagram illustrates this process using the process of
+copying the images from the summit as an example.  The metadata
+database table of interest in this case is the list of pending images,
+with entries supplied by a job which queries the summit data systems.
+The job which is actually performed is a remote copy of the image file
+from the location specified by the summit data system to the
+appropriate location within the IPP Image Server (Nebulous).  (As an
+alternative to the above, the 'pending images' table may be part of
+the summit database system, and the 'get images' command may query the
+summit directly.  In this scenario, the 'copy image' command reports
+to the summit data system that an individual image file has been
+copied.)
+
+In the rest of this document, the use of psched internal queues to
+manage the temporary data states is glossed over and assumed part of
+the tasks defined in the process.
+
+---
+
+Summit Copy
+
+This diagram illustrates the MDDB tables used to copy data (images and
+metadata tables) from the summit.  The left-hand portion of the
+diagram shows the tables involved in copying images from the summit
+system.  The table of pending image files lists the URLs of the
+individual image files available for transfer, along with their
+associated exposure ID and the camera which generated the image.  Two
+other entries assist in interpreting the file: the class and the class
+ID.  The final entry in this table is the current copy state of the
+file, can have the value of 'ready' or 'copied'.
+
+The class defines the data grouping represented by this image file and
+may have values of: FPA, Chip, Cell.  This value indicates that the
+provided image file represents the specified portion of the camera
+FPA.  If the value is FPA, the file represents data from a complete
+FPA, though the file may contain pixel data in multiple extensions or
+other groupings to be identified later.  If the value is chip, the
+file contains only data for a single chip, presumably of multiple
+chips available, and equivalently for Cell.  Further discussion of the
+FPA image hierarchy is given in the IPP documents (eg, Modules SDRS).
+The class ID gives the identifier used to name the class level
+corresponding to this file.  This value is necessary to make decisions
+on how to copy the data based on the chip / cell before the data is
+available to IPP components.  Here are likely values for the class and
+class ID for some common cameras:
+
+camera   class  classID
+GPC	 chip   chip02
+skyprobe fpa	sp01
+Megacam  fpa	MegacamSpliced
+Suprime	 chip	chip0
+
+and so forth.  The system described is sufficiently flexible to allow
+us to transfer the GPC images by cell if we eventually decide that is
+more efficient.  
+
+The copy process copies the file from the given URL to the appropriate
+IPP node and adds an entry to the table of new image files, consisting
+of the same information as the pending image file table, though with a
+new value for the URL.  This URL may be an explicit filename, a
+reference to an entry in the image server, or a web address, or
+located on the image server (marked with file:, neb:, and http:,
+respectively).  (TBD: other possible file storage types?  perhaps the
+path could be abstracted without going to the level of the image
+server?  eg: ref:DIR0001/file0001.fits might be in a directory which
+is defined in a table of directories.) After an image file is
+successfully copied, the corresponding state in the 'pending chip'
+table is updated from 'ready' to 'copied'.
+
+The right hand portion of this diagram illustrates the process of
+copying a metadata table.  The table of pending tables lists the URLs
+for the tables which are ready, a unique table ID for each table, and
+the table type.  The copy function copies the listed table and uploads
+the data to the IPP version of the same metadata database.  Two
+examples of metadata tables needed by the IPP for the basic image
+processing system are illustrated: the table of new exposures and the
+table of pending matches.  The first lists the exposures which are
+avilable from the summit system, and all represent entries which are
+available from the Image server.  the second represents the matches
+between exposure IDs and chips
+
+---
+
+Phase 0
+
+This diagram illustrates phase 0, in which the image files are
+categorised, examined for summary information and basic statistics,
+and moved to the tables used to trigger further analysis.  The process
+first examines the 'new image files' table.  It selects images from
+this table which have not yet been examined (state is 'new').  The
+file header is examined and relevant metadata is extracted (eg, RA,
+DEC, times, and so forth to be defined later).  The process may also
+select a portion of the image pixel data to determine a rough bias and
+background level.  These statistics, whether derived from the header
+or the pixel values, are placed along with image summary information
+in the 'raw image files' table, and the state field of the 'new image
+files' table is set to 'ready'.  
+
+The process is also responsible for moving the exposures to the tables
+used for triggering the analysis process.  If the image class is FPA,
+the image can be advanced without waiting for any other image files.
+If the class is Chip or Cell, the process must also examine the 'new
+exposure' table for this exposure ID.  The number of class files
+available for this exposure is listed in this table.  The process must
+the select all image files matching the exposure ID with state of
+'ready' and compare the number avalable to the number expected.  If
+the two match, then a new exposure is ready.  Based on the image type
+(from the most recently examined image file header or new exp table?),
+the exposure is added to the 'raw exposure' table for images of that
+type.  The allowed types are 'detrend', (all bias, dark, flat images),
+'object', 'focus'(??), etc.  (** The different tables represent
+different analysis modes.  This process also adds an entry to the exp
+ID / image file match **).  This process also adds all science
+(OBJECT) exposures to the P1 exposure table (for mosaic data) or the
+P2 chip table (for single detector data).  These tables are used to
+trigger the Phase 1 and Phase 2 analysis stages.
+
+---
+
+Phase 1
+
+This diagram shows the tables involved in running the P1 analysis
+stage.  There are paths for exposures to enter the analysis
+automatically from the P0 analysis (arrow on left) or to be added
+manually based on a selection from the raw exposure table.  Exposures
+to be analysed by Phase 1 are added to the P1 exposure table with the
+state 'new'.  Exposures may be added multiple times for processing and
+reprocessing. The P1 exp table keeps a record of the old attempts for
+debugging and analysis.  Each time an exposure is added to the P1 exp
+table, it is given a new, unique version number, allowing the system
+as a whole to track different analysis attempts.  This method is used
+in all of the image analysis stages (and extrapolated to iterations in
+the detrend analysis steps below).  The top portion of the diagram
+represents the user-space tool which may be used to re-submit an
+exposure or a group of exposures, potentially selected on the basis of
+a query from the raw SCIENCE exposure table.
+
+The P1 exposure table is examined to select the new exposures, these
+are then used to generate the P1 analysis jobs.  Within the analysis
+job, the chips (image files) associated with the exposure are select
+from the raw image file table.  The analysis examines the contents of
+these files, either extract the guide star information from the image
+files (GS table extension) or searches for and centroids the pixels on
+appropriate bright stars.  The analysis results in astrometric
+calibration terms which are written to the astrometric calibration
+file for this exposure.  The location of the astrometric calibration
+file and the statistics of the measurement are written back to the P1
+exposure table.  The images associated with exposures which are
+successfully processed by P1 are then added to the P2 image table,
+which is used to trigger the Phase 2 analysis.
+
+---
+
+Phase 2
+
+This diagram shows the tables involved in running the P2 analysis
+stage.  There are paths for images to enter the analysis automatically
+from the P1 analysis (arrow on left) or to be added manually based on
+a selection from the raw exposure and raw image file tables.  Images
+to be analysed by Phase 2 are added to the P2 image table with the
+state 'new'.  When images are added to this table, a single entry is
+also added to the P2 exposure table listing the P1 and P2 versions for
+this exposure.  These version numbers must be integers starting with
+1.  If this image did not have a P1 analysis, the P1 version is set to
+0.  Exposures may be added multiple times for processing and
+reprocessing. The P2 image table keeps a record of the old attempts
+for debugging and analysis.  As with P1, each time a collection of
+associated images from an exposure is added to the P2 image table, it
+is given a new, unique version number, allowing the system as a whole
+to track different analysis attempts.  The top portion of the diagram
+represents the user-space tool which may be used to re-submit the
+images for an exposure or a group of exposures, potentially selected
+on the basis of a query from the raw SCIENCE exposure and raw image
+file tables.
+
+The P2 image table is examined to select the 'new' images.  These
+images are used to generate P2 analysis jobs.  The P2 analysis uses
+the input url to find and load the image file.  The url may be a file
+on disk, an entry in the image server, Nebulous, etc.  The master
+detrend images matching the specific science image and the conditions
+are selected by examining the table of master detrend frames.  The
+specific detrend image files are selected by using the master detrend
+ID to select the matching the entries in the table of master detrend
+files.  After the analysis, the output image, mask, and FITS table of
+objects, including the astrometry calibration, are written to the P2
+image table, along with summary statistics from the P2 analysis.  The
+state is also updated (to 'done').  
+
+Whenever the exposure is completed, the value of Ndone in the P2
+exposure table is incremented.  If all P2 images matching the P2
+exposure version have been completed, the value of Ndone will match
+Nclass, and in this case, the process adds an entry to the P3 exposure
+table.
+
+---
+
+Phase 3
+
+This diagram illustrates the tables involved in the Phase 3 analysis.
+The P3 exposure table lists the exposure ID, the P3 analysis version,
+the P2 analysis version to be used as input to this P3 analysis, and
+the recipe to be used.  The P2 exposure and image tables are used, in
+conjunction with the P2 version information, to select the P2 output
+measured objects and the astrometric calibrations from P2 and P1.
+These measured objects are matched with the reference catalog objects,
+and calibrated astrometry and photometry is produced for the full
+exposure.  The location of the resulting astometry calibration table
+is stored back in the P3 exposure table.  If the recipe file
+specifies, the 2-D photometric and background / fringe corrections may
+also be performed at this stage.  Since these analyses require
+reference data, the recipe may also be used to skip these analysis if
+such reference data is unavailable or unreliable.  At the end of Phase
+3, the objects from the exposure are inserted into the photometry
+database (this is not shown). 
+
+The astrometric calibration portion of Phase 3 is principally needed
+for a mosaic camera.  For single-chip cameras, Phase 3 may be used to
+perform the photometric calibration and simply pass the astrometric
+results along to the output file to be listed in the P3 exposure
+table.  In this way, later stages of the analysis (ie, Phase 4) can
+use the P3 exposure table as input for all cameras, even if all the
+funcionality of Phase 3 is not necessary for that camera.  This would
+be the case for the skyprobe camera, for example.
+
+---
+
+Phase 4
+
+At the end of Phase 3, the images are ready for Phase 4.  The Phase 3
+diagram shows the output line adding the exposures to be processed by
+Phase 4 to a Phase 4 table.  However, this line is just for
+illustration purposes.  The rules for initiating a Phase 4 run are
+somewhat more complicated than those for running Phases 1-3.  Groups
+of exposures which have an appropriate overlap should be chosen for
+the Phase 4 analysis.  In the steady-state period of PS-4, it may be
+straightforward to choose the exposure groups: they would simply be
+the exposures obtained nearly simultaneously by the four separate
+cameras.  The circumstance for PS-1 will be much more complicated (and
+even PS-4 will probably be more complex than it seems at first
+glance).  For example, in PS-1, we will not have a static sky for most
+of the AP Survery.  In this circumstance, we cannot run P4, at least
+until after the complete AP Reference catalog is built, and
+potentially all exposures re-run through Phase 3.  It may be useful
+for the AP Survey data to split the Phase 4 analysis into two stages:
+image combination and image differencing.  It may even be the case
+that only the combination portion of Phase 4 is performed on the AP
+Survey data.  
+
+More generally, the image groups selected for Phase 4 analysis may be
+chosen on the basis of a query of the AP Database (DVO) with some
+rules.  This may be 
+
+--
+
+Note that each of the stages P1-P4 refer to the processing version
+from the previous stage.  This allows the processing stage to request
+the correct version of the results from the previous stage, and makes
+it possible to run and re-run the analysis at any stage without
+deleting the earlier results.  As different analysis attempts are
+performed for a given image, the versions branch out, like the diagram
+in Figure NNN.  
+
+Also note that at every stage, the entries include a recipe
+identifier.  This is used to select the analysis recipe which should
+be used for this version.  By default, the recipe should be set to the
+current best recipe (use a default name for this?).  This feature
+allows the user to run test analyses with variations on the recipe
+without altering the analysis system.  For example, it is possible to
+use a different flat-field set by specifying alternate rules for the
+flat-field selection in a recipe file.  If it is necessary to run the
+P1-P3 analysis with the raw master flats, for example, the user simply
+defines that selection in the recipe file and submits the images of
+interest to P1 (or P2, etc), with the corresponding entry for the
+recipe.
+
+The recipe file may also be used to specify alternative analysis paths
+and desitinations.  For example, it is not necessary that all analysis
+stops with P4: the recipe file may be used to halt the analysis at P2
+or P3.  In addition, the recipe file may be used to specify an
+alternative destination for the output results.  For example, to
+generate the photometric flat-field correction frame from a collection
+of dithered images, the user may not want the photometry results in
+the main DVO database.  By using the recipe to set an alternative DVO
+database target, and by specifying the use of the raw master flat
+rather than the corrected one, the analysis of the dithered images is
+kept isolated from the other photometry database results.  The
+resulting photometry may be used to construct the new, corrected
+flat-field images, and the processing of the same images using the new
+flat-field images may be sent to the master DVO database.  
+
+---
+
+mkflat
+
+This diagram illustrates the tables needed for the generic detrend
+construction process, using the flat-field construction as an example.
+This diagram is somewhat more complex than the preceeding versions.
+In this diagram, both single jobs and multiple jobs are represented by
+the process elements (the blue ellipses).  In some cases, more that
+one task will be needed to perform the function illustrated by a
+single process task.  The complexity of this diagram is enhanced by
+the need for multiple iterations and both single chip and full mosaic
+processing.  At the moment, the distinction between mosaic and single
+chip cameras is not specifically discussed.  Finally, the triggers
+which initiate a specific detrend analysis are glossed over.
+
+The detrend analysis is initiated by choosing a type of detrend image
+to be constructed and by specifying the criteria which will be used to
+select the input raw detrend frames for the construction.  For
+example, these criteria could specify that all twilight flat images
+over a certain period of days, perhaps with restrictions on the flux
+levels or the time-from-sunset of the images.  The detrend analysis
+run is given an ID (det ID) which will also be used to identify the
+resulting master detrend frame.  
+
+Given the definition of a master detrend run, the input exposures are
+selected from the raw detrend exposure table, and written to the input
+detrend exposure table.  In the next step, the corresponding image
+files are selected from the table of raw image files.  Since there
+will be a different set of input raw images for each attempt at
+creating a master detrend image, and since any given attempt may use
+some of the same input images as any other attempt, a separate table
+of input raw images is constructed.  
+
+Each of the input raw images may be pre-processed before it may be
+used to construct the detrend frame.  For example, the input
+flat-field images should (probably) be dark- and bias-corrected before
+they are stacked.  The information about these input processed images
+is written to the input images table.  If no processing is needed,
+this step simply copies the appropriate information to the table, and
+points back to the raw image, rather than a processed version.  
+
+The input processed images are combined (stacked) to create a master
+detrend image for the particular data element defined by the image
+class (chip/cell/fpa).  At this stage, not all input images should
+necessarily be included in the stack.  If residual statistics have
+been measured for the input images (say, using a prior stack), then
+some of the input image may be excluded.  The table of residual images
+is used to guide this process.  The information describing the
+resulting master image is written to the master images table.  
+
+The statistics of the master detrend images must examined so that any
+necessary renormalizations may be performed.  For example, after
+stacking the individual flat images, the resulting stacks must be
+renomalized to account for the different ranges of input image fluxes.
+This analysis is least-squares solution in which an appropriate scale
+is determined for each input exposure and a separate gain is
+determined for each of the chips or cells in the camera.  This
+analysis can only performed after all image stacks (ie, for all chips)
+have been constructed.  The resulting information is written to the
+table of master detrend frames.  
+
+Once the master detrend is constructed, the master detrend images may
+be used to construct residual images for each of the input images.
+These residual statistics, as well as the locations of the residual
+images and other related data products (jpeg thumbnails?) are written
+to the residual image table.  Note the red arrow which by-passes the
+stack construction and merge steps and skips directly to the residual
+analysis.  In some cases, it may be useful to have the input images
+confronted with an existing detrend image, and the resulting residual
+values used to guide the rest of the process.  For example, in the
+flat-field analysis, applying an earlier flat can result in a very
+good first-pass rejection of poor input images.  The logic to make
+this leap must be part of the scheduler, since each of the individual
+blocks represent complete processing jobs.
+
+Finally, the residual statistics from the complete mosaic (all input
+images, all chips) are used to assess the quality of the newly
+constructed master detrend image, and to potentially modify the
+selection of input images.  This latter process is performed by
+marking the state of the residual images from this iteration.  The
+stacking process always examines the state information for the
+residual images from the previous iteration, if it exists, when
+constructing the master stack.  Once a master detrend frame has been
+judged of high enough quality, the state of the entry for the frame in
+the master detrend frames table is set to an appropriate value to tell
+the other routines that this image should be used as a master detrend.
+The exact choice of which master detrend frame is used for a given
+science image depends on the recipe along with information such as the
+time period used or the conditions used.
+
+Note that, although this discussion focuses on the construction of
+flat-field images, the same structure should be capable of
+constructing the biases, dark, fringes, etc.  In some cases, as noted
+above, the 'process' stage is a null operation.