


This collection of diagrams shows the IPP tasks and the MDDB tables
needed to manage the flow of data through the system.  

---

The first diagram illustrates the recommended interaction between the
metadata database tables and the scheduler internal queues.  Some
table in the metadata database defines a list of data items which are
to be processed by some analysis job.  The scheduler uses a two-step
approach to define the analysis jobs based on this list.  First, one
scheduler task queries the MDDB for a list of pending items, adds the
returned items to an internal scheduler queue.  The process of adding
the elements to the queue is defined so that only unique items are
added: already existing items are skipped.  The entries in the queue
consist of the data items of interest and an internal temporary state.
At first, this would be 'pending'.  A second scheduler tasks pops
'pending' entries one-by-one from this internal queue, submits a job
based on the entry, and sets the temporary state in the internal queue
to 'running'.  The internal state is needed to prevent the scheduler
from re-submitting a job for the same data item before the first job
is done or assessed.  Since the job make take an arbitrary amount of
time, the scheduler requires a mechanism to remember which data items
it has already submitted.  When the job eventually completes, the
metadata database table is updated noting the completion.  This may be
done either by the job itself or by the scheduler.  In addition, the
state of the entry in the queue can be set to either 'done' or the
entry can be simply removed from the queue.

The purpose of this interaction is to maintain the temporary state
information within non-persistent elements of the scheduler rather
than using the metadata database tables to store this information.
This concept has two advantages.  First, the scheduler internal queues
are in memory and relatively small, thus interfacing with them is
quite fast for the scheduler -- this should reduce the system latency.
Second, by keeping this information non-persistent, the system
responds correctly to stopping and restarting the scheduler.  Any jobs
which have not been completed will not be marked in the database, and
will be restarted naturally by the scheduler.  The alternative, of
writing a temporary state marker in the database would require a
restarted scheduler to initially clean all database tables of these
temporary state markers.

The second diagram illustrates this process using the process of
copying the images from the summit as an example.  The metadata
database table of interest in this case is the list of pending images,
with entries supplied by a job which queries the summit data systems.
The job which is actually performed is a remote copy of the image file
from the location specified by the summit data system to the
appropriate location within the IPP Image Server (Nebulous).  (As an
alternative to the above, the 'pending images' table may be part of
the summit database system, and the 'get images' command may query the
summit directly.  In this scenario, the 'copy image' command reports
to the summit data system that an individual image file has been
copied.)

In the rest of this document, the use of psched internal queues to
manage the temporary data states is glossed over and assumed part of
the tasks defined in the process.

---

Summit Copy

This diagram illustrates the MDDB tables used to copy data (images and
metadata tables) from the summit.  The left-hand portion of the
diagram shows the tables involved in copying images from the summit
system.  The table of pending image files lists the URLs of the
individual image files available for transfer, along with their
associated exposure ID and the camera which generated the image.  Two
other entries assist in interpreting the file: the class and the class
ID.  The final entry in this table is the current copy state of the
file, can have the value of 'ready' or 'copied'.

The class defines the data grouping represented by this image file and
may have values of: FPA, Chip, Cell.  This value indicates that the
provided image file represents the specified portion of the camera
FPA.  If the value is FPA, the file represents data from a complete
FPA, though the file may contain pixel data in multiple extensions or
other groupings to be identified later.  If the value is chip, the
file contains only data for a single chip, presumably of multiple
chips available, and equivalently for Cell.  Further discussion of the
FPA image hierarchy is given in the IPP documents (eg, Modules SDRS).
The class ID gives the identifier used to name the class level
corresponding to this file.  This value is necessary to make decisions
on how to copy the data based on the chip / cell before the data is
available to IPP components.  Here are likely values for the class and
class ID for some common cameras:

camera   class  classID
GPC	 chip   chip02
skyprobe fpa	sp01
Megacam  fpa	MegacamSpliced
Suprime	 chip	chip0

and so forth.  The system described is sufficiently flexible to allow
us to transfer the GPC images by cell if we eventually decide that is
more efficient.  

The copy process copies the file from the given URL to the appropriate
IPP node and adds an entry to the table of new image files, consisting
of the same information as the pending image file table, though with a
new value for the URL.  This URL may be an explicit filename, a
reference to an entry in the image server, or a web address, or
located on the image server (marked with file:, neb:, and http:,
respectively).  (TBD: other possible file storage types?  perhaps the
path could be abstracted without going to the level of the image
server?  eg: ref:DIR0001/file0001.fits might be in a directory which
is defined in a table of directories.) After an image file is
successfully copied, the corresponding state in the 'pending chip'
table is updated from 'ready' to 'copied'.

The right hand portion of this diagram illustrates the process of
copying a metadata table.  The table of pending tables lists the URLs
for the tables which are ready, a unique table ID for each table, and
the table type.  The copy function copies the listed table and uploads
the data to the IPP version of the same metadata database.  Two
examples of metadata tables needed by the IPP for the basic image
processing system are illustrated: the table of new exposures and the
table of pending matches.  The first lists the exposures which are
avilable from the summit system, and all represent entries which are
available from the Image server.  the second represents the matches
between exposure IDs and chips

---

Phase 0

This diagram illustrates phase 0, in which the image files are
categorised, examined for summary information and basic statistics,
and moved to the tables used to trigger further analysis.  The process
first examines the 'new image files' table.  It selects images from
this table which have not yet been examined (state is 'new').  The
file header is examined and relevant metadata is extracted (eg, RA,
DEC, times, and so forth to be defined later).  The process may also
select a portion of the image pixel data to determine a rough bias and
background level.  These statistics, whether derived from the header
or the pixel values, are placed along with image summary information
in the 'raw image files' table, and the state field of the 'new image
files' table is set to 'ready'.  

The process is also responsible for moving the exposures to the tables
used for triggering the analysis process.  If the image class is FPA,
the image can be advanced without waiting for any other image files.
If the class is Chip or Cell, the process must also examine the 'new
exposure' table for this exposure ID.  The number of class files
available for this exposure is listed in this table.  The process must
the select all image files matching the exposure ID with state of
'ready' and compare the number avalable to the number expected.  If
the two match, then a new exposure is ready.  Based on the image type
(from the most recently examined image file header or new exp table?),
the exposure is added to the 'raw exposure' table for images of that
type.  The allowed types are 'detrend', (all bias, dark, flat images),
'object', 'focus'(??), etc.  (** The different tables represent
different analysis modes.  This process also adds an entry to the exp
ID / image file match **).  This process also adds all science
(OBJECT) exposures to the P1 exposure table (for mosaic data) or the
P2 chip table (for single detector data).  These tables are used to
trigger the Phase 1 and Phase 2 analysis stages.

---

Phase 1

This diagram shows the tables involved in running the P1 analysis
stage.  There are paths for exposures to enter the analysis
automatically from the P0 analysis (arrow on left) or to be added
manually based on a selection from the raw exposure table.  Exposures
to be analysed by Phase 1 are added to the P1 exposure table with the
state 'new'.  Exposures may be added multiple times for processing and
reprocessing. The P1 exp table keeps a record of the old attempts for
debugging and analysis.  Each time an exposure is added to the P1 exp
table, it is given a new, unique version number, allowing the system
as a whole to track different analysis attempts.  This method is used
in all of the image analysis stages (and extrapolated to iterations in
the detrend analysis steps below).  The top portion of the diagram
represents the user-space tool which may be used to re-submit an
exposure or a group of exposures, potentially selected on the basis of
a query from the raw SCIENCE exposure table.

The P1 exposure table is examined to select the new exposures, these
are then used to generate the P1 analysis jobs.  Within the analysis
job, the chips (image files) associated with the exposure are select
from the raw image file table.  The analysis examines the contents of
these files, either extract the guide star information from the image
files (GS table extension) or searches for and centroids the pixels on
appropriate bright stars.  The analysis results in astrometric
calibration terms which are written to the astrometric calibration
file for this exposure.  The location of the astrometric calibration
file and the statistics of the measurement are written back to the P1
exposure table.  The images associated with exposures which are
successfully processed by P1 are then added to the P2 image table,
which is used to trigger the Phase 2 analysis.

---

Phase 2

This diagram shows the tables involved in running the P2 analysis
stage.  There are paths for images to enter the analysis automatically
from the P1 analysis (arrow on left) or to be added manually based on
a selection from the raw exposure and raw image file tables.  Images
to be analysed by Phase 2 are added to the P2 image table with the
state 'new'.  When images are added to this table, a single entry is
also added to the P2 exposure table listing the P1 and P2 versions for
this exposure.  These version numbers must be integers starting with
1.  If this image did not have a P1 analysis, the P1 version is set to
0.  Exposures may be added multiple times for processing and
reprocessing. The P2 image table keeps a record of the old attempts
for debugging and analysis.  As with P1, each time a collection of
associated images from an exposure is added to the P2 image table, it
is given a new, unique version number, allowing the system as a whole
to track different analysis attempts.  The top portion of the diagram
represents the user-space tool which may be used to re-submit the
images for an exposure or a group of exposures, potentially selected
on the basis of a query from the raw SCIENCE exposure and raw image
file tables.

The P2 image table is examined to select the 'new' images.  These
images are used to generate P2 analysis jobs.  The P2 analysis uses
the input url to find and load the image file.  The url may be a file
on disk, an entry in the image server, Nebulous, etc.  The master
detrend images matching the specific science image and the conditions
are selected by examining the table of master detrend frames.  The
specific detrend image files are selected by using the master detrend
ID to select the matching the entries in the table of master detrend
files.  After the analysis, the output image, mask, and FITS table of
objects, including the astrometry calibration, are written to the P2
image table, along with summary statistics from the P2 analysis.  The
state is also updated (to 'done').  

Whenever the exposure is completed, the value of Ndone in the P2
exposure table is incremented.  If all P2 images matching the P2
exposure version have been completed, the value of Ndone will match
Nclass, and in this case, the process adds an entry to the P3 exposure
table.

---

Phase 3

This diagram illustrates the tables involved in the Phase 3 analysis.
The P3 exposure table lists the exposure ID, the P3 analysis version,
the P2 analysis version to be used as input to this P3 analysis, and
the recipe to be used.  The P2 exposure and image tables are used, in
conjunction with the P2 version information, to select the P2 output
measured objects and the astrometric calibrations from P2 and P1.
These measured objects are matched with the reference catalog objects,
and calibrated astrometry and photometry is produced for the full
exposure.  The location of the resulting astometry calibration table
is stored back in the P3 exposure table.  If the recipe file
specifies, the 2-D photometric and background / fringe corrections may
also be performed at this stage.  Since these analyses require
reference data, the recipe may also be used to skip these analysis if
such reference data is unavailable or unreliable.  At the end of Phase
3, the objects from the exposure are inserted into the photometry
database (this is not shown). 

The astrometric calibration portion of Phase 3 is principally needed
for a mosaic camera.  For single-chip cameras, Phase 3 may be used to
perform the photometric calibration and simply pass the astrometric
results along to the output file to be listed in the P3 exposure
table.  In this way, later stages of the analysis (ie, Phase 4) can
use the P3 exposure table as input for all cameras, even if all the
funcionality of Phase 3 is not necessary for that camera.  This would
be the case for the skyprobe camera, for example.

---

Phase 4

At the end of Phase 3, the images are ready for Phase 4.  The Phase 3
diagram shows the output line adding the exposures to be processed by
Phase 4 to a Phase 4 table.  However, this line is just for
illustration purposes.  The rules for initiating a Phase 4 run are
somewhat more complicated than those for running Phases 1-3.  Groups
of exposures which have an appropriate overlap should be chosen for
the Phase 4 analysis.  In the steady-state period of PS-4, it may be
straightforward to choose the exposure groups: they would simply be
the exposures obtained nearly simultaneously by the four separate
cameras.  The circumstance for PS-1 will be much more complicated (and
even PS-4 will probably be more complex than it seems at first
glance).  For example, in PS-1, we will not have a static sky for most
of the AP Survery.  In this circumstance, we cannot run P4, at least
until after the complete AP Reference catalog is built, and
potentially all exposures re-run through Phase 3.  It may be useful
for the AP Survey data to split the Phase 4 analysis into two stages:
image combination and image differencing.  It may even be the case
that only the combination portion of Phase 4 is performed on the AP
Survey data.  

More generally, the image groups selected for Phase 4 analysis may be
chosen on the basis of a query of the AP Database (DVO) with some
rules.  This may be 

--

Note that each of the stages P1-P4 refer to the processing version
from the previous stage.  This allows the processing stage to request
the correct version of the results from the previous stage, and makes
it possible to run and re-run the analysis at any stage without
deleting the earlier results.  As different analysis attempts are
performed for a given image, the versions branch out, like the diagram
in Figure NNN.  

Also note that at every stage, the entries include a recipe
identifier.  This is used to select the analysis recipe which should
be used for this version.  By default, the recipe should be set to the
current best recipe (use a default name for this?).  This feature
allows the user to run test analyses with variations on the recipe
without altering the analysis system.  For example, it is possible to
use a different flat-field set by specifying alternate rules for the
flat-field selection in a recipe file.  If it is necessary to run the
P1-P3 analysis with the raw master flats, for example, the user simply
defines that selection in the recipe file and submits the images of
interest to P1 (or P2, etc), with the corresponding entry for the
recipe.

The recipe file may also be used to specify alternative analysis paths
and desitinations.  For example, it is not necessary that all analysis
stops with P4: the recipe file may be used to halt the analysis at P2
or P3.  In addition, the recipe file may be used to specify an
alternative destination for the output results.  For example, to
generate the photometric flat-field correction frame from a collection
of dithered images, the user may not want the photometry results in
the main DVO database.  By using the recipe to set an alternative DVO
database target, and by specifying the use of the raw master flat
rather than the corrected one, the analysis of the dithered images is
kept isolated from the other photometry database results.  The
resulting photometry may be used to construct the new, corrected
flat-field images, and the processing of the same images using the new
flat-field images may be sent to the master DVO database.  

---

mkflat

This diagram illustrates the tables needed for the generic detrend
construction process, using the flat-field construction as an example.
This diagram is somewhat more complex than the preceeding versions.
In this diagram, both single jobs and multiple jobs are represented by
the process elements (the blue ellipses).  In some cases, more that
one task will be needed to perform the function illustrated by a
single process task.  The complexity of this diagram is enhanced by
the need for multiple iterations and both single chip and full mosaic
processing.  At the moment, the distinction between mosaic and single
chip cameras is not specifically discussed.  Finally, the triggers
which initiate a specific detrend analysis are glossed over.

The detrend analysis is initiated by choosing a type of detrend image
to be constructed and by specifying the criteria which will be used to
select the input raw detrend frames for the construction.  For
example, these criteria could specify that all twilight flat images
over a certain period of days, perhaps with restrictions on the flux
levels or the time-from-sunset of the images.  The detrend analysis
run is given an ID (det ID) which will also be used to identify the
resulting master detrend frame.  

Given the definition of a master detrend run, the input exposures are
selected from the raw detrend exposure table, and written to the input
detrend exposure table.  In the next step, the corresponding image
files are selected from the table of raw image files.  Since there
will be a different set of input raw images for each attempt at
creating a master detrend image, and since any given attempt may use
some of the same input images as any other attempt, a separate table
of input raw images is constructed.  

Each of the input raw images may be pre-processed before it may be
used to construct the detrend frame.  For example, the input
flat-field images should (probably) be dark- and bias-corrected before
they are stacked.  The information about these input processed images
is written to the input images table.  If no processing is needed,
this step simply copies the appropriate information to the table, and
points back to the raw image, rather than a processed version.  

The input processed images are combined (stacked) to create a master
detrend image for the particular data element defined by the image
class (chip/cell/fpa).  At this stage, not all input images should
necessarily be included in the stack.  If residual statistics have
been measured for the input images (say, using a prior stack), then
some of the input image may be excluded.  The table of residual images
is used to guide this process.  The information describing the
resulting master image is written to the master images table.  

The statistics of the master detrend images must examined so that any
necessary renormalizations may be performed.  For example, after
stacking the individual flat images, the resulting stacks must be
renomalized to account for the different ranges of input image fluxes.
This analysis is least-squares solution in which an appropriate scale
is determined for each input exposure and a separate gain is
determined for each of the chips or cells in the camera.  This
analysis can only performed after all image stacks (ie, for all chips)
have been constructed.  The resulting information is written to the
table of master detrend frames.  

Once the master detrend is constructed, the master detrend images may
be used to construct residual images for each of the input images.
These residual statistics, as well as the locations of the residual
images and other related data products (jpeg thumbnails?) are written
to the residual image table.  Note the red arrow which by-passes the
stack construction and merge steps and skips directly to the residual
analysis.  In some cases, it may be useful to have the input images
confronted with an existing detrend image, and the resulting residual
values used to guide the rest of the process.  For example, in the
flat-field analysis, applying an earlier flat can result in a very
good first-pass rejection of poor input images.  The logic to make
this leap must be part of the scheduler, since each of the individual
blocks represent complete processing jobs.

Finally, the residual statistics from the complete mosaic (all input
images, all chips) are used to assess the quality of the newly
constructed master detrend image, and to potentially modify the
selection of input images.  This latter process is performed by
marking the state of the residual images from this iteration.  The
stacking process always examines the state information for the
residual images from the previous iteration, if it exists, when
constructing the master stack.  Once a master detrend frame has been
judged of high enough quality, the state of the entry for the frame in
the master detrend frames table is set to an appropriate value to tell
the other routines that this image should be used as a master detrend.
The exact choice of which master detrend frame is used for a given
science image depends on the recipe along with information such as the
time period used or the conditions used.

Note that, although this discussion focuses on the construction of
flat-field images, the same structure should be capable of
constructing the biases, dark, fringes, etc.  In some cases, as noted
above, the 'process' stage is a null operation.
