
2007.01.08

  I have updated pantasks to used named tags in named groups to make
  life much easier when changing the schema of the db tables.  The
  function ipptool2book convernts the output from an ippTool query in
  the form of a set of metadata structures (saved in stdout) into a
  book format.

  dettool -tonormalizedstat     DetrendNormStat
  dettool -tonormalize          DetrendNorm
  dettool -tonormalizedexp      DetrendNormExp
  dettool -toprocessedimfile    DetrendProcessImfiles
  dettool -toprocessedexp       DetrendProcessExposures
  dettool -residdetrun          DetrendRejectExp
  dettool -toresidimfile        DetrendResidImfiles
  dettool -toresidexp           DetrendResidExposures
  dettool -tostacked            DetrendStackClass
  p0tool -pendingimfile         Phase0Imfiles
  p0tool -pendingexp            Phase0Exposures
  p2tool -pendingimfile         Phase2Imfiles
  p3tool -pendingexp            Phase3Exposures

2006.08.28

- ipptools need to use expID:camera:telescope as unique key (ie, not expID alone)
- p0tool -pendingimfile -simple needs to report the expKey (ie, not expID)
- 


This directory contains the PanTasks scripts which define the steps
performed by the IPP.  These scripts use the ippTools to decide what
images need to be processed, and to update the Metadata Database with
the current state of the analysis.  The analysis is performed using
IPP programs which are distributed to the cluster computers as needed.

1. The Detrend Creation Scripts

The detrend creation process is among the more complex elements of the
IPP.  There are N major steps which must be performed:

1.1. Define New Detrend Runs

The script 'detrend.mkruns.pro' defines tasks which initiate new
Detrend Runs.  There is a separate task for each camera automatically
processed by the system (mkdetruns.CAMERA).  These tasks are executed
on a regular, but infrequent basis.  The tasks call external helper
programs which actually run the command dettools -define.  The helper
programs are thus responsible for identifying all of the types of
detrend images to be run for a given camera, as well as the correct
set of filters to apply.  These tasks are run locally and only affect
the database.  PanTasks does not track the detrend runs which are
created.  

- we may need to define these within the PanTasks scripts
- how frequently are these tasks run (1/day?)
- if no raw images are selected by the query, no new detrun should be
  created.
- we may need different schedules for the dark (bias/dark) and the
  light (flat/fringe) data.  And yet another schedule for sky data
  (skyflat/skyfringe).

1.2. Process new raw imfiles

The script 'detrend.process.pro' defines tasks which processes the new
raw imfiles for any detrend run.  This script defines two tasks,
'dettool.raw.load' and 'dettool.raw.process'.  The first script
occasionally polls the Metadata Database to select raw imfiles which
have been identified as part of a detrend run.  The identified raw
imfiles are added to an internal queue, DetrendImfilesToProcess.  Any
raw imfile already in the list is ignored.  The imfiles are added to
the queue with a state of NEW.  The second task examines the queue for
imfiles with state NEW and defines a ppImage analysis job for that
imfile.  The job is sent to pcontrol for processing on the distributed
cluster.  At this point, the corresponding queue entry is set to the
state RUN. Upon success, the queue entry is set to DONE and the
database is notified of the successful operation.  The DONE entries
are occasionally pruned from the queue; this must be done in the same
task as the initial query and the update of the list based on the
database results.  

- we need to define the columns in the output of dettool -raw
- we need a method to associate an imfile with a node in the cluster
- we need to perform some appropriate action when the queries fail.
- we need a method to generate the output URIs
- we need a method to identify the correct recipe (function of type?)

1.3. Stack new processed imfiles

The script 'detrend.stack.pro' defines tasks which stack the newly
processed imfiles for any detrend run iteration.  This script defines
two tasks, 'dettool.stack.load and 'dettool.stack.process.  The first
script occasionally polls the Metadata Database to select class IDs
for detrend runs for which the input imfiles have all been processed.
The identified detRun/classID combination is added to an internal
queue, DetrendClassIDtoStack.  The entries are added to the queue with
a state of NEW.  The second task examines the queue for entries with a
state of NEW.  It defines a ppMerge analysis job for that
detID/classID.  The helper script selects the corresponding input
imfiles and passes them to ppMerge.  The job is sent to pcontrol for
processing on the distributed cluster.  At this point, the
corresponding queue entry is set to the state RUN. Upon success, the
queue entry is set to DONE and the database is notified of the
successful operation.  The DONE entries are occasionally pruned from
the queue; this must be done in the same task as the initial query and
the update of the list based on the database results.

- we need to define the columns in the output of dettool -chip -processed
- we need a method to associate a classID with a node in the cluster
- we need to perform some appropriate action when the queries fail.
- we need a method to generate the output URIs (not a function of the imfiles...)
- we need a method to identify the correct recipe (function of type?)


1.4. Normalize stacked masters

The script 'detrend.norm.pro' defines tasks which normalize the newly
processed stacks for any detrend run iteration.  This script defines
two tasks, 'dettool.norm.load and 'dettool.norm.process.  The first
script occasionally polls the Metadata Database to select detrun IDs
for which the classes have all been stacked.  The identified
detRunID/iteration combination is added to an internal queue,
DetrendRunToNormalize.  The entries are added to the queue with a
state of NEW.  The second task examines the queue for entries with a
state of NEW.  It defines a ppNorm analysis job for that
detID/iteration.  The job is sent to pcontrol for processing on the
distributed cluster.  At this point, the corresponding queue entry is
set to the state RUN. Upon success, the queue entry is set to DONE and
the database is notified of the successful operation.  The DONE
entries are occasionally pruned from the queue; this must be done in
the same task as the initial query and the update of the list based on
the database results.

- we need to define the columns in the output of dettool
- we need to perform some appropriate action when the queries fail.
- we need a method to generate the output URIs (not a function of the imfiles...)
- we need a method to identify the correct recipe (function of type?)
- we need to define the ppNorm operation inputs and outputs better


1.5. Generate Resid Imfiles

The script 'detrend.resid.pro' defines tasks which apply the newly
generated detrend image to the corresponding input images to generate
the residual images.  This script defines two tasks,
'dettool.resid.load and 'dettool.resid.process.  The first script
occasionally polls the Metadata Database to select imfiles for which
the master detrend images has been normalized.  The identified imfiles
are added to an internal queue, DetrendResidImfiles.  The entries are
added to the queue with a state of NEW.  The second task examines the
queue for entries with a state of NEW.  It defines a ppImage analysis
job for that imfile.  The job is sent to pcontrol for processing on
the distributed cluster.  At this point, the corresponding queue entry
is set to the state RUN. Upon success, the queue entry is set to DONE
and the database is notified of the successful operation.  The DONE
entries are occasionally pruned from the queue; this must be done in
the same task as the initial query and the update of the list based on
the database results.

- we need to define the columns in the output of dettool
- can we select the imfiles here without apriori knowledge that the
  master has been normalized?
- we need to perform some appropriate action when the queries fail.
- we need a method to generate the output URIs (not a function of the imfiles...)
- we need a method to identify the correct recipe (function of type?)
- we need a method to pass the matching detrend image to ppImage in
  the correct context (ie, a bias needs to be supplied to the -bias element...)

1.6. Assess Detrend Results

The script 'detrend.assess.pro' defines tasks which assess the
residual statistics for completed detrend runs.  This script defines
two tasks, 'dettool.assess.load and 'dettool.assess.process.  The
first script occasionally polls the Metadata Database to select
detrunIDs/iterations for which the resid imfiles have all been
generated.  The identified detruns are added to an internal queue,
DetrendRunToAssess.  The entries are added to the queue with a state
of NEW.  The second task examines the queue for entries with a state
of NEW.  It defines an analysis job for that collection.  The job is
sent to pcontrol for processing on the distributed cluster.  At this
point, the corresponding queue entry is set to the state RUN. Upon
success, the queue entry is set to DONE and the database is notified
of the successful operation.  The DONE entries are occasionally pruned
from the queue; this must be done in the same task as the initial
query and the update of the list based on the database results.

- the assess tool needs to trigger a new run, if needed.
- what is the command to run on the resid statistics?
- we need to define the columns in the output of dettool
- can we select the detrend runs here?
- we need to perform some appropriate action when the queries fail.
- we need a method to generate the output URIs (not a function of the imfiles...)
