Pan-Tasks has three modes.  The first determines what (if anything)
needs to be done, the second does what needs to be done, and the third
cleans up.  Our operations fit broadly into this pattern: get a list
of files to process, process them, and then update the database.
However, we have the choice as to whether to group the processing and
the database update into a single operation, or to allow Pan-Tasks to
handle these separately.  We choose to group them together using Perl
(or other such) scripts, which will allow the tasks to be performed
more easily in a manual fashion.


--------------------------------------------------------------------------------

The "dettool -pending" command returns a list of unassociated
rawDetrendExp (rawDetrendExps that don't correspond to a exp_tag in
detInputExp).

	dettool -pending -exp_type <type> -telescope <telescope> -camera <camera>

*** To do: other options, e.g., search by time period?

With a list of exposures in hand, a detrend run may be initiated by listing the
exposure ids:

	dettool -definebyexp -det_type <bias|dark|flat|fringe> -exp_tag 1 -exp_tag 2 ...

The above two commands ("dettool -pending" and "dettool -definebyexp")
are envisioned as being principally used manually, and not by
Pan-Tasks.


At certain intervals (e.g., at the start of each night), Pan-Tasks
will run the following command to define a new detrend run:

	dettool -definebyquery -det_type <bias|dark|flat|fringe> -camera <camera> -time <time>

"dettool -definebyquery" will define a new detRun semi-automatically
by selecting the component exps with the specified search criteria.

*** To do: other options?


--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -toprocess

This command returns a list of rawImfiles that are associated with an
detInputExp that do not appear in detProcessedImfiles.  This list
includes the detrend id, exposure id, class id, detrend type (used to
generate the recipe for processing) and the uri.  These rawImfiles are
processed by:

	ppImage -file <imfile_uri> <processed_uri> -recipe PPIMAGE <ppImage_recipe> -stat <temporary_statistics_file>

In addition to removing the nominated instrumental signatures, this
also generates binned images to create a JPEG in the next step.  Once
processing has been performed, "dettool -addprocessedimfile" adds an
imfile to detProcessedImfiles:

	dettool -addprocessedimfile -det_id <det_id> -exp_tag <exp_tag> -class_id <class_id> -uri <processed_uri> -recip <ppImage_recipe> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> -b1_uri <binned1_uri> -b2_uri <binned2_uri>

In practise, the above "ppImage" and "dettool -addprocessedimfile"
commands will be performed by a script, "detrend_process_imfile.pl".

--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -toprocessedexp

This command returns a list of exposures that have had all imfiles
processed.  This list includes the exposure id and detrend id.  Using
this exposure id, a list of imfiles may be gathered using:

	dettool -processedimfile -exp_tag <exp_tag>

The output includes class id, b1_uri, b2_uri, bg, bg_stdev and
bg_mean_stdev for the processed imfiles.  These are combined into a
JPEG by:

	ppImage -list <file_list> <output_uri> -recipe PPIMAGE PPIMAGE_J

The resultant image, along with summary statistics (calculated from
the imfile statistics) is added into the database for the exposure:

	dettool -addprocessedexp -det_id <det_id> -iteration <iteration> -exp_tag
<exp_tag> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> -b1_uri <jpeg_uri>
-b2_uri <jpeg_uri>

In practise, the above "dettool -processedimfile", "ppImage" and "dettool
-addprocessedexp" commands will be performed by a script,
"detrend_process_exp.pl".

--------------------------------------------------------------------------------

<Iteration starts here>

--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -tostack

which returns a list of stacks that are ready to be performed.  This
list includes the detrend id, iteration, class id and detrend types
(note that class id is unique per detrend id).  In terms of the
database, the command returns the list where the detrend id, iteration
and class id triplet do not have an entry in detStackedImfile; the
results are masked by entries in detResidExpAnalysis


*** To do: may need to check detResidExp's even on iter 0 - ???

For each stack that is to be performed, we obtain the list of input
imfiles.  "dettool -processedimfile" returns the list of all processed
imfiles from different exposures with the same class_id:

	dettool -processedimfile -det_id <det_id> -class_id <class_id>

The output includes the exposure id and uri for each of the input
imfiles.  These are used for the stacking:

	ppMerge <stacked_uri> <in_uri> <in_uri> <in_uri> .... -recipe PPMERGE <ppMerge_recipe> -stat STAT_FILE

ppMerge returns statistics of the stack in the STAT_FILE.

Once the stack has been completed, "dettool -addstacked" adds a
stacked class_id:

	dettool -addstacked -det_id <det_id> -iter <iter> -class_id <class_id> -recip <recipe> -uri <stacked_uri> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat>

In practise, the above three commands ("dettool -processedimfile", "ppMerge"
and "dettool -addstacked") will be performed by a Perl script,
"detrend_stack.pl".


--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -tonormalizedstat

This returns a list of detrend id, iteration and detrend type for
which the entire set of class ids have been processed.

For each of these detrend ids, we get the list of processed imfiles:

	dettool -processedimfile -det_id <det_id>

This returns the complete list of processed imfiles for the specified
det_id, providing exposure id, class id, statistic which we use in the
normalisation:

	ppNormCalc INPUT.mdc

where INPUT.mdc is a metadata config file with the stats for each
imfile.  ppNormCalc outputs to stdout a metadata config format
with the normalisation for each class id.

*** To do: Get rms of solution (for both gain and exposure level) out
	   of ppNormCalc.

Then for each imfile that comprises the stack, we push the normalisations
into the database:

	dettool -addnormalizedstat -det_id <det_id> -iter <iter> -class_id <class_id> -norm <normalisation> 

In practise, the above three commands ("dettool -processedimfile",
"ppNormCalc" and "dettool -addnormalizedstat") will be performed by a Perl
script, "detrend_norm_calc.pl".

If the detrend type is not suitable for normalisation, then
normalisation values of unity will be generated without solving the
matrix with ppNormCalc.

*** To do: possible upgrade path is to have "dettool -addnormalizedstat"
	   parse a metadata config file from ppNorm to get the
	   multiple normalisations, instead of running it several
	   times (one for each class id).


--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -tonormalize

which returns a list of imfiles (with corresponding detrend id,
iteration, class id, uri, statistic and normalisation) from
detNormStats, which have no corresponding class_id inserted into
detNormalizedImfile.

For each imfile, we apply the normalisation, generate binned images
and statistics:

	ppImage -file <in_uri> <out_uri> -norm <normalisation> -stat <statictics_file>

The normalised imfile (along with binned versions) is then inserted
into detNormalizedImfile:

	dettool -addnormalizedimfile -det_id <det_id> -iter <iter> -class_id <class_id> -uri <out_uri> -b1_uri <b1_uri> -b2_uri <b2_uri> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat>

In practise, the above two commands ("ppImage" and "dettool
-addnormalizedimfile") will be performed by a Perl script,
"detrend_norm_apply.pl".


--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -tonormalizedexp

which returns a list of stack exposures which have had normalised
imfile summaries added.  This includes the detrend id and iteration.

For each exposure, we get the list of component imfiles:

	dettool -normalizedimfile -det_id <det_id> -iteration <iter>

This includes the class ids, statistics (bg, bg_stdev, bg_mean_stdev)
and the URIs for the binned images.  From these, we generate summary
statistics for the exposure, and produce a JPEG:

	ppImage -list <input_list> <out_uri> -recipe PPIMAGE PPIMAGE_J

The statistics and JPEG uri are pushed into the database:

	dettool -addnormalizedexp -det_id <det_id> -iteration <iteration> -b1_uri <jpeg1_uri> -b2_uri <jpeg2_uri> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat>

In practise, the above three commands ("dettool -stackimfile",
"ppImage" and "dettool -addnormalizedexp") will be performed by a Perl
script, "detrend_norm_exp.pl".

--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -toresid

which returns a list of processed imfiles (with corresponding detrend
id, iteration, class id, uri, type) for which the appropriate stacked
imfile has been normalised, and which have not been masked out by
detResidImfileAnalysis; it also returns stackedImfiles that have
"-pleasenormalize" set to false.

For each of these imfiles, we generate the residual image, and binned images:

	ppImage -file <processed_uri> <residual_uri> -recipe PPIMAGE <ppImage_residual_recipe> -stat <temporary_statistics_file>

The residual is then inserted into the database:

	dettool -addresidimfile -det_id <det_id> -exp_tag <exp_tag> -class_id <class_id> -recip <recipe> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> -b1_uri <b1_fits> -b2_uri <b2_fits> 

In practise, the above two commands ("ppImage" and "dettool
-addresidimfile") will be performed by a Perl script,
"detrend_resid.pl".

XXX Note that this block is not currently dep. on the results of the
normlaized exp block above so they can run async with respect to each other.

--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -toresidexp

which returns a list of exposures for which all component class ids
have had residuals created.  This list includes the detrend id,
iteration, exposure id, detrend type, and whether the exposure was
included in the stack for this iteration.

We then obtain the list of components (with corresponding statistic)
for that exposure:

	dettool -residimfile -det_id <det_id> -iter <iter> -exp_tag <exp_tag>

For these, we create the mosaic jpeg images:

	ppImage -file <b1_fits> <something> -recipe PPIMAGE PPIMAGE_J
	ppImage -file <b2_fits> <something> -recipe PPIMAGE PPIMAGE_J

We also use the statistics from each of the components to decide if
the exposure is behaving, and we generate summary statistics for that
exposure.  If there are any problems with any of the components, we
reject the entire exposure (not just the component chip that's causing
problems; these could be removed by other means):

	dettool -addresidexp -det_id <det_id> -iter <iter> -exp_tag <exp_tag> -recip <recipe> -b1_uri <uri_jpeg1> -b2_uri <uri_jpeg2> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> [-reject]

where the "-reject" flag indicates that the exposure is not to be
included in the stack for the next iteration.  If "-reject" is not
specified, it sets the "accept" boolean to true.

In practise, the above three commands ("dettool -residimfile",
"ppImage" and "dettool -addresidexp") and the decision about whether to
reject exposures would be performed by a Perl script, detrend_reject_imfile.pl"

--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -toresiddetrun

which returns a list of detrend runs (with detrend id, iteration and
detrend type) which have completed all residexps.

For each of these detrend runs, we get the list of all residexps
(specified by their exposure id, along with the summary statistics and
whether they were included in the stack for this iteration):

	dettool -residexp -det_id <det_id> -iter <iter>

For this list of exposures, we inspect the statistics for the group as
a whole, decide which exposures are to be rejected, and generate a
summary statistic for the detrend iteration.  For each exposure, we
run:

	dettool -updateresidexp -det_id <det_id> -iter <iter> -exp_tag <exp_tag> [-reject]

where the "-reject" flag indicates that the exposure is not to be
included in the stack for the next iteration.  If "-reject" is not
specified, it sets the "accept" boolean to true.  Then for the current
iteration as a whole we can run:

	dettool -adddetrunsummary -det_id <det_id> -iter <iter> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> [-accept]

Which allows per detrend run iteration summary statics to be
registered and can flag the iteration's results as being acceptable
for use as a master (if -accept is indicated).

Once this has been done, a decision is made whether to make another
iteration:

	dettool -updatedetrun -det_id <det_id> -iter <iter> [-again|-state <state>] 

where "-rerun" causes a new iteration to be started and "-state" sets the
detrun into other of these states: "stop", "run", "reg".  The "stop" states
causes all iteration to cease (and potentially alerts the human overlords).

In practise, the above three commands ("dettool -residexp", "detool
-updateresid" and "dettool -updatedetrun"), as well as the decisions
about what to reject, and whether to rerun, accept or stop will be
made by a Perl script, "detrend_reject_exp.pl".

--------------------------------------------------------------------------------

The detrend run may also be restarted manually by specifying exposures
to accept and reject:

	dettool -rerun -det_id <det_id> -iter <iter> -accept <exp_tag> -accept <exp_tag> ... -reject <exp_tag> -reject <exp_tag> ...

--------------------------------------------------------------------------------

Registering pre-created detrends

    dettool -register_detrend ...

Which creates a new detrend run set to the state of "reg".

    detool -register_detrend_imfile -det_id <det_id> ...

registers a new imfile into the detRun.  Note that this operation is a
little dangerous as the validity of the det_id run isn't checked.

    detool -updatedetrun -state stop

Sets the dettool run's state to "stop" so that it will become visiable
to detselect.


--------------------------------------------------------------------------------

Detrend Correction analysis (for flat-field corrections)

To build a correction, we need to initial a p2/p3 analysis on a
collection of images, supplying information so the analysis tools know
to use the appropriate recipe (ie, which specifies that, for
flat-field analysis, ppImage should use the 'RAW' version of a flat).  

possible p2tool lines:

  p2tool -queuerawexp [constraints] -set_label DVOCORR

  p2tool -queuerawexp [constraints] -recipe DVOCORR -set_label DVOCORR.megacam.0001

* should we tie the output DVO database to the recipe?
* should we require the output DVO database as a db value?

We need to wait until all of the selected images have been processed
before we trigger the DVO analysis.  this could be done by blocking
until all exposures with the specified label have successfully (or
perhaps unsuccessfully) finished p3.

* do we need to define a table which track these dvo_correction
  'runs'?

--------------------------------------------------------------------------------

Detrend Correction application (for flat-field corrections)

Assuming there is already a flat-field correction in the db for a
given camera and filter, we need to (optionally) apply the correction
to flat-field images as they are built by the detrend creation system.

The first step is to select a flat-field image which requires a
flat-field correction:

  dettool -tocorrectrun -det_type skyflat.raw

returns a list of detrun ids and iterations which have valid masters
and for which there are no children.

  detselect -select -det_id det_id -iteration iter 

returns a list of imfiles to be corrected

  dettool -register_detrend -parent det_id

  dettool -register_detrend_imfile -parent det_id etc

inserts the completed imfiles

