Pan-Tasks has three modes.  The first determines what (if anything)
needs to be done, the second does what needs to be done, and the third
cleans up.  Our operations fit broadly into this pattern: get a list
of files to process, process them, and then update the database.
However, we have the choice as to whether to group the processing and
the database update into a single operation, or to allow Pan-Tasks to
handle these separately.  We choose to group them together using Perl
(or other such) scripts, which will allow the tasks to be performed
more easily in a manual fashion.


--------------------------------------------------------------------------------

The "dettool -pending" command returns a list of unassociated
rawDetrendExp (rawDetrendExps that don't correspond to a exp_id in
detInputExp).

	dettool -pending -exp_type <type> -telescope <telescope> -camera <camera>

*** To do: other options, e.g., search by time period?

With a list of exposures in hand, a detrend run may be initiated by listing the
exposure ids:

	dettool -definebyexp -det_type <bias|dark|flat|fringe> -exp_id 1 -exp_id 2 ...

The above two commands ("dettool -pending" and "dettool -definebyexp")
are envisioned as being principally used manually, and not by
Pan-Tasks.


At certain intervals (e.g., at the start of each night), Pan-Tasks
will run the following command to define a new detrend run:

	dettool -definebyquery -det_type <bias|dark|flat|fringe> -camera <camera> -time <time>

"dettool -definebyquery" will define a new detRun semi-automatically
by selecting the component exps with the specified search criteria.

*** To do: other options?


--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -raw

This command returns a list of rawImfiles that are associated with an
detInputExp that do not appear in detProcessedImfiles.  This list
includes the detrend id, exposure id, class id, detrend type (used to
generate the recipe for processing) and the uri.  These rawImfiles are
processed by:

	ppImage -file <imfile_uri> <processed_uri> -recipe PPIMAGE <ppImage_recipe> -stat <temporary_statistics_file>

One processing has been performed, "dettool -addprocessed" adds an
imfile to detProcessedImfiles:

	dettool -addprocessed -det_id <det_id> -exp_id <exp_id> -class_id <class_id> -uri <processed_uri> -recip <ppImage_recipe> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat>

In practise, the above "ppImage" and "dettool -addprocessed" commands
will be performed by a script, "detrend_process.pl".

--------------------------------------------------------------------------------

<Iteration starts here>

--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -tostack

which returns a list of stacks that are ready to be performed.  This
list includes the detrend id, iteration, class id and detrend types
(note that class id is unique per detrend id).  In terms of the
database, the command returns the list where the detrend id, iteration
and class id triplet do not have an entry in detStackedImfile; the
results are masked by entries in detResidExpAnalysis


*** To do: may need to check detResidExp's even on iter 0 - ???

For each stack that is to be performed, we obtain the list of input
imfiles.  "dettool -processed" returns the list of all processed
imfiles from different exposures with the same class_id:

	dettool -processed -det_id <det_id> -iter <iter> -class_id <class_id>

The output includes the exposure id and uri for each of the input
imfiles.  These are used for the stacking:

	ppMerge <stacked_uri> <in_uri> <in_uri> <in_uri> .... -recipe PPMERGE <ppMerge_recipe> -stat STAT_FILE

ppMerge returns statistics of the stack in the STAT_FILE.

Once the stack has been completed, "dettool -addstacked" adds a
stacked class_id and and optionally marks it for normalization:

	dettool -addstacked [-pleasenormalize] -det_id <det_id> -iter <iter> -class_id <class_id> -recip <recipe> -uri <stacked_uri> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> 

The "-pleasenormalize" flag indicates that the normalisation steps are
to be performed (in the case of flat-fields); if this flag is missing,
then the normalisation steps are to be skipped and the stacked imfiles
are inserted in table of normalised imfiles.

In practise, the above three commands ("dettool -processed
-wholestack", "ppMerge" and "dettool -addstacked") will be performed
by a Perl script, "detrend_stack.pl".


--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -tonormalize

This returns a list of detrend id and iteration for which the entire
set of class ids have been processed.

For each of these detrend ids, we get the list of processed imfiles:

	dettool -processed -unmask -det_id <det_id> -iter <iter>

This returns the complete list of processed imfiles for the specified
det_id, providing exposure id, class id, statistic, which we use in
the normalisation:

	ppNormCalc INPUT.mdc

where INPUT.mdc is a metadata config file with the stats for each
imfile.  ppNormCalc outputs to stdout a metadata config format
with the normalisation for each class id.

*** To do: Get rms of solution (for both gain and exposure level) out
	   of ppNormCalc.

Then for each imfile that comprises the stack, we push the normalisations
into the database:

	dettool -addnormstat -det_id <det_id> -iter <iter> -class_id <class_id> -norm <normalisation> 

In practise, the above three commands ("dettool -processed -unmask",
"ppNorm" and "dettool -addnormstat") will be performed by a Perl
script, "detrend_calc_norm.pl".

*** To do: possible upgrade path is to have "dettool -addnormstat"
	   parse a metadata config file from ppNorm to get the
	   multiple normalisations, instead of running it several
	   times (one for each class id).


--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -normstat

which returns a list of imfiles (with corresponding detrend id,
iteration, class id, uri, statistic and normalisation) from
detNormStats, which have no corresponding class_id inserted into
detNormalizedImfile.

For each imfile, we apply the normalisation:

	ppNormApply -file <in_uri> <out_uri> -norm <normalisation>

The normalised imfile is then inserted into detNormalizedImfile:

	dettool -addnormalizedimfile -det_id <det_id> -iter <iter> -class_id <class_id> -uri <out_uri>

In practise, the above two commands ("ppNormApply" and "dettool
-addnormalizedimfile") will be performed by a Perl script,
"detrend_apply_norm.pl".


--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -toresid

which returns a list of processed imfiles (with corresponding detrend
id, iteration, class id, uri, type) for which the appropriate stacked
imfile has been normalised, and which have not been masked out by
detResidImfileAnalysis; it also returns stackedImfiles that have
"-pleasenormalize" set to false.

For each of these imfiles, we generate the residual image, and binned images:

	ppImage -file <processed_uri> <residual_uri> -recipe PPIMAGE <ppImage_residual_recipe> -stat <temporary_statistics_file>

The residual is then inserted into the database:

	dettool -addresidimfile -det_id <det_id> -exp_id <exp_id> -class_id <class_id> -recip <recipe> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> -b1_uri <b1_fits> -b2_uri <b2_fits> 

In practise, the above two commands ("ppImage" and "dettool
-addresidimfile") will be performed by a Perl script,
"detrend_create_resid.pl".

--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -toresidexp

which returns a list of exposures for which all component class ids
have had residuals created.  This list includes the detrend id,
iteration, exposure id, detrend type, and whether the exposure was
included in the stack for this iteration.

We then obtain the list of components (with corresponding statistic)
for that exposure:

	dettool -residimfile -det_id <det_id> -iter <iter> -exp_id <exp_id>

For these, we create the mosaic jpeg images:

	ppImage -file <b1_fits> <something> -recipe PPIMAGE <some_jpeg_recipe>

*** To do: Mode for ppImage (or new program) just to generate a mosaic
           jpeg.

We also use the statistics from each of the components to decide if
the exposure is behaving, and we generate summary statistics for that
exposure.  If there are any problems with any of the components, we
reject the entire exposure (not just the component chip that's causing
problems; these could be removed by other means):

	dettool -addresidexp -det_id <det_id> -iter <iter> -exp_id <exp_id> -recip <recipe> -b1_uri <uri_jpeg1> -b2_uri <uri_jpeg2> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> [-reject]

where the "-reject" flag indicates that the exposure is not to be
included in the stack for the next iteration.  If "-reject" is not
specified, it sets the "accept" boolean to true.

In practise, the above three commands ("dettool -residimfile",
"ppImage" and "dettool -addresidexp") and the decision about whether to
reject exposures would be performed by a Perl script, detrend_reject_imfile.pl"

--------------------------------------------------------------------------------

Pan-Tasks regularly runs:

	dettool -residdetrun

which returns a list of detrend runs (with detrend id, iteration and
detrend type) which have completed all residexps.

For each of these detrend runs, we get the list of all residexps
(specified by their exposure id, along with the summary statistics and
whether they were included in the stack for this iteration):

	dettool -residexp -det_id <det_id> -iter <iter>

For this list of exposures, we inspect the statistics for the group as
a whole, decide which exposures are to be rejected, and generate a
summary statistic for the detrend iteration.  For each exposure, we
run:

	dettool -updateresidexp -det_id <det_id> -iter <iter> -exp_id <exp_id> [-reject]

where the "-reject" flag indicates that the exposure is not to be
included in the stack for the next iteration.  If "-reject" is not
specified, it sets the "accept" boolean to true.  Then for the current
iteration as a whole we can run:

	dettool -adddetrunsummary -det_id <det_id> -iter <iter> -bg <stat> -bg_stdev <stat> -bg_mean_stdev <stat> [-reject]

Which allows per detrend run iteration summary statics to be registered and
can flag the iteration's results as being acceptable for use as a master.

Once this has been done, a decision is made whether to make another
iteration:

	dettool -updatedetrun -det_id <det_id> -iter <iter> [-again|-stop]

where "-rerun" causes a new iteration to be started and "-stop" causes all
iteration to cease (and potentially alerts the human overlords).

In practise, the above three commands ("dettool -residexp", "detool
-updateresid" and "dettool -updatedetrun"), as well as the decisions
about what to reject, and whether to rerun, accept or stop will be
made by a Perl script, "detrend_reject_exp.pl".

--------------------------------------------------------------------------------

The detrend run may also be restarted manually by specifying exposures
to accept and reject:

	dettool -rerun -det_id <det_id> -iter <iter> -accept <exp_id> -accept <exp_id> ... -reject <exp_id> -reject <exp_id> ...


