Index: /trunk/doc/ipptools/dettool.txt
===================================================================
--- /trunk/doc/ipptools/dettool.txt	(revision 8276)
+++ /trunk/doc/ipptools/dettool.txt	(revision 8276)
@@ -0,0 +1,287 @@
+Pan-Tasks has three modes.  The first determines what (if anything)
+needs to be done, the second does what needs to be done, and the third
+cleans up.  Our operations fit broadly into this pattern: get a list
+of files to process, process them, and then update the database.
+However, we have the choice as to whether to group the processing and
+the database update into a single operation, or to allow Pan-Tasks to
+handle these separately.  We choose to group them together using Perl
+(or other such) scripts, which will allow the tasks to be performed
+more easily in a manual fashion.
+
+
+--------------------------------------------------------------------------------
+
+The "dettool -pending" command returns a list of unassociated
+rawDetrendExp (rawDetrendExps that don't correspond to a exp_id in
+detInputExp).
+
+	dettool -pending -exp_type <type> -telescope <telescope> -camera <camera>
+
+*** To do: other options, e.g., search by time period?
+
+With a list of exposures in hand, a detrend run may be initiated by listing the
+exposure ids:
+
+	dettool -definebyexp -det_type <bias|dark|flat|fringe> -exp_id 1 -exp_id 2 ...
+
+The above two commands ("dettool -pending" and "dettool -definebyexp")
+are envisioned as being principally used manually, and not by
+Pan-Tasks.
+
+
+At certain intervals (e.g., at the start of each night), Pan-Tasks
+will run the following command to define a new detrend run:
+
+	dettool -definebyquery -det_type <bias|dark|flat|fringe> -camera <camera> -time <time>
+
+"dettool -definebyquery" will define a new detRun semi-automatically
+by selecting the component exps with the specified search criteria.
+
+*** To do: other options?
+
+
+--------------------------------------------------------------------------------
+
+Pan-Tasks regularly runs:
+
+	dettool -raw
+
+This command returns a list of rawImfiles that are associated with an
+detInputExp that do not appear in detProcessedImfiles.  This list
+includes the detrend id, iteration, exposure id, class id, detrend
+type (used to generate the recipe for processing) and the uri.  These
+rawImfiles are processed by:
+
+	ppImage -file <imfile_uri> <processed_uri> -recipe PPIMAGE <ppImage_recipe>
+
+*** To do: ppImage needs to output statistics
+
+One processing has been performed, "dettool -addprocessed" adds an
+imfile to detProcessedImfiles:
+
+	dettool -addprocessed -det_id <det_id> -iter <iter> -exp_id <exp_id> -class_id <class_id> -uri <processed_uri> -recipe <ppImage_recipe> -stats <stats>
+
+In practise, the above "ppImage" and "dettool -addprocessed" commands
+will be performed by a script, "detrend_process.pl".
+
+--------------------------------------------------------------------------------
+
+Pan-Tasks regularly runs:
+
+	dettool -tostack
+
+which returns a list of stacks that are ready to be performed.  This
+list includes the detrend id, iteration, class id and detrend types
+(note that class id is unique per detrend id).  In terms of the
+database, the command returns the list where the detrend id, iteration
+and class id triplet do not have an entry in detStackedImfile; the
+results are masked by entries in detResidExpAnalysis
+
+
+*** To do: may need to check detResidExp's even on iter 0 - ???
+
+For each stack that is to be performed, we obtain the list of input
+imfiles.  "dettool -processed -wholestack" returns the list of all
+processed imfiles from different exposures with the same class_id:
+
+	dettool -processed -wholestack -det_id <det_id> -iter <iter> -class_id <class_id>
+
+The output includes the exposure id and uri for each of the input
+imfiles.  These are used for the stacking:
+
+	ppMerge <stacked_uri> <in_uri> <in_uri> <in_uri> .... -recipe PPMERGE <ppMerge_recipe>
+
+Once the stack has been completed, "dettool -addstacked" adds a
+stacked class_id and and optionally marks it for normalization:
+
+	dettool -addstacked [-pleasenormalize] -det_id <det_id> -iter <iter> -class_id <class_id> -uri <stacked_uri>
+
+The "-pleasenormalize" flag indicates that the normalisation steps are
+to be performed (in the case of flat-fields); if this flag is missing,
+then the normalisation steps are to be skipped and the stacked imfiles
+are inserted in table of normalised imfiles.
+
+In practise, the above three commands ("dettool -processed
+-wholestack", "ppMerge" and "dettool -addstacked") will be performed
+by a Perl script, "detrend_stack.pl".
+
+
+--------------------------------------------------------------------------------
+
+Pan-Tasks regularly runs:
+
+	dettool -tonormalize
+
+This returns a list of detrend id and iteration for which the entire
+set of class ids have been processed.
+
+For each of these detrend ids, we get the list of processed imfiles:
+
+	dettool -processed -unmask -det_id <det_id> -iter <iter>
+
+This returns the complete list of processed imfiles for the specified
+det_id, providing exposure id, class id, statistic, which we use in
+the normalisation:
+
+	ppNormCalc <some parameters>
+
+*** To do: decide input form for ppNorm (likely a metdata config file
+           containing the stats for each imfile).
+
+*** To do: decide output form for ppNorm (likely a metadata config
+	   file containing the normalisation for each class id of the
+	   stack).
+
+Then for each imfile that comprises the stack, we push the normalisations
+into the database:
+
+	dettool -addnormstat -det_id <det_id> -iter <iter> -class_id <class_id> -norm <normalisation> 
+
+In practise, the above three commands ("dettool -processed -unmask",
+"ppNorm" and "dettool -addnormstat") will be performed by a Perl
+script, "detrend_calc_norm.pl".
+
+*** To do: possible upgrade path is to have "dettool -addnormstat"
+	   parse a metadata config file from ppNorm to get the
+	   multiple normalisations, instead of running it several
+	   times (one for each class id).
+
+
+--------------------------------------------------------------------------------
+
+Pan-Tasks regularly runs:
+
+	dettool -normstat
+
+which returns a list of imfiles (with corresponding detrend id,
+iteration, class id, uri and normalisation) from detNormStats, which
+have no corresponding class_id inserted into detNormalizedImfile.
+
+For each imfile, we apply the normalisation:
+
+	ppNormApply -file <in_uri> <out_uri> -norm <normalisation>
+
+The normalised imfile is then inserted into detNormalizedImfile:
+
+	dettool -addnormalized -det_id <det_id> -iter <iter> -class_id <class_id> -uri <out_uri>
+
+In practise, the above two commands ("ppNormApply" and "dettool
+-addnormalized") will be performed by a Perl script,
+"detrend_apply_norm.pl".
+
+
+--------------------------------------------------------------------------------
+
+Pan-Tasks regularly runs:
+
+	dettool -toresid
+
+which returns a list of processed imfiles (with corresponding detrend
+id, iteration, class id, uri, type) for which the appropriate stacked
+imfile has been normalised, and which have not been masked out by
+detResidImfileAnalysis; it also returns stackedImfiles that have
+"-pleasenormalize" set to false.
+
+For each of these imfiles, we generate the residual image, and binned images:
+
+	ppImage -file <processed_uri> <residual_uri> -recipe PPIMAGE <ppImage_residual_recipe>
+
+*** To do: ppImage needs to return a statistic so we can judge how
+           good the residual is.
+
+The residual is then inserted into the database:
+
+	dettool -addresidimfile -det_id <det_id> -exp_id <exp_id> -class_id <class_id> -stat <stat> -b1 <b1_fits> -b2 <b2_fits>
+
+In practise, the above two commands ("ppImage" and "dettool
+-addresidimfile") will be performed by a Perl script,
+"detrend_create_resid.pl".
+
+--------------------------------------------------------------------------------
+
+Pan-Tasks regularly runs:
+
+	dettool -toresidexp
+
+which returns a list of exposures for which all component class ids
+have had residuals created.  This list includes the detrend id,
+iteration, exposure id, detrend type, and whether the exposure was
+included in the stack for this iteration.
+
+We then obtain the list of components (with corresponding statistic)
+for that exposure:
+
+	dettool -residimfile -det_id <det_id> -iter <iter> -exp_id <exp_id>
+
+For these, we create the mosaic jpeg images:
+
+	ppImage -file <b1_fits> <something> -recipe PPIMAGE <some_jpeg_recipe>
+
+*** To do: Mode for ppImage (or new program) just to generate a mosaic
+           jpeg.
+
+We also use the statistics from each of the components to decide if
+the exposure is behaving, and we generate summary statistics for that
+exposure.  If there are any problems with any of the components, we
+reject the entire exposure (not just the component chip that's causing
+problems; these could be removed by other means):
+
+	dettool -addresidexp -det_id <det_id> -iter <iter> -exp_id <exp_id> -jpeg1 <uri_jpeg1> -jpeg2 <uri_jpeg2> -stats <stats> [-reject]
+
+where the "-reject" flag indicates that the exposure is not to be
+included in the stack for the next iteration.  If "-reject" is not
+specified, it sets the "accept" boolean to true.
+
+In practise, the above three commands ("dettool -residimfile",
+"ppImage" and "dettool -addresidexp") and the decision about whether to
+reject exposures would be performed by a Perl script, detrend_reject_imfiles.pl"
+
+--------------------------------------------------------------------------------
+
+Pan-Tasks regularly runs:
+
+	dettool -residdetrun
+
+which returns a list of detrend runs (with detrend id, iteration and
+detrend type) which have completed all residexps.
+
+For each of these detrend runs, we get the list of all residexps
+(specified by their exposure id, along with the summary statistics and
+whether they were included in the stack for this iteration):
+
+	dettool -residexp -det_id <det_id> -iter <iter>
+
+For this list of exposures, we inspect the statistics for the group as
+a whole, decide which exposures are to be rejected, and generate a
+summary statistic for the detrend iteration.  For each exposure, we
+run:
+
+	dettool -updateresid -det_id <det_id> -iter <iter> -exp_id <exp_id> [-reject]
+
+where the "-reject" flag indicates that the exposure is not to be
+included in the stack for the next iteration.  If "-reject" is not
+specified, it sets the "accept" boolean to true.
+
+Once this has been done, a decision is made whether to make another
+iteration:
+
+	dettool -updatedetrun -det_id <det_id> -iter <iter> -stat <stats> [-rerun|-ok|-stop]
+
+where "-rerun" causes a new iteration to be started, "-ok" causes the
+(normalised) stack to be elevated to a master (i.e., for use) and
+"-stop" causes all iteration to cease (and potentially alerts the
+human overlords).
+
+In practise, the above three commands ("dettool -residexp", "detool
+-updateresid" and "dettool -updatedetrun"), as well as the decisions
+about what to reject, and whether to rerun, accept or stop will be
+made by a Perl script, "detrend_reject_exp.pl".
+
+--------------------------------------------------------------------------------
+
+The detrend run may also be restarted manually by specifying exposures
+to accept and reject:
+
+	dettool -rerun -det_id <det_id> -iter <iter> -accept <exp_id> -accept <exp_id> ... -reject <exp_id> -reject <exp_id> ...
+
+