Index: /branches/eam_branches/eam_branch_20090303/doc/misc/detnorm.rework.txt
===================================================================
--- /branches/eam_branches/eam_branch_20090303/doc/misc/detnorm.rework.txt	(revision 23206)
+++ /branches/eam_branches/eam_branch_20090303/doc/misc/detnorm.rework.txt	(revision 23206)
@@ -0,0 +1,75 @@
+
+I've been running the detrend analysis on the new gpc1 flats, and
+discovered an important issue.  These chips have significant gain
+variations across the devices.  
+
+We need to have detrend.normstats run on the results of the
+detResidImfiles, rather than detProcessedImfiles.  This requires
+altering the processing sequence from this:
+
+* process each of the input images (detrend.process.imfile)
+* generate a stack for each chip (detrend.stack.imfile)
+* use the stats from the processed chips to determine per-chip
+renormalizations for the stacks. the goal in this step is to yield a
+flat-field image for which a flattened input image is flat across the
+entire mosaic, not just each chip.  (detrend.normstats.imfile)
+* apply the normalization to each of the stacks (detrend.norm.imfile)
+* generate mosaic-wide stats and jpegs for the normalized flats
+(detrend.norm.exp)
+* apply the normalized flats to the processed images (detrend.resid.imfile)
+* generate per-exposure stats and jpegs for the residual images.
+(detrend.resid.exp)
+
+The problem is that, with a large gain variation across each chip, the
+measurement of the per-chip stats from the processed images is not a
+good measurement of the flattened mean value on that chip.  We really
+should be measuring the statistics from the flattened images.  Paul
+and I discussed the pipeline organization and have concluded that we
+can re-work the detrend pipeline without too much work to achieve this
+if we perform the flat-fielding before the normalization, and do the
+following steps:
+
+* process each of the input images (detrend.process.imfile)
+* generate a stack for each chip (detrend.stack.imfile)
+* apply the flats to the processed images (detrend.resid.imfile)
+* use the stats from the residual chips to determine per-chip
+renormalizations for the stacks (detrend.normstats.imfile)
+* apply the normalization to each of the stacks (detrend.norm.imfile)
+* generate mosaic-wide stats and jpegs for the normalized flats
+(detrend.norm.exp)
+* generate per-exposure stats and jpegs for the residual images,
+applying the normalizations on-the-fly. (detrend.resid.exp)
+
+--
+
+Tasks to do:
+
+* ppStack could do an approximate normalization on the fly (perhaps
+  not needed; perhaps already done? what is the output scaling of the
+  resulting flat stack?)
+
+* dettool -toresidimfile should trigger on completed detStackedImfile
+  (not detNormalizedImfile) (DONE)
+
+* dettool -tonormalizedstat should trigger on completed
+  detResidImfile, not detStackedImfile (DONE)
+
+* detrend_norm_calc.pl needs to read the stats from detResidImfile,
+  not detProcessedImfile (DONE)
+
+* detrend.norm.imfile uses the results of detrend_norm_calc.pl as
+  before (DONE)
+
+* detrend.resid.exp applies the normalizations on-the-fly to the b1 and
+  b2 input images, and to the exposure-wide stats pushed into the db.
+
+* detrend.norm.exp is the same?
+
+*** the 'verify' analysis needs to be modified.  I think we can keep
+    it as it was (detrend.process.imfile -> detrend.resid.imfile ->
+    detrend.resid.exp) where the resid imfile is generated using the
+    detNormalizedImfile rather than the detStackedImfile.  Does
+    detrend.resid.exp need to be modified for 'verify' mode to *avoid*
+    applying the normalization?
+
+