Changes between Initial Version and Version 1 of Bad_Pixel_Masks

Timestamp:

Feb 24, 2009, 4:23:53 PM (17 years ago)

Author:

trac

Comment:

--

Bad_Pixel_Masks

@@ +1 @@
+=== Building a DARKMASK ===
+
+The IPP can construct a bad pixel mask using either a set of darks or a set of flats as the input images. From the user perspective, there are two steps to the process: 
+
+1) The pre-mask detRun : build a composite dark/flat image to serve as the baseline image
+
+2) The mask-building detRun : 
+ * use the baseline dark/flat to detrend a set of darks/flats. 
+ * Examine each detrended image from the input set separately, flagging pixels that deviate from the mean in each cell.  
+ * Examine the stack of detrended input images, flagging pixels that have a high dispersion across the stack. 
+
+The example below is for a DARKMASK, but it applies equally well for a FLATMASK.
+
+==== Baseline Image : Pre-Mask Dark ====
+
+We start by building a baseline DARK without rejecting any pixels.  Turn off all the rejection rules in the camera- or site-level rejections.config  file:
+
+ DARK METADATA
+   FILTER            STR  *
+   EXPECTED          F32  0.0
+   IMFILE.MEAN       F32  2.0
+   IMFILE.STDEV       F32 10.0
+   IMFILE.MEANSTDEV   F32  0.5
+   IMFILE.SKEWNESS    F32  0.0
+   IMFILE.KURTOSIS    F32  0.0
+   IMFILE.SNR         F32  0.0
+   IMFILE.BIN.STDEV   F32  0.0
+   IMFILE.BIN.SNR     F32  0.0
+   IMFILE.FLUX        F32  0.0
+   EXP.MEAN           F32  1.0
+   EXP.STDEV          F32 10.0
+   EXP.MEANSTDEV      F32  0.5
+   EXP.SNR            F32  0.0
+   EXP.BIN.STDEV      F32  0.0
+   EXP.BIN.SNR        F32  0.0
+   EXP.FLUX           F32  0.0
+   ENSEMBLE.MEAN      F32  3.0
+   ENSEMBLE.STDEV     F32  3.0
+   ENSEMBLE.MEANSTDEV F32  0.0
+ END
+
+ ## UNCOMMENT THIS SECTION WHEN MAKING A 
+ ## PRE-MASK (NO REJECTIONS) DARK
+ #DARK METADATA
+ #  FILTER                   STR  *
+ #  EXPECTED                 F32  0.0
+ #  IMFILE.MEAN              F32  0.0
+ #  IMFILE.STDEV       F32  0.0
+ #  IMFILE.MEANSTDEV   F32  0.0
+ #  IMFILE.SKEWNESS    F32  0.0
+ #  IMFILE.KURTOSIS    F32  0.0
+ #  IMFILE.SNR         F32  0.0
+ #  IMFILE.BIN.STDEV   F32  0.0
+ #  IMFILE.BIN.SNR     F32  0.0
+ #  IMFILE.FLUX        F32  0.0
+ #  EXP.MEAN           F32  0.0
+ #  EXP.STDEV          F32  0.0
+ #  EXP.MEANSTDEV      F32  0.0
+ #  EXP.SNR            F32  0.0
+ #  EXP.BIN.STDEV      F32  0.0
+ #  EXP.BIN.SNR        F32  0.0
+ #  EXP.FLUX           F32  0.0
+ #  ENSEMBLE.MEAN      F32  0.0
+ #  ENSEMBLE.STDEV     F32  0.0
+ #  ENSEMBLE.MEANSTDEV F32  0.0
+ #END
+
+Choose a set of darks to be your input data. This can be the same set that will be used to flag bad pixels in the next step.  Run dettool to create the detRun, and let pantasks build the DARK.  For example:
+
+ dettool -simple -definebyquery -det_type DARK -dbname essence  -filelevel FPA -workdir file://data/ipp003.0/rodney/sresswork/2003/ -telescope BLANCO -inst MOSAIC2 -select_exp_type dark -select_dateobs_begin 2003-09-25T12:00:00 -select_dateobs_end 2004-01-02T12:00:00 -time_begin 2003-09-25T12:00:00 -time_end 2004-01-02T12:00:00
+
+==== Final image : the DARKMASK ====
+
+Now we're ready to run dettool to create the DARKMASK detRun, and let pantasks build our DARKMASK. e.g.:
+
+ dettool -simple -definebyquery -det_type DARKMASK -dbname essence  -filelevel FPA -workdir file://data/ipp003.0/rodney/sresswork/2003/ -telescope BLANCO -inst MOSAIC2 -select_exp_type dark -select_dateobs_begin 2003-09-25T12:00:00 -select_dateobs_end 2004-01-02T12:00:00 -time_begin 2003-09-25T12:00:00 -time_end 2004-01-02T12:00:00
+
+The baseline DARK is subtracted from each of the input dark images.  Statistics are measured for each chip (or each readout) in each of the detrended images, and pixels that deviate from the expected values are flagged.  See below for more details.  
+
+==== Combining Masks ====
+
+One may want to create bad pixel masks from several different data sets. For example, a DARKMASK and a series of FLATMASKs. Only one mask is applied to each image in the detrend processing, so your multiple masks must be combined together, usually by finding the union of the masks. To do this we use a serial perl script, outside of the pantasks environment, that can combine multiple detrend images with logical operators such as 'OR' or 'AND.'  For example:
+
+ipp_detrend_combine.pl --det_type MASK --filelevel FPA --inst SIMTEST --telescope SimScope --det_id1 3 --iteration1 0 --det_id2 4 --iteration2 0 --workdir path://SIMTEST/detrend/ --operation '|' --mask > ipp_detrend_combine.log
+
+  
+=== Which pixels are flagged? ===
+
+If all the pixels were perfect, none of these dark-subtracted darks would have any features, and a given pixel would be the same in every dark.  We can define a bad pixel, then, as anything that stands out from the others in its cell (a 'cell-deviant') or any pixel that is not consistent across the input stack (a 'stack-deviant').   At the moment, the ipp is only flagging cell-deviant pixels.
+
+The mask creation relies on the assumption that the input images should be perfectly flat (modulo noise).  The appropriate inputs are therefore
+darks with a master dark subtracted (DARKMASK), or flat-fielded images with a range of exposure times (either of flats, or actual object
+images; FLATMASK). A first pass is made through the inputs, flagging pixels in each image that are more than MASK.SUSPECT standard deviations from the background(any pixel that sticks up or down).  These "suspect" pixels may be bad, or noise, or astrophysical objects.  The suspect maps are summed, resulting in an image whose pixels contain the number of times that pixel was flagged as suspect.  The MASK.BAD threshold is then applied to this image to identify pixels which are bad in a majority of the input images.  The standard deviation of the suspect pixel image may be calculated by examining the image, or by assuming Poisson statistics (MASK.MODE).  The resulting mask is grown by MASK.GROW.NPIX, and the pixels set to MASKVAL.
+
+==== Parameters ====
+
+The relevant thresholds and flags for mask building are set in the ppMerge.config file:
+
+ MASK.SUSPECT    F32     5.0             # Threshold for suspect pixels (sigma)
+ MASK.BAD        F32     4.0             # Threshold for bad pixels (sigma)
+ MASK.MODE       STR     POISSON         # How to determine the stdev of the suspect pix image. poisson=assume poisson statistics. 
+ STATS.BY.CHIP   BOOL    TRUE            # measure stats for masking by chip (or by readout)
+ MASK.GROW.NPIX  S32     3               # (??)
+ MASKVAL         STR     SAT,BAD         # Mask value for input data
+
+The values which are assigned to masked pixels in the mask file are defined in the masks.config file:
+
+ ### Recipe specifying values for various mask concepts
+ BLANK           U8      0x01            # The pixel is blank or has no (valid) data
+ FLAT            U8      0x02            # The pixel is non-positive in the flat-field
+ DETECTOR        U8      0x02            # The detector pixel is bad (e.g., bad column, charge trap)
+ SAT             U8      0x04            # The pixel is saturated in the image of interest
+ BAD             U8      0x04            # The pixel is low in the image of interest
+ RANGE           U8      0x04            # The pixel is out of range in the image of interest
+ CR              U8      0x08            # The pixel is probably a CR
+ SUSPECT         U8      0x40            # The pixel is suspected of being bad, but may not be
+ MARK            U8      0x80            # The pixel is marked as temporarily ignored
+
+
+----