Building a DARKMASK

The IPP can construct a bad pixel mask using either a set of darks or a set of flats as the input images. From the user perspective, there are two steps to the process:

1) The pre-mask detRun : build a composite dark/flat image to serve as the baseline image

2) The mask-building detRun :

  • use the baseline dark/flat to detrend a set of darks/flats.
  • Examine each detrended image from the input set separately, flagging pixels that deviate from the mean in each cell.
  • Examine the stack of detrended input images, flagging pixels that have a high dispersion across the stack.

The example below is for a DARKMASK, but it applies equally well for a FLATMASK.

Baseline Image : Pre-Mask Dark

We start by building a baseline DARK without rejecting any pixels. Turn off all the rejection rules in the camera- or site-level rejections.config file:

DARK METADATA

FILTER STR * EXPECTED F32 0.0 IMFILE.MEAN F32 2.0 IMFILE.STDEV F32 10.0 IMFILE.MEANSTDEV F32 0.5 IMFILE.SKEWNESS F32 0.0 IMFILE.KURTOSIS F32 0.0 IMFILE.SNR F32 0.0 IMFILE.BIN.STDEV F32 0.0 IMFILE.BIN.SNR F32 0.0 IMFILE.FLUX F32 0.0 EXP.MEAN F32 1.0 EXP.STDEV F32 10.0 EXP.MEANSTDEV F32 0.5 EXP.SNR F32 0.0 EXP.BIN.STDEV F32 0.0 EXP.BIN.SNR F32 0.0 EXP.FLUX F32 0.0 ENSEMBLE.MEAN F32 3.0 ENSEMBLE.STDEV F32 3.0 ENSEMBLE.MEANSTDEV F32 0.0

END

## UNCOMMENT THIS SECTION WHEN MAKING A ## PRE-MASK (NO REJECTIONS) DARK #DARK METADATA # FILTER STR * # EXPECTED F32 0.0 # IMFILE.MEAN F32 0.0 # IMFILE.STDEV F32 0.0 # IMFILE.MEANSTDEV F32 0.0 # IMFILE.SKEWNESS F32 0.0 # IMFILE.KURTOSIS F32 0.0 # IMFILE.SNR F32 0.0 # IMFILE.BIN.STDEV F32 0.0 # IMFILE.BIN.SNR F32 0.0 # IMFILE.FLUX F32 0.0 # EXP.MEAN F32 0.0 # EXP.STDEV F32 0.0 # EXP.MEANSTDEV F32 0.0 # EXP.SNR F32 0.0 # EXP.BIN.STDEV F32 0.0 # EXP.BIN.SNR F32 0.0 # EXP.FLUX F32 0.0 # ENSEMBLE.MEAN F32 0.0 # ENSEMBLE.STDEV F32 0.0 # ENSEMBLE.MEANSTDEV F32 0.0 #END

Choose a set of darks to be your input data. This can be the same set that will be used to flag bad pixels in the next step. Run dettool to create the detRun, and let pantasks build the DARK. For example:

dettool -simple -definebyquery -det_type DARK -dbname essence -filelevel FPA -workdir file://data/ipp003.0/rodney/sresswork/2003/ -telescope BLANCO -inst MOSAIC2 -select_exp_type dark -select_dateobs_begin 2003-09-25T12:00:00 -select_dateobs_end 2004-01-02T12:00:00 -time_begin 2003-09-25T12:00:00 -time_end 2004-01-02T12:00:00

Final image : the DARKMASK

Now we're ready to run dettool to create the DARKMASK detRun, and let pantasks build our DARKMASK. e.g.:

dettool -simple -definebyquery -det_type DARKMASK -dbname essence -filelevel FPA -workdir file://data/ipp003.0/rodney/sresswork/2003/ -telescope BLANCO -inst MOSAIC2 -select_exp_type dark -select_dateobs_begin 2003-09-25T12:00:00 -select_dateobs_end 2004-01-02T12:00:00 -time_begin 2003-09-25T12:00:00 -time_end 2004-01-02T12:00:00

The baseline DARK is subtracted from each of the input dark images. Statistics are measured for each chip (or each readout) in each of the detrended images, and pixels that deviate from the expected values are flagged. See below for more details.

Combining Masks

One may want to create bad pixel masks from several different data sets. For example, a DARKMASK and a series of FLATMASKs. Only one mask is applied to each image in the detrend processing, so your multiple masks must be combined together, usually by finding the union of the masks. To do this we use a serial perl script, outside of the pantasks environment, that can combine multiple detrend images with logical operators such as 'OR' or 'AND.' For example:

ipp_detrend_combine.pl --det_type MASK --filelevel FPA --inst SIMTEST --telescope SimScope? --det_id1 3 --iteration1 0 --det_id2 4 --iteration2 0 --workdir path://SIMTEST/detrend/ --operation '|' --mask > ipp_detrend_combine.log

Which pixels are flagged?

If all the pixels were perfect, none of these dark-subtracted darks would have any features, and a given pixel would be the same in every dark. We can define a bad pixel, then, as anything that stands out from the others in its cell (a 'cell-deviant') or any pixel that is not consistent across the input stack (a 'stack-deviant'). At the moment, the ipp is only flagging cell-deviant pixels.

The mask creation relies on the assumption that the input images should be perfectly flat (modulo noise). The appropriate inputs are therefore darks with a master dark subtracted (DARKMASK), or flat-fielded images with a range of exposure times (either of flats, or actual object images; FLATMASK). A first pass is made through the inputs, flagging pixels in each image that are more than MASK.SUSPECT standard deviations from the background(any pixel that sticks up or down). These "suspect" pixels may be bad, or noise, or astrophysical objects. The suspect maps are summed, resulting in an image whose pixels contain the number of times that pixel was flagged as suspect. The MASK.BAD threshold is then applied to this image to identify pixels which are bad in a majority of the input images. The standard deviation of the suspect pixel image may be calculated by examining the image, or by assuming Poisson statistics (MASK.MODE). The resulting mask is grown by MASK.GROW.NPIX, and the pixels set to MASKVAL.

Parameters

The relevant thresholds and flags for mask building are set in the ppMerge.config file:

MASK.SUSPECT F32 5.0 # Threshold for suspect pixels (sigma) MASK.BAD F32 4.0 # Threshold for bad pixels (sigma) MASK.MODE STR POISSON # How to determine the stdev of the suspect pix image. poisson=assume poisson statistics. STATS.BY.CHIP BOOL TRUE # measure stats for masking by chip (or by readout) MASK.GROW.NPIX S32 3 # (??) MASKVAL STR SAT,BAD # Mask value for input data

The values which are assigned to masked pixels in the mask file are defined in the masks.config file:

### Recipe specifying values for various mask concepts BLANK U8 0x01 # The pixel is blank or has no (valid) data FLAT U8 0x02 # The pixel is non-positive in the flat-field DETECTOR U8 0x02 # The detector pixel is bad (e.g., bad column, charge trap) SAT U8 0x04 # The pixel is saturated in the image of interest BAD U8 0x04 # The pixel is low in the image of interest RANGE U8 0x04 # The pixel is out of range in the image of interest CR U8 0x08 # The pixel is probably a CR SUSPECT U8 0x40 # The pixel is suspected of being bad, but may not be MARK U8 0x80 # The pixel is marked as temporarily ignored