Yet More Notes : Aggressive Masking for Magic's Sake : 2008.11.12

Magic was being overwhelmed with false positive streaks due to the large number of artifacts still in the images. Some of these were correctable in the static mask, so Paul and I made another pass to the complete camera, masking non-linear dark structures. We were especially concerned with long, linear features such as the hot columns caused by glow points, but we also were trying to clean things more generally. This pass increased somewhat the total masked area:

masked fraction (full focal plane): 0.185


masked fraction (unvignetted region): 0.126


Additional Notes : Masking for Bad CTE : 2008.07.18

I have made another pass to the GPC1 mask, this time adding in regions which appear to have bad CTE, based on the level of correlated noise. These regions largely coincide with regions where the dark current is high and non-linear (the 'windows' in the chip corners). My selection of regions with bad CTE is quite subjective. I was looking for regions where the noise is smoothed by the charge being dropped. I tried to be fairly conservative, since these areas of bad CTE play havoc with the photometry and the astrometry. However, it was not always easy to judge. In some cases, I am probably too generous, leaving behind too many of the bad regions; in other cases, I am probably too aggressive, excising more than necessary. The particular chips are worrying: chips XY26, XY31, and XY37 may have substantially more bad CTE everywhere. Chip XY14 is also quite problematic, with charge bleeding in odd ways.

Below are two images showing the fraction of masked for each cell, first for the total focal plane, and second only showing the regions inside the nominal unvignetted 1.5 degree radius region. The end result is that, in the unvignetted area, the total number of masked pixels is 11.5%. If we have to exclude all pixels on the four chips listed above, the total would rise by about 3%.

masked fraction (full focal plane): 0.174


masked fraction (unvignetted region): 0.115


Initial Write-Up : 2008.06.24

Within the IPP, the detrend analysis system can build masks based on the dark residual structures and the flat residual structures. When a DARK or FLAT master image is generated, the input images can be used to assess the quality of the correction, and pixels which are outliers can be excluded. This process starts be applying the master DARK or FLAT to the input images, yielding a collection of residual images. These images are examined on a chip-by-chip basis to identify pixels which are not well corrected. In the case of the DARK residual images, the mean value should be 0.0, with a scatter dependent on the read noise for the image; in the case of the FLAT residual images, the mean value should be a constant, large value with scatter dependent on the Poisson statistics of the input image. Note that input FLAT images are normally required to have counts somewhere in the range 10k - 30k, but low enough to avoid saturation. The MASK analysis examines the statistics of each chip and flags suspect pixels within that image which deviate from the mean by a specified amount (normally a number of sigmas). After each residual image has been examined (for the given chip), pixels for which more than a specified number of pixels were considered suspect are marked as BAD and masked.

The automatic analysis of the bad pixels using the DARK and FLAT residuals helps to identify a large fraction of the bad pixels. However, it is difficult to set the rejection thresholds accurately enough to robustly reject all bad regions without rejecting too many good pixels. Thus, I have supplemented the automatic analysis with a manual inspection of a single flat residual image, and have marked by hand the remaining outliers which were not caught by the automatic analysis.

I have generated a GPC1 DARKMASK using a set of dark images taken in May 2008 covering a range of exposure times and detector temperatures. I previously used these to build a dark frame, then I used a subset of the input images to build the DARKMASK. I have built a FLATMASK using a set of i-band twilight flat images previously used to generatet an i-band flat, only using those which were not rejected by the flat-field construction (usually those with obvious clouds).

I OR-ed these two masks together, then applied this mask to one of the residual flat images. I supplemented the automatic mask with a fixed mask on the boundary: 6 pixels on the low x and low y, 7 pixels on the high x and high 7 sides. These numbers were chosen based on examining a number of cells. Pixels which were masked were set to a very low value; pixels which were outliers relative to the mean (abs(value - mean) > 500) were set to a very high value, the remaining pixels were left intact. These images were displayed with a colormap to make the masked and unmasked outliers very obvious. Here is an example residual image (full chip).



I examined each cell and marked the obvious bad structures. I did not normally mark the areas which appear to have bad CTE. I also ignored some of the smallish hot blobs of pixels; I would like to see better statistics before rejecting these.

Below are two images showing the fraction of masked pixels in each cell (black = 0, white = 1). The second image excludes the pixels (cells) which are beyond the 1.5 degree radius. If I consider all cells, the total fraction of masked pixels in the full focal plane is 13%; restricting to the unvignetted pixels, the fraction of masked pixels is 7.4%. This number will certainly go up if I mask the regions of bad CTE more aggressively, though I note that many of those regions are also masked by their effect on the flat or dark.

masked fraction (full focal plane): 0.130


masked fraction (unvignetted region): 0.074