February 9, 2012 Results

Continuity

After some development work and debugging, a new PATTERN.CONTINUITY correction has been implemented that is applied to all OTAs and supersedes the previous PATTERN.CELL correction. As proposed below, the median value along each edge of each cell is calculated, and a set of cell-by-cell offsets is calculated from the differences between the edges of adjacent cells. These offsets are applied, with the assumption that by minimizing the cell-to-cell offsets, we will supply the background fitting code with a smooth and continuous function to fit. Applying this correction to the example case used previously:

After testing that this method works as intended, a new processing of the SAS g-filter footprint data was run using this correction (label czw.footprint_continuity2.bkg, data_group czw.20120209.footprint.g). Shown below is a comparison between the original IPP processing, the updated PATTERN.ROW "czw" reduction, and the newest continuity applied reduction. To make this easier, I've chosen the skycell used in Nigel's discussion on the DRAVG page (skycell 1405.030).

The continuity correction does seem to resolve the cell-level banding that is visible in the previous reductions. The entire set of stacks will be available on the distribution server by the end of today. There are still some discontinuities at the edges of OTAs (this skycell shows the corner where four OTAs join), but the majority of the offsets are gone.

Ripples

Although the cell-scale issues seem to be resolved by this correction, a lower level camera feature can be clearly seen in these new stacks. We use the PATTERN.ROW correction to remove row-by-row bias offsets that are present on the detectors (in the warps and stacks, rows run vertically on these images). We apply this correction sparingly, as it creates the "butterfly" patterns around bright stars, so it is limited to only the cells that have the worst row-by-row offsets. Previously, our assumption was that these offsets were random, and so any variations on cells without the PATTERN.ROW correction would be eliminated in the stacks. However, the new stacks show that there are larger correlations between these offsets that are reinforced by the stacking, showing up as wide (~5 pixel) ripples in the background. Looking at the input warps for this stack skycell shows that there does not seem to be a time trend in the amplitude of the ripples (such that sequential exposures have similar structure), but it does seem that the peaks and troughs are more likely to fall in the same location than not. Because of this amplification, as the number of input warps increases, the signal to noise of these ripples increases, making peaks likely sources of false detections.

We currently do not understand the source of these ripples, nor do we have a solution at the moment.

January 2012 Study

The main source of background fluctuations printing through the stack images seems to be caused by discontinuities in the background level between cells. These discontinuities seem to generally be enhanced by the PATTERN.ROW (which corrects the row-by-row bias offsets) and PATTERN.CELL (which corrects the sagging observed in some entire cells). In December 2011, a check was made to identify cells which are receiving these corrections but no longer seem to require them. In addition, an attempt was made to augment the PATTERN.ROW correction to return the x- and y- dimension gradients to the data after the correction was applied. This seems to have been generally successful, as the LAP stacks produced after these changes were added to the working tag (ipp-20111222) appear to have smoother and more consistent background levels. However, it is still obvious that there are some discontinuities that continue to print through into the stacks when the number of inputs is low.

As OTA27 has cells that both have the PATTERN.ROW correction (columns 0-3,7) and do not have this correction (columns 4-6), it was chosen as a test case to see what could be done to ensure the cell-to-cell continuity between these cells. Exposure o5479g0287o was arbitrarily chosen based on JPEG images that showed background issues. Shown below is an image of this OTA (cell 00 is located at the bottom left corner, and increment up and to the right), along with a profile slice along a column and row of cells. All the regular processing stages were enabled (MASK/DARK/NONLINEARITY/FLAT/PATTERN.ROW/BACKGROUND). The large jumps between columns 3-4 and 6-7 show that the discontinuity between these columns due to the different corrections applied cause the background fitting code to arrive at bad background levels.

Disabling the background correction (but leaving the PATTERN.ROW correction enabled) shows that there are discontinuities in the cell-to-cell background levels, as well as trends across the cells. These trends are evident even on columns where no PATTERN.ROW correction has been applied.

The proposed fix for this is to calculate a set of cell-to-cell offsets that will minimize the edge background level differences between adjacent cells. This will provide a "smooth" function for the background fitting code to model, which should remove the cell-to-cell variations currently observed. This may create a case where the discontinuities are shifted to the OTA-to-OTA level, but that should influence a smaller number of pixels than the current cell-to-cell issues.

This proposed correction will also replace the PATTERN.CELL correction. This correction currently attempts to match the median background level of a cell that requires correction to the median across the OTA. This often causes issues where bright stars in the cell to be corrected (or gradients across the field) create a bogus result that does not match that cell to the expected level. By forcing cell edges to match, the new correction should obtain a better estimate of the correct level for the sagging cells.

2011-11-20 Stack questions

Due to the 2011-07-29 stack containing data from two nights, the "reverse checkmark" near the center of this frame is dithered and almost fully removed. Similarly, the small camera defects are cleaned compared to the 2011-12-14 stack which only contains the 12 expected input exposures.