March 5, 2012

Dark model investigations

Due to the large number of images, I've moved this to a new page: Background_Dark_Model

March 1-2, 2012

Similar to Nigel's work, I've been looking at the gradients, however starting from the individual frames. Choosing one of the input exposures that contributes the bulk of the data to skycell.1405.022, I've placed a 300 pixel wide profile along one row of cells (a vertical column as seen in the stack image below). For reference, my row of cells is the 6th from the left side in the stack image below. The first set of profiles shows the data for the raw image (no continuity correction or background models), the raw image corrected by the continuity correction only, and finally the continuity and background corrected image, which is equivalent to what would have gone into the stack.

Raw Continuity Corrected Final

Based on these profiles, it's clear that the initial image has some gradient included, the continuity correction does a reasonable job of aligning these gradients into a large slope, and then the background correction removes this large slope, with a few DN peak-to-peak residual gradient remaining. I checked that this gradient was not due entirely to the increased row-to-row variations evident on the high edge of the chip, and that does not appear to be the case. The best description is that there is an underlying gradient, and as the gradient increases, the row-to-row variations produce higher spikes that trigger the false detections. It is interesting to note that the fifth cell in this profile (with a small slope and low residual gradient) matches a cell with few false detections.

As one aspect of these gradients is to determine how the dark influences the pattern, I looked at the profile for a dark detresid image (a dark frame processed using the current dark model; note that this detresid was taken ~250 days after the footprint data, and therefore is not a direct comparison) to see what residual patterns are visible.

This detresid image shows a similar saw-tooth pattern visible along that same row of cells on the same OTA. Checking the detproc image (a dark frame that has not had the dark model applied) shows a different profile shape:

This suggests that the current dark model assumes a given detector response along the cell pixel columns that possibly no longer matches the real detector response. I am now checking a large sample of dark images to determine how this has changed over time.

March 2, 2012

From Nigel's email: "Example gradients across cells with streaks attached. If you look at the image I sent round with false detections on, full1415022.jpg (which is really 1405022!), these are the cells from the bottom right corner of the top left skycell. i.e. 5 rows down from the top and 5/6/7 in from the left side.

The graphs attached show the mean pixel value as a function of y-coord (vertical on full1415022.jpg) for a box about 100 x pixels wide."

February 17, 2012

Example skycell false detections

Footprint g filter, skycell.1405.022

Footprint g filter, skycell.1315.071

Footprint g filter, skycell.1405.020

Footprint g filter, skycell.1315.092

February 15, 2012

False Positive rates

Nigel sent around an image made by Peter showing the results of psphot on the new footprint g stack. Left shows the old results, and the right shows the new. This shows that there are significant improvements (quoting the email: "The number of false detections has dropped by about 40% at g~23.5 (CAL_PSF_MAG), which is good, although they are still higher than the number of matched (to Stripe82) detections faintward of g~23.4."), although stripes of false detections are still visible. I suspect some of the remaining concentrations are caused by corner glows/dark features.

Independently of this, I've made quick DVOs of the individual exposures for each of the three reductions to see how many times an object is detected in the set of 12 exposures:

Assuming all single detection objects are false positives, the improved continuity correction is only removing about 10% of these false objects from individual exposures.

February 9, 2012 Results


After some development work and debugging, a new PATTERN.CONTINUITY correction has been implemented that is applied to all OTAs and supersedes the previous PATTERN.CELL correction. As proposed below, the median value along each edge of each cell is calculated, and a set of cell-by-cell offsets is calculated from the differences between the edges of adjacent cells. These offsets are applied, with the assumption that by minimizing the cell-to-cell offsets, we will supply the background fitting code with a smooth and continuous function to fit. Applying this correction to the example case used previously:

After testing that this method works as intended, a new processing of the SAS g-filter footprint data was run using this correction (label czw.footprint_continuity2.bkg, data_group czw.20120209.footprint.g). Shown below is a comparison between the original IPP processing, the updated PATTERN.ROW "czw" reduction, and the newest continuity applied reduction. To make this easier, I've chosen the skycell used in Nigel's discussion on the DRAVG page (skycell 1405.030).

The continuity correction does seem to resolve the cell-level banding that is visible in the previous reductions. The entire set of stacks will be available on the distribution server by the end of today. There are still some discontinuities at the edges of OTAs (this skycell shows the corner where four OTAs join), but the majority of the offsets are gone.


Although the cell-scale issues seem to be resolved by this correction, a lower level camera feature can be clearly seen in these new stacks. We use the PATTERN.ROW correction to remove row-by-row bias offsets that are present on the detectors (in the warps and stacks, rows run vertically on these images). We apply this correction sparingly, as it creates the "butterfly" patterns around bright stars, so it is limited to only the cells that have the worst row-by-row offsets. Previously, our assumption was that these offsets were random, and so any variations on cells without the PATTERN.ROW correction would be eliminated in the stacks. However, the new stacks show that there are larger correlations between these offsets that are reinforced by the stacking, showing up as wide (~5 pixel) ripples in the background. Looking at the input warps for this stack skycell shows that there does not seem to be a time trend in the amplitude of the ripples (such that sequential exposures have similar structure), but it does seem that the peaks and troughs are more likely to fall in the same location than not. Because of this amplification, as the number of input warps increases, the signal to noise of these ripples increases, making peaks likely sources of false detections.

This comparison shows the input warps for this skycell (labeled with the image number for that night: o5744g0NNNo) along with the convolved and unconvolved stacks.

We currently do not understand the source of these ripples, nor do we have a solution at the moment.

January 2012 Study

The main source of background fluctuations printing through the stack images seems to be caused by discontinuities in the background level between cells. These discontinuities seem to generally be enhanced by the PATTERN.ROW (which corrects the row-by-row bias offsets) and PATTERN.CELL (which corrects the sagging observed in some entire cells). In December 2011, a check was made to identify cells which are receiving these corrections but no longer seem to require them. In addition, an attempt was made to augment the PATTERN.ROW correction to return the x- and y- dimension gradients to the data after the correction was applied. This seems to have been generally successful, as the LAP stacks produced after these changes were added to the working tag (ipp-20111222) appear to have smoother and more consistent background levels. However, it is still obvious that there are some discontinuities that continue to print through into the stacks when the number of inputs is low.

As OTA27 has cells that both have the PATTERN.ROW correction (columns 0-3,7) and do not have this correction (columns 4-6), it was chosen as a test case to see what could be done to ensure the cell-to-cell continuity between these cells. Exposure o5479g0287o was arbitrarily chosen based on JPEG images that showed background issues. Shown below is an image of this OTA (cell 00 is located at the bottom left corner, and increment up and to the right), along with a profile slice along a column and row of cells. All the regular processing stages were enabled (MASK/DARK/NONLINEARITY/FLAT/PATTERN.ROW/BACKGROUND). The large jumps between columns 3-4 and 6-7 show that the discontinuity between these columns due to the different corrections applied cause the background fitting code to arrive at bad background levels.

Disabling the background correction (but leaving the PATTERN.ROW correction enabled) shows that there are discontinuities in the cell-to-cell background levels, as well as trends across the cells. These trends are evident even on columns where no PATTERN.ROW correction has been applied.

The proposed fix for this is to calculate a set of cell-to-cell offsets that will minimize the edge background level differences between adjacent cells. This will provide a "smooth" function for the background fitting code to model, which should remove the cell-to-cell variations currently observed. This may create a case where the discontinuities are shifted to the OTA-to-OTA level, but that should influence a smaller number of pixels than the current cell-to-cell issues.

This proposed correction will also replace the PATTERN.CELL correction. This correction currently attempts to match the median background level of a cell that requires correction to the median across the OTA. This often causes issues where bright stars in the cell to be corrected (or gradients across the field) create a bogus result that does not match that cell to the expected level. By forcing cell edges to match, the new correction should obtain a better estimate of the correct level for the sagging cells.

2011-11-20 Stack questions

Due to the 2011-07-29 stack containing data from two nights, the "reverse checkmark" near the center of this frame is dithered and almost fully removed. Similarly, the small camera defects are cleaned compared to the 2011-12-14 stack which only contains the 12 expected input exposures.