October 8, 2013

Notes on fftP pattern correction/row-to-row bias offset issue.

The mathematical process for fftP is as follows, for an input image I:

  • Remove median:
    • R = I - median(I)
  • Clip signals:
    • S = R * (R > 5 * sigma_MAD(R))
    • R = R - S
  • Taper:
    • R = R * sinc((x - 0.5 xsize)/xsize/2) * sinc((y - 0.5 ysize)/ysize/2)
  • FFT:
    • C = FFT(R)
  • Censor:
    • C[abs(u) < Ulimit][abs(v) > Vlimit] = 0.0;
  • IFFT:
    • T = IFFT(C)
  • Untaper:
    • T = T / (sinc(x..) * sinc(y..))
  • Restore signals:
    • T = T + S

The images below show a set of IPP chip processed mosaics (top panels) and fftP processed raw mosaics (bottom panels) for the OTA/cell combinations listed in the tables. The fftP processed images have not been detrended in any way, and so retain burntool trails, corner glows, etc. The fftP processing has an odd set of NAN value masks that appear due to some error in my quick coding (these mirror the censored locations on the C image, so that's a likely culprit). I do not believe that these impact the rest of the pixels. The overscan region in the fftP frames is set to zero in the initial "R" image, but this is not repeated after the inverse FFT, so residual row-to-row "corrections" are visible in the region. All images have been scaled to the flux range of [-10:50], so the colorbars match.

o6567g0110o OTA12 OTA27 OTA32
chip processed cell xy45 xy45 xy45
chip PATTERN.ROW? No No No (xy3v,xy7v only)
fftP processed raw cell xy45 xy45 xy45

chip processed cell xy13 xy13 xy13
chip PATTERN.ROW? No Yes No
fftP processed raw cell xy45 xy45 xy45

May 16, 2013

The notation used below is that FFT(g(x,y)) = G(U,V).

The following image shows the results of playing with my test FFT filtering code. The top left panel shows the original overscan subtracted science image. The top central panel shows the results of a 3-sigma clipping that reduces the power of the deviant frequency by a factor of ten. This clipping is performed for U > 100, V < 3. There are some minor changes, but most of the noise remains. I'm unclear at this point if the issue is the clipping threshold or the power reduction factor. The top right panel applies a harsh clipping, setting the value of all components in the clipping region to 0.0. This clearly removes more of the noise, but is likely excessive. In addition, corner glows have a banded nature introduced. This probably suggests that this algorithm should be applied post-DARK calibration.

The bottom row shows a series of "obvious mistakes" that can be generated with this kind of filtering. All used the harsh clipping described above. The bottom left panel shows the results when U is clipped down to U > 11. This clears up a lot of the noise, but also introduces the "butterfly effect" that has been seen with the PATTERN.ROW results. This makes sense, as PATTERN.ROW is effectively removing all information around V~0, including the portions that define the PSF shape. The bottom middle panel shows a similar but decreased butterfly issue where U > 33. Finally, U > 100, V < 50 is shown in the bottom right. This is similar to the top right panel, but by extending the clipping in the V dimension, we create a checkerboard ringing.

May 14, 2013

Now that stack to chip unwarping is available, I've been looking into this problem using that as an additional view point. I've again constructed 2D FFT power spectra, using the central 500x500 square of a set of individual cells.

The following table shows a series of three cells on OTA27 at different stages of processing. The top panel shows the "quickmosaic" processing, which removes the overscan, but does no further detrending. Next is the "chip" processed image, which has had all detrends applied, including the PATTERN.ROW, which is applied to cells A and B, but not cell C (as is evident from the remaining horizontal banding). The bottom panel shows the "defect" image, constructed by scaling and subtracting the unwarped stack from the chip image. The stars that fall in the masked regions are visible as over subtracted objects, and the cores of some bright stars remain. In addition, this image presents are relatively pure example of the row-to-row offsets in cell C.


Further analysis will consider only cell A and cell C. The next table shows the power spectra for these two cells for the three stages of processing (the top half, as the bottom is identical due to symmetry). The main features to notice:

  • vertical stripe at k_x ~= 0. This corresponds to DC offsets to a given row (k_x = 0), as well as power in the lowest frequency terms (abs(k_x) < ~5). These are certainly the frequency components that create the row-to-row offsets. There are a few peaks (k_y = 103, 170, 225) whose relative power likely control the widths and amplitudes of the individual stripes. Note that there is no obvious peak at k_y = 250, which would be the case if the row-to-row variations were entirely independent (each row arbitrarily different than its neighbors).
  • smaller horizontal stripe at k_y ~= 0. I believe this represents the corner glows visible on these cells.
  • constellation of three peaks in the +k_x, +k_y quadrant. These peaks are common to both cells, and likely represent a low level noise component that is undetectable to the eye in most exposures. I have not investigated multiple OTAs at this point to determine if this noise signal is found in all devices.

For cell A, the transition from raw to chip processing whows the removal of the first two features listed above. This suggests that the PATTERN.ROW correction is effectively filtering the vertical stripe on this cell (as is visible in the real image). The horizontal bar is removed as well, suggesting that the cause of this is removed by the detrending process. Finally, subtracting the stack removes the "cloud" around k_x = k_y = 0 that is caused by the presence of real objects (objects are all roughly PSF like, and therefore somewhat Gaussian, so their transform is also a Gaussian). The unknown noise pattern represented by the constellation is unchanged, as we do nothing that should alter it.

Cell C shows a similar reduction in the horizontal stripe and object cloud, but as we have not applied PATTERN.ROW to this cell, the vertical stripe is left unchanged. Therefore, an ideal replacement for PATTERN.ROW is an algorithm that removes the vertical stripe in the image power spectrum, while retaining the other properties of the image. One way to visualize what we're doing is shown in the bottom row. This shows the difference of the power spectra between the raw and chip stage (detdiff = log10(raw) - log10(chip) = size of the reduction in power at this frequency). This shows that we are reducing the vertical stripe in cell A, but there is no change in cell C.

Cell A Cell C

July 25, 2012

I looked into Fourier transforms of bias data, based on reports (see http://asd.gsfc.nasa.gov/Bernard.Rauscher/Research/Publications_files/article.pdf) that the read noise can be improved via a Fourier process. The method described assumes that the correlated noise arises due to a time dependent component that is difficult to detect. Therefore, in order to examine this, I asked Sidik for the read out times that could be used to flag each pixel with a given time. Sidik responded that the pixels are read out as

 T(x,y) = x * dt + y * (dt2 + e2 + delta2)

where x and y are the pixel coordinates on a cell, dt is the readout time for a pixel (dt = 2030ns), dt2 is the time to shift rows down (dt2 = 89880ns), e2 is a constant unknown error in dt2 (e2 <= 100ns), and delta2 is a jitter component (delta2 = +/- 10-20ns). As e2 and delta2 are small relative to dt2, they were ignored entirely.

To prepare the pixel data to allow for an FFT, the image mean was subtracted (to zero the DC component), and the pixels were indexed based on dt, such that

 i(x,y) = x + int(y * 89880 / 2030) / y

with all unpopulated values of i zero-padded.

Two OTAs were chosen for study, OTA33 (the "good" OTA), and OTA67 (the "bad" OTA). These two were selected based on the row-by-row bias properties that have been noticed before. A variety of cells on the OTA were checked, all of which seem to have similar results, so cell xy01 is presented here. A bias exposure was selected more or less at random, and the FFT calculated.


This image shows a selection of the FFT power for a section of the lowest frequencies. The spikes correspond with the ~600 pixel spacing of rows.

Zooming in on the lowest frequencies shows that there is basically no difference between the bias from OTA33 and the bias from OTA67. No clear "row-to-row" noise signal seems to pop up, although it is somewhat clear that the 600-pixel spike peaks are lower in the OTA67 bias, suggesting that some of that power is dispersed among other frequencies.

The difference in the power, showing that other than the difference in spike peaks, the two signals are very similar.

Repeating the experiment with a science image instead of a bias. The y-axis is again Power, not the difference. For the science images, at low frequencies, the spikes are spread out on OTA67 compared to OTA33.

This effect is minimal at the highest frequencies, suggesting that in science images, there is a difference in the low frequency.

To satisfy my own curiosity, I calculated the 2d image power. Clockwise from top left, this shows the OTA33 bias, the OTA67 bias, the power in one quadrant of the OTA67 FFT, and the same quadrant of the OTA33 FFT (zero frequencies are at the cross center in the lower left). These images show a similar result to the 1d-FFTs. The spikes that we see in the 1d FFT corresponds with the vertical bar at f_x = 0. Although this bar does spread out in some places (corresponding to the row-to-row structure visible in the bias), there is again no major difference.

Same arrangement of images for a science image. This is somewhat complicated by the fact that astronomical sources do have some effect on the output FFT. However, it is clear that the low-frequency region for OTA67 has significantly more power than the same region in OTA33.