Return

March 30, 2012

SDSS match comparison

Using Nigel's SDSS catalog, I matched the noisemap results against the SDSS Stripe 82 data, and did the same detection histograms as before.

The numbers are smaller than the previous table, as Stripe 82 only extends to +/- 1.25 degrees declination.

Reduction	All N=1	All N=12	Good flags N=1	Good flags N=12	QF Perfect N=1	QF Perfect N=12	SDSS Match N=1	SDSS Match N=12
noisemap	153892	367454	126529	362838	65470	351742	10166	345433

March 27, 2012

Normalized single-detection x-position CDF plots for all exposures used in the czw.footprint.noisemap test:

This still shows the ramp effect caused by position dependence in the false positives. I've checked that the noisemap was used, and it was, and an examination of the variance map shows the added variance due to the noisemap so it appears that the detrend creation and the recipes are correct.

Comparing the detection histograms for the various reductions (dropping the oldipp reduction, as it appears to have different flag definitions than later reductions):

The noisemap does remove a significant number of false detections, as shown in the following table, although even with the best exclusions, there is still a ~20% contamination rate.

Reduction	All N=1	All N=12	Good flags N=1	Good flags N=12	QF Perfect N=1	QF Perfect N=12
oldipp	299374	420635	38	30	194169 (ignore flags)	404422 (ignore flags)
czwreduction	298849	420891	264070	414845	165624	401662
continuity	271480	417811	237525	411810	142463	398707
new dark	223680	422322	191487	416508	98467	403106
noisemap	180204	416785	149269	411294	76966	398206

March 22, 2012

Normalized single-detection x-position CDF plots for all exposures used in the czw.footprint.dark test:

Example NOISEMAP detproc image showing the gradients in observed noise as a function of OTA/cell/position. The cells that show jumps in the above CDFs also have gradients in the image noise as shown below.

March 21, 2012

As the observed image noise does not match the noise model as stored in the variance image, a noisemap detrend is being constructed to include the correct position dependent noise. This noisemap measures the local pixel sigma on a 20x20 grid of positions across each cell. Bias frames are used for this measurement, as they do not have any sky or dark signal that would influence this measurement. This noisemap is then used in the processing of science images instead of the read noise, as the direct measurement includes any read noise that would be observed.

March 20, 2012

Variance comparison

To investigate why we seem to have position dependent false positive rates, with biases to one side of the detector cells over the other (as shown below), I made profiles of the image variance in two ways. First, I calculated a mean profile of the chip stage variance image. Second, I calculated an "observed" variance profile by calculating the clipped standard deviation of the chip science image. The results are shown below:

The second image shows the ratio of the "observed" to variance image variances, along with a CDF of the singly detected objects present in the box considered for the profile as a function of x-position. It appears that we find more of these objects (which are believed to be largely false detections) in areas where the variance image underestimates the true image variance. This can be seen in an example cell from OTA67 (from which the above plots are taken): As the x-coordinate increases, the row-by-row variations increase as well, creating a higher-than expected variance.

March 19, 2012

False detection study

March 8, 2012

The new dark has finished, and although there does seem to be a weak trend in the residual slopes with dateobs, these slopes have been greatly decreased relative to the previous dark iteration:

New dark footprint results

I've reprocessed the g-filter footprint data with the new dark (including the continuity correction), and it is currently stacking (label czw.footprint.dark,data_group = czw.20120308.footprint.g). The individual frames appear to have a smoother background, and this is reflected in an improved number of orphan detections. Directly comparing to the previous footprint reductions as before:

shows that the new dark model significantly decreases the number of singly detected objects.

Footprint stack comparison

Comparing the footprint stacked images of skycell.1315.071 from the previous reduction (which enabled continuity correction) shows that the cell level gradients have largely been eliminated.

March 7, 2012

I've reduced a series of dark exposures taken since February 01 2011, as this was the time the current dark model was constructed. I've fit the slope in cell xy10 of OTA67 as a proxy measure of the dark quality, as this cell shows an introduced gradient that appears to contribute to false positives. Shown below is a profile cut across this cell and the others in that OTA and cell row (cells xy10-xy16). The code used to extract the profile does not normalize by the width, so the profile residual values and slopes plotted below need to be divided by 300 to switch to counts. The two profiles shown are from subsequent nights at the beginning of the date range considered, and show different residual patterns. Broadly, we can group nights with increasing gradients with x-pixel on a cell as Mode A, and those with decreasing gradients as Mode B.

The slopes of the first cell were checked for four data samples:

1. A random sampling of dark frames from the night long dark series (night of UTC 2012-03-07).
2. A selection of 30s dark frames from each night, chosen from the second dark sequences of the morning and evening.
3. A random sampling of all parameter space.
4. All darks taken on MJD 5591 and 5592, to investigate inter-night changes.

It appears from these data that prior to about 2011-05-01, the camera (or at least OTA67) flipped between the two dark modes without any obvious pattern. The data from MJD 5591 and 5592 show that even adjacent nights can have different patterns. However, after 2011-05-01, the camera settled into what appears to be a single mode, which has persisted since.

Based on this observation, I'm currently constructing a new dark master from data taken after 2011-05-01, which should fit all exposures after this point. However, there is no clear idea why this mode has become dominant. A voltage change was done around this time to solve the STS Astrometry bug, but OTA67 was not one of those with a change.

March 5, 2012

As discussed in the March 2 status, I've been looking into how the dark model may be introducing the gradients observed in the science data. To see what (if any) functional dependence these gradients had, I selected dark frames used in the construction of the current dark model, and calculated the detproc (overscan corrected, no dark applied) and detresid (overscan and current dark model applied) images. I used only XY67 in this study, as it has clear science image gradients, and was a useful test case. For each variable that could influence the dark, I selected two exposures that spanned the range, while attempting to keep the remaining variables constant. Here are the profile plots for these tests (profile the same as that used in the previous study of XY67: a large 300 pixel box covering the row of cells xy10-xy17):

dateobs

exp_name	dateobs	exp_time	ccd_temp	detproc	detresid
o5605g0022d	2011-02-13	10	-78.455
o5743g0645d	2011-07-01	10	-79.6983

exp_time

exp_name	dateobs	exp_time	ccd_temp	detproc	detresid
o5641g0690d	2011-03-21	0.001	-87.2183
o5638g0035d	2011-03-18	300	-87.0017

ccd_temp

exp_name	dateobs	exp_time	ccd_temp	detproc	detresid
o5630g0494d	2011-03-10	30	-88.4883
o5736g0606d	2011-06-24	30	-77.745

ccd_temp2

exp_name	dateobs	exp_time	ccd_temp	detproc	detresid
o5654g0593d	2011-04-03	10	-86.765
o5612g0382d	2011-02-20	10	-72.1067

30s test

exp_name	dateobs	exp_time	ccd_temp	detproc	detresid
o5666g0013d	2011-04-15	30	-84.43
o5666g0645d	2011-04-15	30	-84.37

Results

The above profiles suggested that 30s darks were somehow different than other exposure times. This seems to be an unfortunate coincidence of the darks selected. Processing a set of evening darks (of all times) from 2011-04-15 all show the downward slopes, and all from 2011-04-03 show the upward slopes. Given this behavior, I selected a set of darks from various dates, processed them, and extracted the slope in the first cell. The following plots show the results of these slopes. The exposure time was chosen to be the same for all of these exposures, and their position in the night was selected to be the same as well (these are the second 30s evening dark taken on each date between the beginning and end of the current dark model inputs).

I've requested that a sequence of darks be taken to more finely cover the range of exposure times, and will use this data to develop a more complete dark model that will hopefully not introduce any residual gradients.