2013-10-25

Validation tests using ppSim images

2013-10-16

Validation tests

2013-09-03

Repeated background subtraction

For a set of 465 i-filter warp images, I ran psphot to obtain a background level estimate. This was done in three passes, to provide the standard background level (what we normally use), a second pass (showing the residual background remaining), and a third pass (checking that the residual had been removed). The following table shows some statistics for these runs:

Pass	Average background	Average background_sigma	Median background	Median background_sigma
1	2.14126	30.48071	1.931080	28.195907
2	-0.0381096	30.51861	-0.030491	28.232888
3	0.00361354	30.52406	0.003190	28.226071

This suggests that we are leaving a residual background on the images, and only after three iterations is this background effectively fully removed.

2013-08-28

As the statistics and fitsio both seem to be working largely as expected, I began looking through the ppStack code to attempt to identify an issue there. It appears that the important sequence in the code is as follows:

1. Input images are matched against the target PSF, and convovled to match that target.
2. After convolution, a single ROBUST_MEDIAN background level is subtracted from the convolved images.
3. The convolved images are written to a temporary disk file.
4. Initial and final convolved stacks are constructed, using the disk files to ease the threading of these steps.
5. For the final unconvolved stacks, the list of input original files is passed to the convolution step. These input files have an unknown (to ppStack), possibly non-zero background level.

I believe this explains the deviations in the background behavior between the convolved and unconvolved stacks. As the warps retain a small non-zero background level, failing to remove this results in a stack background level that scales as the number of exposures/exposure time/input proxy. The solution is to retain a set of background level values for the unconvolved inputs, and optionally apply them during the stacking stage. This will require a bit of development work to implement.

For an illustration of the input dependence, I've plotted up the measured background level for a set of SAS_22 stacks, separated by filter.

Convolved:

Unconvolved:

From the behavior of the convolved stack background levels, it appears that there is a small statistical issue remaining, on the order of 0.25DN/input (0.5DN/input in y?).

2013-08-27

Summary of previous discussion of this issue ==

2013-05-13 ​Confluence page

2013-05-20 ​Non-zero SKY on stacks

2013-06-03 ​Update to non-zero sky issue

2013-06-17 ​More on SKY residuals on stacks

Initial checks

The initial thought for this issue is that the stack sky levels are incorrect either due to an error in the sky statistics, or due to an error in the fitsio data storage. To investigate this, I used the set of warps for a very deep stack (~450 inputs) and pulled statistics out of logs and compared them with direct measurements.

1. Internal background code check. Manually run psphot to fit the background level, and confirm that this matches the value recorded in the stack log.

2. External background code check. Compare the psphot background levels to a Gaussian fit to the histogram of image values.

3. Database check. The GPC1 database contains a background value recorded for the warps. I do not have a good explanation as to why these match so badly.

4. BSCALE check. If the fitsio routines are to blame, it's likely an issue with the quantization. There's a correlation here, but it is more likely due to the fact that the background level is correlated with the distribution width (bg_sigma), as is BSCALE.