Background Regeneration

Instructions on how to run the code

NGC4303

The following two images show the g (top image) and z (bottom) results for the bad galaxy Nigel noted in ​Taiwan. The columns show the model, the restored stack, and the original stack from left to right. The top row is the PV1 result, and the bottom is the PV2 results.

As Nigel mentioned on the DRAVG telecon page, PV2 greatly improves the stack quality. However, the pattern related artifacts are not improved significantly by the new background restoration code. This is disappointing, but somewhat expected. First, the pattern artifacts are on a scale reasonably small relative to the background sampling, which limits how much of that signal is retained in the background model. Secondly, the poor model determined for the pv1/g stack suggests that when the data is too noisy, almost all information is clipped out of the output stack background model.

Example restoration: 2014-04-03

An implementation of the method described below is in place, with the results shown above for M33 (from the LAP.ThreePi.20130717 processing, explaining some of the improved masking relative to the set below. Displayed above are the results in all five filters (sorted alphabetically instead of by wavelength) with the composite background model on top, the restored image in the middle, and the original stack image on the bottom.

All models used the same OTA dependent models, derived from the SAS g-band data. This was chosen to ensure the models would not be contaminated with excessive noise, and to minimize the effect of uneven sky levels between exposures.

The gri results look promising, although there are some bad samples in the r filter. This is even more pronounced in the the y and z filters, creating some checkerboard patterns. These features tend to cluster near the edge of the image, as the current implementation does not sample outside of the stack field of view, resulting in more dramatic ramps.

Color

Gene suggested making RGB images from these restored images. I've done so in DS9, using blue for the g-filter, red for the r-filter, and green for the i-filter (chosen to force the HII regions to show up as red). I've tried two different color scalings.

image	scaling	range	filters (RGB)
A	log scale	max set to 1e-7 histogram peak	rig
B	sqrt scale	min set to background sky value, max set to 1e-6 histogram peak	rig
C	log scale	min as before, max set to 15k (a bright star value)	rgz

New thoughts: 2014-01-13

After the Taiwan meeting, I spent some time thinking about this problem again, and came to the following conclusions.

The reason the background restored images do not look usable below is due to the fact that the background is comprised of more components that the original assumption allowed. I'm now fairly confident that the background model solution is

B_model = B_OTA + B_astronomical + B_transparency

where B_OTA is an OTA-specific background model that is created by the PATTERN.CONTINUITY code. This forces all the cells to have a co-planar background, in order to prevent cell-edge background residuals. This B_OTA is assumed to be stable, and some quick checks indicate this isn't completely unreasonable. Therefore, in order to combine the background models, this needs to be removed from the model (possibly by simply removing the common plane calculated for each OTA).

The next conclusion was that it's probably excess work to construct warp stage models. We can instead use the chip stage products and transform the individual points from each to the warp grid. This will create an irregular grid of points, which leads me to suggest using thin plate splines to do the interpolation to the stack grid. This interpolation also allows for outlier rejection by constraining the allowed bending energy of the spline.

Gene has mentioned that B_transparency may also complicate this. However, if we fold this into the ppStack code, we can use the fact that we measure the relative zeropoints for each input, and filter the inputs to ensure that the differences in B_transparency do not skew the calculated B_astronomical.

Status: 2012-12-14

Checking the results of the automatically generated background models made it clear that these models were doing more harm than good. This figure shows the automatic models (top row) along with some manually created models (bottom row). The large color jumps in the automatic models are on the order of ~100 counts, whereas for the manual models it is less than 10.

The change in the manual models is that each input warp model is matched to the model from the warp with the best zeropoint by measuring the offset and scale necessary for reference = offset + scale * input. This ensures that the models have similar zeropoints, and prevents the large discontinuities when some inputs are masked.

This image shows the input stack (asinh scaling removed) in the top row, along with the background restored version in the bottom row. The changes are not very dramatic, which is somewhat expected given the reasonably smooth manual background models.

I've made all the data used in this test available on the rsync server under watersc1-demo-data/bkgrest/m33. Each filter is in a separate subdirectory, with the following files:

g
|-- bkgrest_convolved.fpa.fits                                        ppBackground output/convolved/image
|-- bkgrest_convolved.fpa.mk.fits                                                                   mask
|-- bkgrest_unconvolved.fpa.fits                                                          unconvolved/image
|-- bkgrest_unconvolved.fpa.mk.fits                                                                   mask
|-- new_model.fits                                                    manual background model
|-- stack_data
|   |-- RINGS.V3.skycell.1935.063.stk.1827279.fits                    convolved stack image
|   |-- RINGS.V3.skycell.1935.063.stk.1827279.mask.fits                               mask
|   |-- RINGS.V3.skycell.1935.063.stk.1827279.mdl.fits                automatic background model
|   |-- RINGS.V3.skycell.1935.063.stk.1827279.unconv.fits             unconvolved stack image
|   |-- RINGS.V3.skycell.1935.063.stk.1827279.unconv.mask.fits                          mask
|   |-- linear_convolved.fits                                         stack image, asinh scaling removed
|   `-- linear_unconvolved.fits
`-- warp_models
    |-- o5482g0379o.239505.wrp.655200.skycell.1935.063.mdl.fits       input warp background model
    |-- o5482g0379o.239505.wrp.655200.skycell.1935.063.scaled.fits    input warp background model with offset and scale applied
    |-- o5482g0380o.239506.wrp.655285.skycell.1935.063.mdl.fits
    |-- o5482g0380o.239506.wrp.655285.skycell.1935.063.scaled.fits
    |-- o5482g0398o.239524.wrp.655201.skycell.1935.063.mdl.fits
    |-- o5482g0398o.239524.wrp.655201.skycell.1935.063.scaled.fits
    |-- o5482g0399o.239526.wrp.655287.skycell.1935.063.mdl.fits
    |-- o5482g0399o.239526.wrp.655287.skycell.1935.063.scaled.fits
    |-- o5498g0110o.246616.wrp.655293.skycell.1935.063.mdl.fits
    |-- o5498g0110o.246616.wrp.655293.skycell.1935.063.scaled.fits
    |-- o5498g0127o.246632.wrp.655299.skycell.1935.063.mdl.fits
    |-- o5498g0127o.246632.wrp.655299.skycell.1935.063.scaled.fits
    |-- o5805g0537o.385621.wrp.655212.skycell.1935.063.mdl.fits
    |-- o5805g0537o.385621.wrp.655212.skycell.1935.063.scaled.fits
    |-- o5805g0544o.385628.wrp.655217.skycell.1935.063.mdl.fits
    |-- o5805g0544o.385628.wrp.655217.skycell.1935.063.scaled.fits
    |-- o6219g0269o.534377.wrp.655222.skycell.1935.063.mdl.fits
    |-- o6219g0269o.534377.wrp.655222.skycell.1935.063.scaled.fits
    |-- o6219g0287o.534385.wrp.655226.skycell.1935.063.mdl.fits
    `-- o6219g0287o.534385.wrp.655226.skycell.1935.063.scaled.fits

Status: 2012-12-12

All the code is finished and committed to the trunk, but I am still checking the outputs.

Chip Stage

The background models are generated at the chip stage, and will be retained permanently, even after the other image data has been cleaned. These are small (13x13) images that contain the background samples that can be interpolated to construct the smooth background model that was subtracted from the science images.

Warp Stage

Previously, we have not generated warp stage background models. The new trunk (and next tag) will include the code to generate these models. They are similar to the chip stage models, which are interpolated onto a finer grid, and warped to match the science data. This increase in resolution is necessary to properly account for the inter-chip gaps. These models are still substantially smaller than the science images (~50x50).

Stack Stage

To construct the stack stage background model, all input warp background models are loaded, and the median value is selected for each model pixel. NAN and masked regions are excluded in this median, which should represent the "average" background model that was removed from all input images in the stack.

Applying a background model to an image

Generating a background restored image can be done using the ppBackground command:

 ppBackground restored_version -background input.mdl.fits -image input.fits -mask input.mask.fits -recipe PPBACKGROUND STACK

This can be done to either the warp or stack stage products.

Status

All the code is finished and committed to the trunk, but I am still checking the outputs.