2013-04-16

While testing the input FWHM cut, I discovered that an image that appeared valid by eye was being flagged during the PSF matching stage as having a overly deviant chi^2 value. A study of the code revealed that the chi^2 value used is calculated in the following way:

1. For all valid stamps in the PSF match, the flux and squared residual are fit with a quadratic function (x0 + x1 * flux + x2 * flux^2).
2. This quadratic function is subtracted from the squared residual, providing a flux-neutral measurement of the squared residuals.
3. The RMS scatter of these remaining residuals is saved as the chi^2 value for the PSF match (there's a sum over all used kernels, but the example case has only a single kernel).

The following table shows the set of PSF match values, along with the parameters of the quadratic function. This table shows that we seem to be needlessly matching PSFs for images already rejected at the input stage. In addition, there appear to be 2 iterations of this function (pmSubtractionRejectStamps) being called for images 7+.

Nstamp	mean	rms	x0	x1	x2	Image	Rejected?
27	-1e-06	60.7719	19.8792	0.000116288	-9.31452e-13	0
119	1e-06	64.4908	57.5848	-8.45089e-05	3.59486e-11	1
26	-1e-06	43.3212	27.068	5.53666e-05	1.7642e-13	2
107	-2e-05	2756.68	1273.49	-0.00181565	1.34383e-10	3	At Input Stage
106	-9e-06	1105.5	614.941	-0.000711782	1.01758e-10	4	Yes
102	-1e-06	105.827	18.7579	0.000123878	-3.35511e-12	5	At Input Stage
101	-0	51.924	27.2631	7.72215e-05	1.07841e-12	6
100	-6e-06	738.86	222.832	-0.000305871	2.12201e-10
99	8e-06	273.861	95.194	-0.000156691	2.03263e-10	7	Yes
124	4e-06	226.235	18.6949	0.000113144	4.15584e-12
122	2e-06	92.0136	18.5325	0.000110472	3.66559e-13	8
123	1e-06	135.964	-1.48911	0.000144198	-2.92106e-12
121	1e-06	77.5818	7.60706	0.000104179	7.44259e-14	9
103	3e-06	330.224	-70.9388	0.000411208	-5.42577e-12
101	-3e-06	53.8796	-7.55995	0.000204406	5.83814e-12	10
116	0	55.2009	15.2849	4.40914e-05	6.08214e-12
116	-0	23.573	32.1768	1.87319e-06	8.4501e-12	11

We can now plot up the data and quadratic fits to validate their quality:

Image	Plot
0
1
2
3
4
5
6
7a
7b
8a
8b
9a
9b
10a
10b
11a
11b

2013-04-08

The new proposed algorithm for the initial ppStack FWHM cuts uses a Gaussian mixture model analysis to attempt to determine the best separation between a possible two component distribution.

• If Punimodal > 0.5 (this distribution is best described by a single Gaussian mode); limit = MAX(FWHM_i)
• Otherwise, assume two Gaussians best describe it
  • If pi1 > pi2 (the smaller FWHM group has the majority of the samples); limit = m1 + k * sigma1
  • If pi2 > pi1 (the larger FWHM group has the majority of the samples); limit = m2 + k * sigma2
  • If abs(pi1 - pi2) < separation threshold (taken as ~0.2. assume this is nearly unimodal); limit = m2 + k * sigma2
• If N <= 4; limit = MAX(FWHM_i)
• If limit > high_threshold; limit = high_threshold
• If limit < low_threshold; limit = low_threshold

The results of this algorithm change are shows below, displaying the prior results along with this proposed change. The first figure shows the fraction of input images accepted into the stack for each filter (g,r,i,z,y). The second shows the distribution of the FWHM limits.

2013-04-04

I have the first set of ppStack changes in place and merged to the trunk. This includes the fix for the PSF target that was resulting in excess detections. I have run a test on the same SAS stack that Ken and Heather were using for IRAF tests. This test has the SAFE flag set to FALSE, allowing single inputs to contribute.

2013-03-07

For the Arp 220 example, of the six input warps that contain the galaxies, two are excluded due to large FWHM, with another warp having a FWHM close to the hard limit of 7.5. This input was excluded in the original processing, but not in my reprocessing of the warps (FWHM for the reprocessing was 7.48). Because of this, only three (four) warps are available. We currently enforce the "safe combination" rules, which do not allow a pixel to be populated from only one input exposure. The following images show the input warps that contain Arp 220, with the top showing the reprocessed set, and the bottom showing the original warps. The red X show which inputs were excluded due to their FWHM being large. Below that is a comparison of the stacks obtained for these warps with the default combination (left panels) and with the FWHM limit set to 100, which allows all warps to contribute (right panels).