CZW: I'm rerunning some of the tests here due to an issue with how they were initially processed.

2013-10-23

Comparison of ipp-20130712 and eam-branch-XXX

After plotting the data from 2013-10-21, I wanted to check that my ipp-20130712 processing truly contained the changes that Gene had made to psphot. To confirm this, I ran the test sample of trails using his test arrangement with both code branches. As shown in the table below, the two versions produce effectively identical trail length results.

CMF file	index	ipp-tag L	L_MOPS	difference	eam-branch L	L_MOPS	difference
./o6392g0238o.s0.fpa.cmf	0	25.5520420074	69.5	-43.9479579926	25.2005901337	69.5	-44.2994098663
./o6392g0238o.s0.fpa.cmf	1	3.18367624283	69.5	-66.3163237572	2.64567708969	69.5	-66.8543229103
./o6392g0238o.s0.fpa.cmf	2	25.1396484375	69.5	-44.3603515625	25.189573288	69.5	-44.310426712
./o6392g0238o.s0.fpa.cmf	3	4.92207574844	69.5	-64.5779242516	5.18088006973	69.5	-64.3191199303
./o6392g0238o.s1.fpa.cmf	0	24.3006839752	69.5	-45.1993160248	25.1601676941	69.5	-44.3398323059
./o6392g0238o.s1.fpa.cmf	1	34.5275001526	69.5	-34.9724998474	33.7674255371	69.5	-35.7325744629
./o6392g0238o.s1.fpa.cmf	2	3.02041530609	69.5	-66.4795846939	3.02506089211	69.5	-66.4749391079
./o6392g0258o.s0.fpa.cmf	0	45.983745575	69.5	-23.516254425	46.0084724426	69.5	-23.4915275574
./o6392g0278o.s0.fpa.cmf	0	3.1019051075	69.5	-66.3980948925	3.10278606415	69.5	-66.3972139359
./o6392g0278o.s0.fpa.cmf	1	39.2214813232	69.5	-30.2785186768	39.3013000488	69.5	-30.1986999512
./o6392g0278o.s0.fpa.cmf	2	43.3110542297	69.5	-26.1889457703	43.5664291382	69.5	-25.9335708618
./o6392g0275o.s0.fpa.cmf	0	38.3529281616	48.8	-10.4470718384	38.3323478699	48.8	-10.4676521301
./o6392g0295o.s0.fpa.cmf	0	3.11816740036	48.8	-45.6818325996	3.10649251938	48.8	-45.6935074806
./o6392g0295o.s0.fpa.cmf	1	3.54833292961	48.8	-45.2516670704	3.52356886864	48.8	-45.2764311314
./o6392g0295o.s0.fpa.cmf	2	42.474155426	48.8	-6.325844574	42.4663696289	48.8	-6.3336303711
./o6392g0295o.s0.fpa.cmf	3	3.51850223541	48.8	-45.2814977646	3.28675270081	48.8	-45.5132472992
./o6391g0270o.s0.fpa.cmf	0	6.88583469391	68.2	-61.3141653061	6.86001157761	68.2	-61.3399884224
./o6391g0270o.s1.fpa.cmf	0	6.94266891479	68.2	-61.2573310852	6.90459632874	68.2	-61.2954036713
./o6391g0292o.s0.fpa.cmf	0	46.8225631714	68.2	-21.3774368286	46.8168869019	68.2	-21.3831130981
./o6391g0314o.s0.fpa.cmf	0	49.1394805908	68.2	-19.0605194092	49.116607666	68.2	-19.083392334
./o6389g0193o.s0.fpa.cmf	0	10.2764625549	47.3	-37.0235374451	10.0571022034	47.3	-37.2428977966
./o6389g0193o.s0.fpa.cmf	1	17.5834236145	47.3	-29.7165763855	17.7657604218	47.3	-29.5342395782
./o6389g0215o.s0.fpa.cmf	0	36.3150405884	47.3	-10.9849594116	36.3397865295	47.3	-10.9602134705
./o6389g0215o.s0.fpa.cmf	1	7.93172502518	47.3	-39.3682749748	7.82549095154	47.3	-39.4745090485
./o6389g0259o.s0.fpa.cmf	0	11.1794672012	47.3	-36.1205327988	11.1941289902	47.3	-36.1058710098
./o6389g0259o.s0.fpa.cmf	1	7.39956760406	47.3	-39.9004323959	7.54179620743	47.3	-39.7582037926
./o6389g0259o.s0.fpa.cmf	2	5.88574028015	47.3	-41.4142597198	5.9170088768	47.3	-41.3829911232
./o6389g0259o.s0.fpa.cmf	3	38.5486068726	47.3	-8.7513931274	38.2509269714	47.3	-9.0490730286
./o6389g0259o.s0.fpa.cmf	4	11.5479698181	47.3	-35.7520301819	11.6650266647	47.3	-35.6349733353
./o6389g0259o.s0.fpa.cmf	5	9.03969860077	47.3	-38.2603013992	9.1590385437	47.3	-38.1409614563
./o6389g0205o.s0.fpa.cmf	0	25.7934494019	27.5	-1.7065505981	25.7746906281	27.5	-1.7253093719
./o6389g0227o.s0.fpa.cmf	0	24.7478179932	27.5	-2.7521820068	24.7023792267	27.5	-2.7976207733
./o6389g0249o.s0.fpa.cmf	0	25.8140125275	27.5	-1.6859874725	25.8200283051	27.5	-1.6799716949
./o6389g0249o.s1.fpa.cmf	0	25.8235092163	27.5	-1.6764907837	25.9838275909	27.5	-1.5161724091

ipp-20130712 trail test results

Based on the check that the ipp-20130712 tag does contain the necessary psphot changes, I've compiled the list of flux/length/angle values for the 0010_trails_ground_truth.txt fit sample. The following table contains the exposure/skycell values for the trails listed in this file, along with the index for extended fit objects around the expected location. Following this are the flux/length/angle values. The IPP angle data has been converted to degrees to match the MOPS values. However, the odd distribution of IPP angle values suggests to me that the angle value recorded in the CMF file may not truly represent the same concept as the MOPS trail angle.

Due to the possibility that psphot has split a single trail into multiple objects, each of which may have a fit, I have included all extended fit objects within ~36 arc seconds of the expected location. This table may also have multiple fits for the same source trail, as trails that fall within the overlap region of the skycells were fit on all skycells that they exist on.

exp_name

skycell_id

index

R

RA_MOPS

DEC_MOPS

RA_IPP

DEC_IPP

flux_MOPS

flux_ipp

L_expect_MOPS

L_veres

L_ipp

angle_MOPS

angle_ipp

o6534g0425o	skycell.032	0	4.73474e-05	334.8249634669	-0.2491540302	334.8249692940	-0.2492010177	1730.74	2280.31	16.36	13.14	13.25	180.00	-87.35
o6520g0329o	skycell.036	1	0.000113854	333.3161206075	-0.4001320878	333.3160870170	-0.4002408744	3568.90	4734.75	13.52	14.95	15.05	27.35	-64.07
o6538g0261o	skycell.1228.032	0	6.19412e-05	344.8488993807	-6.6066866393	344.8488729560	-6.6067426612	7040.99	8048.65	43.73	42.03	42.58	154.38	64.60
o6506g0467o	skycell.025	2	7.07829e-05	333.9698254868	-0.7509315681	333.9698772420	-0.7509798550	2890.01	4444.63	16.21	12.82	13.73	23.84	-63.66
o6506g0467o	skycell.024	0	0.00593998	333.9698254868	-0.7509315681	333.9640007950	-0.7497669719	2890.01	35905.10	16.21	12.82	21.56	23.84	-38.89
o6520g0325o	skycell.042	0	7.03597e-05	334.8905148181	0.2605107092	334.8904649130	0.2604611111	2740.86	9131.66	17.26	10.57	20.76	116.32	32.60
o6513g0468o	skycell.075	0	1.27888e-06	352.2698350848	0.5561100456	352.2698358820	0.5561090456	6474.23	7753.83	21.21	21.71	21.79	148.93	58.28
o6513g0300o	skycell.1047.096	0	1.69009e-05	330.8337948710	-12.4115068095	330.8337951010	-12.4114899102	2134.62	1765.98	15.88	12.96	14.92	76.59	-17.11
o6513g0300o	skycell.1047.086	0	1.26342e-05	330.8337948710	-12.4115068095	330.8337913590	-12.4114946732	2134.62	3012.34	15.88	12.96	13.45	76.59	-17.27
o6520g0326o	skycell.046	0	0.000180428	333.4237588936	0.2579521730	333.4235894310	0.2580141134	7162.12	10697.20	18.77	20.13	20.08	114.13	24.55
o6513g0461o	skycell.026	1	3.21561e-05	333.3754432749	-0.9252219897	333.3754443070	-0.9251898502	7036.24	20899.40	16.01	13.99	25.37	120.60	36.55
o6530g0195o	skycell.057	0	5.3773e-05	333.1121916547	0.2757630986	333.1121764380	0.2758146737	3064.19	30980.60	12.38	11.98	9.63	114.72	21.33
o6530g0195o	skycell.047	0	5.59436e-06	333.1121916547	0.2757630986	333.1121916130	0.2757575044	3064.19	44480.80	12.38	11.98	9.93	114.72	24.12
o6516g0385o	skycell.083	0	4.62758e-05	334.5864569983	1.5418775126	334.5864792610	1.5419180813	4234.67	8203.63	15.51	16.85	15.54	33.73	-54.47
o6519g0489o	skycell.045	3	1.01678e-05	333.8032284110	0.0796392809	333.8032256990	0.0796294815	6955.41	10522.40	19.26	17.29	16.89	116.24	25.31
o6522g0367o	skycell.2284.046	1	0.000761114	330.9841607004	45.9391807053	330.9834576130	45.9388892214	67156.08	37878.40	34.50	35.16	16.35	57.47	-30.98
o6520g0324o	skycell.042	0	2.11746e-05	334.8918900343	0.2599152873	334.8919086790	0.2599052504	4631.95	8622.54	17.26	18.47	17.19	112.37	25.06
o6534g0428o	skycell.032	0	9.45728e-05	334.8243605286	-0.2532694412	334.8243659590	-0.2531750244	1072.66	1693.16	16.36	13.29	12.88	12.60	-81.97
o6535g0478o	skycell.041	1	9.89073e-06	353.5278173532	-0.6575027194	353.5278166430	-0.6575125846	5309.39	11462.70	18.85	16.91	17.79	21.22	-62.12
o6535g0478o	skycell.042	1	3.76824e-05	353.5278173532	-0.6575027194	353.5277893320	-0.6575279141	5309.39	10712.50	18.85	16.91	18.68	21.22	-64.14
o6520g0323o	skycell.036	0	0.000101973	333.3186398787	-0.3946247407	333.3186221200	-0.3947251553	4720.30	8129.15	13.52	15.04	8.41	22.08	-68.45
o6538g0142o	skycell.053	0	8.06451e-05	334.4226087889	0.4930885178	334.4225293090	0.4930748585	7351.21	13944.70	16.55	6.50	3.13	115.38	12.06
o6534g0308o	skycell.0958.081	0	4.46979e-05	327.9886783618	-16.5601752257	327.9886855940	-16.5601311168	2921.30	2690.73	29.47	30.87	29.56	38.42	-50.55
o6516g0382o	skycell.083	0	5.15161e-05	334.5885168618	1.5450019654	334.5885444490	1.5450454723	4281.74	4697.93	15.51	16.74	18.24	34.40	-55.92
o6508g0350o	skycell.0873.092	0	0.00828483	331.1632034473	-20.0217368875	331.1667115470	-20.0142314470	10211.30	10769.40	48.80	49.59	4.00	130.24	70.66
o6540g0162o	skycell.022	0	0.000172419	334.8283989123	-0.8650867819	334.8284762790	-0.8649326957	5975.93	8512.42	20.64	17.78	18.38	23.72	-66.44
o6528g0447o	skycell.043	0	1.19322e-05	334.5376145954	0.0013937997	334.5376171900	0.0014054464	30322.66	25898.30	16.74	14.79	15.32	111.45	20.54
o6536g0464o	skycell.076	0	8.99261e-06	35.2745755776	-3.2839948200	35.2745843059	-3.2839926558	5807.19	18380.50	14.83	8.51	19.60	150.26	67.67
o6528g0448o	skycell.043	0	3.72984e-05	334.5360395899	0.0019243782	334.5360186210	0.0019552242	18499.53	15249.90	16.74	14.37	17.06	107.00	19.86
o6508g0127o	skycell.015	0	5.92126e-05	242.4306925220	53.6334619536	242.4307358830	53.6335022768	13498.69	10126.60	42.57	15.14	17.32	169.11	80.50
o6536g0451o	skycell.074	0	0.000307443	352.5356031867	0.7862988513	352.5354821940	0.7860162171	5513.12	3484.16	19.04	15.43	13.03	21.67	-59.43
o6536g0451o	skycell.084	1	2.32227e-05	352.5356031867	0.7862988513	352.5356152060	0.7862789810	5513.12	8548.25	19.04	15.43	16.25	21.67	-66.30
o6519g0492o	skycell.045	0	3.34842e-05	333.7993567774	0.0814677268	333.7993318550	0.0814900889	7139.41	9455.61	19.26	15.54	16.81	129.81	38.21
o6536g0458o	skycell.086	0	2.73791e-05	351.7277190825	0.8975516311	351.7277342340	0.8975288266	9213.20	11397.50	20.87	18.00	18.93	20.40	-69.91
o6513g0472o	skycell.048	0	2.11654e-05	351.0088260831	-0.5534961481	351.0088078510	-0.5534853980	4425.86	5883.01	17.52	16.01	16.41	144.89	54.53
o6520g0322o	skycell.045	0	0.000205211	333.7248167441	0.1754294480	333.7246676040	0.1752884906	5011.05	8281.34	18.61	16.52	16.95	35.89	-53.10
o6520g0328o	skycell.036	1	7.12847e-05	333.3165155875	-0.3992668645	333.3165020980	-0.3993368612	3212.55	5361.43	13.52	13.65	11.17	22.76	-72.38
o6522g0350o	skycell.2284.046	2	8.57995e-06	330.9239048632	45.9095775070	330.9238977330	45.9095727346	60875.60	74109.40	33.64	30.26	31.37	53.94	-35.08
o6538g0145o	skycell.041	0	0.0002407	335.2137065385	0.0239765253	335.2135476660	0.0237957048	3581.63	7609.06	18.97	1.20	8.12	62.51	-68.21
o6519g0507o	skycell.053	3	1.15309e-05	36.2724517898	-4.0298922447	36.2724413381	-4.0298971155	8386.84	7678.79	15.70	25.37	16.37	112.08	37.82
o6519g0507o	skycell.054	0	1.34911e-05	36.2724517898	-4.0298922447	36.2724633353	-4.0298992240	8386.84	9238.33	15.70	25.37	8.44	112.08	31.51
o6506g0465o	skycell.025	3	1.22285e-05	333.9709288732	-0.7485155838	333.9709407280	-0.7485185837	3425.68	4935.89	16.21	16.95	17.72	23.42	-65.38
o6506g0465o	skycell.024	3	1.79985e-05	333.9709288732	-0.7485155838	333.9709468410	-0.7485166347	3425.68	3512.78	16.21	16.95	17.40	23.42	-66.07
o6522g0400o	skycell.037	0	0.000166903	35.0015784520	-4.9655424903	35.0017167029	-4.9656359967	3681.78	6680.88	15.45	16.17	16.28	127.64	38.31
o6513g0473o	skycell.075	1	3.38906e-05	352.2658976587	0.5626687151	352.2659136490	0.5626388340	5737.87	9998.26	21.21	18.43	16.33	148.86	61.18
o6513g0474o	skycell.083	1	0.000217133	352.7936180114	0.8341065016	352.7935884290	0.8338913928	3102.28	5632.77	17.55	19.65	17.40	1.08	-82.00
o6513g0474o	skycell.048	0	5.31946e-05	351.0101321229	-0.5557906864	351.0101721730	-0.5557556771	3918.99	6146.29	17.52	15.31	15.79	153.62	65.92
o6513g0464o	skycell.031	1	5.23332e-05	335.2405267218	-0.2036347419	335.2404743940	-0.2036354917	7471.43	15678.20	14.19	9.38	3.78	114.54	6.51
o6538g0440o	skycell.057	0	0.000185879	34.9714771758	-3.8804560396	34.9713828302	-3.8802958838	2432.61	4902.86	11.71	13.89	7.37	150.81	55.57
o6508g0329o	skycell.0959.004	0	3.62706e-05	331.0905492437	-19.9635879044	331.0905340270	-19.9635549801	8756.38	10485.10	48.80	50.21	51.62	129.62	39.56
o6536g0452o	skycell.086	2	2.93206e-05	351.7307567304	0.9063104206	351.7307302200	0.9062978947	9703.21	13223.70	20.87	19.97	19.32	17.42	-76.50
o6536g0448o	skycell.042	0	2.42146e-05	335.1426040518	0.2587375050	335.1426281810	0.2587395369	2474.03	7121.82	17.10	6.20	11.68	112.31	8.83
o6519g0505o	skycell.053	0	2.74041e-05	36.2702689750	-4.0283006181	36.2702594997	-4.0282749042	3690.30	13251.90	15.70	7.33	19.44	120.53	39.00
o6519g0505o	skycell.054	0	4.60829e-05	36.2702689750	-4.0283006181	36.2703138521	-4.0283110907	3690.30	11152.70	15.70	7.33	18.32	120.53	41.47
o6519g0488o	skycell.056	0	3.78374e-05	333.2191409642	0.4265666737	333.2191058730	0.4265808257	6421.02	7588.58	13.99	15.49	15.27	107.52	17.23
o6519g0488o	skycell.046	0	4.86837e-05	333.4774256537	0.0496105352	333.4774740620	0.0496157063	4386.01	7346.69	14.93	8.67	11.47	105.37	17.20
o6522g0397o	skycell.037	0	5.78439e-05	34.9989144642	-4.9636135148	34.9989537920	-4.9636559322	2485.28	3800.70	15.45	17.30	17.09	122.55	34.12
o6544g0175o	skycell.042	2	1.89072e-05	334.8320977949	0.1981124608	334.8321166100	0.1981143246	6591.16	11120.60	16.34	14.87	14.97	112.86	21.55

table break line.

Angle
Angle vs L
Flux
Flux vs L
L1
L2

2013-10-21

The following table shows the results of fitting the MOPS test trail data set using three different runs. The left column shows the default "current tag" version of the code. The right column shows the comparison between the "czw" run (current trunk, with possible psStats errors I've introduced) and "ipp" run (current tag) with the appropriate LMM options toggled. Plotted are the angle, flux, and length (L1 and L2 being the two length values supplied by MOPS).

The trail window does appear to be updated in all runs, as messages like:

update window : 1144.585815 1019.293091 : 14.631915 -> 29.263830
update window : 1219.900879 1063.883179 : 5.000000 -> 10.000000
update window : 1219.900879 1063.883179 : 10.000000 -> 20.000000
update window : 1241.800659 1062.214478 : 47.373280 -> 94.746559

are printed.

psphot call	-recipe PSPHOT DIFF	-recipe PSPHOT DIFF -Db LMM_FIT_CHISQ_CONVERGENCE F -Di LMM_FIT_GAIN_FACTOR_MODE 2
Angle
Angle vs L
Flux
Flux vs L
L1
L2

2013-02-27

The initial guess for L is in general too large for point-like objects. First, there's an error in the code that chooses between the elliptical major axis and the first radial moment, resulting in a large initial guess. Fixing this doesn't completely work, so the idea was to introduce an offset, such that an object with a size similar to the width is probably point like, so we should normalize by subtracting that width off. This shrinks the length again, but insufficiently. Looking at large numbers of objects (click on full skycell tables in attachments below), it's clear that dL/dIteration is almost universally positive, suggesting an issue with the calculation (typo in the code?) or algorithm.

Example: Original:

    float size = (axes.major > source->moments->Mrf) ? axes.major : source->moments->Mrf;
    PAR[PM_PAR_LENGTH] = 1.5*2.35*size; // a tophat of length L has L = 1.5 * 2.35 * sigma                                  

#X         Y           ID    Mxx       Mxx      Myy       Mr1        EXT_F     EXT_L     EXT_W    EXT_Q     PSF_CHISQ   PSF_NDOF    EXT_CHISQ  EXT_NDOF
77.558716 2633.799072 -98 : 11.551100 1.738350 10.857200 6.066510 : 16.974348 32.950000 1.984060 0.725825 : 3286.150000 2600.000000 3207.900000 2892.000000
       #IT F         L         W        Q          oChi        nChi        delt     dlin      rho      lambda
         0 62.666386 21.384449 1.984059 0.686898 : 5817.757812 3362.864990 nan 1238.471436 1.982196 0.001000
         1 16.869297 22.949951 1.984059 0.686511 : 3362.864990 3327.216797 0.849738 22.782276 1.564734 0.000250
         2 16.829279 24.949951 1.984059 0.689350 : 3327.216797 3292.971436 0.012339 20.549458 1.666485 0.000063
         3 16.904570 26.949951 1.984059 0.697310 : 3292.971436 3260.270020 0.011854 18.246912 1.792162 0.000016
         4 16.923841 28.949951 1.984059 0.710121 : 3260.270020 3231.121338 0.011319 16.398548 1.777516 0.000004
         5 16.916101 30.949951 1.984059 0.721472 : 3231.121338 3207.896484 0.010090 12.160172 1.909912 0.000001
        XX 16.974512 32.949951 1.984059 0.725825 : 3207.896484

Sqrt:

    float size = (axes.major > sqrt(source->moments->Mrf)) ? axes.major : sqrt(source->moments->Mrf);                 
    PAR[PM_PAR_LENGTH] = 1.5*2.35*size; // a tophat of length L has L = 1.5 * 2.35 * sigma                                  

#X         Y           ID    Mxx       Mxx      Myy       Mr1        EXT_F     EXT_L     EXT_W    EXT_Q      PSF_CHISQ   PSF_NDOF    EXT_CHISQ  EXT_NDOF
77.559525 2633.799561 -103 : 11.602900 1.715950 10.896300 6.099250 : 17.214936 29.877100 1.992040 0.654534 : 3400.710000 2600.000000 3371.800000 2917.000000
       #IT F         L         W        Q          oChi        nChi        delt     dlin      rho      lambda
         0 62.998722 12.710316 1.992041 0.683864 : 4785.610352 3625.804443 nan 586.573059 1.977257 0.001000
         1 19.728992 13.877067 1.992041 0.690744 : 3625.804443 3593.194092 0.398012 14.094786 2.313647 0.000250
         2 19.357067 15.877067 1.992041 0.688212 : 3593.194092 3559.262207 0.011191 14.204083 2.388882 0.000063
         3 18.790325 17.877068 1.992041 0.654087 : 3559.262207 3522.943359 0.011644 17.820257 2.038065 0.000016
         4 18.215334 19.877068 1.992041 0.623698 : 3522.943359 3488.975830 0.012464 18.180664 1.868333 0.000004
         5 18.069387 21.877068 1.992041 0.611393 : 3488.975830 3458.826904 0.011657 14.959837 2.015325 0.000001
         6 18.013340 23.877068 1.992041 0.616009 : 3458.826904 3429.208984 0.010346 14.725613 2.011320 0.000000
         7 17.703453 25.877068 1.992041 0.629182 : 3429.208984 3399.503418 0.010164 16.940977 1.753474 0.000000
         8 17.366875 27.877068 1.992041 0.643639 : 3399.503418 3371.803467 0.010194 17.004745 1.628954 0.000000
        XX 17.215097 29.877068 1.992041 0.654534 : 3371.803467

Adjustment:

    float size = (axes.major > sqrt(source->moments->Mrf)) ? axes.major : sqrt(source->moments->Mrf);                 
    if (size > psfAxes.major) { size -= psfAxes.major; }                                                              
    else { size = psfAxes.major; }                                                                                        
    PAR[PM_PAR_LENGTH] = 1.5*2.35*size; // a tophat of length L has L = 1.5 * 2.35 * sigma                                  
#X         Y           ID    Mxx       Mxx      Myy       Mr1        EXT_F     EXT_L     EXT_W    EXT_Q   PSF_CHISQ   PSF_NDOF    EXT_CHISQ  EXT_NDOF
77.559189 2633.799561 -98 : 11.558200 1.736740 10.870700 6.072910 : 5.257746 7.728160 1.978440 0.787694 : 2631.840000 2600.000000 836.549000 824.000000
       #IT F         L         W        Q          oChi        nChi        delt     dlin      rho      lambda
         0 62.719723 5.728158 1.978445 0.687694 : 3845.604736 3837.298584 nan 124.292549 0.066827 0.001000
        XX 5.257778 7.728158 1.978445 0.787694 : 3837.298584