Index: trunk/doc/release.2015/ps1.analysis/analysis.tex =================================================================== --- trunk/doc/release.2015/ps1.analysis/analysis.tex (revision 40586) +++ trunk/doc/release.2015/ps1.analysis/analysis.tex (revision 40588) @@ -112,5 +112,4 @@ % * add example for sky model % * Kaiser optimal detection reference -% * find a brighter-fatter reference % * define more tests and generate examples % * simulation example of background subtraction @@ -119,4 +118,8 @@ % * check all references % \end{verbatim} + +\note{the beginning section needs to be updated to mention the DR1 and + DR2 releases, the PV0-PV3 analysis versions, and the basic idea of + the IPP stages). This is the fourth in a series of seven papers describing the @@ -258,4 +261,6 @@ stand-alone C program, or as a set of library functions which may be integrated into other programs + +\note{quick discussion of the IPP analysis stages; PV0-PV3; DR1 \& DR2} Several variants of \code{psphot} have been used in the PS1 PV3 @@ -764,10 +769,11 @@ artifacts) and (2) the brighter stars are themselves subject to additional biases due to saturation and other non-linear effects -(c.f., ``the Brighter-Fatter'' effect, \note{REF}). To make a robust -choice for $\sigma_w$, we choose a value such that the measured value -of $\sigma^{\prime}_{\rm PSF}$ is 65\% of $\sigma_w$. The resulting second -moment values are biased somewhat low (\approx 75\% of the truth value -for the PS1 PSF profile), but are relatively unbiased as a function of -brightness. +\citep[c.f., ``the Brighter-Fatter'' + effect,][]{2014JInst...9C3048A,2015JInst..10C5032G}. To make a +robust choice for $\sigma_w$, we choose a value such that the measured +value of $\sigma^{\prime}_{\rm PSF}$ is 65\% of $\sigma_w$. The +resulting second moment values are biased somewhat low (\approx 75\% +of the truth value for the PS1 PSF profile), but are relatively +unbiased as a function of brightness. To choose the value of $\sigma_w$, we try a sequence of values @@ -787,4 +793,9 @@ an aperture with a radius of 4$\sigma_w$ to select the pixels for the measurement of the moments. + +%% comfirmed: PSF_MOMENTS_RADIUS = 4 * MOMENTS_GAUSS_SIGMA (\sigma_w) +%% factor of 4 is hard-wired in psphotSourceStats.c where MOMENTS_GAUSS_SIGMA is set. +%% PSF_MOMENTS_RADIUS used for: moments analysis, Kron analysis (starting radius), +%% radial profile wings (starting radius) Once $\sigma_w$ has been determined, moments are measured as defined @@ -917,5 +928,5 @@ some of the observed PSF variations in the images -\note{write up the fitting process to define the grid?} +% \note{write up the fitting process to define the grid?} Several analytical functions which are likely candidates to describe @@ -980,5 +991,5 @@ interpolated to the center of the model pixel. -Pixels for a given star which are more than XX sigma +Pixels for a given star which are more than a number of sigmas (PSF.RESIDUALS.NSIGMA = 3.0) deviant from the median value of the pixels from all stars are rejected. @@ -1030,10 +1041,4 @@ ignored. -% \note{is the pixel scale $0.1 \sigma_w$ or PSF_CLUMP_GRID_SCALE = 0.2?} -% psphotSourceStats sets PSF_CLUMP_GRID_SCALE to 0.1 \sigma_w^2, set -% to 0.2 by default (before \sigma_w is known). -% pmSource uses PSF_CLUMP_GRID_SCALE. note that the image is in Mxx -% (\sigma_x^2) not \sigma_x,\sigma_y) - Once a peak has been detected in this plane, the centroid and second moments of this peak are measured. All sources which land within 2 @@ -1112,5 +1117,4 @@ \end{table} - All of the PSF-candidate sources are then re-fitted using the PSF model to specify the PSF-dependent model parameter values for each @@ -1122,14 +1126,18 @@ the PSF model for this particular image. -The metric used by \code{psphot} to assess the PSF model is the scatter in -the differences between the aperture and fit magnitudes for the PSF -sources. This difference is a critical parameter for any PSF modeling -software as it is a measurement of how well the PSF model captures the -flux of the star. An approximate correction is measured here, with a -more detailed correction measured after all source analysis is -performed (see Section~\ref{sec:aperture.correction}). The PSF model -with the best consistency of the aperture correction is judged to be -the best model. \note{are we making a decision on the order or - anything based on apresid?} +The metric used by \code{psphot} to assess the PSF model is the +scatter in the differences between the aperture and fit magnitudes for +the PSF sources. This difference is a critical parameter for any PSF +modeling software as it is a measurement of how well the PSF model +captures the flux of the star. Aperture photometry is measured for a +circular aperture with a radius of \code{PSF_APERTURE_SCALE} (= 4.5 +for the PV3 $3\pi$ analysis) times $\sigma_w$ +(Section~\ref{sec:moments}). The average aperture correction ($m_{\rm + AP} - m_{\rm PSF}$) is measured and, if multiple PSF model types are +selected, the PSF model with the minimum clipped scatter in this +statistic is chosen for the image. An approximate aperture correction +is measured here, with a more detailed correction measured after all +source analysis is performed (see +Section~\ref{sec:aperture.correction}). \subsection{Bright Source Analysis} @@ -1314,9 +1322,14 @@ M_{\rm minor} = \frac{1}{2}(M_{xx} + M_{yy}) - \frac{1}{2}\sqrt{(M_{xx} - M_{yy})^2 + 4 M_{xy}^2} \] -If $M_{\rm minor} < 1.2$ pixels$^2$ and the instrumental Kron -magnitude is $< -5.5$, then the source is identified as a cosmic ray -and the associated pixels are masked. - -\note{how are / were these parameters set?} +If $M_{\rm minor} < 0.8$ pixels$^2$ and the signal-to-noise of the +flux measured in the moments analysis $> 7$, then the source is +identified as a cosmic ray and the associated pixels are masked. +These values are tuned empirically for the PV3 analysis based on +cosmic rays identified in the GPC1 images. + +% Mminor < 0.8 && SN > 7 + +% for dynamic CR parameters, use object with Mminor < 1.2 and Mkron < +% -5.5 to assess the distribution \subsubsection{Full PSF Model Fitting} @@ -1345,8 +1358,9 @@ For the PSF model fitting, only pixels within a circular aperture scaled based on the seeing are used. The radius of the circular -aperture is set to be a fixed multiple of $\sigma_w$, the width of the -Gaussian window function determined based on the analysis of the -second moments (see Section~\ref{sec:moments}). For the PV3 $3\pi$ -analysis, the PSF fit window radius is $7 \times \sigma_w$. +aperture is set to be a fixed multiple (\code{PSF_FIT_RADIUS_SCALE}) +of $\sigma_w$, the width of the Gaussian window function determined +based on the analysis of the second moments (see +Section~\ref{sec:moments}). For the PV3 $3\pi$ analysis, the PSF fit +window radius is $7 \times \sigma_w$. Sources which are blended with other sources are fitted together as a @@ -1854,5 +1868,5 @@ % \note{is the first convolution done with the Alard-Lupton technique?} -\subsection{Aperture Correction} +\subsection{Aperture Correction and Total Aperture Fluxes} \label{sec:aperture.correction} @@ -1871,11 +1885,10 @@ least within some range of normal image conditions. So, for example, two images with different image quality, or with different tracking -and focus errors, will have different PSF models. Since an analytical -model will always fail to represent the flux of the star at some -level, the measured flux of the same source in the two images will be -different (even assuming all other atmospheric and instrumental -effects have been corrected). The amplitude of the error will by -determined by how inconsistently the models represent the actual -source flux. +and focus errors, will have different PSF models. To the extent the +PSF model is inaccurate, the measured flux of the same source in the +two images will be different (even assuming all other atmospheric and +instrumental effects have been corrected). The amplitude of the error +will by determined by how inconsistently the models represent the +actual source flux. Aperture photometry attempts to avoid these problems, but introduces @@ -1891,5 +1904,6 @@ in the atmosphere. The amplitude and distribution of these two scattering functions do not change significantly or quickly for a -single telescope and site. +single telescope and site. Aperture photometry can then be used to +correct the PSF photometry. The difficulty for aperture photometry is the need to make an accurate @@ -1901,7 +1915,55 @@ number of very bright stars is limited in any image, and of course the brighter stars are more likely to suffer from non-linearity or -saturation. \code{psphot} measures the aperture correction ({\em ApResid}) -for every PSF candidate source and applies this correction to the PSF -model photometry. +saturation. + +In order to thread the needle between these effects, \code{psphot} +measures the aperture photometry on a modest-sized aperture, and then +uses the PSF model to extrapolate to a large aperture. When the PSF +fluxes are calculated, the aperture flux for the modest-sized aperture +is also determined. The aperture is a circular aperture with radius +set to a fixed multiple (\code{PSF_APERTURE_SCALE}) of $\sigma_w$, the +width of the Gaussian window function determined based on the analysis +of the second moments (see Section~\ref{sec:moments}). For the PV3 +$3\pi$ analysis, the aperture window radius is $4.5 \times \sigma_w$, +while the large reference aperture radius is set to 25 pixels +(\code{PSF_REF_RADIUS} = 6\farcs4). These corrected aperture +magnitudes are saved in the output catalogs as \code{AP_MAG}, the +uncorrected aperture magnitudes are saved as \code{AP_MAG_RAW}, and +the radius used to measure the raw aperture flux is saved as +\code{AP_MAG_RADIUS}. The corresponding flux and the flux error are +saved as \code{AP_FLUX} and \code{AP_FLUX_SIG}. + +With these aperture magnitudes in hand, it is now possible to make an +average correction to the PSF magnitudes to bring the PSF and aperture +magnitudes to the same system. This correction is measured using the +same stars from which the PSF model is measured, as long as the PSF +magnitude error for the star is less than 0.03 mag. The correction is +calculated using the weighted average of the values $m_{\rm AP} - +m_{\rm PSF}$. Since the PSF may vary across the image, the correction +is determined as a function of position in the image. Like the PSF +model, the 2D variations of the aperture correction may be modeled as +a polynomial or via interpolation in a grid. For the $3\pi$ PV3 +analysis, a grid with a maximum of $6\times 6$ samples per GPC1 chip +image was used. The reported PSF magnitudes for all objects have this +aperture correction applied. + +% growth curve analysis in psphot: +% in psphotChoosePSF : call psphotMakeGrowthCurve +% in psphotMakeGrowthCurve : boolean GROWTH_FROM_SOURCES, use +%% pmGrowthCurveGenerateFromSources or +%% pmGrowthCurveGenerate (uses PSF model only) +%% GROWTH_FROM_SOURCES is set to TRUE for default recipe + +%% ApTrend: +%% in psphotApResid, called by psphotReadout near the end of the +%% analysis +%% ApTrend = f(x,y) for (apMag - psfMag) for psfMagErr <= 0.03 +%% apMag is growth curve corrected +%% psfMag is raw + +%% raw psfMag and raw apMag are measured +%% apMag = apMagRaw + growth curve correction (from apRadius to 25 pix +%% = PSF_REF_RADIUS) +%% psfMag = psfMagRaw + aperture trend ( + growth curve) % How important is this effect? Consider a typical bright source with a @@ -2012,5 +2074,5 @@ Any measurement which relies on a good knowledge of the PSF at the location of an object either needs to determine the PSF variations -present in the \ippstage{stack} image, or the measurement will be +present in the \ippstage{stack} image or the measurement will be somewhat degraded. The highly textured PSF variations make this a very challenging problem: not only would such a PSF model require an @@ -2032,51 +2094,36 @@ %% images for a given stack. -The PV3 $3\pi$ analysis solves this problem by using the sources +The IPP analysis solves this problem by starting with the sources detected in the stack images and performing forced photometry on the individual warp images used to generate the stack. This -\ippstage{fullforce} analysis is performed on all warps for a single -skycell and filter as a single unit, as this matches the arrangement -of the input source catalog from the \ippstage{skycal} stage. When -processing is queued for this stage, an entry is added to the -\ippdbtable{fullForceRun} primary database table linking to the -specific \ippdbcolumn{skycal_id} entry that will be used as the -catalog for the photometry. The \ippdbcolumn{warp_id} values for the -input \ippstage{warp} stage images that contributed to the -\ippstage{stack} associated with that \ippdbcolumn{skycal_id} are -then added to the \ippdbtable{fullForceInput} table, linked to the -primary table by the \ippdbcolumn{ff_id} identifier. The individual -jobs for each warp are then run, which passes the \ippstage{warp} -stage image products along with the \ippstage{skycal} catalog to the -\ippprog{psphotFullForce} program. - -In this program, the positions of sources are loaded from the input -catalog. PSF stars are pre-identified \note{how?} and a PSF model -generated for each \ippstage{warp} image based on those stars, using -the same stars for all warps to the extent possible (PSF stars which -are excessively masked on a particular image are not used to model the -PSF). The PSF model is fitted to all of the known source positions in -the warp images. Aperture magnitudes, Kron magnitudes, and moments -are also measured at this stage for each warp. Note that the flux -measurement for a faint, but significant, source from the stack image -may be at a low significance (less than the $5\sigma$ criterion used -when the photometry is not run in this forced mode) in any individual -warp image; the flux may even be negative for specific warps. When -combined together, these low-significance measurements will result in -a signficant measurement as the signal-to-noise increases by the -square root of the number of measurements. - -Upon completion of the forced photometry (for point sources as well as -galaxies, discussed below), an entry is added to the -\ippdbtable{fullForceResult} table with the processing statistics for -that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}. -Once all of the entries in the \ippdbtable{fullForceInput} table have -finished, a summary operation is run to generate an appropriate -average value for each measurement, by combining the measurements from -each of the inputs. The output catalogs listed in the -\ippdbtable{fullForceResult} table are passed to the -\ippprog{psphotFullForceSummary} to do this averaging. \note{describe - what is done} When this completes, an entry is added to the -\ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} entry -is marked as completed. +forced-photometry analysis is performed using the +\ippprog{psphotFullForce} variant of \ippprog{psphot}. + +In this program, the positions of sources are loaded from the output +catalog of the stack photometry. Candidates PSF stars are +pre-identified as those stars used to generate the PSF model in the +stack photometry analysis. A PSF model is generated for each input +warp image based on those stars; PSF stars which are excessively +masked on a particular image are not used to model the PSF. The PSF +model is fitted to all of the known source positions in the warp +images. Aperture magnitudes, Kron magnitudes, and moments are also +measured at this stage for each warp. Note that the flux measurement +for a faint, but significant, source from the stack image may be at a +low significance (less than the $5\sigma$ criterion used when the +photometry is not run in this forced mode) in any individual warp +image; the measured flux may even be negative due to statistical +fluctuations. When combined together, these low-significance +measurements will result in a signficant measurement as the +signal-to-noise increases with the combination of more data. + +Individual warp images are processed independently with separate +executions of the \ippprog{psphotFullForce} program. Once all of the +forced photometry measurements (for point sources as well as galaxies, +discussed below) are completed for all of the warps which contributed +to a stack image, the measurements are combined together by other +portions of the IPP system. The PSF photometry measurements are +combined in the context of the DVO database system +\citep{magnier2017.datasystem}, including recalibration of the zero +points for the individual warp. \subsection{Forced Galaxy Models} @@ -2087,5 +2134,5 @@ this analysis, the galaxy models determined by the \ippstage{staticsky} photometry analysis are used to seed the analysis -in the individual \ippstage{warp} images. The purpose of this +in the individual \ippstage{warp} images. The motivation of this analysis is the same as the \ippstage{fullforce} PSF photometry: the PSF of the \ippstage{stack} image is poorly determined due to the @@ -2101,37 +2148,65 @@ elliptical shape, and thus the best galaxy magnitude value. -For each \ippstage{warp} image, the \ippstage{staticsky} value for the -major and minor axis are used as the center of a $7\times{} 7$ grid +For each \ippstage{warp} image, the \ippstage{staticsky} values for +the major and minor axis are used as the center of a $5 \times 5$ grid search of the major and minor axis parameter values. The grid spacing is defined as a function of the signal-to-noise of the galaxy in the stack image so that bright galaxies are measured with a much finer -grid spacing that faint galaxies \note{need to quantify this}. For -each grid point, the major and minor axis values at that point are -determined for the model. The model is then generated and convolved -with the PSF model for the \ippstage{warp} image at that point. The -resulting model is then compared to the \ippstage{warp} pixel data -values and the best fit normalization value is defined. The -normalization and the $\chi^2$ value for each grid point is recorded. - -For a given galaxy, the result is a collection of $\chi^2$ values for -each of the grid points spanning all \ippstage{warp} images. A single -$\chi^2$ grid can then be made by combining each grid point across the -inputs. The combined $\chi^2$ for a single grid point is simply the -sum of all $\chi^2$ values at that point. If, for a single \ippstage{warp} -image, the galaxy model is excessively masked, then that image will be -dropped for all grid points for that galaxy. The reduced $\chi^2$ -values can be determined by tracking the total number of pixels -used across all inputs to generate the combined $\chi^2$ values. From -the combined grid of $\chi^2$ values, the point in the grid with the -minimum $\chi^2$ is found. Quadratic interpolation is used to -determine the major, minor axis values for the interpolated minimum -$\chi^2$ value. The errors on these two parameters is then found by -determining the contour at which the \note{reduced?} $\chi^2$ -increases by 1. - -Thus the \ippstage{fullforce} galaxy analysis uses the PSF information -from each \ippstage{warp} to determine a best set of convovled galaxy -models for each object in the \ippstage{skycal} catalog. -\note{discuss the subset of galaxy models and objects}. +grid spacing than faint galaxies. For both the major and minor axis +directions, values of ($1 - \frac{3.0}{S/N}$, $1 - \frac{1.5}{S/N}$, +1.0, $1 + \frac{1.5}{S/N}$, $1 + \frac{3.0}{S/N}$) times the dimension +are tested. For each grid point, the major and minor axis values at +that point are used to generate the model. The model is then +convolved with the PSF model for the \ippstage{warp} image at that +point. The resulting convolved model is then compared to the +\ippstage{warp} pixel data values and the best fit normalization value +is determined. The integrated flux, flux error, and the $\chi^2$ +value for each grid point are recorded. + +For a given galaxy, the result is a collection of $\chi^2$ values, +fluxes, and flux errors for each of the grid points spanning all +\ippstage{warp} images. A single $\chi^2$ grid can then be made by +combining each grid point across the inputs. The combined $\chi^2$ +for a single grid point is simply the sum of all $\chi^2$ values at +that point. If, for a single \ippstage{warp} image, the galaxy model +is excessively masked, then that image will be dropped for all grid +points for that galaxy. The reduced $\chi^2$ values can be determined +by tracking the total number of pixels used across all inputs to +generate the combined $\chi^2$ values. From the combined grid of +$\chi^2$ values, the point in the grid with the minimum $\chi^2$ is +found. Quadratic interpolation is used to determine the major, minor +axis values for the interpolated minimum $\chi^2$ value. The errors +on these two parameters is then found by determining the contour at +which the \note{reduced?} $\chi^2$ increases by 1. + +In this way, the \ippstage{fullforce} galaxy analysis uses the PSF +information from each \ippstage{warp} to determine a best set of +convolved galaxy models for each object in the \ippstage{skycal} +catalog. + +% galaxy model fits performed based on limits set in psphotChooseAnalysisOptions.c + +% petrosian analysis performed on same objects as galaxy model fits +% if EXTENDED_SOURCE_PETROSIAN == TRUE (TRUE for PV3 stack - STACKPHOT) + +% galaxy model fits are performed on : +% all if (PSPHOT.EXT.FIT.ALL.SOURCES == TRUE) (FALSE for PV3 stack) +% (even so, if density is higher than PSPHOT.EXT.FIT.ALL.THRESH, skip) + +% only extended sources (based on EXT.NSIGMA) if EXT.NSIGMA.LIMIT.USE +% == TRUE (FALSE for PV3 stacks) + +% fit sources / measure petrosian to fixed flux limit if limits are +% defined (they are for PV3) + +% mag limits by filter, e.g., : petro 25, extfit 21.5 +% are translated to flux in counts and compared to Kron flux + +% SN limit is used only if fixed flux limits are not set +% SN limit set to EXTENDED_SOURCE_SN_LIM (10.0 for PV3) +% S/N limit for Kron flux, + +% galaxy coordinate limits: +% if |b| > 20.0 + 15.0 exp(-long^2 / (2 * 50^2)) \section{Difference Image Photometry}