Changeset 4529 for trunk/doc/pslib/psLibADD.tex

Timestamp:

Jul 8, 2005, 4:25:56 PM (21 years ago)

Author:

eugene

Message:

modified stats section: split robust into robust and fitted stats

File:

• trunk/doc/pslib/psLibADD.tex (modified) (4 diffs)

trunk/doc/pslib/psLibADD.tex

@@ -1 @@
-%%% $Id: psLibADD.tex,v 1.81 2005-07-08 20:50:52 price Exp $
+%%% $Id: psLibADD.tex,v 1.82 2005-07-09 02:25:56 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 
@@ -262 +262 @@
 \subsubsection{Robust Statistics}
 
-The robust version of the statistics provides estimators of basic
-statistical concepts which are reliable even for data samples with
-significant contamination.  A typical case is the situation in which
-the data of interest represent a primary population of interest with a
+Robust statistics algorithms provide estimators of basic statistical
+concepts which are reliable even for data samples with significant
+contamination.  A typical case is the situation in which the data of
+interest represent a primary population of interest with a
 single-valued mean and standard deviation and a secondary population
 of data with a substantially different distribution.  For example, an
@@ -273 +273 @@
 sources, but many are significantly higher (contributed by stars) or
 significantly lower (dead pixels).  If we want to measure the mean of
-the background, a robust mean is necessary as the outliers will
-strongly bias the sample statistics.
-
-The robust statistics are calculated by constructing a histogram of
-the values and performing the measurements on the histogram.  The
-choice of a bin size requires some care.  If the data are integer
-valued, the natural bin size is an integer.  Otherwise, the bin should
-be a fraction of an estimate of the standard deviation.  Use the
-$3\sigma$ clipped standard deviation as an estimator of the standard
-deviation.  The bin size shall be set at $\sigma_e / 10$.  The
-remaining steps of the algorithm are as follows:
+the background, a robust mean is necessary as the outliers will bias
+the sample statistics.  We define two levels of robust statistical
+estimators: robust statistics using the cumulative histogram, and
+statistics measured by fitting the differential histogram.
+
+The robust statistics from the cumulative are calculated by
+constructing a cumulative histogram of the values and performing
+simple measurements of the histogram data distribution.  The use of
+the cumulative histogram by this algorithm reduces the sensitivity to
+the exact bin size.  The initial bin size is set to 1/1000 of the
+total data range. The data values are found using quadratic
+interpolation between the bin of interest and its two neighbors.
 
 \begin{itemize}
 \item Construct the histogram with the specified bin size.
-\item Smooth the histogram by a Gaussian with $\sigma_s = \sigma_e /
-  4$.  
-\item Find the bin with the peak value in the range $L_{\frac{1}{4}}$
-  to $U_{\frac{1}{4}}$; this is the robust mode, $\mbox{mode}_r$.  
-\item Determine $dL = (U_{\frac{1}{4}} - L_{\frac{1}{4}}) / 4$.
-\item Fit a Gaussian to the bins in the range $\mbox{mode}_r - dL$ to
-  $\mbox{mode}_r + dL$.
-\item The resulting fit parameters are the robust mean,
-$\mbox{mean}_r$ and the robust standard deviation, $\sigma_r$.
+\item Construct the cumulative histogram from the specific histogram
+\item Find the bin which contains the 50\% data point.  
+\item Interpolate to the exact 50\% position: this is the robust
+  histogram median.
+\item Find the bins which contains the 15.8655\% and 84.1345\% data
+  points.  
+\item Interpolate to find these two positions exactly: these are the
+  $\pm 1\sigma$ positions.
+\item Determine $\sigma$ as 1/2 of the distance between these
+  positions.
+\item If the measured $\sigma$ is less than 2 times the bin size,
+  exclude points which are more than 25 bins from the median,
+  recalculate the bin size, and perform the algorithm again.
+\item Find the bins which contains the 25\% and 75\% data points.
+\item Interpolate to find these two positions exactly: these are the
+  upper and lower quartile positions.
 \end{itemize}
-
-To determine the robust median, construct the cumulative histogram
-from the histogram above. Select the bin which contains the 50th
-percentile value and its two neighbors.  Fit a quadratic to these
-three points.  The robust median value is the coordinate of the
-quadratic which returns the 50\% value.  For the upper and lower
-quartile points, the same process should be used, choosing the three
-bins in the vicinity of the upper and lower quartile points.
 
 If the errors in the input values are known, then the same approach is
@@ -314 +314 @@
 value (as opposed to a Gaussian), where the boxcar width is equal to
 $2 \sqrt{2 \ln 2}$ times the error and each input value contributes
-constant area.  Then the mean, median, mode, standard deviation and
-quartiles are estimated in the same manner as above.
+constant area.  Then the robust median and standard deviation are
+estimated in the same manner as above.
+
+\subsubsection{Fitted Statistics}
+
+The fitted statistics algorithm starts with the histogram used for the
+robust statistics and determines the population statistics by fitting
+a Gaussian model to the histogram.  The algorithm is as follows:
+
+\begin{itemize}
+\item Perform the Robust Histogram Statistics algorithm above
+\item Smooth the resulting histogram with a Gaussian with $\sigma_s$ =
+  1 bin.
+\item Find the bin with the peak value in the range $\pm 2 \sigma$ of
+  the robust histogram median.
+\item Fit a Gaussian to the bins in the range $\pm 2 \sigma$ of
+  the robust histogram median.
+\item The robust mean $\mbox{mean}_r$ is derived directly from the
+  fitted Gaussian mean.  
+\item The robust standard deviation, $\sigma_r$, is determined by
+  subtracting the smoothing scale in quadrature: $\sigma_r^2 =
+  \sigma^2 - \sigma_s^2$
+\end{itemize}
 
 \subsubsection{Histograms}