Context Navigation

← Previous Change
Next Change →

Changeset 40634 for trunk

Timestamp:

Mar 5, 2019, 4:12:58 PM (7 years ago)

Author:

eugene

Message:

reorg of the sections, cleanup Gaia discusions

File:

: 1 edited

trunk/doc/release.2015/ps1.calibration/calibration.tex (modified) (43 diffs)

Legend:

: Unmodified
: Added
: Removed

trunk/doc/release.2015/ps1.calibration/calibration.tex

-              r40632
+              r40634
  see][]{magnier2017.datasystem} were used internally for pipeline
 optimization and the development of the initial photometric and
 astrometric reference catalog \citep{magnier2017.calibration}.  The
+astrometric reference catalog.  The
 products from these reductions were not publicly released, but have
 been used to produce a wide range of scientific papers from the
 …
 Images obtained by \PSONE\ are automatically processed in real time by
+the \PSONE\ Image Processing Pipeline \citep[IPP,][]{magnier2017.datasystem}.
+Real-time analysis goals are aimed at feeding the discovery pipelines
+of the asteroid search and supernova search teams.  The data obtained
+for the \PSONE\ Science Survey has also been used in three additional
+complete re-processing of the data: Processing Versions 1, 2, and 3
+(PV1, PV2, and PV3).  The real-time processing of the data is
+considered ``PV0''.  Except as otherwise noted, this article describes
+the calibration of the PV3 analysis of the data.  Between the first
+(DR1) and second (DR2) data releases, improvements were made to the
+calibration of both the photometry and astrometry, as described in
+this article.
+The pipeline data processing steps are described in detail by
+\cite{waters2017} and
+\cite{magnier2017.datasystem,magnier2017.analysis}.  In summary,
+individual images are detrended: non-linearity and bias corrections
+are applied, a dark current model is subtracted and flat-field
+corrections are applied.  The \yps-band images are also corrected for
+fringing: a master fringe pattern is scaled to match the observed
+fringing and subtracted.  Mask and variance image arrays are generated
+with the detrend analysis and carried forward at each stage of the IPP
+processing.  Source detection and photometry are performed for each
+chip independently.  As discussed below, preliminary astrometric and
+photometric calibrations are performed for all chips in a single
+exposure in a single analysis.  We refer to these measurements as the
+``chip'' photometry and astrometry products.
+the \PSONE\ Image Processing Pipeline (IPP, see Paper II).  Real-time
+analysis goals are aimed at feeding the discovery pipelines of the
+asteroid search and supernova search teams.  The data obtained for the
+\PSONE\ Science Survey has also been used in three additional complete
+re-processing of the data: Processing Versions 1, 2, and 3 (PV1, PV2,
+and PV3).  The real-time processing of the data is considered ``PV0''.
+Except as otherwise noted, this article describes the calibration of
+the PV3 analysis of the data.  Between the first (DR1) and second
+(DR2) data releases, improvements were made to the calibration of both
+the photometry and astrometry, as described in this article.
+The pipeline data processing steps are described in detail in Papers
+II, III, and IV.  In summary, individual images are detrended:
+non-linearity and bias corrections are applied, a dark current model
+is subtracted and flat-field corrections are applied.  The \yps-band
+images are also corrected for fringing: a master fringe pattern is
+scaled to match the observed fringing and subtracted.  Mask and
+variance image arrays are generated with the detrend analysis and
+carried forward at each stage of the IPP processing.  Source detection
+and photometry are performed for each chip independently.  As
+discussed below, preliminary astrometric and photometric calibrations
+are performed for all chips in a single exposure in a single analysis.
+We refer to these measurements as the ``chip'' photometry and
+astrometry products.
 Chip images are geometrically transformed based on the astrometric
 …
 Astronomical objects are detected and characterized in the stack
 images.  The details of the analysis of the sources in the stack
 images are discussed in \cite{magnier2017.analysis}, but in brief
+images are discussed in Paper IV, but in brief
 these include PSF photometry, along with a range of measurements
 driven by the goals of understanding the galaxies in the images.
 …
 fluxes from the individual warp images are averaged, a reliable
 measurement of the faint source flux is determined.  The details of
 this analysis are described in detail in \cite{magnier2017.analysis}.
+this analysis are described in detail in Paper IV.
 The data products from the chip photometry, stack photometry, and
 forced-warp photometry analysis stages are ingested into the internal
 calibration database called the Desktop Virtual Observatory, or DVO
+\citep[see Section~4 in][]{magnier2017.datasystem} and used for
+photometric and astrometric calibrations.  In this article, we discuss
+the photometric calibration of the individual exposures, the stacks,
+and the warp images.  We also discuss the astrometric calibration of
 the individual exposures and the stack images.
+(see Section~4 in Paper II) and used for photometric and astrometric
+calibrations.  In this article, we discuss the photometric calibration
+of the individual exposures, the stacks, and the warp images.  We also
+discuss the astrometric calibration of the individual exposures and
+the stack images.
 \section{Pipeline Calibration}
 …
 Coordinates and calibrated magnitudes of stars from the reference
 database are loaded by \code{pasastro}.  A model for the positions of
+database are loaded by \ippprog{pasastro}.  A model for the positions of
 the 60 chips in the focal plane is used to determine the expected
 astrometry for each chip based on the boresite coordinates and
 …
 tangent-plane coordinate system.  The transforming polynomials are of
 the form:
+% P & = & \sum_{i,j} C^P_{i,j} X^i_{\rm chip} Y^j_{\rm chip} \\
+% Q & = & \sum_{i,j} C^Q_{i,j} X^i_{\rm chip} Y^j_{\rm chip}
 \begin{eqnarray}
 P & = & \sum_{i,j} C^P_{i,j} X^i_{\rm chip} Y^j_{\rm chip} \\
 Q & = & \sum_{i,j} C^Q_{i,j} X^i_{\rm chip} Y^j_{\rm chip}
+P & = & \sum_{i,j} C^P_{i,j} X^i Y^j \\
+Q & = & \sum_{i,j} C^Q_{i,j} X^i Y^j
 \end{eqnarray}
 where $P,Q$ are the tangent plane coordinates, $X_{\rm chip}, Y_{\rm
+  chip}$ are the coordinates on the 60 GPC1 chips, and $C^P_{i,j}, C^Q_{i,j}$
+are the polynomial coefficients for each order.  In the \ippprog{psastro}
+analysis, $i + j <= N_{\rm order}$ where the order of the fit, $N_{\rm
+  order}$, may be 1 to 3, under the restriction that sufficient stars
+are needed to constrain the order.
+where $P,Q$ are the tangent plane coordinates, $X, Y$ are the
+coordinates on the 60 GPC1 chips, and $C^P_{i,j}, C^Q_{i,j}$ are the
+polynomial coefficients for each order $i, j$.  In the
+\ippprog{psastro} analysis, $i + j <= N_{\rm order}$ where the order
+of the fit, $N_{\rm order}$, may be 1 to 3, under the restriction that
+sufficient stars are needed to constrain the order.
 A second form of astrometry model which yields somewhat higher
 …
 coordinate system:
 \begin{eqnarray}
 L & = & \sum_{i,j} C^L_{i,j} X^i_{\rm chip} Y^j_{\rm chip} \\
 M & = & \sum_{i,j} C^M_{i,j} X^i_{\rm chip} Y^j_{\rm chip}
+L & = & \sum_{i,j} C^L_{i,j} X^i Y^j \\
+M & = & \sum_{i,j} C^M_{i,j} X^i Y^j
 \end{eqnarray}
 …
 transformation may be written as:
 \begin{eqnarray}
   L & = & C^L_{0,0} + C^L_{1,0} X_{\rm chip} + C^L_{0,1} Y_{\rm chip} + \delta L(X_{\rm chip}, Y_{\rm chip}) \\
   M & = & C^M_{0,0} + C^M_{1,0} X_{\rm chip} + C^M_{0,1} Y_{\rm chip} + \delta M(X_{\rm chip}, Y_{\rm chip})
+  L & = & C^L_{0,0} + C^L_{1,0} X + C^L_{0,1} Y + \delta L(X, Y) \\
+  M & = & C^M_{0,0} + C^M_{1,0} X + C^M_{0,1} Y + \delta M(X, Y)
 \end{eqnarray}
 …
 the reference stars and the detected objects.  \ippprog{psastro} uses 2D
 cross correlation to search for the match.  The guess astrometry
+calibration is used to define a predicted set of $X^{\rm ref}_{\rm
+  chip}, Y^{\rm ref}_{\rm chip}$ values for the reference catalog
+calibration is used to define a predicted set of $X^{\rm ref}, Y^{\rm ref}$ values for the reference catalog
 stars.  For all possible pairs between the two lists, the values of
 \begin{eqnarray}
 \Delta X & = & X^{\rm ref}_{\rm chip} - X^{\rm obs}_{\rm chip}\\
 \Delta Y & = & Y^{\rm ref}_{\rm chip} - Y^{\rm obs}_{\rm chip}
+\Delta X & = & X^{\rm ref} - X^{\rm obs}\\
+\Delta Y & = & Y^{\rm ref} - Y^{\rm obs}
 \end{eqnarray}
 are generated.  The collection of $\Delta X, \Delta Y$ values are
 …
 %% \note{option to downweight based on photometric inconsistency : not used in PS1 analysis}
 \subsection{Chip Polynomial Fits}
+\subsection{Pipeline Astrometric Calibration}
 The astrometry solution from the cross correlation step above is again
 used to select matches between the reference stars and observed
 stars in the image.  The matching radius starts off quite large, and a
+used to select matches between the reference stars and observed stars
+in the image.  The matching radius starts off quite large, and a
 series of fits is performed to generate the transformation between
 chip and tangent plane coordinates.  Three clipping iterations are
 …
 here $\sigma$ is determined from the distribution of the residuals in
 each dimension (X,Y) independently.  After each fit cycle, the matches
+are redetermined using a smaller radius and the fit re-tried.
+\subsection{Mosaic Astrometry Polynomial Fits}
+are redetermined using a smaller radius and the fit re-tried.
 The astrometry solutions from the independent chip fits are used to
 …
+%
 Table~\ref{tab:measure_mask_values} lists the flags specific to
+individual measurements.  These values are stored in the DVO database in the
+field \code{Measure.dbFlags} and exposed in the public database \citep[PSPS][]{flewelling2017}
+in the fields \code{Detection.infoFlag3},
+\code{StackObjectThin.XinfoFlag3} (where \code{X} is one of
+     {$grizy$}), and \code{ForcedWarpMeasurement.FinfoFlag3}.
+individual measurements.  These values are stored in the DVO database
+in the field \ippdbtable{Measure}\ippdbcolumn{.dbFlags} and exposed in
+the public database (PSPS, Paper VI) in the fields
+\ippdbtable{Detection}\ippdbcolumn{.infoFlag3},
+\ippdbtable{StackObjectThin}{\ippdbcolumn.XinfoFlag3} (where
+\ippdbcolumn{X} is one of {$grizy$}), and
+\ippdbtable{ForcedWarpMeasurement}\ippdbcolumn{.FinfoFlag3}.
+%
 Table~\ref{tab:secf_mask_values} lists the flags which are set for
 each filter for individual objects in the database.  These values are
 recorded in the DVO database field \code{SecFilt.flags} and are
+recorded in the DVO database field \ippdbtable{SecFilt}\ippdbcolumn{.flags} and are
 exposed in PSPS in the fields
 \code{MeanObject.XFlags} and \code{StackObjectThin.XinfoFlag4}, where
 \code{X} in both cases is one of {$grizy$}.
+\ippdbtable{MeanObject}\ippdbcolumn{.XFlags} and \ippdbtable{StackObjectThin}\ippdbcolumn{.XinfoFlag4}, where
+\ippdbcolumn{X} in both cases is one of {$grizy$}.
+%
 Table~\ref{tab:object_mask_values} lists the flags specific to an
 object as a whole.  These values are stored in the DVO database field
 \code{Average.flags} and are exposed in PSPS in
 the field \code{MeanObject.objInfoFlag}.
+\ippdbtable{Average}\ippdbcolumn{.flags} and are exposed in PSPS in
+the field \ippdbtable{MeanObject}\ippdbcolumn{.objInfoFlag}.
+%
 Table~\ref{tab:image_mask_values} lists the flags raised for images.
 These flags are stored in the DVO database field \code{Image.flags}
 and are exposed in PSPS in the field \code{ImageMeta.qaFlags}.
+These flags are stored in the DVO database field \ippdbtable{Image}\ippdbcolumn{.flags}
+and are exposed in PSPS in the field \ippdbtable{ImageMeta}\ippdbcolumn{.qaFlags}.
+%
 The type of conditions which are recorded by these bits range from
 …
 Photometric nights are selected and all other exposures are ignored.
 Each night is allowed to have a single fitted zero point
 (corresponding to the sum $zp_{\rm nominal} + M_{cal}$ below) and a
+(corresponding to the sum $zp_{\rm ref} + M_{cal}$ below) and a
 single fitted value for the airmass extinction coefficient ($K_{\rm
   \lambda}$) per filter.  The zero points and extinction terms are
 …
 The ubercal zero points and the flat-field correction data are loaded
 into the PV3 DVO database using the program \code{setphot}.  This
+into the PV3 DVO database using the program \ippprog{setphot}.  This
 program converts the reported zero point and flat field values to the
 DVO internal representation in which the zero point of each image is
 split into three main components:
 \begin{equation}
 zp_{\rm total} = zp_{\rm nominal} + M_{cal} + K_{\rm \lambda}(\sec \zeta - 1)
+zp_{\rm total} = zp_{\rm ref} + M_{cal} + K_{\rm \lambda}(\sec \zeta - 1)
 \end{equation}
 where $zp_{\rm nominal}$ and $K_{\rm \lambda}$ are static values for
 each filter representing respectively the nominal zero point and the
+where $zp_{\rm ref}$ and $K_{\rm \lambda}$ are static values for
+each filter representing respectively the nominal reference zero point and the
 slope of the trend with respect to the airmass ($\zeta$) for each
 filter.  These static values are listed in Table~\ref{tab:zpts}.  When
 \code{setphot} was run, these static zero points have been adjusted by
+\ippprog{setphot} was run, these static zero points have been adjusted by
 the Calspec offsets listed in Table~\ref{tab:zpts} based on the
 analysis of Calspec standards by \cite{2015ApJ...815..117S}.  These
 …
 in a table of flat-field offsets as a function of time, filter, and
 camera position.  Each image which is part of the ubercal subset is
 marked with a bit in the field \code{Image.flags}:
+marked with a bit in the field \ippdbtable{Image}\ippdbcolumn{.flags}:
 \code{ID_IMAGE_PHOTOM_UBERCAL = 0x00000200}.
 …
 \end{table}
 When \code{setphot} applies the ubercal information to the image
+When \ippprog{setphot} applies the ubercal information to the image
 tables, it also updates the individual measurements associated with
 those images.  In the DVO database schema, the normalized instrumental
 magnitude, $M_{\rm inst} = -2.5log_{10} (DN / sec)$ is stored
+magnitude, $m_{\rm inst} = -2.5log_{10} (DN / sec)$ is stored
 for each measurement, with an arbitrary (but fixed)
 constant offset of 25 to place the modified instrumental magnitudes into
 …
 as:
 \begin{equation}
+M_{\rm rel} = M_{\rm inst} + zp_{\rm ref} + M_{\rm cal} + M_{\rm flat} + K_\lambda (sec \zeta - 1).
+M_{\rm rel} = m_{\rm inst} + zp_{\rm ref} + M_{\rm cal} + M_{\rm flat} + K_\lambda (sec \zeta - 1).
+\label{eqn:Mrel}
 \end{equation}
 The calibration offsets, $M_{\rm cal}$ and $M_{\rm flat}$, represent
 …
 When the ubercal zero points and flat-field data are loaded,
 \code{setphot} updates the $M_{\rm cal}$ values for all measurements
+\ippprog{setphot} updates the $M_{\rm cal}$ values for all measurements
 which have been derived from the ubercal images.  These measurements
 are also marked in the field \code{Measure.dbFlags} with the bit
+are also marked in the field \ippdbtable{Measure}\ippdbcolumn{.dbFlags} with the bit
 \code{ID_MEAS_PHOTOM_UBERCAL = 0x00008000}.  At this stage,
 \code{setphot} also updates the values of $M_{\rm flat}$ for all GPC1
+\ippprog{setphot} also updates the values of $M_{\rm flat}$ for all GPC1
 measurements in the appropriate filters.
 …
 As described above, the instrumental magnitude and the calibrated magnitude
 are related by arithmetic magnitude offsets which account for effects
 such as the instrumental variations and atmospheric attenuation.
 \begin{equation}
 M_{rel} = m_{inst} + ZP + M_{cal}
+\end{equation}
+such as the instrumental variations and atmospheric attenuation (Eqn~\ref{eqn:Mrel}).
+%% \begin{equation}
+%% % M_{rel} = m_{inst} + zp_{\rm ref} + M_{cal}
+%% M_{\rm rel} = m_{\rm inst} + zp_{\rm ref} + M_{\rm cal} + M_{\rm flat} + K_\lambda (sec \zeta - 1).
+%% \end{equation}
 From the collection of measurements, we can generate an average
 magnitude for a single star (or other object):
 …
 $M_{\rm cal}$ offset for each exposure:
 \begin{equation}
+  \chi^2 = \frac{\sum_{i,j} (m_{inst}[i,j] + ZP + K \zeta +
+    M_{clouds}[i] - M_{ave}[j]) w_{i,j}}{\sum_{i,j} w_{i,j}}
+  \chi^2 = \frac{\sum_{i,j} (M_{\rm rel}[i,j] - M_{\rm ave}[j]) w_{i,j}}{\sum_{i,j} w_{i,j}}
 \end{equation}
 …
 %% These extractions should be used for the paper (EAM 2019.02.15)
-\begin{figure*}[htbp]
-  \begin{center}
-%width=\hsize
- \includegraphics[height=\vsize,clip]{{pics/allsky.photom.v1}.png}
-  \caption{\label{fig:allsky.photom.sigma} Consistency of photometry
-    measurements across the sky.  Each panel shows a map of the
-    standard deviation of photometry residuals for stars in each
-    pixel.  The median value of the measure standard deviations across
-    the sky is $(\sigma_g, \sigma_r, \sigma_i, \sigma_z, \sigma_y) =
-    (14, 14, 15, 15, 18)$ millimags.  These values reflect the typical
-    single-measurement errors for bright stars.}
-  \end{center}
-\end{figure*}
 \subsubsection{Photometric Flat-field}
 \label{sec:phot.flat}
 …
 flat-field residual with much finer resolution: 124 x 124 flat-field
 values for each GPC1 chip (40x40 pixels per point).  We then used
 \code{setphot} to apply this new flat-field correction, as well as the
+\ippprog{setphot} to apply this new flat-field correction, as well as the
 ubercal flat-field corrections, to the data in the database.  At this
 point, we re-ran the entire relphot analysis to determine zero points
 …
 to follow the changes in the PSF.
+\subsubsection{Stack and Warp Photometric Calibration}
+\label{sec:phot.stack}
 For stacks and warps, the image calibrations were determined after the
 relative photometry was performed on the individual chips.  Each stack
 and each warp was tied via relative photometry to the average
 magnitudes from the chip photometry.  In this case, no flat-field
 corrections were applied.  For the stacks, such a correction would not
+be possible after the stack has been generated because multiple chip
+coordinates contribute to each stack pixel coordinate.  For the warps,
+it is in principle possible to map back to the corresponding chip, but
 the information was not available in the DVO database, and thus it was
+not possible at this time to determine the flat-field correction
+appropriate for a given warp.  This latter effect is one of several
+which degrade the warp photometry compared to the chip photometry at
 the bright end.
+magnitudes from the chip photometry, as described below.  In this
+case, no flat-field corrections were applied.  For the stacks, such a
+correction would not be possible after the stack has been generated
+because multiple chip coordinates contribute to each stack pixel
+coordinate.  For the warps, it is in principle possible to map back to
+the corresponding chip, but the information was not available in the
+DVO database, and thus it was not possible at this time to determine
+the flat-field correction appropriate for a given warp.  This latter
+effect is one of several which degrade the warp photometry compared to
+the chip photometry at the bright end.
 For the stack calibration, we calculate two separate zero points: one
 …
 the PSF magnitudes to the average of the chip photometry PSF
 magnitudes, but the aperture-like magnitudes are tied by equating the
+stack Kron magnitudes to the average chip Kron magnitudes.  {\em Note
+  that for DR1, this split zero point calibration was {\bf not} used; instead
+  all stack photometry was tied to the average chip photometry via the
+  PSF magnitudes.}  The result of using a single zero point is that
+the stack PSF magnitudes are consistent across the sky with the chip
+PSF magnitudes, but the aperture-like magnitudes show significant
+spatial variations.  Figure~\ref{fig:stack.bad.kron} illustrates the
+impact of using a single PSF zero point for the stack photometry.
+This split is not needed for the forced-warp photometry since the
+individual warps have well-defined PSfs.
+stack Kron magnitudes to the average chip Kron magnitudes.
 %% XXX generate a figure to illustrate the Kron vs PSF mags in stacks (DR1 & DR2)
-\subsection{Photometry Calibration Quality}
-Figure~\ref{fig:allsky.photom.sigma} shows the standard deviations of
-the mean residual photometry for bright stars as a function of
-position across the sky.  For each pixel in these images, we selected
-all objects with (14.5, 14.5, 14.5, 14.0, 13.0) $<$ ($g,r,i,z,y$) $<$
-(17, 17, 17, 16.5, 15.5), with at least 3 measurements in $i$-band (to
-reject artifacts detected in a pair of exposures from the same night),
-with \code{PSF_QF} $> 0.85$ (to reject excessively-masked objects),
-and with $mag_{\rm PSF} - mag_{\rm Kron} < 0.1$ (to reject galaxies).
-We then generated histograms of the difference between the average
-magnitude and the apparent magnitude in an individual image for each
-filter for all stars in a given pixel in the images.  From these
-residual histograms, we can then determine the median and the 68\%-ile
-range to calculate a robust standard deviation.  This represents the
-bright-end systematic error floor for a measurement from a single
-exposure.  The standard deviations are then plotted in
-Figure~\ref{fig:allsky.photom.sigma}.
-The 5 panels in Figure~\ref{fig:allsky.photom.sigma} show several
-features.  The Galactic bulge is clearly seen in all five filters,
-with the impact strongest in the reddest bands.  We attribute this to
-the effects of crowding and contamination of the photometry by
-neighbors.  Large-scale, roughly square features \approx 10 degrees on
-a side in these images can be attributed to the vagaries of weather:
-these patches correspond to the observing chunks.  These images
-include both photometric and non-photometric exposures.  It seems
-plausible that the non-photometric images from relatively poor quality
-nights elevate the typical errors.  On small scales, there are
-circular patterns \approx 3 degrees in diameter corresponding to
-individual exposures; these represent residual flat-fields structures
-not corrected by our stellar flat-fielding.  The median of the
-standard deviations in the five filters are
-$(\sigma_g,\sigma_r,\sigma_i,\sigma_z,\sigma_y) = (14, 14, 15, 15,
-)$ millimagnitudes.
 \subsection{Object Photometry}
 …
 \begin{itemize}
 \item {\bf rank 0 :} perfect measurement (no quality concerns)
+\item {\bf rank 1 :} PSF ``perfect pixel'' quality factor (\code{PSF_QF_PERFECT}) $< 0.85$.  \code{PSF_QF_PERFECT} measures the PSF-weighted fraction of pixels which are not masked \citep[see][]{magnier2017.analysis}.
+\item {\bf rank 1 :} PSF ``perfect pixel'' quality factor
+  (\code{PSF_QF_PERFECT}) $< 0.85$.  \code{PSF_QF_PERFECT} measures
+  the PSF-weighted fraction of pixels which are not masked (see Paper IV).
 \item {\bf rank 2 :} Photometry analysis flag field (\code{photFlags}) has one of the ``poor quality'' bits raised.  These bits are listed below; OR-ed together they have the hexadecimal value \code{0xe0440130}
 \begin{itemize}
 …
   those pixels on ghosts, diffraction spikes, bright star bleeds, and
   the mildly-saturated cores of bright stars.  Suspect values may have
   some use in measuring a flux, but with caution
   \citep[see][]{magnier2017.analysis,waters2017}.
+  some use in measuring a flux, but with caution (see Papers II and
+  III).
 \item {\bf rank 5 :} Photometric calibration of the GPC1 exposure is
   determined by relphot to be poor.  This situation occurs if there
 …
 Pan-STARRS\,1 detections have a relatively high rate of non-Gaussian
 outliers, partly because of the wide range of instrumental features
 affecting the data \citep[see][]{waters2017}.  We have used a
+affecting the data (see Paper III).  We have used a
 technique called Iteratively Reweighted Least Squares (IRLS) fitting
 to reduce the sensitivity of the fits to outlier measurements.  We
 …
 % from: /data/kukui.3/eugene/pv3.stats.20161202/
-\begin{figure*}[htbp]
-  \begin{center}
- \includegraphics[width=\hsize,clip]{{pics/KHexample}.png}
-  \caption{\label{fig:KHexample} Illustration of the Koppenh\"ofer Effect
-    on OTA04.  {\bf Bottom left} X-direction before correction.  The solid line shows the measured
-    mean residual for stars detected on this chip as a function of the
-    instrumental magnitude / FWHM$^2$.
-{\bf Bottom right} Y-direction before correction.
-{\bf Top left} X-direction after correction.
-{\bf Top right} Y-direction after correction.  }
-  \end{center}
-\end{figure*}
-% from: /data/kukui.3/eugene/pv3.stats.20161202/
-\begin{figure}[htbp]
-  \begin{center}
- \includegraphics[width=\hsize,clip]{{pics/KHmap}.png}
-  \caption{\label{fig:KHmap} Map of the amplitude of the
-    Koppenh\"ofer Effect on chips across the focal plane.  In the
-    affected chips, bright stars are up to 0.2 arcsec deviant
-    from their expected positions. {\bf Bottom left} X-direction before
-    correction.  {\bf Bottom right} Y-direction before correction.  {\bf
-      Top left} X-direction after correction.  {\bf Top right}
-    Y-direction after correction.}
-  \end{center}
-\end{figure}
 \subsubsection{Object Photometry Flags}
 …
 \code{ID_OBJ_MOST_SOLSYS_DET} is set.
+\subsection{Photometry Calibration Quality}
+\begin{figure*}[htbp]
+  \begin{center}
+%width=\hsize
+ \includegraphics[height=\vsize,clip]{{pics/allsky.photom.v1}.png}
+  \caption{\label{fig:allsky.photom.sigma} Consistency of photometry
+    measurements across the sky.  Each panel shows a map of the
+    standard deviation of photometry residuals for stars in each
+    pixel.  The median value of the measure standard deviations across
+    the sky is $(\sigma_g, \sigma_r, \sigma_i, \sigma_z, \sigma_y) =
+    (14, 14, 15, 15, 18)$ millimags.  These values reflect the typical
+    single-measurement errors for bright stars.}
+  \end{center}
+\end{figure*}
+Figure~\ref{fig:allsky.photom.sigma} shows the standard deviations of
+the mean residual photometry for bright stars as a function of
+position across the sky.  For each pixel in these images, we selected
+all objects with (14.5, 14.5, 14.5, 14.0, 13.0) $<$ ($g,r,i,z,y$) $<$
+(17, 17, 17, 16.5, 15.5), with at least 3 measurements in $i$-band (to
+reject artifacts detected in a pair of exposures from the same night),
+with \code{PSF_QF} $> 0.85$ (to reject excessively-masked objects),
+and with $mag_{\rm PSF} - mag_{\rm Kron} < 0.1$ (to reject galaxies).
+We then generated histograms of the difference between the average
+magnitude and the apparent magnitude in an individual image for each
+filter for all stars in a given pixel in the images.  From these
+residual histograms, we can then determine the median and the 68\%-ile
+range to calculate a robust standard deviation.  This represents the
+bright-end systematic error floor for a measurement from a single
+exposure.  The standard deviations are then plotted in
+Figure~\ref{fig:allsky.photom.sigma}.
+The 5 panels in Figure~\ref{fig:allsky.photom.sigma} show several
+features.  The Galactic bulge is clearly seen in all five filters,
+with the impact strongest in the reddest bands.  We attribute this to
+the effects of crowding and contamination of the photometry by
+neighbors.  Large-scale, roughly square features \approx 10 degrees on
+a side in these images can be attributed to the vagaries of weather:
+these patches correspond to the observing chunks.  These images
+include both photometric and non-photometric exposures.  It seems
+plausible that the non-photometric images from relatively poor quality
+nights elevate the typical errors.  On small scales, there are
+circular patterns \approx 3 degrees in diameter corresponding to
+individual exposures; these represent residual flat-fields structures
+not corrected by our stellar flat-fielding.  The median of the
+standard deviations in the five filters are
+$(\sigma_g,\sigma_r,\sigma_i,\sigma_z,\sigma_y) = (14, 14, 15, 15,
+)$ millimagnitudes.
+\begin{figure*}[htbp]
+  \begin{center}
+  \includegraphics[width=\hsize,clip]{{pics/photom.pv3.3v4}.png}
+  \caption{\label{fig:photom.pv3.3v4} Sample comparison of PV3.3 and
+    PV3.4 photometry illustrating the impact of the issues identified
+    in the PV3.3 stack and warp photometry.  All figures use \ips-band
+    photometry.  The left panels use data from PV3.3 while the right
+    use PV3.4.  The top row shows the mean difference between the
+    average photometry from individual exposures (``chip'') and the
+    stack photometry using Kron magnitudes.  The middle row shows the
+    mean difference between the average photometry from individual
+    exposures (``chip'') and the average forced-warp photometry, again
+    using Kron magnitudes.  The bottom row shows the mean difference
+    between the average photometry from individual exposures
+    (``chip'') and the average forced-warp photometry, using PSF
+    magnitudes.  See Section~\ref{sec:discussion} for a description of
+    the calibration change in PV3.4.}
+\end{center}
+\end{figure*}
+As discussed above (Section~\ref{sec:phot.stack}), the DR2 stack
+calibration used separate zero points for PSF-like and aperture-like
+photometry.  For DR1, this split zero point calibration was {\bf not}
+used.  Instead all stack photometry was tied to the average chip
+photometry via the PSF magnitudes.  The result of using a single zero
+point is that the stack PSF magnitudes are consistent across the sky
+with the chip PSF magnitudes, but the aperture-like magnitudes show
+significant spatial variations.  A second issue identified in DR1 and
+corrected in DR2 is due to the application of the high-resolution
+photometric flat-field correction.  For the initial processing of the
+PV3 calibration, this flat-field correction was applied with the wrong
+sign.  For DR1, the error was corrected for the \ippstage{chip}-stage
+photometry.  However, the stack and warp photometry had been tied to
+the chip-stage photometry before this correction and they were not
+recalibrated before the DR1 release.  After this error was noticed,
+the stack and warp photometry were recalibrated for DR2.
+Figure~\ref{fig:photom.pv3.3v4} illustrates the impact of using a
+single PSF zero point for the stack photometry and the impact of the
+flat-field error.  This zero point split is not needed for the
+forced-warp photometry since the individual warps have well-defined
+PSFs.
 \section{Astrometry Calibration}
+\begin{figure*}[htbp]
+  \begin{center}
+ \includegraphics[width=\hsize,clip]{{pics/KHexample}.png}
+  \caption{\label{fig:KHexample} Illustration of the Koppenh\"ofer Effect
+    on OTA04.  {\bf Bottom left} X-direction before correction.  The solid line shows the measured
+    mean residual for stars detected on this chip as a function of the
+    instrumental magnitude / FWHM$^2$.
+{\bf Bottom right} Y-direction before correction.
+{\bf Top left} X-direction after correction.
+{\bf Top right} Y-direction after correction.  }
+  \end{center}
+\end{figure*}
+% from: /data/kukui.3/eugene/pv3.stats.20161202/
+\begin{figure}[htbp]
+  \begin{center}
+ \includegraphics[width=\hsize,clip]{{pics/KHmap}.png}
+  \caption{\label{fig:KHmap} Map of the amplitude of the
+    Koppenh\"ofer Effect on chips across the focal plane.  In the
+    affected chips, bright stars are up to 0.2 arcsec deviant
+    from their expected positions. {\bf Bottom left} X-direction before
+    correction.  {\bf Bottom right} Y-direction before correction.  {\bf
+      Top left} X-direction after correction.  {\bf Top right}
+    Y-direction after correction.}
+  \end{center}
+\end{figure}
 Once the full PV3 dataset loaded into the master PV3 DVO database,
 …
 measured the mean X and Y displacements of the astrometric residuals
 as function of the instrumental magnitude of the star divided by the
+FWHM$^2$.  We measured the trend for all chips in a
+number of different time ranges and found the effect to be quite
+stable, in the period where it was present.  The effect only appeared
+in the serial direction.  Figure~\ref{fig:KHexample} shows the KE
+trend for a typical affected chip both before and after the
+correction.  For the PV3 dataset, we re-measured the KE trends using
+stars in the Galactic pole regions after an initial relative
+astrometry calibration pass: the Galactic pole is necessary because
+the real-time astrometric calibration relies largely on the fainter
+stars which are not affected by the KE.  The trend is then stored in a
+form which can be applied to the database measurements.
+FWHM$^2$.  We measured the trend for all chips in a number of
+different time ranges and found the effect to be quite stable, in the
+period where it was present.  The effect only appeared in the serial
+direction.  Figure~\ref{fig:KHexample} shows the KE trend for a
+typical affected chip both before and after the correction.
+Figure~\ref{fig:KHmap} shows the maximum impact of the Koppenh\"ofer
+Effect as a function of chip position in the focal plane.  For the PV3
+dataset, we re-measured the KE trends using stars in the Galactic pole
+regions after an initial relative astrometry calibration pass: the
+Galactic pole is necessary because the real-time astrometric
+calibration relies largely on the fainter stars which are not affected
+by the KE.  The trend is then stored in a form which can be applied to
+the database measurements.
 \subsubsection{Differential Chromatic Refraction}
 …
 the tangent of the zenith distance:
 \begin{eqnarray}
 \delta_{\rm blue} = \alpha \left[(g - i)_{\rm ref} - (g - i)\right] \tan \zeta \\
 \delta_{\rm red} = \alpha \left[(z - y)_{\rm ref} - (z - y)\right] \tan \zeta
+\delta_{\rm blue} & = & \alpha \left[(g - i)_{\rm ref} - (g - i)\right] \tan \zeta \\
+\delta_{\rm red} & = & \alpha \left[(z - y)_{\rm ref} - (z - y)\right] \tan \zeta
 \end{eqnarray}
 where $(g-i)_{\rm ref}$ and $(z-y)_{\rm ref}$ are the median colors of the
 …
 discussed in detail in \cite{2018PASP..130f5002M}.
-% generate (or plot) astrometric flat-field images for DR2 (PV3.X)
-\begin{figure*}[htbp]
-  \begin{center}
-  \includegraphics[width=\hsize,clip]{{pics/astroflat.repair}.png}
-  \caption{\label{fig:astroflat.repair} Comparison of the
-    high-resolution astrometric flat-field images used for PV3.2
-    (left) and for PV3.3 (right).  These examples show the \gps-band
-    astrometric flat-field corrections for the $X$ direction as seen
-    in the focal plane coordinate system.  Note the elevated noise in
-    the PV3.2 image due to insufficient numbers of stars used in the analysis.
+}
-\end{center}
-\end{figure*}
-% numbers of stars used:
-%% mana: load.stars astroflat.20151205/astroflat.20151205.v1.Npt.fits
-%% filter g : 2591421 stars
-%% filter r : 3497036 stars
-%% filter i : 16241986 stars
-%% filter z : 7153595 stars
-%% filter y : 4509749 stars
-%% mana: load.stars astroflat.20170217/astroflat.20170217.Npt.fits
-%% filter g : 17590560 stars
-%% filter r : 31000135 stars
-%% filter i : 82648850 stars
-%% filter z : 62166619 stars
-%% filter y : 42867074 stars
-\note{move the discussion of the DR1 & DR2 scatter to the end of the
-  astrom section?}
-Figure~\ref{fig:allsky.astrom.sigma} shows the standard deviations of
-the mean residual astrometry in $(\alpha,\delta)$ for bright stars as
-a function of position across the sky based on the DR2 calibration.  For each
-pixel in these images, we selected all objects with $15 < i < 17$,
-with at least 3 measurements in $i$-band (to reject artifacts detected
-in a pair of exposures from the same night), with \code{PSF_QF} $>
-.85$ (to reject excessively-masked objects), and with $mag_{\rm PSF}
-- mag_{\rm Kron} < 0.1$ (to reject galaxies).  We then generated
-histograms of the difference between the object position predicted for
-the epoch of each measurement (based on the proper motion and parallax
-fit) and the observed position of that measurement, in both the Right
-Ascension and Declination directions (in linear arcseconds), for all
-stars in a given pixel in the images.  From these residual histograms,
-we can then determine the median and the 68\%-ile range to calculate a
-robust version of the standard deviation.  This represents the
-bright-end systematic error floor for a measurement from a single
-exposure.  The standard deviations are then plotted in
-Figure~\ref{fig:allsky.photom.sigma}.  The median value of the
-standard deviations across the sky in both $(\sigma_\alpha,
-\sigma_\delta)$ is 16 milliarcseconds.
-The Galactic plane is clearly apparently in these images.  Like
-photometry, we attribute this to failure of the PSF fitting due to
-crowding.  The celestial North pole regions have somewhat elevated
-errors in both R.A. and DEC, with some specifc structures.  Some of
-these structures may be due to the larger typical seeing at these high
-airmass regions, but some are due to astrometric failures which stem
-from the reference catalog based on the PV2 analysis (see
-Section~\ref{sec:pole.problems} for further details).  Several
-features can be seen which appear to be an effect of the tie to the
-Gaia astrometry: the stripes near the center of the DEC image and the
-right side of the R.A. image.  The mesh of circular outlines one the 2
-degree scale is due to the outer edge of the focal plane where the
-astrometric calibration is poorly determined.
-The DR1 astrometric calibration suffered from degraded astrometry due
-to a problem with the astrometric flat-field correction identified too
-late to be repaired for DR1.
+%
-The astrometric flat-field images used
-for that release had too few stars to measure the correction with
-sufficient signal-to-noise.  As a result, those corrections had
-significant pixel-to-pixel noise which can be seen in
-Figure~\ref{fig:astroflat.repair}.  As a result, the astrometric
-flat-field correction reduces systematic structures on large spatial
-scales, but at the expense of degrading the quality of individual
-measurements.  Only the $i$-band flat had sufficient signal-to-noise
-per pixel to avoid significantly increasing the per-measurement
-position errors.
-For DR2, we recalculated the astrometric flat-field correction using
-many more stars.  For the DR1 release, the number of stars per filter
-was (\grizy) = (2.6M, 3.5M, 16M, 7M, 4.5M), while for the DR2 release,
-the number of stars per filter was (\grizy) = (18M, 31M, 83M, 62M,
-M).  We also reduced the resolution of the astrometric flat-field,
-using $80 \times 80$ superpixels, rather than the $40 \times 40$
-superpixels used for DR1.  Because of the degraded astrometric
-flat-field correction, the median per-measurement error floor of DR1
-is \approx 22 mas, significantly worse than both DR2 and the earlier
-PV2 analysis.  Figure~\ref{fig:allsky.astro.histogram} shows
-histograms of the astrometric residual scatter across the sky for DR1
-and DR2, illustrating the improvement.
-\begin{figure*}[htbp]
-  \begin{center}
-  \includegraphics[width=\hsize,clip]{{pics/allsky.histogram.astrom.compare}.png}
-  \caption{\label{fig:allsky.astro.histogram} Illustration of the
-    impact of the astrometric flat-field correction used for PV3.2 vs
-    PV3.3.  The blue histograms show the distribution of astrometric
-    residuals for bright stars from the PV3.2 analysis while the red
-    histograms show the distribution for the PV3.3 analysis.  The
-    median standard deviation for PV3.2 is 22 milliarcseconds in R.A.
-    (23 mas in Declination).  Using the higher signal-to-noise
-    flat-field correction images reduces the median values to 16 mas
-    for both R.A. and Declination directions in PV3.3.
+}
-\end{center}
-\end{figure*}
-% older version of this figure:
-% pv2_0 : /data/ipp060.0/eugene/pv2.astrom.20150126/astromap.20150127/dDsig.im.fits
-% pv2_1 : /data/ipp060.0/eugene/pv2.astrom.20150126/astromap.20150429/dDsig.im.fits
-% NOTE:
-% the pv2 versions used:  resize 1800 920; region 0 0 85 ait
-% the pv3 versions used:  resize 1800 950; region 180 0 90 ait
-% thus we cannot directly compare map pixels, without re-extracting the measurements
-% (we can do that if we decide it is needed to generate the best plots)
-% original version of figure: pv3.stats.20161202/allsky.astrom.sigma.png
-%   based on /data/kukui.3/eugene/pv3.stats.20161202/maps.measure/pv3.v1.*.sigma.fits
-%   based on /data/ipp094.0/eugene/pv3.stats.20161202/cdhist.measure/cdmerge.v1.dD.fits (& dR)
-%   plot script /data/kukui.3/eugene/pv3.stats.20161202/scatter.sh
-%   catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master (PV3.2)
-% regenerate using fits image in pv3.stats.20170413
-\begin{figure*}[htbp]
-  \begin{center}
- \includegraphics[width=\hsize,clip]{{pics/allsky.astrom.pv3.3}.png}
-  \caption{\label{fig:allsky.astrom.sigma} Consistency of astrometry
-    measurements across the sky.  Each panel shows a map of the
-    standard deviation of astrometry residuals for stars in each
-    pixel.  The median value of the standard deviations across the sky
-    is $(\sigma_\alpha, \sigma_\delta) = (22, 23)$ milliarcseconds.
-    These values reflect the typical single-measurement errors for
-    bright stars.  See discussion regarding the astrometric flat which
-    is likely responsible for these elevated value. }
-  \end{center}
-\end{figure*}
 After the initial analysis to measure the KE corrections, DCR
 corrections, and astrometric flat-field corrections, we applied these
 …
 \label{sec:galactic.rotation}
 The initial analysis of the PV2 astrometry used the 2MASS positions as
+The analysis of the PV2 astrometry used the 2MASS positions as
 an inertial constraint: the 2MASS coordinates were included in the
 calculation of the mean positions for the objects in the database,
 …
 distance to our reference stars was \approx 500 pc.
 For PV3, we desired to address this bias by including our knowledge
+about the distances to the reference stars and the expected typical
+proper motions for stars at those distances.  With some constraint on
+the distance to each star, we can determine the expected proper motion
+based on a model of the Galactic rotation and solar motions.  We can
+then calculate the mean positions for the objects keeping the assumed
+proper motion fixed.  When calibrating a specific image, the reference
 star mean position is then translated to the expected position at the
 epoch of that image.  The image calibration is then performed relative
+to these predicted positions.  This process naturally accounts for the
+proper motion of the reference stars.  In order to make the
+calibrations consistent with the observed coordinates of an external
 inertial reference, we perform the iterative fits using the technique
+as described, but assign very high weights in the initial iterations
+to the inertial reference, and reduce the weights as the astrometric
 calibration iterations proceed.
+For the PV3 analysis, we desired to address this bias by including our
+knowledge about the distances to the reference stars and the expected
+typical proper motions for stars at those distances.  With some
+constraint on the distance to each star, we can determine the expected
+proper motion based on a model of the Galactic rotation and solar
+motions.  We can then calculate the mean positions for the objects
+keeping the assumed proper motion fixed.  When calibrating a specific
+image, the reference star mean position is then translated to the
+expected position at the epoch of that image.  The image calibration
+is then performed relative to these predicted positions.  This process
+naturally accounts for the proper motion of the reference stars.  In
+order to make the calibrations consistent with the observed
+coordinates of an external inertial reference, we perform the
+iterative fits using the technique as described, but assign very high
+weights in the initial iterations to the inertial reference, and
+reduce the weights as the astrometric calibration iterations proceed.
 In order to perform this analysis, we need estimated distances for
 …
 value of 500pc.
+\subsection{Gaia Constraint}
+\note{move comparisons to Gaia to the discussion, limit this section
+  to the Gaia astrometric tie}
+After the full relative astrometry analysis was performed for the PV3
+database, the Gaia Data Release 1 became available
+\citep{2016AA...595A...2G,2016AA...595A...4L}.  This afforded us
+the opportunity to constrain the astrometry on the basis of the Gaia
+observations.  Gaia DR1 objects which are bright enough to have proper
+motion and parallax solutions are in general saturated in the PS1
+observations.  Thus, we are limited to using the Gaia mean positions
+reported for the fainter stars.  We extracted all Gaia sources not
+marked as a duplicate from the Gaia archive and generated a DVO
+database from this dataset.  We then merged the Gaia DVO into the PV3
+master DVO database.  We re-ran the complete relative astrometry
+analysis using Gaia as an additional measurement.  We applied the
+analysis described above, applying the estimated distances to
+determine preliminary proper motions.  The Gaia mean epoch is reported
+as 2015.0, so all Gaia measurements were assigned this epoch.  We
+wanted to ensure the Gaia measurements dominated the astrometric
+solutions, so we made the weight very high for the Gaia points:
+$\times$ the nominal weight in the initial fits (to lock down the
+reference frame), decreasing to 100$\times$ the nominal weight for the
+last fits.  We also retained the 2MASS measurements in the analysis,
+but gave them somewhat lower weights than Gaia: while the 2MASS data
+does not have the accuracy of Gaia, the coverage is known to be quite
+complete, while the Gaia DR1 has clear gaps and holes.  Having 2MASS,
+even at a lower weight, helps to tile over those gaps.
+\begin{figure*}[htbp]
+  \begin{center}
+  \includegraphics[width=\hsize,clip]{{pics/gaia.photom}.png}
+  \caption{\label{fig:gaia.photom} Comparison with Gaia
+    photometry. {\bf Left} Mean of PS1 - Gaia, {\bf Right} Standard
+    deviation of PS1 - Gaia.  For pixels with $|b| > 30$ and $\delta >
+    -30$, the standard deviation of the PS1 - Gaia mean values is 7
+    millimagnitudes, while the median of the standard deviations is 12
+    millimagnitudes.  The former is a statement about the consistency
+    of the Gaia and Pan-STARRS\,1 photometry, while the latter
+    reflects the combined bright-end errors for both systems.  }
+  \end{center}
+\end{figure*}
+Figure~\ref{fig:gaia.photom} shows a comparison between the Pan-STARRS
+photometry in $g,r,i$ and the Gaia photometry in the $G$-band.  To
+compare the PS1 photometry to the very broadband Gaia G filter, we
+have determined a transformation based on a 3rd order polynomial fit
+to $g-r$ and $g-i$ colors.  This transformation reproduces Gaia
+photometry reasonably well for stars which are not too red.  For a
+comparison, we have selected all PS1 stars with Gaia measurements
+meeting the following criteria: $14 < i < 19$, with at least 10 total
+measurements, within a modest color range $0.2 < g - r < 0.9$.  We
+also restricted to objects with $i_{\rm PSF} - i_{\rm Kron} < 0.1$,
+using the average $i$ magnitudes determined from the individual
+exposures.
+For Figure~\ref{fig:gaia.photom}, we calculate the difference between
+the estimated $G$-band magnitude based on PS1 $g,r,i$ photometry and
+the $G$-band photometry reported by Gaia.  For each pixel, we
+determine the histogram of these differences and calculate the median
+and the 68\%-ile range.  In Figure~\ref{fig:gaia.photom}, these
+values are plotted as a color scale.
+The Galactic plane is clearly poorly matched between the two
+photometry systems.  This may in part be due to the difficulty of
+predicting $G$-band magnitudes for stars which are significantly
+extincted: the $G$-band includes significant flux from the PS1
+$z$-band which was not used in our transformation.  Many other large
+scale feature in the median differences have structures similar to the
+Gaia scanning pattern (large arcs and long parallel lines.  There are
+also structures related to the PS1 exposure footprint.  These show up
+as a mottling on the \approx 3 degree scale (e.g., lower right below
+the Galactic plane).  The amplitude of the residual structures is
+fairly modest.  The standard devition of the median difference values
+is 7 millimagnitudes.  This number gives an indication of the overall
+photometric consistency of both Gaia and PS1 and implies that the
+systematic error floor for each survey is less than 7 millimags.
+% set Gr = -0.090 + gr*gi*0.229 + gi*(-0.207+gi*(gi*0.015 - 0.250)) + gr*(0.491+gr*(-0.021*gr - 0.052))
+%\begin{equation}
+%G - r = -0.09 + 0.229(g-r)(g-r) + (g-i)((
+\begin{figure*}[htbp]
+  \begin{center}
+  \includegraphics[width=\hsize,clip]{{pics/gaia.astrom}.png}
+  \caption{\label{fig:gaia.astrom} Comparison with Gaia
+    astrometry. {\bf Left} Mean of PS1 - Gaia, {\bf Right} Standard
+    deviation of PS1 - Gaia.  The median value of the standard
+    deviations is $(\sigma_\alpha, \sigma_\delta) = (4, 3)$
+    milliarcseconds. }
+  \end{center}
+\end{figure*}
+Figure~\ref{fig:gaia.astrom} shows a comparison between the Pan-STARRS
+mean astrometry positions in $\alpha,\delta$ and the Gaia astrometry.
+For this comparison, we have seleted all PS1 stars with Gaia
+measurements with $14 < i < 19$ and with at least 10 total
+measurements.  For Figure~\ref{fig:gaia.astrom}, we calculate the
+difference between the position predicted by PS1 at the Gaia epoch
+(using the proper motion and parallax fit) and the position reported
+by Gaia.  For each pixel, we determine the histogram of these
+differences in the R.A\. and DEC directions, and calculate the median
+and the 68\%-ile range.  In Figure~\ref{fig:gaia.astrom}, these
+values are plotted as a color scale.
+There is good consistency between the PS1 and Gaia astrometry.  There
+are patterns from the Galactic plane (though not very strongly at the
+bulge).  There are also clear features due to the PS1 exposure
+footprint (ring structure on \approx 3 degree scales).  In the plots
+of the scatter, there are patterns which are related to the Gaia
+scanning rule.  These are presumably regions with relatively low
+signal to noise in Gaia; they were also apparent in the plots of the
+statisics of the per-exposure measurement residuals
+(Figure~\ref{fig:allsky.astrom.sigma}.  The standard deviations of the
+median differences are ($\sigma_\alpha, \sigma_\delta) = (4, 3)$
+milliarcseconds.
+For a future data release, we will recalibrate the Pan-STARRS $3\pi$
+astrometry using the Gaia DR2 release.  The addition of Gaia-measured
+proper motions will obviate the need to correct for the Galactic rotation.
+For the initial PV3 analysis, we again used the 2MASS coordinates as
+an external astrometric reference.  After the DR1 object parameters
+were ingested into the PSPS database, the Gaia DR1 astrometry was
+released \citep{2016AA...595A...4L}.  This gave us the option to use
+the Gaia positions for the external astrometric reference.  We re-did
+the astrometric analysis and generated a Gaia-based astrometry table
+for the Pan-STARRS DR1.  For Pan-STARRS DR2, the average object
+coordinates are based on the analysis using the Gaia coordinates.  The
+Gaia DR1 coordinates used a fixed 2015 epoch.  Coordinates were
+propagated from that epoch to the epoch for each PS1 image as
+described above.
 \subsection{Object Astrometry}
 …
 were available for an object, {\em all} measurements for that object
 are marked with the bit-flag \code{ID_MEAS_OBJECT_HAS_2MASS} or
 \code{ID_MEAS_OBJECT_HAS_GaIA} as appropriate.  The Tycho 2.0
+\code{ID_MEAS_OBJECT_HAS_GAIA} as appropriate.  The Tycho 2.0
 measurements were not included in this analysis and objects with Tycho
 measurements are therefore not marked.
 …
 will be set for the object.
+\subsection{Astrometry Calibration Quality}
+\begin{figure*}[htbp]
+  \begin{center}
+ \includegraphics[width=\hsize,clip]{{pics/allsky.astrom.pv3.3}.png}
+  \caption{\label{fig:allsky.astrom.sigma} Consistency of astrometry
+    measurements across the sky.  Each panel shows a map of the
+    standard deviation of astrometry residuals for stars in each
+    pixel.  The median value of the standard deviations across the sky
+    is $(\sigma_\alpha, \sigma_\delta) = (22, 23)$ milliarcseconds.
+    These values reflect the typical single-measurement errors for
+    bright stars.  See discussion regarding the astrometric flat which
+    is likely responsible for these elevated value. }
+  \end{center}
+\end{figure*}
+\begin{figure*}[htbp]
+  \begin{center}
+  \includegraphics[width=\hsize,clip]{{pics/astroflat.repair}.png}
+  \caption{\label{fig:astroflat.repair} Comparison of the
+    high-resolution astrometric flat-field images used for PV3.2
+    (left) and for PV3.3 (right).  These examples show the \gps-band
+    astrometric flat-field corrections for the $X$ direction as seen
+    in the focal plane coordinate system.  Note the elevated noise in
+    the PV3.2 image due to insufficient numbers of stars used in the analysis.
+}
+\end{center}
+\end{figure*}
+% numbers of stars used:
+%% mana: load.stars astroflat.20151205/astroflat.20151205.v1.Npt.fits
+%% filter g : 2591421 stars
+%% filter r : 3497036 stars
+%% filter i : 16241986 stars
+%% filter z : 7153595 stars
+%% filter y : 4509749 stars
+%% mana: load.stars astroflat.20170217/astroflat.20170217.Npt.fits
+%% filter g : 17590560 stars
+%% filter r : 31000135 stars
+%% filter i : 82648850 stars
+%% filter z : 62166619 stars
+%% filter y : 42867074 stars
+\begin{figure*}[htbp]
+  \begin{center}
+  \includegraphics[width=\hsize,clip]{{pics/allsky.histogram.astrom.compare}.png}
+  \caption{\label{fig:allsky.astro.histogram} Illustration of the
+    impact of the astrometric flat-field correction used for PV3.2 vs
+    PV3.3.  The blue histograms show the distribution of astrometric
+    residuals for bright stars from the PV3.2 analysis while the red
+    histograms show the distribution for the PV3.3 analysis.  The
+    median standard deviation for PV3.2 is 22 milliarcseconds in R.A.
+    (23 mas in Declination).  Using the higher signal-to-noise
+    flat-field correction images reduces the median values to 16 mas
+    for both R.A. and Declination directions in PV3.3.
+}
+\end{center}
+\end{figure*}
+% generate (or plot) astrometric flat-field images for DR2 (PV3.X)
+Figure~\ref{fig:allsky.astrom.sigma} shows the standard deviations of
+the mean residual astrometry in $(\alpha,\delta)$ for bright stars as
+a function of position across the sky based on the DR2 calibration.  For each
+pixel in these images, we selected all objects with $15 < i < 17$,
+with at least 3 measurements in $i$-band (to reject artifacts detected
+in a pair of exposures from the same night), with \code{PSF_QF} $>
+.85$ (to reject excessively-masked objects), and with $mag_{\rm PSF}
+- mag_{\rm Kron} < 0.1$ (to reject galaxies).  We then generated
+histograms of the difference between the object position predicted for
+the epoch of each measurement (based on the proper motion and parallax
+fit) and the observed position of that measurement, in both the Right
+Ascension and Declination directions (in linear arcseconds), for all
+stars in a given pixel in the images.  From these residual histograms,
+we can then determine the median and the 68\%-ile range to calculate a
+robust version of the standard deviation.  This represents the
+bright-end systematic error floor for a measurement from a single
+exposure.  The standard deviations are then plotted in
+Figure~\ref{fig:allsky.astrom.sigma}.  The median value of the
+standard deviations across the sky in both $(\sigma_\alpha,
+\sigma_\delta)$ is 16 milliarcseconds.
+The Galactic plane is clearly apparently in these images.  Like
+photometry, we attribute this to failure of the PSF fitting due to
+crowding.  The celestial North pole regions have somewhat elevated
+errors in both R.A. and DEC, with some specifc structures.  Some of
+these structures may be due to the larger typical seeing at these high
+airmass regions, but some are due to astrometric failures which stem
+from the reference catalog based on the PV2 analysis (see
+Section~\ref{sec:pole.problems} for further details).  Several
+features can be seen which appear to be an effect of the tie to the
+Gaia astrometry: the stripes near the center of the DEC image and the
+right side of the R.A. image.  The mesh of circular outlines one the 2
+degree scale is due to the outer edge of the focal plane where the
+astrometric calibration is poorly determined.
+The DR1 astrometric calibration suffered from degraded astrometry due
+to a problem with the astrometric flat-field correction identified too
+late to be repaired for DR1.
+%
+The astrometric flat-field images used
+for that release had too few stars to measure the correction with
+sufficient signal-to-noise.  As a result, those corrections had
+significant pixel-to-pixel noise which can be seen in
+Figure~\ref{fig:astroflat.repair}.  As a result, the astrometric
+flat-field correction reduces systematic structures on large spatial
+scales, but at the expense of degrading the quality of individual
+measurements.  Only the $i$-band flat had sufficient signal-to-noise
+per pixel to avoid significantly increasing the per-measurement
+position errors.
+For DR2, we recalculated the astrometric flat-field correction using
+many more stars.  For the DR1 release, the number of stars per filter
+was (\grizy) = (2.6M, 3.5M, 16M, 7M, 4.5M), while for the DR2 release,
+the number of stars per filter was (\grizy) = (18M, 31M, 83M, 62M,
+M).  We also reduced the resolution of the astrometric flat-field,
+using $80 \times 80$ superpixels, rather than the $40 \times 40$
+superpixels used for DR1.  Because of the degraded astrometric
+flat-field correction, the median per-measurement error floor of DR1
+is \approx 22 mas, significantly worse than both DR2 and the earlier
+PV2 analysis.  Figure~\ref{fig:allsky.astro.histogram} shows
+histograms of the astrometric residual scatter across the sky for DR1
+and DR2, illustrating the improvement.
+% older version of this figure:
+% pv2_0 : /data/ipp060.0/eugene/pv2.astrom.20150126/astromap.20150127/dDsig.im.fits
+% pv2_1 : /data/ipp060.0/eugene/pv2.astrom.20150126/astromap.20150429/dDsig.im.fits
+% NOTE:
+% the pv2 versions used:  resize 1800 920; region 0 0 85 ait
+% the pv3 versions used:  resize 1800 950; region 180 0 90 ait
+% thus we cannot directly compare map pixels, without re-extracting the measurements
+% (we can do that if we decide it is needed to generate the best plots)
+% original version of figure: pv3.stats.20161202/allsky.astrom.sigma.png
+%   based on /data/kukui.3/eugene/pv3.stats.20161202/maps.measure/pv3.v1.*.sigma.fits
+%   based on /data/ipp094.0/eugene/pv3.stats.20161202/cdhist.measure/cdmerge.v1.dD.fits (& dR)
+%   plot script /data/kukui.3/eugene/pv3.stats.20161202/scatter.sh
+%   catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master (PV3.2)
+% regenerate using fits image in pv3.stats.20170413
 \section{Discussion}
 \label{sec:discussion}
+\subsection{Calibration Versions}
 The calibration of the PV3 DVO database required several iterations.
 For completeness, we discuss these steps and their implications for
 the DR1 and DR2 releases.
+\begin{itemize}
 \item[PV3.0] The first calibrated PV3 database is identified as PV3.0.
+\paragraph{PV3.0}
+The first calibrated PV3 database is identified as PV3.0.
   This calibration predates the Gaia DR1 release and uses the 2MASS
   catalog as a reference.  After internal testing, an error in the
 …
   with the wrong sign to the measurements.
 \item[PV3.1] After the above error was identified, the photometric
+\paragraph{PV3.1} After the above error was identified, the photometric
   flat-field correction was applied in the correct sense to the
   measurements and the average photometry was recalculated.  The
 …
   (but see below regarding the mean positions).
 \item[PV3.2] The Gaia DR1 release motivated a recalibration of the
+\paragraph{PV3.2} The Gaia DR1 release motivated a recalibration of the
   astrometry using the Gaia DR1 position information, combined with
   photometric distance estimates and a model for the Galactic and
 …
   release.
 \item[PV3.3] After the DR1 release, we identified a problem with the
+\paragraph{PV3.3} After the DR1 release, we identified a problem with the
   astrometric flat-field corrections (see
   Section~\ref{sec:astro.flat}): for all but the \ips\ filter, the
 …
   noticable improvement in the astrometric scatter for bright stars.
 \item[PV3.4] Two errors were identified in the PV3.3 calibration
+\paragraph{PV3.4} Two errors were identified in the PV3.3 calibration
   before the DR2 release was completed.  First, we discovered that the
   repair applied to the photometric flat-field correction for PV3.1,
 …
   these issue in the PV3.4 calibration of the DVO database.  This
   database was then used to generate the values in the DR2 PSPS
+  database tables.  \note{what about P2, those were done first, right?}
+\end{itemize}
+  database tables.
+\subsection{Comparison to Gaia}
+After the full relative astrometry analysis was performed for the PV3
+database, the Gaia Data Release 1 became available
+\citep{2016AA...595A...2G,2016AA...595A...4L}.  This afforded us
+the opportunity to constrain the astrometry on the basis of the Gaia
+observations.  Gaia DR1 objects which are bright enough to have proper
+motion and parallax solutions are in general saturated in the PS1
+observations.  Thus, we are limited to using the Gaia mean positions
+reported for the fainter stars.  We extracted all Gaia sources not
+marked as a duplicate from the Gaia archive and generated a DVO
+database from this dataset.  We then merged the Gaia DVO into the PV3
+master DVO database.  We re-ran the complete relative astrometry
+analysis using Gaia as an additional measurement.  We applied the
+analysis described above, applying the estimated distances to
+determine preliminary proper motions.  The Gaia mean epoch is reported
+as 2015.0, so all Gaia measurements were assigned this epoch.  We
+wanted to ensure the Gaia measurements dominated the astrometric
+solutions, so we made the weight very high for the Gaia points:
+$\times$ the nominal weight in the initial fits (to lock down the
+reference frame), decreasing to 100$\times$ the nominal weight for the
+last fits.  We also retained the 2MASS measurements in the analysis,
+but gave them somewhat lower weights than Gaia: while the 2MASS data
+does not have the accuracy of Gaia, the coverage is known to be quite
+complete, while the Gaia DR1 has clear gaps and holes.  Having 2MASS,
+even at a lower weight, helps to tile over those gaps.
 \begin{figure*}[htbp]
   \begin{center}
+  \includegraphics[width=\hsize,clip]{{pics/photom.pv3.3v4}.png}
+  \caption{\label{fig:photom.pv3.3v4} Sample comparison of PV3.3 and
+    PV3.4 photometry illustrating the impact of the issues identified
+    in the PV3.3 stack and warp photometry.  All figures use \ips-band
+    photometry.  The left panels use data from PV3.3 while the right
+    use PV3.4.  The top row shows the mean difference between the
+    average photometry from individual exposures (``chip'') and the
+    stack photometry using Kron magnitudes.  The middle row shows the
+    mean difference between the average photometry from individual
+    exposures (``chip'') and the average forced-warp photometry, again
+    using Kron magnitudes.  The bottom row shows the mean difference
+    between the average photometry from individual exposures
+    (``chip'') and the average forced-warp photometry, using PSF
+    magnitudes.  See Section~\ref{sec:discussion} for a description of
+    the calibration change in PV3.4.}
+\end{center}
+  \includegraphics[width=\hsize,clip]{{pics/gaia.photom}.png}
+  \caption{\label{fig:gaia.photom} Comparison with Gaia
+    photometry. {\bf Left} Mean of PS1 - Gaia, {\bf Right} Standard
+    deviation of PS1 - Gaia.  For pixels with $|b| > 30$ and $\delta >
+    -30$, the standard deviation of the PS1 - Gaia mean values is 7
+    millimagnitudes, while the median of the standard deviations is 12
+    millimagnitudes.  The former is a statement about the consistency
+    of the Gaia and Pan-STARRS\,1 photometry, while the latter
+    reflects the combined bright-end errors for both systems.  }
+  \end{center}
 \end{figure*}
+Figure~\ref{fig:gaia.photom} shows a comparison between the Pan-STARRS
+photometry in $g,r,i$ and the Gaia photometry in the $G$-band.  To
+compare the PS1 photometry to the very broadband Gaia G filter, we
+have determined a transformation based on a 3rd order polynomial fit
+to $g-r$ and $g-i$ colors.  This transformation reproduces Gaia
+photometry reasonably well for stars which are not too red.  For a
+comparison, we have selected all PS1 stars with Gaia measurements
+meeting the following criteria: $14 < i < 19$, with at least 10 total
+measurements, within a modest color range $0.2 < g - r < 0.9$.  We
+also restricted to objects with $i_{\rm PSF} - i_{\rm Kron} < 0.1$,
+using the average $i$ magnitudes determined from the individual
+exposures.
+For Figure~\ref{fig:gaia.photom}, we calculate the difference between
+the estimated $G$-band magnitude based on PS1 $g,r,i$ photometry and
+the $G$-band photometry reported by Gaia.  For each pixel, we
+determine the histogram of these differences and calculate the median
+and the 68\%-ile range.  In Figure~\ref{fig:gaia.photom}, these
+values are plotted as a color scale.
+The Galactic plane is clearly poorly matched between the two
+photometry systems.  This may in part be due to the difficulty of
+predicting $G$-band magnitudes for stars which are significantly
+extincted: the $G$-band includes significant flux from the PS1
+$z$-band which was not used in our transformation.  Many other large
+scale feature in the median differences have structures similar to the
+Gaia scanning pattern (large arcs and long parallel lines.  There are
+also structures related to the PS1 exposure footprint.  These show up
+as a mottling on the \approx 3 degree scale (e.g., lower right below
+the Galactic plane).  The amplitude of the residual structures is
+fairly modest.  The standard devition of the median difference values
+is 7 millimagnitudes.  This number gives an indication of the overall
+photometric consistency of both Gaia and PS1 and implies that the
+systematic error floor for each survey is less than 7 millimags.
+% set Gr = -0.090 + gr*gi*0.229 + gi*(-0.207+gi*(gi*0.015 - 0.250)) + gr*(0.491+gr*(-0.021*gr - 0.052))
+%\begin{equation}
+%G - r = -0.09 + 0.229(g-r)(g-r) + (g-i)((
+\begin{figure*}[htbp]
+  \begin{center}
+  \includegraphics[width=\hsize,clip]{{pics/gaia.astrom}.png}
+  \caption{\label{fig:gaia.astrom} Comparison with Gaia
+    astrometry. {\bf Left} Mean of PS1 - Gaia, {\bf Right} Standard
+    deviation of PS1 - Gaia.  The median value of the standard
+    deviations is $(\sigma_\alpha, \sigma_\delta) = (4, 3)$
+    milliarcseconds. }
+  \end{center}
+\end{figure*}
+Figure~\ref{fig:gaia.astrom} shows a comparison between the Pan-STARRS
+mean astrometry positions in $\alpha,\delta$ and the Gaia astrometry.
+For this comparison, we have seleted all PS1 stars with Gaia
+measurements with $14 < i < 19$ and with at least 10 total
+measurements.  For Figure~\ref{fig:gaia.astrom}, we calculate the
+difference between the position predicted by PS1 at the Gaia epoch
+(using the proper motion and parallax fit) and the position reported
+by Gaia.  For each pixel, we determine the histogram of these
+differences in the R.A\. and DEC directions, and calculate the median
+and the 68\%-ile range.  In Figure~\ref{fig:gaia.astrom}, these
+values are plotted as a color scale.
+There is good consistency between the PS1 and Gaia astrometry.  There
+are patterns from the Galactic plane (though not very strongly at the
+bulge).  There are also clear features due to the PS1 exposure
+footprint (ring structure on \approx 3 degree scales).  In the plots
+of the scatter, there are patterns which are related to the Gaia
+scanning rule.  These are presumably regions with relatively low
+signal to noise in Gaia; they were also apparent in the plots of the
+statisics of the per-exposure measurement residuals
+(Figure~\ref{fig:allsky.astrom.sigma}.  The standard deviations of the
+median differences are ($\sigma_\alpha, \sigma_\delta) = (4, 3)$
+milliarcseconds.
+For a future data release, we will recalibrate the Pan-STARRS $3\pi$
+astrometry using the Gaia DR2 release.  The addition of Gaia-measured
+proper motions will obviate the need to correct for the Galactic rotation.
 \section{Conclusion}
+\note{WRITE THIS}
 \acknowledgments
 …
 under Grant No. AST-1238877, the University of Maryland, and Eotvos
 Lorand University (ELTE) and the Los Alamos National Laboratory.
 \note{colormaps by Peter Kovesi. Good Colour Maps: How to Design Them.
+arXiv:1509.03700 [cs.GR] 2015.  add ref}
+Colormaps for Figures \ref{fig:photflat},
+\ref{fig:allsky.photom.sigma}, \ref{fig:photom.pv3.3v4},
+\ref{fig:astroflat.gri}, \ref{fig:astroflat.zy},
+\ref{fig:allsky.astrom.sigma}, and \ref{fig:astroflat.repair} from
+Peter Kovesi \citep[Good Colour Maps: How to Design Them.][]{2015arXiv150903700K}.
 \bibliographystyle{apj}
 % \bibliography{lib}{}
 \input{calibration.bbl}
+\bibliography{lib}{}
+% \input{calibration.bbl}
 \end{document}

Note: See TracChangeset for help on using the changeset viewer.

Context Navigation

Changeset 40634 for trunk

Legend:

trunk/doc/release.2015/ps1.calibration/calibration.tex

Download in other formats: