Index: trunk/doc/release.2015/ps1.calibration/calibration.tex =================================================================== --- trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 39839) +++ trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 39840) @@ -1,12 +1,13 @@ -\documentclass[iop,floatfix]{emulateapj} +% \documentclass[iop,floatfix]{emulateapj} % \pdfoutput=1 % see latex.readme.txt for notes on using the PS1 template -%\documentclass[12pt,preprint]{aastex} +\documentclass[12pt,preprint]{aastex} %\documentclass[manuscript]{aastex} %\documentclass[preprint2]{aastex} %\documentclass[preprint2,longabstract]{aastex} \RequirePackage{color} -% \input{astro.sty} +\RequirePackage{code} +\input{astro.sty} % online version may use color, but print version needs b/w @@ -86,5 +87,5 @@ % insert additional keywords as appropriate: -\keywords{Surveys:\PSONE } +%\keywords{Surveys:\PSONE } \section{Introduction}\label{sec:intro} @@ -141,10 +142,9 @@ images are also corrected for fringing: a master fringe pattern is scaled to match the observed fringing and subtracted. Mask and -variance image arrays are generated with the \changed{detrend - analysis} and carried forward at each stage of the IPP processing. -Source detection and photometry are performed for each chip -independently. As discussed below, preliminary astrometric and -photometric calibrations are performed for all chips in a single -exposure in a single analysis. +variance image arrays are generated with the detrend analysis and +carried forward at each stage of the IPP processing. Source detection +and photometry are performed for each chip independently. As +discussed below, preliminary astrometric and photometric calibrations +are performed for all chips in a single exposure in a single analysis. Chip images are geometrically transformed based on the astrometric @@ -266,5 +266,5 @@ A third form of the astrometry model is used in the context of the calibration determined within the DVO database system. We retain the -two levels of transformations (chip $\rtarrow$ focal plane $\rtarrow$ +two levels of transformations (chip $\rightarrow$ focal plane $\rightarrow$ tangent plane), but the relationship between the chip and focal plane is represented with only the linear terms in the polynomial, @@ -359,6 +359,6 @@ stars. For all possible pairs between the two lists, the values of \[ -$\Delta X = X^{\rm ref}_{\rm chip} - X^{\rm obs}_{\rm chip}\\ -$\Delta Y = Y^{\rm ref}_{\rm chip} - Y^{\rm obs}_{\rm chip} +\Delta X = X^{\rm ref}_{\rm chip} - X^{\rm obs}_{\rm chip}\\ +\Delta Y = Y^{\rm ref}_{\rm chip} - Y^{\rm obs}_{\rm chip} \] are generated. The collection of $\Delta X, \Delta Y$ values are @@ -404,6 +404,6 @@ The astrometry solutions from the independent chip fits are used to generate a single model for the camera-wide distortion terms. The -goal is to determine the two stage fit (chip $\rtarrow$ focal plane -$\rtarrow$ tangent plane). There are a number of degenerate terms +goal is to determine the two stage fit (chip $\rightarrow$ focal plane +$\rightarrow$ tangent plane). There are a number of degenerate terms between these two levels of transformation, most obviously between the parameters which define the constant offset from chip to focal plane @@ -424,4 +424,6 @@ \note{describe the output smf file?} + +\note{discuss the real-time photometric calibration} \section{DVO Description} @@ -522,5 +524,5 @@ contains 9 times as many rows as the Average table. The order of the table is fixed in relation to the Average table: row $i$ of Average -defines the object with photometry contained in rows $9i \rtarrow 9i + +defines the object with photometry contained in rows $9i \rightarrow 9i + 8$ ($i$ is zero counting). @@ -716,5 +718,5 @@ such as the instrumental variations and atmospheric attenuation. \[ -M_{rel} & = & m_{inst} + ZP + M_{cal} \\ +M_{rel} = m_{inst} + ZP + M_{cal} \] @@ -877,5 +879,5 @@ values for each GPC1 chip (40x40 pixels per point). \note{show the flat-field residual images, discuss the features?}. We then used -\node{setphot} to apply this new flat-field correction, as well as the +\code{setphot} to apply this new flat-field correction, as well as the ubercal flat-field corrections, to the data in the database. At this point, we re-ran the entire relphot analysis to determine zero points @@ -896,15 +898,316 @@ the bright end. \note{recommendation} +\section{PV3 DVO Master Database} + +Data from the GPC1 chip images, the stack images, and the warp images +are loaded into DVO using the real-time analysis astrometric +calibration to guide the association of detections into objects. +After the full PV3 DVO database was constructed, including all of the +chip, stack, and warp detections, several external catalogs were +merged into the database. First, the complete 2MASS PSC was loaded +into a stand-alone DVO database, which was then merged into the PV3 +master database. Next the DVO database of synthetic photometry in +the PS1 bands (see Section~\ref{sec:synthdb}) was merged in. Next, +the full Tycho database was added, followed by the AllWISE database. +After the Gaia release in August 2016, we generated a DVO database of +the Gaia positional and photometric information and merged that into +the master DVO database. + +\note{need to describe the assignment of flags, etc, for the external + data sources}. + \section{Astrometry Analysis} -\begin{verbatim} -* initial astrometry based on real-time calibration -* relative astrometry calibration of images - * bright objects, images -* first pass to deter -\end{verbatim} - -\section{Systematic Residuals} - -\subsection{Camera-Scale Trends} + +Once the full PV3 dataset loaded into the master PV3 DVO database, +along with supporting databases, and the photometric calibrations were +performed, relative astrometry could be performed on the database to +improve the overall astrometric calibration. + +In many respects the relative astrometric analysis is similar to the +relative photometric analysis: the repeated measurements of the same +object in different images are used to determine a high quality +average position for the object. The new average positions are then +used to determine improved astrometric calibrations for each of the +images. These improved calibrations are used to set the observed +coordinates of the measurements from those images, which are in turn +used to improve the average positions of the objects. The whole +process is repeated for several iterations. Like the photometric +analysis, the astrometric analysis is performed in a parallel fashion +with the same concept that specific machines are responsible for +exposures and objects which land within their regions of +responsibility, defined on the basis of lines of constant RA and DEC. +Between iteration steps, the astrometric calibrations are shared +between the parallel machines as are the improved positions for +objects controlled by one machine but detect in images controlled by +another machine. Like the photometric analysis, the entire sky is +processed in one pass. However, there are some important differences +in the details. + +\subsection{Systematic Effects} + +First, the astrometric calibration has a larger number of systematic +effects which must be performed. These consist of: 1) the +Koppenh\"offer Effect, 2) Differential Chromatic Refraction, 3) Static +deviations in the camera. We discuss each of these in turn below. + +\subsubsection{Koppenh\"offer Effect} + +The Koppenh\"offer Effect was first identified (DATE) by Johannes +Koppenh\"offer (MPE) as part of the effort to search for planet +transists in the Stellar Transit Survey data. He noticed that the +astromety of bright stars and faint stars disagreed on overlapping +chips at the boundary between the STS fields. After some exploration, +it was determined that the X coordinate of the brightest stars was +offset from the expected location based on the faint stars for a +subset of the GPC1 chips. The essence of the effect was that the +bright stars were advanced along the serial register more quickly than +they should have been. The brighter the star, the more the charge +cloud was pushed ahead on the serial register. The amplitude of the +effect was at most \note{XXX}. Only the \note{2-phase} chips suffered +from this effect. By adjusting the \note{which?} voltages on the +camera, the effect was prevented in exposures after \note{DATE}. +However, this left \note{XXX,XXX exposures (XX\%)} already +contaminated by the effect. + +We measured the Koppenh\"offer Effect by accumulating the residual +astrometry statistics for \note{how many} stars. For each chip, we +measured the mean X and Y displacements of the astrometric residuals +as function of the instrumental magnitude of the star divided by the +FWHM$^2$. \note{was there is significant difference using a surface + brightness version?} We measured the trend for all chips in a +number of different time ranges and found the effect to be quite +stable, in the period where it was present. The effect only appeared +in the serial direction. Figure~\ref{fig:koppenhoefer} shows the KE +trend for a typical affected chip both before and after the +correction. For the PV3 dataset, we re-measured the KE trends using +stars in the Galactic pole regions after an initial relative +astrometry calibration pass: the Galactic pole is necessary because +the real-time astrometric calibration relies largely on the fainter +stars which are not affected by the KE. The trend is then stored in a +form which can be applied to the database measurements. + +\subsubsection{Differential Chromatic Refraction} + +Differential Chromatic Refraction (DCR) affects astrometry because the +reference stars used the calibrate the images are not the same color +(SED) as the rest of the stars in the image. For a given star of a +color different from the reference stars, as exposures are taken at +higher airmass, the apparent position of the star will be biased along +the parallactic angle. While it is possible to build a model for the +DCR impact based on the filter response functions and atmospheric +refraction, we have instead elected to use an empirical correction for +the DCR present in the PV3 database. We have measured the DCR trend +using the astrometric residuals of millions of stars after performing +an initial relative astrometry calibration. We define a blue DCR +color ($g-i$) to be used when correcting the filters \gps,\rps,\ips, and a red +DCR color ($z - y$) to be used when correcting the filters $zy$. In +the process of performing the relative astrometry calibration, we +record the median red and blue colors of the reference stars used to +measure the astrometry calibration for each image. As we determine +the astrometry parameters for each object in the database, we record +the median red and blue reference star colors for all images used to +determine the astrometry for a given object. For each star in the +database, we know both the color of the star and the typical color of +the reference stars used to calibrate the astrometry for that star. + +We measure the mean deviation of the residuals in the parallactic +angle direction and the direction perpendicular to the parallactic +angle. For each filter, we determine the DCR trend as a function of +the difference between the star color and the reference star color, +using the red or blue color approriate to the particular filter, times +the tangent of the zenith distance. Figure~\ref{fig:DCR} shows the +DCR trend for the 5 filters \grizy, as well as the measured +displacement in the direction perpendicular to the parallactic angle. +We represent the trend with a spline fitted to this dataset. The DCR +trend has an amplitude of \note{XXX - XXX} in the five filters. + +\note{write down the DCR formalae for reference}. + +\subsubsection{Astrometric Flat-field} + +After correction for both KE and DCR, we observe persistent residual +astrometric deviations which depend on the position in the camera. We +construct an astrometric ``flat-field'' response by determining the +mean residual displacement in the X and Y (chip) directions as a +function of position in the focal plane. We have measured the +astrometric flat using a sampling resolution of 40x40 pixels, matching +the photometric flat-field correction images. +Figure~\ref{fig:astroflat} shows the astrometric flat-field images for +the five filters \grizy\ in each of the two coordinate directions. +These plots show several types of features. + +The dominant pattern in the astrometric residual is roughly a series +of concentric rings. The pattern is similar to the pattern of the +focal surface residuals measured by (REF), which also has a concentric +series of rings with similar spacing. The ``tent'' in the center of +the focal surface reflected in these astrometry residual plots. Our +interpretation of the structure is that the deviations of the focal +plane from the ideal focal surface introduces small-scale PSF changes, +presumably coupled to the optical aberrations, which result in small +changes in the centroid of the object relative to the PSF model at +that location. Since the PSF model shape parameters are only able to +vary at the level of a 6x6 grid per chips, the finer structures are +not included in the PSF model. The PV2 analysis shows the ring +structure more clearly, with a pattern much more closely following the +focal surface deviations. In the PV2 analysis, the PSF model used at +most a 3x3 grid per chip to follow the shape variations, so any +changes caused by the optical aberrations would be less well modeled in +the PV2 analysis, as we observe. + +A second pattern which is weakly seen in several chips consists of +consistent displacements in the X (serial) direction for certain +cells. This effect can be seen most clearly in chips XY45 and XY46. +In the PV2 analysis, this pattern is also more clearly seen. In this +case, the fact that the astrometric model used polynomials with a +maximum of 3rd order per chip means the deviation of individual cells +cannot be followed by the astrometric model. + +A third effect is seen at the edge of the chips, where there appears +to be a tendency for the residual to follow the chip edge. The origin +of this is unclear, but likely caused by the astrometry model failing +to follow the underlying variations because of the need to extrapolate +to the edge pixels. Finally, we also identify an interesting effect +{\em not} visible at the resolution of these astrometric flat-field +images. Fine structures are observed at the \approx 10 pixel scale +similar to the ``tree rings'' reported by the DES team and others +(G. Berstein REF \& REFS). We explore these tree rings in detail in +\note{SECTION or REF?}. + +After the initial analysis to measure the KE corrections, DCR +corrections, and astrometric flat-field corrections, we applied these +corrections to the entire database. Within the schema of the +database, each measurement has the raw chip coordinates +(\code{Measure.Xccd,Yccd}) as well as the offset for that object based on each of +these three corrections: \code{Measure.XoffKH,YoffKH, + Measure.XoffDCR,YoffDCR, Measure.XoffCAM,YoffCAM}. The offsets are +calculated for each measurement based on the observed instrumental +chip magnitudes and FWHM for the Koppenhoffer Effect, on the average +chip colors and the altitude \& azimuth of each measurement for the +DCR correction, and on the chip coordinates for the astrometric +flat-field corrections. The corrections are combined and applied to +the raw chip coordinates and saved back in the database in the fields +\code{Measure.Xfix,Yfix}. At this point, we are ready to run the +full astrometric calibration. + +\subsection{Galactic Rotation and Solar Motion} + +The initial analysis of the PV2 astrometry used the 2MASS positions as +an inertial constraint: the 2MASS coordiates were included in the +calculation of the mean positions for the objects in the database, +with weight corresponding to the reported astrometric errors. In this +analysis, the object positions used to determine the calibrations of +the image parameters ignored proper motion and parallax. After the +image calibrations were determined, then individual objects were +fitted for proper motion and possibly parallax, as discussed in detail +below. + +Using the PV2 analysis of the astrometry calibration, we discovered +large-scale systematic trends in the reported proper motions of +background quasars. This motion had an amplitude of 10 - 15 +milliarcseconds per year and clear trends with Galactic longitude. We +also observed systematic errors of the mean positions with respect to +the ICRF milliarcsecond radio quasar positions, with an amplitude of +\approx 60 milliarcseconds, again with trends associated with Galactic +longitude. Since the 2MASS data were believed to have minimal average +deviations relative to the ICRF quasars, this latter seemed to be a +real effect. + +We realized that both the proper motion and the mean position biases +could be caused by a single common effect: the proper motion of the +stars used as reference stars between the 2MASS epoch (\approx 2000) +and PS1 epoch (\approx 2012). Since we are fitting the image +calibrations without fitting for the proper motions of the stars, we +are in essencence forcing those stars to have proper motions of 0.0. +The background quasars would then be observed to have proper motions +corresponding to the proper motions of the reference stars, but in the +opposite direction. We demonstrated that the observed quasar proper +motions agreed well with the distribution expected if the median +distance to our reference stars was \approx 500 pc. + +For PV3, we desired to address this bias by including our knowledge +about the distances to the reference stars and the expected typical +proper motions for stars at those distances. With some constraint on +the distance to each star, we can determine the expected proper motion +based on a model of the Galactic rotation and solar motions. We can +then calculate the mean positions for the objects keeping the assumed +proper motion fixed. When calibrating a specific image, the reference +star mean position is then translated to the expected position at the +epoch of that image. The image calibration is then performed relative +to these predicted postions. This process naturally accounts for the +proper motion of the reference stars. In order to make the +calibrations consistent with the observed coordinates of an external +inertial reference, we perform the iterative fits using the technique +as described, but assign very high weights in the initial iterations +to the inertial reference, and reduce the weights as the astrometric +calibration iterations proceed. + +In order to perform this analysis, we need estimated distances for +every reference star used in the analysis. Green et al (REF) +performed SED fitting for 800M stars in the 3$\pi$ region using PV2 +data. The goal of this work was to determine the 3D structure of the +dust in the galaxy. By fitting model SEDs to \note{all?} stars +meeting a basic data quality cut \note{(describe)}, they determined +the best spectral type, and thus $T_{\rm eff}$, absolute $r$-band +magnitude, distance modulus, and extinction $A_V$ (the desired output +and used to determine the dust extinction as a function of distance +throughout the galaxy). We use the distance modulus determined in +this analysis to predict the proper motions. + +To convert the distances to proper motions, we use the Galactic +rotation parameters ($A,B$) = (14.82,-12.37) km sec$^{-1}$ pc$^{-1}$ +and Solar motion parameters ($U_{\rm sol}, V_{\rm sol}, W_{\rm sol}$) += (9.32, 11.18, 7.61) km sec$^{-1}$ as determined by Feast \& +Whitelock (REF) using Hipparchos data. Proper motions are determined +from the following: +\begin{eqnarray} +\mu^{\rm gal}_{l} & = & (A \cos (2 l) + B) \cos (b) \\ +\mu^{\rm gal}_{b} & = & \frac{-A \sin (2 l) \sin (2 b)}{2} \\ +\mu^{\rm sol}_{l} & = & \frac{U \sin(l) - V \cos(l)}{d} \\ +\mu^{\rm sol}_{b} & = & \frac{(U \cos(l) + V \sin(l)) \sin(b) - W \cos(b)}{d} +\end{eqnarray} +where $d$ is the distance and $l,b$ are the Galactic coordintes of the +star. \note{some reference?} Note that the proper motion induced by +the Galactic rotation is independent of distance while the reflex +motion induced by the solar motion decreases with increasing +distance. Also note that this model assumes a flat rotation curve for +objects in the thin disk; any reference stars which are part of +the halo population will have proper motions which are not +described by this model; the mostly random nature of the halo motions +should act to increase the noise in the measurement, but should not +introduce detectable motion biases. Also, if the distance modulus is +not well determined, we can assume the object is simply following the +Galactic rotation curve and set a fixed proper motion. If we do not +have a distance modulus from the Green et al analysis, we assume a +value of 500pc. + +\note{plots to show how well this worked for PV3 pre Gaia} + +\subsection{Gaia Constraint} + +After the full relative astrometry analysis was performed for the PV3 +database, the Gaia Data Release 1 became available. This afforded us +the opportunity to constrain the astrometry on the basis of the Gaia +observations. Gaia DR1 objects which are bright enough to have proper +motion and parallax solutions are in general saturated in the PS1 +observations. Thus, we are limited to using the Gaia mean positions +reported for the fainter stars. We extracted all Gaia sources +\note{not marked as a duplicate} from \note{where?} and generated a +DVO database from this dataset. We then merged the Gaia DVO into the +PV3 master DVO database. We re-ran the complete relative astrometry +analysis using Gaia as an additional measurement. We applied the +analysis described above, applying the estimated distances to +determine preliminary proper motions. The Gaia mean epoch is reported +as 2015.0, so all Gaia measurements were assigned this epoch. We +wanted to ensure the Gaia measurements dominated the astrometric +solutions, so we made the weight very high for the Gaia points: +1000$\times$ the nominal weight in the initial fits (to lock down the +reference frame), decreasing to 100$\times$ the nominal weight for the +last fits. We also retained the 2MASS measurements in the analysis, +but gave them somewhat lower weights than Gaia: while the 2MASS data +does not have the accuracy of Gaia, the coverage is known to be quite +complete, while the Gaia DR1 has clear gaps and holes. Having 2MASS, +even at a lower weight, helps to tile over those gaps. + +\note{Figures showing the Gaia residuals} \section{Discussion}