Index: trunk/doc/release.2015/ps1.calibration/calibration.tex =================================================================== --- trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 40630) +++ trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 40632) @@ -18,4 +18,6 @@ \def\plotext{pdf} %\def\plotext{ps} + +%% NOTE: 2019 Feb versions of the figures are generated in /data/kukui.1/eugene/cal.paper.20190217 %\def\picdir{/home/eugene/chipresid.20140404} @@ -148,5 +150,7 @@ readout time of 7 seconds for a full unbinned image \citep{2008SPIE.7014E..0DO}. The active, usable pixels cover $\sim -80$\% of the FOV. +80$\% of the FOV. Figure~\ref{fig:gpc1.layout} illustrates the +physical layout of the devices in the camera with respect to the +parity of the sky. Nightly observations are conducted remotely from the Advanced @@ -240,4 +244,22 @@ %% submission and refereeing process.}} +%% this figure comes from kukui.1/~/czw.paper.images.20181130 +\begin{figure} + \centering + \includegraphics[width=0.9\hsize,angle=0,clip]{{pics/gpc1.layout}.pdf} + \caption{Diagram illustrating layout of OTA devices in GPC1. The + blue dots mark the locations of the amplifiers for xy00 cells in + each chip. When cells are mosaicked to a single pixel grid, the + pixel in this corner is at chip coordinate (0,0). The figure + illustrates the orientation of the OTA devices relative to the + parity of the sky. An exposure taken with North at the top of the + field-of-view will have East to the left when the OTA devices are + mosaicked as shown. Note that the devices OTA0Y - OTA3Y are + rotated by 180\degrees\ relative to the other half of the camera. + The labeling of the non-existent corner OTAs is provided to orient + the focal plane.} + \label{fig:gpc1.layout} +\end{figure} + \section{Pan-STARRS\,1 Data Analysis} @@ -255,17 +277,18 @@ this article. -The data processing steps are described in detail by \cite{waters2017} -and \cite{magnier2017.datasystem,magnier2017.analysis}. In summary, individual images -are detrended: non-linearity and bias corrections are applied, a dark -current model is subtracted and flat-field corrections are applied. -The \yps-band images are also corrected for fringing: a master fringe -pattern is scaled to match the observed fringing and subtracted. Mask -and variance image arrays are generated with the detrend analysis and -carried forward at each stage of the IPP processing. Source detection -and photometry are performed for each chip independently. As -discussed below, preliminary astrometric and photometric calibrations -are performed for all chips in a single exposure in a single analysis. -We refer to these measurements as the ``chip'' photometry and -astrometry products. +The pipeline data processing steps are described in detail by +\cite{waters2017} and +\cite{magnier2017.datasystem,magnier2017.analysis}. In summary, +individual images are detrended: non-linearity and bias corrections +are applied, a dark current model is subtracted and flat-field +corrections are applied. The \yps-band images are also corrected for +fringing: a master fringe pattern is scaled to match the observed +fringing and subtracted. Mask and variance image arrays are generated +with the detrend analysis and carried forward at each stage of the IPP +processing. Source detection and photometry are performed for each +chip independently. As discussed below, preliminary astrometric and +photometric calibrations are performed for all chips in a single +exposure in a single analysis. We refer to these measurements as the +``chip'' photometry and astrometry products. Chip images are geometrically transformed based on the astrometric @@ -321,8 +344,73 @@ the individual exposures and the stack images. -\section{Astrometric Models} - -% \note{include projection math?} -% \note{reference discussion somewhere on cell vs chip} +\section{Pipeline Calibration} + +\subsection{Overview} + +As images are processed by the data analysis system, every exposure is +calibrated individually with respect to a photometric and astrometric +reference database. The goal of this calibration step is to generate +a preliminary astrometric calibration, to be used by the warping +analysis to determine the geometric transformation of the pixels, and +a preliminary photometric transformation, to be used by the stacking +analysis to ensure the warps are combined using consistent flux units. + +The program used for the pipeline calibration, \ippprog{psastro}, +loads the measurements of the chip detections from their individual +output catalog files. It uses the header information populated at the +telescope to determine an initial astrometric calibration guess based +on the position of the telescope boresite right ascension, declination +and position angle as reported by the telescope \& camera subsystems. +Using the initial guess, \ippprog{psastro} loads astrometric and +photometric data from the reference database. + +\subsection{Reference Catalogs} +\label{sec:synthdb} + +During the course of the PS1SC Survey, several reference databases +have been used. For the first 20 months of the survey, +\ippprog{psastro} used a reference catalog with synthetic PS1 +\grizy\ photometry generated by the Pan-STARRS IPP team based on based +combined photometry from Tycho (B, V), USNO \citep[red, blue, + IR][]{2003AJ....125..984M}, and 2MASS +$J, H, K$ \citep{2006AJ....131.1163S}. The astrometry in the database was from 2MASS +\citep{2006AJ....131.1163S}. After 2012 May, a reference catalog +generated from internal re-calibration of the PV0 analysis of PS1 +photometry and astrometry was used for the reference catalog. + +Coordinates and calibrated magnitudes of stars from the reference +database are loaded by \code{pasastro}. A model for the positions of +the 60 chips in the focal plane is used to determine the expected +astrometry for each chip based on the boresite coordinates and +position angle reported by the header. Reference stars are selected +from the full field of view of the GPC1 camera, padded by an +additional 25\% to ensure a match can be determined even in the +presence of substantial errors in the boresite coordinates. It is +important to choose an appropriate set of reference stars: if too few +are selected, the chance of finding a match between the reference and +observed stars is diminished. In addition, since stars are loaded in +brightness order, a selection which is too small is likely to contain +only stars which are saturated in the GPC1 images. On the other hand, +if too many reference stars are chosen, there is a higher chance of a +false-positive match, especially as many of the reference stars may +not be detected in the GPC1 image. The selection of the reference +stars includes a limit on the brightest and faintest magnitudes of the +stars selected. + +The astrometric analysis is necessarily performed first; after the +astrometry is determined, an automatic byproduct is a reliable match +between reference and observed stars, allowing a comparison of the +magnitudes to determine the photometric calibration. + +%% The astrometric calibration is performed in two major stages: first, +%% the chips are fitted independently with independent models for each +%% chip. This fit is sufficient to ensure a reliable match between +%% reference stars and observed sources in the image. Next, the set of +%% chip calibrations are used to define the transformation between the +%% focal plane coordinate system and the tangent plane coordinate +%% system. The chip-to-focal plane transformations are then determined +%% under the single common focal plane to tangent plane transformation. + +\subsection{Astrometric Models} Three somewhat distinct astrometric models are employed within the IPP @@ -344,6 +432,4 @@ order}$, may be 1 to 3, under the restriction that sufficient stars are needed to constrain the order. - -% \note{describe a bit better: this is automatically selected based on the number of stars} A second form of astrometry model which yields somewhat higher @@ -420,74 +506,4 @@ %% \end{verbatim} -\section{Real-time Calibration} - -\subsection{Overview} - -As images are processed by the data analysis system, every exposure is -calibrated individually with respect to a photometric and astrometric -database. The goal of this calibration step is to generate a preliminary -astrometric calibration, to be used by the warping analysis to determine -the geometric transformation of the pixels, and preliminary -photometric transformation, to be used by the stacking analysis to -ensure the warps are combined using consistent flux units. - -The program used for the real-time calibration, \ippprog{psastro}, -loads the measurements of the chip detections from their individual -output catalog files. It uses the header information populated at the -telescope to determine an initial astrometric calibration guess based -on the position of the telescope boresite right ascension, declination -and position angle as reported by the telescope \& camera subsystems. -Using the initial guess, \ippprog{psastro} loads astrometric and -photometric data from the reference database. - -\subsection{Reference Catalogs} -\label{sec:synthdb} - -During the course of the PS1SC Survey, several reference databases -have been used. For the first 20 months of the survey, -\ippprog{psastro} used a reference catalog with synthetic PS1 -\grizy\ photometry generated by the Pan-STARRS IPP team based on based -combined photometry from Tycho (B, V), USNO \citep[red, blue, - IR][]{2003AJ....125..984M}, and 2MASS -$J, H, K$ \citep{2006AJ....131.1163S}. The astrometry in the database was from 2MASS -\citep{2006AJ....131.1163S}. After 2012 May, a reference catalog -generated from internal re-calibration of the PV0 analysis of PS1 -photometry and astrometry was used for the reference catalog. - -% \note{discuss history of the different refcats?} - -Coordinates and calibrated magnitudes of stars from the reference -database are loaded by \code{pasastro}. A model for the positions of -the 60 chips in the focal plane is used to determine the expected -astrometry for each chip based on the boresite coordinates and -position angle reported by the header. Reference stars are selected -from the full field of view of the GPC1 camera, padded by an -additional 25\% to ensure a match can be determined even in the -presence of substantial errors in the boresite coordinates. It is -important to choose an appropriate set of reference stars: if too few -are selected, the chance of finding a match between the reference and -observed stars is diminished. In addition, since stars are loaded in -brightness order, a selection which is too small is likely to contain -only stars which are saturated in the GPC1 images. On the other hand, -if too many reference stars are chosen, there is a higher chance of a -false-positive match, especially as many of the reference stars may -not be detected in the GPC1 image. The selection of the reference -stars includes a limit on the brightest and faintest magnitudes of the -stars selected. - -The astrometric analysis is necessarily performed first; after the -astrometry is determined, an automatic byproduct is a reliable match -between reference and observed stars, allowing a comparison of the -magnitudes to determine the photometric calibration. - -The astrometric calibration is performed in two major stages: first, -the chips are fitted independently with independent models for each -chip. This fit is sufficient to ensure a reliable match between -reference stars and observed sources in the image. Next, the set of -chip calibrations are used to define the transformation between the -focal plane coordinate system and the tangent plane coordinate -system. The chip-to-focal plane transformations are then determined -under the single common focal plane to tangent plane transformation. - \subsection{Cross-Correlation Search} @@ -587,9 +603,9 @@ %% \note{quality of the fits as a result of this stage}. -\subsection{Real-time Photometric Calibration} +\subsection{Pipeline Photometric Calibration} %% \note{define / describe the robust median} -After the astrometric calibration has finished, the photometric +After the astrometric calibration is determined, the photometric calibration is performed by \ippprog{psastro}. When the reference stars are loaded, the apparent magnitude in the filter of interest is @@ -598,20 +614,19 @@ points by comparison with the instrumental magnitudes. For the PV3 analysis, an outlier-rejecting median is used to measure the zero -point. For early versions of the real-time analysis, when the -reference catalog used synthetic magnitudes, it was necessary to -search for the blue edge of the distribution: the synthetic magnitude -poorly predicted the magnitudes of stars in the presence of -significant extinction or for the very red stars, making the blue edge -somewhat more reliable as a reference than the mean. Once the -calibration was based on a reference catalog generated from -\PSONE\ photometry, this methods was no longer needed. Note that we -do not fit for the airmass slope in this analysis. The nominal -airmass slope is used for each filter; any deviation from the nominal -value is effectively folded into the observed zero point. The zero -point may be measured separately for each chip or as a single value -for the entire exposure; the latter option was used for the PV3 -analysis. - -\subsection{Real-time outputs} +point. For early versions of the pipeline analysis, when the reference +catalog used synthetic magnitudes, it was necessary to search for the +blue edge of the distribution: the synthetic magnitude poorly +predicted the magnitudes of stars in the presence of significant +extinction or for the very red stars, making the blue edge somewhat +more reliable as a reference than the mean. Once the calibration was +based on a reference catalog generated from \PSONE\ photometry, this +methods was no longer needed. Note that we do not fit for the airmass +slope in this analysis. The nominal airmass slope is used for each +filter; any deviation from the nominal value is effectively folded +into the observed zero point. The zero point may be measured +separately for each chip or as a single value for the entire exposure; +the latter option was used for the PV3 analysis. + +\subsection{Outputs} The calibrations determined by \ippprog{psastro} are saved as part of @@ -646,5 +661,5 @@ the data from the exposure are loaded into the DVO database. -\section{PV3 DVO Master Database} +\section{Calibration Database} Data from the GPC1 chip images, the stack images, and the warp images @@ -712,23 +727,23 @@ \hline ID\_MEAS\_NOCAL & 0x00000001 & detection ignored for this analysis (photcode, time range) -- internal only \\ -ID\_MEAS\_POOR\_PHOTOM & 0x00000002 & detection is photometry outlier (not used PV3) \\ -ID\_MEAS\_SKIP\_PHOTOM & 0x00000004 & detection was ignored for photometry measurement (not used PV3) \\ -ID\_MEAS\_AREA & 0x00000008 & detection near image edge (not used PV3) \\ +ID\_MEAS\_POOR\_PHOTOM & 0x00000002 & detection is photometry outlier (not used for PV3) \\ +ID\_MEAS\_SKIP\_PHOTOM & 0x00000004 & detection was ignored for photometry measurement (not used for PV3) \\ +ID\_MEAS\_AREA & 0x00000008 & detection near image edge (not used for PV3) \\ ID\_MEAS\_POOR\_ASTROM & 0x00000010 & detection is astrometry outlier \\ -ID\_MEAS\_SKIP\_ASTROM & 0x00000020 & detection was ignored for astrometry measurement \\ +ID\_MEAS\_SKIP\_ASTROM & 0x00000020 & detection was not used for image calibration (not reported for PV3) \\ ID\_MEAS\_USED\_OBJ & 0x00000040 & detection was used during update objects \\ -ID\_MEAS\_USED\_CHIP & 0x00000080 & detection was used during update chips (not saved PV3) \\ -ID\_MEAS\_BLEND\_MEAS & 0x00000100 & detection is within radius of multiple objects \\ -ID\_MEAS\_BLEND\_OBJ & 0x00000200 & multiple detections within radius of object \\ +ID\_MEAS\_USED\_CHIP & 0x00000080 & detection was used during update chips (not saved for PV3) \\ +ID\_MEAS\_BLEND\_MEAS & 0x00000100 & detection is within radius of multiple objects (not used for PV3) \\ +ID\_MEAS\_BLEND\_OBJ & 0x00000200 & multiple detections within radius of object (not used for PV3) \\ ID\_MEAS\_WARP\_USED & 0x00000400 & measurement used to find mean warp photometry \\ ID\_MEAS\_UNMASKED\_ASTRO & 0x00000800 & measurement was not masked in final astrometry fit \\ -ID\_MEAS\_BLEND\_MEAS\_X & 0x00001000 & detection is within radius of multiple objects across catalogs \\ -ID\_MEAS\_ARTIFACT & 0x00002000 & detection is thought to be non-astronomical \\ -ID\_MEAS\_SYNTH\_MAG & 0x00004000 & magnitude is synthetic \\ +ID\_MEAS\_BLEND\_MEAS\_X & 0x00001000 & detection is within radius of multiple objects across catalogs (not used for PV3) \\ +ID\_MEAS\_ARTIFACT & 0x00002000 & detection is thought to be non-astronomical (not used for PV3) \\ +ID\_MEAS\_SYNTH\_MAG & 0x00004000 & magnitude is synthetic (not used for DR2) \\ ID\_MEAS\_PHOTOM\_UBERCAL & 0x00008000 & externally-supplied zero point from ubercal analysis \\ ID\_MEAS\_STACK\_PRIMARY & 0x00010000 & this stack measurement is in the primary skycell \\ ID\_MEAS\_STACK\_PHOT\_SRC & 0x00020000 & this measurement supplied the stack photometry \\ -ID\_MEAS\_ICRF\_QSO & 0x00040000 & this measurement is an ICRF reference position \\ -ID\_MEAS\_IMAGE\_EPOCH & 0x00080000 & this measurement is registered to the image epoch (not tied to ref catalog epoch) \\ +ID\_MEAS\_ICRF\_QSO & 0x00040000 & this measurement is an ICRF reference position (not used for PV3) \\ +ID\_MEAS\_IMAGE\_EPOCH & 0x00080000 & this measurement is registered to the image epoch (not used for PV3) \\ ID\_MEAS\_PHOTOM\_PSF & 0x00100000 & this measurement is used for the mean psf mag \\ ID\_MEAS\_PHOTOM\_APER & 0x00200000 & this measurement is used for the mean ap mag \\ @@ -754,12 +769,12 @@ {\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\ \hline -ID\_SECF\_STAR\_FEW & 0x00000001 & Used within relphot: skip star \\ -ID\_SECF\_STAR\_POOR & 0x00000002 & Used within relphot: skip star \\ -ID\_SECF\_USE\_SYNTH & 0x00000004 & Synthetic photometry used in average measurement \\ +ID\_SECF\_STAR\_FEW & 0x00000001 & Used within relphot: skip star (not reported for PV3) \\ +ID\_SECF\_STAR\_POOR & 0x00000002 & Used within relphot: skip star (not reported for PV3) \\ +ID\_SECF\_USE\_SYNTH & 0x00000004 & Synthetic photometry used in average measurement (not used in PV3) \\ ID\_SECF\_USE\_UBERCAL & 0x00000008 & Ubercal photometry used in average measurement \\ ID\_SECF\_HAS\_PS1 & 0x00000010 & PS1 photometry used in average measurement \\ ID\_SECF\_HAS\_PS1\_STACK & 0x00000020 & PS1 stack photometry exists \\ -ID\_SECF\_HAS\_TYCHO & 0x00000040 & Tycho photometry used for synth mags \\ -ID\_SECF\_FIX\_SYNTH & 0x00000080 & Synth mags repaired with zpt map \\ +ID\_SECF\_HAS\_TYCHO & 0x00000040 & Tycho photometry used for synth mags (not used in PV3) \\ +ID\_SECF\_FIX\_SYNTH & 0x00000080 & Synth mags repaired with zpt map (not used in PV3) \\ ID\_SECF\_RANK\_0 & 0x00000100 & Average magnitude uses rank 0 values \\ ID\_SECF\_RANK\_1 & 0x00000200 & Average magnitude uses rank 1 values \\ @@ -772,8 +787,8 @@ ID\_SECF\_STACK\_PRIMDET & 0x00010000 & PS1 stack primary measurement is a detection (not forced) \\ ID\_SECF\_STACK\_PRIMARY\_MULTIPLE & 0x00020000 & PS1 stack object has multiple primary measurements \\ -ID\_SECF\_HAS\_SDSS & 0x00100000 & This photcode has SDSS photometry \\ -ID\_SECF\_HAS\_HSC & 0x00200000 & This photcode has HSC photometry \\ -ID\_SECF\_HAS\_CFH & 0x00400000 & This photcode has CFH photometry (mostly Megacam) \\ -ID\_SECF\_HAS\_DES & 0x00800000 & This photcode has DES photometry \\ +ID\_SECF\_HAS\_SDSS & 0x00100000 & This photcode has SDSS photometry (not used for PV3) \\ +ID\_SECF\_HAS\_HSC & 0x00200000 & This photcode has HSC photometry (not used for PV3) \\ +ID\_SECF\_HAS\_CFH & 0x00400000 & This photcode has CFH photometry (not used for PV3) \\ +ID\_SECF\_HAS\_DES & 0x00800000 & This photcode has DES photometry (not used for PV3) \\ ID\_SECF\_OBJ\_EXT & 0x01000000 & Extended in this band \\ \hline @@ -791,7 +806,7 @@ {\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\ \hline -ID\_OBJ\_FEW & 0x00000001 & used within relphot: skip star \\ -ID\_OBJ\_POOR & 0x00000002 & used within relphot: skip star \\ -ID\_OBJ\_ICRF\_QSO & 0x00000004 & object IDed with known ICRF quasar (may have ICRF position measurement) \\ +ID\_OBJ\_FEW & 0x00000001 & used within relphot: skip star (not reported for PV3) \\ +ID\_OBJ\_POOR & 0x00000002 & used within relphot: skip star (not reported for PV3) \\ +ID\_OBJ\_ICRF\_QSO & 0x00000004 & object IDed with known ICRF quasar (not used for PV3) \\ ID\_OBJ\_HERN\_QSO\_P60 & 0x00000008 & identified as likely QSO \citep{2016ApJ...817...73H}, $P_{\rm QSO} \geq 0.60$ \\ ID\_OBJ\_HERN\_QSO\_P05 & 0x00000010 & identified as possible QSO \citep{2016ApJ...817...73H}, $P_{\rm QSO} \geq 0.05$ \\ @@ -799,8 +814,8 @@ ID\_OBJ\_HERN\_RRL\_P05 & 0x00000040 & identified as possible RR Lyra \citep{2016ApJ...817...73H}, $P_{\rm RRLyra} \geq 0.05$ \\ ID\_OBJ\_HERN\_VARIABLE & 0x00000080 & identified as a variable by \cite{2016ApJ...817...73H} \\ -ID\_OBJ\_TRANSIENT & 0x00000100 & identified as a non-periodic (stationary) transient \\ +ID\_OBJ\_TRANSIENT & 0x00000100 & identified as a non-periodic (stationary) transient (not used for PV3) \\ ID\_OBJ\_HAS\_SOLSYS\_DET & 0x00000200 & identified with a known solar-system object (asteroid or other) \\ ID\_OBJ\_MOST\_SOLSYS\_DET & 0x00000400 & most detections from a known solar-system object \\ -ID\_OBJ\_LARGE\_PM & 0x00000800 & star with large proper motion \\ +ID\_OBJ\_LARGE\_PM & 0x00000800 & star with large proper motion (not used for PV3) \\ ID\_OBJ\_RAW\_AVE & 0x00001000 & simple weighted average position was used (no IRLS fitting) \\ ID\_OBJ\_FIT\_AVE & 0x00002000 & average position was fitted \\ @@ -840,9 +855,9 @@ ID\_IMAGE\_NEW & 0x00000000 & no calibrations yet attempted \\ ID\_IMAGE\_PHOTOM\_NOCAL & 0x00000001 & user-set value used within relphot: ignore \\ -ID\_IMAGE\_PHOTOM\_POOR & 0x00000002 & relphot says image is bad (dMcal > limit) \\ +ID\_IMAGE\_PHOTOM\_POOR & 0x00000002 & relphot says image is bad (dMcal $>$ limit) \\ ID\_IMAGE\_PHOTOM\_SKIP & 0x00000004 & user-set value: assert that this image has bad photometry \\ ID\_IMAGE\_PHOTOM\_FEW & 0x00000008 & currently too few measurements for photometry \\ ID\_IMAGE\_ASTROM\_NOCAL & 0x00000010 & user-set value used within relastro: ignore \\ -ID\_IMAGE\_ASTROM\_POOR & 0x00000020 & relastro says image is bad (dR,dD > limit) \\ +ID\_IMAGE\_ASTROM\_POOR & 0x00000020 & relastro says image is bad (dR,dD $>$ limit) \\ ID\_IMAGE\_ASTROM\_FAIL & 0x00000040 & relastro fit diverged, fit not applied \\ ID\_IMAGE\_ASTROM\_SKIP & 0x00000080 & user-set value: assert that this image has bad astrometry \\ @@ -861,10 +876,8 @@ \subsection{Ubercal Analysis} -% \note{clean up and re-word the pieces below} - The photometric calibration of the DVO database starts with the ``ubercal'' analysis technique as described by \cite{2012ApJ...756..158S}. This analysis is performed by the group -at Harvard, loading data from the \code{smf} files into their instance +at Harvard, loading data from the raw detection files into their instance of the Large Scale Database \citep[LSD,][]{2011AAS...21743319J}, a system similar to DVO used to manage the detections and determine the @@ -894,6 +907,4 @@ additional flat-field seasons. -%% \note{something for PV4}. - By excluding non-photometric data and only fitting 2 parameters for each night, the Ubercal solution is robust and rigid. It is not @@ -907,8 +918,4 @@ millimags in (\grizy). As we discuss below, this conclusion is reinforced by our external comparison. - -%% \note{do I have a measurement -%% of the bright end stability in PV3? basically, what is the scatter -%% per star as a function of position in the camera and magnitude?} The overall zero point for each filter is not naturally determined by @@ -929,12 +936,7 @@ \cite{2012ApJ...756..158S}. -%% \note{The calspec spectrophotometry values have also been re-examined -%% by REF; using these new measurements, \cite{2015ApJ...815..117S} -%% determine new zero points for the PS1 system, which we have applied -%% (see below).} - % http://iopscience.iop.org/article/10.1088/0004-637X/815/2/117/pdf -\subsection{Applying the Ubercal Zero Points : Setphot} +\subsection{Apply Zero Points} The ubercal analysis above results in a table of zero points for all @@ -976,6 +978,6 @@ \hline \hline -{\bf Filter} & {\bf Zero Point} & {\bf Zero Point} & {\bf Airmass Slope} \\ - & {\bf (Raw)} & {\bf (Calspec)} & \\ +{\bf Filter} & {\bf Zero Point} & {\bf Zero Point} & {\bf Airmass} \\ + & {\bf (Raw)} & {\bf (Calspec)} & {\bf Slope} \\ \hline \gps & 24.563 & 24.583 & 0.147 \\ @@ -988,6 +990,4 @@ \end{center} \end{table} - -%% \note{give airmass formula for completeness?}. When \code{setphot} applies the ubercal information to the image @@ -1077,5 +1077,5 @@ the offsets converge to the milli-magnitude level within 8 iterations. -Only brighter, high quality measurements are used in the relative +Only high quality measurements are used in the relative photometry analysis of the exposure zero points. We use only the brighter objects, limiting the density to a maximum of 4000 objects @@ -1147,5 +1147,5 @@ The calculation of the relative photometry zero points is performed for the entire $3\pi$ data set in a single, highly parallelized -analysis. As discussed above, the measurement and object data in the +analysis. The measurement and object data in the DVO database are distributed across a large number of computers in the IPP cluster: for PV3, 100 parallel hosts are used. These machines by @@ -1164,13 +1164,32 @@ of responsibility. +%% plots made using scripts and data in +% /data/kukui.3/eugene/pv3.cam.20150607: +% photflat.20151127.fix/photflat.20151127.fix.0.*.fits +% based on extractions in: +% /data/ipp094.0/eugene/pv3.cam.20150607/astrom.corrections/ +% measurements are in photflat.extract.*.fits +% tdhistograms in photflat.20151127/ +% script: photflat.sh +% catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master +% measurement extraction was done ~ 2015.11.25-27 +% this is PV3.0 [pre-calibrations] + \begin{figure*}[htbp] \begin{center} \begin{minipage}{0.85\linewidth} - \includegraphics[width=\textwidth,clip]{{pics/photflat.example.sm}.png} + \includegraphics[width=\textwidth,clip]{{pics/photflat.example.v1}.png} \end{minipage} - \hspace{-2.75in} + \hspace{-3.0in} \begin{minipage}{0.4\linewidth} - \vspace{3.25in} - \caption{\label{fig:photflat} High-resolution flat-field correction images for the 5 filters $grizy$.} + \vspace{6.0in} + \caption{\label{fig:photflat} High-resolution flat-field correction + images for the 5 filters $grizy$. These images are shown in + standard camera orientation with OTA00 in the lower-left + corner and OTA07 in the upper-right corner. Fine + ``tree-ring'' structures are visible in several chips especially + in the bluer bands. The effect of the central ``tent'' on the + photometry, presumably due to the rapidly-varying PSF in this + region may also be seen. } \end{minipage} \end{center} @@ -1220,8 +1239,23 @@ analysis. +%% figure made using scripts and data in: +% /data/kukui.3/eugene/pv3.stats.20161202 +% scatter.sh : allsky.scatter.photom +% maps.measure/pv3.v1.dmag_*.sigma.fits +% cdhist.measure/cdmerge.v1.dmag_*.fits +% +%% cdhist.measure from: +% /data/ipp094.0/eugene/pv3.stats.20161202/ +% measures.sh : extract.allsky +% used catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master +% data was extracted 2016.12.11 : PV3.2 calibration + +%% the mean camera photometry was not modified after this date +%% These extractions should be used for the paper (EAM 2019.02.15) + \begin{figure*}[htbp] \begin{center} %width=\hsize - \includegraphics[height=\vsize,clip]{{pics/allsky.photom.sigma.sm}.png} + \includegraphics[height=\vsize,clip]{{pics/allsky.photom.v1}.png} \caption{\label{fig:allsky.photom.sigma} Consistency of photometry measurements across the sky. Each panel shows a map of the @@ -1233,6 +1267,4 @@ \end{center} \end{figure*} - -%% \note{need to discuss the process of setting the final mean magnitudes} \subsubsection{Photometric Flat-field} @@ -1291,11 +1323,7 @@ variable charge diffusion. -Other features include some poorly responding cells (e.g., in XY14) +Other features include some poorly responding cells (e.g., in OTA14) and effects at the edges of chips, possibly where the PSF model fails to follow the changes in the PSF. - -%% XXX : need to refer to system paper on the central tent? - -%% \note{show the flat-field residual images, discuss the features?}. For stacks and warps, the image calibrations were determined after the @@ -1323,5 +1351,5 @@ magnitudes, but the aperture-like magnitudes are tied by equating the stack Kron magnitudes to the average chip Kron magnitudes. {\em Note - that for DR1, this split zero point calibration was used; instead + that for DR1, this split zero point calibration was {\bf not} used; instead all stack photometry was tied to the average chip photometry via the PSF magnitudes.} The result of using a single zero point is that @@ -1332,4 +1360,6 @@ This split is not needed for the forced-warp photometry since the individual warps have well-defined PSfs. + +%% XXX generate a figure to illustrate the Kron vs PSF mags in stacks (DR1 & DR2) \subsection{Photometry Calibration Quality} @@ -1369,7 +1399,5 @@ 18)$ millimagnitudes. -%% \note{recommendation} - -\subsection{Calculation of Object Photometry} +\subsection{Object Photometry} Once the image photometric calibrations (zero points and flat-field @@ -1400,4 +1428,5 @@ \subsubsection{Selection of Measurements} +\label{sec:measurement.quality} To choose the measurements which will be used in the analysis, we @@ -1486,5 +1515,10 @@ raised identifying which rank was used. These bit are called \code{ID_SECF_RANK_0} through \code{ID_SECF_RANK_4} (see -Table~\ref{tab:secf_mask_values}). +Table~\ref{tab:secf_mask_values}). This assessment of the valid +measurements is performed independently for PSF, Kron, and +seeing-matched total aperture magnitudes. All measurements which are +retained to determine the average value are marked with bit-flags: \code{ID_MEAS_PHOTOM_PSF}, +\code{ID_MEAS_PHOTOM_KRON}, or \code{ID_MEAS_PHOTOM_APER} depending on +which average magnitude is being calculated. %% where do these go? analyis? @@ -1510,5 +1544,7 @@ underlying constant value. The discussion below applies to both the average of the chip photometry magnitudes and the forced-warp -photometry fluxes. +photometry fluxes. This technique is used to calculate the average +magnitudes for all three types of photometry stored in the DVO +database: PSF, Kron, and seeing-matched total aperture photometry. The IRLS analysis starts with an ordinary least squares fit, using the @@ -1550,6 +1586,4 @@ converge. -% \note{did this happen for any of our targets?} - To calculate a fit $\chi^2$ value and to determine an appropriate set of errors for the model parameters, it is necessary to transform the @@ -1558,5 +1592,8 @@ ($\omega^\prime < 0.3 <\omega>$) then the point is treated as clipped. The $\chi^2$ is determined from the {\em unclipped} points using the -standard Poisson errors. +standard Poisson errors. Data points which are so excluded are marked +with bit-flags: \code{ID_MEAS_MASKED_PSF}, +\code{ID_MEAS_MASKED_KRON}, or \code{ID_MEAS_MASKED_APER} depending on +which average magnitude is being calculated. Bootstrap-resampling analysis is used to assess the errors on the fit @@ -1575,4 +1612,19 @@ photometry. +One detail related to the above analysis concerns the measurements +from images which were included in the ubercal analysis. These images +were determined to have been taken in good quality (photometric) +weather, and have had their zero points determined with a robust +analysis. We therefore over-weight these data points to ensure the +average photometry is dominated by the ubercal values. In the IRLS +analysis above, the ubercal points are given 10 times the weight of +the non-ubercal points. This over-weighting is applied independently +of the calculation of the reweighting based on the deviation from the +model. Thus, the increased weight is {\em not} applied by reducing +the errorbars by a factor of 10 since that would increase the chance +that the ubercal measurements would be given reduced weight. If the +average photometry of an object in a filter includes ubercal +measurements, the per-filter bit flag \code{ID_SECF_USE_UBERCAL} is set. + % mask values for which wt < threshold (0.3 * median wt) % we record the min and max values of the unmasked / unclipped subset @@ -1580,9 +1632,6 @@ % bootstrap: use only unclipped subset and raw weights to estimate errors -% \note{bootstrap uses unclipped values and the raw weights? confirmed} - -% \note{reported error is max of bootstrap and formal error? confirmed} - \subsubsection{Stack Photometry} +\label{sec:stack.phot} For the stack photometry, the assessment is different from the chip @@ -1596,4 +1645,7 @@ detections of the same object. This situation is discussed in more detail below. + +% generate from : +% /data/kukui.1/eugene/czw.paper.images.20181130 (see .dvo) \begin{figure*}[htbp] @@ -1661,5 +1713,5 @@ Since the ``primary'' identification is purely based on the skycell geometry and the coordinate of the object, there is no guarantee that -any primary measurement is in fact a good or even the best measurement +any primary measurement is in fact the best or even a good measurement of the object. While the different overlapping pixels should be essentially identical, it is possible (due to some of the edge cases @@ -1687,5 +1739,63 @@ split should not be common (and in fact reflects a failure of the algorithm), but we have defined the rules to allows us to choose an -acceptable measurement even in these cases. +acceptable measurement even in these cases. Also note that the +``best'' measurement is not guarateed to be a good measurement. + +Stack measurements which are in the ``primary'' skycell have the bit +flag \code{ID_MEAS_STACK_PRIMARY}. The measurement which was +identified as the ``best'' measurement gets the bit flag +\code{ID_MEAS_STACK_PHOT_SRC}. If a ``primary'' measurement exists +for a given filter, then the per-filter bit flag +\code{ID_SECF_STACK_PRIMARY} is set for that filter. If multiple +primary stack measurements exist for a given filter, then the +per-filter bit flag \code{ID_SECF_STACK_PRIMARY_MULTIPLE} is also set +for that filter. +% +If the ``best'' measurement for a filter is a significant detection +(not forced from another band), then the per-filter bit flag +\code{ID_SECF_STACK_BESTDET} is set. +% +If any of the ``primary'' measurements for a filter is a significant +detection (not forced from another band), then the per-filter bit flag +\code{ID_SECF_STACK_PRIMDET} is set. +% +If any stack measurements exist for a given filter, then the +per-filter bit flag \code{ID_SECF_HAS_PS1_STACK} is set. + +The ``best'' stack measurements are examined across the filters. If +for all five filters, the ``best'' stack measurement is a ``primary'' +measurement, then the object bit flag \code{ID_OBJ_BEST_STACK} is set. +% +If the ``best'' stack measurement in a filter has signal-to-noise less +than 5, has any of the ``bad quality'' bits raised (see +Section~\ref{sec:measurement.quality}, rank 6), or has a \code{PSF_QF} +value less than 0.85 (or NAN) is considered to be ``bad''. +% +It it has any of the ``poor quality'' bits raised (see +Section~\ref{sec:measurement.quality}, rank 2), or has a +\code{PSF_QF_PERFECT} value less than 0.85 is considered to be +``suspect''. +% +Otherwise, the measurement is considered to be ``good''. For an +object detected in the stacks, if at least 2 of the filters have +``good'' stack measurements, then the object is considered to be +``good'', \ie, likely to be a valid astronomical object, and the +object bit flag \code{ID_OBJ_GOOD_STACK} is set. If no more than one +filter measurement is good, and there are at least two good or suspect +measurements, then the object is considered to be ``suspect'' and the +object bit flag \code{ID_OBJ_SUSPECT_STACK} is set. If at most a +single measurement is either good or suspect, then the object is +considered to be ``bad'' and the object bit flag +\code{ID_OBJ_BAD_STACK} is set. Note, however, that a high redshift +quasar which is well detected in the \yps-band but undetected in the +other bands would be labeled ``bad''; caution is required as always. + +In the public science database (PSPS) available through the MAST +interface includes two fields in the \ippdbtable{StackObjectThin} +table, \ippdbcolumn{primaryDetection} and \ippdbcolumn{bestDetection}. +These fields have an error in their definition and should not be used +for either DR1 or DR2. An update to the database will define fields +for each object which encapsulate the information about the ``primary'' +and ``best'' detections. \subsubsection{Warp Photometry} @@ -1706,4 +1816,8 @@ been selected, the same quality cuts are applied to the measurements as are applied to the chip measurements, as discussed above. +Forced-warp measurements actually used to calculate the average for a +filter are marked with the bit flag \code{ID_MEAS_WARP_USED}. + +% from: /data/kukui.3/eugene/pv3.stats.20161202/ \begin{figure*}[htbp] @@ -1711,5 +1825,5 @@ \includegraphics[width=\hsize,clip]{{pics/KHexample}.png} \caption{\label{fig:KHexample} Illustration of the Koppenh\"ofer Effect - on chip XY04. {\bf Bottom left} X-direction before correction. The solid line shows the measured + on OTA04. {\bf Bottom left} X-direction before correction. The solid line shows the measured mean residual for stars detected on this chip as a function of the instrumental magnitude / FWHM$^2$. @@ -1719,4 +1833,6 @@ \end{center} \end{figure*} + +% from: /data/kukui.3/eugene/pv3.stats.20161202/ \begin{figure}[htbp] @@ -1732,4 +1848,64 @@ \end{center} \end{figure} + +\subsubsection{Object Photometry Flags} + +Certain object-level bit flags are set based on the +\ippstage{chip}-stage measurements. If any object has at least one +PS1 measurement from rank 0 - 2 +(Section~\ref{sec:measurement.quality}), then the object is marked +with the bit flag \code{ID_OBJ_GOOD}. Each measurement is also +checked for consistency with a PSF or an extended source morphology: +if the difference between the PSF magnitude and the seeing-matched +full aperture magnitude is less than a specific cut-off (2.5$\sigma$ +added in quadrature to a floor of 0.1 magnitudes), then the +measurement is considered ``PSF-like''. Otherwise, the measurement is +counted as extended. If more of the PS1 measurements are extended +than PSF-like, the object bit flag \code{ID_OBJ_EXT} is raised. If +more than half of the PS1 \ippstage{chip}-stage measurements within a +single filter are extended, then the per-filter bit flag +\code{ID_SEC_OBJ_EXT} and \code{ID_SEC_OBJ_EXT_PSPS} are set. The +latter bit is a duplicate bit defined because the high bit in a 32-bit +integer is difficult to handle within the context of SQL server. Any +object which has any \ippstage{chip}-stage measurements for one of the +five filters has the per-filter bit flag \code{ID_SECF_HAS_PS1} set. + +In addition, if the object has measurements from the 2MASS point +source catalog, the quality of these measurements are check. If the +2MASS quality flag \code{ph_qual} has a value of A,B, or C, then the +object is considered to be a good 2MASS object and the bit flag +\code{ID_OBJ_GOOD_ALT} is set. If the 2MASS extended source flag, +\code{gal_contam}, has a value of 1 or 2 then the object bit flag +\code{ID_OBJ_EXT_ALT} is set. + +%% the flags below were in fact correctly set -- verified 2019.02.26 +%% for 3pi.pv3.20170919 (and logs say they were set 2016.04.12 in +%% /data/ipp094.0/eugene/hernitschek.20151125 + +We also set certain object-level bit flags based on additional +analysis of the Pan-STARRS data. \cite{2016ApJ...817...73H} used +measurements from the $3\pi$ survey to identify potentially +interesting variable sources. They examined the characteristics of +the varying fluxes in the 5 bands to distinguish two classes of +variable sources: RR Lyrae stars and QSOs. They present two +classifier statistics, $P_{\rm QSO}$ and $P_{\rm RRLyrae}$ which can +be used to select candidates with varying levels of quality and +completeness. Using this catalog, we have marked objects with a set +of bits to specify the possible varibility information as identified +by \cite{2016ApJ...817...73H}: +\begin{itemize} +\item \code{ID_OBJ_HERN_QSO_P60} : identified as likely QSO, $P_{\rm QSO} \geq 0.60$ +\item \code{ID_OBJ_HERN_QSO_P05} : identified as possible QSO, $P_{\rm QSO} \geq 0.05$ +\item \code{ID_OBJ_HERN_RRL_P60} : identified as likely RR Lyra, $P_{\rm RRLyra} \geq 0.60$ +\item \code{ID_OBJ_HERN_RRL_P05} : identified as possible RR Lyra, $P_{\rm RRLyra} \geq 0.05$ +\item \code{ID_OBJ_HERN_VARIABLE} : identified as a variable by \cite{2016ApJ...817...73H} +\end{itemize} +In addition, the Pan-STARRS MOPS team has identified solar-system +objects within the $3\pi$ dataset. We have used a list of 14.7M such +detections recorded by MOPS from the $3\pi$ survey. Any object which +contains one of these detections has the object bit flag +\code{ID_OBJ_HAS_SOLSYS_DET} set. If 50\% or more of the detections +for an object are solar-system objects, then the bit flag +\code{ID_OBJ_MOST_SOLSYS_DET} is set. \section{Astrometry Calibration} @@ -1794,6 +1970,4 @@ % ALL 322922 1163377 27.76 -% \note{was there is significant difference using a surface brightness version?} - We measured the Koppenh\"ofer Effect by accumulating the residual astrometry statistics for stars in the database. For each chip, we @@ -1827,5 +2001,5 @@ define a blue DCR color ($g-i$) to be used when correcting the filters \gps,\rps,\ips, and a red DCR color ($z - y$) to be used when -correcting the filters $zy$. In the process of performing the +correcting the filters \zps\ and \yps. In the process of performing the relative astrometry calibration, we record the median red and blue colors of the reference stars used to measure the astrometry @@ -1842,8 +2016,25 @@ the difference between the star color and the reference star color, using the red or blue color appropriate to the particular filter, times -the tangent of the zenith distance. Figure~\ref{fig:DCRexample} shows the -DCR trend for the 5 filters \grizy, as well as the measured -displacement in the direction perpendicular to the parallactic angle. -We represent the trend with a spline fitted to this dataset. +the tangent of the zenith distance: +\begin{eqnarray} +\delta_{\rm blue} = \alpha \left[(g - i)_{\rm ref} - (g - i)\right] \tan \zeta \\ +\delta_{\rm red} = \alpha \left[(z - y)_{\rm ref} - (z - y)\right] \tan \zeta +\end{eqnarray} +where $(g-i)_{\rm ref}$ and $(z-y)_{\rm ref}$ are the median colors of the +stars used the calibrate a specific blue- or red-filter image, +respecitively, while $\zeta$ is the zenith distance. +Figure~\ref{fig:DCRexample} shows the DCR trend for the 5 filters +\grizy, as well as the measured displacement in the direction +perpendicular to the parallactic angle. We represent the trend with a +spline fitted to this dataset. + +% figure from /data/kukui.3/eugene/dcr.20141205 +% based on /data/ipp064.0/eugene/dcr.20141205 +% script: dvo.dcr.sh +% catdir /data/stsci19.2/eugene/addstar.20141016/lap.pv2.subset.catdir +% XXX THIS IS A PV2 analysis! +% +% Generate new figure using: +% /data/ipp094.0/eugene/pv3.cam.20150607/astrom.corrections/dcr.meas.20151203.0.fits \begin{figure}[htbp] @@ -1856,17 +2047,39 @@ \end{figure} -The amplitude of the DCR trend in the five filters is $(g,r,i,z,y) = -(0.010, 0.001, -0.003, -0.017, -0.021)$ arcsec airmass$^{-1}$ -magnitude$^{-1}$. We saturate the DCR correction if the term $color -TAN (\zeta)$ for a given measurement is outside a range where the -DCR correction is well measured. The maximum DCR correction applied -to the five filters is $(g,r,i,z,y) = (0.019, 0.002, 0.003, 0.006, -0.008)$ arcseconds. - -%% \note{write down the DCR formalae for reference}. +The amplitude of the DCR trend, $\alpha$, in the five filters is +$(g,r,i,z,y) = (0.010, 0.001, -0.003, -0.017, -0.021)$ arcsec +airmass$^{-1}$ magnitude$^{-1}$. We saturate the DCR correction if +the term $\left[gi_{\rm ref} - (g - i)\right] \tan \zeta$ or +$\left[zy_{\rm ref} - (z - y)\right] \tan \zeta$ for a given +measurement is outside of the range where the DCR correction is +measured. The maximum DCR correction applied to the five filters is +$(g,r,i,z,y) = (0.019, 0.002, 0.003, 0.006, 0.008)$ arcseconds. + +%% plots made using scripts and data in +% /data/kukui.3/eugene/pv3.cam.20150607: +% astroflat.20151205/astroflat.20151205.v2.$dir.$filter.fits +% +% based on extractions in: +% /data/ipp094.0/eugene/pv3.cam.20150607/astrom.corrections/ +% measurements are in astroflat.0.fits - astroflat.3.fits +% +% script: dvo.astroflat.sh +% catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master +% measurement extraction was done 2015.12.04 +% this is PV3.0 [pre-calibrations] +% +% NOTE: the extraction generated 4 meas tables, but the flat-field +% was built with only 1 (the .0.fits version) +% +% 2017.02.17 : I generated a new set of flats based on all 4 extractions +% this is in /data/ipp105.0/eugene/astrom.20170225/astroflat.20170217/ +% and was applied to the database 2017.02.25 (../run.setastrom) +% +% generate new astrometric flat-field images based on e.g.: +% /data/ipp105.0/eugene/astrom.20170225/astroflat.20170217/astroflat.20170217.med.cam.dX.g.fits \begin{figure*}[htbp] \begin{center} - \includegraphics[width=0.85\textwidth,clip]{{pics/astroflat.gri.sm}.png} + \includegraphics[width=0.85\textwidth,clip]{{pics/astroflat.gri.v1}.png} \caption{\label{fig:astroflat.gri} High-resolution astrometric flat-field correction images for $gri$.} \end{center} @@ -1875,5 +2088,5 @@ \begin{figure*}[htbp] \begin{center} - \includegraphics[width=0.85\textwidth,clip]{{pics/astroflat.zy.sm}.png} + \includegraphics[width=0.85\textwidth,clip]{{pics/astroflat.zy.v1}.png} \caption{\label{fig:astroflat.zy} High-resolution astrometric flat-field correction images for $zy$.} \end{center} @@ -1881,12 +2094,12 @@ \subsubsection{Astrometric Flat-field} +\label{sec:astro.flat} After correction for both KE and DCR, we observe persistent residual astrometric deviations which depend on the position in the camera. We construct an astrometric ``flat-field'' response by determining the -mean residual displacement in the X and Y (chip) directions as a +mean residual displacement in the $X$ and $Y$ (chip) directions as a function of position in the focal plane. We have measured the -astrometric flat using a sampling resolution of 40x40 pixels, matching -the photometric flat-field correction images. +astrometric flat using a sampling resolution of $80 \times 80$ pixels. Figures~\ref{fig:astroflat.gri} and \ref{fig:astroflat.zy} show the astrometric flat-field images for the five filters \grizy\ in each of @@ -1900,24 +2113,27 @@ The dominant pattern in the astrometric residual is roughly a series of concentric rings. The pattern is similar to the pattern of the -focal surface residuals measured by \cite{2008SPIE.7014E..0DO}, which also has -a concentric series of rings with similar spacing. The ``tent'' in -the center of the focal surface is reflected in these astrometry -residual plots. Our interpretation of the structure is that the -deviations of the focal plane from the ideal focal surface introduces -small-scale PSF changes, presumably coupled to the optical +focal surface residuals measured by \cite{2008SPIE.7014E..0DO}, which +also has a concentric series of rings with similar spacing. The +``tent'' in the center of the focal surface is reflected in these +astrometry residual plots. Our interpretation of the structure is +that the deviations of the focal plane from the ideal focal surface +introduces small-scale PSF changes, presumably coupled to the optical aberrations, which result in small changes in the centroid of the object relative to the PSF model at that location. Since the PSF -model shape parameters are only able to vary at the level of a 6x6 -grid per chips, the finer structures are not included in the PSF -model. The PV2 analysis shows the ring structure more clearly, with a -pattern much more closely following the focal surface deviations. In -the PV2 analysis, the PSF model used at most a 3x3 grid per chip to -follow the shape variations, so any changes caused by the optical -aberrations would be less well modeled in the PV2 analysis, as we -observe. +model shape parameters are only able to vary at the level of a $6 +\times 6$ grid per chips, the finer structures are not included in the +PSF model. + +The PV2 analysis shows this circular pattern more clearly than the PV3 +analysis, with a pattern much more closely following the focal surface +deviations. In the PV2 analysis, the PSF model used at most a +$3\times 3$ grid per chip to follow the shape variations, so any +changes caused by the optical aberrations would be less well modeled +in the PV2 analysis than the PV3 analysis. For PV3, some of these +patterns are suppressed by the higher-resolution PSF model. A second pattern which is weakly seen in several chips consists of -consistent displacements in the X (serial) direction for certain -cells. This effect can be seen most clearly in chips XY45 and XY46. +consistent displacements in the $X$ (serial) direction for certain +cells. This effect can be seen most clearly in chips OTA45 and OTA46. In the PV2 analysis, this pattern is also more clearly seen. In this case, the fact that the astrometric model used polynomials with a @@ -1929,5 +2145,7 @@ of this is unclear, but likely caused by the astrometry model failing to follow the underlying variations because of the need to extrapolate -to the edge pixels. Finally, we also mention an interesting effect +to the edge pixels. + +Finally, we also mention an interesting effect {\em not} visible at the resolution of these astrometric flat-field images. Fine structures are observed at the \approx 10 pixel scale @@ -1949,68 +2167,146 @@ average solution, resulting in residual astrometric structures. The gradient of the astrometric displacement results in an apparent -expansion or compression of the pixel sizes, resulting in a signal +expansion or compression of the pixel sizes, generating a signal which can be observed in the flat-field images. For GPC1, unlike the DES detectors, the amplitude of these flat-field variations are much smaller than the photometric variations caused by the changing PSF -sized, caused in turn by varying electron diffusion rates. These +sizez, caused in turn by varying electron diffusion rates. These features, and the related vertical electron diffusion variations are discussed in detail in \cite{2018PASP..130f5002M}. -Unfortunately, we discovered a problem with the astrometric flat-field -correction too late to be repaired for DR1. As can be seen by -inspection of Figures~\ref{fig:astroflat.gri} and -\ref{fig:astroflat.zy}, there is significant pixel-to-pixel noise in -the the astrometric flat-field images. This pixel-to-pixel noise is -caused by too few stars used in the measurement of the flat-field -structure for the high-resolution sampling. As a result, the -astrometric flat-field correction reduces systematic structures on -large spatial scales, but at the expense of degrading the quality of -an individual measurement. Only $i$-band has sufficient -signal-to-noise per pixel to avoid significantly increasing the -per-measurement position errors. +% generate (or plot) astrometric flat-field images for DR2 (PV3.X) + +\begin{figure*}[htbp] + \begin{center} + \includegraphics[width=\hsize,clip]{{pics/astroflat.repair}.png} + \caption{\label{fig:astroflat.repair} Comparison of the + high-resolution astrometric flat-field images used for PV3.2 + (left) and for PV3.3 (right). These examples show the \gps-band + astrometric flat-field corrections for the $X$ direction as seen + in the focal plane coordinate system. Note the elevated noise in + the PV3.2 image due to insufficient numbers of stars used in the analysis. +} +\end{center} +\end{figure*} + +% numbers of stars used: +%% mana: load.stars astroflat.20151205/astroflat.20151205.v1.Npt.fits +%% filter g : 2591421 stars +%% filter r : 3497036 stars +%% filter i : 16241986 stars +%% filter z : 7153595 stars +%% filter y : 4509749 stars +%% mana: load.stars astroflat.20170217/astroflat.20170217.Npt.fits +%% filter g : 17590560 stars +%% filter r : 31000135 stars +%% filter i : 82648850 stars +%% filter z : 62166619 stars +%% filter y : 42867074 stars + +\note{move the discussion of the DR1 & DR2 scatter to the end of the + astrom section?} Figure~\ref{fig:allsky.astrom.sigma} shows the standard deviations of the mean residual astrometry in $(\alpha,\delta)$ for bright stars as -a function of position across the sky. For each pixel in these -images, we selected all objects with $15 < i < 17$, with at least 3 -measurements in $i$-band (to reject artifacts detected in a pair of -exposures from the same night), with \code{PSF_QF} $> 0.85$ (to reject -excessively-masked objects), and with $mag_{\rm PSF} - mag_{\rm Kron} -< 0.1$ (to reject galaxies). We then generated histograms of the -difference between the object position predicted for the epoch of each -measurement (based on the proper motion and parallax fit) and the -observed position of that measurement, in both the Right Ascension and -Declination directions (in linear arcseconds), for all stars in a -given pixel in the images. From these residual histograms, we can -then determine the median and the 68\%-ile range to calculate a robust -version of the standard deviation. This represents the bright-end -systematic error floor for a measurement from a single exposure. The -standard deviations are then plotted in +a function of position across the sky based on the DR2 calibration. For each +pixel in these images, we selected all objects with $15 < i < 17$, +with at least 3 measurements in $i$-band (to reject artifacts detected +in a pair of exposures from the same night), with \code{PSF_QF} $> +0.85$ (to reject excessively-masked objects), and with $mag_{\rm PSF} +- mag_{\rm Kron} < 0.1$ (to reject galaxies). We then generated +histograms of the difference between the object position predicted for +the epoch of each measurement (based on the proper motion and parallax +fit) and the observed position of that measurement, in both the Right +Ascension and Declination directions (in linear arcseconds), for all +stars in a given pixel in the images. From these residual histograms, +we can then determine the median and the 68\%-ile range to calculate a +robust version of the standard deviation. This represents the +bright-end systematic error floor for a measurement from a single +exposure. The standard deviations are then plotted in Figure~\ref{fig:allsky.photom.sigma}. The median value of the -standard deviations across the sky is $(\sigma_\alpha, \sigma_\delta) -= (22, 23)$ milliarcseconds. +standard deviations across the sky in both $(\sigma_\alpha, +\sigma_\delta)$ is 16 milliarcseconds. The Galactic plane is clearly apparently in these images. Like photometry, we attribute this to failure of the PSF fitting due to crowding. The celestial North pole regions have somewhat elevated -errors in both R.A. and DEC. This may be due to the larger typical -seeing at these high airmass regions, but without further exploration -this interpretation is uncertain. Several features can be seen which -appear to be an effect of the tie to the Gaia astrometry: the stripes -near the center of the DEC image and the right side of the R.A. image. -The mesh of circular outlines is due to the outer edge of the focal -plane where the astrometric calibration is poorly determined. As -discussed above, the median values in the images are higher than -expected based on our PV2 analysis of the astrometry: the median -per-measurement error floor of \approx 22 mas is significantly worse -than the \approx 17 mas value in that earlier analysis. We attribute -this degradation to the noise introduced by the astrometric -flat-field. This noise has been addressed for the DR2 release -of the individual measurement data. - -\begin{figure}[htbp] +errors in both R.A. and DEC, with some specifc structures. Some of +these structures may be due to the larger typical seeing at these high +airmass regions, but some are due to astrometric failures which stem +from the reference catalog based on the PV2 analysis (see +Section~\ref{sec:pole.problems} for further details). Several +features can be seen which appear to be an effect of the tie to the +Gaia astrometry: the stripes near the center of the DEC image and the +right side of the R.A. image. The mesh of circular outlines one the 2 +degree scale is due to the outer edge of the focal plane where the +astrometric calibration is poorly determined. + +The DR1 astrometric calibration suffered from degraded astrometry due +to a problem with the astrometric flat-field correction identified too +late to be repaired for DR1. +% +The astrometric flat-field images used +for that release had too few stars to measure the correction with +sufficient signal-to-noise. As a result, those corrections had +significant pixel-to-pixel noise which can be seen in +Figure~\ref{fig:astroflat.repair}. As a result, the astrometric +flat-field correction reduces systematic structures on large spatial +scales, but at the expense of degrading the quality of individual +measurements. Only the $i$-band flat had sufficient signal-to-noise +per pixel to avoid significantly increasing the per-measurement +position errors. + +For DR2, we recalculated the astrometric flat-field correction using +many more stars. For the DR1 release, the number of stars per filter +was (\grizy) = (2.6M, 3.5M, 16M, 7M, 4.5M), while for the DR2 release, +the number of stars per filter was (\grizy) = (18M, 31M, 83M, 62M, +43M). We also reduced the resolution of the astrometric flat-field, +using $80 \times 80$ superpixels, rather than the $40 \times 40$ +superpixels used for DR1. Because of the degraded astrometric +flat-field correction, the median per-measurement error floor of DR1 +is \approx 22 mas, significantly worse than both DR2 and the earlier +PV2 analysis. Figure~\ref{fig:allsky.astro.histogram} shows +histograms of the astrometric residual scatter across the sky for DR1 +and DR2, illustrating the improvement. + +\begin{figure*}[htbp] \begin{center} - \includegraphics[width=\hsize,clip]{{pics/allsky.astrom.sigma}.png} - \caption{\label{fig:allsky.astrom.sigma} Consistency of photometry + \includegraphics[width=\hsize,clip]{{pics/allsky.histogram.astrom.compare}.png} + \caption{\label{fig:allsky.astro.histogram} Illustration of the + impact of the astrometric flat-field correction used for PV3.2 vs + PV3.3. The blue histograms show the distribution of astrometric + residuals for bright stars from the PV3.2 analysis while the red + histograms show the distribution for the PV3.3 analysis. The + median standard deviation for PV3.2 is 22 milliarcseconds in R.A. + (23 mas in Declination). Using the higher signal-to-noise + flat-field correction images reduces the median values to 16 mas + for both R.A. and Declination directions in PV3.3. +} +\end{center} +\end{figure*} + +% older version of this figure: +% pv2_0 : /data/ipp060.0/eugene/pv2.astrom.20150126/astromap.20150127/dDsig.im.fits +% pv2_1 : /data/ipp060.0/eugene/pv2.astrom.20150126/astromap.20150429/dDsig.im.fits + +% NOTE: +% the pv2 versions used: resize 1800 920; region 0 0 85 ait +% the pv3 versions used: resize 1800 950; region 180 0 90 ait + +% thus we cannot directly compare map pixels, without re-extracting the measurements +% (we can do that if we decide it is needed to generate the best plots) + +% original version of figure: pv3.stats.20161202/allsky.astrom.sigma.png +% based on /data/kukui.3/eugene/pv3.stats.20161202/maps.measure/pv3.v1.*.sigma.fits +% based on /data/ipp094.0/eugene/pv3.stats.20161202/cdhist.measure/cdmerge.v1.dD.fits (& dR) +% plot script /data/kukui.3/eugene/pv3.stats.20161202/scatter.sh +% catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master (PV3.2) + +% regenerate using fits image in pv3.stats.20170413 + +\begin{figure*}[htbp] + \begin{center} + \includegraphics[width=\hsize,clip]{{pics/allsky.astrom.pv3.3}.png} + \caption{\label{fig:allsky.astrom.sigma} Consistency of astrometry measurements across the sky. Each panel shows a map of the standard deviation of astrometry residuals for stars in each @@ -2021,27 +2317,26 @@ is likely responsible for these elevated value. } \end{center} -\end{figure} - -% plot of the astrometric error floor per filter? - -% \note{SECTION or REF?}. +\end{figure*} After the initial analysis to measure the KE corrections, DCR corrections, and astrometric flat-field corrections, we applied these corrections to the entire database. Within the schema of the -database, each measurement has the raw chip coordinates -(\code{Measure.Xccd,Yccd}) as well as the offset for that object based on each of -these three corrections: \code{Measure.XoffKH,YoffKH, - Measure.XoffDCR,YoffDCR, Measure.XoffCAM,YoffCAM}. The offsets are -calculated for each measurement based on the observed instrumental -chip magnitudes and FWHM for the Koppenh\"ofer Effect, on the average -chip colors and the altitude \& azimuth of each measurement for the -DCR correction, and on the chip coordinates for the astrometric -flat-field corrections. The corrections are combined and applied to -the raw chip coordinates and saved back in the database in the fields -\code{Measure.Xfix,Yfix}. At this point, we are ready to run the -full astrometric calibration. - -\subsection{Galactic Rotation and Solar Motion} +database, each measurement in the \ippdbtable{Measure} table has the +raw chip coordinates (\ippdbcolumn{Xccd}, \ippdbcolumn{Yccd}) as well +as the offset for that object based on each of the three corrections +discussed above (\ippdbcolumn{XoffKH}, \ippdbcolumn{YoffKH}; +\ippdbcolumn{XoffDCR}, \ippdbcolumn{YoffDCR}; \ippdbcolumn{XoffCAM}, +\ippdbcolumn{YoffCAM}). The offsets are calculated for each +measurement based on the observed instrumental chip magnitudes and +FWHM for the Koppenh\"ofer Effect, on the average chip colors and the +altitude \& azimuth of each measurement for the DCR correction, and on +the chip coordinates for the astrometric flat-field corrections. The +corrections are combined and applied to the raw chip coordinates and +saved back in the database in the fields \ippdbcolumn{Xfix}, +\ippdbcolumn{Yfix}. At this point, we are ready to run the full +astrometric calibration. + +\subsection{Absolute Calibration} +\label{sec:galactic.rotation} The initial analysis of the PV2 astrometry used the 2MASS positions as @@ -2121,5 +2416,4 @@ where $d$ is the distance and $l,b$ are the Galactic coordinates of the star. Note that the proper motion induced by -%% \note{some reference for this?} the Galactic rotation is independent of distance while the reflex motion induced by the solar motion decreases with increasing @@ -2135,7 +2429,8 @@ value of 500pc. -%% \note{plots to show how well this worked for PV3 pre Gaia} - \subsection{Gaia Constraint} + +\note{move comparisons to Gaia to the discussion, limit this section + to the Gaia astrometric tie} After the full relative astrometry analysis was performed for the PV3 @@ -2164,6 +2459,4 @@ even at a lower weight, helps to tile over those gaps. -%% \note{Figures showing the Gaia residuals} - \begin{figure*}[htbp] \begin{center} @@ -2259,23 +2552,31 @@ proper motions will obviate the need to correct for the Galactic rotation. -\subsection{Calculation of Object Astrometry} +\subsection{Object Astrometry} + +After the image astrometric parameters have been determined and +applied to the measurements from each image, we attempt to find the +best astrometric parameters (position, parallax and proper motions) +for all objects in the database. Only good quality measurements are +kept for the astrometric analysis: PS1 chip detections with +\code{PSF_QF} $< 0.85$ are rejected, as are any detections for which +the magnitude or magnitude error were reported as \code{NAN}. Only +PS1 \ippstage{chip}-stage measurements were used for the astrometry +measurement (no stack or forced-warp measurements). If available, the +2MASS and Gaia astrometry for an object was also used in the +calculation of the astrometry. Measurements which were kept for the +astrometric fit for an object were marked with the bit-flags +\code{ID_MEAS_USED_OBJ}. Some detections were identified as extreme +outliers if their position deviated from the mean object coordinate by +more than 2 arcseconds. These detections were ignored and marked with +the bit flag \code{ID_MEAS_POOR_ASTROM}. + +If 2MASS or Gaia astrometry measurements +were available for an object, {\em all} measurements for that object +are marked with the bit-flag \code{ID_MEAS_OBJECT_HAS_2MASS} or +\code{ID_MEAS_OBJECT_HAS_GaIA} as appropriate. The Tycho 2.0 +measurements were not included in this analysis and objects with Tycho +measurements are therefore not marked. \subsubsection{Iteratively Reweighted Least Squares Fitting} - -After the image astrometric parameters have been determined and -applied to the measurements from each image, we attempt to find -the best astrometric parameters (position, parallax and proper -motions) for all objects in the database. We require a minimum of 5 -detections and 1 year of data for any object in order for it to be -fitted for just proper motion. For a parallax and proper-motion fit, -we require at least 7 detections, 1 year of data, and a parallax -factor range of at least 0.25; no object is fitted to parallax without -proper motion as well. If an object is fitted for parallax, it is -also fitted with a model including only proper motion and only a mean -position. The chisq for all three fits is saved. Currently, the -highest order fit allowed is saved in the database, regardless of the -significance of the improvement in adding parameters. The resulting -parallax and proper motion measurements are inserted back into the DVO -database for use by science queries. With an automatic process applied to hundreds of millions of stars, it @@ -2339,9 +2640,6 @@ fractional change is less than some tolerance ($10^{-4}$), then iterations are halted and the last fitted parameters are used. If -convergence is not reached in 10 iterations, the process is halted in -any case and a flag raised for the object to note that IRLS did not -converge. - -% \note{did this happen for any of our targets?} +convergence is not reached in 10 iterations, the process is halted and +the analysis is rejected. To calculate a fit $\chi^2$ value and to determine an appropriate set @@ -2354,5 +2652,8 @@ either used to calculate both RA and Declination terms, or neither). The $\chi^2$ is determined from the unclipped points in the standard -way. Bootstrap analysis is used to assess the errors on the fit +way. These measurements are marked with the bit flag +\code{ID_MEAS_UNMASKED_ASTRO}. + +Bootstrap-resampling analysis is used to assess the errors on the fit parameters: A number of measurements equal to the number of unclipped data points are randomly selected from the set of unclipped data @@ -2360,9 +2661,130 @@ then used to fit for the astrometric parameters, using ordinary least squares fitting. The parameters are recorded and the process re-run -100 times. For each astrometric parameter, the error is determined as +300 times. For each astrometric parameter, the error is determined as half of the 68\% confidence range for the distribution of fitted parameter values. +\subsubsection{Object Astrometry Flags} + +We require a minimum of 5 detections and 1 year of data for any object +in order for it to be fitted for just proper motion. For a parallax +and proper-motion fit, we require at least 7 detections, 1 year of +data, and a parallax factor range of at least 0.25; no object is +fitted to parallax without proper motion as well. If an object is +fitted for parallax, it is also fitted with a model including only +proper motion and only a mean position. The chisq for all three fits +is saved. Currently, the highest order fit allowed is saved in the +database, regardless of the significance of the improvement in adding +parameters. The resulting parallax and proper motion measurements are +inserted back into the DVO database for use by science queries. If +one of the three types of fits were attempted, the corresponding bit +flags are set: \code{ID_OBJ_FIT_PAR} for the full parallax fit, +\code{ID_OBJ_FIT_PM} for the proper-motion fit, \code{ID_OBJ_FIT_AVE} +for the mean position. The fit which was used to provide the reported +astrometric parameters is noted with one of the three object bit +flags: \code{ID_OBJ_USE_PAR}, \code{ID_OBJ_USE_PM}, +\code{ID_OBJ_USE_AVE}. If the IRLS analysis for all three types of +fits fails to converge, the raw weighted average position is reported +and the bit flag \code{ID_OBJ_RAW_AVE} is set. If the proper-motion +model was attempted and failed, the bit flag \code{ID_OBJ_BAD_PM} is +set. + +Objects for which there is no valid chip-stage measurement (\eg., +faint sources below the single-exposure detection limit) will use the +position from the stack for the mean position. In this case, the bit +flag \code{ID_OBJ_STACK_FOR_MEAN} will be raised. Stack astrometry is +reported to the PSPS database. The stack astrometry is calculated +based on the median of stack measurements. The stack measurements are +not statistically independent (see Section~\ref{sec:stack.phot}), so +there an average of the stack measurements does not improve the +statistical significance of the position measurement. In addition, +the stack astrometry is expected to be degraded relative to the +chip-stage astrometry, in part because of the geometric re-warping +required to generate the stack images and in part because of the +spatially variable stack PSFs. If stack measurements exist but for +some reason cannot be used for astrometry (\eg., poor quality) the +values reported to the PSPS database will be derived from the average +of the chip detections and the bit flag \code{ID_OBJ_MEAN_FOR_STACK} +will be set for the object. + \section{Discussion} +\label{sec:discussion} + +The calibration of the PV3 DVO database required several iterations. +For completeness, we discuss these steps and their implications for +the DR1 and DR2 releases. +\begin{itemize} + +\item[PV3.0] The first calibrated PV3 database is identified as PV3.0. + This calibration predates the Gaia DR1 release and uses the 2MASS + catalog as a reference. After internal testing, an error in the + photometry calibration was identified in this DVO version: the + high-resolution photometric flat-field correction measured using the + stellar photometry (see Section~\ref{sec:phot.flat}) was applied + with the wrong sign to the measurements. + +\item[PV3.1] After the above error was identified, the photometric + flat-field correction was applied in the correct sense to the + measurements and the average photometry was recalculated. The + resulting PV3.1 version of the database was used for the DR1 release + (but see below regarding the mean positions). + +\item[PV3.2] The Gaia DR1 release motivated a recalibration of the + astrometry using the Gaia DR1 position information, combined with + photometric distance estimates and a model for the Galactic and + Solar motion to correct the absolute proper motion (see + Section~\ref{sec:galactic.rotation}). We identify the resulting + database as PV3.1. This database was used to generate the positions + in the \ippdbtable{gaiaObject} table, which are exposed in the DR1 + release. + +\item[PV3.3] After the DR1 release, we identified a problem with the + astrometric flat-field corrections (see + Section~\ref{sec:astro.flat}): for all but the \ips\ filter, the + analysis of the flat-field used too few stars. The measurement of + the systematic astrometric corrections therefore had a low + signal-to-noise. Instead of reducing the scatter in the astrometric + measurements, the application of these flat-fields {\em increased} + the scatter. Recognizing this error, we re-measured the astrometric + flat-fields with a larger number of stars and applied the improve + versions to the database. The resulting PV3.3 calibration has a + noticable improvement in the astrometric scatter for bright stars. + +\item[PV3.4] Two errors were identified in the PV3.3 calibration + before the DR2 release was completed. First, we discovered that the + repair applied to the photometric flat-field correction for PV3.1, + reversing the sign of the correction, was not propagated to the + stack or warp photometry calibrations. Although the measurements + from these stages are not corrected by those flat-fields, they are + affected by this calibration since they are tied to the average of + the chip-stage measurements. Second, we determined that the + aperture-like photometry (e.g., Kron magnitudes) and photomety + which depends on the PSF model for the stack measurements need to be + independently tied to the average exposure photometry (see + discussion in Section~\ref{sec:phot.flat}). We addressed both of + these issue in the PV3.4 calibration of the DVO database. This + database was then used to generate the values in the DR2 PSPS + database tables. \note{what about P2, those were done first, right?} +\end{itemize} + +\begin{figure*}[htbp] + \begin{center} + \includegraphics[width=\hsize,clip]{{pics/photom.pv3.3v4}.png} + \caption{\label{fig:photom.pv3.3v4} Sample comparison of PV3.3 and + PV3.4 photometry illustrating the impact of the issues identified + in the PV3.3 stack and warp photometry. All figures use \ips-band + photometry. The left panels use data from PV3.3 while the right + use PV3.4. The top row shows the mean difference between the + average photometry from individual exposures (``chip'') and the + stack photometry using Kron magnitudes. The middle row shows the + mean difference between the average photometry from individual + exposures (``chip'') and the average forced-warp photometry, again + using Kron magnitudes. The bottom row shows the mean difference + between the average photometry from individual exposures + (``chip'') and the average forced-warp photometry, using PSF + magnitudes. See Section~\ref{sec:discussion} for a description of + the calibration change in PV3.4.} +\end{center} +\end{figure*} \section{Conclusion} @@ -2386,4 +2808,9 @@ Lorand University (ELTE) and the Los Alamos National Laboratory. +\note{colormaps by Peter Kovesi. Good Colour Maps: How to Design Them. +arXiv:1509.03700 [cs.GR] 2015. add ref} + + + \bibliographystyle{apj} % \bibliography{lib}{} @@ -2416,5 +2843,5 @@ * kh.data.20151203.v1/spline.final.fits : spline fits to the KH data * kh.data.20151203.v1.fits : densify images of residuals per chip : (dX,dY) & (T0, T1) = (pre fix, post fix) - * mana.sh : kh.example - plot of XY04 + * mana.sh : kh.example - plot of OTA04 * mana.sh : khmap (needs cleanup) * ipp094:/data/ipp094.0/eugene/pv3.cam.20150607/astrom.corrections : extractions and original scripts to make spline, etc