IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Ignore:
Timestamp:
Mar 4, 2019, 3:17:12 PM (7 years ago)
Author:
eugene
Message:

some re-org of the sections to smooth the flow; updates to some figures; added gpc1 layout figure

File:
1 edited

Legend:

Unmodified
Added
Removed
  • trunk/doc/release.2015/ps1.calibration/calibration.tex

    r40614 r40632  
    1818\def\plotext{pdf}
    1919%\def\plotext{ps}
     20
     21%% NOTE: 2019 Feb versions of the figures are generated in /data/kukui.1/eugene/cal.paper.20190217
    2022
    2123%\def\picdir{/home/eugene/chipresid.20140404}
     
    148150readout time of 7 seconds for a full unbinned image
    149151\citep{2008SPIE.7014E..0DO}.  The active, usable pixels cover $\sim
    150 80$\% of the FOV.
     15280$\% of the FOV.  Figure~\ref{fig:gpc1.layout} illustrates the
     153physical layout of the devices in the camera with respect to the
     154parity of the sky.
    151155
    152156Nightly observations are conducted remotely from the Advanced
     
    240244%%     submission and refereeing process.}}
    241245
     246%% this figure comes from kukui.1/~/czw.paper.images.20181130
     247\begin{figure}
     248  \centering
     249  \includegraphics[width=0.9\hsize,angle=0,clip]{{pics/gpc1.layout}.pdf}
     250  \caption{Diagram illustrating layout of OTA devices in GPC1.  The
     251    blue dots mark the locations of the amplifiers for xy00 cells in
     252    each chip.  When cells are mosaicked to a single pixel grid, the
     253    pixel in this corner is at chip coordinate (0,0).  The figure
     254    illustrates the orientation of the OTA devices relative to the
     255    parity of the sky.  An exposure taken with North at the top of the
     256    field-of-view will have East to the left when the OTA devices are
     257    mosaicked as shown.  Note that the devices OTA0Y - OTA3Y are
     258    rotated by 180\degrees\ relative to the other half of the camera.
     259    The labeling of the non-existent corner OTAs is provided to orient
     260    the focal plane.}
     261  \label{fig:gpc1.layout}
     262\end{figure}
     263
    242264\section{Pan-STARRS\,1 Data Analysis}
    243265
     
    255277this article.
    256278
    257 The data processing steps are described in detail by \cite{waters2017}
    258 and \cite{magnier2017.datasystem,magnier2017.analysis}.  In summary, individual images
    259 are detrended: non-linearity and bias corrections are applied, a dark
    260 current model is subtracted and flat-field corrections are applied.
    261 The \yps-band images are also corrected for fringing: a master fringe
    262 pattern is scaled to match the observed fringing and subtracted.  Mask
    263 and variance image arrays are generated with the detrend analysis and
    264 carried forward at each stage of the IPP processing.  Source detection
    265 and photometry are performed for each chip independently.  As
    266 discussed below, preliminary astrometric and photometric calibrations
    267 are performed for all chips in a single exposure in a single analysis.
    268 We refer to these measurements as the ``chip'' photometry and
    269 astrometry products.
     279The pipeline data processing steps are described in detail by
     280\cite{waters2017} and
     281\cite{magnier2017.datasystem,magnier2017.analysis}.  In summary,
     282individual images are detrended: non-linearity and bias corrections
     283are applied, a dark current model is subtracted and flat-field
     284corrections are applied.  The \yps-band images are also corrected for
     285fringing: a master fringe pattern is scaled to match the observed
     286fringing and subtracted.  Mask and variance image arrays are generated
     287with the detrend analysis and carried forward at each stage of the IPP
     288processing.  Source detection and photometry are performed for each
     289chip independently.  As discussed below, preliminary astrometric and
     290photometric calibrations are performed for all chips in a single
     291exposure in a single analysis.  We refer to these measurements as the
     292``chip'' photometry and astrometry products.
    270293
    271294Chip images are geometrically transformed based on the astrometric
     
    321344the individual exposures and the stack images.
    322345
    323 \section{Astrometric Models}
    324 
    325 % \note{include projection math?} 
    326 % \note{reference discussion somewhere on cell vs chip}
     346\section{Pipeline Calibration}
     347
     348\subsection{Overview}
     349
     350As images are processed by the data analysis system, every exposure is
     351calibrated individually with respect to a photometric and astrometric
     352reference database.  The goal of this calibration step is to generate
     353a preliminary astrometric calibration, to be used by the warping
     354analysis to determine the geometric transformation of the pixels, and
     355a preliminary photometric transformation, to be used by the stacking
     356analysis to ensure the warps are combined using consistent flux units.
     357
     358The program used for the pipeline calibration, \ippprog{psastro},
     359loads the measurements of the chip detections from their individual
     360output catalog files.  It uses the header information populated at the
     361telescope to determine an initial astrometric calibration guess based
     362on the position of the telescope boresite right ascension, declination
     363and position angle as reported by the telescope \& camera subsystems.
     364Using the initial guess, \ippprog{psastro} loads astrometric and
     365photometric data from the reference database.
     366
     367\subsection{Reference Catalogs}
     368\label{sec:synthdb}
     369
     370During the course of the PS1SC Survey, several reference databases
     371have been used.  For the first 20 months of the survey,
     372\ippprog{psastro} used a reference catalog with synthetic PS1
     373\grizy\ photometry generated by the Pan-STARRS IPP team based on based
     374combined photometry from Tycho (B, V), USNO \citep[red, blue,
     375  IR][]{2003AJ....125..984M}, and 2MASS
     376$J, H, K$ \citep{2006AJ....131.1163S}.  The astrometry in the database was from 2MASS
     377\citep{2006AJ....131.1163S}.  After 2012 May, a reference catalog
     378generated from internal re-calibration of the PV0 analysis of PS1
     379photometry and astrometry was used for the reference catalog.
     380
     381Coordinates and calibrated magnitudes of stars from the reference
     382database are loaded by \code{pasastro}.  A model for the positions of
     383the 60 chips in the focal plane is used to determine the expected
     384astrometry for each chip based on the boresite coordinates and
     385position angle reported by the header.  Reference stars are selected
     386from the full field of view of the GPC1 camera, padded by an
     387additional 25\% to ensure a match can be determined even in the
     388presence of substantial errors in the boresite coordinates.  It is
     389important to choose an appropriate set of reference stars: if too few
     390are selected, the chance of finding a match between the reference and
     391observed stars is diminished.  In addition, since stars are loaded in
     392brightness order, a selection which is too small is likely to contain
     393only stars which are saturated in the GPC1 images.  On the other hand,
     394if too many reference stars are chosen, there is a higher chance of a
     395false-positive match, especially as many of the reference stars may
     396not be detected in the GPC1 image.  The selection of the reference
     397stars includes a limit on the brightest and faintest magnitudes of the
     398stars selected.
     399
     400The astrometric analysis is necessarily performed first; after the
     401astrometry is determined, an automatic byproduct is a reliable match
     402between reference and observed stars, allowing a comparison of the
     403magnitudes to determine the photometric calibration. 
     404
     405%% The astrometric calibration is performed in two major stages: first,
     406%% the chips are fitted independently with independent models for each
     407%% chip.  This fit is sufficient to ensure a reliable match between
     408%% reference stars and observed sources in the image.  Next, the set of
     409%% chip calibrations are used to define the transformation between the
     410%% focal plane coordinate system and the tangent plane coordinate
     411%% system.  The chip-to-focal plane transformations are then determined
     412%% under the single common focal plane to tangent plane transformation. 
     413
     414\subsection{Astrometric Models}
    327415
    328416Three somewhat distinct astrometric models are employed within the IPP
     
    344432  order}$, may be 1 to 3, under the restriction that sufficient stars
    345433are needed to constrain the order. 
    346 
    347 % \note{describe a bit better: this is automatically selected based on the number of stars}
    348434
    349435A second form of astrometry model which yields somewhat higher
     
    420506%% \end{verbatim}
    421507
    422 \section{Real-time Calibration}
    423 
    424 \subsection{Overview}
    425 
    426 As images are processed by the data analysis system, every exposure is
    427 calibrated individually with respect to a photometric and astrometric
    428 database.  The goal of this calibration step is to generate a preliminary
    429 astrometric calibration, to be used by the warping analysis to determine
    430 the geometric transformation of the pixels, and preliminary
    431 photometric transformation, to be used by the stacking analysis to
    432 ensure the warps are combined using consistent flux units.
    433 
    434 The program used for the real-time calibration, \ippprog{psastro},
    435 loads the measurements of the chip detections from their individual
    436 output catalog files.  It uses the header information populated at the
    437 telescope to determine an initial astrometric calibration guess based
    438 on the position of the telescope boresite right ascension, declination
    439 and position angle as reported by the telescope \& camera subsystems.
    440 Using the initial guess, \ippprog{psastro} loads astrometric and
    441 photometric data from the reference database.
    442 
    443 \subsection{Reference Catalogs}
    444 \label{sec:synthdb}
    445 
    446 During the course of the PS1SC Survey, several reference databases
    447 have been used.  For the first 20 months of the survey,
    448 \ippprog{psastro} used a reference catalog with synthetic PS1
    449 \grizy\ photometry generated by the Pan-STARRS IPP team based on based
    450 combined photometry from Tycho (B, V), USNO \citep[red, blue,
    451   IR][]{2003AJ....125..984M}, and 2MASS
    452 $J, H, K$ \citep{2006AJ....131.1163S}.  The astrometry in the database was from 2MASS
    453 \citep{2006AJ....131.1163S}.  After 2012 May, a reference catalog
    454 generated from internal re-calibration of the PV0 analysis of PS1
    455 photometry and astrometry was used for the reference catalog.
    456 
    457 % \note{discuss history of the different refcats?} 
    458 
    459 Coordinates and calibrated magnitudes of stars from the reference
    460 database are loaded by \code{pasastro}.  A model for the positions of
    461 the 60 chips in the focal plane is used to determine the expected
    462 astrometry for each chip based on the boresite coordinates and
    463 position angle reported by the header.  Reference stars are selected
    464 from the full field of view of the GPC1 camera, padded by an
    465 additional 25\% to ensure a match can be determined even in the
    466 presence of substantial errors in the boresite coordinates.  It is
    467 important to choose an appropriate set of reference stars: if too few
    468 are selected, the chance of finding a match between the reference and
    469 observed stars is diminished.  In addition, since stars are loaded in
    470 brightness order, a selection which is too small is likely to contain
    471 only stars which are saturated in the GPC1 images.  On the other hand,
    472 if too many reference stars are chosen, there is a higher chance of a
    473 false-positive match, especially as many of the reference stars may
    474 not be detected in the GPC1 image.  The selection of the reference
    475 stars includes a limit on the brightest and faintest magnitudes of the
    476 stars selected.
    477 
    478 The astrometric analysis is necessarily performed first; after the
    479 astrometry is determined, an automatic byproduct is a reliable match
    480 between reference and observed stars, allowing a comparison of the
    481 magnitudes to determine the photometric calibration. 
    482 
    483 The astrometric calibration is performed in two major stages: first,
    484 the chips are fitted independently with independent models for each
    485 chip.  This fit is sufficient to ensure a reliable match between
    486 reference stars and observed sources in the image.  Next, the set of
    487 chip calibrations are used to define the transformation between the
    488 focal plane coordinate system and the tangent plane coordinate
    489 system.  The chip-to-focal plane transformations are then determined
    490 under the single common focal plane to tangent plane transformation. 
    491 
    492508\subsection{Cross-Correlation Search}
    493509
     
    587603%% \note{quality of the fits as a result of this stage}.
    588604
    589 \subsection{Real-time Photometric Calibration}
     605\subsection{Pipeline Photometric Calibration}
    590606
    591607%% \note{define / describe the robust median}
    592608
    593 After the astrometric calibration has finished, the photometric
     609After the astrometric calibration is determined, the photometric
    594610calibration is performed by \ippprog{psastro}.  When the reference
    595611stars are loaded, the apparent magnitude in the filter of interest is
     
    598614points by comparison with the instrumental magnitudes.  For the PV3
    599615analysis, an outlier-rejecting median is used to measure the zero
    600 point. For early versions of the real-time analysis, when the
    601 reference catalog used synthetic magnitudes, it was necessary to
    602 search for the blue edge of the distribution: the synthetic magnitude
    603 poorly predicted the magnitudes of stars in the presence of
    604 significant extinction or for the very red stars, making the blue edge
    605 somewhat more reliable as a reference than the mean.  Once the
    606 calibration was based on a reference catalog generated from
    607 \PSONE\ photometry, this methods was no longer needed.  Note that we
    608 do not fit for the airmass slope in this analysis.  The nominal
    609 airmass slope is used for each filter; any deviation from the nominal
    610 value is effectively folded into the observed zero point.  The zero
    611 point may be measured separately for each chip or as a single value
    612 for the entire exposure; the latter option was used for the PV3
    613 analysis.
    614 
    615 \subsection{Real-time outputs}
     616point. For early versions of the pipeline analysis, when the reference
     617catalog used synthetic magnitudes, it was necessary to search for the
     618blue edge of the distribution: the synthetic magnitude poorly
     619predicted the magnitudes of stars in the presence of significant
     620extinction or for the very red stars, making the blue edge somewhat
     621more reliable as a reference than the mean.  Once the calibration was
     622based on a reference catalog generated from \PSONE\ photometry, this
     623methods was no longer needed.  Note that we do not fit for the airmass
     624slope in this analysis.  The nominal airmass slope is used for each
     625filter; any deviation from the nominal value is effectively folded
     626into the observed zero point.  The zero point may be measured
     627separately for each chip or as a single value for the entire exposure;
     628the latter option was used for the PV3 analysis.
     629
     630\subsection{Outputs}
    616631
    617632The calibrations determined by \ippprog{psastro} are saved as part of
     
    646661the data from the exposure are loaded into the DVO database.
    647662
    648 \section{PV3 DVO Master Database}
     663\section{Calibration Database}
    649664
    650665Data from the GPC1 chip images, the stack images, and the warp images
     
    712727\hline
    713728ID\_MEAS\_NOCAL              & 0x00000001 & detection ignored for this analysis (photcode, time range) -- internal only \\
    714 ID\_MEAS\_POOR\_PHOTOM       & 0x00000002 & detection is photometry outlier (not used PV3) \\
    715 ID\_MEAS\_SKIP\_PHOTOM       & 0x00000004 & detection was ignored for photometry measurement (not used PV3) \\
    716 ID\_MEAS\_AREA               & 0x00000008 & detection near image edge (not used PV3) \\
     729ID\_MEAS\_POOR\_PHOTOM       & 0x00000002 & detection is photometry outlier (not used for PV3) \\
     730ID\_MEAS\_SKIP\_PHOTOM       & 0x00000004 & detection was ignored for photometry measurement (not used for PV3) \\
     731ID\_MEAS\_AREA               & 0x00000008 & detection near image edge (not used for PV3) \\
    717732ID\_MEAS\_POOR\_ASTROM       & 0x00000010 & detection is astrometry outlier \\
    718 ID\_MEAS\_SKIP\_ASTROM       & 0x00000020 & detection was ignored for astrometry measurement \\
     733ID\_MEAS\_SKIP\_ASTROM       & 0x00000020 & detection was not used for image calibration (not reported for PV3) \\
    719734ID\_MEAS\_USED\_OBJ          & 0x00000040 & detection was used during update objects \\
    720 ID\_MEAS\_USED\_CHIP         & 0x00000080 & detection was used during update chips (not saved PV3) \\
    721 ID\_MEAS\_BLEND\_MEAS        & 0x00000100 & detection is within radius of multiple objects \\
    722 ID\_MEAS\_BLEND\_OBJ         & 0x00000200 & multiple detections within radius of object \\
     735ID\_MEAS\_USED\_CHIP         & 0x00000080 & detection was used during update chips (not saved for PV3) \\
     736ID\_MEAS\_BLEND\_MEAS        & 0x00000100 & detection is within radius of multiple objects (not used for PV3) \\
     737ID\_MEAS\_BLEND\_OBJ         & 0x00000200 & multiple detections within radius of object (not used for PV3) \\
    723738ID\_MEAS\_WARP\_USED         & 0x00000400 & measurement used to find mean warp photometry \\
    724739ID\_MEAS\_UNMASKED\_ASTRO    & 0x00000800 & measurement was not masked in final astrometry fit \\
    725 ID\_MEAS\_BLEND\_MEAS\_X     & 0x00001000 & detection is within radius of multiple objects across catalogs \\
    726 ID\_MEAS\_ARTIFACT           & 0x00002000 & detection is thought to be non-astronomical \\
    727 ID\_MEAS\_SYNTH\_MAG         & 0x00004000 & magnitude is synthetic \\
     740ID\_MEAS\_BLEND\_MEAS\_X     & 0x00001000 & detection is within radius of multiple objects across catalogs (not used for PV3) \\
     741ID\_MEAS\_ARTIFACT           & 0x00002000 & detection is thought to be non-astronomical (not used for PV3) \\
     742ID\_MEAS\_SYNTH\_MAG         & 0x00004000 & magnitude is synthetic (not used for DR2) \\
    728743ID\_MEAS\_PHOTOM\_UBERCAL    & 0x00008000 & externally-supplied zero point from ubercal analysis \\
    729744ID\_MEAS\_STACK\_PRIMARY     & 0x00010000 & this stack measurement is in the primary skycell \\
    730745ID\_MEAS\_STACK\_PHOT\_SRC   & 0x00020000 & this measurement supplied the stack photometry \\
    731 ID\_MEAS\_ICRF\_QSO          & 0x00040000 & this measurement is an ICRF reference position \\
    732 ID\_MEAS\_IMAGE\_EPOCH       & 0x00080000 & this measurement is registered to the image epoch (not tied to ref catalog epoch) \\
     746ID\_MEAS\_ICRF\_QSO          & 0x00040000 & this measurement is an ICRF reference position (not used for PV3) \\
     747ID\_MEAS\_IMAGE\_EPOCH       & 0x00080000 & this measurement is registered to the image epoch (not used for PV3) \\
    733748ID\_MEAS\_PHOTOM\_PSF        & 0x00100000 & this measurement is used for the mean psf mag \\
    734749ID\_MEAS\_PHOTOM\_APER       & 0x00200000 & this measurement is used for the mean ap mag \\
     
    754769{\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\
    755770\hline
    756 ID\_SECF\_STAR\_FEW                & 0x00000001 & Used within relphot: skip star \\
    757 ID\_SECF\_STAR\_POOR               & 0x00000002 & Used within relphot: skip star \\
    758 ID\_SECF\_USE\_SYNTH               & 0x00000004 & Synthetic photometry used in average measurement \\
     771ID\_SECF\_STAR\_FEW                & 0x00000001 & Used within relphot: skip star (not reported for PV3) \\
     772ID\_SECF\_STAR\_POOR               & 0x00000002 & Used within relphot: skip star (not reported for PV3) \\
     773ID\_SECF\_USE\_SYNTH               & 0x00000004 & Synthetic photometry used in average measurement (not used in PV3) \\
    759774ID\_SECF\_USE\_UBERCAL             & 0x00000008 & Ubercal photometry used in average measurement \\
    760775ID\_SECF\_HAS\_PS1                 & 0x00000010 & PS1 photometry used in average measurement \\
    761776ID\_SECF\_HAS\_PS1\_STACK          & 0x00000020 & PS1 stack photometry exists \\
    762 ID\_SECF\_HAS\_TYCHO               & 0x00000040 & Tycho photometry used for synth mags \\
    763 ID\_SECF\_FIX\_SYNTH               & 0x00000080 & Synth mags repaired with zpt map \\
     777ID\_SECF\_HAS\_TYCHO               & 0x00000040 & Tycho photometry used for synth mags (not used in PV3) \\
     778ID\_SECF\_FIX\_SYNTH               & 0x00000080 & Synth mags repaired with zpt map (not used in PV3) \\
    764779ID\_SECF\_RANK\_0                  & 0x00000100 & Average magnitude uses rank 0 values \\
    765780ID\_SECF\_RANK\_1                  & 0x00000200 & Average magnitude uses rank 1 values \\
     
    772787ID\_SECF\_STACK\_PRIMDET           & 0x00010000 & PS1 stack primary measurement is a detection (not forced) \\
    773788ID\_SECF\_STACK\_PRIMARY\_MULTIPLE & 0x00020000 & PS1 stack object has multiple primary measurements \\
    774 ID\_SECF\_HAS\_SDSS                & 0x00100000 & This photcode has SDSS photometry \\
    775 ID\_SECF\_HAS\_HSC                 & 0x00200000 & This photcode has HSC  photometry \\
    776 ID\_SECF\_HAS\_CFH                 & 0x00400000 & This photcode has CFH  photometry (mostly Megacam) \\
    777 ID\_SECF\_HAS\_DES                 & 0x00800000 & This photcode has DES  photometry \\
     789ID\_SECF\_HAS\_SDSS                & 0x00100000 & This photcode has SDSS photometry (not used for PV3) \\
     790ID\_SECF\_HAS\_HSC                 & 0x00200000 & This photcode has HSC  photometry (not used for PV3) \\
     791ID\_SECF\_HAS\_CFH                 & 0x00400000 & This photcode has CFH  photometry (not used for PV3) \\
     792ID\_SECF\_HAS\_DES                 & 0x00800000 & This photcode has DES  photometry (not used for PV3) \\
    778793ID\_SECF\_OBJ\_EXT                 & 0x01000000 & Extended in this band \\
    779794\hline
     
    791806{\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\
    792807\hline
    793 ID\_OBJ\_FEW               & 0x00000001 & used within relphot: skip star \\
    794 ID\_OBJ\_POOR              & 0x00000002 & used within relphot: skip star \\
    795 ID\_OBJ\_ICRF\_QSO         & 0x00000004 & object IDed with known ICRF quasar (may have ICRF position measurement) \\
     808ID\_OBJ\_FEW               & 0x00000001 & used within relphot: skip star (not reported for PV3) \\
     809ID\_OBJ\_POOR              & 0x00000002 & used within relphot: skip star (not reported for PV3) \\
     810ID\_OBJ\_ICRF\_QSO         & 0x00000004 & object IDed with known ICRF quasar (not used for PV3) \\
    796811ID\_OBJ\_HERN\_QSO\_P60    & 0x00000008 & identified as likely QSO \citep{2016ApJ...817...73H}, $P_{\rm QSO} \geq 0.60$ \\
    797812ID\_OBJ\_HERN\_QSO\_P05    & 0x00000010 & identified as possible QSO \citep{2016ApJ...817...73H}, $P_{\rm QSO} \geq 0.05$ \\
     
    799814ID\_OBJ\_HERN\_RRL\_P05    & 0x00000040 & identified as possible RR Lyra \citep{2016ApJ...817...73H}, $P_{\rm RRLyra} \geq 0.05$ \\
    800815ID\_OBJ\_HERN\_VARIABLE    & 0x00000080 & identified as a variable by \cite{2016ApJ...817...73H} \\
    801 ID\_OBJ\_TRANSIENT         & 0x00000100 & identified as a non-periodic (stationary) transient \\
     816ID\_OBJ\_TRANSIENT         & 0x00000100 & identified as a non-periodic (stationary) transient (not used for PV3) \\
    802817ID\_OBJ\_HAS\_SOLSYS\_DET  & 0x00000200 & identified with a known solar-system object (asteroid or other) \\
    803818ID\_OBJ\_MOST\_SOLSYS\_DET & 0x00000400 & most detections from a known solar-system object \\
    804 ID\_OBJ\_LARGE\_PM         & 0x00000800 & star with large proper motion \\
     819ID\_OBJ\_LARGE\_PM         & 0x00000800 & star with large proper motion (not used for PV3) \\
    805820ID\_OBJ\_RAW\_AVE          & 0x00001000 & simple weighted average position was used (no IRLS fitting) \\
    806821ID\_OBJ\_FIT\_AVE          & 0x00002000 & average position was fitted \\
     
    840855ID\_IMAGE\_NEW             & 0x00000000 & no calibrations yet attempted \\
    841856ID\_IMAGE\_PHOTOM\_NOCAL   & 0x00000001 & user-set value used within relphot: ignore \\
    842 ID\_IMAGE\_PHOTOM\_POOR    & 0x00000002 & relphot says image is bad (dMcal > limit) \\
     857ID\_IMAGE\_PHOTOM\_POOR    & 0x00000002 & relphot says image is bad (dMcal $>$ limit) \\
    843858ID\_IMAGE\_PHOTOM\_SKIP    & 0x00000004 & user-set value: assert that this image has bad photometry \\
    844859ID\_IMAGE\_PHOTOM\_FEW     & 0x00000008 & currently too few measurements for photometry \\
    845860ID\_IMAGE\_ASTROM\_NOCAL   & 0x00000010 & user-set value used within relastro: ignore \\
    846 ID\_IMAGE\_ASTROM\_POOR    & 0x00000020 & relastro says image is bad (dR,dD > limit) \\
     861ID\_IMAGE\_ASTROM\_POOR    & 0x00000020 & relastro says image is bad (dR,dD $>$ limit) \\
    847862ID\_IMAGE\_ASTROM\_FAIL    & 0x00000040 & relastro fit diverged, fit not applied \\
    848863ID\_IMAGE\_ASTROM\_SKIP    & 0x00000080 & user-set value: assert that this image has bad astrometry \\
     
    861876\subsection{Ubercal Analysis}
    862877
    863 % \note{clean up and re-word the pieces below}
    864 
    865878The photometric calibration of the DVO database starts with the
    866879``ubercal'' analysis technique as described by
    867880\cite{2012ApJ...756..158S}.  This analysis is performed by the group
    868 at Harvard, loading data from the \code{smf} files into their instance
     881at Harvard, loading data from the raw detection files into their instance
    869882of the Large Scale Database \citep[LSD,][]{2011AAS...21743319J}, a
    870883system similar to DVO used to manage the detections and determine the
     
    894907additional flat-field seasons.
    895908
    896 %% \note{something for PV4}.
    897 
    898909By excluding non-photometric data and only fitting 2 parameters for
    899910each night, the Ubercal solution is robust and rigid.  It is not
     
    907918millimags in (\grizy).  As we discuss below, this conclusion is
    908919reinforced by our external comparison. 
    909 
    910 %% \note{do I have a measurement
    911 %% of the bright end stability in PV3?  basically, what is the scatter
    912 %% per star as a function of position in the camera and magnitude?}
    913920
    914921The overall zero point for each filter is not naturally determined by
     
    929936\cite{2012ApJ...756..158S}.
    930937
    931 %% \note{The calspec spectrophotometry values have also been re-examined
    932 %%   by REF; using these new measurements, \cite{2015ApJ...815..117S}
    933 %%   determine new zero points for the PS1 system, which we have applied
    934 %%   (see below).}
    935 
    936938% http://iopscience.iop.org/article/10.1088/0004-637X/815/2/117/pdf
    937939
    938 \subsection{Applying the Ubercal Zero Points : Setphot}
     940\subsection{Apply Zero Points}
    939941
    940942The ubercal analysis above results in a table of zero points for all
     
    976978\hline
    977979\hline
    978 {\bf Filter} & {\bf Zero Point} & {\bf Zero Point} & {\bf Airmass Slope} \\
    979  & {\bf (Raw)} & {\bf (Calspec)} & \\
     980{\bf Filter} & {\bf Zero Point} & {\bf Zero Point} & {\bf Airmass} \\
     981 & {\bf (Raw)} & {\bf (Calspec)} & {\bf Slope} \\
    980982\hline
    981983\gps & 24.563 & 24.583 & 0.147 \\
     
    988990\end{center}
    989991\end{table}
    990 
    991 %% \note{give airmass formula for completeness?}.
    992992
    993993When \code{setphot} applies the ubercal information to the image
     
    10771077the offsets converge to the milli-magnitude level within 8 iterations.
    10781078
    1079 Only brighter, high quality measurements are used in the relative
     1079Only high quality measurements are used in the relative
    10801080photometry analysis of the exposure zero points.  We use only the
    10811081brighter objects, limiting the density to a maximum of 4000 objects
     
    11471147The calculation of the relative photometry zero points is performed
    11481148for the entire $3\pi$ data set in a single, highly parallelized
    1149 analysis.  As discussed above, the measurement and object data in the
     1149analysis.  The measurement and object data in the
    11501150DVO database are distributed across a large number of computers in the
    11511151IPP cluster: for PV3, 100 parallel hosts are used.  These machines by
     
    11641164of responsibility. 
    11651165
     1166%% plots made using scripts and data in
     1167% /data/kukui.3/eugene/pv3.cam.20150607:
     1168% photflat.20151127.fix/photflat.20151127.fix.0.*.fits
     1169% based on extractions in:
     1170% /data/ipp094.0/eugene/pv3.cam.20150607/astrom.corrections/
     1171% measurements are in photflat.extract.*.fits
     1172% tdhistograms in photflat.20151127/
     1173% script: photflat.sh
     1174% catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master
     1175% measurement extraction was done ~ 2015.11.25-27
     1176% this is PV3.0 [pre-calibrations]
     1177
    11661178\begin{figure*}[htbp]
    11671179 \begin{center}
    11681180  \begin{minipage}{0.85\linewidth}
    1169    \includegraphics[width=\textwidth,clip]{{pics/photflat.example.sm}.png}
     1181   \includegraphics[width=\textwidth,clip]{{pics/photflat.example.v1}.png}
    11701182  \end{minipage}
    1171   \hspace{-2.75in}
     1183  \hspace{-3.0in}
    11721184  \begin{minipage}{0.4\linewidth}
    1173    \vspace{3.25in}
    1174    \caption{\label{fig:photflat} High-resolution flat-field correction images for the 5 filters $grizy$.}
     1185   \vspace{6.0in}
     1186   \caption{\label{fig:photflat} High-resolution flat-field correction
     1187     images for the 5 filters $grizy$.  These images are shown in
     1188     standard camera orientation with OTA00 in the lower-left
     1189     corner and OTA07 in the upper-right corner.  Fine
     1190     ``tree-ring'' structures are visible in several chips especially
     1191     in the bluer bands.  The effect of the central ``tent'' on the
     1192     photometry, presumably due to the rapidly-varying PSF in this
     1193     region may also be seen. }
    11751194  \end{minipage}
    11761195 \end{center}
     
    12201239analysis.
    12211240
     1241%% figure made using scripts and data in:
     1242% /data/kukui.3/eugene/pv3.stats.20161202
     1243% scatter.sh : allsky.scatter.photom
     1244% maps.measure/pv3.v1.dmag_*.sigma.fits
     1245% cdhist.measure/cdmerge.v1.dmag_*.fits
     1246%
     1247%% cdhist.measure from:
     1248% /data/ipp094.0/eugene/pv3.stats.20161202/
     1249% measures.sh : extract.allsky
     1250% used catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master
     1251% data was extracted 2016.12.11 : PV3.2 calibration
     1252
     1253%% the mean camera photometry was not modified after this date
     1254%% These extractions should be used for the paper (EAM 2019.02.15)
     1255
    12221256\begin{figure*}[htbp]
    12231257  \begin{center}
    12241258%width=\hsize
    1225  \includegraphics[height=\vsize,clip]{{pics/allsky.photom.sigma.sm}.png}
     1259 \includegraphics[height=\vsize,clip]{{pics/allsky.photom.v1}.png}
    12261260  \caption{\label{fig:allsky.photom.sigma} Consistency of photometry
    12271261    measurements across the sky.  Each panel shows a map of the
     
    12331267  \end{center}
    12341268\end{figure*}
    1235 
    1236 %% \note{need to discuss the process of setting the final mean magnitudes}
    12371269
    12381270\subsubsection{Photometric Flat-field}
     
    12911323variable charge diffusion.
    12921324
    1293 Other features include some poorly responding cells (e.g., in XY14)
     1325Other features include some poorly responding cells (e.g., in OTA14)
    12941326and effects at the edges of chips, possibly where the PSF model fails
    12951327to follow the changes in the PSF.
    1296 
    1297 %% XXX : need to refer to system paper on the central tent?
    1298 
    1299 %% \note{show the flat-field residual images, discuss the features?}. 
    13001328
    13011329For stacks and warps, the image calibrations were determined after the
     
    13231351magnitudes, but the aperture-like magnitudes are tied by equating the
    13241352stack Kron magnitudes to the average chip Kron magnitudes.  {\em Note
    1325   that for DR1, this split zero point calibration was used; instead
     1353  that for DR1, this split zero point calibration was {\bf not} used; instead
    13261354  all stack photometry was tied to the average chip photometry via the
    13271355  PSF magnitudes.}  The result of using a single zero point is that
     
    13321360This split is not needed for the forced-warp photometry since the
    13331361individual warps have well-defined PSfs.
     1362
     1363%% XXX generate a figure to illustrate the Kron vs PSF mags in stacks (DR1 & DR2)
    13341364
    13351365\subsection{Photometry Calibration Quality}
     
    1369139918)$ millimagnitudes.
    13701400
    1371 %% \note{recommendation}
    1372 
    1373 \subsection{Calculation of Object Photometry}
     1401\subsection{Object Photometry}
    13741402
    13751403Once the image photometric calibrations (zero points and flat-field
     
    14001428
    14011429\subsubsection{Selection of Measurements}
     1430\label{sec:measurement.quality}
    14021431
    14031432To choose the measurements which will be used in the analysis, we
     
    14861515raised identifying which rank was used.  These bit are called
    14871516\code{ID_SECF_RANK_0} through \code{ID_SECF_RANK_4} (see
    1488 Table~\ref{tab:secf_mask_values}). 
     1517Table~\ref{tab:secf_mask_values}).  This assessment of the valid
     1518measurements is performed independently for PSF, Kron, and
     1519seeing-matched total aperture magnitudes.  All measurements which are
     1520retained to determine the average value are marked with bit-flags: \code{ID_MEAS_PHOTOM_PSF},
     1521\code{ID_MEAS_PHOTOM_KRON}, or \code{ID_MEAS_PHOTOM_APER} depending on
     1522which average magnitude is being calculated.
    14891523
    14901524%% where do these go? analyis?
     
    15101544underlying constant value.  The discussion below applies to both the
    15111545average of the chip photometry magnitudes and the forced-warp
    1512 photometry fluxes.
     1546photometry fluxes.  This technique is used to calculate the average
     1547magnitudes for all three types of photometry stored in the DVO
     1548database: PSF, Kron, and seeing-matched total aperture photometry. 
    15131549
    15141550The IRLS analysis starts with an ordinary least squares fit, using the
     
    15501586converge.
    15511587
    1552 % \note{did this happen for any of our targets?}
    1553 
    15541588To calculate a fit $\chi^2$ value and to determine an appropriate set
    15551589of errors for the model parameters, it is necessary to transform the
     
    15581592($\omega^\prime < 0.3 <\omega>$) then the point is treated as clipped.
    15591593The $\chi^2$ is determined from the {\em unclipped} points using the
    1560 standard Poisson errors.
     1594standard Poisson errors.  Data points which are so excluded are marked
     1595with bit-flags: \code{ID_MEAS_MASKED_PSF},
     1596\code{ID_MEAS_MASKED_KRON}, or \code{ID_MEAS_MASKED_APER} depending on
     1597which average magnitude is being calculated.
    15611598
    15621599Bootstrap-resampling analysis is used to assess the errors on the fit
     
    15751612photometry.
    15761613
     1614One detail related to the above analysis concerns the measurements
     1615from images which were included in the ubercal analysis.  These images
     1616were determined to have been taken in good quality (photometric)
     1617weather, and have had their zero points determined with a robust
     1618analysis.  We therefore over-weight these data points to ensure the
     1619average photometry is dominated by the ubercal values.  In the IRLS
     1620analysis above, the ubercal points are given 10 times the weight of
     1621the non-ubercal points.  This over-weighting is applied independently
     1622of the calculation of the reweighting based on the deviation from the
     1623model.  Thus, the increased weight is {\em not} applied by reducing
     1624the errorbars by a factor of 10 since that would increase the chance
     1625that the ubercal measurements would be given reduced weight.  If the
     1626average photometry of an object in a filter includes ubercal
     1627measurements, the per-filter bit flag \code{ID_SECF_USE_UBERCAL} is set.   
     1628
    15771629% mask values for which wt < threshold (0.3 * median wt)
    15781630% we record the min and max values of the unmasked / unclipped subset
     
    15801632% bootstrap: use only unclipped subset and raw weights to estimate errors
    15811633
    1582 % \note{bootstrap uses unclipped values and the raw weights? confirmed}
    1583 
    1584 % \note{reported error is max of bootstrap and formal error?  confirmed}
    1585 
    15861634\subsubsection{Stack Photometry}
     1635\label{sec:stack.phot}
    15871636
    15881637For the stack photometry, the assessment is different from the chip
     
    15961645detections of the same object.  This situation is discussed in more
    15971646detail below. 
     1647
     1648% generate from :
     1649% /data/kukui.1/eugene/czw.paper.images.20181130 (see .dvo)
    15981650
    15991651\begin{figure*}[htbp]
     
    16611713Since the ``primary'' identification is purely based on the skycell
    16621714geometry and the coordinate of the object, there is no guarantee that
    1663 any primary measurement is in fact a good or even the best measurement
     1715any primary measurement is in fact the best or even a good measurement
    16641716of the object.  While the different overlapping pixels should be
    16651717essentially identical, it is possible (due to some of the edge cases
     
    16871739split should not be common (and in fact reflects a failure of the
    16881740algorithm), but we have defined the rules to allows us to choose an
    1689 acceptable measurement even in these cases.
     1741acceptable measurement even in these cases.  Also note that the
     1742``best'' measurement is not guarateed to be a good measurement.
     1743
     1744Stack measurements which are in the ``primary'' skycell have the bit
     1745flag \code{ID_MEAS_STACK_PRIMARY}.  The measurement which was
     1746identified as the ``best'' measurement gets the bit flag
     1747\code{ID_MEAS_STACK_PHOT_SRC}.  If a ``primary'' measurement exists
     1748for a given filter, then the per-filter bit flag
     1749\code{ID_SECF_STACK_PRIMARY} is set for that filter.  If multiple
     1750primary stack measurements exist for a given filter, then the
     1751per-filter bit flag \code{ID_SECF_STACK_PRIMARY_MULTIPLE} is also set
     1752for that filter.
     1753%
     1754If the ``best'' measurement for a filter is a significant detection
     1755(not forced from another band), then the per-filter bit flag
     1756\code{ID_SECF_STACK_BESTDET} is set.
     1757%
     1758If any of the ``primary'' measurements for a filter is a significant
     1759detection (not forced from another band), then the per-filter bit flag
     1760\code{ID_SECF_STACK_PRIMDET} is set.
     1761%
     1762If any stack measurements exist for a given filter, then the
     1763per-filter bit flag \code{ID_SECF_HAS_PS1_STACK} is set.
     1764
     1765The ``best'' stack measurements are examined across the filters. If
     1766for all five filters, the ``best'' stack measurement is a ``primary''
     1767measurement, then the object bit flag \code{ID_OBJ_BEST_STACK} is set.
     1768%
     1769If the ``best'' stack measurement in a filter has signal-to-noise less
     1770than 5, has any of the ``bad quality'' bits raised (see
     1771Section~\ref{sec:measurement.quality}, rank 6), or has a \code{PSF_QF}
     1772value less than 0.85 (or NAN) is considered to be ``bad''.
     1773%
     1774It it has any of the ``poor quality'' bits raised (see
     1775Section~\ref{sec:measurement.quality}, rank 2), or has a
     1776\code{PSF_QF_PERFECT} value less than 0.85 is considered to be
     1777``suspect''. 
     1778%
     1779Otherwise, the measurement is considered to be ``good''.  For an
     1780object detected in the stacks, if at least 2 of the filters have
     1781``good'' stack measurements, then the object is considered to be
     1782``good'', \ie, likely to be a valid astronomical object, and the
     1783object bit flag \code{ID_OBJ_GOOD_STACK} is set.  If no more than one
     1784filter measurement is good, and there are at least two good or suspect
     1785measurements, then the object is considered to be ``suspect'' and the
     1786object bit flag \code{ID_OBJ_SUSPECT_STACK} is set.  If at most a
     1787single measurement is either good or suspect, then the object is
     1788considered to be ``bad'' and the object bit flag
     1789\code{ID_OBJ_BAD_STACK} is set.  Note, however, that a high redshift
     1790quasar which is well detected in the \yps-band but undetected in the
     1791other bands would be labeled ``bad''; caution is required as always.
     1792
     1793In the public science database (PSPS) available through the MAST
     1794interface includes two fields in the \ippdbtable{StackObjectThin}
     1795table, \ippdbcolumn{primaryDetection} and \ippdbcolumn{bestDetection}.
     1796These fields have an error in their definition and should not be used
     1797for either DR1 or DR2.  An update to the database will define fields
     1798for each object which encapsulate the information about the ``primary''
     1799and ``best'' detections.
    16901800
    16911801\subsubsection{Warp Photometry}
     
    17061816been selected, the same quality cuts are applied to the measurements
    17071817as are applied to the chip measurements, as discussed above.
     1818Forced-warp measurements actually used to calculate the average for a
     1819filter are marked with the bit flag \code{ID_MEAS_WARP_USED}.
     1820
     1821% from: /data/kukui.3/eugene/pv3.stats.20161202/
    17081822
    17091823\begin{figure*}[htbp]
     
    17111825 \includegraphics[width=\hsize,clip]{{pics/KHexample}.png}
    17121826  \caption{\label{fig:KHexample} Illustration of the Koppenh\"ofer Effect
    1713     on chip XY04.  {\bf Bottom left} X-direction before correction.  The solid line shows the measured
     1827    on OTA04.  {\bf Bottom left} X-direction before correction.  The solid line shows the measured
    17141828    mean residual for stars detected on this chip as a function of the
    17151829    instrumental magnitude / FWHM$^2$. 
     
    17191833  \end{center}
    17201834\end{figure*}
     1835
     1836% from: /data/kukui.3/eugene/pv3.stats.20161202/
    17211837
    17221838\begin{figure}[htbp]
     
    17321848  \end{center}
    17331849\end{figure}
     1850
     1851\subsubsection{Object Photometry Flags}
     1852
     1853Certain object-level bit flags are set based on the
     1854\ippstage{chip}-stage measurements.  If any object has at least one
     1855PS1 measurement from rank 0 - 2
     1856(Section~\ref{sec:measurement.quality}), then the object is marked
     1857with the bit flag \code{ID_OBJ_GOOD}.  Each measurement is also
     1858checked for consistency with a PSF or an extended source morphology:
     1859if the difference between the PSF magnitude and the seeing-matched
     1860full aperture magnitude is less than a specific cut-off (2.5$\sigma$
     1861added in quadrature to a floor of 0.1 magnitudes), then the
     1862measurement is considered ``PSF-like''.  Otherwise, the measurement is
     1863counted as extended.  If more of the PS1 measurements are extended
     1864than PSF-like, the object bit flag \code{ID_OBJ_EXT} is raised.  If
     1865more than half of the PS1 \ippstage{chip}-stage measurements within a
     1866single filter are extended, then the per-filter bit flag
     1867\code{ID_SEC_OBJ_EXT} and \code{ID_SEC_OBJ_EXT_PSPS} are set.  The
     1868latter bit is a duplicate bit defined because the high bit in a 32-bit
     1869integer is difficult to handle within the context of SQL server.  Any
     1870object which has any \ippstage{chip}-stage measurements for one of the
     1871five filters has the per-filter bit flag \code{ID_SECF_HAS_PS1} set.
     1872
     1873In addition, if the object has measurements from the 2MASS point
     1874source catalog, the quality of these measurements are check.  If the
     18752MASS quality flag \code{ph_qual} has a value of A,B, or C, then the
     1876object is considered to be a good 2MASS object and the bit flag
     1877\code{ID_OBJ_GOOD_ALT} is set.  If the 2MASS extended source flag,
     1878\code{gal_contam}, has a value of 1 or 2 then the object bit flag
     1879\code{ID_OBJ_EXT_ALT} is set.
     1880
     1881%% the flags below were in fact correctly set -- verified 2019.02.26
     1882%% for 3pi.pv3.20170919 (and logs say they were set 2016.04.12 in
     1883%% /data/ipp094.0/eugene/hernitschek.20151125
     1884
     1885We also set certain object-level bit flags based on additional
     1886analysis of the Pan-STARRS data.  \cite{2016ApJ...817...73H} used
     1887measurements from the $3\pi$ survey to identify potentially
     1888interesting variable sources.  They examined the characteristics of
     1889the varying fluxes in the 5 bands to distinguish two classes of
     1890variable sources: RR Lyrae stars and QSOs.  They present two
     1891classifier statistics, $P_{\rm QSO}$ and $P_{\rm RRLyrae}$ which can
     1892be used to select candidates with varying levels of quality and
     1893completeness.  Using this catalog, we have marked objects with a set
     1894of bits to specify the possible varibility information as identified
     1895by \cite{2016ApJ...817...73H}:
     1896\begin{itemize}
     1897\item \code{ID_OBJ_HERN_QSO_P60} : identified as likely QSO, $P_{\rm QSO} \geq 0.60$
     1898\item \code{ID_OBJ_HERN_QSO_P05} : identified as possible QSO, $P_{\rm QSO} \geq 0.05$
     1899\item \code{ID_OBJ_HERN_RRL_P60} : identified as likely RR Lyra, $P_{\rm RRLyra} \geq 0.60$
     1900\item \code{ID_OBJ_HERN_RRL_P05} : identified as possible RR Lyra, $P_{\rm RRLyra} \geq 0.05$
     1901\item \code{ID_OBJ_HERN_VARIABLE} : identified as a variable by \cite{2016ApJ...817...73H}
     1902\end{itemize}
     1903In addition, the Pan-STARRS MOPS team has identified solar-system
     1904objects within the $3\pi$ dataset.  We have used a list of 14.7M such
     1905detections recorded by MOPS from the $3\pi$ survey.  Any object which
     1906contains one of these detections has the object bit flag
     1907\code{ID_OBJ_HAS_SOLSYS_DET} set.  If 50\% or more of the detections
     1908for an object are solar-system objects, then the bit flag
     1909\code{ID_OBJ_MOST_SOLSYS_DET} is set.
    17341910
    17351911\section{Astrometry Calibration}
     
    17941970% ALL             322922            1163377   27.76
    17951971
    1796 % \note{was there is significant difference using a surface brightness version?} 
    1797 
    17981972We measured the Koppenh\"ofer Effect by accumulating the residual
    17991973astrometry statistics for stars in the database.  For each chip, we
     
    18272001define a blue DCR color ($g-i$) to be used when correcting the filters
    18282002\gps,\rps,\ips, and a red DCR color ($z - y$) to be used when
    1829 correcting the filters $zy$.  In the process of performing the
     2003correcting the filters \zps\ and \yps.  In the process of performing the
    18302004relative astrometry calibration, we record the median red and blue
    18312005colors of the reference stars used to measure the astrometry
     
    18422016the difference between the star color and the reference star color,
    18432017using the red or blue color appropriate to the particular filter, times
    1844 the tangent of the zenith distance.  Figure~\ref{fig:DCRexample} shows the
    1845 DCR trend for the 5 filters \grizy, as well as the measured
    1846 displacement in the direction perpendicular to the parallactic angle.
    1847 We represent the trend with a spline fitted to this dataset. 
     2018the tangent of the zenith distance:
     2019\begin{eqnarray}
     2020\delta_{\rm blue} = \alpha \left[(g - i)_{\rm ref} - (g - i)\right] \tan \zeta \\
     2021\delta_{\rm red} = \alpha \left[(z - y)_{\rm ref} - (z - y)\right] \tan \zeta
     2022\end{eqnarray}
     2023where $(g-i)_{\rm ref}$ and $(z-y)_{\rm ref}$ are the median colors of the
     2024stars used the calibrate a specific blue- or red-filter image,
     2025respecitively, while $\zeta$ is the zenith distance.
     2026Figure~\ref{fig:DCRexample} shows the DCR trend for the 5 filters
     2027\grizy, as well as the measured displacement in the direction
     2028perpendicular to the parallactic angle.  We represent the trend with a
     2029spline fitted to this dataset.
     2030
     2031% figure from /data/kukui.3/eugene/dcr.20141205
     2032% based on /data/ipp064.0/eugene/dcr.20141205
     2033% script: dvo.dcr.sh
     2034% catdir /data/stsci19.2/eugene/addstar.20141016/lap.pv2.subset.catdir
     2035% XXX THIS IS A PV2 analysis!
     2036%
     2037% Generate new figure using:
     2038% /data/ipp094.0/eugene/pv3.cam.20150607/astrom.corrections/dcr.meas.20151203.0.fits
    18482039
    18492040\begin{figure}[htbp]
     
    18562047\end{figure}
    18572048
    1858 The amplitude of the DCR trend in the five filters is $(g,r,i,z,y) =
    1859 (0.010, 0.001, -0.003, -0.017, -0.021)$ arcsec airmass$^{-1}$
    1860 magnitude$^{-1}$.  We saturate the DCR correction if the term $color
    1861 TAN (\zeta)$ for a given measurement is outside a range where the
    1862 DCR correction is well measured.  The maximum DCR correction applied
    1863 to the five filters is $(g,r,i,z,y) = (0.019, 0.002, 0.003, 0.006,
    1864 0.008)$ arcseconds.
    1865 
    1866 %% \note{write down the DCR formalae for reference}.
     2049The amplitude of the DCR trend, $\alpha$, in the five filters is
     2050$(g,r,i,z,y) = (0.010, 0.001, -0.003, -0.017, -0.021)$ arcsec
     2051airmass$^{-1}$ magnitude$^{-1}$.  We saturate the DCR correction if
     2052the term $\left[gi_{\rm ref} - (g - i)\right] \tan \zeta$ or
     2053$\left[zy_{\rm ref} - (z - y)\right] \tan \zeta$ for a given
     2054measurement is outside of the range where the DCR correction is
     2055measured.  The maximum DCR correction applied to the five filters is
     2056$(g,r,i,z,y) = (0.019, 0.002, 0.003, 0.006, 0.008)$ arcseconds.
     2057
     2058%% plots made using scripts and data in
     2059% /data/kukui.3/eugene/pv3.cam.20150607:
     2060% astroflat.20151205/astroflat.20151205.v2.$dir.$filter.fits
     2061%
     2062% based on extractions in:
     2063% /data/ipp094.0/eugene/pv3.cam.20150607/astrom.corrections/
     2064% measurements are in astroflat.0.fits - astroflat.3.fits
     2065%
     2066% script: dvo.astroflat.sh
     2067% catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master
     2068% measurement extraction was done 2015.12.04
     2069% this is PV3.0 [pre-calibrations]
     2070%
     2071% NOTE: the extraction generated 4 meas tables, but the flat-field
     2072% was built with only 1 (the .0.fits version)
     2073%
     2074% 2017.02.17 : I generated a new set of flats based on all 4 extractions
     2075% this is in /data/ipp105.0/eugene/astrom.20170225/astroflat.20170217/
     2076% and was applied to the database 2017.02.25 (../run.setastrom)
     2077%
     2078% generate new astrometric flat-field images based on e.g.:
     2079% /data/ipp105.0/eugene/astrom.20170225/astroflat.20170217/astroflat.20170217.med.cam.dX.g.fits
    18672080
    18682081\begin{figure*}[htbp]
    18692082 \begin{center}
    1870  \includegraphics[width=0.85\textwidth,clip]{{pics/astroflat.gri.sm}.png}
     2083 \includegraphics[width=0.85\textwidth,clip]{{pics/astroflat.gri.v1}.png}
    18712084 \caption{\label{fig:astroflat.gri} High-resolution astrometric flat-field correction images for $gri$.}
    18722085 \end{center}
     
    18752088\begin{figure*}[htbp]
    18762089 \begin{center}
    1877  \includegraphics[width=0.85\textwidth,clip]{{pics/astroflat.zy.sm}.png}
     2090 \includegraphics[width=0.85\textwidth,clip]{{pics/astroflat.zy.v1}.png}
    18782091 \caption{\label{fig:astroflat.zy} High-resolution astrometric flat-field correction images for $zy$.}
    18792092 \end{center}
     
    18812094
    18822095\subsubsection{Astrometric Flat-field}
     2096\label{sec:astro.flat}
    18832097
    18842098After correction for both KE and DCR, we observe persistent residual
    18852099astrometric deviations which depend on the position in the camera.  We
    18862100construct an astrometric ``flat-field'' response by determining the
    1887 mean residual displacement in the X and Y (chip) directions as a
     2101mean residual displacement in the $X$ and $Y$ (chip) directions as a
    18882102function of position in the focal plane.  We have measured the
    1889 astrometric flat using a sampling resolution of 40x40 pixels, matching
    1890 the photometric flat-field correction images.
     2103astrometric flat using a sampling resolution of $80 \times 80$ pixels.
    18912104Figures~\ref{fig:astroflat.gri} and \ref{fig:astroflat.zy} show the
    18922105astrometric flat-field images for the five filters \grizy\ in each of
     
    19002113The dominant pattern in the astrometric residual is roughly a series
    19012114of concentric rings. The pattern is similar to the pattern of the
    1902 focal surface residuals measured by \cite{2008SPIE.7014E..0DO}, which also has
    1903 a concentric series of rings with similar spacing.  The ``tent'' in
    1904 the center of the focal surface is reflected in these astrometry
    1905 residual plots.  Our interpretation of the structure is that the
    1906 deviations of the focal plane from the ideal focal surface introduces
    1907 small-scale PSF changes, presumably coupled to the optical
     2115focal surface residuals measured by \cite{2008SPIE.7014E..0DO}, which
     2116also has a concentric series of rings with similar spacing.  The
     2117``tent'' in the center of the focal surface is reflected in these
     2118astrometry residual plots.  Our interpretation of the structure is
     2119that the deviations of the focal plane from the ideal focal surface
     2120introduces small-scale PSF changes, presumably coupled to the optical
    19082121aberrations, which result in small changes in the centroid of the
    19092122object relative to the PSF model at that location.  Since the PSF
    1910 model shape parameters are only able to vary at the level of a 6x6
    1911 grid per chips, the finer structures are not included in the PSF
    1912 model.  The PV2 analysis shows the ring structure more clearly, with a
    1913 pattern much more closely following the focal surface deviations.  In
    1914 the PV2 analysis, the PSF model used at most a 3x3 grid per chip to
    1915 follow the shape variations, so any changes caused by the optical
    1916 aberrations would be less well modeled in the PV2 analysis, as we
    1917 observe.
     2123model shape parameters are only able to vary at the level of a $6
     2124\times 6$ grid per chips, the finer structures are not included in the
     2125PSF model.
     2126
     2127The PV2 analysis shows this circular pattern more clearly than the PV3
     2128analysis, with a pattern much more closely following the focal surface
     2129deviations.  In the PV2 analysis, the PSF model used at most a
     2130$3\times 3$ grid per chip to follow the shape variations, so any
     2131changes caused by the optical aberrations would be less well modeled
     2132in the PV2 analysis than the PV3 analysis.  For PV3, some of these
     2133patterns are suppressed by the higher-resolution PSF model.
    19182134
    19192135A second pattern which is weakly seen in several chips consists of
    1920 consistent displacements in the X (serial) direction for certain
    1921 cells.  This effect can be seen most clearly in chips XY45 and XY46.
     2136consistent displacements in the $X$ (serial) direction for certain
     2137cells.  This effect can be seen most clearly in chips OTA45 and OTA46.
    19222138In the PV2 analysis, this pattern is also more clearly seen.  In this
    19232139case, the fact that the astrometric model used polynomials with a
     
    19292145of this is unclear, but likely caused by the astrometry model failing
    19302146to follow the underlying variations because of the need to extrapolate
    1931 to the edge pixels.  Finally, we also mention an interesting effect
     2147to the edge pixels.
     2148
     2149Finally, we also mention an interesting effect
    19322150{\em not} visible at the resolution of these astrometric flat-field
    19332151images.  Fine structures are observed at the \approx 10 pixel scale
     
    19492167average solution, resulting in residual astrometric structures.  The
    19502168gradient of the astrometric displacement results in an apparent
    1951 expansion or compression of the pixel sizes, resulting in a signal
     2169expansion or compression of the pixel sizes, generating a signal
    19522170which can be observed in the flat-field images.  For GPC1, unlike the
    19532171DES detectors, the amplitude of these flat-field variations are much
    19542172smaller than the photometric variations caused by the changing PSF
    1955 sized, caused in turn by varying electron diffusion rates.  These
     2173sizez, caused in turn by varying electron diffusion rates.  These
    19562174features, and the related vertical electron diffusion variations are
    19572175discussed in detail in \cite{2018PASP..130f5002M}.
    19582176
    1959 Unfortunately, we discovered a problem with the astrometric flat-field
    1960 correction too late to be repaired for DR1.  As can be seen by
    1961 inspection of Figures~\ref{fig:astroflat.gri} and
    1962 \ref{fig:astroflat.zy}, there is significant pixel-to-pixel noise in
    1963 the the astrometric flat-field images.  This pixel-to-pixel noise is
    1964 caused by too few stars used in the measurement of the flat-field
    1965 structure for the high-resolution sampling.  As a result, the
    1966 astrometric flat-field correction reduces systematic structures on
    1967 large spatial scales, but at the expense of degrading the quality of
    1968 an individual measurement.  Only $i$-band has sufficient
    1969 signal-to-noise per pixel to avoid significantly increasing the
    1970 per-measurement position errors. 
     2177% generate (or plot) astrometric flat-field images for DR2 (PV3.X)
     2178
     2179\begin{figure*}[htbp]
     2180  \begin{center}
     2181  \includegraphics[width=\hsize,clip]{{pics/astroflat.repair}.png}
     2182  \caption{\label{fig:astroflat.repair} Comparison of the
     2183    high-resolution astrometric flat-field images used for PV3.2
     2184    (left) and for PV3.3 (right).  These examples show the \gps-band
     2185    astrometric flat-field corrections for the $X$ direction as seen
     2186    in the focal plane coordinate system.  Note the elevated noise in
     2187    the PV3.2 image due to insufficient numbers of stars used in the analysis.
     2188}
     2189\end{center}
     2190\end{figure*}
     2191
     2192% numbers of stars used:
     2193%% mana: load.stars astroflat.20151205/astroflat.20151205.v1.Npt.fits
     2194%% filter g : 2591421 stars
     2195%% filter r : 3497036 stars
     2196%% filter i : 16241986 stars
     2197%% filter z : 7153595 stars
     2198%% filter y : 4509749 stars
     2199%% mana: load.stars astroflat.20170217/astroflat.20170217.Npt.fits
     2200%% filter g : 17590560 stars
     2201%% filter r : 31000135 stars
     2202%% filter i : 82648850 stars
     2203%% filter z : 62166619 stars
     2204%% filter y : 42867074 stars
     2205
     2206\note{move the discussion of the DR1 & DR2 scatter to the end of the
     2207  astrom section?}
    19712208
    19722209Figure~\ref{fig:allsky.astrom.sigma} shows the standard deviations of
    19732210the mean residual astrometry in $(\alpha,\delta)$ for bright stars as
    1974 a function of position across the sky.  For each pixel in these
    1975 images, we selected all objects with $15 < i < 17$, with at least 3
    1976 measurements in $i$-band (to reject artifacts detected in a pair of
    1977 exposures from the same night), with \code{PSF_QF} $> 0.85$ (to reject
    1978 excessively-masked objects), and with $mag_{\rm PSF} - mag_{\rm Kron}
    1979 < 0.1$ (to reject galaxies).  We then generated histograms of the
    1980 difference between the object position predicted for the epoch of each
    1981 measurement (based on the proper motion and parallax fit) and the
    1982 observed position of that measurement, in both the Right Ascension and
    1983 Declination directions (in linear arcseconds), for all stars in a
    1984 given pixel in the images.  From these residual histograms, we can
    1985 then determine the median and the 68\%-ile range to calculate a robust
    1986 version of the standard deviation.  This represents the bright-end
    1987 systematic error floor for a measurement from a single exposure.  The
    1988 standard deviations are then plotted in
     2211a function of position across the sky based on the DR2 calibration.  For each
     2212pixel in these images, we selected all objects with $15 < i < 17$,
     2213with at least 3 measurements in $i$-band (to reject artifacts detected
     2214in a pair of exposures from the same night), with \code{PSF_QF} $>
     22150.85$ (to reject excessively-masked objects), and with $mag_{\rm PSF}
     2216- mag_{\rm Kron} < 0.1$ (to reject galaxies).  We then generated
     2217histograms of the difference between the object position predicted for
     2218the epoch of each measurement (based on the proper motion and parallax
     2219fit) and the observed position of that measurement, in both the Right
     2220Ascension and Declination directions (in linear arcseconds), for all
     2221stars in a given pixel in the images.  From these residual histograms,
     2222we can then determine the median and the 68\%-ile range to calculate a
     2223robust version of the standard deviation.  This represents the
     2224bright-end systematic error floor for a measurement from a single
     2225exposure.  The standard deviations are then plotted in
    19892226Figure~\ref{fig:allsky.photom.sigma}.  The median value of the
    1990 standard deviations across the sky is $(\sigma_\alpha, \sigma_\delta)
    1991 = (22, 23)$ milliarcseconds.
     2227standard deviations across the sky in both $(\sigma_\alpha,
     2228\sigma_\delta)$ is 16 milliarcseconds.
    19922229
    19932230The Galactic plane is clearly apparently in these images.  Like
    19942231photometry, we attribute this to failure of the PSF fitting due to
    19952232crowding.  The celestial North pole regions have somewhat elevated
    1996 errors in both R.A. and DEC.  This may be due to the larger typical
    1997 seeing at these high airmass regions, but without further exploration
    1998 this interpretation is uncertain.  Several features can be seen which
    1999 appear to be an effect of the tie to the Gaia astrometry: the stripes
    2000 near the center of the DEC image and the right side of the R.A. image.
    2001 The mesh of circular outlines is due to the outer edge of the focal
    2002 plane where the astrometric calibration is poorly determined.  As
    2003 discussed above, the median values in the images are higher than
    2004 expected based on our PV2 analysis of the astrometry: the median
    2005 per-measurement error floor of \approx 22 mas is significantly worse
    2006 than the \approx 17 mas value in that earlier analysis.  We attribute
    2007 this degradation to the noise introduced by the astrometric
    2008 flat-field.  This noise has been addressed for the DR2 release
    2009 of the individual measurement data.
    2010 
    2011 \begin{figure}[htbp]
     2233errors in both R.A. and DEC, with some specifc structures.  Some of
     2234these structures may be due to the larger typical seeing at these high
     2235airmass regions, but some are due to astrometric failures which stem
     2236from the reference catalog based on the PV2 analysis (see
     2237Section~\ref{sec:pole.problems} for further details).  Several
     2238features can be seen which appear to be an effect of the tie to the
     2239Gaia astrometry: the stripes near the center of the DEC image and the
     2240right side of the R.A. image.  The mesh of circular outlines one the 2
     2241degree scale is due to the outer edge of the focal plane where the
     2242astrometric calibration is poorly determined. 
     2243
     2244The DR1 astrometric calibration suffered from degraded astrometry due
     2245to a problem with the astrometric flat-field correction identified too
     2246late to be repaired for DR1.
     2247%
     2248The astrometric flat-field images used
     2249for that release had too few stars to measure the correction with
     2250sufficient signal-to-noise.  As a result, those corrections had
     2251significant pixel-to-pixel noise which can be seen in
     2252Figure~\ref{fig:astroflat.repair}.  As a result, the astrometric
     2253flat-field correction reduces systematic structures on large spatial
     2254scales, but at the expense of degrading the quality of individual
     2255measurements.  Only the $i$-band flat had sufficient signal-to-noise
     2256per pixel to avoid significantly increasing the per-measurement
     2257position errors.
     2258
     2259For DR2, we recalculated the astrometric flat-field correction using
     2260many more stars.  For the DR1 release, the number of stars per filter
     2261was (\grizy) = (2.6M, 3.5M, 16M, 7M, 4.5M), while for the DR2 release,
     2262the number of stars per filter was (\grizy) = (18M, 31M, 83M, 62M,
     226343M).  We also reduced the resolution of the astrometric flat-field,
     2264using $80 \times 80$ superpixels, rather than the $40 \times 40$
     2265superpixels used for DR1.  Because of the degraded astrometric
     2266flat-field correction, the median per-measurement error floor of DR1
     2267is \approx 22 mas, significantly worse than both DR2 and the earlier
     2268PV2 analysis.  Figure~\ref{fig:allsky.astro.histogram} shows
     2269histograms of the astrometric residual scatter across the sky for DR1
     2270and DR2, illustrating the improvement.
     2271
     2272\begin{figure*}[htbp]
    20122273  \begin{center}
    2013  \includegraphics[width=\hsize,clip]{{pics/allsky.astrom.sigma}.png}
    2014   \caption{\label{fig:allsky.astrom.sigma} Consistency of photometry
     2274  \includegraphics[width=\hsize,clip]{{pics/allsky.histogram.astrom.compare}.png}
     2275  \caption{\label{fig:allsky.astro.histogram} Illustration of the
     2276    impact of the astrometric flat-field correction used for PV3.2 vs
     2277    PV3.3.  The blue histograms show the distribution of astrometric
     2278    residuals for bright stars from the PV3.2 analysis while the red
     2279    histograms show the distribution for the PV3.3 analysis.  The
     2280    median standard deviation for PV3.2 is 22 milliarcseconds in R.A.
     2281    (23 mas in Declination).  Using the higher signal-to-noise
     2282    flat-field correction images reduces the median values to 16 mas
     2283    for both R.A. and Declination directions in PV3.3.
     2284}
     2285\end{center}
     2286\end{figure*}
     2287
     2288% older version of this figure:
     2289% pv2_0 : /data/ipp060.0/eugene/pv2.astrom.20150126/astromap.20150127/dDsig.im.fits
     2290% pv2_1 : /data/ipp060.0/eugene/pv2.astrom.20150126/astromap.20150429/dDsig.im.fits
     2291
     2292% NOTE:
     2293% the pv2 versions used:  resize 1800 920; region 0 0 85 ait
     2294% the pv3 versions used:  resize 1800 950; region 180 0 90 ait
     2295
     2296% thus we cannot directly compare map pixels, without re-extracting the measurements
     2297% (we can do that if we decide it is needed to generate the best plots)
     2298
     2299% original version of figure: pv3.stats.20161202/allsky.astrom.sigma.png
     2300%   based on /data/kukui.3/eugene/pv3.stats.20161202/maps.measure/pv3.v1.*.sigma.fits
     2301%   based on /data/ipp094.0/eugene/pv3.stats.20161202/cdhist.measure/cdmerge.v1.dD.fits (& dR)
     2302%   plot script /data/kukui.3/eugene/pv3.stats.20161202/scatter.sh
     2303%   catdir /data/ipp094.0/eugene/pv3.cam.20150607/catdir.master (PV3.2)
     2304
     2305% regenerate using fits image in pv3.stats.20170413
     2306
     2307\begin{figure*}[htbp]
     2308  \begin{center}
     2309 \includegraphics[width=\hsize,clip]{{pics/allsky.astrom.pv3.3}.png}
     2310  \caption{\label{fig:allsky.astrom.sigma} Consistency of astrometry
    20152311    measurements across the sky.  Each panel shows a map of the
    20162312    standard deviation of astrometry residuals for stars in each
     
    20212317    is likely responsible for these elevated value. }
    20222318  \end{center}
    2023 \end{figure}
    2024 
    2025 % plot of the astrometric error floor per filter?
    2026 
    2027 % \note{SECTION or REF?}.
     2319\end{figure*}
    20282320
    20292321After the initial analysis to measure the KE corrections, DCR
    20302322corrections, and astrometric flat-field corrections, we applied these
    20312323corrections to the entire database.  Within the schema of the
    2032 database, each measurement has the raw chip coordinates
    2033 (\code{Measure.Xccd,Yccd}) as well as the offset for that object based on each of
    2034 these three corrections: \code{Measure.XoffKH,YoffKH,
    2035   Measure.XoffDCR,YoffDCR, Measure.XoffCAM,YoffCAM}.  The offsets are
    2036 calculated for each measurement based on the observed instrumental
    2037 chip magnitudes and FWHM for the Koppenh\"ofer Effect, on the average
    2038 chip colors and the altitude \& azimuth of each measurement for the
    2039 DCR correction, and on the chip coordinates for the astrometric
    2040 flat-field corrections.  The corrections are combined and applied to
    2041 the raw chip coordinates and saved back in the database in the fields
    2042 \code{Measure.Xfix,Yfix}.  At this point, we are ready to run the
    2043 full astrometric calibration.
    2044 
    2045 \subsection{Galactic Rotation and Solar Motion}
     2324database, each measurement in the \ippdbtable{Measure} table has the
     2325raw chip coordinates (\ippdbcolumn{Xccd}, \ippdbcolumn{Yccd}) as well
     2326as the offset for that object based on each of the three corrections
     2327discussed above (\ippdbcolumn{XoffKH}, \ippdbcolumn{YoffKH};
     2328\ippdbcolumn{XoffDCR}, \ippdbcolumn{YoffDCR}; \ippdbcolumn{XoffCAM},
     2329\ippdbcolumn{YoffCAM}).  The offsets are calculated for each
     2330measurement based on the observed instrumental chip magnitudes and
     2331FWHM for the Koppenh\"ofer Effect, on the average chip colors and the
     2332altitude \& azimuth of each measurement for the DCR correction, and on
     2333the chip coordinates for the astrometric flat-field corrections.  The
     2334corrections are combined and applied to the raw chip coordinates and
     2335saved back in the database in the fields \ippdbcolumn{Xfix},
     2336\ippdbcolumn{Yfix}.  At this point, we are ready to run the full
     2337astrometric calibration.
     2338
     2339\subsection{Absolute Calibration}
     2340\label{sec:galactic.rotation}
    20462341
    20472342The initial analysis of the PV2 astrometry used the 2MASS positions as
     
    21212416where $d$ is the distance and $l,b$ are the Galactic coordinates of the
    21222417star. Note that the proper motion induced by
    2123 %% \note{some reference for this?} 
    21242418the Galactic rotation is independent of distance while the reflex
    21252419motion induced by the solar motion decreases with increasing
     
    21352429value of 500pc. 
    21362430
    2137 %% \note{plots to show how well this worked for PV3 pre Gaia}
    2138 
    21392431\subsection{Gaia Constraint}
     2432
     2433\note{move comparisons to Gaia to the discussion, limit this section
     2434  to the Gaia astrometric tie}
    21402435
    21412436After the full relative astrometry analysis was performed for the PV3
     
    21642459even at a lower weight, helps to tile over those gaps.
    21652460
    2166 %% \note{Figures showing the Gaia residuals}
    2167 
    21682461\begin{figure*}[htbp]
    21692462  \begin{center}
     
    22592552proper motions will obviate the need to correct for the Galactic rotation.
    22602553
    2261 \subsection{Calculation of Object Astrometry}
     2554\subsection{Object Astrometry}
     2555
     2556After the image astrometric parameters have been determined and
     2557applied to the measurements from each image, we attempt to find the
     2558best astrometric parameters (position, parallax and proper motions)
     2559for all objects in the database.  Only good quality measurements are
     2560kept for the astrometric analysis: PS1 chip detections with
     2561\code{PSF_QF} $< 0.85$ are rejected, as are any detections for which
     2562the magnitude or magnitude error were reported as \code{NAN}.  Only
     2563PS1 \ippstage{chip}-stage measurements were used for the astrometry
     2564measurement (no stack or forced-warp measurements).  If available, the
     25652MASS and Gaia astrometry for an object was also used in the
     2566calculation of the astrometry.  Measurements which were kept for the
     2567astrometric fit for an object were marked with the bit-flags
     2568\code{ID_MEAS_USED_OBJ}.  Some detections were identified as extreme
     2569outliers if their position deviated from the mean object coordinate by
     2570more than 2 arcseconds.  These detections were ignored and marked with
     2571the bit flag \code{ID_MEAS_POOR_ASTROM}.
     2572
     2573If 2MASS or Gaia astrometry measurements
     2574were available for an object, {\em all} measurements for that object
     2575are marked with the bit-flag \code{ID_MEAS_OBJECT_HAS_2MASS} or
     2576\code{ID_MEAS_OBJECT_HAS_GaIA} as appropriate.  The Tycho 2.0
     2577measurements were not included in this analysis and objects with Tycho
     2578measurements are therefore not marked.
    22622579
    22632580\subsubsection{Iteratively Reweighted Least Squares Fitting}
    2264 
    2265 After the image astrometric parameters have been determined and
    2266 applied to the measurements from each image, we attempt to find
    2267 the best astrometric parameters (position, parallax and proper
    2268 motions) for all objects in the database.  We require a minimum of 5
    2269 detections and 1 year of data for any object in order for it to be
    2270 fitted for just proper motion.  For a parallax and proper-motion fit,
    2271 we require at least 7 detections, 1 year of data, and a parallax
    2272 factor range of at least 0.25; no object is fitted to parallax without
    2273 proper motion as well.  If an object is fitted for parallax, it is
    2274 also fitted with a model including only proper motion and only a mean
    2275 position.  The chisq for all three fits is saved.  Currently, the
    2276 highest order fit allowed is saved in the database, regardless of the
    2277 significance of the improvement in adding parameters.  The resulting
    2278 parallax and proper motion measurements are inserted back into the DVO
    2279 database for use by science queries.
    22802581
    22812582With an automatic process applied to hundreds of millions of stars, it
     
    23392640fractional change is less than some tolerance ($10^{-4}$), then
    23402641iterations are halted and the last fitted parameters are used.  If
    2341 convergence is not reached in 10 iterations, the process is halted in
    2342 any case and a flag raised for the object to note that IRLS did not
    2343 converge.
    2344 
    2345 % \note{did this happen for any of our targets?}
     2642convergence is not reached in 10 iterations, the process is halted and
     2643the analysis is rejected. 
    23462644
    23472645To calculate a fit $\chi^2$ value and to determine an appropriate set
     
    23542652either used to calculate both RA and Declination terms, or neither).
    23552653The $\chi^2$ is determined from the unclipped points in the standard
    2356 way.  Bootstrap analysis is used to assess the errors on the fit
     2654way.  These measurements are marked with the bit flag
     2655\code{ID_MEAS_UNMASKED_ASTRO}.
     2656
     2657Bootstrap-resampling analysis is used to assess the errors on the fit
    23572658parameters: A number of measurements equal to the number of unclipped
    23582659data points are randomly selected from the set of unclipped data
     
    23602661then used to fit for the astrometric parameters, using ordinary least
    23612662squares fitting.  The parameters are recorded and the process re-run
    2362 100 times.  For each astrometric parameter, the error is determined as
     2663300 times.  For each astrometric parameter, the error is determined as
    23632664half of the 68\% confidence range for the distribution of fitted
    23642665parameter values.
    23652666
     2667\subsubsection{Object Astrometry Flags}
     2668
     2669We require a minimum of 5 detections and 1 year of data for any object
     2670in order for it to be fitted for just proper motion.  For a parallax
     2671and proper-motion fit, we require at least 7 detections, 1 year of
     2672data, and a parallax factor range of at least 0.25; no object is
     2673fitted to parallax without proper motion as well.  If an object is
     2674fitted for parallax, it is also fitted with a model including only
     2675proper motion and only a mean position.  The chisq for all three fits
     2676is saved.  Currently, the highest order fit allowed is saved in the
     2677database, regardless of the significance of the improvement in adding
     2678parameters.  The resulting parallax and proper motion measurements are
     2679inserted back into the DVO database for use by science queries.  If
     2680one of the three types of fits were attempted, the corresponding bit
     2681flags are set: \code{ID_OBJ_FIT_PAR} for the full parallax fit,
     2682\code{ID_OBJ_FIT_PM} for the proper-motion fit, \code{ID_OBJ_FIT_AVE}
     2683for the mean position.  The fit which was used to provide the reported
     2684astrometric parameters is noted with one of the three object bit
     2685flags: \code{ID_OBJ_USE_PAR}, \code{ID_OBJ_USE_PM},
     2686\code{ID_OBJ_USE_AVE}.  If the IRLS analysis for all three types of
     2687fits fails to converge, the raw weighted average position is reported
     2688and the bit flag \code{ID_OBJ_RAW_AVE} is set.  If the proper-motion
     2689model was attempted and failed, the bit flag \code{ID_OBJ_BAD_PM} is
     2690set.
     2691
     2692Objects for which there is no valid chip-stage measurement (\eg.,
     2693faint sources below the single-exposure detection limit) will use the
     2694position from the stack for the mean position.  In this case, the bit
     2695flag \code{ID_OBJ_STACK_FOR_MEAN} will be raised.  Stack astrometry is
     2696reported to the PSPS database.  The stack astrometry is calculated
     2697based on the median of stack measurements.  The stack measurements are
     2698not statistically independent (see Section~\ref{sec:stack.phot}), so
     2699there an average of the stack measurements does not improve the
     2700statistical significance of the position measurement.  In addition,
     2701the stack astrometry is expected to be degraded relative to the
     2702chip-stage astrometry, in part because of the geometric re-warping
     2703required to generate the stack images and in part because of the
     2704spatially variable stack PSFs.  If stack measurements exist but for
     2705some reason cannot be used for astrometry (\eg., poor quality) the
     2706values reported to the PSPS database will be derived from the average
     2707of the chip detections and the bit flag \code{ID_OBJ_MEAN_FOR_STACK}
     2708will be set for the object.
     2709
    23662710\section{Discussion}
     2711\label{sec:discussion}
     2712
     2713The calibration of the PV3 DVO database required several iterations.
     2714For completeness, we discuss these steps and their implications for
     2715the DR1 and DR2 releases.
     2716\begin{itemize}
     2717
     2718\item[PV3.0] The first calibrated PV3 database is identified as PV3.0.
     2719  This calibration predates the Gaia DR1 release and uses the 2MASS
     2720  catalog as a reference.  After internal testing, an error in the
     2721  photometry calibration was identified in this DVO version: the
     2722  high-resolution photometric flat-field correction measured using the
     2723  stellar photometry (see Section~\ref{sec:phot.flat}) was applied
     2724  with the wrong sign to the measurements.
     2725
     2726\item[PV3.1] After the above error was identified, the photometric
     2727  flat-field correction was applied in the correct sense to the
     2728  measurements and the average photometry was recalculated.  The
     2729  resulting PV3.1 version of the database was used for the DR1 release
     2730  (but see below regarding the mean positions).
     2731
     2732\item[PV3.2] The Gaia DR1 release motivated a recalibration of the
     2733  astrometry using the Gaia DR1 position information, combined with
     2734  photometric distance estimates and a model for the Galactic and
     2735  Solar motion to correct the absolute proper motion (see
     2736  Section~\ref{sec:galactic.rotation}).  We identify the resulting
     2737  database as PV3.1.  This database was used to generate the positions
     2738  in the \ippdbtable{gaiaObject} table, which are exposed in the DR1
     2739  release.
     2740
     2741\item[PV3.3] After the DR1 release, we identified a problem with the
     2742  astrometric flat-field corrections (see
     2743  Section~\ref{sec:astro.flat}): for all but the \ips\ filter, the
     2744  analysis of the flat-field used too few stars.  The measurement of
     2745  the systematic astrometric corrections therefore had a low
     2746  signal-to-noise.  Instead of reducing the scatter in the astrometric
     2747  measurements, the application of these flat-fields {\em increased}
     2748  the scatter.  Recognizing this error, we re-measured the astrometric
     2749  flat-fields with a larger number of stars and applied the improve
     2750  versions to the database.  The resulting PV3.3 calibration has a
     2751  noticable improvement in the astrometric scatter for bright stars.
     2752
     2753\item[PV3.4] Two errors were identified in the PV3.3 calibration
     2754  before the DR2 release was completed.  First, we discovered that the
     2755  repair applied to the photometric flat-field correction for PV3.1,
     2756  reversing the sign of the correction, was not propagated to the
     2757  stack or warp photometry calibrations.  Although the measurements
     2758  from these stages are not corrected by those flat-fields, they are
     2759  affected by this calibration since they are tied to the average of
     2760  the chip-stage measurements.  Second, we determined that the
     2761  aperture-like photometry (e.g., Kron magnitudes) and photomety
     2762  which depends on the PSF model for the stack measurements need to be
     2763  independently tied to the average exposure photometry (see
     2764  discussion in Section~\ref{sec:phot.flat}).  We addressed both of
     2765  these issue in the PV3.4 calibration of the DVO database.  This
     2766  database was then used to generate the values in the DR2 PSPS
     2767  database tables.  \note{what about P2, those were done first, right?}
     2768\end{itemize}
     2769
     2770\begin{figure*}[htbp]
     2771  \begin{center}
     2772  \includegraphics[width=\hsize,clip]{{pics/photom.pv3.3v4}.png}
     2773  \caption{\label{fig:photom.pv3.3v4} Sample comparison of PV3.3 and
     2774    PV3.4 photometry illustrating the impact of the issues identified
     2775    in the PV3.3 stack and warp photometry.  All figures use \ips-band
     2776    photometry.  The left panels use data from PV3.3 while the right
     2777    use PV3.4.  The top row shows the mean difference between the
     2778    average photometry from individual exposures (``chip'') and the
     2779    stack photometry using Kron magnitudes.  The middle row shows the
     2780    mean difference between the average photometry from individual
     2781    exposures (``chip'') and the average forced-warp photometry, again
     2782    using Kron magnitudes.  The bottom row shows the mean difference
     2783    between the average photometry from individual exposures
     2784    (``chip'') and the average forced-warp photometry, using PSF
     2785    magnitudes.  See Section~\ref{sec:discussion} for a description of
     2786    the calibration change in PV3.4.}
     2787\end{center}
     2788\end{figure*}
    23672789
    23682790\section{Conclusion}
     
    23862808Lorand University (ELTE) and the Los Alamos National Laboratory.
    23872809
     2810\note{colormaps by Peter Kovesi. Good Colour Maps: How to Design Them.
     2811arXiv:1509.03700 [cs.GR] 2015.  add ref}
     2812
     2813
     2814
    23882815\bibliographystyle{apj}
    23892816% \bibliography{lib}{}
     
    24162843    * kh.data.20151203.v1/spline.final.fits : spline fits to the KH data
    24172844    * kh.data.20151203.v1.fits : densify images of residuals per chip : (dX,dY) & (T0, T1) = (pre fix, post fix)
    2418     * mana.sh : kh.example - plot of XY04
     2845    * mana.sh : kh.example - plot of OTA04
    24192846    * mana.sh : khmap (needs cleanup)
    24202847  * ipp094:/data/ipp094.0/eugene/pv3.cam.20150607/astrom.corrections : extractions and original scripts to make spline, etc
Note: See TracChangeset for help on using the changeset viewer.