Context Navigation

-              r40081
+              r40483
 \begin{abstract}
 The Pan-STARRS1 Science Consortium have carried out a set of imaging
 surveys using the 1.4 giga-pixel GPC1 camera on the PS1 telescope.  As
+The Pan-STARRS1 Science Consortium has carried out a set of imaging
+surveys using the 1.4 gigapixel GPC1 camera on the PS1 telescope.  As
 this camera is composed of many individual electronic readouts, and
 covers a very large field of view, great care was taken to ensure that
 …
 the data reduction techniques and the resulting data products. This paper (Paper III)
 describes the details of the pixel processing algorithms, including
 detrending, warping, and adding (to create stacked images) and subtracting
 (to create difference images) and resulting image products and their
+detrending, warping, adding (to create stacked images), and subtracting
+(to create difference images), along with the resulting image products and their
 properties.
 \citet[][Paper I]{chambers2017} provides an overview of the Pan-STARRS System, the
 …
 %Magnier et al. 2017 (Paper II)
 %Pan-STARRS Data Processing Stages
 \citet[][Paper II]{magnier2017a}
+\citet[][Paper II]{magnier2017.datasystem}
 describes how the various data processing stages are organized and
 implemented
 …
 %Magnier et al. 2017 (Paper IV)
 %Pan-STARRS Pixel Analysis : Source Detection
 \citet[][Paper IV]{magnier2017b}
+\citet[][Paper IV]{magnier2017.analysis}
 describes the details of the source detection and photometry, including
 point-spread-function and extended source fitting models, and the
 …
 %Magnier et al. 2017 (Paper V)
 %Pan-STARRS Photometric and Astrometric Calibration
 \citet[][Paper V]{magnier2017c}
+\citet[][Paper V]{magnier2017.calibration}
 describes the final calibration process, and the resulting photometric and
 astrometric quality.
 …
 The Pan-STARRS 1 Science Survey uses the 1.4 giga-pixel GPC1 camera
+The Pan-STARRS 1 Science Survey uses the 1.4 gigapixel GPC1 camera
 with the PS1 telescope on Haleakala Maui to image the sky north of
 $-30^\circ$ declination.  The GPC1 camera is composed of 60 orthogonal
 transfer array (OTA) devices, each of which is an $8\times{}8$ grid of
+readout cells.  This parallelizes the readout process, reducing the
+overhead in each exposure.  However, as a consequence of this large
+number of individual detector readouts, many calibrations are needed
+to ensure the response is consistent across the entire field of view.
+The Processing Version 3 (PV3) reduction represents the third full
+reduction of the Pan-STARRS archival data.  The first two reductions
+were used internally for pipeline optimization and the development of
+the initial photometric and astrometric reference catalog \citep{magnier2017c}.  The
+products from these reductions were not publicly released, but have
+been used to produce a wide range of scientific papers from the
+readout cells.  The large number of cells parallelizes the readout
+process, reducing the overhead in each exposure.  However, as a
+consequence, many calibration operations are needed to ensure the
+response is consistent across the entire seven square degree field of
+view.
+%The Processing Version 3 (PV3) reduction represents the third full
+DR1 contains the results of the third full reduction of the Pan-STARRS
+archival data.  The first two reductions were used internally for
+pipeline optimization and the development of the initial photometric
+and astrometric reference catalog \citep{magnier2017.calibration}.
+The products from these reductions were not publicly released, but
+have been used to produce a wide range of scientific papers from the
 Pan-STARRS 1 Science Consortium members.
 The Pan-STARRS image processing pipeline (IPP) is described elsewhere
+\citep{magnier2017a}, but a short summary follows.  The
+archive of raw exposures is stored on disk, with a database storing
+the metadata of exposure parameters.  For the PV3 processing, large
+contiguous regions were defined, and the images for all exposures
+within that region launched for the \IPPstage{chip} stage processing.
+This stage performs the image detrending (described below in section
+\citep{magnier2017.datasystem}, but a short summary follows.  The raw
+image data is stored on the processing cluster, with a database
+containing the metadata of exposure parameters.  These raw images can
+be launched for the initial \IPPstage{chip} stage processing.  This
+stage performs the image detrending (described below in section
 \ref{sec:detrending}), as well as the single epoch photometry
 \citep{magnier2017b}, in parallel on the individual OTA device data.
+Following the \IPPstage{chip} stage is the \IPPstage{camera} stage, in
+which the astrometry and photometry for the entire exposure is
 calibrated by matching the detections against the reference catalog.
+\citep{magnier2017.analysis}, in parallel on the individual OTA device
+data.  Following the \IPPstage{chip} stage is the \IPPstage{camera}
+stage, in which the astrometry and photometry for the entire exposure
+is calibrated by matching the detections against a reference catalog.
 This stage also performs masking updates based on the now-known
 positions and brightnesses of stars that create dynamic features (see
 …
 \IPPstage{chip} stage images onto common sky oriented images that have
 fixed sky projections (Section \ref{sec:warping}).  When all
 \IPPstage{warp} stage processing is done for the region of the sky,
+\IPPstage{warp} stage processing is done for a region of the sky,
 \IPPstage{stack} processing is performed (Section \ref{sec:stacking})
 to construct deeper, fully populated images from the set of
+\IPPstage{warp} images that cover that region of the sky.  Beyond the
+\IPPstage{stack} stage, a series of additional stages are done that
+are more fully described in other papers.  Transient features are
+identified in the \IPPstage{diff} stage, which takes input
+\IPPstage{warp} and/or \IPPstage{stack} data and performs image
+\IPPstage{warp} images that cover that region of the sky.  Transient
+features are identified in the \IPPstage{diff} stage, which takes
+input \IPPstage{warp} and/or \IPPstage{stack} data and performs image
 differencing (Section \ref{sec:diffs}).  Further photometry is
 performed in the \IPPstage{staticsky} and \IPPstage{skycal} stages,
 which add extended source fitting to the point source photometry of
 objects detected in the \IPPstage{stack} images, and calibrate the
+results against the reference catalog.  The \IPPstage{fullforce} stage
+takes the catalog output of the \IPPstage{skycal} stage, and uses the
 objects detected in that to perform forced photometry on the
+objects detected in the \IPPstage{stack} images, and again calibrate
+the results against a reference catalog.  The \IPPstage{fullforce}
+stage takes the catalog output of the \IPPstage{skycal} stage, and
+uses the objects detected in that to perform forced photometry on the
 individual \IPPstage{warp} stage images.  The details of these stages
+are provided in \citet{magnier2017b}.
+The same reduction procedure described above is also performed in real
+time on new exposures as they are observed by the telescope.  This
+process is largely automatic, with new exposures being downloaded from
+the summit to the main IPP processing cluster at the Maui Research and
+Technology Center in Kihei, and registered into the processing
+database.  This triggers a new \IPPstage{chip} stage reduction for
+science exposures, advancing processing upon completion through to the
+\IPPstage{diff} stage.  This allows the ongoing solar system moving
+object search to identify candidates for follow up observations within
+hours of the initial set of observations \citep{2015IAUGA..2251124W}.
+are provided in \citet{magnier2017.analysis}.
+A limited version of same reduction procedure described above is also
+performed in real time on new exposures as they are observed by the
+telescope.  This process is automatic, with new exposures being
+downloaded from the summit to the main IPP processing cluster at the
+Maui Research and Technology Center in Kihei, and registered into the
+processing database.  New \IPPstage{chip} stage reductions are
+launched for science exposures, advancing processing upon completion
+through to the \IPPstage{diff} stage, skipping the additional stack
+and forced warp photometry stages.  This automatic processing allows
+the ongoing solar system moving object search to identify candidates
+for follow up observations within 24 hours of the initial set of
+observations \citep{2015IAUGA..2251124W}.
 Section \ref{sec:detrending} provides an overview of the detrending
 process that corrects the instrumental signatures of GPC1, with
 details of the construction of those detrends in Section
+\ref{sec:detrend construction}.  An analysis of the algorithms used to
 complete the \IPPstage{warp} (section \ref{sec:warping}),
+details of the construction of the reference detrend templates in
+Section \ref{sec:detrend construction}.  An analysis of the algorithms
+used to perform the \IPPstage{warp} (section \ref{sec:warping}),
 \IPPstage{stack} (section \ref{sec:stacking}), and \IPPstage{diff}
+(section \ref{sec:diffs}) stage transformations of the image data to
+from the detector frame to a common sky frame, and the co-adding of
+those common sky frame images continues after the list of detrend
+steps.  Finally, a discussion of the remaining issues and possible
+future improvements is presented in section \ref{sec:discussion}.
+Image products presented in figures have been
+mosaicked to arrange pixels as follows.  Single cell images are
+arranged such that pixel $(1,1)$ is at the lower left corner.  Images
+mosaicked to the OTA level have cell xy00 in the lower left corner,
+with cells xy10, xy20, etc. sequentially to the right, and cells xy01,
+xy02, etc. sequentially to the top of this cell.  Again, pixel $(1,1)$
+of cell xy00 is located in the lower left corner of the image.  For
+mosaicks of the full field of view, the OTAs are arranged as they see
+the sky.  The lower left corner is the empty location where OTA70
+would exist.  Toward the right, the OTA labels decrease in $X$ label,
+with the empty OTA00 located in the lower right.  The OTA $Y$ labels
+increase upward in the mosaic.  The OTAs to the left of the midplane
+(OTA4Y-OTA7Y) are oriented with cell xy00 and pixel $(1,1)$ to the
+lower left of their position.  Due to the electronic connections of
+(section \ref{sec:diffs}) stage transformations of the image data
+follows after the list of detrend steps.  Finally, a discussion of the
+remaining issues and possible future improvements is presented in
+section \ref{sec:discussion}.
+Image products presented in figures have been mosaicked to arrange
+pixels as follows.  Single cell images are arranged such that pixel
+$(1,1)$ is at the lower right corner (for example Figure
+\ref{fig:burntool images}).  This corrects the parity difference
+between the raw data and the sky.  Images mosaicked to show a full OTA
+detector are arranged as they are on the focal plane (as in Figure
+\ref{fig:dark image}.  The OTAs to the left of the midplane
+(OTA4Y-OTA7Y) are oriented with cell xy00 and pixel $(590,1)$ to the
+lower right of their position.  Due to the electronic connections of
 the OTAs in the focal plane, the OTAs to the right of the midplane
 (OTA0Y-OTA3Y) are rotated 180 degrees, and are oriented with cell xy00
+and pixel $(1,1)$ to the top right of their position.
+and pixel $(590,1)$ to the top left of their position. For mosaics of
+the full field of view, the OTAs are arranged as they see the sky,
+with the cells arranged as in the single OTA images (Figure \ref{fig:optical ghosts}).  The lower left
+corner is the empty location where OTA70 would exist.  Toward the
+right, the OTA labels decrease in $X$ label, with the empty OTA00
+located in the lower right.  The OTA $Y$ labels increase upward in the
+mosaic.
 \textit{Note: These papers are being placed on the arXiv.org to
 …
 level, dark frame subtraction to remove temperature and exposure time
 dependent detector glows, and flat field correction to remove pixel to
+pixel response functions.  We also construct fringe correction for the
+reddest data in the \yps{} filter, to remove the interference patterns that
+arise in that filter due to the variations in the thickness of the
+detector surface.
+These corrections, however, assume that the detector response is
+linear across the full range of values.  This is not universally the
+case with GPC1, and this requires an additional set of detrending
+steps to remove these non-linear responses.  The first of these is the
+\IPPprog{burntool} correction, which removes the persistence trails
+caused by the incomplete transfer of charge along the readout columns.
+This bright-end nonlinearity is generally only evident for the
+brightest stars, as only pixels that are at or beyond the saturation
+point of the detector have this issue.  More widespread is the
+non-linearity at the faint end of the pixel range.  Some readout cells
+and some readout cell edge pixels experience a sag relative to linear
+at low illumination, such that faint pixels appear fainter than
+expected.  The correction to this requires amplifying the pixel values
+in these regions to match the expected model.
+The final non-linear response issue has no good option for correction.
+pixel response functions.  We also perform fringe correction for the
+reddest data in the \yps{} filter to remove the interference patterns
+that arise in that filter due to the variations in the thickness of
+the detector surface.
+These corrections assume that the detector response is linear across
+the full range of values.  This assumption is not universally true for
+GPC1, and an additional set of detrending steps are required to remove
+these artifacts.  The first of these is the \IPPprog{burntool}
+correction, which removes the flux trails left by the incomplete
+transfer of charge along the readout columns.  These trails are
+generally only evident for the brightest stars, as only pixels that
+are at or beyond the saturation point of the detector leave residual
+charge.  More widespread is the non-linearity at the faint end of the
+pixel range.  Some readout cells and some readout cell edge pixels
+experience a sag relative to linear at low illumination, such that
+faint pixels appear fainter than expected.  The correction to this
+requires amplifying the pixel values in these regions to match the
+expected model.
 Large regions of some OTA cells experience significant charge transfer
 issues, making them unusable for science observations.  These regions
 are therefore masked in processing, with these CTE regions making up
 the largest fraction of masked pixels on the detector.  Other regions
+are masked for other regions, such as static bad pixel features or
+temporary readout masking caused by issues in the camera electronics
+that make these regions unreliable.  These all contribute to the
+detector mask, which is augmented in each exposure for dynamic
+are masked for reasons such as static bad pixel features or temporary
+readout masking caused by issues in the camera electronics that make
+these regions unreliable.  These all contribute to the detector mask,
+a 16 bit value which records the reason a pixel is masked based on the
+value added.  This mask is augmented in each exposure for dynamic
 features that are masked based on the astronomical features within the
 field of view.
 For the PV3 processing, all detrending is done by the
+\IPPprog{ppImage} program.  This program applies the detrends to the
+individual cells, and then an OTA level mosaic is constructed for the
+science image, the mask image, and the variance map image.  The single
+epoch photometry is done at this stage as well.  The following
+subsections (\ref{sec:burntool} - \ref{sec:background}) detail these
+detrending steps, presented in the order in which they are applied to
+the individual OTA image data.
+\subsection{Burntool / Persistence effect}
+\label{sec:burntool}
+Pixels that approach the saturation point on GPC1, which varies by
+readout with common values around 60000 DN, cause persistence problems
+on that and subsequent images.  During the read out process of an
+image with such a bright pixel, some of the charge associated with it
+is not fully shifted down the detector column toward the amplifier.
+As a result, this charge remains in the starting cell, and is
+partially collected in subsequent shifts, resulting in a ``burn
+trail'' that extends from the center of the bright source away from
+the amplifier (vertically along the pixel columns toward the top of
+the cell).
+This incomplete charge shifting in nearly full wells continues as each
+row is read out.  This results in a remnant charge being deposited in
+the pixels that the full well was shifted through.  In following
+exposures, this remnant charge leaks out, resulting in a trail that
+extends from the initial location of the bright source on the previous
+image towards the amplifier (vertically down along the pixel column).
+This remnant charge can remain on the detector for up to thirty
+minutes, requiring the locations of these ``burns'' be retained
+between exposures.
+Both of these types of persistence trails are measured and optionally
+repaired via the \IPPprog{burntool} program.  This program does an
+initial scan of the images, and identifies objects with pixel values
+brighter than a conservative threshold of 30000 DN.  The trail from
+the peak of that object is fit with a one-dimensional power law in
+each pixel column above the threshold, based on empirical evidence
+that this is the functional form of this persistence effect.  This
+also matches the expectation that a constant fraction of charge is
+incompletely transferred at each shift beyond the persistence
+threshold.  Once this fit is done, the model can be subtracted from
+the image, and the location of the star is stored in a table along
+with the exposure PONTIME, which denotes the number of seconds since
+the detector was last powered on, and provides an internally consistent
+time scale.
+For subsequent exposures, the table associated with the previous image
+is read in, and after correcting trails from the stars on the new
+image, the positions of the bright stars from the table are used to
+check for remnant trails on the image.  These are fit and subtracted
+using a one-dimensional exponential model, again based on empirical
+studies.  If a significant model is found, then this location is
+retained in the image output table.  If not, the old burn is allowed
+to expire.
+The main concern with this method of correcting the persistence trails
+is that it is based on fits to the raw image data, which may have
+other signal sources not determined by the persistence effect.  The
+presence of other stars or artifacts along the path of the burn can
+result in a poor model to be fit, resulting in either an over- or
+under-subtraction of the persistence burn.  For this reason, the image
+mask is marked with a value indicating that this correction has been
+applied.  These pixels are not fully excluded, but they are marked as
+suspect, which allows them to be excluded from consideration in
+subsequent stages, such as image stacking.
+Another concern is that the cores of very bright stars are deformed by
+this process, as the burntool fitting subtracts flux
+from only one side of the star.  As most stars that result in burns already
+have saturated cores, they are already ignored for the purpose of
+PSF determination and are flagged as saturated by the photometry
+reduction.
+\begin{figure}
+  \centering
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt_trail.png}
+%    \caption{(a)}
+%  \end{subfigure}%
+%  \begin{subfigure}[]{.45\hsize}
+  \end{minipage}%
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt_trail.png}
+%    \caption{(b)}
+%  \end{subfigure}
+  \end{minipage}
+  \caption{Example of a profile cut along the y-axis through a bright star on exposure o5677g0123o OTA11 in cell xy60 (left panel) and on the subsequent exposure o5677g0124o (right panel).  In both figures, the green points show the image corrected with all appropriate detrending steps, but without burntool applied, illustrating the amplitude of the persistence trails.  The red points show the same data after the burntool correction, which reduces the impact of these features.  Both exposures are in the \gps{} filter with exposure times of 43s}
+\end{figure}
+\begin{figure}
+  \centering
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_nobt.png}
+%    \caption{(a)}
+%  \end{subfigure}%
+%  \begin{subfigure}[]{.45\hsize}
+  \end{minipage}%
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_nobt.png}
+%    \caption{(b)}
+%  \end{subfigure}
+  \end{minipage}
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt.png}
+%    \caption{(a)}
+%  \end{subfigure}%
+%  \begin{subfigure}[]{.45\hsize}
+  \end{minipage}%
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt.png}
+%    \caption{(b)}
+%  \end{subfigure}
+  \end{minipage}
+  \caption{Example of OTA11 cell xy60 on exposures o5677g0123o (left) and o5677g0124o (right).  The top panels show the image with all appropriate detrending steps, but without burntool, and the bottom show the same with burntool applied.  There is some slight over subtraction in fitting the initial trail, but the impact of the trail is greatly reduced in both exposures.}
+\end{figure}
+\IPPprog{ppImage} program.  This program applies the detrend
+corrections to the individual cells, and then an OTA-level mosaic is
+constructed for the signal image, the mask image, and the variance map
+image.  The single epoch photometry is done at this stage as well.
+The following subsections (\ref{sec:overscan} - \ref{sec:background})
+detail the detrending process used on GPC1 that are common to other
+detectors.  The GPC1 specific detrending steps are included after,
+explaining these additional steps that remove the instrument signature.
 \subsection{Overscan}
 …
 Each cell on GPC1 has an overscan region that covers the first 34
 columns of each row, and the last 10 rows of each column.  No light
+lands on these pixels, so the image region is trimmed to exclude them.
+Each row has an overscan value subtracted, calculated by finding the
+median value of that row's overscan pixels and then smoothing between
+rows with a three-row boxcar median.
+\subsection{Non-linearity Correction}
+\label{sec:nonlinearity}
+The pixels of GPC1 are not uniformly linear at all flux levels.  In
+particular, at low flux levels, some pixels have a tendency to sag
+relative to the expected linear value.  This effect is most pronounced
+along the edges of the detector cells, although some entire cells show
+evidence of this effect.
+To correct this sag, we studied the flux behavior of a series of flat
+frames for a ramp of exposure times with approximate logarithmically
+equal spacing between 0.01s and 57.04s.  As the exposure time
+increases, the flux on each pixel also increases in what is expected
+to be a linear manner.  Each of these flat exposures in this ramp is
+overscan corrected, and then the median is calculated for each cell,
+as well as for the rows and columns within ten pixels of the edge of
+the science region.  From these median values at each exposure time
+value, we can construct the expected trend by fitting a linear model,
+$f_{region} = G * t_{exp} + B$, to determine the gain, $G$, and the
+bias, $B$, for the region considered.  This fitting was limited to only
+the range of fluxes between 12000 and 38000 counts, as these ranges
+were found to match the linear model well.  This range avoids the
+non-linearity at low fluxes, as well as the possibility of high-flux
+non-linearity effects.
+We store the average flux measurement and deviation from the linear
+fit for each exposure time for all regions on all detector cells in
+the linearity detrend look up tables.  When this is applied to science
+data, these lookup tables are loaded, and a linear interpolation is
+performed to determine the correction needed for the flux in that
+pixel.  This look up is performed for both the row and column of each
+pixel, to allow the edge correction to be applied where applicable,
+and the full cell correction elsewhere.  The average of these two
+values is then applied to the pixel value, reducing the effects of
+pixel nonlinearity.
+This non-linearity effect appears to be stable in time for the
+majority of the detector pixels, with little evident change over the
+survey duration.  However, as the non-linearity is most pronounced at
+the edges of the detector cells, those are the regions where the
+correction is most likely to be incomplete.  Because of this fact,
+most pixels in the static mask with either the DARKMASK or FLATMASK
+bit set are found along these edges.  As the non-linearity correction
+is unable to reliably restore these pixels, they produce inconsistent
+values after the dark and flat have been applied, and are therefore
+rejected.
+\begin{figure}
+  \centering
+  \includegraphics[width=0.9\hsize,angle=0,clip]{images/linearity_XY27_xy16.png}
+  \caption{Example plot of the linearity correction as a fraction of observed flux for OTA27, cell xy16.}
+\end{figure}
+lands on these pixels, so the science region is trimmed to exclude
+them.  Each row has an overscan value subtracted, calculated by
+finding the median value of that row's overscan pixels and then
+smoothing between rows with a three-row boxcar median.
 \subsection{Dark/Bias Subtraction}
 \label{sec:dark}
+The dark model we make for GPC1 considers each pixel individually,
+independent of any neighbors.  To construct this model, we fit a
+multi-dimensional model to the array of input pixels from a randomly
+selected set of 100-150 overscan and non-linearity corrected dark
+frames chosen from a given date range.  The model fits each pixel as a
+function of the exposure time $t_{exp}$ and the detector temperature
+$T_{chip}$ of the input images such that $\mathrm{dark} = a_0 + a_1
+t_{exp} + a_2 T_{chip} t_{exp} + a_3 T_{chip}^2 t_{exp}$.  This
+fitting uses two iterations to produce a clipped fit, rejecting at the
+$3\sigma$ level.  The final coefficients $a_i$ for the dark model are
+stored in the detrend image.  The constant $a_0$ term includes the
+residual bias signal after overscan subtraction, and as such, a
+separate bias subtraction is not necessary.
+The dark current in the GPC1 detectors has significant variations
+across each cell.  The model we make to remove this signal considers
+each pixel individually, independent of any neighbors.  To construct
+this model, we fit a multi-dimensional model to the array of input
+pixels from a randomly selected set of 100-150 overscan and
+non-linearity corrected dark frames chosen from a given date range.
+The model fits each pixel as a function of the exposure time $t_{exp}$
+and the detector temperature $T_{chip}$ of the input images such that
+$\mathrm{dark} = a_0 + a_1 t_{exp} + a_2 T_{chip} t_{exp} + a_3
+T_{chip}^2 t_{exp}$.  This fitting uses two iterations to produce a
+clipped fit, rejecting at the $3\sigma$ level.  The final coefficients
+$a_i$ for the dark model are stored in the detrend image.  The
+constant $a_0$ term includes the residual bias signal after overscan
+subtraction, and as such, a separate bias subtraction is not
+necessary.
 Applying the dark model is simply a matter of calculating the response
 to the exposure time and detector temperature for the image to be
+for the exposure time and detector temperature of the image to be
 corrected, and subtracting the resulting dark signal from the image.
+Figure \ref{fig:dark image} shows the results of the dark subtraction.
 \subsubsection{Time evolution}
 …
 significant drift over the course of multiple months.  Some of the
 changes in the dark can be attributed to changes in the voltage
+settings of the GPC1 controller electronics, but the majority seem to
+be the result of some unknown parameter.  We can separate the dark
+model history of GPC1 into three epochs.  The first epoch covers all
+data taken prior to 2010-01-23.  This epoch used a different header
+keyword for the detector temperature, making data from this epoch
+incompatible with later dark models.
+settings of the GPC1 controller electronics, but the causes of others
+are unknown.  We can separate the dark model history of GPC1 into
+three epochs.  The first epoch covers all data taken prior to
+-01-23.  This epoch used a different header keyword for the
+detector temperature, making data from this epoch incompatible with
+later dark models.  In addition, the temperatures recorded in this
+value were not fully calibrated, making the dark model generated less
+reliable.
 The second epoch covers data between 2010-01-23 and 2011-05-01, and is
 characterized by a largely stable but oscillatory dark solution.
 There are two modes that the dark model switches between apparently at
+The dark model switches between two modes apparently at
 random.  No clear cause has been established for the switching, but
 there are clear differences between the two modes that require the
 …
 After 2011-05-01, the two-mode behavior of the dark disappears, and is
 replaced with a slow observation date dependent drift in the magnitude
+replaced with a slow observation-date-dependent drift in the magnitude
 of the gradient.  This drift is sufficiently slow that we have modeled
+it using three observation date independent dark model for different
+date ranges.  These darks cover the range from 2011-05-01 to
+-08-01, 2011-08-01 to 2011-11-01, and 2011-11-01 and on.  The
+reason for this time evolution is unknown, but as it is correctable
+with a small number of dark models, this does not significantly impact
+detrending.
+it by generating models for different date ranges.  These darks cover
+the range from 2011-05-01 to 2011-08-01, 2011-08-01 to 2011-11-01, and
+-11-01 and on.  The reason for this time evolution is unknown, but
+as it is correctable with a small number of dark models, this does not
+significantly impact detrending.
 \begin{figure}
   \centering
-%  \begin{subfigure}[]{.45\hsize}
   \begin{minipage}{0.45\hsize}
     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_M_OS_NL_XY23_b1.jpg}
-%    \caption{(a)}
-%  \end{subfigure}%
-%  \begin{subfigure}[]{.45\hsize}
   \end{minipage}%
   \begin{minipage}{0.45\hsize}
     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_to_DARK_XY23_b1.jpg}
-%    \caption{(b)}
-%  \end{subfigure}
   \end{minipage}
+  \caption{An example of the dark model application to exposure o5677g0123o, OTA23 (2011-04-26, 43s \gps{} filter).  The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, and the detector non-linearity corrected.  The right panel, shows the same exposure with the dark applied in addition to the processing shown on the left.}
+  \caption{An example of the dark model application to exposure o5677g0123o, OTA23 (2011-04-26, 43s \gps{} filter).  The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, and the detector non-linearity corrected.  The right panel, shows the same exposure with the dark applied in addition to the processing shown on the left, removing the amplifier glows in the cell corners.}
+  \label{fig:dark image}
 \end{figure}
 …
   \centering
   \includegraphics[width=0.9\hsize,angle=0,clip]{images/B_profile_ex.png}
   \caption{Example showing a profile cut across exposure o5676g0195, OTA67 (2011-04-25, 43s \gps{} filter).  The entire first row of cells (xy00-xy07) have had a median calculated along each pixel column on the OTA mosaicked image.  Arbitrary offsets have been applied to shift the curves to not overlap.  The top curve (in purple) shows the initial raw profile, with no dark model applied.  The next curve (in green) shows the smoother profile after applying the correct B-mode dark model.  Applying the incorrect A-mode dark results in the blue curve, which shows a significant increase in gradients across the cells.  The orange curve shows the result of the PATTERN.CONTINUITY correction.  Although this creates a larger gradient across the mosaicked images, it decreases the cell-to-cell level changes.  The final yellow curve shows the final image profile after all detrending and background subtraction, and has not had an offset applied.  The bright source at the cell xy00 to xy01 transition is a result of a large optical ghost, which due to the area covered, increases the median level more than the field stars.}
+  \caption{Example showing a profile cut across exposure o5676g0195, OTA67 (2011-04-25, 43s \gps{} filter).  The entire first row of cells (xy00-xy07) have had a median calculated along each pixel column on the OTA mosaicked image.  Arbitrary offsets have been applied to shift the curves to not overlap.  The top curve (in purple) shows the initial raw profile, with no dark model applied.  The next curve (in green) shows the smoother profile after applying the appropriate B-mode dark model.  Applying the A-mode dark instead results in the blue curve, which shows a significant increase in gradients across the cells.  The orange curve shows the result of the PATTERN.CONTINUITY correction.  Although this creates a larger gradient across the mosaicked images, it decreases the cell-to-cell level changes.  The final yellow curve shows the final image profile after all detrending and background subtraction, and has not had an offset applied.  The bright source at the cell xy00 to xy01 transition is a result of a large optical ghost, which due to the area covered, increases the median level more than the field stars.}
   \label{fig:dark switching}
 \end{figure}
 …
 \label{sec:video_darks}
+The dark signal is stronger in cell corners due to glow from the
+read-out amplifiers.  The standard dark model corrects this for most
+observations.  However, as mentioned above, when a cell is repeatedly
+read in video mode, the dark model for the OTA containing it changes.
+Surprisingly, added reads for the video cell do not amplify the
+amplifier glow, but rather decrease the dark signal in these regions.
+As a result, using the standard dark model on the data for these OTAs
+results in oversubtraction of the corner glow.
+Video darks have been constructed to eliminate the effect this
+observational change has on the final image quality.  This was done by
+running the standard dark construction process on a series of dark
+frames that have had the video signal enabled for some cells.  GPC1
+can only run video signals on a subset of the OTAs at a given time.
+This requires two passes to enable the video signal across the full
+set of OTAs that support video cells.  This is convenient for the
+process of creating darks, as those OTAs that do not have video
+signals enabled create standard dark models, while the video dark is
+created for those that do.
+Individual cells on GPC1 can be repeatedly read to create a video
+signal used for telescope guiding.  However, when a cell is used for
+this purpose, the dark signal for the entire OTA is changed.  The most
+noticeable feature of this change is that the glows in cell corners
+caused by the read-out amplifiers are suppressed.  As a result, using
+the standard dark model on the data for these OTAs results in
+oversubtraction of the corner glow.
+To generate a correction for this change, a set of video dark models
+were created by running the standard dark construction process on a
+series of dark frames that had the video signal enabled for some
+cells.  GPC1 can only run video signals on a subset of the OTAs at a
+given time.  This requires two passes to enable the video signal
+across the full set of OTAs that support video cells.  This is
+convenient for the process of creating darks, as those OTAs that do
+not have video signals enabled create standard dark models, while the
+video dark is created for those that do.
 This simultaneous construction of video and standard dark models is
+useful, as it provides the ability to isolate the response on the
+standard dark from the video signals.  Isolating this response is
+essential for attempting to create archival video darks.  We only have
+raw video dark frame data after 2012-05-16, when this problem was
+initially identified, so any data prior to that can not be directly
+corrected for the video dark signal.  Isolating the video signal
+response allows linear corrections to the pre-existing standard dark
+models for archival data.  Testing this shows that constructing a
+video dark for older data simply as $VD_{2009} = D_{2009} - D_{Modern}
++ VD_{Modern}$ produces a satisfactory result that does not
+over subtract the amplifier glow.  This is shown in figure
+\ref{fig:video_darks}, which shows video cells from before 2012-05-16,
+corrected with both the standard and video darks, with the early video
+dark constructed in such a manner.
+useful, as it provides a way to isolate the response on the standard
+dark from the video signals.  If the standard and video dark signals
+are separable, then archival video darks can be constructed by
+applying the video dark response to the previously constructed dark
+models.  Raw video dark frame data only exists after 2012-05-16, when
+this problem was initially identified, so any data prior to that can
+not be directly corrected for the video dark signal.  Testing the
+separability shows that constructing a video dark for older data
+simply as $VD_{Old} = D_{Old} - D_{Modern} + VD_{Modern}$ produces a
+satisfactory result that does not over subtract the amplifier glow.
+This is shown in figure \ref{fig:video_darks}, which shows video cells
+from before 2012-05-16, corrected with both the standard and video
+darks, with the early video dark constructed in such a manner.
 \begin{figure}
   \centering
-%  \begin{subfigure}[]{.45\hsize}
   \begin{minipage}{0.45\hsize}
     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_Rdark_XY22_b1.jpg}
-%    \caption{(a)}
-%  \end{subfigure}%
-%  \begin{subfigure}[]{.45\hsize}
   \end{minipage}%
   \begin{minipage}{0.45\hsize}
     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_VDdark_XY22_b1.jpg}
-%    \caption{(b)}
-%  \end{subfigure}
   \end{minipage}
   \caption{An example of the video dark model application to exposure o5677g0123o, OTA22 (2011-04-26, 43s \gps{} filter), which has a video cell located in cell xy16.  The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, the detector non-linearity corrected, and a regular dark applied.  The right panel, shows the same exposure with a video dark applied instead of the standard dark.  The main impact of this change is the improved correction of the corner glows, which are over subtracted with the standard dark.}
 …
 Based on a study of the positional dependence of all detected sources,
 we have discovered that the cells in GPC1 do not have uniform noise
+we discovered that the cells in GPC1 do not have uniform noise
 characteristics.  Instead, there is a gradient along the pixel rows,
 with the noise generally higher away from the read out amplifier
 (higher cell x pixel positions).  This is likely an effect of the
+row-by-row bias issue discussed below.  This gradient causes the read
+noise to increase as the row is read out.  As a result of this
+row-by-row bias issue discussed below.  As a result of this
 increased noise, more sources are detected in the higher noise regions
+when the read noise is assumed constant across the readout.  Read noise is the
+To
+mitigate this noise gradient, we constructed an initial set of
+noisemap images by measuring the median variance on bias frames.  The
+variance is calculated in boxes of 20x20 pixels, and then linearly
+interpolated to cover the full image.
+when the read noise is assumed constant across the readout.
+To mitigate this noise gradient, we constructed an initial set of
+noisemap images by measuring the median variance on bias frames
+processed as science images.  The variance is calculated in boxes of
+x20 pixels, and then linearly interpolated to cover the full image.
 Unfortunately, due to correlations within this noise, the variance
 …
 contaminating the final object catalogs.
+In the detection process, we expect false positives at a rate equal to
+the one-tailed probability beyond the detection threshold.  For these
+tests, only detections measured at the $\sigma_{thresh} = 5\sigma$
+level are used, to match that used in the photometry on science data.
+This probability can be converted into a number of false number by
+considering a given area.  As the detections must be isolated to not
+be detected as an extended object, this area must be reduced by the
+area a given PSF occupies.  Combining this, we find that we expect a
+probability $P = 1 - \Phi_{normal}(5) = \frac{1}{2}
+\erfcinv\left(\frac{5}{\sqrt{2}}\right)$, and an area given $N$
+exposures of area $X\times Y$, $A = \frac{X \times Y \times
+  N}{A_{PSF}}$.  For a typical $1"$ seeing, $A_{PSF}$ is approximately
+pixels.  Using this model for the false positives, we found that
+the added read noise was insufficient to account for the observed
+false positive rate.  Inverting this relation, we can measure
+$\sigma_{obs}$, the true threshold level based on the number of false
+positives observed.  This $\sigma_{obs}$ is the combined to form a
+boost factor $B = \sigma_{thresh} / \sigma_{obs}$ that amplifies the
+  noisemap to match the observed false detection rate.
+The row-to-row variations that contribute to the extra noise are
+related to the dark model, and because of this, as the dark model
+changes, the effective noise also changes.  To ensure that the
+noisemap accurately matches the true noise level, we have created
+different noisemap models for the three major time ranges of the dark
+model.  We do not see any strong evidence that the noisemaps have the
+A/B modes visible in the dark, and so we do not generate different
+models for each individual dark model.  The additional pixel-to-pixel
+variance from this noisemap is added to the Poissonian variance to
+form the science variance image generated by the \IPPstage{chip}
+processing.
+In the detection process, we expect false positives at a low rate,
+given that all sources are required to be significant at the $5\sigma$
+level.  Since the observed false positive rate was significantly
+higher than expected, we implemented an empirical ``boost'' to
+increase the noisemap to more accurately account for the position
+dependent read noise.  By binning the number of false positives
+measured on the bias frames on the noisemap inputs using 20 pixel
+boxes in the cell x-axis, and comparing this to the number expected
+from random Gaussian noise, we estimated the true read noise level.
+As the noisemap uses bias frames that have had a dark model
+subtracted, we constructed noisemaps for each dark model used for
+science processing.  There is some evidence that the noise has changed
+over time as measured on full cells, so matching the noisemap to the
+dark model allows for these changes to be tracked.  There is no
+evidence that the noisemap has the A/B modes found in the dark, so we
+do not generate separate models for that time period.
+The noisemap detrend is not directly applied to the science image.
+Instead, it is used to construct the weight image that contains the
+pixel-by-pixel variance for the \IPPstage{chip} stage image.  The
+initial weight image is constructed by dividing the science image by
+the cell gain (approximately 1.0 e$^{-} /$ DN).  This weight image
+contains the expected Poissonian variance in electrons measured.  The
+square of the noisemap is then added to this initial weight, adding
+the additional empirical variance term in place of a single read noise
+value.
 \subsection{Flat}
 …
 correction to remove the effect of the illumination differences over
 the detector surface.  This is done by dithering a series of science
+exposures with a given pointing.  By fully calibrating these exposures
+exposures with a given pointing, as described in
+\citet{2004PASP..116..449M}.  By fully calibrating these exposures
 with the initial flat model, and then comparing the measured fluxes
 for the same star as a function of position on the detector, we can
 …
 In addition to this flat field applied to the individual images, the
 ubercal process used to calibrate the database of all detections
+\citep{2012ApJ...756..158S} constructs internal ``flat field'' corrections.
+Although a single set of image flat fields was used for the entire PV3
+survey, five separate ``seasons'' of database flat fields were needed
+to ensure proper calibration.  This indicates that the flat field
+response is not completely fixed in time.  More details on this
+process are contained in \citet{magnier2017c}.
+\subsection{Pattern correction}
+\label{sec:pattern}
+Due to detector specific issues that are not cleanly removed by the
+dark model, we have a set of ``pattern'' corrections that are applied
+to some selection of the OTAs in the camera.  This is done to reduce
+the effect that detector differences have on the measured astronomical
+signal that are not stable enough to be corrected with a static model.
+Because of this, the pattern corrections attempt to identify and
+correct the detector issues based on appropriate filtering the
+individual science exposures.
+The PATTERN.ROW correction is used to remove any remaining row-by-row
+bias variation, and the PATTERN.CONTINUITY correction attempts to
+ensure that the cells of a given OTA are consistent with the other
+cells on that OTA.
+\subsubsection{Pattern Row}
+%% Statistics so I have them written down somewhere
+%% chipProcessedImfile.bg/bg_stdev by filter for XY33 (a good chip)
+%% filter  bg_mean stdev median Qsig                              bg_stdev_mean stdev median Qsig
+%% g        36.37422026669   64.64175104057  32.693   6.10284     14.696938349131  78.80460307171  8.8401  0.5417843
+%% r       200.96143304525  471.87743546238 117.105  94.55608     33.854672792146  79.01642728089 13.4564  5.3771355
+%% i       447.00504994458  938.38517801037 286.810 154.71397     57.298335510188  99.38392923935 20.0217 24.2254723
+%% z       317.54933679054  390.38930252748 241.014 114.13316     48.359069000176  94.44452756094 17.9404  9.1535209
+%% y       371.09019536218  293.57439970375 288.481 133.38769     43.724342221691 135.04286534327 19.9029  7.5396461
+As discussed above in the dark and noisemap sections, certain
+detectors have significant bias offsets between adjacent rows, caused
+by noise in the camera control electronics.  The magnitude of these
+offsets increases as the distance from the readout amplifier
+increases, resulting in horizontal streaks that are more pronounced
+along the large x pixel edge of the cell.  As the level of the offset
+is apparently random between exposures, the dark correction cannot
+fully remove this structure from the images, and the noisemap value
+only indicates the level of the average variance added by these bias
+offsets.  Therefore, we apply the PATTERN.ROW correction in an attempt
+to mitigate the offsets and correct the image values.  To force the
+rows to agree, a second order clipped polynomial is fit to each row in
+the cell.  Four fit iterations are run, and pixels $2.5\sigma$ deviant
+are excluded from subsequent fits, to minimize the effect stars and
+other astronomical signals have.  This final trend is then subtracted
+from that row.  Simply doing this subtraction will also have the
+effect of removing the background sky level.  To prevent this, the
+constant and linear terms for each row are stored, and linear fits are
+made to these parameters as a function of row, perpendicular to the
+initial fits.  This produces a plane that is added back to the image
+to restore the background offset and any linear ramp that exists in
+the sky.
+These row-by-row variations have the largest impact on data taken in
+the \gps{} filter, as the read noise is the dominant noise source in
+that filter.  At longer wavelengths, the noise from the Poissonian
+variation in the sky level increases.  Although the PATTERN.ROW correction is still applied to data taken in the other filters,
+This correction was required on all cells on all OTAs prior to
+-12-01, at which point a modification of the camera electronics
+reduced the scale of the row-by-row offsets for the majority of the
+OTAs.  As a result, we only apply this correction to the cells where
+it is still necessary, as shown in Figure \ref{fig: pattern row
+  cells}.  A list of these cells is listed in Table
+\ref{tab:pattern_row_cells}.
+Although this correction does largely resolve the row-by-row offset
+issue in a satisfactory way, large and bright astronomical objects can
+bias the fit significantly.  This results in an oversubtraction of the
+offset near these objects.  As the offsets are calculated on the pixel
+rows, this oversubtraction is not uniform around the object, but is
+preferentially along the horizontal x axis of the object.  Most
+astronomical objects are not significantly distorted by this, with
+this only becoming on issue for only bright objects comparable to the
+size of the cell (598 pixels = 150").
+\begin{deluxetable}{lcccc}
+  \tablecolumns{3}
+  \tablewidth{0pc}
+  \tablecaption{Cells which have PATTERN.ROW correction applied}
+  \tablehead{\colhead{OTA} & \colhead{Cell columns} & \colhead{Additional cells}}
+  \startdata
+  OTA11 &  & xy02, xy03, xy04, xy07 \\
+  OTA14 &  & xy23 \\
+  OTA15 & 0 & \\
+  OTA27 & 0, 1, 2, 3, 7 & \\
+  OTA31 & 7 & \\
+  OTA32 & 3, 7 & \\
+  OTA45 & 3, 7 & \\
+  OTA47 & 0, 3, 5, 7 & \\
+  OTA57 & 0, 1, 2, 6, 7 & \\
+  OTA60 &  & xy55 \\
+  OTA74 & 2, 7 & \\
+  \enddata
+  \label{tab:pattern_row_cells}
+\end{deluxetable}
+\begin{figure}
+  \centering
+  \includegraphics[width=0.9\hsize,angle=0,clip]{images/pattern_row_edit.png}
+  \caption{Diagram illustrating in red which cells on GPC1 require the PATTERN.ROW correction to be applied.  The footprint of each OTA is outlined, and cell xy00 is marked with either a filled box or an outline.  The labeling of the non-existent corner OTAs is provided to orient the focal plane.}
+  \label{fig: pattern row cells}
+\end{figure}
+\begin{figure}
+  \centering
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_nopat.png}
+%    \caption{(a)}
+%  \end{subfigure}%
+%  \begin{subfigure}[]{.45\hsize}
+  \end{minipage}%
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_pat.png}
+%    \caption{(b)}
+%  \end{subfigure}
+  \end{minipage}
+  \caption{Example of the PATTERN.ROW correction on exposure o5379g0103o OTA57 cell xy00 (\ips{} filter 45s).  The left panel shows the cell with all appropriate detrending except the PATTERN.ROW, and the right shows the same cell with PATTERN.ROW applied.  The correction reduces the correlated noise on the right side, which is most distant from the read out amplifier.  There is a slight over subtraction along the rows near the bright star.}
+\end{figure}
+\subsubsection{Pattern Continuity}
+After previous attempts to ensure that adjacent cells on an OTA
+matched background levels were insufficient in many situations, we
+designed a replacement correction that would reduce the background
+distortion for large objects.  In addition, studies of the background
+level illustrated that the row-by-row bias can introduce small
+background gradient variations along the rows of the cells that is not
+stable enough to be completely fit by the dark model.  This common
+feature across the columns of cells results in a ``saw tooth'' pattern
+horizontally across an OTA, and as the background model fits a smooth
+sky level, this induces over and under subtraction at the cell
+boundaries.
+The PATTERN.CONTINUITY correction, attempts to match the edges of a
+cell to those of its neighbors.  For each cell, a thin box 10 pixels
+wide on each edge is extracted and the median value of unmasked values
+calculated for that box.  These median values are then used to
+construct a vector of differences $\Delta_i = \sum_{j} \mathrm{Edge}_{i} -
+\mathrm{Edge}_{j}$, along with a matrix of associations $A_{i,i'} = \sum_{j}
+\delta(i,j) \delta(j,i')$ denoting which cell boundaries are adjacent.
+By solving the system $A x = \Delta$, we find the set of offsets $x_i$
+to be applied to each cell to ensure the minimum differences between
+all cell edges and their neighbors.
+For OTAs that initially show the saw tooth pattern, the effect of this
+correction is to align the cells into a single ramp, at the expense of
+the absolute background level.  However, as we subtract off a smooth
+background model prior to doing photometry, these deviations from an
+absolute sky level are unimportant.  The fact that the final ramp is
+smoother than it would be otherwise also allows for the background
+subtracted image to more closely match the astronomical sky, without
+significant errors at cell boundaries.  An example of the effect of
+this correction on an image profile is shown in Figure \ref{fig:dark switching}.
+\citep{2012ApJ...756..158S} constructs ``in catalog'' flat field
+corrections.  Although a single set of image flat fields was used for
+the entire PV3 survey, five separate ``seasons'' of database flat
+fields were needed to ensure proper calibration.  This indicates that
+the flat field response is not completely fixed in time.  More details
+on this process are contained in \citet{magnier2017.calibration}.
 \subsection{Fringe correction}
 …
 input images with two iteration of clipping at the $3\sigma$ level.
 A course background model for each cell is constructed by calculating
+A coarse background model for each cell is constructed by calculating
 the median on a 3x3 grid (approximately 200x200 pixels each).  A set
 of 1000 randomly selected points are then selected on the fringe image
+for each cell, and a median calculated for this position in a 10x10
 pixel box, with the background level subtracted.  These sample
+locations provide scale points to allow the amplitude of the measured
 fringe to be compared to that found on science images.
+of 1000 points are randomly selected from the fringe image for each
+cell, and a median calculated for this position in a 10x10 pixel box,
+with the background level subtracted.  These sample locations provide
+scale points to allow the amplitude of the measured fringe to be
+compared to that found on science images.
 To apply the fringe, the same sample locations are measured on the
 …
 \label{sec:static_masks}
+Due to the large size of the detector, it is expected that there
+are a number of pixel defects that do not have the detection
+sensitivity on par with their neighbors.  To remove these pixels, we
+have constructed a static mask that identifies the known defects.
+This mask is constructed in three phases.
+First, a CTEMASK is constructed to mask out regions in which the
+charge transfer efficiency is low compared to the rest of the
+detector.  Twenty-five of the sixty OTAs in GPC1 show some evidence of
+CTE issues, with this pattern appearing (to varying degrees) in
+roughly triangular patches on the OTA due to defects in the
+semiconductor manufacturing.  To generate the mask for these regions,
+a sample set of 26 evenly illuminated flat field images were measured
+to produce a map of the image variance in 20x20 pixel bins.  As the
+flat image is expected to illuminate the image uniformly, the expected
+variances in each bin should be Poissonian distributed with the flux
+level.  However, in regions with CTE issues, adjacent pixels are not
+independent, as the charge in those pixels is more free to spread.
+This reduces the pixel-to-pixel differences, resulting in a lower than
+expected variance.  All regions with variance less than half the
+average image level are added to the static CTEMASK.
+Due to the large size of the detector, it is expected that there are a
+number of pixels that respond poorly.  To remove these pixels, we have
+constructed a mask that identifies the known defects.  This mask is
+referred to as the ``static'' mask, as it is applied to all images
+processed.  The ``dynamic'' mask (Section \ref{sec:dynamic_masks}) is
+calculated based on objects in the field, and so changes between
+images.  Construction of the static mask consists of three phases.
+First, regions in which the charge transfer efficiency (CTE) is low
+compared to the rest of the detector are identified.  Twenty-five of
+the sixty OTAs in GPC1 show some evidence of poor CTE, with this
+pattern appearing (to varying degrees) in roughly triangular patches.
+During the manufacture of the devices, an improperly tuned
+semiconductor process step resulted in a radial pattern of poor
+performance on some silicon wafers.  When the OTAs were cut from these
+wafers, the outer corners exhibited the issue.  To generate the mask
+for these regions, a sample set of 26 evenly-illuminated flat-field
+images were measured to produce a map of the image variance in 20x20
+pixel bins.  As the flat screen is expected to illuminate the image
+uniformly on this scale, the expected variances in each bin should be
+Poissonian distributed with the flux level.  However, in regions with
+poor CTE, adjacent pixels are not independent, as the charge in those
+pixels is more free to spread along the image columns.  This reduces
+the pixel-to-pixel differences, resulting in a lower than expected
+variance.  All regions with variance less than half the average image
+level are added to the static mask.
 The next step of mask construction is to examine the flat and dark
 …
 removing the pixels that cannot be corrected to a linear response.
+% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/StaticMasks20101215
 The final step of mask construction is to examine the detector for
 bright columns and other static pixel issues.  This is first done by
 …
   \centering
   \includegraphics[width=0.9\hsize,angle=0,clip]{images/gpc1_mask_indexed.png}
-  \label{fig:static mask}
   \caption{Image map of the GPC1 static mask.  The CTE regions are clearly visible as roughly triangular patches covering the corners of some OTAs.  Some entire cells are masked, including an entire column of cells on OTA14.  Calcite cells remove large areas from OTA17 AND OTA76.}
+  \label{fig:static mask}
 \end{figure}
 …
 nature, and do not completely exclude the pixel from further
 processing consideration.  The first of these dynamic masks is the
 burntool advisory mask mentioned above.  These pixels are included for
+burntool advisory mask described below.  These pixels are included for
 photometry, but are rejected more readily in the stacking and
 difference image construction, as they are more likely to have small
 deviations due to imperfections in the burntool correction.
 The remaining dynamic masks are not generated until the IPP
+\IPPstage{camera} stage, at which point all object photometry is
+complete, and an astrometric solution is known for the exposure.  This
+added information provides the positions of bright sources based on
 the reference catalog, including those that fall slightly out of the
+The remaining dynamic masks are generated in the IPP \IPPstage{camera}
+stage, at which point all object photometry is complete, and an
+astrometric solution is known for the exposure.  This added
+information provides the positions of bright sources based on the
+reference catalog, including those that fall slightly out of the
 detector field of view or within the inter chip gaps, where internal
 photometry may not identify them.  These bright sources are the origin
 …
 Table \ref{tab:crosstalk_rules} summarizes the list of known crosstalk
 rules, with an estimate of the magnitude difference between the source
 and ghost.  For all of the rules, any cell $v$ within the specified
+and ghost.  For all of the rules, any source cell $v$ within the specified
 column of cells on any of the OTAs in the specified column of OTAs $Y$
 creates the ghost in the same $v$ and $Y$ in the target column of
+can create a ghost in the same cell $v$ and OTA $Y$ in the target column of
 cells and OTAs.  In each of these cases, a source object with an
 instrumental magnitude brighter than -14.47 creates a ghost object
 …
   \label{tab:crosstalk_rules}
 \end{deluxetable}
 \subsubsubsection{Optical ghosts}
 \label{sec:optical_ghosts}
+Due to imperfections in the anti-reflective coating on the optical
+surfaces of GPC1, bright sources can also result in large out of focus
+objects, particularly in the \gps{} filter data.  These objects are the
+result of light reflecting back off the surface of the detector,
+reflecting again off the lower surfaces of the optics (particularly
+the L1 corrector lens), and then back down onto the focal plane.  Due
+to the extra travel distance, the resulting source is out of focus and
+The anti-reflective coating on the optical surfaces of GPC1 is less
+effective at shorter wavelengths, which can allow bright sources to
+reflect back onto the focal plane and generate large out-of-focus
+objects.  Due to the wavelength dependence, these objects are most
+prominent in the \gps{} filter data.  These objects are the result of
+light reflecting back off the surface of the detector, reflecting
+again off the lower surfaces of the optics (particularly the L1
+corrector lens), and then back down onto the focal plane.  Due to the
+extra travel distance, the resulting source is out of focus and
 elongated along the radial direction of the camera focal plane. These
 optical ghosts can be modeled in the focal plane coordinates (L,M)
 which has its origin at the center of the focal plane.  In this
 system, a bright object at location (L,M) on the focal plane creates a
 reflection ghost on the opposite side of the optical axis at (-L,-M).
+The exact location is fit as a third order polynomial in the focal
+plane L and M directions (as listed in Table \ref{tab:ghost_centers}).
+An elliptical annulus mask is constructed at the expected ghost
+location, with the major and minor axes defined by linear functions of
+the ghost distance from the optical axis, and oriented with the
+ellipse major axis is along the radial direction (Table
+\ref{tab:ghost_radii}).  All stars brighter than a filter-dependent
+threshold (listed in Table \ref{tab:ghost_magnitudes}) have such masks
 constructed.
+reflection ghost on the opposite side of the optical axis near
+(-L,-M).  The exact location is fit as a third order polynomial in the
+focal plane L and M directions (as listed in Table
+\ref{tab:ghost_centers}).  An elliptical annulus mask is constructed
+at the expected ghost location, with the major and minor axes defined
+by linear functions of the ghost distance from the optical axis, and
+oriented with the ellipse major axis is along the radial direction
+(Table \ref{tab:ghost_radii}).  All stars brighter than a
+filter-dependent threshold (listed in Table
+\ref{tab:ghost_magnitudes}) have such masks constructed.
 \begin{deluxetable}{lcc}
 …
 \end{deluxetable}
 \begin{figure}
   \centering
   \includegraphics[width=0.9\hsize,angle=0,clip]{images/full_fpa_ghosts.jpg}
   \caption{Example of the full GPC1 field of view illustrating the sources and destinations of optical ghosts on exposure o5677g0123o (2011-04-26, 43s \gps{} filter).  The bright stars on OTA33 and OTA44 result in nearly circular ghosts on the opposite OTA.  In contrast, the trio of stars on OTA11 result in very elongated ghosts on OTA66.}
+  \label{fig:optical ghosts}
 \end{figure}
 …
 detector surface in a long narrow glint.  This surface was physically
 masked on 2010-08-24, removing the possibility of glints in subsequent
+data, but that taken prior have an advisory dynamic mask constructed
+when a reference source falls on the focal plane within one degree of
+the detector edge.  This mask is 150 pixels wide, with length $L =
+\left(-20 - m_{inst}\right)$ pixels.  These glint masks are
+constructed by selecting sufficiently bright sources in the reference
+catalog that fall within rectangular regions around each edge of the
+GPC1 camera.  These regions are separated from the edge of the camera
+by 17 arcminutes, and extend outwards an additional degree.
+data, but images that were taken prior to this date have an advisory
+dynamic mask constructed when a reference source falls on the focal
+plane within one degree of the detector edge.  This mask is 150 pixels
+wide, with length $L = 2500 \left(-20 - m_{inst}\right)$ pixels.
+These glint masks are constructed by selecting sufficiently bright
+sources in the reference catalog that fall within rectangular regions
+around each edge of the GPC1 camera.  These regions are separated from
+the edge of the camera by 17 arcminutes, and extend outwards an
+additional degree.
 \begin{figure}
 …
   \includegraphics[width=0.9\hsize,angle=0,clip]{images/glint_example_o5379g0103o.jpg}
   \caption{Example of a glint on exposure o5379g0103o (2010-07-02, 45s \ips{} filter).  The source star out of the field of view creates a long reflection that extends through OTA73 and OTA63.}
+  \label{fig:optical glints}
 \end{figure}
 …
 mask is constructed, as the source is likely too faint to produce the
 feature.  These spikes are dependent on the camera rotation, and are
 oriented at $\theta = n * \frac{\pi}{2} - \mathrm{ROTANGLE} + 0.798$,
 based on the header keyword.
+oriented based on the header keyword at $\theta = n * \frac{\pi}{2} -
+\mathrm{ROTANGLE} + 0.798$, for $n = {0,1,2,3}$.
 The cores of stars that are saturated are masked as well, with a
 …
 \label{sec:masking_fraction}
+Although there are a large number of masked pixels within the sixty
+OTAs of GPC1, the camera was designed to move chips with problematic
+areas (most notably CTE issues) to the edges of the detector.  Because
+of this, the main analysis of the mask fraction is based not on the
+total footprint of the detector, but upon a circular reference field
+of view with a radius of 1.5 degrees.  This field of view corresponds
+approximately to half the width and height of the detector.  This
+field of view underestimates the unvignetted region of GPC1.  A second
+``maximum'' field of view is also used to estimate the mask fraction
+within a larger 1.628 degree radius.  This larger radius includes far
+larger missing fractions due to the circular regions outside region
+populated with OTAs, but does include the contribution from
+well-illuminated pixels that are ignored by the reference radius.
+The results of simulating simulating the footprint of the detector as
+a grid of uniformly sized pixels of $0\farcs{}258$ size are provided
+in Table \ref{tab:mask fraction}.  Both fields of view contain
+circular segments outside of the footprint of the detector, which
+increase the area estimate that is unpopulated.  This category also
+accounts for the inter-OTA and inter-cell gaps.  The regions with poor
+CTE also account for a significant fraction of the masked pixels.  The
+The GPC1 camera was designed such that where possible, OTAs with CTE
+issues were placed towards the edge of the detector.  Because of this,
+the main analysis of the mask fraction is based not on the total
+footprint of the detector, but upon a circular reference field of view
+with a radius of 1.5 degrees.  This radius corresponds approximately
+to half the width and height of the detector.  This field of view
+underestimates the unvignetted region of GPC1.  A second ``maximum''
+field of view is also used to estimate the mask fraction within a
+larger 1.628 degree radius.  This larger radius includes far larger
+missing fractions due to the circular regions outside region populated
+with OTAs, but does include the contribution from well-illuminated
+pixels that are ignored by the reference radius.
+The results of simulating the footprint of the detector as a grid of
+uniformly sized pixels of $0\farcs{}258$ size are provided in Table
+\ref{tab:mask fraction}.  Both fields of view contain circular
+segments outside of the footprint of the detector, which increase the
+area estimate that is unpopulated.  This category also accounts for
+the inter-OTA and inter-cell gaps.  The regions with poor CTE also
+contribute to a significant fraction of the masked pixels.  The
 remaining mask category accounts for known bad columns, cells that do
 not calibrate well, and vignetting.  There are also a small fraction
 …
 input masks already account for the inter-cell gaps).  This estimate
 does not include the circular segments outside of the detector
 footprint.  This is minor for the reference field of view (1\%
 difference), but does underestimate the static mask fraction for the
 maximum radius by 7.3\%.  This analysis does provide a the observed
+dynamic and advisory mask fractions, which are 0.03\% and 3\%
+respectively.  The significant advisory value is a result of applying
 such masks to all burntool corrected pixels.
+footprint.  This difference is minor for the reference field of view
+(1\% difference), but underestimates the static mask fraction for the
+maximum radius by 7.3\%.  This analysis provides the observed dynamic
+and advisory mask fractions, which are 0.03\% and 3\% respectively.
+The significant advisory value is a result of applying such masks to
+all burntool corrected pixels.
 \begin{deluxetable}{lcc}
 …
 background model for the full OTA is then determined prior to the
 photometric analysis.  The mosaicked image is subdivided into
 $800\times{}800$ pixel segments that define each pixel of the
 background model, with the segments centered on the image center, and
+overlapping adjacent subdivisions by 400 pixels.  These overlaps help
 smooth the background model, as adjacent model pixels share input
+$800\times{}800$ pixel segments that define each superpixel of the
+background model, with the superpixels centered on the image center
+and overlapping adjacent superpixels by 400 pixels.  These overlaps
+help smooth the background model, as adjacent model pixels share input
 pixels.
 From each subdivision, 10000 random unmasked pixels are drawn.  In the
+From each segment, 10000 random unmasked pixels are drawn.  In the
 case where the mask fraction is large (such as on OTAs near the edge
 of the field of view), and there are insufficient unmasked pixels to
 meet this criterion, all possible unmasked pixels are used instead.
 If this number is still small (less than 100 good pixels), the
 subdivision does not have a background model calculated, and instead,
+superpixel does not have a background model calculated.  Instead,
 the value assigned to that model pixel is set as the average of the
 adjacent model pixels.  This allows up to eight neighboring background
 values to be used to patch these bad pixels.
 For the remaining subdivisions that have sufficient unmasked pixels
+for the background to be measured, the pixel values are used to
+calculate a set of robust statistics for the initial background guess.
 The minimum and maximum of the values are found, and checked to ensure
+For the subdivisions that have sufficient unmasked pixels for the
+background to be measured, the pixel values are used to calculate a
+set of robust statistics for the initial background guess.  The
+minimum and maximum of the values are found, and checked to ensure
 that these are not the same value, which would indicate some problem
 with the input values.  The values are then inserted into a histogram
 with 1000 bins between the minimum and maximum values, and again
+checked for issues with the inputs by ensuring that the bin with the
+most input pixels does not contain more than half of the input values.
+In this case, the minimum and maximum do not constrain the true
+distribution of the input values well, and any values outside of the
+bins closest to the bin with the peak are masked for future
 consideration.  A cumulative distribution is then constructed from the
+with 1000 bins between the minimum and maximum values.  If the bin
+with the most input pixels contains more than half of the input
+values, the bin size is too coarse for the population of interest.  In
+this case, a new histogram is constructed using a range corresponding
+to the 20 bins closes to the peak, again dividing the range into 1000
+bins.  This process is iterated up to 20 times until a binsize is
+determined.  A cumulative distribution is then constructed from the
 histogram, which saves the computational cost of sorting all the input
 values.  The bins containing the 50-percentile point, as well as the
 …
 If this measured standard deviation is smaller than 3 times the bin
 size, then all points more than 25 bins away from the calculated
 median are masked, and the process is repeated until the bin size is
+sufficiently small to ensure that the distribution width is well
+sampled.  Once this iterative process converges, or 20 iterations are
+run, the 25- and 75-percentile values are found by interpolating the 5
+bins around the expected bin as well, and the count of the number of
+input values within this inner 50-percentile region, $N_{50}$ is
 calculated.
+median are masked, and the process is repeated with a new 1000 bin
+histogram until the bin size is sufficiently small to ensure that the
+distribution width is well sampled.  Once this iterative process
+converges, or 20 iterations are run, the 25- and 75-percentile values
+are found by interpolating the 5 bins around the expected bin as well,
+and the count of the number of input values within this inner
+-percentile region, $N_{50}$ is calculated.
 These initial statistics are then used as the starting guesses for a
 …
 distribution with a Gaussian.  All pixels that were masked in the
 initial calculation are unmasked, and a histogram is again constructed
 of the values, with a bin size set to $\sigma_{guess} / \left( N_{50} /
+from the values, with a bin size set to $\sigma_{guess} / \left( N_{50} /
 \right)$.  With this bin size, we expect that a bin at $\pm 2
 \sigma$ will have approximately 50 input points, which gives a
 Poissonian signal to noise estimate around 7.  In the case where
+Poissonian signal-to-noise estimate around 7.  In the case where
 $N_{50}$ is small (due to a poorly populated input image), this bin
 size is fixed to be no larger than the guess of the standard
 …
 Two second order polynomial fits are then performed to the logarithm
 of the histogram counts set at the midpoint of each bin.  The first
 fit considers the ``lower half'' of the distribution, under the
+fit considers the lower portion of the distribution, under the
 assumption that deviations from a normal distribution are caused by
 real astrophysical sources that will be brighter than the true
 …
 fit.  The Gaussian mean and standard deviation are calculated from the
 polynomial coefficients, and the symmetric fit results are accepted
+unless the lower-half fit results in a smaller mean.  This process is
+repeated again if the calculated standard deviation is not larger than
+\% of the initial guess (suggesting an issue with the initial bin
+size).
+unless the lower-half fit results in a smaller mean.  This histogram
+and polynomial fit process is repeated again, with updated bin size
+based on the previous iteration standard deviation, if the calculated
+standard deviation is not larger than 75\% of the initial guess
+(suggesting an issue with the initial bin size).
 With this two-stage calculation performed across all subdivisions of
 the mosaicked OTA image, and missing model pixels filled with the
 average of their neighbors, the final background model is stored on
 disk as a $13\times{}13$ image with header entries listing the binning
+used.  The full scale background image is then constructed by
+bilinearly interpolating this binned model, and this is subtracted
+from the science image.  Each object in the photometric catalog has a
+SKY and SKY\_SIGMA value that is the evaluation of this model at the
 location of that object.
+disk as a $13\times{}13$ image for the GPC1 chips with header entries
+listing the binning used.  The full scale background image is then
+constructed by bilinearly interpolating this binned model, and this is
+subtracted from the science image.  Each object in the photometric
+catalog has a SKY and SKY\_SIGMA value determined from the background
+model mean and standard deviation.
 Although this background modeling process works well for most of the
 …
 of GPC1, the measured background was added back to the \IPPstage{chip}
 stage images, but this special processing was not used for the large
+scale $3\Pi$ PV3 reduction.
+scale $3\pi$ PV3 reduction.
+\subsection{Burntool / Persistence effect}
+\label{sec:burntool}
+Pixels that approach the saturation point on GPC1, which varies by
+cell with common values around 60000 DN, introduce ``persistent
+charge'' on that and subsequent images.  During the read out process
+of a cell with such a bright pixel, some of the charge remains in the
+undepleted region of the silicon and is not shifted down the detector
+column toward the amplifier.  This charge remains in the starting
+pixel and slowly leaks out of the undepleted region, contaminating
+subsequent pixels during the read out process, resulting in a ``burn
+trail'' that extends from the center of the bright source away from
+the amplifier (vertically along the pixel columns toward the top of
+the cell).
+This incomplete charge shifting in nearly full wells continues as each
+row is read out.  This results in a remnant charge being deposited in
+the pixels that the full well was shifted through.  In following
+exposures, this remnant charge leaks out, resulting in a trail that
+extends from the initial location of the bright source on the previous
+image towards the amplifier (vertically down along the pixel column).
+This remnant charge can remain on the detector for up to thirty
+minutes.
+Both of these types of persistence trails are measured and optionally
+repaired via the \IPPprog{burntool} program.  This program does an
+initial scan of the image, and identifies objects with pixel values
+higher than a conservative threshold of 30000 DN.  The trail from the
+peak of that object is fit with a one-dimensional power law in each
+pixel column above the threshold, based on empirical evidence that
+this is the functional form of this persistence effect.  This fit also
+matches the expectation that a constant fraction of charge is
+incompletely transferred at each shift beyond the persistence
+threshold.  Once the fit is done, the model can be subtracted from
+the image.  The location of the source is stored in a table along
+with the exposure PONTIME, which denotes the number of seconds since
+the detector was last powered on and provides an internally
+consistent time scale.
+For subsequent exposures, the table associated with the previous image
+is read in, and after correcting trails from the stars on the new
+image, the positions of the bright stars from the table are used to
+check for remnant trails from previous exposures on the image.  These
+are fit and subtracted using a one-dimensional exponential model,
+again based on empirical studies.  The output table retains this
+remnant position for 2000 seconds after the initial PONTIME recorded.
+This allows fits to be attempted well beyond the nominal lifetime of
+these trails.  Figure \ref{fig:burntool images} shows an example of a
+cell with a persistence trail from a bright star, the post-correction
+result, as well as the pre and post correction versions of the same
+cell on the subsequent exposure.  The profiles along the detector
+columns for these two exposures are presented in Figure
+\ref{fig:burntool plot}.
+Using this method of correcting the persistence trails has the
+challenge that it is based on fits to the raw image data, which may
+have other signal sources not determined by the persistence effect.
+The presence of other stars or artifacts in the detector column can
+result in a poor model to be fit, resulting in either an over- or
+under-subtraction of the trail.  For this reason, the image mask is
+marked with a value indicating that this correction has been applied.
+These pixels are not fully excluded, but they are marked as suspect,
+which allows them to be excluded from consideration in subsequent
+stages, such as image stacking.
+The cores of very bright stars can also be deformed by this process,
+as the burntool fitting subtracts flux from only one side of the star.
+As most stars that result in persistence trails already have saturated
+cores, they are already ignored for the purpose of PSF determination
+and are flagged as saturated by the photometry reduction.
+\begin{figure}
+  \centering
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_nobt.png}
+  \end{minipage}%
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_nobt.png}
+  \end{minipage}
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt.png}
+  \end{minipage}%
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt.png}
+  \end{minipage}
+  \caption{Example of OTA11 cell xy50 on exposures o5677g0123o (left) and o5677g0124o (right).  The top panels show the image with all appropriate detrending steps, but without burntool, and the bottom show the same with burntool applied.  There is some slight over subtraction in fitting the initial trail, but the impact of the trail is greatly reduced in both exposures.}
+  \label{fig:burntool images}
+\end{figure}
+\begin{figure}
+  \centering
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt_trail.png}
+  \end{minipage}%
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt_trail.png}
+  \end{minipage}
+  \caption{Example of a profile cut along the y-axis through a bright star on exposure o5677g0123o OTA11 in cell xy50 (left panel) and on the subsequent exposure o5677g0124o (right panel).  In both figures, the green points show the image corrected with all appropriate detrending steps, but without burntool applied, illustrating the amplitude of the persistence trails.  The red points show the same data after the burntool correction, which reduces the impact of these features.  Both exposures are in the \gps{} filter with exposure times of 43s}
+  \label{fig:burntool plot}
+\end{figure}
+\subsection{Non-linearity Correction}
+\label{sec:nonlinearity}
+The pixels of GPC1 are not uniformly linear at all flux levels.  In
+particular, at low flux levels, some pixels have a tendency to sag
+relative to the expected linear value.  This effect is most pronounced
+along the edges of the detector cells, although some entire cells show
+evidence of this effect.
+To correct this sag, we studied the behavior of a series of flat
+frames for a ramp of exposure times with approximate logarithmically
+equal spacing between 0.01s and 57.04s.  As the exposure time
+increases, the signal on each pixel also increases in what is expected
+to be a linear manner.  Each of the flat exposures in this ramp is
+overscan corrected, and then the median is calculated for each cell,
+as well as for the rows and columns within ten pixels of the edge of
+the science region.  From these median values at each exposure time
+value, we can construct the expected trend by fitting a linear model
+for the region considered.  This fitting was limited to only the range
+of fluxes between 12000 and 38000 counts, as these ranges were found
+to match the linear model well.  This range avoids the non-linearity
+at low fluxes, as well as the possibility of high-flux non-linearity
+effects.
+We store the average flux measurement and deviation from the linear
+fit for each exposure time for each region on all detector cells in
+the linearity detrend look up tables.  An example of this data is
+shown in figure \ref{fig: nonlinearity}.  When this correction is
+applied to science data, these lookup tables are loaded, and a linear
+interpolation is performed to determine the correction needed for the
+flux in that pixel.  This look up is performed for both the row and
+column of each pixel, to allow the edge correction to be applied where
+applicable, and the full cell correction elsewhere.  The average of
+these two values is then applied to the pixel value, reducing the
+effects of pixel nonlinearity.
+This non-linearity effect appears to be stable in time for the
+majority of the detector pixels, with little evident change over the
+survey duration.  However, as the non-linearity is most pronounced at
+the edges of the detector cells, those are the regions where the
+correction is most likely to be incomplete.  Because of this fact,
+most pixels in the static mask with either the DARKMASK or FLATMASK
+bit set are found along these edges.  As the non-linearity correction
+is unable to reliably restore these pixels, they produce inconsistent
+values after the dark and flat have been applied, and are therefore
+rejected.
+\begin{figure}
+  \centering
+  \includegraphics[width=0.9\hsize,angle=0,clip]{images/linearity_XY27_xy16.png}
+  \caption{Example plot of the linearity correction as a fraction of observed flux for OTA27, cell xy16.}
+  \label{fig: nonlinearity}
+\end{figure}
+\subsection{Pattern correction}
+\label{sec:pattern}
+\subsubsection{Pattern Row}
+%% Statistics so I have them written down somewhere
+%% chipProcessedImfile.bg/bg_stdev by filter for XY33 (a good chip)
+%% filter  bg_mean stdev median Qsig                              bg_stdev_mean stdev median Qsig
+%% g        36.37422026669   64.64175104057  32.693   6.10284     14.696938349131  78.80460307171  8.8401  0.5417843
+%% r       200.96143304525  471.87743546238 117.105  94.55608     33.854672792146  79.01642728089 13.4564  5.3771355
+%% i       447.00504994458  938.38517801037 286.810 154.71397     57.298335510188  99.38392923935 20.0217 24.2254723
+%% z       317.54933679054  390.38930252748 241.014 114.13316     48.359069000176  94.44452756094 17.9404  9.1535209
+%% y       371.09019536218  293.57439970375 288.481 133.38769     43.724342221691 135.04286534327 19.9029  7.5396461
+As discussed above in the dark and noisemap sections, certain
+detectors have significant bias offsets between adjacent rows, caused
+by drifts in the bias level due to cross talk.  The magnitude of these
+offsets increases as the distance from the readout amplifier and
+overscan region increases, resulting in horizontal streaks that are
+more pronounced along the large x pixel edge of the cell.  As the
+level of the offset is apparently random between exposures, the dark
+correction cannot fully remove this structure from the images, and the
+noisemap value only indicates the level of the average variance added
+by these bias offsets.  Therefore, we apply the PATTERN.ROW correction
+in an attempt to mitigate the offsets and correct the image values.
+To force the rows to agree, a second order clipped polynomial is fit
+to each row in the cell.  Four fit iterations are run, and pixels
+$2.5\sigma$ deviant are excluded from subsequent fits, to minimize the
+effect stars and other astronomical signals have.  This final trend is
+then subtracted from that row.  Simply doing this subtraction will
+also have the effect of removing the background sky level.  To prevent
+this, the constant and linear terms for each row are stored, and
+linear fits are made to these parameters as a function of row,
+perpendicular to the initial fits.  This produces a plane that is
+added back to the image to restore the background offset and any
+linear ramp that exists in the sky.
+These row-by-row variations have the largest impact on data taken in
+the \gps{} filter, as the read noise is the dominant noise source in
+that filter.  At longer wavelengths, the noise from the Poissonian
+variation in the sky level increases.  The PATTERN.ROW correction is
+still applied to data taken in the other filters, as the increase in
+sky noise does not fully obscure the row-by-row noise.
+This correction was required on all cells on all OTAs prior to
+-12-01, at which point a modification of the camera electronics
+reduced the scale of the row-by-row offsets for the majority of the
+OTAs.  \czw{describe modification} As a result, we only apply this
+correction to the cells where it is still necessary, as shown in
+Figure \ref{fig: pattern row cells}.  A list of these cells is in
+Table \ref{tab:pattern_row_cells}.
+Although this correction largely resolves the row-by-row offset issue
+in a satisfactory way, large and bright astronomical objects can bias
+the fit significantly.  This results in an oversubtraction of the
+offset near these objects.  As the offsets are calculated on the pixel
+rows, this oversubtraction is not uniform around the object, but is
+preferentially along the horizontal x axis of the object.  Most
+astronomical objects are not significantly distorted by this, with
+this only becoming on issue for only bright objects comparable to the
+size of the cell (598 pixels = 150").  Figure \ref{fig: pattern row example}
+shows an example of a cell pre- and post-correction.
+\begin{deluxetable}{lcccc}
+  \tablecolumns{3}
+  \tablewidth{0pc}
+  \tablecaption{Cells which have PATTERN.ROW correction applied}
+  \tablehead{\colhead{OTA} & \colhead{Cell columns} & \colhead{Additional cells}}
+  \startdata
+  OTA11 &  & xy02, xy03, xy04, xy07 \\
+  OTA14 &  & xy23 \\
+  OTA15 & 0 & \\
+  OTA27 & 0, 1, 2, 3, 7 & \\
+  OTA31 & 7 & \\
+  OTA32 & 3, 7 & \\
+  OTA45 & 3, 7 & \\
+  OTA47 & 0, 3, 5, 7 & \\
+  OTA57 & 0, 1, 2, 6, 7 & \\
+  OTA60 &  & xy55 \\
+  OTA74 & 2, 7 & \\
+  \enddata
+  \label{tab:pattern_row_cells}
+\end{deluxetable}
+\begin{figure}
+  \centering
+  \includegraphics[width=0.9\hsize,angle=0,clip]{images/pattern_row_edit.png}
+  \caption{Diagram illustrating in red which cells on GPC1 require the PATTERN.ROW correction to be applied.  The footprint of each OTA is outlined, and cell xy00 is marked with either a filled box or an outline.  The labeling of the non-existent corner OTAs is provided to orient the focal plane.}
+  \label{fig: pattern row cells}
+\end{figure}
+\begin{figure}
+  \centering
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_nopat.png}
+  \end{minipage}%
+  \begin{minipage}{0.45\hsize}
+    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_pat.png}
+  \end{minipage}
+  \caption{Example of the PATTERN.ROW correction on exposure o5379g0103o OTA57 cell xy01 (\ips{} filter 45s).  The left panel shows the cell with all appropriate detrending except the PATTERN.ROW, and the right shows the same cell with PATTERN.ROW applied.  The correction reduces the correlated noise on the right side, which is most distant from the read out amplifier.  There is a slight over subtraction along the rows near the bright star.}
+  \label{fig: pattern row example}
+\end{figure}
+\subsubsection{Pattern Continuity}
+The background levels of cells on a single OTA do not always have the
+same value.  Even with dark and flat corrections applied, adjacent
+cells may not match.  In addition, studies of the background level
+indicate that the row-by-row bias can introduce small background
+gradient variations along the rows of the cells that are not stable.
+This common feature across the columns of cells results in a ``saw
+tooth'' pattern horizontally across an the mosaicked OTA, and as the
+background model fits a smooth sky level, this induces over and under
+subtraction at the cell boundaries.
+The PATTERN.CONTINUITY correction, attempts to match the edges of a
+cell to those of its neighbors.  For each cell, a thin box 10 pixels
+wide running the full length of each edge is extracted and the median
+value of unmasked values calculated for that box.  These median values
+are then used to construct a vector of the sum of the differences
+between that cell's edges and the corresponding edge on any adjacent
+cell $\Delta$.  A matrix $A$ of these associations is also
+constructed, with the diagonal containing the number of cells adjacent
+to that cell, and the off-diagonal values being set to -1 for each
+pair of adjacent cells.  The offsets needed for each chip, $x$ can
+then be found by solving the system $A x = \Delta$. A cell with the
+maximum number of neighbors, usually cell xy11, the first cell not on
+the edge of the OTA, is used to constrain the system, ensuring that
+that cell has zero correction and that there is a single solution.
+For OTAs that initially show the saw tooth pattern, the effect of this
+correction is to align the cells into a single ramp, at the expense of
+the absolute background level.  However, as we subtract off a smooth
+background model prior to doing photometry, these deviations from an
+absolute sky level do not affect photometry for point sources and
+extended sources smaller than a single cell.  The fact that the
+final ramp is smoother than it would be otherwise also allows for the
+background subtracted image to more closely match the astronomical
+sky, without significant errors at cell boundaries.  An example of the
+effect of this correction on an image profile is shown in Figure
+\ref{fig:dark switching}.
 \section{GPC1 Detrend Construction}
 \label{sec:detrend construction}
+The various detrends for GPC1 are constructed in similar ways.  A
+series of appropriate exposures is selected from the database, and
+processed with the \IPPprog{ppImage} program.  This program is used
+for the \IPPstage{chip} stage processing as well, and is designed to
+do multiple image processing operations.  The extent of this
+processing is dependent on the order in which the detrend to be
+constructed is applied to science data.  In general, the input
+exposures to the detrend have all prior stages of detrend processing
+applied.  Table \ref{tab:detrend ppImage} summarizes stages applied
+for the detrends we construct.
+The various master detrend images for GPC1 are constructed using a
+common approach.  A series of appropriate exposures is selected from
+the database, and processed with the \IPPprog{ppImage} program, which
+is designed to do multiple image processing operations.  The
+processing steps applied to the images depend on the type of master
+detrend to be constructed.  In general, the input exposures to the
+detrend have all prior stages of detrend processing applied.  Table
+\ref{tab:detrend ppImage} summarizes stages applied for the detrends
+we construct.
 Once the input data has been prepared, the \IPPprog{ppMerge} program
 is used to construct some sort of ``average'' of the inputs.  This
 step need not be a mathematical average, but is used to combine the
+signal from the individual exposures into a single output product.
 Table \ref{tab:detrend ppMerge} lists some of the properties of the
+process for the detrends, including how discrepant values are removed
+and the combination method used.  The outputs from this step have the
+format of the detrend under construction, and after construction, are
+applied to the processed input data.  This creates a set of residual
+files that are checked to determine if the newly created detrend
 correctly removes the detector dependent signal.
+is used to combine the inputs.  In some cases, this is the
+mathematical average, but in other cases it is a fit across the
+inputs.  Table \ref{tab:detrend ppMerge} lists some of the properties
+of the process for the detrends, including how discrepant values are
+removed and the combination method used.  The outputs from this step
+have the format of the detrend under construction.  After
+construction, these combined outputs are applied to the processed
+input data.  This creates a set of residual files that are checked to
+determine if the newly created detrend correctly removes the detector
+dependent signal.
 This process of detrend construction and testing can be iterated, with
 individual exposures excluded if they are found to be contaminating
+the output.  If the final detrend has sufficiently small residuals,
+then the iterations are stopped and the detrend is finalized by
+selecting the date range to which it applies.  This allows subsequent
+science processing to select the detrends needed based on the
+observation date.  Table \ref{tab:detrend list} lists the set of
+detrends used in the PV3 processing.
+the output.  The construction of detrends is largely automatic, but
+manual intervention is needed to accept the detrend for use on science
+data.  If the final detrend has sufficiently small residuals, then the
+iterations are stopped and the detrend is finalized by selecting the
+date range to which it applies.  This allows subsequent science
+processing to select the detrends needed based on the observation
+date.  Table \ref{tab:detrend list} lists the set of detrends used in
+the PV3 processing.
 \begin{deluxetable}{lcccc}
 …
   \startdata
   LINEARITY & Y & & & \\
+%%  DARKMASK  & Y & Y & Y & \\
+%%  FLATMASK  & Y & Y & Y & Y \\
+%%  CTEMASK   & Y & Y & Y & Y \\
+  DARK      & Y & Y & & \\
+%%  NOISEMAP  & Y & Y & & \\
+  FLAT      & Y & Y & Y & \\
+  FRINGE    & Y & Y & Y & Y \\
   DARKMASK  & Y & Y & Y & \\
   FLATMASK  & Y & Y & Y & Y \\
   CTEMASK   & Y & Y & Y & Y \\
-  DARK      & Y & Y & & \\
   NOISEMAP  & Y & Y & & \\
-  FLAT      & Y & Y & Y & \\
-  FRINGE    & Y & Y & Y & Y \\
   \enddata
   \label{tab:detrend ppImage}
 …
 \section{Warping}
 \label{sec:warping}
+To provide a consistent and uniform set of images for co-added image
+stacking and differences, the individual mosaicked OTA images are
+projected onto a common set of tangent plane projected regions called
+projection cells.  These projection cells are $4\times{}4$ degree
+fields spaced onto a set of centers that fully cover the sky.  They are
+arranged into rings of constant declination, and allowed to overlap as
+$|\delta|$ increases.  Each projection cell is further subdivided into
+$10\times{}10$ sky cells with fixed $0.25"$ resolution pixels, and
+constant overlap regions between adjacent skycells of $60"$.  These
+skycells are the main image unit used for processing image data beyond
+the initial chip stage.  The coordinate system used for these images
+matches the parity of the sky, with north in the positive y direction
+and east to the negative x direction.
+To provide a consistent and uniform set of coordinates for image
+combination (including stacking and differences), the individual
+mosaicked OTA images are projected onto a common pixel grids, called
+tessellations.  A tessellation can contain any number of tangent plane
+projections, with those designed for single pointing surveys using
+only one, while the tessellation used for the $3\pi$ survey contains
+tangent plane projection centers.  These ``projection cells'' are
+$4\times{}4$ degree fields spaced onto a set of centers that fully
+cover the sky.  They are arranged into rings of constant declination,
+and allowed to overlap as $|\delta|$ increases.  Each projection cell
+is further subdivided into $10\times{}10$ ``skycells'' with fixed
+$0.25"$ resolution pixels, and constant overlap regions between
+adjacent skycells of $60"$.  These skycells are the main image unit
+used for processing image data beyond the initial chip stage.  The
+coordinate system used for these images matches the parity of the sky,
+with north in the positive y direction and east to the negative x
+direction.
 After the detrending and photometry, the detection catalog for the
 full camera is fit to the reference catalog, producing third-order
+astrometric solutions that map the detector focal plane to the sky,
+and map the individual OTA pixels to the detector focal plane.  This
+solution is then used to determine which skycells the exposure OTAs
 overlap.
+full camera is fit to the reference catalog, producing astrometric
+solutions that map the detector focal plane to the sky, and map the
+individual OTA pixels to the detector focal plane
+\citep[see][]{magnier2017.calibration}.  This solution is then used to
+determine which skycells the exposure OTAs overlap.
 For each output skycell, all overlapping OTAs and the calibrated
+catalog are read into the \IPPprog{pswarp} program.  Each input image
+is examined in order, and the same transformation performed.  This
+transformation breaks the output warp image into $128\times{}128$
+pixel grid boxes.  Each grid box has a locally linear map calculated
+that converts the output warp image coordinates to the input chip
+image coordinates.  By doing the transformation in this direction,
+each output pixel has a unique sampling position on the input image
+catalog are read into the \IPPprog{pswarp} program.  The output warp
+image is broken into $128\times{}128$ pixel grid boxes.  For purposes
+of speed, each grid box has a locally linear map calculated that
+converts the output warp image coordinates to the input chip image
+coordinates.  By doing the transformation in this direction, each
+output pixel has a unique sampling position on the input image
 (although it may be off the image frame and therefore not populated),
 preventing gaps in the output image due to the spacing of the input
 pixels.
+With the locally linear grid defined, Lanczos interpolation with
+filter size parameter $a = 3$ on the input image is used to determine
+the values to assign to the output pixel location.  The output
+locations are shifted by 0.5 pixels to let the interpolation select
+the value that would be assigned to the center of the output
+pixel. This process is repeated for all grid boxes, for all input
+With the locally linear grid defined, Lanczos interpolation
+\citep{lanczos1956applied} with filter size parameter $a = 3$ on the input
+image is used to determine the values to assign to the output pixel
+location.  This process is repeated for all grid boxes, for all input
 images, and for each output image product: the science image, the
 variance, and the mask.  The image values are scaled by the absolute
 value of the Jacobian determinant of the transformation.  This
+corrects the pixel values for the possible change in pixel area due to
+the transformation.  Similarly, the variance image is scaled by the
 square of this value, again to correctly account for the pixel area
 change.
 As the interpolation constructs the output pixels from more than one
 input pixel, there is a covariance term that is must be included.  For
 each locally linear grid box, the covariance is calculated from the
+value of the Jacobian determinant of the transformation for each grid
+box.  This corrects the pixel values for the possible change in pixel
+area due to the transformation.  Similarly, the variance image is
+scaled by the square of this value, again to correctly account for the
+pixel area change.
+The interpolation constructs the output pixels from more than one
+input pixel, which introduces covariance between pixels.  For each
+locally-linear grid box, the covariance matrix is calculated from the
 kernel in the center of the 128 pixel range.  Once the image has been
 fully populated, this set of individual covariance matrices is
+fully populated, this set of individual covariance matrices are
 averaged to create the final covariance for the full image.
 …
 catalog, including only those objects that fall on the new warped image.
 These detections are transformed to match the new image location, and
 to scale the position errors based on the new orientation.
+to scale the position uncertainties based on the new orientation.
 The output image also contains header keywords SRC\_0000, SEC\_0000,
 MPX\_0000, and MPY\_0000 that contain the mappings from the warped
+MPX\_0000, and MPY\_0000 that define the mappings from the warped
 pixel space to the input image.  The SRC keyword lists the input OTA
 name, and the SEC keyword lists the image section corresponding to the
+locally linear grid box.  The MPX and MPY contain the transformation
+parameters for the locally linear grid.  These parameters are stored
 in a string listing the reference position in the chip coordinate
+frame, the slope of the relation in the warp x axis, and the slope of
+the relation in the warp y axis.  From these keywords, any position in
+the warp can be mapped back to the location in any of the input OTA
 images.
+name, and the SEC keyword lists the image section that the mapping
+covers.  The MPX and MPY contain the back-transformation linearized
+across the full chip.  These parameters are stored in a string listing
+the reference position in the chip coordinate frame, the slope of the
+relation in the warp x axis, and the slope of the relation in the warp
+y axis.  From these keywords, any position in the warp can be mapped
+back to the location in any of the input OTA images, with some
+reduction in accuracy.
 \begin{figure}
 …
 Once individual exposures have been warped onto a common projection
 system, they can then be combined pixel-by-pixel regardless of their
+system, they can be combined pixel-by-pixel regardless of their
 original orientation.  Creating a stacked image by co-adding the
 individual warps increases the signal to noise, allowing for the
+detection of objects that would not be sufficiently significant to be measured from a single image.
+Creating this stack also allows a complete image to be
+constructed that does not have regions masked due to the gaps between
+cells and OTAs.  This fully populated static sky image can also be
+used as a template for subtraction to find transient sources.
+The stacked image is comprised of all warp frames for a given skycell
+in a single filter.  The source catalogs and image components are
+loaded into the \IPPprog{ppStack} program to prepare the inputs and
+stack the frames.
+detection of objects that would not be sufficiently significant to be
+measured from a single image.  Creating this stack also allows a more
+complete image to be constructed that has fewer regions masked due to
+the gaps between cells and OTAs.  This deeper and more complete image
+can also be used as a template for subtraction to find transient
+sources.
+For the $3\pi$ survey, the stacked image is comprised of all warp
+frames for a given skycell in a single filter.  The source catalogs
+and image components are loaded into the \IPPprog{ppStack} program to
+prepare the inputs and stack the frames.
 Once all files are ingested, the first step is to measure the size and
 …
 FWHM greater than 10 pixels (2.5 arcseconds), as those images have the
 seeing far worse than average, and would degrade the final output
 stack.  For the PV3 $3\Pi$ survey, this size represents a PSF larger
+stack.  For the PV3 $3\pi$ survey, this size represents a PSF larger
 than the $97$th percentile in all filters.  A target PSF for the stack
 is constructed by finding the maximum envelope of all input PSFs,
 …
 input images when matched to the target.
 The input images also need to have their fluxes normalized to prevent
+differences in seeing and sky transparency from causing discrepancies
+during pixel rejection.  From the reference catalog calibrated input
+catalogs, we have the instrumental magnitudes of all sources, along
+with the airmass, image exposure time, and zeropoint.  All output
+stacks are calibrated to a zeropoint of 25.0 in all filters, and to
+have an airmass of 1.0.  The output exposure time is set to the sum of
+the input exposure times, regardless of if those inputs are rejected
 later in the combination process.  We can determine the relative
+The input image fluxes are normalized to prevent differences in seeing
+and sky transparency from causing discrepancies during pixel
+rejection.  From the reference catalog calibrated input catalogs, we
+have the instrumental magnitudes of all sources, along with the
+airmass, image exposure time, and zeropoint.  All output stacks are
+constructed to a target zeropoint of 25.0 in all filters, and to have
+an airmass of 1.0.  The output exposure time is set to the sum of the
+input exposure times, regardless of whether those inputs are rejected later
+in the combination process.  We can determine the relative
 transparency for each input image by comparing the magnitudes of
 matched sources between the different images.  Each image then has a
+normalization factor defined, equal to $\mathrm{norm}_{input} = (ZP_\mathrm{input}
+- ZP_\mathrm{target}) - \mathrm{transparency}_\mathrm{input} - 2.5 *
+\log_{10} (t_\mathrm{target} / t_\mathrm{input}) -
+\mathrm{F}_\mathrm{airmass} * (\mathrm{airmass}_\mathrm{input} -
+\mathrm{airmass}_\mathrm{target})$.  For the PV3 processing, the
+airmass factor $\mathrm{F}_\mathrm{airmass}$ was set to zero, such
+that all flux differences from differing exposure airmasses are
+assumed to be included in the zeropoint and transparency values.
+normalization factor defined, equal to $\mathrm{norm}_{input} =
+(ZP_\mathrm{input} - ZP_\mathrm{target}) -
+\mathrm{transparency}_\mathrm{input} - 2.5 * \log_{10}
+(t_\mathrm{target} / t_\mathrm{input}) - \mathrm{F}_\mathrm{airmass} *
+(\mathrm{airmass}_\mathrm{input} - \mathrm{airmass}_\mathrm{target})$.
+For the PV3 processing, the airmass factor
+$\mathrm{F}_\mathrm{airmass}$ was set to zero, such that all flux
+differences from differing exposure airmasses are assumed to be
+included in the zeropoint and transparency values.
 The zeropoint calibration performed here uses the calibration of the
 …
 the entire region of the sky imaged.  This further calibration is not
 available at the time of stacking, and so there may be small residuals
 in the transparency values as a result of this \citet{magnier2017c}.
+in the transparency values as a result of this \citet{magnier2017.calibration}.
 With the flux normalization factors and target PSF chosen, the
 …
 the kernel, and the residual with the target PSF used to update the
 parameters of the kernel via least squares optimization.  Stamps that
 significantly deviate are rejected, but as the squared residual
+significantly deviate are rejected, although the squared residual
 difference will increase with increasing source flux.  To mitigate
 this effect, a parabola is fit to the distribution of squared
 …
 convolution kernel is returned.
 This convolution may change the image flux scaling, so a normalization
 factor is used to correct this.  This normalization factor is equal to
+This convolution may change the image flux scaling, so the kernel is
+normalized to account for this.  The normalization factor is equal to
 the ratio of $10^{-0.4 \mathrm{norm}_{input}}$ to the sum of the
 kernel.  The image is multiplied by this factor, and the variance by
 …
 Once the convolution kernels are defined for each image, they are used
 to convolve the image to match the target PSF.  Any input image that
+has a kernel match $\chi^2$ value greater than 4.0$\sigma$ larger than
+the median value is rejected from the stack.  Each image also has a
+weight assigned, based on the image variance after convolution.  A
+full image weight is then calculated for each input, with the weight,
+$W_\mathrm{input}$ is equal to the inverse of the median of the image
+variance multiplied by the peak of the image covariance (due to the
+warping process).
+has a kernel match $\chi^2$ value (defined as the sum of the RMS error
+across the kernel) 4.0$\sigma$ or larger than the median
+value is rejected from the stack.  Each image also has a weight
+assigned, based on the image variance after convolution.  A full image
+weight is then calculated for each input, with the weight,
+$W_\mathrm{input}$ equal to the inverse of the median of the image
+variance multiplied by the peak of the image covariance (from the
+warping process).  This ensures that low signal-to-noise images are
+down-weighted in the final combination.
 Following the convolution, an initial stack is constructed.  For a
 given pixel coordinate, the values at that coordinate are extracted
+from all input images.  Images that have a suspect mask bit (including
+the SUSPECT, BURNTOOL, SPIKE, STREAK, STARCORE, and CONV.POOR bit
+values) are appended to a suspect pixel list for preferential
+exclusion.  Following this, the pixel values are combined and tested
+to attempt to identify discrepant input values that should be excluded.
+from all input images, with pixels masked excluded from consideration.
+Images that only have a suspect mask bit (including the SUSPECT,
+BURNTOOL, SPIKE, STREAK, STARCORE, and CONV.POOR bit values) are
+appended to a suspect pixel list for preferential exclusion.
+Following this, the pixel values are combined and tested to attempt to
+identify discrepant input values that should be excluded.
 If only a single input is available, the initial stack contains the
 value from that single input.  If there are only two inputs, the
 average of the two is used.  These cases should occur only rarely in
 the $3\Pi$ survey, as there are many input exposures that overlap each
+the $3\pi$ survey, as there are many input exposures that overlap each
 point on the sky.  For the more common case of three or more inputs, a
 weighted average from the inputs is used, with the weight for each
 …
 there were no valid inputs, in which case the BLANK mask bit is set.
 Due to the various non-astronomical ghosts that can occur on GPC1, and
 the fact that they may not be fully masked to ensure all bad pixels
+are removed, it is expected that some of the inputs for a given stack
+pixel are not in agreement with the others.  In general, there is the
+population of input pixel values around the correct astronomical
+level, as well as possible populations at lower pixel value (such as
 due to an over-subtracted burntool trail) and at higher pixel values
+(such as that caused by an incompletely masked optical ghost).  Due to
 the observation strategy to image a given field twice to allow for
+Due to uncorrected artifacts that can occur on GPC1, and the fact that
+they may not be fully masked to ensure all bad pixels are removed, it
+is expected that some of the inputs for a given stack pixel are not in
+agreement with the others.  In general, there is the population of
+input pixel values around the correct astronomical level, as well as
+possible populations at lower pixel value (such as due to an
+over-subtracted burntool trail) and at higher pixel values (such as
+that caused by an incompletely masked optical ghost).  Due to the
+observation strategy to observe a given field twice to allow for
 warp-warp difference images to be constructed to identify transient
 detections, higher pixel values that come from sources like optical
+ghosts that depend on the telescope pointing will come in pairs as well.
+The higher pixel value contaminants are also potentially problematic
+as they may appear to be real sources, prompting photometry to be
+performed on false objects.  Because of the expectation that there are
+more bright contaminants than faint ones, there is a slight preference
+to reject higher pixel values than lower pixel values.
+ghosts that depend on the telescope pointing will come in pairs.
+Detector artifacts will  appear in pairs as well.  The higher
+pixel value contaminants are also potentially problematic as they may
+appear to be real sources, prompting photometry to be performed on
+false objects.  Because of the expectation that there are more positive
+deviations than negative ones, there is a slight preference to  reject
+higher pixel value outliers than lower pixel values, as described below.
 Following the initial combination, a ``testing'' loop iterates in an
 …
 and both are flagged for rejection
 If the number of inputs is larger than 6, then a Gaussian mixture
 model analysis is run on the inputs to fit two sub populations, and
 determine an the likelihood that the distribution is best described by
+If the number of input pixels is larger than 6, then a Gaussian mixture
+model analysis is run on the inputs fitting two sub populations, to
+determine the likelihood that the distribution is best described by
 an uni-modal model.  If this probability is less than $5\%$, then the
 mean is taken from the bimodal sub population with the largest
 …
 comprised of high pixel value outliers.
 If this is not the case, and the distribution is likely unimodal, or
+if there are insufficient inputs for this mixture model analysis, the
 input values are passed to an Olympic weighted mean calculation.  We
+reject $20\%$ of the number of inputs through this process.  The
+number of bad inputs is set to $N_\mathrm{bad} = 0.2 *
+N_\mathrm{input} + 0.5$, with the 0.5 term ensuring at least one input
+is rejected.  This number is further separated into the number of low
+values to exclude $N_\mathrm{low} = N_\mathrm{bad} / 2$, which will
+default to zero if there are few inputs, and $N_\mathrm{high} =
+N_\mathrm{input} + N_\mathrm{low} - N_\mathrm{bad}$.  After sorting
+the input values to determine which values fall into the low and high
+groups, the remaining input values are used in a weighted mean using
 the image weights above.
+If the unimodal probability is greater than $5\%$ (indicating the
+distribution is likely to be unimodal), or if there are insufficient
+inputs for this mixture model analysis, the input values are passed to
+an Olympic weighted mean calculation.  We reject $20\%$ of the number
+of inputs through this process.  The number of bad inputs is set to
+$N_\mathrm{bad} = 0.2 * N_\mathrm{input} + 0.5$, with the 0.5 term
+ensuring at least one input is rejected.  This number is further
+separated into the number of low values to exclude, $N_\mathrm{low} =
+N_\mathrm{bad} / 2$, which will default to zero if there are few
+inputs, and $N_\mathrm{high} = N_\mathrm{low} - N_\mathrm{bad}$.
+After sorting the input values to determine which values fall into the
+low and high groups, the remaining input values are used in a weighted
+mean using the image weights above.
 A systematic variance term is necessary to correctly scale how
 discrepant points can be from the ensemble mean.  If the mixture model
 analysis was run, the Gaussian sigma from the largest sub population
+is squared and used.  If this is not available, a $10\%$ systematic
+error on the input values is used.  Each point then has a limit
 calculated using a $4\sigma$ rejection
+analysis has been run, the Gaussian sigma from the largest sub
+population is squared and used.  Otherwise, a $10\%$ systematic error
+on the input values is used.  Each point then has a limit calculated
+using a $4\sigma$ rejection
 \begin{eqnarray}
 …
 \mathrm{mean})^2$ exceeding this limit is identified.  If there are
 suspect pixels in the set, those pixels are marked for rejection,
 otherwise this worst pixel is marked for rejection.  Following this,
 the combine and test loop is repeated for until no more pixels are
+otherwise this worst pixel is marked for rejection.  Following this step,
+the combine and test loop is repeated until no more pixels are
 rejected, up to a maximum number of iterations equal to $50\%$ of the
 number of inputs.
 …
 the entire image is rejected as it likely has some systematic issue.
 Finally, a second pass at rejecting pixels is conducted, by growing the
 current list to include pixels that are neighbors to many rejected
+Finally, a second pass at rejecting pixels is conducted, by extending
+the current list to include pixels that are neighbors to many rejected
 pixels.  The ISIS kernel used in the previous step is again used to
 determine the largest square box that contains under the limit of
+determine the largest square box that does not exceed the limit of
 $0.25 * \sum_{x,y} kernel^2$.  This square box is then convolved with
 the rejected pixel mask to reject the neighboring pixels.  This final
 …
 a map of the number of inputs per pixel.
+These convolved stack products are not retained, as the convolution
+reduces the resolution of the final image.  Instead, we apply the
+normalizations and rejected pixel maps generated from the convolved
+stack process to the original unconvolved input images.  This produces
+an unconvolved stack that has the optimum image quality possible from
+the input images.  Not convolving does mean that the PSF shape changes
+across the image, as the different PSF widths of the input images
+print through in the different regions to which they have contributed.
+These convolved stack products are not retained, as the convolution is
+used to ensure that the pixel rejection uses seeing-matched images.
+This prevents any differences in the input PSF shape from skewing the
+input pixel rejection.  We apply the normalizations and rejected pixel
+maps generated from the convolved stack process to the original
+unconvolved input images.  This produces an unconvolved stack that has
+the optimum image quality possible from the input images.  Not
+convolving does mean that the PSF shape changes across the image, as
+the different PSF widths of the input images print through in the
+different regions to which they have contributed.
 %% Asinh compression
+Due to the expected large range of data values in the final stacked
+image, saving them as compressed 16-bit integer images with linear
+BSCALE and BZERO scaling values is likely to offer poor
+reconstructions of the stacked image.  This will lead either to
+truncation of the extrema of the image, or quantized values that are
+poorly spaced for the image histogram.  Saving the images as 32-bit
+floating point values would alleviate this quantization issue, at the
+cost of a large increase in the disk space required for the stacked
+images.
+Transforming the data prior to writing to disk by taking the logarithm
+of the pixel values can resolve this, with the complication that all
+data values must first be made positive, which then sets the highest
+quantization sampling near the lowest values in the image.  Following
+techniques used by SDSS \citep{2000AJ....120.1579Y}, we have instead opted to use the
+inverse hyperbolic sine function to transform the data.  The domain of
+this function allows any input value to be converted.  In addition,
+the quantization sampling can be tuned by placing the zero of the
+inverse hyperbolic sine function at a value where the highest sampling
+is desired.
+While IPP image products from single exposures use compressed 16-bit
+integer images, this dynamic range is insufficient for the expected
+scale of the stacked images.  This will lead either to truncation of
+the extrema of the image, or quantized values that poorly sample the
+image noise distribution.  Saving the images as 32-bit floating point
+values would alleviate this quantization issue, at the cost of a large
+increase in the disk space required for the stacked images.
+Inspired by techniques used by SDSS \citep{1999AJ....118.1406L}, we
+use the inverse hyperbolic sine function to transform the data.  The
+domain of this function allows any input value to be converted.  In
+addition, the quantization sampling can be tuned by placing the zero
+of the inverse hyperbolic sine function at a value where the highest
+sampling is desired.
 Formally, prior to being written to disk, the pixel values are
 transformed by $C = \alpha \asinh\left(\frac{L - \mathrm{BOFFSET}}{2.0
   \cdot \mathrm{BSOFTEN}}\right)$, where $L$ are the linear input
 pixel values, $C$ the transformed values, $\alpha = 2.5 \log_{10}(e)$.
+pixel values, $C$ the transformed values, and $\alpha = 2.5 \log_{10}(e)$.
 BOFFSET centers the transformed values, and the mean of the linear
 input pixel values is used.  BSOFTEN controls the stretch of the
 …
 \label{sec:diffs}
+Constructing difference images is essentially the same as that used in
+the stacking process.  An image is chosen as a template, another image
 as the input, and after matching sources to determine the scaling and
 transparency, convolution kernels are defined that are used to
+convolve one or both of the images to a target PSF.  The images are
 then subtracted, and as they should now share a common PSF, static
 sources are largely subtracted (completely in an ideal case), whereas
+sources that are not static between the two images leave a significant
+remnant.  More information on the difference image construction is
+contained in \citet{price2017}.  The follow section contains a
+overview of the difference image construction used for the data in
 DR2.
+The image matching process used in constructing difference images is
+essentially the same as for the stacking process.  An image is chosen
+as a template, another image as the input, and after matching sources
+to determine the scaling and transparency, convolution kernels are
+defined that are used to convolve one or both of the images to a
+target PSF.  The images are then subtracted, and as they should now
+share a common PSF, static sources are largely subtracted (completely
+in an ideal case), whereas sources that are not static between the two
+images leave a significant remnant.  More information on the
+difference image construction is contained in \citet{price2017}.  The
+following section contains an overview of the difference image
+construction used for the data in DR2.
 The images used to construct difference images can be either
 …
 minus stack) and inverse (stack minus warp) to allow for the
 photometry of the difference image to detect sources that both rise
 and fall relative to the stack.  Note that the convolution process
+grows the mask fraction of pixels relative to the warp (the largest
+source of masked pixels in these warp stack differences).  Any pixel
 that after convolution has any contribution from a masked pixel is
 masked as well, ensuring only fully unmasked pixels are used.
+and fall relative to the stack.  The convolution process grows the
+mask fraction of pixels relative to the warp (the largest source of
+masked pixels in these warp stack differences).  Any pixel that after
+convolution has any contribution from a masked pixel is masked as
+well, ensuring only fully unmasked pixels are used.
 For warp-warp differences, such as those used for the ongoing Solar
+System moving object search in nightly observations \citep{2013PASP..125..357D}, the
+warp that was taken first is used as the template.  As there is less
+certainty in which of the two input images will have better seeing, a
+``dual'' convolution method is used.  Both inputs are convolved to a
+target PSF that is not identical to either input.  This intermediate
+target is essential for the case in which the PSFs of the two inputs
+have been distorted in orthogonal directions.  Simply convolving one
+to match the other would require some degree of deconvolution along
+one axis.  As this convolution method by necessity uses more free
+parameters, the ISIS kernels used are chosen to be simpler than those
+used in the warp-stack differences.  The ISIS widths are kept the same
+(1.5, 3.0, 6.0 pixel FWHMs), but each Gaussian kernel is constrained
+to only use a second-order polynomial.  As with the warp-stack
+differences, the mask fraction grows between the input warp and the
+final difference image due to the convolution.  For the warp-warp
+differences, each image mask grows based on the appropriate
+convolution kernel, so the final usable image area is highly dependent
+on ensuring that the telescope pointings are as close to identical as
+possible.  The observing strategy to enable this is discussed in more
+detail in \citet{chambers2017}.
+\section{Discussion}
+System moving object search in nightly observations
+\citep{2013PASP..125..357D}, the warp that was taken first is used as
+the template.  As there is less certainty in which of the two input
+images will have better seeing, a ``dual'' convolution method is used.
+Both inputs are convolved to a target PSF that is not identical to
+either input.  This intermediate target is essential for the case in
+which the PSFs of the two inputs have been distorted in orthogonal
+directions.  Simply convolving one to match the other would require
+some degree of deconvolution along one axis.  As this convolution
+method by necessity uses more free parameters, the ISIS kernels used
+are chosen to be simpler than those used in the warp-stack
+differences.  The ISIS widths are kept the same (1.5, 3.0, 6.0 pixel
+FWHMs), but each Gaussian kernel is constrained to only use a
+second-order polynomial.  As with the warp-stack differences, the mask
+fraction grows between the input warp and the final difference image
+due to the convolution.  For the warp-warp differences, each image
+mask grows based on the appropriate convolution kernel, so the final
+usable image area is highly dependent on ensuring that the telescope
+pointings are as close to identical as possible.  The observing
+strategy to enable this is discussed in more detail in
+\citet{chambers2017}.
+\section{Future Plans}
 \label{sec:discussion}
 …
 dependent on focal plane position.
 An obvious way to make use of the PV3 catalog is to do a statistical
+One obvious way to make use of the PV3 catalog is to do a statistical
 search for electronic crosstalk ghosts that do not match a known rule.
 Given that bright stars do not equally populate all fields, choosing
 …
 There is some evidence that we have not fully identified all of these
 crosstalk rules, based on a study of PV3 images.  For example,
 extremely bright stars may be able to create crosstalk ghosts between the second
+cell column of OTA01 and OTA21, with possibly fainter ghosts appearing
+on OTA11.  Despite the symmetry observed in the main ghost rules,
+there do not appear to be clear examples of a similar ghost between
+OTA47 and OTA66.  Examining this further based on the PV3 catalog
+should provide a clear answer to this, as well as clarify brightness
 limits below which the ghost does not appear.
+extremely bright stars may be able to create crosstalk ghosts between
+the second cell column of OTA01 and OTA21, with possibly fainter
+ghosts appearing on OTA11.  Despite the symmetry observed in the main
+ghost rules, there do not appear to be clear examples of a similar
+ghost between OTA47 and OTA66.  Examining this further based on the
+PV3 catalog should provide a clear answer to this, as well as clarify
+brightness limits below which the ghost does not appear.
 The PV3 catalog may also allow better determination of which date
 ranges we should use to build the dark model.  The date ranges
 currently in use are based on limited sampling of exposures, and do
 not have strong tests indicating that they are the best.  By examining
+not have strong tests indicating that they are optimal.  By examining
 the scatter between the detections on a given exposure and the catalog
 average, we can attempt to look for increases in scatter that might
 …
 to isolate and remove this signal in the Fourier domain.  Preliminary
 investigations have shown that there is a small peak visible in the
 power spectrum of a single cell, but determining the optimal way to
+power spectrum of a single cell, but determining the best way to
 clip this peak to reduce the noise in the image space is not clear.
 …
 The Pan-STARRS1 PV3 processing has reduced an unprecedented volume of
 image data, and has produced a catalog for the $3\Pi$ Survey
+image data, and has produced a catalog for the $3\pi$ Survey
 containing hundreds of billions of individual measurements of
 three billion astronomical objects.  Accurately calibrating

Note: See TracChangeset for help on using the changeset viewer.

Context Navigation

Changeset 40483 for branches/czw_branch/20170908/doc/release.2015/ps1.detrend/detrend.tex

Legend:

branches/czw_branch/20170908

branches/czw_branch/20170908/doc/release.2015/ps1.detrend/detrend.tex

Download in other formats: