IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Ignore:
Timestamp:
Nov 17, 2016, 6:06:50 PM (10 years ago)
Author:
watersc1
Message:

Pre-vacation draft.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/release.2015/ps1.detrend/detrend.tex

    r39799 r39817  
    181181\czwdraft{Should there be a discussion of any header keywords/OTA file formats?}
    182182
    183 Section \ref{sec:detrend construction} provides an overview of the
    184 detrend creation process for GPC1, with details of the application of
    185 those detrends to correct particular issues in Section
    186 \ref{sec:detrending}.  An analysis of the algorithms used to complete
    187 the \ippstage{warp} (section \ref{sec:warping}) and \ippstage{stack}
    188 (section \ref{sec:stacking}) stage transformations of the image data
    189 to from the detector frame to a common sky frame, and the co-adding of
    190 those common sky frame images continues after the list of detrend
    191 steps.  Finally, a discussion of the remaining issues and possible
    192 future improvements is presented in section \ref{sec:discussion}.
     183Section \ref{sec:detrending} provides an overview of the detrending
     184process that corrects the instrumental signatures of GPC1, with
     185details of the construction of those detrends in Section
     186\ref{sec:detrend construction}.  An analysis of the algorithms used to
     187complete the \ippstage{warp} (section \ref{sec:warping}) and
     188\ippstage{stack} (section \ref{sec:stacking}) stage transformations of
     189the image data to from the detector frame to a common sky frame, and
     190the co-adding of those common sky frame images continues after the
     191list of detrend steps.  Finally, a discussion of the remaining issues
     192and possible future improvements is presented in section
     193\ref{sec:discussion}.
    193194
    194195
    … …  
    216217%\section{General Detrend Discussion}
    217218%\label{sec:detrending}
     219
     220
     221\section{GPC1 Detrend Details}
     222\label{sec:detrending}
     223
     224Ensuring a consistent and uniform detector response across the
     225three-degree diameter field of view of the GPC1 camera is essential to
     226a well calibrated survey.  Many standard image detrending steps are
     227done for GPC1, with overscan subtraction removing the detector bias
     228level, dark frame subtraction to remove temperature and exposure time
     229dependent detector glows, and flat field correction to remove pixel to
     230pixel response functions.  We also construct fringe correction for the
     231reddest data in the y filter, to remove the interference patterns that
     232arise in that filter due to the variations in the thickness of the
     233detector surface.
     234
     235These corrections, however, assume that the detector response is
     236linear across the full range of values.  This is not universally the
     237case with GPC1, and this requires an additional set of detrending
     238steps to remove these non-linear responses.  The first of these is the
     239\ippprog{burntool} correction, which removes the persistence trails
     240caused by the incomplete transfer of charge along the readout columns.
     241This bright-end nonlinearity is generally only evident for the
     242brightest stars, as only pixels that are at or beyond the saturation
     243point of the detector have this issue.  More widespread is the
     244non-linearity at the faint end of the pixel range.  Some readout cells
     245and some readout cell edge pixels experience a sag relative to linear
     246at low illumination, such that faint pixels appear fainter than
     247expected.  The correction to this requires amplifying the pixel values
     248in these regions to match the expected model.
     249
     250The final non-linear response issue has no good option for correction.
     251Large regions of some OTA cells experience significant charge transfer
     252issues, making them unusable for science observations.  These regions
     253are therefore masked in processing, with these CTE regions making up
     254the largest fraction of masked pixels on the detector.  Other regions
     255are masked for other regions, such as static bad pixel features or
     256temporary readout masking caused by issues in the camera electronics
     257that make these regions unreliable.  These all contribute to the
     258detector mask, which is augmented in each exposure for dynamic
     259features that are masked based on the astronomical features within the
     260field of view.
     261
     262For the PV3 processing, all detrending is done by the
     263\ippprog{ppImage} program.  This program applies the detrends to the
     264individual cells, and then an OTA level mosaic is constructed for the
     265science image, the mask image, and the variance map image.  The single
     266epoch photometry is done at this stage as well.  The following
     267subsections (\ref{sec:burntool} - \ref{sec:background}) detail these
     268detrending steps, presented in the order in which they are applied to
     269the individual OTA image data.
     270
     271\subsection{Burntool / Persistence effect}
     272\label{sec:burntool}
     273
     274Pixels that approach the saturation point on GPC1, which varies by
     275readout with common values around 60000 DN, cause persistence problems
     276on that and subsequent images.  During the read out process of an
     277image with such a bright pixel, some of the charge associated with it
     278is not fully shifted down the detector column toward the amplifier.
     279As a result, this charge remains in the starting cell, and is
     280partially collected in subsequent shifts, resulting in a ``burn
     281trail'' that extends from the center of the bright source away from
     282the amplifier (vertically along the pixel columns toward the top of
     283the cell).
     284
     285This incomplete charge shifting in nearly full wells continues as each
     286row is read out.  This results in a remnant charge being deposited in
     287the pixels that the full well was shifted through.  In following
     288exposures, this remnant charge leaks out, resulting in a trail that
     289extends from the initial location of the bright source on the previous
     290image towards the amplifier (vertically down along the pixel column).
     291This remnant charge can remain on the detector for up to thirty
     292minutes, requiring the locations of these ``burns'' be retained
     293between exposures.
     294
     295Both of these types of persistance trails are measured and optionally
     296repaired via the \ippprog{burntool} program.  This program does an
     297initial scan of the images, and identifies objects with pixel values
     298brighter than a conservative threshold of 30000 DN.  The trail from
     299the peak of that object is fit with a one-dimensional power law in
     300each pixel column above the threshold, based on empirical evidence
     301that this is the functional form of this persistence effect.  This
     302also matches the expectation that a constant fraction of charge is
     303incompletely transfered at each shift beyond the persistence
     304threshold.  Once this fit is done, the model can be subtracted from
     305the image, and the location of the star is stored in a table along
     306with the exposure PONTIME, which denotes the number of seconds since
     307the detector was last powered on, and provides an internally consistent
     308time scale.
     309
     310For subsequent exposures, the table associated with the previous image
     311is read in, and after correcting trails from the stars on the new
     312image, the positions of the bright stars from the table are used to
     313check for remnant trails on the image.  These are fit and subtracted
     314using a one-dimensional exponential model, again based on empirical
     315studies.  If a significant model is found, then this location is
     316retained in the image output table.  If not, the old burn is allowed
     317to expire.
     318
     319The main concern with this method of correcting the persistance trails
     320is that it is based on fits to the raw image data, which may have
     321other signal sources not determined by the persistence effect.  The
     322presence of other stars or artifacts along the path of the burn can
     323result in a poor model to be fit, resulting in either an over- or
     324under-subtraction of the persistance burn.  For this reason, the image
     325mask is marked with a value indicating that this correction has been
     326applied.  These pixels are not fully excluded, but they are marked as
     327suspect, which allows them to be excluded from consideration in
     328subsequent stages, such as image stacking.
     329
     330Another concern is that the cores of very bright stars are deformed by
     331this process, as the burntool fitting subtracts flux
     332from only one side of the star.  As most stars that result in burns already
     333have saturated cores, they are already ignored for the purpose of
     334PSF determination and are flagged as saturated by the photometry
     335reduction.
     336
     337\begin{figure}
     338  \centering
     339  \begin{minipage}{0.45\hsize}
     340    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt_trail.png}
     341%    \caption{(a)}
     342%  \end{subfigure}%
     343%  \begin{subfigure}[]{.45\hsize}
     344  \end{minipage}%
     345  \begin{minipage}{0.45\hsize}
     346    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt_trail.png}
     347%    \caption{(b)}
     348%  \end{subfigure}
     349  \end{minipage}
     350
     351  \caption{Example of a profile cut along the y-axis through a bright star on exposure o5677g0123o OTA11 in cell xy60 (left panel) and on the subsequent exposure o5677g0124o (right panel).  In both figures, the green points show the image corrected with all appropriate detrending steps, but without burntool applied, illustrating the amplitude of the persistence trails.  The red points show the same data after the burntool correction, which reduces the impact of these features.  Both exposures are in the g-filter with exposure times of 43s}
     352\end{figure}
     353
     354\begin{figure}
     355  \centering
     356  \begin{minipage}{0.45\hsize}
     357    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_nobt.png}
     358%    \caption{(a)}
     359%  \end{subfigure}%
     360%  \begin{subfigure}[]{.45\hsize}
     361  \end{minipage}%
     362  \begin{minipage}{0.45\hsize}
     363    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_nobt.png}
     364%    \caption{(b)}
     365%  \end{subfigure}
     366  \end{minipage}
     367  \begin{minipage}{0.45\hsize}
     368    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt.png}
     369%    \caption{(a)}
     370%  \end{subfigure}%
     371%  \begin{subfigure}[]{.45\hsize}
     372  \end{minipage}%
     373  \begin{minipage}{0.45\hsize}
     374    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt.png}
     375%    \caption{(b)}
     376%  \end{subfigure}
     377  \end{minipage}
     378  \caption{Example of OTA11 cell xy60 on exposures o5677g0123o (left) and o5677g0124o (right).  The top panels show the image with all appropriate detrending steps, but without burntool, and the bottom show the same with burntool applied.  There is some slight over subtraction in fitting the initial trail, but the impact of the trail is greatly reduced in both exposures.}
     379\end{figure}
     380
     381
     382\subsection{Overscan}
     383\label{sec:overscan}
     384
     385Each cell on GPC1 has an overscan region that covers the first 34
     386columns of each row, and the last 10 rows of each column.  No light
     387lands on these pixels, so the image region is trimmed to exclude them.
     388Each row has an overscan value subtracted, calculated by finding the
     389median value of that row's overscan pixels and then smoothing between
     390rows with a three-row boxcar median.
     391
     392\subsection{Non-linearity Correction}
     393\label{sec:nonlinearity}
     394% check notebook, 2010-07/08
     395
     396The pixels of GPC1 are not uniformly linear at all flux levels.  In
     397particular, at low flux levels, some pixels have a tendency to sag
     398relative to the expected linear value.  This effect is most pronounced
     399along the edges of the detector cells, although some entire cells show
     400evidence of this effect.
     401
     402To correct this sag, we studied the flux behavior of a series of flat
     403frames for a ramp of exposure times with approximate logarithmically
     404equal spacing between 0.01s and 57.04s.  As the exposure time
     405increases, the flux on each pixel also increases in what is expected
     406to be a linear manner.  Each of these flat exposures in this ramp is
     407overscan corrected, and then the median is calculated for each cell,
     408as well as for the rows and columns within ten pixels of the edge of
     409the science region.  From these median values at each exposure time
     410value, we can construct the expected trend by fitting a linear model,
     411$f_{region} = G * t_{exp} + B$, to determine the gain, $G$, and the
     412bias, $B$, for the region considered.  This fitting was limited to only
     413the range of fluxes between 12000 and 38000 counts, as these ranges
     414were found to match the linear model well.  This range avoids the
     415non-linearity at low fluxes, as well as the possibility of high-flux
     416non-linearity effects.
     417
     418We store the average flux measurement and deviation from the linear
     419fit for each exposure time for all regions on all detector cells in
     420the linearity detrend look up tables.  When this is applied to science
     421data, these lookup tables are loaded, and a linear interpolation is
     422performed to determine the correction needed for the flux in that
     423pixel.  This look up is performed for both the row and column of each
     424pixel, to allow the edge correction to be applied where applicable,
     425and the full cell correction elsewhere.  The average of these two
     426values is then applied to the pixel value, reducing the effects of
     427pixel nonlinearity.
     428
     429This non-linearity effect appears to be stable in time for the
     430majority of the detector pixels, with little evident change over the
     431survey duration.  However, as the non-linearity is most pronounced at
     432the edges of the detector cells, those are the regions where the
     433correction is most likely to be incomplete.  Because of this fact,
     434most pixels in the static mask with either the DARKMASK or FLATMASK
     435bit set are found along these edges.  As the non-linearity correction
     436is unable to reliably restore these pixels, they produce inconsistent
     437values after the dark and flat have been applied, and are therefore
     438rejected.
     439
     440%% exptime n_included/det_id = 372
     441%% clearly this isn't the one used, as 3-12 spans three data points, poorly.x
     442%% 0.01 2
     443%% 0.14 2
     444%% 0.27 2
     445%% 0.49 2
     446%% 0.72 2
     447%% 1.06 2
     448%% 1.41 2
     449%% 2.02 2
     450%% 2.63 2
     451%% 3.94 2
     452%% 5.25 2
     453%% 8.74 2
     454%% 13.09 2
     455%% 17.4 2
     456%% 20.86 2
     457%% 24.3 2
     458%% 27.78 2
     459%% 31.24 2
     460%% 34.65 2
     461%% 38.12 2
     462%% 42.41 2
     463%% 46.69 2
     464%% 51.89 2
     465%% 57.04 2
     466
     467
     468%http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearity_AllEdges
     469%http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearityArchive
     470
     471\begin{figure}
     472  \centering
     473  \includegraphics[width=0.9\hsize,angle=0,clip]{images/linearity_XY27_xy16.png}
     474  \caption{Example plot of the linearity correction as a fraction of observed flux for OTA27, cell xy16.}
     475\end{figure}
     476
     477\subsection{Dark/Bias Subtraction}
     478\label{sec:dark}
     479% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Dark_Model
     480
     481The dark model we make for GPC1 considers each pixel individually,
     482independent of any neighbors.  To construct this model, we fit a
     483multi-dimensional model to the array of input pixels from a randomly
     484selected set of 100-150 overscan and non-linearity corrected dark
     485frames chosen from a given date range.  The model fits each pixel as a
     486function of the exposure time $t_{exp}$ and the detector temperature
     487$T_{chip}$ of the input images such that $\mathrm{dark} = a_0 + a_1
     488t_{exp} + a_2 T_{chip} t_{exp} + a_3 T_{chip}^2 t_{exp}$.  This
     489fitting uses two iterations to produce a clipped fit, rejecting at the
     490$3\sigma$ level.  The final coefficients $a_i$ for the dark model are
     491stored in the detrend image.  The constant $a_0$ term includes the
     492residual bias signal after overscan subtraction, and as such, a
     493separate bias subtraction is not necessary.
     494
     495Applying the dark model is simply a matter of calculating the response
     496to the exposure time and detector temperature for the image to be
     497corrected, and subtracting the resulting dark signal from the image.
     498
     499\subsubsection{Time evolution}
     500
     501The dark model is not consistently stable over the full survey, with
     502significant drift over the course of multiple months.  Some of the
     503changes in the dark can be attributed to changes in the voltage
     504settings of the GPC1 controller electronics, but the majority seem to
     505be the result of some unknown parameter.  We can separate the dark
     506model history of GPC1 into three epochs.  The first epoch covers all
     507data taken prior to 2010-01-23.  This epoch used a different header
     508keyword for the detector temperature, making data from this epoch
     509incompatible with later dark models.
     510
     511The second epoch covers data between 2010-01-23 and 2011-05-01, and is
     512characterized by a largely stable but oscillatory dark solution.
     513There are two modes that the dark model switches between apparently at
     514random.  No clear cause has been established for the switching, but
     515there are clear differences between the two modes that require the
     516observation dates to be split to use the model that is most
     517appropriate.
     518
     519The initial evidence of these two modes comes from the discovery of a
     520slight gradient along the rows of certain cells.  This is a result of
     521a drift in the bias level of the detector as it is read out.  An
     522appropriate dark model should remove this gradient entirely.  For
     523these two modes, the direction of this bias drift is different, so a
     524single dark model generated from all dark images in the time range
     525over corrects the positive-gradient mode, and under corrects the
     526negative-gradient mode.  Upon identifying this two-mode behavior, and
     527determining the dates each mode was dominant, two separate dark
     528models were constructed from appropriate ``A'' and ``B'' mode dark
     529frames.  Using the appropriate dark minimizes the effect of this bias
     530gradient in the dark corrected data. 
     531
     532The bias drift gradients of the mode switching can be visualized in
     533Figure \ref{fig:dark switching}.  This figure shows the image profile
     534along the x-pixel axis binned along the full y-axis of the first row
     535of cells.  The raw data is shown, illustrating the positional
     536depenendence the dark signal has on the image values.  In addition,
     537both the correct B-mode dark and incorrect A-mode dark have been
     538applied to this image, showing that although both correct the bulk of
     539the dark signal, using the incorrect mode creates larger intensity
     540gradients.
     541
     542After 2011-05-01, the two-mode behavior of the dark disappears, and is
     543replaced with a slow observation date dependent drift in the magnitude
     544of the gradient.  This drift is sufficiently slow that we have modeled
     545it using three observation date independent dark model for different
     546date ranges.  These darks cover the range from 2011-05-01 to
     5472011-08-01, 2011-08-01 to 2011-11-01, and 2011-11-01 and on.  The
     548reason for this time evolution is unknown, but as it is correctable
     549with a small number of dark models, this does not significantly impact
     550detrending.
     551
     552\begin{figure}
     553  \centering
     554%  \begin{subfigure}[]{.45\hsize}
     555  \begin{minipage}{0.45\hsize}
     556    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_M_OS_NL_XY23_b1.jpg}
     557%    \caption{(a)}
     558%  \end{subfigure}%
     559%  \begin{subfigure}[]{.45\hsize}
     560  \end{minipage}%
     561  \begin{minipage}{0.45\hsize}
     562    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_to_DARK_XY23_b1.jpg}
     563%    \caption{(b)}
     564%  \end{subfigure}
     565  \end{minipage}
     566  \caption{An example of the dark model application to exposure o5677g0123o, OTA23 (2011-04-26, 43s g-filter).  The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, and the detector non-linearity corrected.  The right panel, shows the same exposure with the dark applied in addition to the processing shown on the left.}
     567\end{figure}
     568
     569\begin{figure}
     570  \centering
     571  \includegraphics[width=0.9\hsize,angle=0,clip]{images/B_profile_ex.png}
     572  \caption{Example showing a profile cut across exposure o5676g0195, OTA67 (2011-04-25, 43s g-filter).  The entire first row of cells (xy00-xy07) have had a median calculated along each pixel column on the OTA mosaicked image.  Arbitrary offsets have been applied to shift the curves to not overlap.  The top curve (in purple) shows the initial raw profile, with no dark model applied.  The next curve (in green) shows the smoother profile after applying the correct B-mode dark model.  Applying the incorrect A-mode dark results in the blue curve, which shows a significant increase in gradients across the cells.  The orange curve shows the result of the PATTERN.CONTINUITY correction.  Although this creates a larger gradient across the mosaicked images, it decreases the cell-to-cell level changes.  The final yellow curve shows the final image profile after all detrending and background subtraction, and has not had an offset applied.  The bright source at the cell xy00 to xy01 transition is a result of a large optical ghost, which due to the area covered, increases the median level more than the field stars.}
     573  \label{fig:dark switching}
     574\end{figure}
     575
     576\subsubsection{Video Dark}
     577\label{sec:video_darks}
     578
     579The dark signal is stronger in cell corners due to glow from the
     580read-out amplifiers.  The standard dark model corrects this for most
     581observations.  However, as mentioned above, when a cell is repeatedly
     582read in video mode, the dark model for the OTA containing it changes.
     583Surprisingly, added reads for the video cell do not amplify the
     584amplifier glow, but rather decrease the dark signal in these regions.
     585As a result, using the standard dark model on the data for these OTAs
     586results in oversubtraction of the corner glow.
     587
     588Video darks have been constructed to eliminate the effect this
     589observational change has on the final image quality.  This was done by
     590running the standard dark construction process on a series of dark
     591frames that have had the video signal enabled for some cells.  GPC1
     592can only run video signals on a subset of the OTAs at a given time.
     593This requires two passes to enable the video signal across the full
     594set of OTAs that support video cells.  This is convenient for the
     595process of creating darks, as those OTAs that do not have video
     596signals enabled create standard dark models, while the video dark is
     597created for those that do.
     598
     599This simultaneous construction of video and standard dark models is
     600useful, as it provides the ability to isolate the response on the
     601standard dark from the video signals.  Isolating this response is
     602essential for attempting to create archival video darks.  We only have
     603raw video dark frame data after 2012-05-16, when this problem was
     604initially identified, so any data prior to that can not be directly
     605corrected for the video dark signal.  Isolating the video signal
     606response allows linear corrections to the pre-existing standard dark
     607models for archival data.  Testing this shows that constructing a
     608video dark for older data simply as $VD_{2009} = D_{2009} - D_{Modern}
     609+ VD_{Modern}$ produces a satisfactory result that does not
     610oversubtract the amplifier glow.  This is shown in figure
     611\ref{fig:video_darks}, which shows video cells from before 2012-05-16,
     612corrected with both the standard and video darks, with the early video
     613dark constructed in such a manner.
     614
     615\begin{figure}
     616  \centering
     617%  \begin{subfigure}[]{.45\hsize}
     618  \begin{minipage}{0.45\hsize}
     619    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_Rdark_XY22_b1.jpg}
     620%    \caption{(a)}
     621%  \end{subfigure}%
     622%  \begin{subfigure}[]{.45\hsize}
     623  \end{minipage}%
     624  \begin{minipage}{0.45\hsize}
     625    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_VDdark_XY22_b1.jpg}
     626%    \caption{(b)}
     627%  \end{subfigure}
     628  \end{minipage}
     629  \caption{An example of the video dark model application to exposure o5677g0123o, OTA22 (2011-04-26, 43s g-filter), which has a video cell located in cell xy16.  The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, the detector non-linearity corrected, and a regular dark applied.  The right panel, shows the same exposure with a video dark applied instead of the standard dark.  The main impact of this change is the improved correction of the corner glows, which are oversubtracted with the standard dark.}
     630  \label{fig:video_darks}
     631\end{figure}
     632
     633\subsection{Noisemap}
     634\label{sec:noisemap}
     635
     636Based on a study of the positional dependence of all detected sources,
     637we have discovered that the cells in GPC1 do not have uniform noise
     638characteristics.  Instead, there is a gradient along the pixel rows,
     639with the noise generally higher away from the read out amplifier
     640(higher cell x pixel positions).  This is likely an effect of the
     641row-by-row bias issue discussed below.  This gradient causes the read
     642noise to increase as the row is read out.  As a result of this
     643increased noise, more sources are detected in the higher noise regions
     644when the read noise is assumed constant across the readout.  To
     645mitigate this noise gradient, we constructed an initial set of
     646noisemap images by measuring the median variance on bias frames.  The
     647variance is calculated in boxes of 20x20 pixels, and then linearly
     648interpolated to cover the full image.
     649
     650Unfortunately, due to correlations within this noise, the variance
     651measured from the bias images does not fully remove the positional
     652dependence of objects that are detected.  This simple noisemap
     653underestimates the noise observed when the image is filtered during
     654the object detection process.  This filtering convolves the background
     655noise with a PSF, which has the effect of amplifying the correlated
     656peaks in the noise.  This amplification can therefore boost background
     657fluctuations above the threshold used to select real objects,
     658contaminating the final object catalogs.
     659
     660In the detection process, we expect false positives at a rate equal to
     661the one-tailed probability beyond the detection threshold.  For these
     662tests, only detections measured at the $\sigma_{thresh} = 5\sigma$
     663level are used, to match that used in the photometry on science data.
     664This probability can be converted into a number of false number by
     665considering a given area.  As the detections must be isolated to not
     666be detected as an extended object, this area must be reduced by the
     667area a given PSF occupies.  Combining this, we find that we expect a
     668probability $P = 1 - \Phi_{normal}(5) = \frac{1}{2}
     669\erfcinv\left(\frac{5}{\sqrt{2}}\right)$, and an area given $N$
     670exposures of area $X\times Y$, $A = \frac{X \times Y \times
     671  N}{A_{PSF}}$.  For a typical $1"$ seeing, $A_{PSF}$ is approximately
     67216 pixels.  Using this model for the false positives, we found that
     673the added read noise was insufficient to account for the observed
     674false positive rate.  Inverting this relation, we can measure
     675$\sigma_{obs}$, the true threshold level based on the number of false
     676positives observed.  This $\sigma_{obs}$ is the combined to form a
     677boost factor $B = \sigma_{thresh} / \sigma_{obs}$ that amplifies the
     678  noisemap to match the observed false detection rate.
     679
     680The row-to-row variations that contribute to the extra noise are
     681related to the dark model, and because of this, as the dark model
     682changes, the effective noise also changes.  To ensure that the
     683noisemap accurately matches the true noise level, we have created
     684different noisemap models for the three major time ranges of the dark
     685model.  We do not see any strong evidence that the noisemaps have the
     686A/B modes visible in the dark, and so we do not generate different
     687models for each individual dark model.  The additional pixel-to-pixel
     688variance from this noisemap is added to the Poissonian variance to
     689form the science variance image generated by the \ippstage{chip}
     690processing.
     691
     692\subsection{Flat}
     693
     694Determining a flat field correction for GPC1 is a challenging
     695endeavor, as the wide field of view makes it difficult to construct a
     696uniformly illuminated image.  Using a dome screen is not possible, as
     697the variations in illumination and screen rigidity create large
     698scatter between different images that are not caused by the detector
     699response function.  Because of this, we use sky flat images taken at
     700twilight, which are more consistently illuminated than screen flats.
     701We calculate the mean of these images to determine the initial flat
     702model.
     703
     704From this starting skyflat model, we construct a photometric
     705correction to remove the effect of the illumination differences over
     706the detector surface.  This is done by dithering a series of science
     707exposures with a given pointing.  By fully calibrating these exposures
     708with the initial flat model, and then comparing the measured fluxes
     709for the same star as a function of position on the detector, we can
     710determine position dependent scaling factors.  From the set of scaling
     711factors for the full catalog of stars observed in the dithered
     712sequence, we can construct a model of the error in the initial flat
     713model as a function of detector position.  Applying a correction that
     714reduces the amplitude of these errors produces a flat field model that
     715better represents the true detector response.
     716
     717\czwdraft{EAM: the flat-field construction part needs to make a clearer discussion of
     718the skyflat vs the photometric correction (photflat) built initially for
     719the survey vs the flat-field corrections determined in the database as part
     720of ubercal (for the latter, you should just mention the concept -- it will
     721also be mentioned in the calibration paper).  The statement that the
     722flat-field response was stable is not true since we did need 5 'seasons'.}
     723
     724In addition to this flat field applied to the individual images, the
     725ubercal process used to calibrate the database of all detections
     726\citep{ubercal} constructs internal ``flat field'' corrections.
     727Although a single set of image flat fields was used for the entire PV3
     728survey, five separate ``seasons'' of database flat fields were needed
     729to ensure proper calibration.  This indicates that the flat field
     730response is not completely fixed in time.
     731
     732\subsection{Pattern correction}
     733\label{sec:pattern}
     734
     735Due to detector specific issues that are not cleanly removed by the
     736dark model, we have a set of ``pattern'' corrections that are applied
     737to some selection of the OTAs in the camera.  This is done to reduce
     738the effect that detector differences have on the measured astronomical
     739signal that are not stable enough to be corrected with a static model.
     740Because of this, the pattern corrections attempt to identify and
     741correct the detector issues based on appropriate filtering the
     742individual science exposures.
     743
     744The PATTERN.ROW correction is used to remove any remaining row-by-row
     745bias variation, and the PATTERN.CELL and PATTERN.CONTINUITY
     746corrections attempt to ensure that the cells of a given OTA are
     747consistent with the other cells on that OTA. 
     748
     749\subsubsection{Pattern Row}
     750% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Bias_Pattern_Study
     751As discussed above in the dark and noisemap sections, certain
     752detectors have significant bias offsets between adjacent rows, caused
     753by noise in the camera control electronics.  The magnitude of these
     754offsets increases as the distance from the readout amplifier
     755increases, resulting in horizontal streaks that are more pronounced
     756along the large x pixel edge of the cell.  As the level of the offset
     757is apparently random between exposures, the dark correction cannot
     758fully remove this structure from the images, and the noisemap value
     759only indicates the level of the average variance added by these bias
     760offsets.  Therefore, we apply the PATTERN.ROW correction in an attempt
     761to mitigate the offsets and correct the image values.  To force the
     762rows to agree, a second order clipped polynomial is fit to each row in
     763the cell.  Four fit iterations are run, and pixels $2.5\sigma$ deviant
     764are excluded from subsequent fits, to minimize the effect stars and
     765other astronomical signals have.  This final trend is then subtracted
     766from that row.  Simply doing this subtraction will also have the
     767effect of removing the background sky level.  To prevent this, the
     768constant and linear terms for each row are stored, and linear fits are
     769made to these parameters as a function of row, perpendicular to the
     770initial fits.  This produces a plane that is added back to the image
     771to restore the background offset and any linear ramp that exists in
     772the sky.
     773
     774This correction was required on all cells on all OTAs prior to
     7752009-12-01, at which point a modification of the camera electronics
     776reduced the scale of the row-by-row offsets for the majority of the
     777OTAs.  As a result, we only apply this correction to the cells where
     778it is still necessary, as shown in Figure \ref{fig: pattern row
     779  cells}.  A list of these cells is listed in Table
     780\ref{tab:pattern_row_cells}.
     781
     782Although this correction does largely resolve the row-by-row offset
     783issue in a satisfactory way, large and bright astronomical objects can
     784bias the fit significantly.  This results in an oversubtraction of the
     785offset near these objects.  As the offsets are calculated on the pixel
     786rows, this oversubtraction is not uniform around the object, but is
     787preferentially along the horizontal x axis of the object.  Most
     788astronomical objects are not significantly distorted by this, with
     789this only becoming on issue for only bright objects comparable to the
     790size of the cell (598 pixels = 150").
     791
     792%% \czwdraft{keep this?}  This row-by-row offset is visible in similar
     793%% camera designs, and has been removed by identifying the noise signal
     794%% in the pixel data stream.  By taking the FFT of the pixels and a
     795%% reference signal, the frequency of this noise can be isolated and
     796%% removed, resulting in a much cleaner image.  However, GPC1 does not
     797%% record the value of the reference signal, instead automatically
     798%% subtracting it from the data values.  Without this comparison signal,
     799%% we have been unable to reproduce this method, as there is no obvious
     800%% FFT component visible.
     801
     802\begin{deluxetable}{lcccc}
     803  \tablecolumns{3}
     804  \tablewidth{0pc}
     805  \tablecaption{Cells which have PATTERN.ROW correction applied}
     806  \tablehead{\colhead{OTA} & \colhead{Cell columns} & \colhead{Additional cells}}
     807  \startdata
     808  OTA11 &  & xy02, xy03, xy04, xy07 \\
     809  OTA14 &  & xy23 \\
     810  OTA15 & 0 & \\
     811  OTA27 & 0, 1, 2, 3, 7 & \\
     812  OTA31 & 7 & \\
     813  OTA32 & 3, 7 & \\
     814  OTA45 & 3, 7 & \\
     815  OTA47 & 0, 3, 5, 7 & \\
     816  OTA57 & 0, 1, 2, 6, 7 & \\
     817  OTA60 &  & xy55 \\
     818  OTA74 & 2, 7 & \\
     819  \enddata
     820  \label{tab:pattern_row_cells}
     821\end{deluxetable}
     822
     823\begin{figure}
     824  \centering
     825  \includegraphics[width=0.9\hsize,angle=0,clip]{images/pattern_row_edit.png}
     826  \caption{Diagram illustrating in red which cells on GPC1 require the PATTERN.ROW correction to be applied.  The footprint of each OTA is outlined, and cell xy00 is marked with either a filled box or an outline.  The labeling of the non-existent corner OTAs is provided to orient the focal plane.}
     827  \label{fig: pattern row cells}
     828\end{figure}
     829
     830\begin{figure}
     831  \centering
     832  \begin{minipage}{0.45\hsize}
     833    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_nopat.png}
     834%    \caption{(a)}
     835%  \end{subfigure}%
     836%  \begin{subfigure}[]{.45\hsize}
     837  \end{minipage}%
     838  \begin{minipage}{0.45\hsize}
     839    \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_pat.png}
     840%    \caption{(b)}
     841%  \end{subfigure}
     842  \end{minipage}
     843  \caption{Example of the PATTERN.ROW correction on exposure o5379g0103o OTA57 cell xy00 (i-filter 45s).  The left panel shows the cell with all appropriate detrending except the PATTERN.ROW, and the right shows the same cell with PATTERN.ROW applied.  The correction reduces the correlated noise on the right side, which is most distant from the read out amplifier.  There is a slight over subtraction along the rows near the bright star.}
     844\end{figure}
     845
     846\subsubsection{Pattern Continuity}
     847
     848As the PATTERN.CELL correction was insufficient in many situations, we
     849designed a replacement correction that would reduce the background
     850distortion for large objects.  In addition, studies of the background
     851level illustrated that the row-by-row bias can introduce small
     852background gradient variations along the rows of the cells that is not
     853stable enough to be completely fit by the dark model.  This common
     854feature across the columns of cells results in a ``saw tooth'' pattern
     855horizontally across an OTA, and as the background model fits a smooth
     856sky level, this induces over and under subtraction at the cell
     857boundaries.  As the PATTERN.CELL was designed to correct changes only
     858in the median value between cells, it could not adequately resolve
     859this higher order issue.
     860
     861The replacement for PATTERN.CELL is the PATTERN.CONTINUITY correction,
     862which attempts to match the edges of a cell to those of its neighbors.
     863For each cell, a thin box 10 pixels wide on each edge is extracted and
     864the median value of unmasked values calculated for that box.  These
     865median values are then used to construct a vector of differences
     866$\Delta_i = \sum_{j} Edge_{i} - Edge_{j}$, along with a matrix of
     867associations $A_{i,i'} = \sum_{j} \delta(i,j) \delta(j,i')$ denoting
     868which cell boundaries are adjacent.  By solving the system $A x =
     869diff$, we find the set of offsets $x_i$ to be applied to each cell to
     870ensure the minimum differences between all cell edges and their
     871neighbors.
     872
     873For OTAs that initially show the saw tooth pattern, the effect of this
     874correction is to align the cells into a single ramp, at the expense of
     875the absolute background level.  However, as we subtract off a smooth
     876background model prior to doing photometry, these deviations from an
     877absolute sky level are unimportant.  The fact that the final ramp is
     878smoother than it would be otherwise also allows for the background
     879subtracted image to more closely match the astronomical sky, without
     880significant errors at cell boundaries.  An example of the effect of
     881this correction on an image profile is shown in Figure \ref{fig:dark switching}.
     882
     883%% \begin{figure}
     884%%   \centering
     885%%   \caption{Continuity example, with background issue.}
     886%%   \label{fig: continuity example}
     887%% \end{figure}
     888
     889\subsection{Fringe correction}
     890\label{sec:fringe}
     891% det_id 296 is the fringe we use.
     892
     893Due to variations in the thickness of the detectors, we observe
     894interference patterns at the infrared end of the filter set, as the
     895wavelength of the light becomes comparable to the thickness of the
     896detectors.  Visually inspecting the images shows that the fringing is
     897most prevalent in the y filter images, with negligible fringing in the
     898other bands.  As a result of this, we only apply a fringe correction
     899to the y filter data.
     900
     901The fringe used for PV3 processing was constructed from a set of 20
     902120s science exposures.  These exposures are overscan subtracted, and
     903corrected for non-linearity, and have the dark and flat models
     904applied.  These images are smoothed with a Gaussian kernel with
     905$\sigma = 2$ pixels to minimize pixel to pixel noise.  The fringe
     906image data is then constructed by calculating the clipped mean of the
     907input images with two iteration of clipping at the $3\sigma$ level.
     908
     909A course background model for each cell is constructed by calculating
     910the median on a 3x3 grid (approximately 200x200 pixels each).  A set
     911of 1000 randomly selected points are then selected on the fringe image
     912for each cell, and a median calculated for this position in a 10x10
     913pixel box, with the background level subtracted.  These sample
     914locations provide scale points to allow the amplitude of the measured
     915fringe to be compared to that found on science images.
     916
     917To apply the fringe, the same sample locations are measured on the
     918science image to determine the relative strength of the fringing in
     919that particular image.  A least squares fit between the fringe
     920measurements and the corresponding measurements on the science image
     921provides the scale factor multiplied to the fringe before it is
     922subtracted from the science image.
     923
     924\begin{figure}
     925  \centering
     926  \begin{minipage}{0.5\hsize}
     927    \includegraphics[width=1.0\hsize,angle=0,clip]{images/o5220g0025o_XY53_nofringe.png}
     928%    \caption{(a)}
     929%  \end{subfigure}%
     930%  \begin{subfigure}[]{.45\hsize}
     931  \end{minipage}%
     932  \begin{minipage}{0.5\hsize}
     933    \includegraphics[width=1.0\hsize,angle=0,clip]{images/o5220g0025o_XY53_fringe.png}
     934%    \caption{(b)}
     935%  \end{subfigure}
     936  \end{minipage}
     937  \caption{Example of the y-filter fringe pattern on exposure o5220g0025o OTA53 (y-filter 30s).  The left panel shows the OTA mosaic with all detrending except the fringe correction, while the right shows the same including the fringe correction.  Both images have been smoothed with a Gaussian with $\sigma = 3$ pixels to highlight the faint and large scale fringe patterns. \czwdraft{See if there's a way to have mana produce images larger than the screen size.}}
     938  \label{fig: fringe example}
     939\end{figure}
     940
     941\subsection{Masking}
     942\label{sec:masking}
     943
     944\subsubsection{Static Masks}
     945\label{sec:static_masks}
     946
     947Due to the large size of the detector, it is expected that there
     948are a number of pixel defects that do not have the detection
     949sensitivity on par with their neighbors.  To remove these pixels, we
     950have constructed a static mask that identifies the known defects.
     951This mask is constructed in three phases.
     952
     953First, a CTEMASK is constructed to mask out regions in which the
     954charge transfer efficiency is low compared to the rest of the
     955detector.  Twenty-five of the sixty OTAs in GPC1 show some evidence of
     956CTE issues, with this pattern appearing (to varying degrees) in
     957roughly triangular patches on the OTA due to defects in the
     958semiconductor manufacturing.  To generate the mask for these regions,
     959a sample set of 26 evenly illuminated flat field images were measured
     960to produce a map of the image variance in 20x20 pixel bins.  As the
     961flat image is expected to illuminate the image uniformly, the expected
     962variances in each bin should be Poissonian distributed with the flux
     963level.  However, in regions with CTE issues, adjacent pixels are not
     964independent, as the charge in those pixels is more free to spread.
     965This reduces the pixel-to-pixel differences, resulting in a lower than
     966expected variance.  All regions with variance less than half the
     967average image level are added to the static CTEMASK.
     968
     969The next step of mask construction is to examine the flat and dark
     970models, and exclude pixels that appear to be poorly corrected by these
     971models.  The DARKMASK process looks for pixels that are more than
     972$8\sigma$ discrepant in $10\%$ of the 100 input dark frame images
     973after those images have had the dark model applied to them.  These
     974pixels are assumed to be unstable with respect to the dark model, and
     975have the DARK bit set in the static mask, indicating that they are
     976unreliable in scientific observing.  Similarly, the FLATMASK process
     977looks for pixels that are $3\sigma$ discrepant in the same fraction of
     97816 input flat field images after both the dark and flat models have
     979been applied.  Those pixels that do not follow the flat field model of
     980the rest of image are assigned the FLAT mask bit in the static mask,
     981removing the pixels that cannot be corrected to a linear response.
     982
     983The final step of mask construction is to examine the detector for
     984bright columns and other static pixel issues.  This is first done by
     985processing a set of 100 i filter science images in the same fashion as
     986for the DARKMASK.  A median image is constructed from these inputs
     987along with the per-pixel variance.  These images are used to identify
     988pixels that have unexpectedly low variation between all inputs, as
     989well as those that significantly deviate from the global median value.
     990Once this initial set of bad pixels is identified, a $3\times{}3$
     991pixel triangular kernel is convolved with the initial set, and any
     992convolved pixel with value greater than 1 is assigned to the static
     993mask.  This does an excellent job of removing the majority of the
     994problem pixels.  A subsequent manual inspection allows human
     995interaction to identify other inconsistent pixels including the
     996vignetted regions around the edge of the detector. 
     997
     998Figure \ref{fig:static mask} shows an example of the static mask for
     999the full GPC1 field of view.  Table \ref{tab:mask_values} lists the
     1000bit mask values used for the different sources of masking.
     1001
     1002\begin{figure}
     1003  \centering
     1004  \includegraphics[width=0.9\hsize,angle=0,clip]{images/gpc1_mask_indexed.png}
     1005  \label{fig:static mask}
     1006 
     1007  \caption{Image map of the GPC1 static mask.  The CTE regions are clearly visible as roughly triangular patches covering the corners of some OTAs.  Some entire cells are masked, including an entire column of cells on OTA14.  Calcite cells remove large areas from OTA17 AND OTA76.}
     1008\end{figure}
     1009
     1010\begin{deluxetable}{ccl}
     1011  \tablecolumns{3}
     1012  \tablewidth{0pc}
     1013  \tablecaption{GPC1 Mask Values}
     1014  \tablehead{\colhead{Mask Name} & \colhead{Mask Value} & \colhead{Description}}
     1015  \startdata
     1016  DETECTOR & 0x0001 & A detector defect is present. \\
     1017  FLAT     & 0x0002 & The flat field model does not calibrate the pixel reliably. \\
     1018  DARK     & 0x0004 & The dark model does not calibrate the pixel reliably. \\
     1019  BLANK    & 0x0008 & The pixel does not contain valid data. \\
     1020  CTE      & 0x0010 & The pixel has poor charge transfer efficiency. \\
     1021  SAT      & 0x0020 & The pixel is saturated. \\
     1022  LOW      & 0x0040 & The pixel has a lower value than expected. \\
     1023  SUSPECT  & 0x0080 & The pixel is suspected of being bad. \\
     1024  BURNTOOL & 0x0080 & The pixel contain an burntool repaired streak. \\
     1025  CR       & 0x0100 & A cosmic ray is present. \\
     1026  SPIKE    & 0x0200 & A diffraction spike is present. \\
     1027  GHOST    & 0x0400 & An optical ghost is present. \\
     1028  STREAK   & 0x0800 & A streak is present. \\
     1029  STARCORE & 0x1000 & A bright star core is present. \\
     1030  CONV.BAD & 0x2000 & The pixel is bad after convolution with a bad pixel. \\
     1031  CONV.POOR& 0x4000 & The pixel is poor after convolution with a bad pixel. \\
     1032  MARK     & 0x8000 & An internal flag for temporarily marking a pixel. \\
     1033  \enddata
     1034  \label{tab:mask_values}
     1035\end{deluxetable}
     1036
     1037\subsubsection{Dynamic masks}
     1038\label{sec:dynamic_masks}
     1039
     1040In addition to the static mask that removes the constant detector
     1041defects, we also generate a set of dynamic masks that change with the
     1042astronomical features in the image.  These masks are advisory in
     1043nature, and do not completely exclude the pixel from further
     1044processing consideration.  The first of these dynamic masks is the
     1045burntool advisory mask mentioned above.  These pixels are included for
     1046photometry, but are rejected more readily in the stacking and
     1047difference image construction, as they are more likely to have small
     1048deviations due to imperfections in the burntool correction.
     1049
     1050The remaining dynamic masks are not generated until the IPP
     1051\ippstage{camera} stage, at which point all object photometry is
     1052complete, and an astrometric solution is known for the exposure.  This
     1053added information provides the positions of bright sources based on
     1054the reference catalog, including those that fall slightly out of the
     1055detector field of view or within the inter chip gaps, where internal
     1056photometry may not identify them.  These bright sources are the origin
     1057for many of the image artifacts that the dynamic mask identifies and
     1058excludes.
     1059
     1060\subsubsubsection{Electronic crosstalk ghosts}
     1061\label{sec:crosstalk}
     1062
     1063Due to electrical crosstalk between the flex cables connecting the
     1064individual detector OTA devices, ghost objects can be created by the
     1065presence of a bright source at a different position on the camera.
     1066Table \ref{tab:crosstalk_rules} summarizes the list of known crosstalk
     1067rules, with an estimate of the magnitude difference between the source
     1068and ghost.  For all of the rules, any cell $v$ within the specified
     1069column of cells on any of the OTAs in the specified column of OTAs $Y$
     1070creates the ghost in the same $v$ and $Y$ in the target column of
     1071cells and OTAs.  In each of these cases, a source object with an
     1072instrumental magnitude brighter than -14.47 creates a ghost object
     1073many orders of magnitude fainter at the target location.  The cell
     1074(x,y) pixel coordinate is identical between source and ghost, as a
     1075result of the transfer occurring as the devices are read.  A circular
     1076mask is added to the ghost location with radius $R = 3.44 \left(-14.47
     1077- m_{source, instrumental}\right)$ pixels.  Any objects in the
     1078photometric catalog found at the location of the ghost mask have the
     1079GHOST mask bit set, marking the object as a likely ghost.  The
     1080majority of the crosstalk rules are bi-directional, with a source in
     1081either position creating a ghost at the corresponding crosstalk target
     1082position.  The two faintest rules are uni-directional, due to
     1083differences in the electronic path for the crosstalk.
     1084
     1085For the very brightest sources ($m_{instrumental} < -15$), there can
     1086be crosstalk ghosts between all columns of cells during the readout.
     1087These ``bleed'' ghosts were originally identified as ghosts of the
     1088saturation bleeds appearing in the neighboring cells, and as such, the
     1089masking for these objects puts a rectangular mask down from top to
     1090bottom of cells in all columns that are in the same row of cells as
     1091the bright source.  The width of this box is a function of the source
     1092magnitude, with $W = 5 * \left(-15 - m_{source, instrumental}\right)$
     1093pixels.
     1094
     1095\begin{deluxetable}{lllc}
     1096  \tablecolumns{4}
     1097  \tablewidth{0pc}
     1098  \tablecaption{GPC1 Crosstalk Rules}
     1099  \tablehead{\colhead{Type}&\colhead{Source OTA/Cell}&\colhead{Ghost OTA/Cell}&\colhead{$\Delta m$}}
     1100  \startdata
     1101  Inter-OTA & OTA2Y XY3v & OTA3Y XY3v & 6.16 \\
     1102            & OTA3Y XY3v & OTA2Y XY3v &      \\
     1103            & OTA4Y XY3v & OTA5Y XY3v &      \\
     1104            & OTA5Y XY3v & OTA4Y XY3v &      \\
     1105  Intra-OTA & OTA2Y XY5v & OTA2Y XY6v & 7.07 \\
     1106            & OTA2Y XY6v & OTA2Y XY5v &      \\
     1107            & OTA5Y XY5v & OTA5Y XY6v &      \\
     1108            & OTA5Y XY6v & OTA5Y XY5v &      \\
     1109  One-way   & OTA2Y XY7v & OTA3Y XY2v & 7.34 \\
     1110            & OTA5Y XY7v & OTA4Y XY2v &      \\
     1111  \enddata
     1112  \label{tab:crosstalk_rules}
     1113\end{deluxetable}
     1114 
     1115%% \begin{figure}
     1116%%   \centering
     1117%%   \caption{Figure of crosstalk ghost and bright star source.  Plot of cut across ghost to illustrate the flat-top shape.}
     1118%% \end{figure}
     1119
     1120\subsubsubsection{Optical ghosts}
     1121\label{sec:optical_ghosts}
     1122% http://arxiv.org/pdf/1207.2513v1.pdf
     1123
     1124Due to imperfections in the anti-reflective coating on the optical
     1125surfaces of GPC1, bright sources can also result in large out of focus
     1126objects, particularly in the g-filter data.  These objects are the
     1127result of light reflecting back off the surface of the detector,
     1128reflecting again off the lower surfaces of the optics (particularly
     1129the L1 corrector lens), and then back down onto the focal plane.  Due
     1130to the extra travel distance, the resulting source is out of focus and
     1131elongated along the radial direction of the camera focal plane. These
     1132optical ghosts can be modeled in the focal plane coordinates (L,M)
     1133which has its origin at the center of the focal plane.  In this
     1134system, a bright object at location (L,M) on the focal plane creates a
     1135reflection ghost on the opposite side of the optical axis at (-L,-M).
     1136The exact location is fit as a third order polynomial in the focal
     1137plane L and M directions (as listed in Table \ref{tab:ghost_centers}).
     1138An elliptical annulus mask is constructed at the expected ghost
     1139location, with the major and minor axes defined by linear functions of
     1140the ghost distance from the optical axis, and oriented with the
     1141ellipse major axis is along the radial direction (Table
     1142\ref{tab:ghost_radii}).  All stars brighter than a filter-dependent
     1143threshold (listed in Table \ref{tab:ghost_magnitudes}) have such masks
     1144constructed.
     1145
     1146\begin{deluxetable}{lcc}
     1147  \tablecolumns{3}
     1148  \tablewidth{0pc}
     1149  \tablecaption{Optical Ghost Center Transformations}
     1150  \tablehead{\colhead{Polynomial Term}&\colhead{L center}&\colhead{M center}}
     1151  \startdata
     1152  $x^0 y^0$ & -1.215661e+02 &  2.422174e+01 \\
     1153  $x^1 y^0$ &  1.321875e-02 &  4.170486e-04 \\
     1154  $x^2 y^0$ & -4.017026e-09 & -1.934260e-08 \\
     1155  $x^3 y^0$ &  1.148288e-10 & -1.173657e-12 \\
     1156  $x^0 y^1$ & -1.908074e-03 &  1.189352e-02 \\
     1157  $x^1 y^1$ &  8.479150e-08 & -9.256748e-08 \\
     1158  $x^2 y^1$ &  1.635732e-11 &  1.140772e-10 \\
     1159  $x^0 y^2$ &  2.625405e-08 &  8.123932e-08 \\
     1160  $x^1 y^2$ &  1.125586e-10 &  1.328378e-11 \\
     1161  $x^0 y^3$ &  2.912432e-12 &  1.170865e-10 \\
     1162  \enddata
     1163  \label{tab:ghost_centers}
     1164\end{deluxetable}
     1165
     1166\begin{deluxetable}{lcccc}
     1167  \tablecolumns{5}
     1168  \tablewidth{0pc}
     1169  \tablecaption{Optical Ghost Annulus Axis Length}
     1170  \tablehead{\colhead{Radial Order}&\colhead{Inner Major Axis}&\colhead{Inner Minor Axis}&    \colhead{Outer Major Axis}&\colhead{Outer Minor Axis}}
     1171  \startdata
     1172  $r^0$ & 3.926693e+01 & 5.287548e+01 & 7.928722e+01 & 1.314265e+02 \\
     1173  $r^1$ & 5.325759e-03 &-2.191669e-03 & 1.722181e-02 & -2.627153e-03 \\
     1174  \enddata
     1175  \label{tab:ghost_radii}
     1176\end{deluxetable}
     1177
     1178\begin{deluxetable}{lc}
     1179  \tablecolumns{2}
     1180  \tablewidth{0pc}
     1181  \tablecaption{Optical Ghost Magnitude Limits}
     1182  \tablehead{\colhead{Filter}&\colhead{$m_{inst}$}}
     1183  \startdata
     1184  g & -16.5 \\
     1185  r & -20.0 \\
     1186  i & -25.0 \\
     1187  z & -25.0 \\
     1188  y & -25.0 \\
     1189  w & -20.0 \\
     1190  \enddata
     1191  \label{tab:ghost_magnitudes}
     1192\end{deluxetable}
     1193
     1194
     1195\begin{figure}
     1196  \centering
     1197  \includegraphics[width=0.9\hsize,angle=0,clip]{images/full_fpa_ghosts.jpg}
     1198  \caption{Example of the full GPC1 field of view illustrating the sources and destinations of optical ghosts on exposure o5677g0123o (2011-04-26, 43s g-filter).  The bright stars on OTA33 and OTA44 result in nearly circular ghosts on the opposite OTA.  In contrast, the trio of stars on OTA11 result in very elongated ghosts on OTA66.}
     1199\end{figure}
     1200
     1201\subsubsubsection{Optical glints}
     1202\label{sec:glints}
     1203
     1204Prior to \czwdraft{DATE}, a reflective surface at the edge of the
     1205camera aperture was incompletely screened to light passing through the
     1206telescope.  Sources brighter than $m_{inst} = -21$ that fell on this
     1207reflective surface resulted in light being scattered across the
     1208detector surface in a long narrow glint.  This surface was physically
     1209masked on \czwdraft{DATE}, removing the possibility of glints in
     1210subsequent data, but that taken prior have a dynamic mask constructed
     1211when a reference source falls on the focal plane within one degree of
     1212the detector edge.  This mask is 150 pixels wide, with length $L =
     12132500 \left(-20 - m_{inst}\right)$ pixels.  These glint masks are
     1214constructed by selecting sufficiently bright sources in the reference
     1215catalog that fall within rectangular regions around each edge of the
     1216GPC1 camera.  These regions are separated from the edge of the camera
     1217by 17 arcminutes, and extend outwards an additional degree.
     1218
     1219%%
     1220%% GLINT_MAX_MAG                   F32 -21.0
     1221%% GLINT.REGION                    MULTI
     1222
     1223%% GLINT.REGION                    METADATA
     1224%%   REGION                        STR  [-38000:-24000,-20000:+20000]
     1225%%   GLINT.TYPE                    STR  LEFT
     1226%% END
     1227
     1228%% GLINT.REGION                    METADATA
     1229%%   REGION                        STR  [+24000:+38000,-20000:+20000]
     1230%%   GLINT.TYPE                    STR  RIGHT
     1231%% END
     1232
     1233%% GLINT.REGION                    METADATA
     1234%%   REGION                        STR  [-20000:+20000,+24000:+38000:]
     1235%%   GLINT.TYPE                    STR  TOP
     1236%% END
     1237
     1238%% GLINT.REGION                    METADATA
     1239%%   REGION                        STR  [-20000:+20000,-38000:-24000]
     1240%%   GLINT.TYPE                    STR  BOTTOM
     1241%% END
     1242
     1243\begin{figure}
     1244  \centering
     1245  \includegraphics[width=0.9\hsize,angle=0,clip]{images/glint_example_o5379g0103o.jpg}
     1246  \caption{Example of a glint on exposure o5379g0103o (2010-07-02, 45s i-filter).  The source star out of the field of view creates a long reflection that extends through OTA73 and OTA63.}
     1247\end{figure}
     1248
     1249\subsubsubsection{Diffraction Spikes and Saturated Stars}
     1250\label{sec:diffraction_spikes}
     1251
     1252Bright sources also form diffraction spikes that are dynamically
     1253masked.  These are filter independent, and are modeled as rectangles
     1254with length $L = 10^{0.096 * (7.35 - m_{instrumental})} - 200$ and
     1255width $W = 8 + (L - 200) * 0.01$, with negative values indicating no
     1256mask is constructed, as the source is likely too faint to produce the
     1257feature.  These spikes are dependent on the camera rotation, and are
     1258oriented at $\theta = n * \frac{\pi}{2} - \mathrm{ROTANGLE} + 0.798$,
     1259based on the header keyword.
     1260
     1261%\subsubsection{Saturated stars}
     1262%\label{sec:saturated_stars}
     1263
     1264The cores of stars that are saturated are masked as well, with a
     1265circular mask radius $r = 10.15 * (-15 - m_{instrumental})$.  An
     1266example of a saturated star, with the masked regions for the
     1267diffraction spikes and core saturation highlighted, is shown in Figure
     1268\ref{fig:saturated star}.
     1269
     1270\begin{figure}
     1271  \centering
     1272  \includegraphics[width=0.9\hsize,angle=0,clip]{images/o6802g0338o_XY51_b1.jpg}
     1273  \caption{Example of saturated star, with diffraction spikes extending from the core on exposure o6802g0338o, OTA51 (2014-05-25, 45s g-filter).}
     1274  \label{fig:saturated star}
     1275\end{figure}
     1276
     1277\subsubsection{Masking Fraction}
     1278\label{sec:masking_fraction}
     1279
     1280For the full field of view that falls on the sixty OTAs, 14.7\% of all
     1281pixels are masked.  The large fraction of this masking is due to
     1282regions that fall within the vignetted region.  Defining the diameter
     1283of the unvignetted region to be 3 degrees, and excluding pixels that
     1284fall beyond this point reduces the static masking fraction to 9.7\%.
     1285
     1286Unfortunately, due to the design of the OTAs and readout cells, a
     1287non-negligible fraction of the field of view falls onto an area that
     1288does not have a detector pixel.  For a given OTA mosaicked to a
     1289$4846\times{}4868$ pixel image, the 64 $590\times{}598$ pixel readout
     1290cells cover 95.7\% of the OTA area, providing an additional 4.3\%
     1291masking in the unvignetted field of view due to the absence of a
     1292detector pixel.
     1293
     1294For the inter-chip gap area loss, we use two field of view
     1295calculations to estimate the masking fraction.  The reference field of
     1296view of GPC1 is 3 degrees, which at the nominal plate scale of 0.258
     1297arcseconds per pixel, translates to a 20930 FPA pixel radius. \czwdraft{I need a percentage here.}
     1298
     1299%% mysql> select filter,AVG(camProcessedExp.maskfrac_ref_static), AVG(camProcessedExp.maskfrac_ref_dynamic), AVG(camProcessedExp.maskfrac_ref_advisory), AVG(camProcessedExp.maskfrac_max_static),AVG(camProcessedExp.maskfrac_max_dynamic),AVG(camProcessedExp.maskfrac_max_advisory) from camRun join camProcessedExp USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.label = 'LAP.PV3.20140730.final' GROUP BY filter;
     1300%% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+
     1301%% | filter  | AVG(camProcessedExp.maskfrac_ref_static) | AVG(camProcessedExp.maskfrac_ref_dynamic) | AVG(camProcessedExp.maskfrac_ref_advisory) | AVG(camProcessedExp.maskfrac_max_static) | AVG(camProcessedExp.maskfrac_max_dynamic) | AVG(camProcessedExp.maskfrac_max_advisory) |
     1302%% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+
     1303%%             static              dynamic                advisory
     1304%% | g.00000 |   0.19642137972007 | 0.00010322263512709 |    0.026838445469766
     1305%%           |   0.20949461794863 |   9.89200027293e-05 |    0.026431927734548 |
     1306%% | r.00000 |   0.19675996201399 | 0.00025214447869606 |    0.032641054600788
     1307%%           |   0.20989768279138 | 0.00023994155711801 |    0.032178525485201 |
     1308%% | i.00000 |   0.19677587604327 | 0.00057470697316504 |    0.038096251937072
     1309%%           |   0.21003570722292 | 0.00053987093278142 |    0.037471018638997 |
     1310%% | z.00000 |    0.1974290315691 | 0.00024758901226967 |     0.03064123748973
     1311%%           |   0.21055007930696 | 0.00023452690039757 |    0.030144453360769 |
     1312%% | y.00000 |   0.19828990634315 | 0.00014523787521897 |    0.021984846417987
     1313%%           |   0.21130344126869 | 0.00013634812877977 |     0.02163070300815 |
     1314
     1315Summing mask fractions from these three contributions within the
     1316unvignetted field of view results in an average of $\sim 20\%$ masking
     1317fraction across the field of view.  Dynamic masking adds an additional
     1318$2-3\%$ on average, with advisory burntool masking contributing the
     1319largest single component.
     1320
     1321\subsection{Background subtraction}
     1322\label{sec:background}
     1323
     1324Once all other detrending is done, the pixels from each cell are
     1325mosaicked into the full $4846\times{}4868$ pixel OTA image.  A
     1326background model for the full OTA is then determined prior to the
     1327photometric analysis.  The mosaicked image is binned into
     1328$800\times{}800$ pixel bins, centered on the image center, and
     1329overlapping by a factor of 2 in both axes.  These bins have 10000
     1330random samples drawn, and a binned cumulative distribution function is
     1331generated.  These bins are interpolated to find the best mean value at
     1332the $50\%$ level, as well as the distribution $\sigma$ by estimating
     1333from the $32\%$ and $68\%$ levels.  Repeating this across all bins
     1334results in a $13\times{}13$ grid of background bins, which are
     1335bilinearly interpolated to generate the background model to subtract.
     1336Each object in the photometric catalog has a SKY and SKY\_SIGMA value
     1337based on this model as well.
     1338
     1339%% * Magic
     1340%% * Warping
     1341%%   * warping kernel
     1342%%   * linear-by-pieces
     1343%%   * Covariance
     1344%%   * def of skycells?
     1345%% * Stacking
     1346%%   * pixel combination rules
     1347%%   * pixel rejections
     1348%%   * convolution for matching (success and failure)
     1349%% * Difference Image analysis
    2181350
    2191351\section{GPC1 Detrend Construction}
    … …  
    3321464\end{deluxetable}
    3331465
    334 \section{GPC1 Detrend Details}
    335 \label{sec:detrending}
    336 
    337 Ensuring a consistent and uniform detector response across the
    338 three-degree diameter field of view of the GPC1 camera is essential to
    339 a well calibrated survey.  Many standard image detrending steps are
    340 done for GPC1, with overscan subtraction removing the detector bias
    341 level, dark frame subtraction to remove temperature and exposure time
    342 dependent detector glows, and flat field correction to remove pixel to
    343 pixel response functions.  We also construct fringe correction for the
    344 reddest data in the y filter, to remove the interference patterns that
    345 arise in that filter due to the variations in the thickness of the
    346 detector surface.
    347 
    348 These corrections, however, assume that the detector response is
    349 linear across the full range of values.  This is not universally the
    350 case with GPC1, and this requires an additional set of detrending
    351 steps to remove these non-linear responses.  The first of these is the
    352 \ippprog{burntool} correction, which removes the persistence trails
    353 caused by the incomplete transfer of charge along the readout columns.
    354 This bright-end nonlinearity is generally only evident for the
    355 brightest stars, as only pixels that are at or beyond the saturation
    356 point of the detector have this issue.  More widespread is the
    357 non-linearity at the faint end of the pixel range.  Some readout cells
    358 and some readout cell edge pixels experience a sag relative to linear
    359 at low illumination, such that faint pixels appear fainter than
    360 expected.  The correction to this requires amplifying the pixel values
    361 in these regions to match the expected model.
    362 
    363 The final non-linear response issue has no good option for correction.
    364 Large regions of some OTA cells experience significant charge transfer
    365 issues, making them unusable for science observations.  These regions
    366 are therefore masked in processing, with these CTE regions making up
    367 the largest fraction of masked pixels on the detector.  Other regions
    368 are masked for other regions, such as static bad pixel features or
    369 temporary readout masking caused by issues in the camera electronics
    370 that make these regions unreliable.  These all contribute to the
    371 detector mask, which is augmented in each exposure for dynamic
    372 features that are masked based on the astronomical features within the
    373 field of view.
    374 
    375 For the PV3 processing, all detrending is done by the
    376 \ippprog{ppImage} program.  This program applies the detrends to the
    377 individual cells, and then an OTA level mosaic is constructed for the
    378 science image, the mask image, and the variance map image.  The single
    379 epoch photometry is done at this stage as well.  The following
    380 subsections (\ref{sec:burntool} - \ref{sec:background}) detail these
    381 detrending steps, presented in the order in which they are applied to
    382 the individual OTA image data.
    383 
    384 \subsection{Burntool / Persistence effect}
    385 \label{sec:burntool}
    386 
    387 Pixels that approach the saturation point on GPC1, which varies by
    388 readout with common values around 60000 DN, cause persistence problems
    389 on that and subsequent images.  During the read out process of an
    390 image with such a bright pixel, some of the charge associated with it
    391 is not fully shifted down the detector column toward the amplifier.
    392 As a result, this charge remains in the starting cell, and is
    393 partially collected in subsequent shifts, resulting in a ``burn
    394 trail'' that extends from the center of the bright source away from
    395 the amplifier (vertically along the pixel columns toward the top of
    396 the cell).
    397 
    398 This incomplete charge shifting in nearly full wells continues as each
    399 row is read out.  This results in a remnant charge being deposited in
    400 the pixels that the full well was shifted through.  In following
    401 exposures, this remnant charge leaks out, resulting in a trail that
    402 extends from the initial location of the bright source on the previous
    403 image towards the amplifier (vertically down along the pixel column).
    404 This remnant charge can remain on the detector for up to thirty
    405 minutes, requiring the locations of these ``burns'' be retained
    406 between exposures.
    407 
    408 Both of these types of persistance trails are measured and optionally
    409 repaired via the \ippprog{burntool} program.  This program does an
    410 initial scan of the images, and identifies objects with pixel values
    411 brighter than a conservative threshold of 30000 DN.  The trail from
    412 the peak of that object is fit with a one-dimensional power law in
    413 each pixel column above the threshold, based on empirical evidence
    414 that this is the functional form of this persistence effect.  This
    415 also matches the expectation that a constant fraction of charge is
    416 incompletely transfered at each shift beyond the persistence
    417 threshold.  Once this fit is done, the model can be subtracted from
    418 the image, and the location of the star is stored in a table along
    419 with the exposure PONTIME, which denotes the number of seconds since
    420 the detector was last powered on, and provides an internally consistent
    421 time scale.
    422 
    423 For subsequent exposures, the table associated with the previous image
    424 is read in, and after correcting trails from the stars on the new
    425 image, the positions of the bright stars from the table are used to
    426 check for remnant trails on the image.  These are fit and subtracted
    427 using a one-dimensional exponential model, again based on empirical
    428 studies.  If a significant model is found, then this location is
    429 retained in the image output table.  If not, the old burn is allowed
    430 to expire.
    431 
    432 The main concern with this method of correcting the persistance trails
    433 is that it is based on fits to the raw image data, which may have
    434 other signal sources not determined by the persistence effect.  The
    435 presence of other stars or artifacts along the path of the burn can
    436 result in a poor model to be fit, resulting in either an over- or
    437 under-subtraction of the persistance burn.  For this reason, the image
    438 mask is marked with a value indicating that this correction has been
    439 applied.  These pixels are not fully excluded, but they are marked as
    440 suspect, which allows them to be excluded from consideration in
    441 subsequent stages, such as image stacking.
    442 
    443 Another concern is that the cores of very bright stars are deformed by
    444 this process, as the burntool fitting subtracts flux
    445 from only one side of the star.  As most stars that result in burns already
    446 have saturated cores, they are already ignored for the purpose of
    447 PSF determination and are flagged as saturated by the photometry
    448 reduction.
    449 
    450 \begin{figure}
    451   \centering
    452   \begin{minipage}{0.45\hsize}
    453     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt_trail.png}
    454 %    \caption{(a)}
    455 %  \end{subfigure}%
    456 %  \begin{subfigure}[]{.45\hsize}
    457   \end{minipage}%
    458   \begin{minipage}{0.45\hsize}
    459     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt_trail.png}
    460 %    \caption{(b)}
    461 %  \end{subfigure}
    462   \end{minipage}
    463 
    464   \caption{Example of a profile cut along the y-axis through a bright star on exposure o5677g0123o OTA11 in cell xy60 (left panel) and on the subsequent exposure o5677g0124o (right panel).  In both figures, the green points show the image corrected with all appropriate detrending steps, but without burntool applied, illustrating the amplitude of the persistence trails.  The red points show the same data after the burntool correction, which reduces the impact of these features.  Both exposures are in the g-filter with exposure times of 43s}
    465 \end{figure}
    466 
    467 \begin{figure}
    468   \centering
    469   \begin{minipage}{0.45\hsize}
    470     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_nobt.png}
    471 %    \caption{(a)}
    472 %  \end{subfigure}%
    473 %  \begin{subfigure}[]{.45\hsize}
    474   \end{minipage}%
    475   \begin{minipage}{0.45\hsize}
    476     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_nobt.png}
    477 %    \caption{(b)}
    478 %  \end{subfigure}
    479   \end{minipage}
    480   \begin{minipage}{0.45\hsize}
    481     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt.png}
    482 %    \caption{(a)}
    483 %  \end{subfigure}%
    484 %  \begin{subfigure}[]{.45\hsize}
    485   \end{minipage}%
    486   \begin{minipage}{0.45\hsize}
    487     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt.png}
    488 %    \caption{(b)}
    489 %  \end{subfigure}
    490   \end{minipage}
    491   \caption{Example of OTA11 cell xy60 on exposures o5677g0123o (left) and o5677g0124o (right).  The top panels show the image with all appropriate detrending steps, but without burntool, and the bottom show the same with burntool applied.  There is some slight over subtraction in fitting the initial trail, but the impact of the trail is greatly reduced in both exposures.}
    492 \end{figure}
    493 
    494 \subsection{Masking}
    495 \label{sec:masking}
    496 
    497 \subsubsection{Static Masks}
    498 \label{sec:static_masks}
    499 
    500 Due to the large size of the detector, it is expected that there
    501 are a number of pixel defects that do not have the detection
    502 sensitivity on par with their neighbors.  To remove these pixels, we
    503 have constructed a static mask that identifies the known defects.
    504 This mask is constructed in three phases.
    505 
    506 First, a CTEMASK is constructed to mask out regions in which the
    507 charge transfer efficiency is low compared to the rest of the
    508 detector.  Twenty-five of the sixty OTAs in GPC1 show some evidence of
    509 CTE issues, with this pattern appearing (to varying degrees) in
    510 roughly triangular patches on the OTA due to defects in the
    511 semiconductor manufacturing.  To generate the mask for these regions,
    512 a sample set of 26 evenly illuminated flat field images were measured
    513 to produce a map of the image variance in 20x20 pixel bins.  As the
    514 flat image is expected to illuminate the image uniformly, the expected
    515 variances in each bin should be Poissonian distributed with the flux
    516 level.  However, in regions with CTE issues, adjacent pixels are not
    517 independent, as the charge in those pixels is more free to spread.
    518 This reduces the pixel-to-pixel differences, resulting in a lower than
    519 expected variance.  All regions with variance less than half the
    520 average image level are added to the static CTEMASK.
    521 
    522 The next step of mask construction is to examine the flat and dark
    523 models, and exclude pixels that appear to be poorly corrected by these
    524 models.  The DARKMASK process looks for pixels that are more than
    525 $8\sigma$ discrepant in $10\%$ of the 100 input dark frame images
    526 after those images have had the dark model applied to them.  These
    527 pixels are assumed to be unstable with respect to the dark model, and
    528 have the DARK bit set in the static mask, indicating that they are
    529 unreliable in scientific observing.  Similarly, the FLATMASK process
    530 looks for pixels that are $3\sigma$ discrepant in the same fraction of
    531 16 input flat field images after both the dark and flat models have
    532 been applied.  Those pixels that do not follow the flat field model of
    533 the rest of image are assigned the FLAT mask bit in the static mask,
    534 removing the pixels that cannot be corrected to a linear response.
    535 
    536 The final step of mask construction is to examine the detector for
    537 bright columns and other static pixel issues.  This is first done by
    538 processing a set of 100 i filter science images in the same fashion as
    539 for the DARKMASK.  A median image is constructed from these inputs
    540 along with the per-pixel variance.  These images are used to identify
    541 pixels that have unexpectedly low variation between all inputs, as
    542 well as those that significantly deviate from the global median value.
    543 Once this initial set of bad pixels is identified, a $3\times{}3$
    544 pixel triangular kernel is convolved with the initial set, and any
    545 convolved pixel with value greater than 1 is assigned to the static
    546 mask.  This does an excellent job of removing the majority of the
    547 problem pixels.  A subsequent manual inspection allows human
    548 interaction to identify other inconsistent pixels including the
    549 vignetted regions around the edge of the detector. 
    550 
    551 Figure \ref{fig:static mask} shows an example of the static mask for
    552 the full GPC1 field of view.  Table \ref{tab:mask_values} lists the
    553 bit mask values used for the different sources of masking.
    554 
    555 \begin{figure}
    556   \centering
    557   \includegraphics[width=0.9\hsize,angle=0,clip]{images/gpc1_mask_indexed.png}
    558   \label{fig:static mask}
    559  
    560   \caption{Image map of the GPC1 static mask.  The CTE regions are clearly visible as roughly triangular patches covering the corners of some OTAs.  Some entire cells are masked, including an entire column of cells on OTA14.  Calcite cells remove large areas from OTA17 AND OTA76.}
    561 \end{figure}
    562 
    563 \begin{deluxetable}{ccl}
    564   \tablecolumns{3}
    565   \tablewidth{0pc}
    566   \tablecaption{GPC1 Mask Values}
    567   \tablehead{\colhead{Mask Name} & \colhead{Mask Value} & \colhead{Description}}
    568   \startdata
    569   DETECTOR & 0x0001 & A detector defect is present. \\
    570   FLAT     & 0x0002 & The flat field model does not calibrate the pixel reliably. \\
    571   DARK     & 0x0004 & The dark model does not calibrate the pixel reliably. \\
    572   BLANK    & 0x0008 & The pixel does not contain valid data. \\
    573   CTE      & 0x0010 & The pixel has poor charge transfer efficiency. \\
    574   SAT      & 0x0020 & The pixel is saturated. \\
    575   LOW      & 0x0040 & The pixel has a lower value than expected. \\
    576   SUSPECT  & 0x0080 & The pixel is suspected of being bad. \\
    577   BURNTOOL & 0x0080 & The pixel contain an burntool repaired streak. \\
    578   CR       & 0x0100 & A cosmic ray is present. \\
    579   SPIKE    & 0x0200 & A diffraction spike is present. \\
    580   GHOST    & 0x0400 & An optical ghost is present. \\
    581   STREAK   & 0x0800 & A streak is present. \\
    582   STARCORE & 0x1000 & A bright star core is present. \\
    583   CONV.BAD & 0x2000 & The pixel is bad after convolution with a bad pixel. \\
    584   CONV.POOR& 0x4000 & The pixel is poor after convolution with a bad pixel. \\
    585   MARK     & 0x8000 & An internal flag for temporarily marking a pixel. \\
    586   \enddata
    587   \label{tab:mask_values}
    588 \end{deluxetable}
    589 
    590 \subsubsection{Dynamic masks}
    591 \label{sec:dynamic_masks}
    592 
    593 In addition to the static mask that removes the constant detector
    594 defects, we also generate a set of dynamic masks that change with the
    595 astronomical features in the image.  These masks are advisory in
    596 nature, and do not completely exclude the pixel from further
    597 processing consideration.  The first of these dynamic masks is the
    598 burntool advisory mask mentioned above.  These pixels are included for
    599 photometry, but are rejected more readily in the stacking and
    600 difference image construction, as they are more likely to have small
    601 deviations due to imperfections in the burntool correction.
    602 
    603 The remaining dynamic masks are not generated until the IPP
    604 \ippstage{camera} stage, at which point all object photometry is
    605 complete, and an astrometric solution is known for the exposure.  This
    606 added information provides the positions of bright sources based on
    607 the reference catalog, including those that fall slightly out of the
    608 detector field of view or within the inter chip gaps, where internal
    609 photometry may not identify them.  These bright sources are the origin
    610 for many of the image artifacts that the dynamic mask identifies and
    611 excludes.
    612 
    613 \subsubsection{Electronic crosstalk ghosts}
    614 \label{sec:crosstalk}
    615 
    616 Due to electrical crosstalk between the flex cables connecting the
    617 individual detector OTA devices, ghost objects can be created by the
    618 presence of a bright source at a different position on the camera.
    619 Table \ref{tab:crosstalk_rules} summarizes the list of known crosstalk
    620 rules, with an estimate of the magnitude difference between the source
    621 and ghost.  For all of the rules, any cell $v$ within the specified
    622 column of cells on any of the OTAs in the specified column of OTAs $Y$
    623 creates the ghost in the same $v$ and $Y$ in the target column of
    624 cells and OTAs.  In each of these cases, a source object with an
    625 instrumental magnitude brighter than -14.47 creates a ghost object
    626 many orders of magnitude fainter at the target location.  The cell
    627 (x,y) pixel coordinate is identical between source and ghost, as a
    628 result of the transfer occurring as the devices are read.  A circular
    629 mask is added to the ghost location with radius $R = 3.44 \left(-14.47
    630 - m_{source, instrumental}\right)$ pixels.  Any objects in the
    631 photometric catalog found at the location of the ghost mask have the
    632 GHOST mask bit set, marking the object as a likely ghost.  The
    633 majority of the crosstalk rules are bi-directional, with a source in
    634 either position creating a ghost at the corresponding crosstalk target
    635 position.  The two faintest rules are uni-directional, due to
    636 differences in the electronic path for the crosstalk.
    637 
    638 For the very brightest sources ($m_{instrumental} < -15$), there can
    639 be crosstalk ghosts between all columns of cells during the readout.
    640 These ``bleed'' ghosts were originally identified as ghosts of the
    641 saturation bleeds appearing in the neighboring cells, and as such, the
    642 masking for these objects puts a rectangular mask down from top to
    643 bottom of cells in all columns that are in the same row of cells as
    644 the bright source.  The width of this box is a function of the source
    645 magnitude, with $W = 5 * \left(-15 - m_{source, instrumental}\right)$
    646 pixels.
    647 
    648 \begin{deluxetable}{lllc}
    649   \tablecolumns{4}
    650   \tablewidth{0pc}
    651   \tablecaption{GPC1 Crosstalk Rules}
    652   \tablehead{\colhead{Type}&\colhead{Source OTA/Cell}&\colhead{Ghost OTA/Cell}&\colhead{$\Delta m$}}
    653   \startdata
    654   Inter-OTA & OTA2Y XY3v & OTA3Y XY3v & 6.16 \\
    655             & OTA3Y XY3v & OTA2Y XY3v &      \\
    656             & OTA4Y XY3v & OTA5Y XY3v &      \\
    657             & OTA5Y XY3v & OTA4Y XY3v &      \\
    658   Intra-OTA & OTA2Y XY5v & OTA2Y XY6v & 7.07 \\
    659             & OTA2Y XY6v & OTA2Y XY5v &      \\
    660             & OTA5Y XY5v & OTA5Y XY6v &      \\
    661             & OTA5Y XY6v & OTA5Y XY5v &      \\
    662   One-way   & OTA2Y XY7v & OTA3Y XY2v & 7.34 \\
    663             & OTA5Y XY7v & OTA4Y XY2v &      \\
    664   \enddata
    665   \label{tab:crosstalk_rules}
    666 \end{deluxetable}
    667  
    668 %% \begin{figure}
    669 %%   \centering
    670 %%   \caption{Figure of crosstalk ghost and bright star source.  Plot of cut across ghost to illustrate the flat-top shape.}
    671 %% \end{figure}
    672 
    673 \subsubsection{Optical ghosts}
    674 \label{sec:optical_ghosts}
    675 % http://arxiv.org/pdf/1207.2513v1.pdf
    676 
    677 Due to imperfections in the anti-reflective coating on the optical
    678 surfaces of GPC1, bright sources can also result in large out of focus
    679 objects, particularly in the g-filter data.  These objects are the
    680 result of light reflecting back off the surface of the detector,
    681 reflecting again off the lower surfaces of the optics (particularly
    682 the L1 corrector lens), and then back down onto the focal plane.  Due
    683 to the extra travel distance, the resulting source is out of focus and
    684 elongated along the radial direction of the camera focal plane. These
    685 optical ghosts can be modeled in the focal plane coordinates (L,M)
    686 which has its origin at the center of the focal plane.  In this
    687 system, a bright object at location (L,M) on the focal plane creates a
    688 reflection ghost on the opposite side of the optical axis at (-L,-M).
    689 The exact location is fit as a third order polynomial in the focal
    690 plane L and M directions (as listed in Table \ref{tab:ghost_centers}).
    691 An elliptical annulus mask is constructed at the expected ghost
    692 location, with the major and minor axes defined by linear functions of
    693 the ghost distance from the optical axis, and oriented with the
    694 ellipse major axis is along the radial direction (Table
    695 \ref{tab:ghost_radii}).  All stars brighter than a filter-dependent
    696 threshold (listed in Table \ref{tab:ghost_magnitudes}) have such masks
    697 constructed.
    698 
    699 \begin{deluxetable}{lcc}
    700   \tablecolumns{3}
    701   \tablewidth{0pc}
    702   \tablecaption{Optical Ghost Center Transformations}
    703   \tablehead{\colhead{Polynomial Term}&\colhead{L center}&\colhead{M center}}
    704   \startdata
    705   $x^0 y^0$ & -1.215661e+02 &  2.422174e+01 \\
    706   $x^1 y^0$ &  1.321875e-02 &  4.170486e-04 \\
    707   $x^2 y^0$ & -4.017026e-09 & -1.934260e-08 \\
    708   $x^3 y^0$ &  1.148288e-10 & -1.173657e-12 \\
    709   $x^0 y^1$ & -1.908074e-03 &  1.189352e-02 \\
    710   $x^1 y^1$ &  8.479150e-08 & -9.256748e-08 \\
    711   $x^2 y^1$ &  1.635732e-11 &  1.140772e-10 \\
    712   $x^0 y^2$ &  2.625405e-08 &  8.123932e-08 \\
    713   $x^1 y^2$ &  1.125586e-10 &  1.328378e-11 \\
    714   $x^0 y^3$ &  2.912432e-12 &  1.170865e-10 \\
    715   \enddata
    716   \label{tab:ghost_centers}
    717 \end{deluxetable}
    718 
    719 \begin{deluxetable}{lcccc}
    720   \tablecolumns{5}
    721   \tablewidth{0pc}
    722   \tablecaption{Optical Ghost Annulus Axis Length}
    723   \tablehead{\colhead{Radial Order}&\colhead{Inner Major Axis}&\colhead{Inner Minor Axis}&    \colhead{Outer Major Axis}&\colhead{Outer Minor Axis}}
    724   \startdata
    725   $r^0$ & 3.926693e+01 & 5.287548e+01 & 7.928722e+01 & 1.314265e+02 \\
    726   $r^1$ & 5.325759e-03 &-2.191669e-03 & 1.722181e-02 & -2.627153e-03 \\
    727   \enddata
    728   \label{tab:ghost_radii}
    729 \end{deluxetable}
    730 
    731 \begin{deluxetable}{lc}
    732   \tablecolumns{2}
    733   \tablewidth{0pc}
    734   \tablecaption{Optical Ghost Magnitude Limits}
    735   \tablehead{\colhead{Filter}&\colhead{$m_{inst}$}}
    736   \startdata
    737   g & -16.5 \\
    738   r & -20.0 \\
    739   i & -25.0 \\
    740   z & -25.0 \\
    741   y & -25.0 \\
    742   w & -20.0 \\
    743   \enddata
    744   \label{tab:ghost_magnitudes}
    745 \end{deluxetable}
    746 
    747 
    748 \begin{figure}
    749   \centering
    750   \includegraphics[width=0.9\hsize,angle=0,clip]{images/full_fpa_ghosts.jpg}
    751   \caption{Example of the full GPC1 field of view illustrating the sources and destinations of optical ghosts on exposure o5677g0123o (2011-04-26, 43s g-filter).  The bright stars on OTA33 and OTA44 result in nearly circular ghosts on the opposite OTA.  In contrast, the trio of stars on OTA11 result in very elongated ghosts on OTA66.}
    752 \end{figure}
    753 
    754 \subsubsection{Optical glints}
    755 \label{sec:glints}
    756 Prior to \czwdraft{DATE}, a reflective surface at the edge of the
    757 camera aperture was incompletely screened to light passing through the
    758 telescope.  Sources brighter than $m = -20$ that fell on this
    759 reflective surface resulted in light being scattered across the
    760 detector surface in a long narrow glint.  This surface was physically
    761 masked on \czwdraft{DATE}, removing the possibility of glints in
    762 subsequent data, but that taken prior have a dynamic mask constructed
    763 when a reference source falls on the focal plane within one degree of
    764 the detector edge.  This mask is 150 pixels wide, with length $L =
    765 2500 \left(-20 - m_{inst}\right)$ pixels.  \czwdraft{Am I correct that
    766   this is basically a one-degree edge around the detector?}
    767 
    768 %%
    769 %% GLINT_MAX_MAG                   F32 -21.0
    770 %% GLINT.REGION                    MULTI
    771 
    772 %% GLINT.REGION                    METADATA
    773 %%   REGION                        STR  [-38000:-24000,-20000:+20000]
    774 %%   GLINT.TYPE                    STR  LEFT
    775 %% END
    776 
    777 %% GLINT.REGION                    METADATA
    778 %%   REGION                        STR  [+24000:+38000,-20000:+20000]
    779 %%   GLINT.TYPE                    STR  RIGHT
    780 %% END
    781 
    782 %% GLINT.REGION                    METADATA
    783 %%   REGION                        STR  [-20000:+20000,+24000:+38000:]
    784 %%   GLINT.TYPE                    STR  TOP
    785 %% END
    786 
    787 %% GLINT.REGION                    METADATA
    788 %%   REGION                        STR  [-20000:+20000,-38000:-24000]
    789 %%   GLINT.TYPE                    STR  BOTTOM
    790 %% END
    791 
    792 \begin{figure}
    793   \centering
    794   \includegraphics[width=0.9\hsize,angle=0,clip]{images/glint_example_o5379g0103o.jpg}
    795   \caption{Example of a glint on exposure o5379g0103o (2010-07-02, 45s i-filter).  The source star out of the field of view creates a long reflection that extends through OTA73 and OTA63.}
    796 \end{figure}
    797 
    798 \subsubsection{Diffraction Spikes and Saturated Stars}
    799 \label{sec:diffraction_spikes}
    800 
    801 Bright sources also form diffraction spikes that are dynamically
    802 masked.  These are filter independent, and are modeled as rectangles
    803 with length $L = 10^{0.096 * (7.35 - m_{instrumental})} - 200$ and
    804 width $W = 8 + (L - 200) * 0.01$, with negative values indicating no
    805 mask is constructed, as the source is likely too faint to produce the
    806 feature.  These spikes are dependent on the camera rotation, and are
    807 oriented at $\theta = n * \frac{\pi}{2} - \mathrm{ROTANGLE} + 0.798$,
    808 based on the header keyword.
    809 
    810 %\subsubsection{Saturated stars}
    811 %\label{sec:saturated_stars}
    812 
    813 The cores of stars that are saturated are masked as well, with a
    814 circular mask radius $r = 10.15 * (-15 - m_{instrumental})$.  An
    815 example of a saturated star, with the masked regions for the
    816 diffraction spikes and core saturation highlighted, is shown in Figure
    817 \ref{fig:saturated star}.
    818 
    819 \begin{figure}
    820   \centering
    821   \includegraphics[width=0.9\hsize,angle=0,clip]{images/o6802g0338o_XY51_b1.jpg}
    822   \caption{Example of saturated star, with diffraction spikes extending from the core on exposure o6802g0338o, OTA51 (2014-05-25, 45s g-filter).}
    823   \label{fig:saturated star}
    824 \end{figure}
    825 
    826 \subsubsection{Video Mask}
    827 \label{sec:video_masks}
    828 
    829 One aspect of the OTAs on GPC1 is that an individual cell can be read
    830 repeatedly while the other cells integrate, resulting in a video
    831 signal from that cell.  This data is used for telescope guiding
    832 purposes, and a single exposure is likely to have a number of these
    833 video cells active on different OTAs.  For the 3PI survey, the median
    834 exposure has 14 video cells being read, although this number ranges
    835 from less than five to more than thirty, depending on the stellar
    836 density and field pointing.  Reading these cells while integrating on
    837 the others changes the characteristic dark model (see Section
    838 \ref{sec:video_darks} below) experienced by the other cells on the
    839 OTA.  The observed effect of this is that the glow associated with the
    840 amplifiers in the corners of the cells is suppressed during the video
    841 readout, relative to the nominal glow.  The standard dark model
    842 oversubtracts this glow, resulting in dark regions in the corners of
    843 the cells on an OTA taking video data.  Before the nature of this
    844 issue was fully understood, these poorly constrained corners were
    845 masked with 25-pixel radius quarter circles, centered on the (1,1)
    846 pixel nearest the cell amplifier.  The other corners of the cell were
    847 masked with a 15-pixel radius quarter circle, as the amplifier
    848 creating the glow is associated with another cell and separated by the
    849 inter-cell spacing, diminishing the area effected.  Due to the large
    850 area that this masking would cover, the PV3 processing used a more
    851 robust video dark model to correct this problem, as described in
    852 section \ref{sec:video_darks} below.
    853 
    854 \subsubsection{Masking Fraction}
    855 \label{sec:masking_fraction}
    856 
    857 For the full field of view that falls on the sixty OTAs, 14.7\% of all
    858 pixels are masked.  The large fraction of this masking is due to
    859 regions that fall within the vignetted region.  Defining the diameter
    860 of the unvignetted region to be 3 degrees, and excluding pixels that
    861 fall beyond this point reduces the static masking fraction to 9.7\%.
    862 
    863 Unfortunately, due to the design of the OTAs and readout cells, a
    864 non-negligible fraction of the field of view falls onto an area that
    865 does not have a detector pixel.  For a given OTA mosaicked to a
    866 $4846\times{}4868$ pixel image, the 64 $590\times{}598$ pixel readout
    867 cells cover 95.7\% of the OTA area, providing an additional 4.3\%
    868 masking in the unvignetted field of view due to the absence of a
    869 detector pixel.
    870 
    871 For the inter-chip gap area loss, we use two field of view
    872 calculations to estimate the masking fraction.  The reference field of
    873 view of GPC1 is 3 degrees, which at the nominal plate scale of 0.258
    874 arcseconds per pixel, translates to a 20930 FPA pixel radius. \czwdraft{I need a percentage here.}
    875 
    876 %% mysql> select filter,AVG(camProcessedExp.maskfrac_ref_static), AVG(camProcessedExp.maskfrac_ref_dynamic), AVG(camProcessedExp.maskfrac_ref_advisory), AVG(camProcessedExp.maskfrac_max_static),AVG(camProcessedExp.maskfrac_max_dynamic),AVG(camProcessedExp.maskfrac_max_advisory) from camRun join camProcessedExp USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.label = 'LAP.PV3.20140730.final' GROUP BY filter;
    877 %% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+
    878 %% | filter  | AVG(camProcessedExp.maskfrac_ref_static) | AVG(camProcessedExp.maskfrac_ref_dynamic) | AVG(camProcessedExp.maskfrac_ref_advisory) | AVG(camProcessedExp.maskfrac_max_static) | AVG(camProcessedExp.maskfrac_max_dynamic) | AVG(camProcessedExp.maskfrac_max_advisory) |
    879 %% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+
    880 %%             static              dynamic                advisory
    881 %% | g.00000 |   0.19642137972007 | 0.00010322263512709 |    0.026838445469766
    882 %%           |   0.20949461794863 |   9.89200027293e-05 |    0.026431927734548 |
    883 %% | r.00000 |   0.19675996201399 | 0.00025214447869606 |    0.032641054600788
    884 %%           |   0.20989768279138 | 0.00023994155711801 |    0.032178525485201 |
    885 %% | i.00000 |   0.19677587604327 | 0.00057470697316504 |    0.038096251937072
    886 %%           |   0.21003570722292 | 0.00053987093278142 |    0.037471018638997 |
    887 %% | z.00000 |    0.1974290315691 | 0.00024758901226967 |     0.03064123748973
    888 %%           |   0.21055007930696 | 0.00023452690039757 |    0.030144453360769 |
    889 %% | y.00000 |   0.19828990634315 | 0.00014523787521897 |    0.021984846417987
    890 %%           |   0.21130344126869 | 0.00013634812877977 |     0.02163070300815 |
    891 
    892 Summing mask fractions from these three contributions within the
    893 unvignetted field of view results in an average of $\sim 20\%$ masking
    894 fraction across the field of view.  Dynamic masking adds an additional
    895 $2-3\%$ on average, with advisory burntool masking contributing the
    896 largest single component.
    897 
    898 \subsection{Overscan}
    899 \label{sec:overscan}
    900 
    901 Each cell on GPC1 has an overscan region that covers the first 34
    902 columns of each row, and the last 10 rows of each column.  No light
    903 lands on these pixels, so the image region is trimmed to exclude them.
    904 Each row has an overscan value subtracted, calculated by finding the
    905 median value of that row's overscan pixels and then smoothing between
    906 rows with a three-row boxcar median.
    907 
    908 \subsection{Non-linearity Correction}
    909 \label{sec:nonlinearity}
    910 % check notebook, 2010-07/08
    911 
    912 The pixels of GPC1 are not uniformly linear at all flux levels.  In
    913 particular, at low flux levels, some pixels have a tendency to sag
    914 relative to the expected linear value.  This effect is most pronounced
    915 along the edges of the detector cells, although some entire cells show
    916 evidence of this effect.
    917 
    918 To correct this sag, we studied the flux behavior of a series of flat
    919 frames for a ramp of exposure times with approximate logarithmically
    920 equal spacing between 0.01s and 57.04s.  As the exposure time
    921 increases, the flux on each pixel also increases in what is expected
    922 to be a linear manner.  Each of these flat exposures in this ramp is
    923 overscan corrected, and then the median is calculated for each cell,
    924 as well as for the rows and columns within ten pixels of the edge of
    925 the science region.  From these median values at each exposure time
    926 value, we can construct the expected trend by fitting a linear model,
    927 $f_{region} = G * t_{exp} + B$, to determine the gain, $G$, and the
    928 bias, $B$, for the region considered.  This fitting was limited to only
    929 the range of fluxes between 12000 and 38000 counts, as these ranges
    930 were found to match the linear model well.  This range avoids the
    931 non-linearity at low fluxes, as well as the possibility of high-flux
    932 non-linearity effects.
    933 
    934 We store the average flux measurement and deviation from the linear
    935 fit for each exposure time for all regions on all detector cells in
    936 the linearity detrend look up tables.  When this is applied to science
    937 data, these lookup tables are loaded, and a linear interpolation is
    938 performed to determine the correction needed for the flux in that
    939 pixel.  This look up is performed for both the row and column of each
    940 pixel, to allow the edge correction to be applied where applicable,
    941 and the full cell correction elsewhere.  The average of these two
    942 values is then applied to the pixel value, reducing the effects of
    943 pixel nonlinearity.
    944 
    945 This non-linearity effect appears to be stable in time for the
    946 majority of the detector pixels, with little evident change over the
    947 survey duration.  However, as the non-linearity is most pronounced at
    948 the edges of the detector cells, those are the regions where the
    949 correction is most likely to be incomplete.  Because of this fact,
    950 most pixels in the static mask with either the DARKMASK or FLATMASK
    951 bit set are found along these edges.  As the non-linearity correction
    952 is unable to reliably restore these pixels, they produce inconsistent
    953 values after the dark and flat have been applied, and are therefore
    954 rejected.
    955 
    956 %% exptime n_included/det_id = 372
    957 %% clearly this isn't the one used, as 3-12 spans three data points, poorly.x
    958 %% 0.01 2
    959 %% 0.14 2
    960 %% 0.27 2
    961 %% 0.49 2
    962 %% 0.72 2
    963 %% 1.06 2
    964 %% 1.41 2
    965 %% 2.02 2
    966 %% 2.63 2
    967 %% 3.94 2
    968 %% 5.25 2
    969 %% 8.74 2
    970 %% 13.09 2
    971 %% 17.4 2
    972 %% 20.86 2
    973 %% 24.3 2
    974 %% 27.78 2
    975 %% 31.24 2
    976 %% 34.65 2
    977 %% 38.12 2
    978 %% 42.41 2
    979 %% 46.69 2
    980 %% 51.89 2
    981 %% 57.04 2
    982 
    983 
    984 %http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearity_AllEdges
    985 %http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearityArchive
    986 
    987 \begin{figure}
    988   \centering
    989   \includegraphics[width=0.9\hsize,angle=0,clip]{images/linearity_XY27_xy16.png}
    990   \caption{Example plot of the linearity correction as a fraction of observed flux for OTA27, cell xy16.}
    991 \end{figure}
    992 
    993 \subsection{Dark/Bias Subtraction}
    994 \label{sec:dark}
    995 % http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Dark_Model
    996 
    997 The dark model we make for GPC1 considers each pixel individually,
    998 independent of any neighbors.  To construct this model, we fit a
    999 multi-dimensional model to the array of input pixels from a randomly
    1000 selected set of 100-150 overscan and non-linearity corrected dark
    1001 frames chosen from a given date range.  The model fits each pixel as a
    1002 function of the exposure time $t_{exp}$ and the detector temperature
    1003 $T_{chip}$ of the input images such that $\mathrm{dark} = a_0 + a_1
    1004 t_{exp} + a_2 T_{chip} t_{exp} + a_3 T_{chip}^2 t_{exp}$.  This
    1005 fitting uses two iterations to produce a clipped fit, rejecting at the
    1006 $3\sigma$ level.  The final coefficients $a_i$ for the dark model are
    1007 stored in the detrend image.  The constant $a_0$ term includes the
    1008 residual bias signal after overscan subtraction, and as such, a
    1009 separate bias subtraction is not necessary.
    1010 
    1011 Applying the dark model is simply a matter of calculating the response
    1012 to the exposure time and detector temperature for the image to be
    1013 corrected, and subtracting the resulting dark signal from the image.
    1014 
    1015 \subsubsection{Time evolution}
    1016 
    1017 The dark model is not consistently stable over the full survey, with
    1018 significant drift over the course of multiple months.  Some of the
    1019 changes in the dark can be attributed to changes in the voltage
    1020 settings of the GPC1 controller electronics, but the majority seem to
    1021 be the result of some unknown parameter.  We can separate the dark
    1022 model history of GPC1 into three epochs.  The first epoch covers all
    1023 data taken prior to 2010-01-23.  This epoch used a different header
    1024 keyword for the detector temperature, making data from this epoch
    1025 incompatible with later dark models.
    1026 
    1027 The second epoch covers data between 2010-01-23 and 2011-05-01, and is
    1028 characterized by a largely stable but oscillatory dark solution.
    1029 There are two modes that the dark model switches between apparently at
    1030 random.  No clear cause has been established for the switching, but
    1031 there are clear differences between the two modes that require the
    1032 observation dates to be split to use the model that is most
    1033 appropriate.
    1034 
    1035 The initial evidence of these two modes comes from the discovery of a
    1036 slight gradient along the rows of certain cells.  This is a result of
    1037 a drift in the bias level of the detector as it is read out.  An
    1038 appropriate dark model should remove this gradient entirely.  For
    1039 these two modes, the direction of this bias drift is different, so a
    1040 single dark model generated from all dark images in the time range
    1041 over corrects the positive-gradient mode, and under corrects the
    1042 negative-gradient mode.  Upon identifying this two-mode behavior, and
    1043 determining the dates each mode was dominant, two separate dark
    1044 models were constructed from appropriate ``A'' and ``B'' mode dark
    1045 frames.  Using the appropriate dark minimizes the effect of this bias
    1046 gradient in the dark corrected data. 
    1047 
    1048 The bias drift gradients of the mode switching can be visualized in
    1049 Figure \ref{fig:dark switching}.  This figure shows image profile
    1050 along the x-pixel axis binned along the full y-axis of dark corrected
    1051 images for OTA67.  These images are from sequential days, and have
    1052 been corrected with a dark model constructed from the full set of dark
    1053 data within the second epoch.  The opposite sign of the slopes of
    1054 these profiles indicates that the average dark model does not correct
    1055 these dates sufficiently, due to the contradictory dark signals
    1056 between the two modes. \czwdraft{this paragraph dependent on that figure.  This doesn't quite match.}
    1057 
    1058 After 2011-05-01, the two-mode behavior of the dark disappears, and is
    1059 replaced with a slow observation date dependent drift in the magnitude
    1060 of the gradient.  This drift is sufficiently slow that we have modeled
    1061 it using three observation date independent dark model for different
    1062 date ranges.  These darks cover the range from 2011-05-01 to
    1063 2011-08-01, 2011-08-01 to 2011-11-01, and 2011-11-01 and on.  The
    1064 reason for this time evolution is unknown, but as it is correctable
    1065 with a small number of dark models, this does not significantly impact
    1066 detrending.
    1067 
    1068 \begin{figure}
    1069   \centering
    1070 %  \begin{subfigure}[]{.45\hsize}
    1071   \begin{minipage}{0.45\hsize}
    1072     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_M_OS_NL_XY23_b1.jpg}
    1073 %    \caption{(a)}
    1074 %  \end{subfigure}%
    1075 %  \begin{subfigure}[]{.45\hsize}
    1076   \end{minipage}%
    1077   \begin{minipage}{0.45\hsize}
    1078     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_to_DARK_XY23_b1.jpg}
    1079 %    \caption{(b)}
    1080 %  \end{subfigure}
    1081   \end{minipage}
    1082   \caption{An example of the dark model application to exposure o5677g0123o, OTA23 (2011-04-26, 43s g-filter).  The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, and the detector non-linearity corrected.  The right panel, shows the same exposure with the dark applied in addition to the processing shown on the left.}
    1083 \end{figure}
    1084 
    1085 \begin{figure}
    1086   \centering
    1087   \includegraphics[width=0.9\hsize,angle=0,clip]{images/B_profile_ex.png}
    1088   \caption{Example showing a profile cut across exposure o5676g0195, OTA67 (2011-04-25, 43s g-filter).  The entire first row of cells (xy00-xy07) have had a median calculated along each pixel column on the OTA mosaicked image.  Arbitrary offsets have been applied to shift the curves to not overlap.  The top curve (in purple) shows the initial raw profile, without no dark model applied.  The next curve (in green) shows the smoother profile after applying the correct B-mode dark model.  Applying the incorrect A-mode dark results in the blue curve, which shows a significant increase in gradients across the cells.  The orange curve shows the result of the PATTERN.CONTINUITY correction.  Although this creates a larger gradient across the mosaicked images, it decreases the cell-to-cell level changes.  The final yellow curve shows the final image profile after all detrending and background subtraction, and has not had an offset applied.  The bright source at the cell xy00 to xy01 transition is a result of a large optical ghost, which due to the area covered, increases the median level more than the field stars.}
    1089   \label{fig:dark switching}
    1090 \end{figure}
    1091 
    1092 \subsubsection{Video Dark}
    1093 \label{sec:video_darks}
    1094 
    1095 The dark signal is stronger in cell corners due to glow from the
    1096 read-out amplifiers.  The standard dark model corrects this for most
    1097 observations.  However, as mentioned above, when a cell is repeatedly
    1098 read in video mode, the dark model for the OTA containing it changes.
    1099 Surprisingly, added reads for the video cell do not amplify the
    1100 amplifier glow, but rather decrease the dark signal in these regions.
    1101 As a result, using the standard dark model on the data for these OTAs
    1102 results in oversubtraction of the corner glow.
    1103 
    1104 Video darks have been constructed to eliminate the effect this
    1105 observational change has on the final image quality.  This was done by
    1106 running the standard dark construction process on a series of dark
    1107 frames that have had the video signal enabled for some cells.  GPC1
    1108 can only run video signals on a subset of the OTAs at a given time.
    1109 This requires two passes to enable the video signal across the full
    1110 set of OTAs that support video cells.  This is convenient for the
    1111 process of creating darks, as those OTAs that do not have video
    1112 signals enabled create standard dark models, while the video dark is
    1113 created for those that do.
    1114 
    1115 This simultaneous construction of video and standard dark models is
    1116 useful, as it provides the ability to isolate the response on the
    1117 standard dark from the video signals.  Isolating this response is
    1118 essential for attempting to create archival video darks.  We only have
    1119 raw video dark frame data after 2012-05-16, when this problem was
    1120 initially identified, so any data prior to that can not be directly
    1121 corrected for the video dark signal.  Isolating the video signal
    1122 response allows linear corrections to the pre-existing standard dark
    1123 models for archival data.  Testing this shows that constructing a
    1124 video dark for older data simply as $VD_{2009} = D_{2009} - D_{Modern}
    1125 + VD_{Modern}$ produces a satisfactory result that does not
    1126 oversubtract the amplifier glow.  This is shown in figure
    1127 \ref{fig:video_darks}, which shows video cells from before 2012-05-16,
    1128 corrected with both the standard and video darks, with the early video
    1129 dark constructed in such a manner.
    1130 
    1131 \begin{figure}
    1132   \centering
    1133 %  \begin{subfigure}[]{.45\hsize}
    1134   \begin{minipage}{0.45\hsize}
    1135     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_Rdark_XY22_b1.jpg}
    1136 %    \caption{(a)}
    1137 %  \end{subfigure}%
    1138 %  \begin{subfigure}[]{.45\hsize}
    1139   \end{minipage}%
    1140   \begin{minipage}{0.45\hsize}
    1141     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_VDdark_XY22_b1.jpg}
    1142 %    \caption{(b)}
    1143 %  \end{subfigure}
    1144   \end{minipage}
    1145   \caption{An example of the video dark model application to exposure o5677g0123o, OTA22 (2011-04-26, 43s g-filter), which has a video cell located in cell xy16.  The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, the detector non-linearity corrected, and a regular dark applied.  The right panel, shows the same exposure with a video dark applied instead of the standard dark.  The main impact of this change is the improved correction of the corner glows, which are oversubtracted with the standard dark.}
    1146   \label{fig:video_darks}
    1147 \end{figure}
    1148 
    1149 \subsection{Noisemap}
    1150 \label{sec:noisemap}
    1151 
    1152 Based on a study of the positional dependence of all detected sources,
    1153 we have discovered that the cells in GPC1 do not have uniform noise
    1154 characteristics.  Instead, there is a gradient along the pixel rows,
    1155 with the noise generally higher away from the read out amplifier
    1156 (higher cell x pixel positions).  This is likely an effect of the
    1157 row-by-row bias issue discussed below.  This gradient causes the read
    1158 noise to increase as the row is read out.  As a result of this
    1159 increased noise, more sources are detected in the higher noise regions
    1160 when the read noise is assumed constant across the readout.  To
    1161 mitigate this noise gradient, we constructed an initial set of
    1162 noisemap images by measuring the median variance on bias frames.  The
    1163 variance is calculated in boxes of 20x20 pixels, and then linearly
    1164 interpolated to cover the full image.
    1165 
    1166 Unfortunately, due to correlations within this noise, the variance
    1167 measured from the bias images does not fully remove the positional
    1168 dependence of objects that are detected.  This simple noisemap
    1169 underestimates the noise observed when the image is filtered during
    1170 the object detection process.  This filtering convolves the background
    1171 noise with a PSF, which has the effect of amplifying the correlated
    1172 peaks in the noise.  This amplification can therefore boost background
    1173 fluctuations above the threshold used to select real objects,
    1174 contaminating the final object catalogs.
    1175 
    1176 In the detection process, we expect false positives at a rate equal to
    1177 the one-tailed probability beyond the detection threshold.  For these
    1178 tests, only detections measured at the $\sigma_{thresh} = 5\sigma$
    1179 level are used, to match that used in the photometry on science data.
    1180 This probability can be converted into a number of false number by
    1181 considering a given area.  As the detections must be isolated to not
    1182 be detected as an extended object, this area must be reduced by the
    1183 area a given PSF occupies.  Combining this, we find that we expect a
    1184 probability $P = 1 - \Phi_{normal}(5) = \frac{1}{2}
    1185 \erfcinv\left(\frac{5}{\sqrt{2}}\right)$, and an area given $N$
    1186 exposures of area $X\times Y$, $A = \frac{X \times Y \times
    1187   N}{A_{PSF}}$.  For a typical $1"$ seeing, $A_{PSF}$ is approximately
    1188 16 pixels.  Using this model for the false positives, we found that
    1189 the added read noise was insufficient to account for the observed
    1190 false positive rate.  Inverting this relation, we can measure
    1191 $\sigma_{obs}$, the true threshold level based on the number of false
    1192 positives observed.  This $\sigma_{obs}$ is the combined to form a
    1193 boost factor $B = \sigma_{thresh} / \sigma_{obs}$ that amplifies the
    1194   noisemap to match the observed false detection rate.
    1195 
    1196 The row-to-row variations that contribute to the extra noise are
    1197 related to the dark model, and because of this, as the dark model
    1198 changes, the effective noise also changes.  To ensure that the
    1199 noisemap accurately matches the true noise level, we have created
    1200 different noisemap models for the three major time ranges of the dark
    1201 model.  We do not see any strong evidence that the noisemaps have the
    1202 A/B modes visible in the dark, and so we do not generate different
    1203 models for each individual dark model.  The additional pixel-to-pixel
    1204 variance from this noisemap is added to the Poissonian variance to
    1205 form the science variance image generated by the \ippstage{chip}
    1206 processing.
    1207 
    1208 \subsection{Flat}
    1209 
    1210 Determining a flat field correction for GPC1 is a challenging
    1211 endeavor, as the wide field of view makes it difficult to construct a
    1212 uniformly illuminated image.  Using a dome screen is not possible, as
    1213 the variations in illumination and screen rigidity create large
    1214 scatter between different images that are not caused by the detector
    1215 response function.  Because of this, we use sky flat images taken at
    1216 twilight, which are more consistently illuminated than screen flats.
    1217 We calculate the mean of these images to determine the initial flat
    1218 model.
    1219 
    1220 From this starting model, we construct a correction to remove the
    1221 effect of the illumination differences over the detector surface.
    1222 This is done by dithering a series of science exposures with a given
    1223 pointing.  By fully calibrating these exposures with the initial flat
    1224 model, and then comparing the measured fluxes for the same star as a
    1225 function of position on the detector, we can determine position
    1226 dependent scaling factors.  From the set of scaling factors for the
    1227 full catalog of stars observed in the dithered sequence, we can
    1228 construct a model of the error in the initial flat model as a function
    1229 of detector position.  Applying a correction that reduces the
    1230 amplitude of these errors produces a flat field model that better
    1231 represents the true detector response.
    1232 
    1233 The flat model appears stable with time, although directly measuring
    1234 this is as difficult as originally constructing the model.  However,
    1235 due to the photometric consistency observed in the final catalog of
    1236 GPC1 measurements \citep{MagnierXXX}, we can be confident that the
    1237 flat model does not have a significant time dependent component.
    1238 
    1239 \subsection{Pattern correction}
    1240 \label{sec:pattern}
    1241 
    1242 Due to detector specific issues that are not cleanly removed by the
    1243 dark model, we have a set of ``pattern'' corrections that are applied
    1244 to some selection of the OTAs in the camera.  This is done to reduce
    1245 the effect that detector differences have on the measured astronomical
    1246 signal that are not stable enough to be corrected with a static model.
    1247 Because of this, the pattern corrections attempt to identify and
    1248 correct the detector issues based on appropriate filtering the
    1249 individual science exposures.
    1250 
    1251 The PATTERN.ROW correction is used to remove any remaining row-by-row
    1252 bias variation, and the PATTERN.CELL and PATTERN.CONTINUITY
    1253 corrections attempt to ensure that the cells of a given OTA are
    1254 consistent with the other cells on that OTA. 
    1255 
    1256 \subsubsection{Pattern Row}
    1257 % http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Bias_Pattern_Study
    1258 As discussed above in the dark and noisemap sections, certain
    1259 detectors have significant bias offsets between adjacent rows, caused
    1260 by noise in the camera control electronics.  The magnitude of these
    1261 offsets increases as the distance from the readout amplifier
    1262 increases, resulting in horizontal streaks that are more pronounced
    1263 along the large x pixel edge of the cell.  As the level of the offset
    1264 is apparently random between exposures, the dark correction cannot
    1265 fully remove this structure from the images, and the noisemap value
    1266 only indicates the level of the average variance added by these bias
    1267 offsets.  Therefore, we apply the PATTERN.ROW correction in an attempt
    1268 to mitigate the offsets and correct the image values.  To force the
    1269 rows to agree, a second order clipped polynomial is fit to each row in
    1270 the cell.  Four fit iterations are run, and pixels $2.5\sigma$ deviant
    1271 are excluded from subsequent fits, to minimize the effect stars and
    1272 other astronomical signals have.  This final trend is then subtracted
    1273 from that row.  Simply doing this subtraction will also have the
    1274 effect of removing the background sky level.  To prevent this, the
    1275 constant and linear terms for each row are stored, and linear fits are
    1276 made to these parameters as a function of row, perpendicular to the
    1277 initial fits.  This produces a plane that is added back to the image
    1278 to restore the background offset and any linear ramp that exists in
    1279 the sky.
    1280 
    1281 This correction was required on all cells on all OTAs prior to
    1282 2009-12-01, at which point a modification of the camera electronics
    1283 reduced the scale of the row-by-row offsets for the majority of the
    1284 OTAs.  As a result, we only apply this correction to the cells where
    1285 it is still necessary, as shown in Figure \ref{fig: pattern row
    1286   cells}.  A list of these cells is listed in Table
    1287 \ref{tab:pattern_row_cells}.
    1288 
    1289 Although this correction does largely resolve the row-by-row offset
    1290 issue in a satisfactory way, large and bright astronomical objects can
    1291 bias the fit significantly.  This results in an oversubtraction of the
    1292 offset near these objects.  As the offsets are calculated on the pixel
    1293 rows, this oversubtraction is not uniform around the object, but is
    1294 preferentially along the horizontal x axis of the object.  Most
    1295 astronomical objects are not significantly distorted by this, with
    1296 this only becoming on issue for only bright objects comparable to the
    1297 size of the cell (598 pixels = 150").
    1298 
    1299 %% \czwdraft{keep this?}  This row-by-row offset is visible in similar
    1300 %% camera designs, and has been removed by identifying the noise signal
    1301 %% in the pixel data stream.  By taking the FFT of the pixels and a
    1302 %% reference signal, the frequency of this noise can be isolated and
    1303 %% removed, resulting in a much cleaner image.  However, GPC1 does not
    1304 %% record the value of the reference signal, instead automatically
    1305 %% subtracting it from the data values.  Without this comparison signal,
    1306 %% we have been unable to reproduce this method, as there is no obvious
    1307 %% FFT component visible.
    1308 
    1309 \begin{deluxetable}{lcccc}
    1310   \tablecolumns{3}
    1311   \tablewidth{0pc}
    1312   \tablecaption{Cells which have PATTERN.ROW correction applied}
    1313   \tablehead{\colhead{OTA} & \colhead{Cell columns} & \colhead{Additional cells}}
    1314   \startdata
    1315   OTA11 &  & xy02, xy03, xy04, xy07 \\
    1316   OTA14 &  & xy23 \\
    1317   OTA15 & 0 & \\
    1318   OTA27 & 0, 1, 2, 3, 7 & \\
    1319   OTA31 & 7 & \\
    1320   OTA32 & 3, 7 & \\
    1321   OTA45 & 3, 7 & \\
    1322   OTA47 & 0, 3, 5, 7 & \\
    1323   OTA57 & 0, 1, 2, 6, 7 & \\
    1324   OTA60 &  & xy55 \\
    1325   OTA74 & 2, 7 & \\
    1326   \enddata
    1327   \label{tab:pattern_row_cells}
    1328 \end{deluxetable}
    1329 
    1330 \begin{figure}
    1331   \centering
    1332   \includegraphics[width=0.9\hsize,angle=0,clip]{images/pattern_row_edit.png}
    1333   \caption{Diagram illustrating in red which cells on GPC1 require the PATTERN.ROW correction to be applied.  The footprint of each OTA is outlined, and cell xy00 is marked with either a filled box or an outline.  The labeling of the non-existent corner OTAs is provided to orient the focal plane.}
    1334   \label{fig: pattern row cells}
    1335 \end{figure}
    1336 
    1337 \begin{figure}
    1338   \centering
    1339   \begin{minipage}{0.45\hsize}
    1340     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_nopat.png}
    1341 %    \caption{(a)}
    1342 %  \end{subfigure}%
    1343 %  \begin{subfigure}[]{.45\hsize}
    1344   \end{minipage}%
    1345   \begin{minipage}{0.45\hsize}
    1346     \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_pat.png}
    1347 %    \caption{(b)}
    1348 %  \end{subfigure}
    1349   \end{minipage}
    1350   \caption{Example of the PATTERN.ROW correction on exposure o5379g0103o OTA57 cell xy00 (i-filter 45s).  The left panel shows the cell with all appropriate detrending except the PATTERN.ROW, and the right shows the same cell with PATTERN.ROW applied.  The correction reduces the correlated noise on the right side, which is most distant from the read out amplifier.  There is a slight over subtraction along the rows near the bright star.}
    1351 \end{figure}
    1352 
    1353 \subsubsection{Pattern Cell}
    1354 
    1355 As the measured background level of a given cell may not exactly match
    1356 that of its neighbors, fitting a smooth background model over the full
    1357 OTA can result in over and under-subtraction of the sky level at the
    1358 cell boundary discontinuities.  The PATTERN.CELL correction was an
    1359 initial attempt to remove this effect on the worst cells, by forcing
    1360 all the cells of an OTA to the same level.  Each cell had the median
    1361 value measured, and then each cell had an offset added that shifts the
    1362 cell to match the median of those medians.
    1363 
    1364 This correction is reasonable when the astronomical signal is smooth,
    1365 with no objects that are large relative to the size of an individual
    1366 cell.  However, the presence of large galaxies (or even bright stars)
    1367 can bias the offsets for some cells from their neighbors.  Because of
    1368 this issue, we no longer apply this correction to any data.
    1369 
    1370 \subsubsection{Pattern Continuity}
    1371 
    1372 As the PATTERN.CELL correction was insufficient in many situations, we
    1373 designed a replacement correction that would reduce the background
    1374 distortion for large objects.  In addition, studies of the background
    1375 level illustrated that the row-by-row bias can introduce small
    1376 background gradient variations along the rows of the cells that is not
    1377 stable enough to be completely fit by the dark model.  This common
    1378 feature across the columns of cells results in a ``saw tooth'' pattern
    1379 horizontally across an OTA, and as the background model fits a smooth
    1380 sky level, this induces over and under subtraction at the cell
    1381 boundaries.  As the PATTERN.CELL was designed to correct changes only
    1382 in the median value between cells, it could not adequately resolve
    1383 this higher order issue.
    1384 
    1385 The replacement for PATTERN.CELL is the PATTERN.CONTINUITY correction,
    1386 which attempts to match the edges of a cell to those of its neighbors.
    1387 For each cell, a thin box 10 pixels wide on each edge is extracted and
    1388 the median value of unmasked values calculated for that box.  These
    1389 median values are then used to construct a vector of differences
    1390 $\Delta_i = \sum_{j} Edge_{i} - Edge_{j}$, along with a matrix of
    1391 associations $A_{i,i'} = \sum_{j} \delta(i,j) \delta(j,i')$ denoting
    1392 which cell boundaries are adjacent.  By solving the system $A x =
    1393 diff$, we find the set of offsets $x_i$ to be applied to each cell to
    1394 ensure the minimum differences between all cell edges and their
    1395 neighbors.
    1396 
    1397 For OTAs that initially show the saw tooth pattern, the effect of this
    1398 correction is to align the cells into a single ramp, at the expense of
    1399 the absolute background level.  However, as we subtract off a smooth
    1400 background model prior to doing photometry, these deviations from an
    1401 absolute sky level are unimportant.  The fact that the final ramp is
    1402 smoother than it would be otherwise also allows for the background
    1403 subtracted image to more closely match the astronomical sky, without
    1404 significant errors at cell boundaries.  An example of the effect of
    1405 this correction on an image profile is shown in Figure \ref{fig:dark switching}.
    1406 
    1407 %% \begin{figure}
    1408 %%   \centering
    1409 %%   \caption{Continuity example, with background issue.}
    1410 %%   \label{fig: continuity example}
    1411 %% \end{figure}
    1412 
    1413 \subsection{Fringe correction}
    1414 \label{sec:fringe}
    1415 % det_id 296 is the fringe we use.
    1416 
    1417 \czwdraft{This is still a mess}
    1418 
    1419 Due to variations in the thickness of the detectors, we observe
    1420 interference patterns at the infrared end of the filter set, as the
    1421 wavelength of the light becomes comparable to the thickness of the
    1422 detectors.  Visually inspecting the images shows that the fringing is
    1423 most prevalent in the y-filter images, with negligible fringing in
    1424 other bands.  As a result of this, we only apply a fringe correction
    1425 to the y filter data.
    1426 
    1427 The fringe used for PV3 processing was constructed from a set of 20
    1428 120s science exposures.  These exposures are overscan subtracted, and
    1429 corrected for non-linearity, and have the dark and flat models
    1430 applied.  These images are smoothed with a Gaussian of $\sigma = 2$
    1431 pixels to minimize pixel to pixel noise.  The fringe image data is
    1432 then constructed by calculating the clipped mean of the input images
    1433 with two iteration of clipping at the $3\sigma$ level.
    1434 
    1435 A course background model is constructed by calculating the median on
    1436 a 3x3 grid (approximately 200x200 pixels each).  A set of 1000
    1437 randomly selected points are selected on the fringe image in each
    1438 cell, and a median calculated for this position in a 10x10 pixel box,
    1439 with the background level subtracted.  These sample locations provide
    1440 scale points to allow the amplitude of the measured fringe to be
    1441 compared to that found on science images.
    1442 
    1443 To apply the fringe, the same sample locations are measured on science
    1444 image to determine the relative strength of the fringing in that
    1445 particular image.  A least squares fit between the fringe measurements
    1446 and the corresponding measurements on the science image provides the
    1447 scale factor multiplied to the fringe before it is subtracted from the
    1448 science image.
    1449 
    1450 \begin{figure}
    1451   \centering
    1452   \begin{minipage}{0.5\hsize}
    1453     \includegraphics[width=1.0\hsize,angle=0,clip]{images/o5220g0025o_XY53_nofringe.png}
    1454 %    \caption{(a)}
    1455 %  \end{subfigure}%
    1456 %  \begin{subfigure}[]{.45\hsize}
    1457   \end{minipage}%
    1458   \begin{minipage}{0.5\hsize}
    1459     \includegraphics[width=1.0\hsize,angle=0,clip]{images/o5220g0025o_XY53_fringe.png}
    1460 %    \caption{(b)}
    1461 %  \end{subfigure}
    1462   \end{minipage}
    1463   \caption{Example of the y-filter fringe pattern on exposure o5220g0025o OTA53 (y-filter 30s).  The left panel shows the OTA mosaic with all detrending except the fringe correction, while the right shows the same including the fringe correction.  Both images have been smoothed with a Gaussian with $\sigma = 3$ pixels to highlight the faint and large scale fringe patterns. \czwdraft{See if there's a way to have mana produce images larger than the screen size.}}
    1464   \label{fig: fringe example}
    1465 \end{figure}
    1466 
    1467 \subsection{Background subtraction}
    1468 \label{sec:background}
    1469 
    1470 Once all other detrending is done, the pixels from each cell are
    1471 mosaicked into the full $4846\times{}4868$ pixel OTA image.  A
    1472 background model for the full OTA is then determined prior to the
    1473 photometric analysis.  The mosaicked image is binned into
    1474 $800\times{}800$ pixel bins, centered on the image center, and
    1475 overlapping by a factor of 2 in both axes.  These bins have 10000
    1476 random samples drawn, and a binned cumulative distribution function is
    1477 generated.  These bins are interpolated to find the best mean value at
    1478 the $50\%$ level, as well as the distribution $\sigma$ by estimating
    1479 from the $32\%$ and $68\%$ levels.  Repeating this across all bins
    1480 results in a $13\times{}13$ grid of background bins, which are
    1481 bilinearly interpolated to generate the background model to subtract.
    1482 Each object in the photometric catalog has a SKY and SKY\_SIGMA value
    1483 based on this model as well.
    1484 
    1485 %% * Magic
    1486 %% * Warping
    1487 %%   * warping kernel
    1488 %%   * linear-by-pieces
    1489 %%   * Covariance
    1490 %%   * def of skycells?
    1491 %% * Stacking
    1492 %%   * pixel combination rules
    1493 %%   * pixel rejections
    1494 %%   * convolution for matching (success and failure)
    1495 %% * Difference Image analysis
    1496 
    1497 
    14981466\section{Warping}
    14991467\label{sec:warping}
Note: See TracChangeset for help on using the changeset viewer.