Index: trunk/doc/release.2015/ps1.detrend/detrend.tex =================================================================== --- trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 39815) +++ trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 39817) @@ -181,14 +181,15 @@ \czwdraft{Should there be a discussion of any header keywords/OTA file formats?} -Section \ref{sec:detrend construction} provides an overview of the -detrend creation process for GPC1, with details of the application of -those detrends to correct particular issues in Section -\ref{sec:detrending}. An analysis of the algorithms used to complete -the \ippstage{warp} (section \ref{sec:warping}) and \ippstage{stack} -(section \ref{sec:stacking}) stage transformations of the image data -to from the detector frame to a common sky frame, and the co-adding of -those common sky frame images continues after the list of detrend -steps. Finally, a discussion of the remaining issues and possible -future improvements is presented in section \ref{sec:discussion}. +Section \ref{sec:detrending} provides an overview of the detrending +process that corrects the instrumental signatures of GPC1, with +details of the construction of those detrends in Section +\ref{sec:detrend construction}. An analysis of the algorithms used to +complete the \ippstage{warp} (section \ref{sec:warping}) and +\ippstage{stack} (section \ref{sec:stacking}) stage transformations of +the image data to from the detector frame to a common sky frame, and +the co-adding of those common sky frame images continues after the +list of detrend steps. Finally, a discussion of the remaining issues +and possible future improvements is presented in section +\ref{sec:discussion}. @@ -216,4 +217,1135 @@ %\section{General Detrend Discussion} %\label{sec:detrending} + + +\section{GPC1 Detrend Details} +\label{sec:detrending} + +Ensuring a consistent and uniform detector response across the +three-degree diameter field of view of the GPC1 camera is essential to +a well calibrated survey. Many standard image detrending steps are +done for GPC1, with overscan subtraction removing the detector bias +level, dark frame subtraction to remove temperature and exposure time +dependent detector glows, and flat field correction to remove pixel to +pixel response functions. We also construct fringe correction for the +reddest data in the y filter, to remove the interference patterns that +arise in that filter due to the variations in the thickness of the +detector surface. + +These corrections, however, assume that the detector response is +linear across the full range of values. This is not universally the +case with GPC1, and this requires an additional set of detrending +steps to remove these non-linear responses. The first of these is the +\ippprog{burntool} correction, which removes the persistence trails +caused by the incomplete transfer of charge along the readout columns. +This bright-end nonlinearity is generally only evident for the +brightest stars, as only pixels that are at or beyond the saturation +point of the detector have this issue. More widespread is the +non-linearity at the faint end of the pixel range. Some readout cells +and some readout cell edge pixels experience a sag relative to linear +at low illumination, such that faint pixels appear fainter than +expected. The correction to this requires amplifying the pixel values +in these regions to match the expected model. + +The final non-linear response issue has no good option for correction. +Large regions of some OTA cells experience significant charge transfer +issues, making them unusable for science observations. These regions +are therefore masked in processing, with these CTE regions making up +the largest fraction of masked pixels on the detector. Other regions +are masked for other regions, such as static bad pixel features or +temporary readout masking caused by issues in the camera electronics +that make these regions unreliable. These all contribute to the +detector mask, which is augmented in each exposure for dynamic +features that are masked based on the astronomical features within the +field of view. + +For the PV3 processing, all detrending is done by the +\ippprog{ppImage} program. This program applies the detrends to the +individual cells, and then an OTA level mosaic is constructed for the +science image, the mask image, and the variance map image. The single +epoch photometry is done at this stage as well. The following +subsections (\ref{sec:burntool} - \ref{sec:background}) detail these +detrending steps, presented in the order in which they are applied to +the individual OTA image data. + +\subsection{Burntool / Persistence effect} +\label{sec:burntool} + +Pixels that approach the saturation point on GPC1, which varies by +readout with common values around 60000 DN, cause persistence problems +on that and subsequent images. During the read out process of an +image with such a bright pixel, some of the charge associated with it +is not fully shifted down the detector column toward the amplifier. +As a result, this charge remains in the starting cell, and is +partially collected in subsequent shifts, resulting in a ``burn +trail'' that extends from the center of the bright source away from +the amplifier (vertically along the pixel columns toward the top of +the cell). + +This incomplete charge shifting in nearly full wells continues as each +row is read out. This results in a remnant charge being deposited in +the pixels that the full well was shifted through. In following +exposures, this remnant charge leaks out, resulting in a trail that +extends from the initial location of the bright source on the previous +image towards the amplifier (vertically down along the pixel column). +This remnant charge can remain on the detector for up to thirty +minutes, requiring the locations of these ``burns'' be retained +between exposures. + +Both of these types of persistance trails are measured and optionally +repaired via the \ippprog{burntool} program. This program does an +initial scan of the images, and identifies objects with pixel values +brighter than a conservative threshold of 30000 DN. The trail from +the peak of that object is fit with a one-dimensional power law in +each pixel column above the threshold, based on empirical evidence +that this is the functional form of this persistence effect. This +also matches the expectation that a constant fraction of charge is +incompletely transfered at each shift beyond the persistence +threshold. Once this fit is done, the model can be subtracted from +the image, and the location of the star is stored in a table along +with the exposure PONTIME, which denotes the number of seconds since +the detector was last powered on, and provides an internally consistent +time scale. + +For subsequent exposures, the table associated with the previous image +is read in, and after correcting trails from the stars on the new +image, the positions of the bright stars from the table are used to +check for remnant trails on the image. These are fit and subtracted +using a one-dimensional exponential model, again based on empirical +studies. If a significant model is found, then this location is +retained in the image output table. If not, the old burn is allowed +to expire. + +The main concern with this method of correcting the persistance trails +is that it is based on fits to the raw image data, which may have +other signal sources not determined by the persistence effect. The +presence of other stars or artifacts along the path of the burn can +result in a poor model to be fit, resulting in either an over- or +under-subtraction of the persistance burn. For this reason, the image +mask is marked with a value indicating that this correction has been +applied. These pixels are not fully excluded, but they are marked as +suspect, which allows them to be excluded from consideration in +subsequent stages, such as image stacking. + +Another concern is that the cores of very bright stars are deformed by +this process, as the burntool fitting subtracts flux +from only one side of the star. As most stars that result in burns already +have saturated cores, they are already ignored for the purpose of +PSF determination and are flagged as saturated by the photometry +reduction. + +\begin{figure} + \centering + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt_trail.png} +% \caption{(a)} +% \end{subfigure}% +% \begin{subfigure}[]{.45\hsize} + \end{minipage}% + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt_trail.png} +% \caption{(b)} +% \end{subfigure} + \end{minipage} + + \caption{Example of a profile cut along the y-axis through a bright star on exposure o5677g0123o OTA11 in cell xy60 (left panel) and on the subsequent exposure o5677g0124o (right panel). In both figures, the green points show the image corrected with all appropriate detrending steps, but without burntool applied, illustrating the amplitude of the persistence trails. The red points show the same data after the burntool correction, which reduces the impact of these features. Both exposures are in the g-filter with exposure times of 43s} +\end{figure} + +\begin{figure} + \centering + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_nobt.png} +% \caption{(a)} +% \end{subfigure}% +% \begin{subfigure}[]{.45\hsize} + \end{minipage}% + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_nobt.png} +% \caption{(b)} +% \end{subfigure} + \end{minipage} + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt.png} +% \caption{(a)} +% \end{subfigure}% +% \begin{subfigure}[]{.45\hsize} + \end{minipage}% + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt.png} +% \caption{(b)} +% \end{subfigure} + \end{minipage} + \caption{Example of OTA11 cell xy60 on exposures o5677g0123o (left) and o5677g0124o (right). The top panels show the image with all appropriate detrending steps, but without burntool, and the bottom show the same with burntool applied. There is some slight over subtraction in fitting the initial trail, but the impact of the trail is greatly reduced in both exposures.} +\end{figure} + + +\subsection{Overscan} +\label{sec:overscan} + +Each cell on GPC1 has an overscan region that covers the first 34 +columns of each row, and the last 10 rows of each column. No light +lands on these pixels, so the image region is trimmed to exclude them. +Each row has an overscan value subtracted, calculated by finding the +median value of that row's overscan pixels and then smoothing between +rows with a three-row boxcar median. + +\subsection{Non-linearity Correction} +\label{sec:nonlinearity} +% check notebook, 2010-07/08 + +The pixels of GPC1 are not uniformly linear at all flux levels. In +particular, at low flux levels, some pixels have a tendency to sag +relative to the expected linear value. This effect is most pronounced +along the edges of the detector cells, although some entire cells show +evidence of this effect. + +To correct this sag, we studied the flux behavior of a series of flat +frames for a ramp of exposure times with approximate logarithmically +equal spacing between 0.01s and 57.04s. As the exposure time +increases, the flux on each pixel also increases in what is expected +to be a linear manner. Each of these flat exposures in this ramp is +overscan corrected, and then the median is calculated for each cell, +as well as for the rows and columns within ten pixels of the edge of +the science region. From these median values at each exposure time +value, we can construct the expected trend by fitting a linear model, +$f_{region} = G * t_{exp} + B$, to determine the gain, $G$, and the +bias, $B$, for the region considered. This fitting was limited to only +the range of fluxes between 12000 and 38000 counts, as these ranges +were found to match the linear model well. This range avoids the +non-linearity at low fluxes, as well as the possibility of high-flux +non-linearity effects. + +We store the average flux measurement and deviation from the linear +fit for each exposure time for all regions on all detector cells in +the linearity detrend look up tables. When this is applied to science +data, these lookup tables are loaded, and a linear interpolation is +performed to determine the correction needed for the flux in that +pixel. This look up is performed for both the row and column of each +pixel, to allow the edge correction to be applied where applicable, +and the full cell correction elsewhere. The average of these two +values is then applied to the pixel value, reducing the effects of +pixel nonlinearity. + +This non-linearity effect appears to be stable in time for the +majority of the detector pixels, with little evident change over the +survey duration. However, as the non-linearity is most pronounced at +the edges of the detector cells, those are the regions where the +correction is most likely to be incomplete. Because of this fact, +most pixels in the static mask with either the DARKMASK or FLATMASK +bit set are found along these edges. As the non-linearity correction +is unable to reliably restore these pixels, they produce inconsistent +values after the dark and flat have been applied, and are therefore +rejected. + +%% exptime n_included/det_id = 372 +%% clearly this isn't the one used, as 3-12 spans three data points, poorly.x +%% 0.01 2 +%% 0.14 2 +%% 0.27 2 +%% 0.49 2 +%% 0.72 2 +%% 1.06 2 +%% 1.41 2 +%% 2.02 2 +%% 2.63 2 +%% 3.94 2 +%% 5.25 2 +%% 8.74 2 +%% 13.09 2 +%% 17.4 2 +%% 20.86 2 +%% 24.3 2 +%% 27.78 2 +%% 31.24 2 +%% 34.65 2 +%% 38.12 2 +%% 42.41 2 +%% 46.69 2 +%% 51.89 2 +%% 57.04 2 + + +%http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearity_AllEdges +%http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearityArchive + +\begin{figure} + \centering + \includegraphics[width=0.9\hsize,angle=0,clip]{images/linearity_XY27_xy16.png} + \caption{Example plot of the linearity correction as a fraction of observed flux for OTA27, cell xy16.} +\end{figure} + +\subsection{Dark/Bias Subtraction} +\label{sec:dark} +% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Dark_Model + +The dark model we make for GPC1 considers each pixel individually, +independent of any neighbors. To construct this model, we fit a +multi-dimensional model to the array of input pixels from a randomly +selected set of 100-150 overscan and non-linearity corrected dark +frames chosen from a given date range. The model fits each pixel as a +function of the exposure time $t_{exp}$ and the detector temperature +$T_{chip}$ of the input images such that $\mathrm{dark} = a_0 + a_1 +t_{exp} + a_2 T_{chip} t_{exp} + a_3 T_{chip}^2 t_{exp}$. This +fitting uses two iterations to produce a clipped fit, rejecting at the +$3\sigma$ level. The final coefficients $a_i$ for the dark model are +stored in the detrend image. The constant $a_0$ term includes the +residual bias signal after overscan subtraction, and as such, a +separate bias subtraction is not necessary. + +Applying the dark model is simply a matter of calculating the response +to the exposure time and detector temperature for the image to be +corrected, and subtracting the resulting dark signal from the image. + +\subsubsection{Time evolution} + +The dark model is not consistently stable over the full survey, with +significant drift over the course of multiple months. Some of the +changes in the dark can be attributed to changes in the voltage +settings of the GPC1 controller electronics, but the majority seem to +be the result of some unknown parameter. We can separate the dark +model history of GPC1 into three epochs. The first epoch covers all +data taken prior to 2010-01-23. This epoch used a different header +keyword for the detector temperature, making data from this epoch +incompatible with later dark models. + +The second epoch covers data between 2010-01-23 and 2011-05-01, and is +characterized by a largely stable but oscillatory dark solution. +There are two modes that the dark model switches between apparently at +random. No clear cause has been established for the switching, but +there are clear differences between the two modes that require the +observation dates to be split to use the model that is most +appropriate. + +The initial evidence of these two modes comes from the discovery of a +slight gradient along the rows of certain cells. This is a result of +a drift in the bias level of the detector as it is read out. An +appropriate dark model should remove this gradient entirely. For +these two modes, the direction of this bias drift is different, so a +single dark model generated from all dark images in the time range +over corrects the positive-gradient mode, and under corrects the +negative-gradient mode. Upon identifying this two-mode behavior, and +determining the dates each mode was dominant, two separate dark +models were constructed from appropriate ``A'' and ``B'' mode dark +frames. Using the appropriate dark minimizes the effect of this bias +gradient in the dark corrected data. + +The bias drift gradients of the mode switching can be visualized in +Figure \ref{fig:dark switching}. This figure shows the image profile +along the x-pixel axis binned along the full y-axis of the first row +of cells. The raw data is shown, illustrating the positional +depenendence the dark signal has on the image values. In addition, +both the correct B-mode dark and incorrect A-mode dark have been +applied to this image, showing that although both correct the bulk of +the dark signal, using the incorrect mode creates larger intensity +gradients. + +After 2011-05-01, the two-mode behavior of the dark disappears, and is +replaced with a slow observation date dependent drift in the magnitude +of the gradient. This drift is sufficiently slow that we have modeled +it using three observation date independent dark model for different +date ranges. These darks cover the range from 2011-05-01 to +2011-08-01, 2011-08-01 to 2011-11-01, and 2011-11-01 and on. The +reason for this time evolution is unknown, but as it is correctable +with a small number of dark models, this does not significantly impact +detrending. + +\begin{figure} + \centering +% \begin{subfigure}[]{.45\hsize} + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_M_OS_NL_XY23_b1.jpg} +% \caption{(a)} +% \end{subfigure}% +% \begin{subfigure}[]{.45\hsize} + \end{minipage}% + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_to_DARK_XY23_b1.jpg} +% \caption{(b)} +% \end{subfigure} + \end{minipage} + \caption{An example of the dark model application to exposure o5677g0123o, OTA23 (2011-04-26, 43s g-filter). The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, and the detector non-linearity corrected. The right panel, shows the same exposure with the dark applied in addition to the processing shown on the left.} +\end{figure} + +\begin{figure} + \centering + \includegraphics[width=0.9\hsize,angle=0,clip]{images/B_profile_ex.png} + \caption{Example showing a profile cut across exposure o5676g0195, OTA67 (2011-04-25, 43s g-filter). The entire first row of cells (xy00-xy07) have had a median calculated along each pixel column on the OTA mosaicked image. Arbitrary offsets have been applied to shift the curves to not overlap. The top curve (in purple) shows the initial raw profile, with no dark model applied. The next curve (in green) shows the smoother profile after applying the correct B-mode dark model. Applying the incorrect A-mode dark results in the blue curve, which shows a significant increase in gradients across the cells. The orange curve shows the result of the PATTERN.CONTINUITY correction. Although this creates a larger gradient across the mosaicked images, it decreases the cell-to-cell level changes. The final yellow curve shows the final image profile after all detrending and background subtraction, and has not had an offset applied. The bright source at the cell xy00 to xy01 transition is a result of a large optical ghost, which due to the area covered, increases the median level more than the field stars.} + \label{fig:dark switching} +\end{figure} + +\subsubsection{Video Dark} +\label{sec:video_darks} + +The dark signal is stronger in cell corners due to glow from the +read-out amplifiers. The standard dark model corrects this for most +observations. However, as mentioned above, when a cell is repeatedly +read in video mode, the dark model for the OTA containing it changes. +Surprisingly, added reads for the video cell do not amplify the +amplifier glow, but rather decrease the dark signal in these regions. +As a result, using the standard dark model on the data for these OTAs +results in oversubtraction of the corner glow. + +Video darks have been constructed to eliminate the effect this +observational change has on the final image quality. This was done by +running the standard dark construction process on a series of dark +frames that have had the video signal enabled for some cells. GPC1 +can only run video signals on a subset of the OTAs at a given time. +This requires two passes to enable the video signal across the full +set of OTAs that support video cells. This is convenient for the +process of creating darks, as those OTAs that do not have video +signals enabled create standard dark models, while the video dark is +created for those that do. + +This simultaneous construction of video and standard dark models is +useful, as it provides the ability to isolate the response on the +standard dark from the video signals. Isolating this response is +essential for attempting to create archival video darks. We only have +raw video dark frame data after 2012-05-16, when this problem was +initially identified, so any data prior to that can not be directly +corrected for the video dark signal. Isolating the video signal +response allows linear corrections to the pre-existing standard dark +models for archival data. Testing this shows that constructing a +video dark for older data simply as $VD_{2009} = D_{2009} - D_{Modern} ++ VD_{Modern}$ produces a satisfactory result that does not +oversubtract the amplifier glow. This is shown in figure +\ref{fig:video_darks}, which shows video cells from before 2012-05-16, +corrected with both the standard and video darks, with the early video +dark constructed in such a manner. + +\begin{figure} + \centering +% \begin{subfigure}[]{.45\hsize} + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_Rdark_XY22_b1.jpg} +% \caption{(a)} +% \end{subfigure}% +% \begin{subfigure}[]{.45\hsize} + \end{minipage}% + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_VDdark_XY22_b1.jpg} +% \caption{(b)} +% \end{subfigure} + \end{minipage} + \caption{An example of the video dark model application to exposure o5677g0123o, OTA22 (2011-04-26, 43s g-filter), which has a video cell located in cell xy16. The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, the detector non-linearity corrected, and a regular dark applied. The right panel, shows the same exposure with a video dark applied instead of the standard dark. The main impact of this change is the improved correction of the corner glows, which are oversubtracted with the standard dark.} + \label{fig:video_darks} +\end{figure} + +\subsection{Noisemap} +\label{sec:noisemap} + +Based on a study of the positional dependence of all detected sources, +we have discovered that the cells in GPC1 do not have uniform noise +characteristics. Instead, there is a gradient along the pixel rows, +with the noise generally higher away from the read out amplifier +(higher cell x pixel positions). This is likely an effect of the +row-by-row bias issue discussed below. This gradient causes the read +noise to increase as the row is read out. As a result of this +increased noise, more sources are detected in the higher noise regions +when the read noise is assumed constant across the readout. To +mitigate this noise gradient, we constructed an initial set of +noisemap images by measuring the median variance on bias frames. The +variance is calculated in boxes of 20x20 pixels, and then linearly +interpolated to cover the full image. + +Unfortunately, due to correlations within this noise, the variance +measured from the bias images does not fully remove the positional +dependence of objects that are detected. This simple noisemap +underestimates the noise observed when the image is filtered during +the object detection process. This filtering convolves the background +noise with a PSF, which has the effect of amplifying the correlated +peaks in the noise. This amplification can therefore boost background +fluctuations above the threshold used to select real objects, +contaminating the final object catalogs. + +In the detection process, we expect false positives at a rate equal to +the one-tailed probability beyond the detection threshold. For these +tests, only detections measured at the $\sigma_{thresh} = 5\sigma$ +level are used, to match that used in the photometry on science data. +This probability can be converted into a number of false number by +considering a given area. As the detections must be isolated to not +be detected as an extended object, this area must be reduced by the +area a given PSF occupies. Combining this, we find that we expect a +probability $P = 1 - \Phi_{normal}(5) = \frac{1}{2} +\erfcinv\left(\frac{5}{\sqrt{2}}\right)$, and an area given $N$ +exposures of area $X\times Y$, $A = \frac{X \times Y \times + N}{A_{PSF}}$. For a typical $1"$ seeing, $A_{PSF}$ is approximately +16 pixels. Using this model for the false positives, we found that +the added read noise was insufficient to account for the observed +false positive rate. Inverting this relation, we can measure +$\sigma_{obs}$, the true threshold level based on the number of false +positives observed. This $\sigma_{obs}$ is the combined to form a +boost factor $B = \sigma_{thresh} / \sigma_{obs}$ that amplifies the + noisemap to match the observed false detection rate. + +The row-to-row variations that contribute to the extra noise are +related to the dark model, and because of this, as the dark model +changes, the effective noise also changes. To ensure that the +noisemap accurately matches the true noise level, we have created +different noisemap models for the three major time ranges of the dark +model. We do not see any strong evidence that the noisemaps have the +A/B modes visible in the dark, and so we do not generate different +models for each individual dark model. The additional pixel-to-pixel +variance from this noisemap is added to the Poissonian variance to +form the science variance image generated by the \ippstage{chip} +processing. + +\subsection{Flat} + +Determining a flat field correction for GPC1 is a challenging +endeavor, as the wide field of view makes it difficult to construct a +uniformly illuminated image. Using a dome screen is not possible, as +the variations in illumination and screen rigidity create large +scatter between different images that are not caused by the detector +response function. Because of this, we use sky flat images taken at +twilight, which are more consistently illuminated than screen flats. +We calculate the mean of these images to determine the initial flat +model. + +From this starting skyflat model, we construct a photometric +correction to remove the effect of the illumination differences over +the detector surface. This is done by dithering a series of science +exposures with a given pointing. By fully calibrating these exposures +with the initial flat model, and then comparing the measured fluxes +for the same star as a function of position on the detector, we can +determine position dependent scaling factors. From the set of scaling +factors for the full catalog of stars observed in the dithered +sequence, we can construct a model of the error in the initial flat +model as a function of detector position. Applying a correction that +reduces the amplitude of these errors produces a flat field model that +better represents the true detector response. + +\czwdraft{EAM: the flat-field construction part needs to make a clearer discussion of +the skyflat vs the photometric correction (photflat) built initially for +the survey vs the flat-field corrections determined in the database as part +of ubercal (for the latter, you should just mention the concept -- it will +also be mentioned in the calibration paper). The statement that the +flat-field response was stable is not true since we did need 5 'seasons'.} + +In addition to this flat field applied to the individual images, the +ubercal process used to calibrate the database of all detections +\citep{ubercal} constructs internal ``flat field'' corrections. +Although a single set of image flat fields was used for the entire PV3 +survey, five separate ``seasons'' of database flat fields were needed +to ensure proper calibration. This indicates that the flat field +response is not completely fixed in time. + +\subsection{Pattern correction} +\label{sec:pattern} + +Due to detector specific issues that are not cleanly removed by the +dark model, we have a set of ``pattern'' corrections that are applied +to some selection of the OTAs in the camera. This is done to reduce +the effect that detector differences have on the measured astronomical +signal that are not stable enough to be corrected with a static model. +Because of this, the pattern corrections attempt to identify and +correct the detector issues based on appropriate filtering the +individual science exposures. + +The PATTERN.ROW correction is used to remove any remaining row-by-row +bias variation, and the PATTERN.CELL and PATTERN.CONTINUITY +corrections attempt to ensure that the cells of a given OTA are +consistent with the other cells on that OTA. + +\subsubsection{Pattern Row} +% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Bias_Pattern_Study +As discussed above in the dark and noisemap sections, certain +detectors have significant bias offsets between adjacent rows, caused +by noise in the camera control electronics. The magnitude of these +offsets increases as the distance from the readout amplifier +increases, resulting in horizontal streaks that are more pronounced +along the large x pixel edge of the cell. As the level of the offset +is apparently random between exposures, the dark correction cannot +fully remove this structure from the images, and the noisemap value +only indicates the level of the average variance added by these bias +offsets. Therefore, we apply the PATTERN.ROW correction in an attempt +to mitigate the offsets and correct the image values. To force the +rows to agree, a second order clipped polynomial is fit to each row in +the cell. Four fit iterations are run, and pixels $2.5\sigma$ deviant +are excluded from subsequent fits, to minimize the effect stars and +other astronomical signals have. This final trend is then subtracted +from that row. Simply doing this subtraction will also have the +effect of removing the background sky level. To prevent this, the +constant and linear terms for each row are stored, and linear fits are +made to these parameters as a function of row, perpendicular to the +initial fits. This produces a plane that is added back to the image +to restore the background offset and any linear ramp that exists in +the sky. + +This correction was required on all cells on all OTAs prior to +2009-12-01, at which point a modification of the camera electronics +reduced the scale of the row-by-row offsets for the majority of the +OTAs. As a result, we only apply this correction to the cells where +it is still necessary, as shown in Figure \ref{fig: pattern row + cells}. A list of these cells is listed in Table +\ref{tab:pattern_row_cells}. + +Although this correction does largely resolve the row-by-row offset +issue in a satisfactory way, large and bright astronomical objects can +bias the fit significantly. This results in an oversubtraction of the +offset near these objects. As the offsets are calculated on the pixel +rows, this oversubtraction is not uniform around the object, but is +preferentially along the horizontal x axis of the object. Most +astronomical objects are not significantly distorted by this, with +this only becoming on issue for only bright objects comparable to the +size of the cell (598 pixels = 150"). + +%% \czwdraft{keep this?} This row-by-row offset is visible in similar +%% camera designs, and has been removed by identifying the noise signal +%% in the pixel data stream. By taking the FFT of the pixels and a +%% reference signal, the frequency of this noise can be isolated and +%% removed, resulting in a much cleaner image. However, GPC1 does not +%% record the value of the reference signal, instead automatically +%% subtracting it from the data values. Without this comparison signal, +%% we have been unable to reproduce this method, as there is no obvious +%% FFT component visible. + +\begin{deluxetable}{lcccc} + \tablecolumns{3} + \tablewidth{0pc} + \tablecaption{Cells which have PATTERN.ROW correction applied} + \tablehead{\colhead{OTA} & \colhead{Cell columns} & \colhead{Additional cells}} + \startdata + OTA11 & & xy02, xy03, xy04, xy07 \\ + OTA14 & & xy23 \\ + OTA15 & 0 & \\ + OTA27 & 0, 1, 2, 3, 7 & \\ + OTA31 & 7 & \\ + OTA32 & 3, 7 & \\ + OTA45 & 3, 7 & \\ + OTA47 & 0, 3, 5, 7 & \\ + OTA57 & 0, 1, 2, 6, 7 & \\ + OTA60 & & xy55 \\ + OTA74 & 2, 7 & \\ + \enddata + \label{tab:pattern_row_cells} +\end{deluxetable} + +\begin{figure} + \centering + \includegraphics[width=0.9\hsize,angle=0,clip]{images/pattern_row_edit.png} + \caption{Diagram illustrating in red which cells on GPC1 require the PATTERN.ROW correction to be applied. The footprint of each OTA is outlined, and cell xy00 is marked with either a filled box or an outline. The labeling of the non-existent corner OTAs is provided to orient the focal plane.} + \label{fig: pattern row cells} +\end{figure} + +\begin{figure} + \centering + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_nopat.png} +% \caption{(a)} +% \end{subfigure}% +% \begin{subfigure}[]{.45\hsize} + \end{minipage}% + \begin{minipage}{0.45\hsize} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_pat.png} +% \caption{(b)} +% \end{subfigure} + \end{minipage} + \caption{Example of the PATTERN.ROW correction on exposure o5379g0103o OTA57 cell xy00 (i-filter 45s). The left panel shows the cell with all appropriate detrending except the PATTERN.ROW, and the right shows the same cell with PATTERN.ROW applied. The correction reduces the correlated noise on the right side, which is most distant from the read out amplifier. There is a slight over subtraction along the rows near the bright star.} +\end{figure} + +\subsubsection{Pattern Continuity} + +As the PATTERN.CELL correction was insufficient in many situations, we +designed a replacement correction that would reduce the background +distortion for large objects. In addition, studies of the background +level illustrated that the row-by-row bias can introduce small +background gradient variations along the rows of the cells that is not +stable enough to be completely fit by the dark model. This common +feature across the columns of cells results in a ``saw tooth'' pattern +horizontally across an OTA, and as the background model fits a smooth +sky level, this induces over and under subtraction at the cell +boundaries. As the PATTERN.CELL was designed to correct changes only +in the median value between cells, it could not adequately resolve +this higher order issue. + +The replacement for PATTERN.CELL is the PATTERN.CONTINUITY correction, +which attempts to match the edges of a cell to those of its neighbors. +For each cell, a thin box 10 pixels wide on each edge is extracted and +the median value of unmasked values calculated for that box. These +median values are then used to construct a vector of differences +$\Delta_i = \sum_{j} Edge_{i} - Edge_{j}$, along with a matrix of +associations $A_{i,i'} = \sum_{j} \delta(i,j) \delta(j,i')$ denoting +which cell boundaries are adjacent. By solving the system $A x = +diff$, we find the set of offsets $x_i$ to be applied to each cell to +ensure the minimum differences between all cell edges and their +neighbors. + +For OTAs that initially show the saw tooth pattern, the effect of this +correction is to align the cells into a single ramp, at the expense of +the absolute background level. However, as we subtract off a smooth +background model prior to doing photometry, these deviations from an +absolute sky level are unimportant. The fact that the final ramp is +smoother than it would be otherwise also allows for the background +subtracted image to more closely match the astronomical sky, without +significant errors at cell boundaries. An example of the effect of +this correction on an image profile is shown in Figure \ref{fig:dark switching}. + +%% \begin{figure} +%% \centering +%% \caption{Continuity example, with background issue.} +%% \label{fig: continuity example} +%% \end{figure} + +\subsection{Fringe correction} +\label{sec:fringe} +% det_id 296 is the fringe we use. + +Due to variations in the thickness of the detectors, we observe +interference patterns at the infrared end of the filter set, as the +wavelength of the light becomes comparable to the thickness of the +detectors. Visually inspecting the images shows that the fringing is +most prevalent in the y filter images, with negligible fringing in the +other bands. As a result of this, we only apply a fringe correction +to the y filter data. + +The fringe used for PV3 processing was constructed from a set of 20 +120s science exposures. These exposures are overscan subtracted, and +corrected for non-linearity, and have the dark and flat models +applied. These images are smoothed with a Gaussian kernel with +$\sigma = 2$ pixels to minimize pixel to pixel noise. The fringe +image data is then constructed by calculating the clipped mean of the +input images with two iteration of clipping at the $3\sigma$ level. + +A course background model for each cell is constructed by calculating +the median on a 3x3 grid (approximately 200x200 pixels each). A set +of 1000 randomly selected points are then selected on the fringe image +for each cell, and a median calculated for this position in a 10x10 +pixel box, with the background level subtracted. These sample +locations provide scale points to allow the amplitude of the measured +fringe to be compared to that found on science images. + +To apply the fringe, the same sample locations are measured on the +science image to determine the relative strength of the fringing in +that particular image. A least squares fit between the fringe +measurements and the corresponding measurements on the science image +provides the scale factor multiplied to the fringe before it is +subtracted from the science image. + +\begin{figure} + \centering + \begin{minipage}{0.5\hsize} + \includegraphics[width=1.0\hsize,angle=0,clip]{images/o5220g0025o_XY53_nofringe.png} +% \caption{(a)} +% \end{subfigure}% +% \begin{subfigure}[]{.45\hsize} + \end{minipage}% + \begin{minipage}{0.5\hsize} + \includegraphics[width=1.0\hsize,angle=0,clip]{images/o5220g0025o_XY53_fringe.png} +% \caption{(b)} +% \end{subfigure} + \end{minipage} + \caption{Example of the y-filter fringe pattern on exposure o5220g0025o OTA53 (y-filter 30s). The left panel shows the OTA mosaic with all detrending except the fringe correction, while the right shows the same including the fringe correction. Both images have been smoothed with a Gaussian with $\sigma = 3$ pixels to highlight the faint and large scale fringe patterns. \czwdraft{See if there's a way to have mana produce images larger than the screen size.}} + \label{fig: fringe example} +\end{figure} + +\subsection{Masking} +\label{sec:masking} + +\subsubsection{Static Masks} +\label{sec:static_masks} + +Due to the large size of the detector, it is expected that there +are a number of pixel defects that do not have the detection +sensitivity on par with their neighbors. To remove these pixels, we +have constructed a static mask that identifies the known defects. +This mask is constructed in three phases. + +First, a CTEMASK is constructed to mask out regions in which the +charge transfer efficiency is low compared to the rest of the +detector. Twenty-five of the sixty OTAs in GPC1 show some evidence of +CTE issues, with this pattern appearing (to varying degrees) in +roughly triangular patches on the OTA due to defects in the +semiconductor manufacturing. To generate the mask for these regions, +a sample set of 26 evenly illuminated flat field images were measured +to produce a map of the image variance in 20x20 pixel bins. As the +flat image is expected to illuminate the image uniformly, the expected +variances in each bin should be Poissonian distributed with the flux +level. However, in regions with CTE issues, adjacent pixels are not +independent, as the charge in those pixels is more free to spread. +This reduces the pixel-to-pixel differences, resulting in a lower than +expected variance. All regions with variance less than half the +average image level are added to the static CTEMASK. + +The next step of mask construction is to examine the flat and dark +models, and exclude pixels that appear to be poorly corrected by these +models. The DARKMASK process looks for pixels that are more than +$8\sigma$ discrepant in $10\%$ of the 100 input dark frame images +after those images have had the dark model applied to them. These +pixels are assumed to be unstable with respect to the dark model, and +have the DARK bit set in the static mask, indicating that they are +unreliable in scientific observing. Similarly, the FLATMASK process +looks for pixels that are $3\sigma$ discrepant in the same fraction of +16 input flat field images after both the dark and flat models have +been applied. Those pixels that do not follow the flat field model of +the rest of image are assigned the FLAT mask bit in the static mask, +removing the pixels that cannot be corrected to a linear response. + +The final step of mask construction is to examine the detector for +bright columns and other static pixel issues. This is first done by +processing a set of 100 i filter science images in the same fashion as +for the DARKMASK. A median image is constructed from these inputs +along with the per-pixel variance. These images are used to identify +pixels that have unexpectedly low variation between all inputs, as +well as those that significantly deviate from the global median value. +Once this initial set of bad pixels is identified, a $3\times{}3$ +pixel triangular kernel is convolved with the initial set, and any +convolved pixel with value greater than 1 is assigned to the static +mask. This does an excellent job of removing the majority of the +problem pixels. A subsequent manual inspection allows human +interaction to identify other inconsistent pixels including the +vignetted regions around the edge of the detector. + +Figure \ref{fig:static mask} shows an example of the static mask for +the full GPC1 field of view. Table \ref{tab:mask_values} lists the +bit mask values used for the different sources of masking. + +\begin{figure} + \centering + \includegraphics[width=0.9\hsize,angle=0,clip]{images/gpc1_mask_indexed.png} + \label{fig:static mask} + + \caption{Image map of the GPC1 static mask. The CTE regions are clearly visible as roughly triangular patches covering the corners of some OTAs. Some entire cells are masked, including an entire column of cells on OTA14. Calcite cells remove large areas from OTA17 AND OTA76.} +\end{figure} + +\begin{deluxetable}{ccl} + \tablecolumns{3} + \tablewidth{0pc} + \tablecaption{GPC1 Mask Values} + \tablehead{\colhead{Mask Name} & \colhead{Mask Value} & \colhead{Description}} + \startdata + DETECTOR & 0x0001 & A detector defect is present. \\ + FLAT & 0x0002 & The flat field model does not calibrate the pixel reliably. \\ + DARK & 0x0004 & The dark model does not calibrate the pixel reliably. \\ + BLANK & 0x0008 & The pixel does not contain valid data. \\ + CTE & 0x0010 & The pixel has poor charge transfer efficiency. \\ + SAT & 0x0020 & The pixel is saturated. \\ + LOW & 0x0040 & The pixel has a lower value than expected. \\ + SUSPECT & 0x0080 & The pixel is suspected of being bad. \\ + BURNTOOL & 0x0080 & The pixel contain an burntool repaired streak. \\ + CR & 0x0100 & A cosmic ray is present. \\ + SPIKE & 0x0200 & A diffraction spike is present. \\ + GHOST & 0x0400 & An optical ghost is present. \\ + STREAK & 0x0800 & A streak is present. \\ + STARCORE & 0x1000 & A bright star core is present. \\ + CONV.BAD & 0x2000 & The pixel is bad after convolution with a bad pixel. \\ + CONV.POOR& 0x4000 & The pixel is poor after convolution with a bad pixel. \\ + MARK & 0x8000 & An internal flag for temporarily marking a pixel. \\ + \enddata + \label{tab:mask_values} +\end{deluxetable} + +\subsubsection{Dynamic masks} +\label{sec:dynamic_masks} + +In addition to the static mask that removes the constant detector +defects, we also generate a set of dynamic masks that change with the +astronomical features in the image. These masks are advisory in +nature, and do not completely exclude the pixel from further +processing consideration. The first of these dynamic masks is the +burntool advisory mask mentioned above. These pixels are included for +photometry, but are rejected more readily in the stacking and +difference image construction, as they are more likely to have small +deviations due to imperfections in the burntool correction. + +The remaining dynamic masks are not generated until the IPP +\ippstage{camera} stage, at which point all object photometry is +complete, and an astrometric solution is known for the exposure. This +added information provides the positions of bright sources based on +the reference catalog, including those that fall slightly out of the +detector field of view or within the inter chip gaps, where internal +photometry may not identify them. These bright sources are the origin +for many of the image artifacts that the dynamic mask identifies and +excludes. + +\subsubsubsection{Electronic crosstalk ghosts} +\label{sec:crosstalk} + +Due to electrical crosstalk between the flex cables connecting the +individual detector OTA devices, ghost objects can be created by the +presence of a bright source at a different position on the camera. +Table \ref{tab:crosstalk_rules} summarizes the list of known crosstalk +rules, with an estimate of the magnitude difference between the source +and ghost. For all of the rules, any cell $v$ within the specified +column of cells on any of the OTAs in the specified column of OTAs $Y$ +creates the ghost in the same $v$ and $Y$ in the target column of +cells and OTAs. In each of these cases, a source object with an +instrumental magnitude brighter than -14.47 creates a ghost object +many orders of magnitude fainter at the target location. The cell +(x,y) pixel coordinate is identical between source and ghost, as a +result of the transfer occurring as the devices are read. A circular +mask is added to the ghost location with radius $R = 3.44 \left(-14.47 +- m_{source, instrumental}\right)$ pixels. Any objects in the +photometric catalog found at the location of the ghost mask have the +GHOST mask bit set, marking the object as a likely ghost. The +majority of the crosstalk rules are bi-directional, with a source in +either position creating a ghost at the corresponding crosstalk target +position. The two faintest rules are uni-directional, due to +differences in the electronic path for the crosstalk. + +For the very brightest sources ($m_{instrumental} < -15$), there can +be crosstalk ghosts between all columns of cells during the readout. +These ``bleed'' ghosts were originally identified as ghosts of the +saturation bleeds appearing in the neighboring cells, and as such, the +masking for these objects puts a rectangular mask down from top to +bottom of cells in all columns that are in the same row of cells as +the bright source. The width of this box is a function of the source +magnitude, with $W = 5 * \left(-15 - m_{source, instrumental}\right)$ +pixels. + +\begin{deluxetable}{lllc} + \tablecolumns{4} + \tablewidth{0pc} + \tablecaption{GPC1 Crosstalk Rules} + \tablehead{\colhead{Type}&\colhead{Source OTA/Cell}&\colhead{Ghost OTA/Cell}&\colhead{$\Delta m$}} + \startdata + Inter-OTA & OTA2Y XY3v & OTA3Y XY3v & 6.16 \\ + & OTA3Y XY3v & OTA2Y XY3v & \\ + & OTA4Y XY3v & OTA5Y XY3v & \\ + & OTA5Y XY3v & OTA4Y XY3v & \\ + Intra-OTA & OTA2Y XY5v & OTA2Y XY6v & 7.07 \\ + & OTA2Y XY6v & OTA2Y XY5v & \\ + & OTA5Y XY5v & OTA5Y XY6v & \\ + & OTA5Y XY6v & OTA5Y XY5v & \\ + One-way & OTA2Y XY7v & OTA3Y XY2v & 7.34 \\ + & OTA5Y XY7v & OTA4Y XY2v & \\ + \enddata + \label{tab:crosstalk_rules} +\end{deluxetable} + +%% \begin{figure} +%% \centering +%% \caption{Figure of crosstalk ghost and bright star source. Plot of cut across ghost to illustrate the flat-top shape.} +%% \end{figure} + +\subsubsubsection{Optical ghosts} +\label{sec:optical_ghosts} +% http://arxiv.org/pdf/1207.2513v1.pdf + +Due to imperfections in the anti-reflective coating on the optical +surfaces of GPC1, bright sources can also result in large out of focus +objects, particularly in the g-filter data. These objects are the +result of light reflecting back off the surface of the detector, +reflecting again off the lower surfaces of the optics (particularly +the L1 corrector lens), and then back down onto the focal plane. Due +to the extra travel distance, the resulting source is out of focus and +elongated along the radial direction of the camera focal plane. These +optical ghosts can be modeled in the focal plane coordinates (L,M) +which has its origin at the center of the focal plane. In this +system, a bright object at location (L,M) on the focal plane creates a +reflection ghost on the opposite side of the optical axis at (-L,-M). +The exact location is fit as a third order polynomial in the focal +plane L and M directions (as listed in Table \ref{tab:ghost_centers}). +An elliptical annulus mask is constructed at the expected ghost +location, with the major and minor axes defined by linear functions of +the ghost distance from the optical axis, and oriented with the +ellipse major axis is along the radial direction (Table +\ref{tab:ghost_radii}). All stars brighter than a filter-dependent +threshold (listed in Table \ref{tab:ghost_magnitudes}) have such masks +constructed. + +\begin{deluxetable}{lcc} + \tablecolumns{3} + \tablewidth{0pc} + \tablecaption{Optical Ghost Center Transformations} + \tablehead{\colhead{Polynomial Term}&\colhead{L center}&\colhead{M center}} + \startdata + $x^0 y^0$ & -1.215661e+02 & 2.422174e+01 \\ + $x^1 y^0$ & 1.321875e-02 & 4.170486e-04 \\ + $x^2 y^0$ & -4.017026e-09 & -1.934260e-08 \\ + $x^3 y^0$ & 1.148288e-10 & -1.173657e-12 \\ + $x^0 y^1$ & -1.908074e-03 & 1.189352e-02 \\ + $x^1 y^1$ & 8.479150e-08 & -9.256748e-08 \\ + $x^2 y^1$ & 1.635732e-11 & 1.140772e-10 \\ + $x^0 y^2$ & 2.625405e-08 & 8.123932e-08 \\ + $x^1 y^2$ & 1.125586e-10 & 1.328378e-11 \\ + $x^0 y^3$ & 2.912432e-12 & 1.170865e-10 \\ + \enddata + \label{tab:ghost_centers} +\end{deluxetable} + +\begin{deluxetable}{lcccc} + \tablecolumns{5} + \tablewidth{0pc} + \tablecaption{Optical Ghost Annulus Axis Length} + \tablehead{\colhead{Radial Order}&\colhead{Inner Major Axis}&\colhead{Inner Minor Axis}& \colhead{Outer Major Axis}&\colhead{Outer Minor Axis}} + \startdata + $r^0$ & 3.926693e+01 & 5.287548e+01 & 7.928722e+01 & 1.314265e+02 \\ + $r^1$ & 5.325759e-03 &-2.191669e-03 & 1.722181e-02 & -2.627153e-03 \\ + \enddata + \label{tab:ghost_radii} +\end{deluxetable} + +\begin{deluxetable}{lc} + \tablecolumns{2} + \tablewidth{0pc} + \tablecaption{Optical Ghost Magnitude Limits} + \tablehead{\colhead{Filter}&\colhead{$m_{inst}$}} + \startdata + g & -16.5 \\ + r & -20.0 \\ + i & -25.0 \\ + z & -25.0 \\ + y & -25.0 \\ + w & -20.0 \\ + \enddata + \label{tab:ghost_magnitudes} +\end{deluxetable} + + +\begin{figure} + \centering + \includegraphics[width=0.9\hsize,angle=0,clip]{images/full_fpa_ghosts.jpg} + \caption{Example of the full GPC1 field of view illustrating the sources and destinations of optical ghosts on exposure o5677g0123o (2011-04-26, 43s g-filter). The bright stars on OTA33 and OTA44 result in nearly circular ghosts on the opposite OTA. In contrast, the trio of stars on OTA11 result in very elongated ghosts on OTA66.} +\end{figure} + +\subsubsubsection{Optical glints} +\label{sec:glints} + +Prior to \czwdraft{DATE}, a reflective surface at the edge of the +camera aperture was incompletely screened to light passing through the +telescope. Sources brighter than $m_{inst} = -21$ that fell on this +reflective surface resulted in light being scattered across the +detector surface in a long narrow glint. This surface was physically +masked on \czwdraft{DATE}, removing the possibility of glints in +subsequent data, but that taken prior have a dynamic mask constructed +when a reference source falls on the focal plane within one degree of +the detector edge. This mask is 150 pixels wide, with length $L = +2500 \left(-20 - m_{inst}\right)$ pixels. These glint masks are +constructed by selecting sufficiently bright sources in the reference +catalog that fall within rectangular regions around each edge of the +GPC1 camera. These regions are separated from the edge of the camera +by 17 arcminutes, and extend outwards an additional degree. + +%% +%% GLINT_MAX_MAG F32 -21.0 +%% GLINT.REGION MULTI + +%% GLINT.REGION METADATA +%% REGION STR [-38000:-24000,-20000:+20000] +%% GLINT.TYPE STR LEFT +%% END + +%% GLINT.REGION METADATA +%% REGION STR [+24000:+38000,-20000:+20000] +%% GLINT.TYPE STR RIGHT +%% END + +%% GLINT.REGION METADATA +%% REGION STR [-20000:+20000,+24000:+38000:] +%% GLINT.TYPE STR TOP +%% END + +%% GLINT.REGION METADATA +%% REGION STR [-20000:+20000,-38000:-24000] +%% GLINT.TYPE STR BOTTOM +%% END + +\begin{figure} + \centering + \includegraphics[width=0.9\hsize,angle=0,clip]{images/glint_example_o5379g0103o.jpg} + \caption{Example of a glint on exposure o5379g0103o (2010-07-02, 45s i-filter). The source star out of the field of view creates a long reflection that extends through OTA73 and OTA63.} +\end{figure} + +\subsubsubsection{Diffraction Spikes and Saturated Stars} +\label{sec:diffraction_spikes} + +Bright sources also form diffraction spikes that are dynamically +masked. These are filter independent, and are modeled as rectangles +with length $L = 10^{0.096 * (7.35 - m_{instrumental})} - 200$ and +width $W = 8 + (L - 200) * 0.01$, with negative values indicating no +mask is constructed, as the source is likely too faint to produce the +feature. These spikes are dependent on the camera rotation, and are +oriented at $\theta = n * \frac{\pi}{2} - \mathrm{ROTANGLE} + 0.798$, +based on the header keyword. + +%\subsubsection{Saturated stars} +%\label{sec:saturated_stars} + +The cores of stars that are saturated are masked as well, with a +circular mask radius $r = 10.15 * (-15 - m_{instrumental})$. An +example of a saturated star, with the masked regions for the +diffraction spikes and core saturation highlighted, is shown in Figure +\ref{fig:saturated star}. + +\begin{figure} + \centering + \includegraphics[width=0.9\hsize,angle=0,clip]{images/o6802g0338o_XY51_b1.jpg} + \caption{Example of saturated star, with diffraction spikes extending from the core on exposure o6802g0338o, OTA51 (2014-05-25, 45s g-filter).} + \label{fig:saturated star} +\end{figure} + +\subsubsection{Masking Fraction} +\label{sec:masking_fraction} + +For the full field of view that falls on the sixty OTAs, 14.7\% of all +pixels are masked. The large fraction of this masking is due to +regions that fall within the vignetted region. Defining the diameter +of the unvignetted region to be 3 degrees, and excluding pixels that +fall beyond this point reduces the static masking fraction to 9.7\%. + +Unfortunately, due to the design of the OTAs and readout cells, a +non-negligible fraction of the field of view falls onto an area that +does not have a detector pixel. For a given OTA mosaicked to a +$4846\times{}4868$ pixel image, the 64 $590\times{}598$ pixel readout +cells cover 95.7\% of the OTA area, providing an additional 4.3\% +masking in the unvignetted field of view due to the absence of a +detector pixel. + +For the inter-chip gap area loss, we use two field of view +calculations to estimate the masking fraction. The reference field of +view of GPC1 is 3 degrees, which at the nominal plate scale of 0.258 +arcseconds per pixel, translates to a 20930 FPA pixel radius. \czwdraft{I need a percentage here.} + +%% mysql> select filter,AVG(camProcessedExp.maskfrac_ref_static), AVG(camProcessedExp.maskfrac_ref_dynamic), AVG(camProcessedExp.maskfrac_ref_advisory), AVG(camProcessedExp.maskfrac_max_static),AVG(camProcessedExp.maskfrac_max_dynamic),AVG(camProcessedExp.maskfrac_max_advisory) from camRun join camProcessedExp USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.label = 'LAP.PV3.20140730.final' GROUP BY filter; +%% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+ +%% | filter | AVG(camProcessedExp.maskfrac_ref_static) | AVG(camProcessedExp.maskfrac_ref_dynamic) | AVG(camProcessedExp.maskfrac_ref_advisory) | AVG(camProcessedExp.maskfrac_max_static) | AVG(camProcessedExp.maskfrac_max_dynamic) | AVG(camProcessedExp.maskfrac_max_advisory) | +%% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+ +%% static dynamic advisory +%% | g.00000 | 0.19642137972007 | 0.00010322263512709 | 0.026838445469766 +%% | 0.20949461794863 | 9.89200027293e-05 | 0.026431927734548 | +%% | r.00000 | 0.19675996201399 | 0.00025214447869606 | 0.032641054600788 +%% | 0.20989768279138 | 0.00023994155711801 | 0.032178525485201 | +%% | i.00000 | 0.19677587604327 | 0.00057470697316504 | 0.038096251937072 +%% | 0.21003570722292 | 0.00053987093278142 | 0.037471018638997 | +%% | z.00000 | 0.1974290315691 | 0.00024758901226967 | 0.03064123748973 +%% | 0.21055007930696 | 0.00023452690039757 | 0.030144453360769 | +%% | y.00000 | 0.19828990634315 | 0.00014523787521897 | 0.021984846417987 +%% | 0.21130344126869 | 0.00013634812877977 | 0.02163070300815 | + +Summing mask fractions from these three contributions within the +unvignetted field of view results in an average of $\sim 20\%$ masking +fraction across the field of view. Dynamic masking adds an additional +$2-3\%$ on average, with advisory burntool masking contributing the +largest single component. + +\subsection{Background subtraction} +\label{sec:background} + +Once all other detrending is done, the pixels from each cell are +mosaicked into the full $4846\times{}4868$ pixel OTA image. A +background model for the full OTA is then determined prior to the +photometric analysis. The mosaicked image is binned into +$800\times{}800$ pixel bins, centered on the image center, and +overlapping by a factor of 2 in both axes. These bins have 10000 +random samples drawn, and a binned cumulative distribution function is +generated. These bins are interpolated to find the best mean value at +the $50\%$ level, as well as the distribution $\sigma$ by estimating +from the $32\%$ and $68\%$ levels. Repeating this across all bins +results in a $13\times{}13$ grid of background bins, which are +bilinearly interpolated to generate the background model to subtract. +Each object in the photometric catalog has a SKY and SKY\_SIGMA value +based on this model as well. + +%% * Magic +%% * Warping +%% * warping kernel +%% * linear-by-pieces +%% * Covariance +%% * def of skycells? +%% * Stacking +%% * pixel combination rules +%% * pixel rejections +%% * convolution for matching (success and failure) +%% * Difference Image analysis \section{GPC1 Detrend Construction} @@ -332,1168 +1464,4 @@ \end{deluxetable} -\section{GPC1 Detrend Details} -\label{sec:detrending} - -Ensuring a consistent and uniform detector response across the -three-degree diameter field of view of the GPC1 camera is essential to -a well calibrated survey. Many standard image detrending steps are -done for GPC1, with overscan subtraction removing the detector bias -level, dark frame subtraction to remove temperature and exposure time -dependent detector glows, and flat field correction to remove pixel to -pixel response functions. We also construct fringe correction for the -reddest data in the y filter, to remove the interference patterns that -arise in that filter due to the variations in the thickness of the -detector surface. - -These corrections, however, assume that the detector response is -linear across the full range of values. This is not universally the -case with GPC1, and this requires an additional set of detrending -steps to remove these non-linear responses. The first of these is the -\ippprog{burntool} correction, which removes the persistence trails -caused by the incomplete transfer of charge along the readout columns. -This bright-end nonlinearity is generally only evident for the -brightest stars, as only pixels that are at or beyond the saturation -point of the detector have this issue. More widespread is the -non-linearity at the faint end of the pixel range. Some readout cells -and some readout cell edge pixels experience a sag relative to linear -at low illumination, such that faint pixels appear fainter than -expected. The correction to this requires amplifying the pixel values -in these regions to match the expected model. - -The final non-linear response issue has no good option for correction. -Large regions of some OTA cells experience significant charge transfer -issues, making them unusable for science observations. These regions -are therefore masked in processing, with these CTE regions making up -the largest fraction of masked pixels on the detector. Other regions -are masked for other regions, such as static bad pixel features or -temporary readout masking caused by issues in the camera electronics -that make these regions unreliable. These all contribute to the -detector mask, which is augmented in each exposure for dynamic -features that are masked based on the astronomical features within the -field of view. - -For the PV3 processing, all detrending is done by the -\ippprog{ppImage} program. This program applies the detrends to the -individual cells, and then an OTA level mosaic is constructed for the -science image, the mask image, and the variance map image. The single -epoch photometry is done at this stage as well. The following -subsections (\ref{sec:burntool} - \ref{sec:background}) detail these -detrending steps, presented in the order in which they are applied to -the individual OTA image data. - -\subsection{Burntool / Persistence effect} -\label{sec:burntool} - -Pixels that approach the saturation point on GPC1, which varies by -readout with common values around 60000 DN, cause persistence problems -on that and subsequent images. During the read out process of an -image with such a bright pixel, some of the charge associated with it -is not fully shifted down the detector column toward the amplifier. -As a result, this charge remains in the starting cell, and is -partially collected in subsequent shifts, resulting in a ``burn -trail'' that extends from the center of the bright source away from -the amplifier (vertically along the pixel columns toward the top of -the cell). - -This incomplete charge shifting in nearly full wells continues as each -row is read out. This results in a remnant charge being deposited in -the pixels that the full well was shifted through. In following -exposures, this remnant charge leaks out, resulting in a trail that -extends from the initial location of the bright source on the previous -image towards the amplifier (vertically down along the pixel column). -This remnant charge can remain on the detector for up to thirty -minutes, requiring the locations of these ``burns'' be retained -between exposures. - -Both of these types of persistance trails are measured and optionally -repaired via the \ippprog{burntool} program. This program does an -initial scan of the images, and identifies objects with pixel values -brighter than a conservative threshold of 30000 DN. The trail from -the peak of that object is fit with a one-dimensional power law in -each pixel column above the threshold, based on empirical evidence -that this is the functional form of this persistence effect. This -also matches the expectation that a constant fraction of charge is -incompletely transfered at each shift beyond the persistence -threshold. Once this fit is done, the model can be subtracted from -the image, and the location of the star is stored in a table along -with the exposure PONTIME, which denotes the number of seconds since -the detector was last powered on, and provides an internally consistent -time scale. - -For subsequent exposures, the table associated with the previous image -is read in, and after correcting trails from the stars on the new -image, the positions of the bright stars from the table are used to -check for remnant trails on the image. These are fit and subtracted -using a one-dimensional exponential model, again based on empirical -studies. If a significant model is found, then this location is -retained in the image output table. If not, the old burn is allowed -to expire. - -The main concern with this method of correcting the persistance trails -is that it is based on fits to the raw image data, which may have -other signal sources not determined by the persistence effect. The -presence of other stars or artifacts along the path of the burn can -result in a poor model to be fit, resulting in either an over- or -under-subtraction of the persistance burn. For this reason, the image -mask is marked with a value indicating that this correction has been -applied. These pixels are not fully excluded, but they are marked as -suspect, which allows them to be excluded from consideration in -subsequent stages, such as image stacking. - -Another concern is that the cores of very bright stars are deformed by -this process, as the burntool fitting subtracts flux -from only one side of the star. As most stars that result in burns already -have saturated cores, they are already ignored for the purpose of -PSF determination and are flagged as saturated by the photometry -reduction. - -\begin{figure} - \centering - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt_trail.png} -% \caption{(a)} -% \end{subfigure}% -% \begin{subfigure}[]{.45\hsize} - \end{minipage}% - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt_trail.png} -% \caption{(b)} -% \end{subfigure} - \end{minipage} - - \caption{Example of a profile cut along the y-axis through a bright star on exposure o5677g0123o OTA11 in cell xy60 (left panel) and on the subsequent exposure o5677g0124o (right panel). In both figures, the green points show the image corrected with all appropriate detrending steps, but without burntool applied, illustrating the amplitude of the persistence trails. The red points show the same data after the burntool correction, which reduces the impact of these features. Both exposures are in the g-filter with exposure times of 43s} -\end{figure} - -\begin{figure} - \centering - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_nobt.png} -% \caption{(a)} -% \end{subfigure}% -% \begin{subfigure}[]{.45\hsize} - \end{minipage}% - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_nobt.png} -% \caption{(b)} -% \end{subfigure} - \end{minipage} - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_XY11_bt.png} -% \caption{(a)} -% \end{subfigure}% -% \begin{subfigure}[]{.45\hsize} - \end{minipage}% - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0124o_XY11_bt.png} -% \caption{(b)} -% \end{subfigure} - \end{minipage} - \caption{Example of OTA11 cell xy60 on exposures o5677g0123o (left) and o5677g0124o (right). The top panels show the image with all appropriate detrending steps, but without burntool, and the bottom show the same with burntool applied. There is some slight over subtraction in fitting the initial trail, but the impact of the trail is greatly reduced in both exposures.} -\end{figure} - -\subsection{Masking} -\label{sec:masking} - -\subsubsection{Static Masks} -\label{sec:static_masks} - -Due to the large size of the detector, it is expected that there -are a number of pixel defects that do not have the detection -sensitivity on par with their neighbors. To remove these pixels, we -have constructed a static mask that identifies the known defects. -This mask is constructed in three phases. - -First, a CTEMASK is constructed to mask out regions in which the -charge transfer efficiency is low compared to the rest of the -detector. Twenty-five of the sixty OTAs in GPC1 show some evidence of -CTE issues, with this pattern appearing (to varying degrees) in -roughly triangular patches on the OTA due to defects in the -semiconductor manufacturing. To generate the mask for these regions, -a sample set of 26 evenly illuminated flat field images were measured -to produce a map of the image variance in 20x20 pixel bins. As the -flat image is expected to illuminate the image uniformly, the expected -variances in each bin should be Poissonian distributed with the flux -level. However, in regions with CTE issues, adjacent pixels are not -independent, as the charge in those pixels is more free to spread. -This reduces the pixel-to-pixel differences, resulting in a lower than -expected variance. All regions with variance less than half the -average image level are added to the static CTEMASK. - -The next step of mask construction is to examine the flat and dark -models, and exclude pixels that appear to be poorly corrected by these -models. The DARKMASK process looks for pixels that are more than -$8\sigma$ discrepant in $10\%$ of the 100 input dark frame images -after those images have had the dark model applied to them. These -pixels are assumed to be unstable with respect to the dark model, and -have the DARK bit set in the static mask, indicating that they are -unreliable in scientific observing. Similarly, the FLATMASK process -looks for pixels that are $3\sigma$ discrepant in the same fraction of -16 input flat field images after both the dark and flat models have -been applied. Those pixels that do not follow the flat field model of -the rest of image are assigned the FLAT mask bit in the static mask, -removing the pixels that cannot be corrected to a linear response. - -The final step of mask construction is to examine the detector for -bright columns and other static pixel issues. This is first done by -processing a set of 100 i filter science images in the same fashion as -for the DARKMASK. A median image is constructed from these inputs -along with the per-pixel variance. These images are used to identify -pixels that have unexpectedly low variation between all inputs, as -well as those that significantly deviate from the global median value. -Once this initial set of bad pixels is identified, a $3\times{}3$ -pixel triangular kernel is convolved with the initial set, and any -convolved pixel with value greater than 1 is assigned to the static -mask. This does an excellent job of removing the majority of the -problem pixels. A subsequent manual inspection allows human -interaction to identify other inconsistent pixels including the -vignetted regions around the edge of the detector. - -Figure \ref{fig:static mask} shows an example of the static mask for -the full GPC1 field of view. Table \ref{tab:mask_values} lists the -bit mask values used for the different sources of masking. - -\begin{figure} - \centering - \includegraphics[width=0.9\hsize,angle=0,clip]{images/gpc1_mask_indexed.png} - \label{fig:static mask} - - \caption{Image map of the GPC1 static mask. The CTE regions are clearly visible as roughly triangular patches covering the corners of some OTAs. Some entire cells are masked, including an entire column of cells on OTA14. Calcite cells remove large areas from OTA17 AND OTA76.} -\end{figure} - -\begin{deluxetable}{ccl} - \tablecolumns{3} - \tablewidth{0pc} - \tablecaption{GPC1 Mask Values} - \tablehead{\colhead{Mask Name} & \colhead{Mask Value} & \colhead{Description}} - \startdata - DETECTOR & 0x0001 & A detector defect is present. \\ - FLAT & 0x0002 & The flat field model does not calibrate the pixel reliably. \\ - DARK & 0x0004 & The dark model does not calibrate the pixel reliably. \\ - BLANK & 0x0008 & The pixel does not contain valid data. \\ - CTE & 0x0010 & The pixel has poor charge transfer efficiency. \\ - SAT & 0x0020 & The pixel is saturated. \\ - LOW & 0x0040 & The pixel has a lower value than expected. \\ - SUSPECT & 0x0080 & The pixel is suspected of being bad. \\ - BURNTOOL & 0x0080 & The pixel contain an burntool repaired streak. \\ - CR & 0x0100 & A cosmic ray is present. \\ - SPIKE & 0x0200 & A diffraction spike is present. \\ - GHOST & 0x0400 & An optical ghost is present. \\ - STREAK & 0x0800 & A streak is present. \\ - STARCORE & 0x1000 & A bright star core is present. \\ - CONV.BAD & 0x2000 & The pixel is bad after convolution with a bad pixel. \\ - CONV.POOR& 0x4000 & The pixel is poor after convolution with a bad pixel. \\ - MARK & 0x8000 & An internal flag for temporarily marking a pixel. \\ - \enddata - \label{tab:mask_values} -\end{deluxetable} - -\subsubsection{Dynamic masks} -\label{sec:dynamic_masks} - -In addition to the static mask that removes the constant detector -defects, we also generate a set of dynamic masks that change with the -astronomical features in the image. These masks are advisory in -nature, and do not completely exclude the pixel from further -processing consideration. The first of these dynamic masks is the -burntool advisory mask mentioned above. These pixels are included for -photometry, but are rejected more readily in the stacking and -difference image construction, as they are more likely to have small -deviations due to imperfections in the burntool correction. - -The remaining dynamic masks are not generated until the IPP -\ippstage{camera} stage, at which point all object photometry is -complete, and an astrometric solution is known for the exposure. This -added information provides the positions of bright sources based on -the reference catalog, including those that fall slightly out of the -detector field of view or within the inter chip gaps, where internal -photometry may not identify them. These bright sources are the origin -for many of the image artifacts that the dynamic mask identifies and -excludes. - -\subsubsection{Electronic crosstalk ghosts} -\label{sec:crosstalk} - -Due to electrical crosstalk between the flex cables connecting the -individual detector OTA devices, ghost objects can be created by the -presence of a bright source at a different position on the camera. -Table \ref{tab:crosstalk_rules} summarizes the list of known crosstalk -rules, with an estimate of the magnitude difference between the source -and ghost. For all of the rules, any cell $v$ within the specified -column of cells on any of the OTAs in the specified column of OTAs $Y$ -creates the ghost in the same $v$ and $Y$ in the target column of -cells and OTAs. In each of these cases, a source object with an -instrumental magnitude brighter than -14.47 creates a ghost object -many orders of magnitude fainter at the target location. The cell -(x,y) pixel coordinate is identical between source and ghost, as a -result of the transfer occurring as the devices are read. A circular -mask is added to the ghost location with radius $R = 3.44 \left(-14.47 -- m_{source, instrumental}\right)$ pixels. Any objects in the -photometric catalog found at the location of the ghost mask have the -GHOST mask bit set, marking the object as a likely ghost. The -majority of the crosstalk rules are bi-directional, with a source in -either position creating a ghost at the corresponding crosstalk target -position. The two faintest rules are uni-directional, due to -differences in the electronic path for the crosstalk. - -For the very brightest sources ($m_{instrumental} < -15$), there can -be crosstalk ghosts between all columns of cells during the readout. -These ``bleed'' ghosts were originally identified as ghosts of the -saturation bleeds appearing in the neighboring cells, and as such, the -masking for these objects puts a rectangular mask down from top to -bottom of cells in all columns that are in the same row of cells as -the bright source. The width of this box is a function of the source -magnitude, with $W = 5 * \left(-15 - m_{source, instrumental}\right)$ -pixels. - -\begin{deluxetable}{lllc} - \tablecolumns{4} - \tablewidth{0pc} - \tablecaption{GPC1 Crosstalk Rules} - \tablehead{\colhead{Type}&\colhead{Source OTA/Cell}&\colhead{Ghost OTA/Cell}&\colhead{$\Delta m$}} - \startdata - Inter-OTA & OTA2Y XY3v & OTA3Y XY3v & 6.16 \\ - & OTA3Y XY3v & OTA2Y XY3v & \\ - & OTA4Y XY3v & OTA5Y XY3v & \\ - & OTA5Y XY3v & OTA4Y XY3v & \\ - Intra-OTA & OTA2Y XY5v & OTA2Y XY6v & 7.07 \\ - & OTA2Y XY6v & OTA2Y XY5v & \\ - & OTA5Y XY5v & OTA5Y XY6v & \\ - & OTA5Y XY6v & OTA5Y XY5v & \\ - One-way & OTA2Y XY7v & OTA3Y XY2v & 7.34 \\ - & OTA5Y XY7v & OTA4Y XY2v & \\ - \enddata - \label{tab:crosstalk_rules} -\end{deluxetable} - -%% \begin{figure} -%% \centering -%% \caption{Figure of crosstalk ghost and bright star source. Plot of cut across ghost to illustrate the flat-top shape.} -%% \end{figure} - -\subsubsection{Optical ghosts} -\label{sec:optical_ghosts} -% http://arxiv.org/pdf/1207.2513v1.pdf - -Due to imperfections in the anti-reflective coating on the optical -surfaces of GPC1, bright sources can also result in large out of focus -objects, particularly in the g-filter data. These objects are the -result of light reflecting back off the surface of the detector, -reflecting again off the lower surfaces of the optics (particularly -the L1 corrector lens), and then back down onto the focal plane. Due -to the extra travel distance, the resulting source is out of focus and -elongated along the radial direction of the camera focal plane. These -optical ghosts can be modeled in the focal plane coordinates (L,M) -which has its origin at the center of the focal plane. In this -system, a bright object at location (L,M) on the focal plane creates a -reflection ghost on the opposite side of the optical axis at (-L,-M). -The exact location is fit as a third order polynomial in the focal -plane L and M directions (as listed in Table \ref{tab:ghost_centers}). -An elliptical annulus mask is constructed at the expected ghost -location, with the major and minor axes defined by linear functions of -the ghost distance from the optical axis, and oriented with the -ellipse major axis is along the radial direction (Table -\ref{tab:ghost_radii}). All stars brighter than a filter-dependent -threshold (listed in Table \ref{tab:ghost_magnitudes}) have such masks -constructed. - -\begin{deluxetable}{lcc} - \tablecolumns{3} - \tablewidth{0pc} - \tablecaption{Optical Ghost Center Transformations} - \tablehead{\colhead{Polynomial Term}&\colhead{L center}&\colhead{M center}} - \startdata - $x^0 y^0$ & -1.215661e+02 & 2.422174e+01 \\ - $x^1 y^0$ & 1.321875e-02 & 4.170486e-04 \\ - $x^2 y^0$ & -4.017026e-09 & -1.934260e-08 \\ - $x^3 y^0$ & 1.148288e-10 & -1.173657e-12 \\ - $x^0 y^1$ & -1.908074e-03 & 1.189352e-02 \\ - $x^1 y^1$ & 8.479150e-08 & -9.256748e-08 \\ - $x^2 y^1$ & 1.635732e-11 & 1.140772e-10 \\ - $x^0 y^2$ & 2.625405e-08 & 8.123932e-08 \\ - $x^1 y^2$ & 1.125586e-10 & 1.328378e-11 \\ - $x^0 y^3$ & 2.912432e-12 & 1.170865e-10 \\ - \enddata - \label{tab:ghost_centers} -\end{deluxetable} - -\begin{deluxetable}{lcccc} - \tablecolumns{5} - \tablewidth{0pc} - \tablecaption{Optical Ghost Annulus Axis Length} - \tablehead{\colhead{Radial Order}&\colhead{Inner Major Axis}&\colhead{Inner Minor Axis}& \colhead{Outer Major Axis}&\colhead{Outer Minor Axis}} - \startdata - $r^0$ & 3.926693e+01 & 5.287548e+01 & 7.928722e+01 & 1.314265e+02 \\ - $r^1$ & 5.325759e-03 &-2.191669e-03 & 1.722181e-02 & -2.627153e-03 \\ - \enddata - \label{tab:ghost_radii} -\end{deluxetable} - -\begin{deluxetable}{lc} - \tablecolumns{2} - \tablewidth{0pc} - \tablecaption{Optical Ghost Magnitude Limits} - \tablehead{\colhead{Filter}&\colhead{$m_{inst}$}} - \startdata - g & -16.5 \\ - r & -20.0 \\ - i & -25.0 \\ - z & -25.0 \\ - y & -25.0 \\ - w & -20.0 \\ - \enddata - \label{tab:ghost_magnitudes} -\end{deluxetable} - - -\begin{figure} - \centering - \includegraphics[width=0.9\hsize,angle=0,clip]{images/full_fpa_ghosts.jpg} - \caption{Example of the full GPC1 field of view illustrating the sources and destinations of optical ghosts on exposure o5677g0123o (2011-04-26, 43s g-filter). The bright stars on OTA33 and OTA44 result in nearly circular ghosts on the opposite OTA. In contrast, the trio of stars on OTA11 result in very elongated ghosts on OTA66.} -\end{figure} - -\subsubsection{Optical glints} -\label{sec:glints} -Prior to \czwdraft{DATE}, a reflective surface at the edge of the -camera aperture was incompletely screened to light passing through the -telescope. Sources brighter than $m = -20$ that fell on this -reflective surface resulted in light being scattered across the -detector surface in a long narrow glint. This surface was physically -masked on \czwdraft{DATE}, removing the possibility of glints in -subsequent data, but that taken prior have a dynamic mask constructed -when a reference source falls on the focal plane within one degree of -the detector edge. This mask is 150 pixels wide, with length $L = -2500 \left(-20 - m_{inst}\right)$ pixels. \czwdraft{Am I correct that - this is basically a one-degree edge around the detector?} - -%% -%% GLINT_MAX_MAG F32 -21.0 -%% GLINT.REGION MULTI - -%% GLINT.REGION METADATA -%% REGION STR [-38000:-24000,-20000:+20000] -%% GLINT.TYPE STR LEFT -%% END - -%% GLINT.REGION METADATA -%% REGION STR [+24000:+38000,-20000:+20000] -%% GLINT.TYPE STR RIGHT -%% END - -%% GLINT.REGION METADATA -%% REGION STR [-20000:+20000,+24000:+38000:] -%% GLINT.TYPE STR TOP -%% END - -%% GLINT.REGION METADATA -%% REGION STR [-20000:+20000,-38000:-24000] -%% GLINT.TYPE STR BOTTOM -%% END - -\begin{figure} - \centering - \includegraphics[width=0.9\hsize,angle=0,clip]{images/glint_example_o5379g0103o.jpg} - \caption{Example of a glint on exposure o5379g0103o (2010-07-02, 45s i-filter). The source star out of the field of view creates a long reflection that extends through OTA73 and OTA63.} -\end{figure} - -\subsubsection{Diffraction Spikes and Saturated Stars} -\label{sec:diffraction_spikes} - -Bright sources also form diffraction spikes that are dynamically -masked. These are filter independent, and are modeled as rectangles -with length $L = 10^{0.096 * (7.35 - m_{instrumental})} - 200$ and -width $W = 8 + (L - 200) * 0.01$, with negative values indicating no -mask is constructed, as the source is likely too faint to produce the -feature. These spikes are dependent on the camera rotation, and are -oriented at $\theta = n * \frac{\pi}{2} - \mathrm{ROTANGLE} + 0.798$, -based on the header keyword. - -%\subsubsection{Saturated stars} -%\label{sec:saturated_stars} - -The cores of stars that are saturated are masked as well, with a -circular mask radius $r = 10.15 * (-15 - m_{instrumental})$. An -example of a saturated star, with the masked regions for the -diffraction spikes and core saturation highlighted, is shown in Figure -\ref{fig:saturated star}. - -\begin{figure} - \centering - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o6802g0338o_XY51_b1.jpg} - \caption{Example of saturated star, with diffraction spikes extending from the core on exposure o6802g0338o, OTA51 (2014-05-25, 45s g-filter).} - \label{fig:saturated star} -\end{figure} - -\subsubsection{Video Mask} -\label{sec:video_masks} - -One aspect of the OTAs on GPC1 is that an individual cell can be read -repeatedly while the other cells integrate, resulting in a video -signal from that cell. This data is used for telescope guiding -purposes, and a single exposure is likely to have a number of these -video cells active on different OTAs. For the 3PI survey, the median -exposure has 14 video cells being read, although this number ranges -from less than five to more than thirty, depending on the stellar -density and field pointing. Reading these cells while integrating on -the others changes the characteristic dark model (see Section -\ref{sec:video_darks} below) experienced by the other cells on the -OTA. The observed effect of this is that the glow associated with the -amplifiers in the corners of the cells is suppressed during the video -readout, relative to the nominal glow. The standard dark model -oversubtracts this glow, resulting in dark regions in the corners of -the cells on an OTA taking video data. Before the nature of this -issue was fully understood, these poorly constrained corners were -masked with 25-pixel radius quarter circles, centered on the (1,1) -pixel nearest the cell amplifier. The other corners of the cell were -masked with a 15-pixel radius quarter circle, as the amplifier -creating the glow is associated with another cell and separated by the -inter-cell spacing, diminishing the area effected. Due to the large -area that this masking would cover, the PV3 processing used a more -robust video dark model to correct this problem, as described in -section \ref{sec:video_darks} below. - -\subsubsection{Masking Fraction} -\label{sec:masking_fraction} - -For the full field of view that falls on the sixty OTAs, 14.7\% of all -pixels are masked. The large fraction of this masking is due to -regions that fall within the vignetted region. Defining the diameter -of the unvignetted region to be 3 degrees, and excluding pixels that -fall beyond this point reduces the static masking fraction to 9.7\%. - -Unfortunately, due to the design of the OTAs and readout cells, a -non-negligible fraction of the field of view falls onto an area that -does not have a detector pixel. For a given OTA mosaicked to a -$4846\times{}4868$ pixel image, the 64 $590\times{}598$ pixel readout -cells cover 95.7\% of the OTA area, providing an additional 4.3\% -masking in the unvignetted field of view due to the absence of a -detector pixel. - -For the inter-chip gap area loss, we use two field of view -calculations to estimate the masking fraction. The reference field of -view of GPC1 is 3 degrees, which at the nominal plate scale of 0.258 -arcseconds per pixel, translates to a 20930 FPA pixel radius. \czwdraft{I need a percentage here.} - -%% mysql> select filter,AVG(camProcessedExp.maskfrac_ref_static), AVG(camProcessedExp.maskfrac_ref_dynamic), AVG(camProcessedExp.maskfrac_ref_advisory), AVG(camProcessedExp.maskfrac_max_static),AVG(camProcessedExp.maskfrac_max_dynamic),AVG(camProcessedExp.maskfrac_max_advisory) from camRun join camProcessedExp USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.label = 'LAP.PV3.20140730.final' GROUP BY filter; -%% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+ -%% | filter | AVG(camProcessedExp.maskfrac_ref_static) | AVG(camProcessedExp.maskfrac_ref_dynamic) | AVG(camProcessedExp.maskfrac_ref_advisory) | AVG(camProcessedExp.maskfrac_max_static) | AVG(camProcessedExp.maskfrac_max_dynamic) | AVG(camProcessedExp.maskfrac_max_advisory) | -%% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+ -%% static dynamic advisory -%% | g.00000 | 0.19642137972007 | 0.00010322263512709 | 0.026838445469766 -%% | 0.20949461794863 | 9.89200027293e-05 | 0.026431927734548 | -%% | r.00000 | 0.19675996201399 | 0.00025214447869606 | 0.032641054600788 -%% | 0.20989768279138 | 0.00023994155711801 | 0.032178525485201 | -%% | i.00000 | 0.19677587604327 | 0.00057470697316504 | 0.038096251937072 -%% | 0.21003570722292 | 0.00053987093278142 | 0.037471018638997 | -%% | z.00000 | 0.1974290315691 | 0.00024758901226967 | 0.03064123748973 -%% | 0.21055007930696 | 0.00023452690039757 | 0.030144453360769 | -%% | y.00000 | 0.19828990634315 | 0.00014523787521897 | 0.021984846417987 -%% | 0.21130344126869 | 0.00013634812877977 | 0.02163070300815 | - -Summing mask fractions from these three contributions within the -unvignetted field of view results in an average of $\sim 20\%$ masking -fraction across the field of view. Dynamic masking adds an additional -$2-3\%$ on average, with advisory burntool masking contributing the -largest single component. - -\subsection{Overscan} -\label{sec:overscan} - -Each cell on GPC1 has an overscan region that covers the first 34 -columns of each row, and the last 10 rows of each column. No light -lands on these pixels, so the image region is trimmed to exclude them. -Each row has an overscan value subtracted, calculated by finding the -median value of that row's overscan pixels and then smoothing between -rows with a three-row boxcar median. - -\subsection{Non-linearity Correction} -\label{sec:nonlinearity} -% check notebook, 2010-07/08 - -The pixels of GPC1 are not uniformly linear at all flux levels. In -particular, at low flux levels, some pixels have a tendency to sag -relative to the expected linear value. This effect is most pronounced -along the edges of the detector cells, although some entire cells show -evidence of this effect. - -To correct this sag, we studied the flux behavior of a series of flat -frames for a ramp of exposure times with approximate logarithmically -equal spacing between 0.01s and 57.04s. As the exposure time -increases, the flux on each pixel also increases in what is expected -to be a linear manner. Each of these flat exposures in this ramp is -overscan corrected, and then the median is calculated for each cell, -as well as for the rows and columns within ten pixels of the edge of -the science region. From these median values at each exposure time -value, we can construct the expected trend by fitting a linear model, -$f_{region} = G * t_{exp} + B$, to determine the gain, $G$, and the -bias, $B$, for the region considered. This fitting was limited to only -the range of fluxes between 12000 and 38000 counts, as these ranges -were found to match the linear model well. This range avoids the -non-linearity at low fluxes, as well as the possibility of high-flux -non-linearity effects. - -We store the average flux measurement and deviation from the linear -fit for each exposure time for all regions on all detector cells in -the linearity detrend look up tables. When this is applied to science -data, these lookup tables are loaded, and a linear interpolation is -performed to determine the correction needed for the flux in that -pixel. This look up is performed for both the row and column of each -pixel, to allow the edge correction to be applied where applicable, -and the full cell correction elsewhere. The average of these two -values is then applied to the pixel value, reducing the effects of -pixel nonlinearity. - -This non-linearity effect appears to be stable in time for the -majority of the detector pixels, with little evident change over the -survey duration. However, as the non-linearity is most pronounced at -the edges of the detector cells, those are the regions where the -correction is most likely to be incomplete. Because of this fact, -most pixels in the static mask with either the DARKMASK or FLATMASK -bit set are found along these edges. As the non-linearity correction -is unable to reliably restore these pixels, they produce inconsistent -values after the dark and flat have been applied, and are therefore -rejected. - -%% exptime n_included/det_id = 372 -%% clearly this isn't the one used, as 3-12 spans three data points, poorly.x -%% 0.01 2 -%% 0.14 2 -%% 0.27 2 -%% 0.49 2 -%% 0.72 2 -%% 1.06 2 -%% 1.41 2 -%% 2.02 2 -%% 2.63 2 -%% 3.94 2 -%% 5.25 2 -%% 8.74 2 -%% 13.09 2 -%% 17.4 2 -%% 20.86 2 -%% 24.3 2 -%% 27.78 2 -%% 31.24 2 -%% 34.65 2 -%% 38.12 2 -%% 42.41 2 -%% 46.69 2 -%% 51.89 2 -%% 57.04 2 - - -%http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearity_AllEdges -%http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearityArchive - -\begin{figure} - \centering - \includegraphics[width=0.9\hsize,angle=0,clip]{images/linearity_XY27_xy16.png} - \caption{Example plot of the linearity correction as a fraction of observed flux for OTA27, cell xy16.} -\end{figure} - -\subsection{Dark/Bias Subtraction} -\label{sec:dark} -% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Dark_Model - -The dark model we make for GPC1 considers each pixel individually, -independent of any neighbors. To construct this model, we fit a -multi-dimensional model to the array of input pixels from a randomly -selected set of 100-150 overscan and non-linearity corrected dark -frames chosen from a given date range. The model fits each pixel as a -function of the exposure time $t_{exp}$ and the detector temperature -$T_{chip}$ of the input images such that $\mathrm{dark} = a_0 + a_1 -t_{exp} + a_2 T_{chip} t_{exp} + a_3 T_{chip}^2 t_{exp}$. This -fitting uses two iterations to produce a clipped fit, rejecting at the -$3\sigma$ level. The final coefficients $a_i$ for the dark model are -stored in the detrend image. The constant $a_0$ term includes the -residual bias signal after overscan subtraction, and as such, a -separate bias subtraction is not necessary. - -Applying the dark model is simply a matter of calculating the response -to the exposure time and detector temperature for the image to be -corrected, and subtracting the resulting dark signal from the image. - -\subsubsection{Time evolution} - -The dark model is not consistently stable over the full survey, with -significant drift over the course of multiple months. Some of the -changes in the dark can be attributed to changes in the voltage -settings of the GPC1 controller electronics, but the majority seem to -be the result of some unknown parameter. We can separate the dark -model history of GPC1 into three epochs. The first epoch covers all -data taken prior to 2010-01-23. This epoch used a different header -keyword for the detector temperature, making data from this epoch -incompatible with later dark models. - -The second epoch covers data between 2010-01-23 and 2011-05-01, and is -characterized by a largely stable but oscillatory dark solution. -There are two modes that the dark model switches between apparently at -random. No clear cause has been established for the switching, but -there are clear differences between the two modes that require the -observation dates to be split to use the model that is most -appropriate. - -The initial evidence of these two modes comes from the discovery of a -slight gradient along the rows of certain cells. This is a result of -a drift in the bias level of the detector as it is read out. An -appropriate dark model should remove this gradient entirely. For -these two modes, the direction of this bias drift is different, so a -single dark model generated from all dark images in the time range -over corrects the positive-gradient mode, and under corrects the -negative-gradient mode. Upon identifying this two-mode behavior, and -determining the dates each mode was dominant, two separate dark -models were constructed from appropriate ``A'' and ``B'' mode dark -frames. Using the appropriate dark minimizes the effect of this bias -gradient in the dark corrected data. - -The bias drift gradients of the mode switching can be visualized in -Figure \ref{fig:dark switching}. This figure shows image profile -along the x-pixel axis binned along the full y-axis of dark corrected -images for OTA67. These images are from sequential days, and have -been corrected with a dark model constructed from the full set of dark -data within the second epoch. The opposite sign of the slopes of -these profiles indicates that the average dark model does not correct -these dates sufficiently, due to the contradictory dark signals -between the two modes. \czwdraft{this paragraph dependent on that figure. This doesn't quite match.} - -After 2011-05-01, the two-mode behavior of the dark disappears, and is -replaced with a slow observation date dependent drift in the magnitude -of the gradient. This drift is sufficiently slow that we have modeled -it using three observation date independent dark model for different -date ranges. These darks cover the range from 2011-05-01 to -2011-08-01, 2011-08-01 to 2011-11-01, and 2011-11-01 and on. The -reason for this time evolution is unknown, but as it is correctable -with a small number of dark models, this does not significantly impact -detrending. - -\begin{figure} - \centering -% \begin{subfigure}[]{.45\hsize} - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_M_OS_NL_XY23_b1.jpg} -% \caption{(a)} -% \end{subfigure}% -% \begin{subfigure}[]{.45\hsize} - \end{minipage}% - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_to_DARK_XY23_b1.jpg} -% \caption{(b)} -% \end{subfigure} - \end{minipage} - \caption{An example of the dark model application to exposure o5677g0123o, OTA23 (2011-04-26, 43s g-filter). The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, and the detector non-linearity corrected. The right panel, shows the same exposure with the dark applied in addition to the processing shown on the left.} -\end{figure} - -\begin{figure} - \centering - \includegraphics[width=0.9\hsize,angle=0,clip]{images/B_profile_ex.png} - \caption{Example showing a profile cut across exposure o5676g0195, OTA67 (2011-04-25, 43s g-filter). The entire first row of cells (xy00-xy07) have had a median calculated along each pixel column on the OTA mosaicked image. Arbitrary offsets have been applied to shift the curves to not overlap. The top curve (in purple) shows the initial raw profile, without no dark model applied. The next curve (in green) shows the smoother profile after applying the correct B-mode dark model. Applying the incorrect A-mode dark results in the blue curve, which shows a significant increase in gradients across the cells. The orange curve shows the result of the PATTERN.CONTINUITY correction. Although this creates a larger gradient across the mosaicked images, it decreases the cell-to-cell level changes. The final yellow curve shows the final image profile after all detrending and background subtraction, and has not had an offset applied. The bright source at the cell xy00 to xy01 transition is a result of a large optical ghost, which due to the area covered, increases the median level more than the field stars.} - \label{fig:dark switching} -\end{figure} - -\subsubsection{Video Dark} -\label{sec:video_darks} - -The dark signal is stronger in cell corners due to glow from the -read-out amplifiers. The standard dark model corrects this for most -observations. However, as mentioned above, when a cell is repeatedly -read in video mode, the dark model for the OTA containing it changes. -Surprisingly, added reads for the video cell do not amplify the -amplifier glow, but rather decrease the dark signal in these regions. -As a result, using the standard dark model on the data for these OTAs -results in oversubtraction of the corner glow. - -Video darks have been constructed to eliminate the effect this -observational change has on the final image quality. This was done by -running the standard dark construction process on a series of dark -frames that have had the video signal enabled for some cells. GPC1 -can only run video signals on a subset of the OTAs at a given time. -This requires two passes to enable the video signal across the full -set of OTAs that support video cells. This is convenient for the -process of creating darks, as those OTAs that do not have video -signals enabled create standard dark models, while the video dark is -created for those that do. - -This simultaneous construction of video and standard dark models is -useful, as it provides the ability to isolate the response on the -standard dark from the video signals. Isolating this response is -essential for attempting to create archival video darks. We only have -raw video dark frame data after 2012-05-16, when this problem was -initially identified, so any data prior to that can not be directly -corrected for the video dark signal. Isolating the video signal -response allows linear corrections to the pre-existing standard dark -models for archival data. Testing this shows that constructing a -video dark for older data simply as $VD_{2009} = D_{2009} - D_{Modern} -+ VD_{Modern}$ produces a satisfactory result that does not -oversubtract the amplifier glow. This is shown in figure -\ref{fig:video_darks}, which shows video cells from before 2012-05-16, -corrected with both the standard and video darks, with the early video -dark constructed in such a manner. - -\begin{figure} - \centering -% \begin{subfigure}[]{.45\hsize} - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_Rdark_XY22_b1.jpg} -% \caption{(a)} -% \end{subfigure}% -% \begin{subfigure}[]{.45\hsize} - \end{minipage}% - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5677g0123o_VIDEODARK_VDim_VDdark_XY22_b1.jpg} -% \caption{(b)} -% \end{subfigure} - \end{minipage} - \caption{An example of the video dark model application to exposure o5677g0123o, OTA22 (2011-04-26, 43s g-filter), which has a video cell located in cell xy16. The left panel shows the image data mosaicked to the OTA level, and has had the static mask applied, the overscan subtracted, the detector non-linearity corrected, and a regular dark applied. The right panel, shows the same exposure with a video dark applied instead of the standard dark. The main impact of this change is the improved correction of the corner glows, which are oversubtracted with the standard dark.} - \label{fig:video_darks} -\end{figure} - -\subsection{Noisemap} -\label{sec:noisemap} - -Based on a study of the positional dependence of all detected sources, -we have discovered that the cells in GPC1 do not have uniform noise -characteristics. Instead, there is a gradient along the pixel rows, -with the noise generally higher away from the read out amplifier -(higher cell x pixel positions). This is likely an effect of the -row-by-row bias issue discussed below. This gradient causes the read -noise to increase as the row is read out. As a result of this -increased noise, more sources are detected in the higher noise regions -when the read noise is assumed constant across the readout. To -mitigate this noise gradient, we constructed an initial set of -noisemap images by measuring the median variance on bias frames. The -variance is calculated in boxes of 20x20 pixels, and then linearly -interpolated to cover the full image. - -Unfortunately, due to correlations within this noise, the variance -measured from the bias images does not fully remove the positional -dependence of objects that are detected. This simple noisemap -underestimates the noise observed when the image is filtered during -the object detection process. This filtering convolves the background -noise with a PSF, which has the effect of amplifying the correlated -peaks in the noise. This amplification can therefore boost background -fluctuations above the threshold used to select real objects, -contaminating the final object catalogs. - -In the detection process, we expect false positives at a rate equal to -the one-tailed probability beyond the detection threshold. For these -tests, only detections measured at the $\sigma_{thresh} = 5\sigma$ -level are used, to match that used in the photometry on science data. -This probability can be converted into a number of false number by -considering a given area. As the detections must be isolated to not -be detected as an extended object, this area must be reduced by the -area a given PSF occupies. Combining this, we find that we expect a -probability $P = 1 - \Phi_{normal}(5) = \frac{1}{2} -\erfcinv\left(\frac{5}{\sqrt{2}}\right)$, and an area given $N$ -exposures of area $X\times Y$, $A = \frac{X \times Y \times - N}{A_{PSF}}$. For a typical $1"$ seeing, $A_{PSF}$ is approximately -16 pixels. Using this model for the false positives, we found that -the added read noise was insufficient to account for the observed -false positive rate. Inverting this relation, we can measure -$\sigma_{obs}$, the true threshold level based on the number of false -positives observed. This $\sigma_{obs}$ is the combined to form a -boost factor $B = \sigma_{thresh} / \sigma_{obs}$ that amplifies the - noisemap to match the observed false detection rate. - -The row-to-row variations that contribute to the extra noise are -related to the dark model, and because of this, as the dark model -changes, the effective noise also changes. To ensure that the -noisemap accurately matches the true noise level, we have created -different noisemap models for the three major time ranges of the dark -model. We do not see any strong evidence that the noisemaps have the -A/B modes visible in the dark, and so we do not generate different -models for each individual dark model. The additional pixel-to-pixel -variance from this noisemap is added to the Poissonian variance to -form the science variance image generated by the \ippstage{chip} -processing. - -\subsection{Flat} - -Determining a flat field correction for GPC1 is a challenging -endeavor, as the wide field of view makes it difficult to construct a -uniformly illuminated image. Using a dome screen is not possible, as -the variations in illumination and screen rigidity create large -scatter between different images that are not caused by the detector -response function. Because of this, we use sky flat images taken at -twilight, which are more consistently illuminated than screen flats. -We calculate the mean of these images to determine the initial flat -model. - -From this starting model, we construct a correction to remove the -effect of the illumination differences over the detector surface. -This is done by dithering a series of science exposures with a given -pointing. By fully calibrating these exposures with the initial flat -model, and then comparing the measured fluxes for the same star as a -function of position on the detector, we can determine position -dependent scaling factors. From the set of scaling factors for the -full catalog of stars observed in the dithered sequence, we can -construct a model of the error in the initial flat model as a function -of detector position. Applying a correction that reduces the -amplitude of these errors produces a flat field model that better -represents the true detector response. - -The flat model appears stable with time, although directly measuring -this is as difficult as originally constructing the model. However, -due to the photometric consistency observed in the final catalog of -GPC1 measurements \citep{MagnierXXX}, we can be confident that the -flat model does not have a significant time dependent component. - -\subsection{Pattern correction} -\label{sec:pattern} - -Due to detector specific issues that are not cleanly removed by the -dark model, we have a set of ``pattern'' corrections that are applied -to some selection of the OTAs in the camera. This is done to reduce -the effect that detector differences have on the measured astronomical -signal that are not stable enough to be corrected with a static model. -Because of this, the pattern corrections attempt to identify and -correct the detector issues based on appropriate filtering the -individual science exposures. - -The PATTERN.ROW correction is used to remove any remaining row-by-row -bias variation, and the PATTERN.CELL and PATTERN.CONTINUITY -corrections attempt to ensure that the cells of a given OTA are -consistent with the other cells on that OTA. - -\subsubsection{Pattern Row} -% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Bias_Pattern_Study -As discussed above in the dark and noisemap sections, certain -detectors have significant bias offsets between adjacent rows, caused -by noise in the camera control electronics. The magnitude of these -offsets increases as the distance from the readout amplifier -increases, resulting in horizontal streaks that are more pronounced -along the large x pixel edge of the cell. As the level of the offset -is apparently random between exposures, the dark correction cannot -fully remove this structure from the images, and the noisemap value -only indicates the level of the average variance added by these bias -offsets. Therefore, we apply the PATTERN.ROW correction in an attempt -to mitigate the offsets and correct the image values. To force the -rows to agree, a second order clipped polynomial is fit to each row in -the cell. Four fit iterations are run, and pixels $2.5\sigma$ deviant -are excluded from subsequent fits, to minimize the effect stars and -other astronomical signals have. This final trend is then subtracted -from that row. Simply doing this subtraction will also have the -effect of removing the background sky level. To prevent this, the -constant and linear terms for each row are stored, and linear fits are -made to these parameters as a function of row, perpendicular to the -initial fits. This produces a plane that is added back to the image -to restore the background offset and any linear ramp that exists in -the sky. - -This correction was required on all cells on all OTAs prior to -2009-12-01, at which point a modification of the camera electronics -reduced the scale of the row-by-row offsets for the majority of the -OTAs. As a result, we only apply this correction to the cells where -it is still necessary, as shown in Figure \ref{fig: pattern row - cells}. A list of these cells is listed in Table -\ref{tab:pattern_row_cells}. - -Although this correction does largely resolve the row-by-row offset -issue in a satisfactory way, large and bright astronomical objects can -bias the fit significantly. This results in an oversubtraction of the -offset near these objects. As the offsets are calculated on the pixel -rows, this oversubtraction is not uniform around the object, but is -preferentially along the horizontal x axis of the object. Most -astronomical objects are not significantly distorted by this, with -this only becoming on issue for only bright objects comparable to the -size of the cell (598 pixels = 150"). - -%% \czwdraft{keep this?} This row-by-row offset is visible in similar -%% camera designs, and has been removed by identifying the noise signal -%% in the pixel data stream. By taking the FFT of the pixels and a -%% reference signal, the frequency of this noise can be isolated and -%% removed, resulting in a much cleaner image. However, GPC1 does not -%% record the value of the reference signal, instead automatically -%% subtracting it from the data values. Without this comparison signal, -%% we have been unable to reproduce this method, as there is no obvious -%% FFT component visible. - -\begin{deluxetable}{lcccc} - \tablecolumns{3} - \tablewidth{0pc} - \tablecaption{Cells which have PATTERN.ROW correction applied} - \tablehead{\colhead{OTA} & \colhead{Cell columns} & \colhead{Additional cells}} - \startdata - OTA11 & & xy02, xy03, xy04, xy07 \\ - OTA14 & & xy23 \\ - OTA15 & 0 & \\ - OTA27 & 0, 1, 2, 3, 7 & \\ - OTA31 & 7 & \\ - OTA32 & 3, 7 & \\ - OTA45 & 3, 7 & \\ - OTA47 & 0, 3, 5, 7 & \\ - OTA57 & 0, 1, 2, 6, 7 & \\ - OTA60 & & xy55 \\ - OTA74 & 2, 7 & \\ - \enddata - \label{tab:pattern_row_cells} -\end{deluxetable} - -\begin{figure} - \centering - \includegraphics[width=0.9\hsize,angle=0,clip]{images/pattern_row_edit.png} - \caption{Diagram illustrating in red which cells on GPC1 require the PATTERN.ROW correction to be applied. The footprint of each OTA is outlined, and cell xy00 is marked with either a filled box or an outline. The labeling of the non-existent corner OTAs is provided to orient the focal plane.} - \label{fig: pattern row cells} -\end{figure} - -\begin{figure} - \centering - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_nopat.png} -% \caption{(a)} -% \end{subfigure}% -% \begin{subfigure}[]{.45\hsize} - \end{minipage}% - \begin{minipage}{0.45\hsize} - \includegraphics[width=0.9\hsize,angle=0,clip]{images/o5379g0103o_XY57_pat.png} -% \caption{(b)} -% \end{subfigure} - \end{minipage} - \caption{Example of the PATTERN.ROW correction on exposure o5379g0103o OTA57 cell xy00 (i-filter 45s). The left panel shows the cell with all appropriate detrending except the PATTERN.ROW, and the right shows the same cell with PATTERN.ROW applied. The correction reduces the correlated noise on the right side, which is most distant from the read out amplifier. There is a slight over subtraction along the rows near the bright star.} -\end{figure} - -\subsubsection{Pattern Cell} - -As the measured background level of a given cell may not exactly match -that of its neighbors, fitting a smooth background model over the full -OTA can result in over and under-subtraction of the sky level at the -cell boundary discontinuities. The PATTERN.CELL correction was an -initial attempt to remove this effect on the worst cells, by forcing -all the cells of an OTA to the same level. Each cell had the median -value measured, and then each cell had an offset added that shifts the -cell to match the median of those medians. - -This correction is reasonable when the astronomical signal is smooth, -with no objects that are large relative to the size of an individual -cell. However, the presence of large galaxies (or even bright stars) -can bias the offsets for some cells from their neighbors. Because of -this issue, we no longer apply this correction to any data. - -\subsubsection{Pattern Continuity} - -As the PATTERN.CELL correction was insufficient in many situations, we -designed a replacement correction that would reduce the background -distortion for large objects. In addition, studies of the background -level illustrated that the row-by-row bias can introduce small -background gradient variations along the rows of the cells that is not -stable enough to be completely fit by the dark model. This common -feature across the columns of cells results in a ``saw tooth'' pattern -horizontally across an OTA, and as the background model fits a smooth -sky level, this induces over and under subtraction at the cell -boundaries. As the PATTERN.CELL was designed to correct changes only -in the median value between cells, it could not adequately resolve -this higher order issue. - -The replacement for PATTERN.CELL is the PATTERN.CONTINUITY correction, -which attempts to match the edges of a cell to those of its neighbors. -For each cell, a thin box 10 pixels wide on each edge is extracted and -the median value of unmasked values calculated for that box. These -median values are then used to construct a vector of differences -$\Delta_i = \sum_{j} Edge_{i} - Edge_{j}$, along with a matrix of -associations $A_{i,i'} = \sum_{j} \delta(i,j) \delta(j,i')$ denoting -which cell boundaries are adjacent. By solving the system $A x = -diff$, we find the set of offsets $x_i$ to be applied to each cell to -ensure the minimum differences between all cell edges and their -neighbors. - -For OTAs that initially show the saw tooth pattern, the effect of this -correction is to align the cells into a single ramp, at the expense of -the absolute background level. However, as we subtract off a smooth -background model prior to doing photometry, these deviations from an -absolute sky level are unimportant. The fact that the final ramp is -smoother than it would be otherwise also allows for the background -subtracted image to more closely match the astronomical sky, without -significant errors at cell boundaries. An example of the effect of -this correction on an image profile is shown in Figure \ref{fig:dark switching}. - -%% \begin{figure} -%% \centering -%% \caption{Continuity example, with background issue.} -%% \label{fig: continuity example} -%% \end{figure} - -\subsection{Fringe correction} -\label{sec:fringe} -% det_id 296 is the fringe we use. - -\czwdraft{This is still a mess} - -Due to variations in the thickness of the detectors, we observe -interference patterns at the infrared end of the filter set, as the -wavelength of the light becomes comparable to the thickness of the -detectors. Visually inspecting the images shows that the fringing is -most prevalent in the y-filter images, with negligible fringing in -other bands. As a result of this, we only apply a fringe correction -to the y filter data. - -The fringe used for PV3 processing was constructed from a set of 20 -120s science exposures. These exposures are overscan subtracted, and -corrected for non-linearity, and have the dark and flat models -applied. These images are smoothed with a Gaussian of $\sigma = 2$ -pixels to minimize pixel to pixel noise. The fringe image data is -then constructed by calculating the clipped mean of the input images -with two iteration of clipping at the $3\sigma$ level. - -A course background model is constructed by calculating the median on -a 3x3 grid (approximately 200x200 pixels each). A set of 1000 -randomly selected points are selected on the fringe image in each -cell, and a median calculated for this position in a 10x10 pixel box, -with the background level subtracted. These sample locations provide -scale points to allow the amplitude of the measured fringe to be -compared to that found on science images. - -To apply the fringe, the same sample locations are measured on science -image to determine the relative strength of the fringing in that -particular image. A least squares fit between the fringe measurements -and the corresponding measurements on the science image provides the -scale factor multiplied to the fringe before it is subtracted from the -science image. - -\begin{figure} - \centering - \begin{minipage}{0.5\hsize} - \includegraphics[width=1.0\hsize,angle=0,clip]{images/o5220g0025o_XY53_nofringe.png} -% \caption{(a)} -% \end{subfigure}% -% \begin{subfigure}[]{.45\hsize} - \end{minipage}% - \begin{minipage}{0.5\hsize} - \includegraphics[width=1.0\hsize,angle=0,clip]{images/o5220g0025o_XY53_fringe.png} -% \caption{(b)} -% \end{subfigure} - \end{minipage} - \caption{Example of the y-filter fringe pattern on exposure o5220g0025o OTA53 (y-filter 30s). The left panel shows the OTA mosaic with all detrending except the fringe correction, while the right shows the same including the fringe correction. Both images have been smoothed with a Gaussian with $\sigma = 3$ pixels to highlight the faint and large scale fringe patterns. \czwdraft{See if there's a way to have mana produce images larger than the screen size.}} - \label{fig: fringe example} -\end{figure} - -\subsection{Background subtraction} -\label{sec:background} - -Once all other detrending is done, the pixels from each cell are -mosaicked into the full $4846\times{}4868$ pixel OTA image. A -background model for the full OTA is then determined prior to the -photometric analysis. The mosaicked image is binned into -$800\times{}800$ pixel bins, centered on the image center, and -overlapping by a factor of 2 in both axes. These bins have 10000 -random samples drawn, and a binned cumulative distribution function is -generated. These bins are interpolated to find the best mean value at -the $50\%$ level, as well as the distribution $\sigma$ by estimating -from the $32\%$ and $68\%$ levels. Repeating this across all bins -results in a $13\times{}13$ grid of background bins, which are -bilinearly interpolated to generate the background model to subtract. -Each object in the photometric catalog has a SKY and SKY\_SIGMA value -based on this model as well. - -%% * Magic -%% * Warping -%% * warping kernel -%% * linear-by-pieces -%% * Covariance -%% * def of skycells? -%% * Stacking -%% * pixel combination rules -%% * pixel rejections -%% * convolution for matching (success and failure) -%% * Difference Image analysis - - \section{Warping} \label{sec:warping}