Index: trunk/doc/release.2015/ps1.detrend/detrend.tex =================================================================== --- trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 37862) +++ trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 37895) @@ -1,7 +1,8 @@ -\documentclass[iop,floatfix]{emulateapj} +%\documentclass[iop,floatfix]{emulateapj} + % \pdfoutput=1 % see latex.readme.txt for notes on using the PS1 template -%\documentclass[12pt,preprint]{aastex} +\documentclass[12pt,preprint]{aastex} %\documentclass[manuscript]{aastex} %\documentclass[preprint2]{aastex} @@ -21,4 +22,14 @@ %\def\picdir{PATH} \def\picdir{ALTPATH} + +% CZW commands from my previous draft. +\newcommand{\czw}[1]{ + \textbf{CZW: }\textcolor{red}{#1} +} +\newcommand{\czwdraft}[1]{ + \textcolor{red}{#1} +} +\newcommand{\erfcinv}{\mathop{\rm erfcinv}\nolimits} + % Pick a terse version of the title here; @@ -129,4 +140,479 @@ \section{INTRODUCTION}\label{sec:intro} +\section{Camera description} + +\czwdraft{60 otas} + +\czwdraft{64 cells per ota} + +\czwdraft{effectively 60x64 different cameras, each with particular gain/noise/etc characteristics} + +\czwdraft{Add summary of detrending steps} + +\section{Burntool / Persistence effect} + +Stars that are nearing saturation \czwdraft{(30000 DN)} cause +persistance problems during the read out of the image, creating trails +of light are left on the image. During the read out process of an +image with a bright star above this threshold, some of the charge +associated with that object is not fully shifted toward the amplifier. +As a result, this charge remains in the starting cell, and is +partially collected in subsequent shifts, resulting in a ``burn +trail'' that extends from the center of the bright source away from +the amplifier (vertically along the pixel columns toward the top of +the cell). + +This incomplete charge shifting in nearly full wells continues as each +row is read out. This results in a remnant charge in the pixels that +the full well was shifted through. In following exposures, this +remnant charge leaks out, resulting in a trail that extends from the +initial location of the bright source on the previous image towards +the amplifier (vertically down along the pixel column). This charge can remain on the detector for up +to thirty minutes, so the locations of these ``burns'' needs to be +retained between exposures. + +Both of these types of persistance trails are corrected via the BURNTOOL program. This +program does an initial scan of the images, and identifies stars +brighter than a given threshold. Then, the trail from that star is +fit with a one-dimensional power law, based on empirical evidence that +this is the functional form of this perseistence effect. Once this +fit is done, the model is subtracted from the image, and the location +of the star is stored in a table along with the exposure PONTIME +\czwdraft{obs time?}. + +For subsequent exposures, the table associated with the previous image +is read in, and after correcting trails from its own stars, it +attempts to find remnant trails from previous images. These are fit +and subtracted using a one-dimensional exponential model, again based +to empirical studies. If no significant model is determined, then +this location is not included in the output table, allowing old burns +to ``expire.'' + +One problem with this method to correct the persistance trails is that +it is based on fits to the image data, which may not be fully +determined by the persistance effect. The presence of other stars or +artifacts along the path of the burn can result in an incorrect model +to be determined, resulting in either an over- or under-subtraction of +the persistance burn. \czwdraft{However, it's better than doing nothing.} +Another issue is that the cores of very bright stars are deformed by +this process, as it preferentially subtracts flux from one side of the +star. As most stars that result in burns already have the cores +saturated, this does not significantly affect PSF determination or +photometry. + +\section{Mask} + +Due to the large size of the detector, it is to be expected that there +will be a number of pixel defects that do not measure light as well as +their neighbors. To remove these pixels, we have constructed a static +mask that contains information about these defects. This mask is +constructed in three phases. + +First, a CTEMASK is constructed to mask out regions in which the +charge transfer efficiency is low compared to the rest of the +detector. Twenty-five of the sixty OTAs in GPC1 show some evidence of +CTE issues, with this pattern showing up (to varying degrees) in +triangular sets of cells on the OTA. \czwdraft{probably a figure would + help explain this?} To generate the mask, a sample set of flat +images are used to generate a map of the image variance with some +binning. As the flat image largely illuminates the image uniformly, +the expected variances should be Poissonian distributed with the flux +level. However, in regions with CTE issues, adjacent pixels are able +to ``share'' their charge, resulting in a lower-than-expected +variance. This allows these regions to be identified and removed from +processing in science images. + +The next step of mask construction is to examine the detector for +bright columns and other pixel issues. This is first done by \czwdraft{I + think Heather wrote a program to do this, but I'm not totally sure + how it works} scanning a set of images for pixels that have values +that do not change throughout the sequence of exposures. Such pixels +cannot be caused by astronomical effects, and must be due to the +detector itself. This does an excellent job of removing the majority +of the problem pixels, and greatly speeds up the manual inspection for +defects. This manual inspection allows human interaction to identify +other odd detector issues that should not be allowed through to +science processing. This is also where the vignetted regions around +the edge of the detector are masked out. \czwdraft{This might be a lie} +As the size of the vignetted region changes with filter, we have been +somewhat aggressive about this, defining the smallest possible +``good'' region by using the g-filter to set this. + +Finally, not all bad regions on the image are due to pixel level +defects. Crosstalk between electronics channels results in the +appearance of faint ``stars'' that appear with the same cell (x,y) +coordinate as a real star, but are shifted to another cell or to +another OTA. We believe we have identified all such crosstalk issues, +and therefore place a mask over the crosstalk ghost when we detect a +sufficiently bright star in a ``source'' location. + +Due to an issue with the anti-reflective coating, we also see large +out of focus objects in the g-filter data. These objects are the +result of a bright source reflecting back off the surface of the +detector, reflecting again off the \czwdraft{No clue} mirror, and then +back down onto the focal plane. These are also somewhat reasonable to +identify, as a bright star in location (X,Y) on the focal plane +creates a reflection ghost at (-X,-Y). The exact location is fit as a +\czwdraft{Nth} order polynomial, and seems to sufficiently cover these +regions. + +\subsection{Optical ghosts} + +%% +%% GHOST.CENTER.X METADATA +%% NORDER_X S32 3 +%% NORDER_Y S32 3 +%% VAL_X00_Y00 F64 -1.215661e+02 +%% VAL_X01_Y00 F64 1.321875e-02 +%% VAL_X02_Y00 F64 -4.017026e-09 +%% VAL_X03_Y00 F64 1.148288e-10 +%% VAL_X00_Y01 F64 -1.908074e-03 +%% VAL_X01_Y01 F64 8.479150e-08 +%% VAL_X02_Y01 F64 1.635732e-11 +%% VAL_X00_Y02 F64 2.625405e-08 +%% VAL_X01_Y02 F64 1.125586e-10 +%% VAL_X00_Y03 F64 2.912432e-12 +%% NELEMENTS S32 10 +%% END + +%% GHOST.CENTER.Y METADATA +%% NORDER_X S32 3 +%% NORDER_Y S32 3 +%% VAL_X00_Y00 F64 2.422174e+01 +%% VAL_X01_Y00 F64 4.170486e-04 +%% VAL_X02_Y00 F64 -1.934260e-08 +%% VAL_X03_Y00 F64 -1.173657e-12 +%% VAL_X00_Y01 F64 1.189352e-02 +%% VAL_X01_Y01 F64 -9.256748e-08 +%% VAL_X02_Y01 F64 1.140772e-10 +%% VAL_X00_Y02 F64 8.123932e-08 +%% VAL_X01_Y02 F64 1.328378e-11 +%% VAL_X00_Y03 F64 1.170865e-10 +%% NELEMENTS S32 10 +%% END +%% # These are the original linear solutions +%% GHOST.INNER.MAJOR METADATA +%% NORDER_X S32 1 +%% VAL_X00 F64 3.926693e+01 +%% VAL_X01 F64 5.325759e-03 +%% NELEMENTS S32 2 +%% END + +%% GHOST.INNER.MINOR METADATA +%% NORDER_X S32 1 +%% VAL_X00 F64 5.287548e+01 +%% VAL_X01 F64 -2.191669e-03 +%% NELEMENTS S32 2 +%% END + +%% GHOST.OUTER.MAJOR METADATA +%% NORDER_X S32 1 +%% VAL_X00 F64 7.928722e+01 +%% VAL_X01 F64 1.722181e-02 +%% NELEMENTS S32 2 +%% END + +%% GHOST.OUTER.MINOR METADATA +%% NORDER_X S32 1 +%% VAL_X00 F64 1.314265e+02 +%% VAL_X01 F64 -2.627153e-03 +%% NELEMENTS S32 2 +%% END + +\subsection{Glints} + +%% +%% GLINT_MAX_MAG F32 -21.0 +%% GLINT.REGION MULTI + +%% GLINT.REGION METADATA +%% REGION STR [-38000:-24000,-20000:+20000] +%% GLINT.TYPE STR LEFT +%% END + +%% GLINT.REGION METADATA +%% REGION STR [+24000:+38000,-20000:+20000] +%% GLINT.TYPE STR RIGHT +%% END + +%% GLINT.REGION METADATA +%% REGION STR [-20000:+20000,+24000:+38000:] +%% GLINT.TYPE STR TOP +%% END + +%% GLINT.REGION METADATA +%% REGION STR [-20000:+20000,-38000:-24000] +%% GLINT.TYPE STR BOTTOM +%% END + + +\czwdraft{Write up something about the masking fraction.} + +\subsection{Video Mask} + +One aspect of the OTAs in GPC1 is that an individual cell can be read +off repeatedly while the other cells integrate, resulting in a video +signal from that cell. This is used for guiding purposes, and a +single exposure is likely to have a number of these video cells. +However, reading these cells while integrating on the others changes +the characteristic dark model (see below) experienced by the other +cells on the OTA. The observational effect of this is that the glow +related to the amplifiers in the corners of the cells is depressed +during the video readout, relative to the nominal glow. Because of +this, the standard dark model oversubtracts this glow. Due to camera +configuration issues \czwdraft{I need to check this}, we are unable to +obtain video dark images, preventing us from correctly modelling this +change in the dark model. Instead, we apply simple masks that block +out these corner anti-glows from the data. This is reasonable, as +other than the corners, most pixels have the same dark model in either +mode. + +\section{Overscan} + +Each cell on GPC1 has an overscan region that covers the +first\czwdraft{?} 34 columns of each row, and the last\czwdraft{?} 10 rows +of each column. No light lands on these pixels, so the image region +is trimmed to exclude them. Each row has an overscan value +subtracted, calculated by finding the median value of that row's +overscan pixels. These medians are then smoothed between rows with a +3-row wide boxcar. + +\section{Non-linearity Correction} + +The pixels of GPC1 are not perfectly linear at all flux levels. +Particularly, at low flux levels, some pixels have a tendency to sag +relative to the expected linear value. This effect is most pronounced +along the edges of the detector, although some entire cells show +evidence of this effect. + +To correct this sag, we study the flux behavior of a series of dark +frames with a ramp of exposure times. As the exposure time increases, +the flux on each pixel also increases in what is expected to be a +linear manner. Each of these dark ramp exposures is overscan +corrected, and then the median is calculated for each cell, as well as +the rows and columns within ten pixels of the edge of the science +region. From these median values at each exposure time value, we can +construct the expected trend by fitting a linear model, $f_{region} = +gain * t_{exp} + bias_0$, to the median fluxes for darks with exposure +times between 3 and 12 seconds. This time interval was selected as it +avoids the non-linearity at low fluxes, as well as the possibility of +high-flux non-linearity effects. From this set of models for each +row, column, or full cell, we construct a table of correction values +by linear interpolating the row and column results to match the full +cell results in the center of the detector. + +This non-linearity effect appears to be stable in time, with no +evident change over a year's worth of data. + +\czwdraft{I have figures at http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearity that might be useful} + +\section{Dark/Bias Subtraction} + +The dark model we make for GPC1 considers each pixel individually, +independent of any neighbors. To create the dark model for each +pixel, we fit an arbitrary dimensional model \czwdraft{clunky} to the +array of input pixels from a selection of dark images. The current +model is linear \czwdraft{really?} in both the exposure time and the +detector temperature. Adding in a constant value for the fit provides +three parameters that define the dark model for that pixel. As this +constant value is effectively the bias value for that pixel, we do not +do a separate bias correction. This model is applied to science +images by fitting the correct dark value based on the exposure time +and detector temperature for that exposure. + +\subsection{Time evolution} + +\czwdraft{The dark model is noticably unstable on time scales of months, and so we have generated a sequence in time to keep the effect of a missed correction low.} + +Unfortunately, the dark model is not consistently stable on the time +span of multiple months. Some of the changes in the dark can be +attributed to changes in the voltage settings of GPC1, but the +majority seem to be the result of some unknown parameter. Largely, we +can separate the dark model history of GPC1 into three epochs. The +first epoch covers all data taken prior to 2010-01-23. This epoch +used a different header keyword for the detector temperature, making +data from this epoch incompatible with later dark models. + +The second epoch covers data between 2010-01-23 and 2011-05-01, and is +characterized by a largely stable but oscillatory dark solution. +There appear to be two modes that the dark model switches between +apparently at random. No clear cause has been established for these +switching, but there are clear differences between the two modes +\czwdraft{figures?}. + +The evidence of these two modes comes from the discovery of a slight +gradient along the rows of certain cells. This is a result of a drift +in the bias level of the detector. Therefore, an appropriate dark +model should remove this gradient entirely. For these two modes, the +magnitude of this bias drift is different, so a single dark model over +corrects the low-magnitude mode, and undercorrects the high-magnitude +mode. Upon identifying this two-mode behavior, and determining the +switching points, two separate darks models were constructed from +appropriate ``A'' and ``B'' mode dark frames. Using the appropriate +dark minimizes the effect of this bias gradient in the dark corrected +data. + +After 2011-05-01, the two-mode behavior of the dark disappears, and is +replaced with a slow dateobs-dependent drift in the magnitude of the +gradient. This drift is sufficiently slow that we have modeled it +using three dateobs-independent dark model for different date ranges. +These darks cover the range from 2011-05-01 to 2011-08-01, 2011-08-01 +to 2011-11-01, and 2011-11-01 and on. The reason for this time +dependent drift is unknown, but we seem to be able to model it with +reasonable accuracy by creating new dark models. + +\section{Noisemap} + +Based on a study of the positional dependence of detected objects, we discovered that the cells in GPC1 do not have uniform noise characteristics. Instead, there is a gradient along the pixel rows, with the noise generally higher away from the read out amplifier. This is likely another effect of the row-by-row bias issue. This gradient has the effect that the read noise increases as the row is read out. To mitigate this noise gradient, we construct a set of noisemap images by measuring the median variance on bias frames. The variance is calculated in boxes of 20x20 pixels, and then linearly interpolated to cover the full image. + +Unfortunately, due to correlations in the row-to-row offsets \czwdraft{in the noise?}, the variance measured from the bias images does not fully remove the positional dependence of objects that are detected. The reason for this is that the simple noisemap underestimates teh noise observed when the image is filtered during the object detection process. This filtering convolves the background noise with a PSF, which has the effect of amplifying the correlated peaks in the noise. This amplification can therefore boost background fluctuations above the threshold used to select real objects, contaminating the final object catalogs. + +To resolve this issue, we chose a typical PSF, and used it to look for detections on a sample of bias images. As the bias has no real sources, all objects found are by definition false, and provides an idea of how much our noisemap estimation deviates from the ``true'' noise observed by the object detection process. For a region of area X*Y, if we find k false detections above our signal-to-noise threshold, then we can estimate how much the noise model deviates from what is observed. The observed noise threshold is defined as $\sigma_{observed} = \sqrt{2} * \erfcinv{2 * k A_{psf} / (X * Y * N_{exp})}$, where $A_{psf}$ is the footprint size of the PSF (taken as 16 pixels), and $N_{exp}$ is the number of exposures examined in this location. From this observed threshold, we scale the noisemap previously calculated by the boost factor $B = \sigma_{thresh} / \sigma_{observed}$. + +The row-to-row variations that contribute to the extra noise are related to the dark model, and because of this, as the dark model changes, the effective noise also changes. Because of this, we have created different noisemap models for the three major time ranges of the dark model. We do not see any evidence that the noisemaps have the A/B modes visible in the dark, and so we do not generate different models. + +\section{Remnance?} + +\czwdraft{Despite the known persistence effects of the detectors, we do not do any remnance correction beyond what is discussed above in the burntool section. Therefore, I probably should just remove this section entirely.} + +\section{Shutter?} + +\czwdraft{I don't believe that we do a shutter correction either. So, again, probably shouldn't include it.} + +\section{Flat} + +\czwdraft{I don't know how the flat calibration code works. We start with flat field images of the sky, but due to the size of the detector, it is difficult to equally illuminate each pixel. Therefore, flat calibration.} + +Determining a flat field correction for GPC1 is a challenging +endeavor, as the wide field of view makes it difficult to construct a +uniformly illuminated image. Using a dome screen is not possible, as +the variations in illumination and screen rigidity create unusably +large scatter between different images. Because of this, we use sky +flat images taken at twilight, which are more consistently illuminated +than screen flats. We calculate the mean of these images to determine +the starting flat model. + +From this initial flat model, we construct a correction to remove the +effect of the problems illuminating the large area. This is done by +dithering a series of exposures across a given pointing. By comparing +the measured fluxes for a given star as a function of position, we can +correct out the errors in the flat model. + +The flat model appears stable with time, although directly measuring +this is as difficult as originally constructing the model. However, +due to the photometric consistency observed in GPC1 measurements, we +can be confident that the flat model is not changing much. + + +\section{Pattern correction} + +Due to the row-by-row bias offsets that are not cleanly removed by the +dark model, we have a set of ``pattern'' corrections that are applied +to some selection of the images. The PATTERN.ROW correction is used +to remove the remaining row-by-row variation, and the PATTERN.CELL and +PATTERN.CONTINUITY corrections attempt to ensure that the cells of a +given OTA are consistent with each other. These corrections are +largely designed to fix issues that are not stable enough with time +for the dark model or flat field model to fully account for the +detector behavior. + +\subsection{Pattern Row} + +As discussed above in the dark and noisemap sections, certain +detectors have significant row-by-row bias offsets. As the level of +the offset is largely random, the dark correction cannot fully remove +this structure from the images. Therefore, we apply the PATTERN.ROW +correction in an attempt to mitigate the offsets. To force the rows +to agree, a \czwdraft{first} order polynomial is fit to each row in the +cell, and that trend subtracted from the data. The median offset +(corresponding to the background level) is then added back to the +image so that the cell matches its neighbors during background +subtraction. + +This correction was required on all cells on all OTAs prior to +\czwdraft{2009-12-01}, at which point a modification of the camera +electronics resolved the row-by-row offsets for the majority of the +detectors. As a result, we only apply this correction where it is +necessary, as shown in figure \czwdraft{X}. + +Although this correction does resolve the row-by-row offset issue in a +satifactory way, large and bright astronomical objects can bias the +fit significantly. This results in an oversubtraction of the offset +near these objects. As the offsets are calculated on the pixel rows, +this oversubtraction is not uniform around the object, but is +preferentially along the $\pm x$ axis of the object. + +\czwdraft{keep this?} This row-by-row offset is visible in similar +camera designs, and has been removed by identifying the noise signal +in the pixel data stream. By taking the FFT of the pixels and a +reference signal, the frequency of this noise can be isolated and +removed, resulting in a much cleaner image. However, GPC1 does not +record the value of the reference signal, instead automatically +subtracting it from the data values. Without this comparison signal, +we have been unable to reproduce this method, as there is no obvious +FFT component visible. + +\subsection{Pattern Cell} + +As the bias level of a given cell may not exactly match that of its +neighbors, fitting a smooth background model results in over and +under-subtraction of the sky level at these discontinuities. The +PATTERN.CELL correction was the first attempt to remove this effect on +the worst cells, by forcing all the cells of an OTA to the same level. +Each cell has the median value measured, and then an offset added that +shifts each cell to match the median of these medians. + +This correction is reasonable when the astronomical signal is smooth, +with no objects that are large relative to the size of an individual +cell. However, the presence of large galaxies (or even bright stars) +can force some cells into a nearly arbitrary offset from their +neighbors. Because of this issue, we no longer apply this correction +to any data. + +\subsection{Pattern Continuity} + +As the PATTERN.CELL correction was clearly defective in many +situations, we designed a replacement correction that would distort +large objects less. In addition, studies of the background level +illustrated that the row-by-row bias introduces a small background +gradient along the rows of the cells. This results in a ``sawtooth'' +pattern across an OTA, and as the background model assumes a smooth +sky level, we saw evidence of over and under subtraction at cell +boundaries. As the PATTERN.CELL was designed to correct mean changes +between cells, it could not adequately resolve this higher order +issue. + +The replacment for PATTERN.CELL was the PATTERN.CONTINUITY correction, +which attempts to match the edges of a cell to those of its neighbors. +For each cell, a thin box on each edge is extracted and the median +value calculated for that box. These median values are then used to +construct a vector of differences $diff_i = \sum_{j,j'} Edge_{i,j} - +Edge)_{i',j'}$, along with a matrix of associations $A_{i,i'} = +\sum_{j,j'} \delta(j,j')$ denoting which cell boundary touches +another. By solving the system $A x = diff$, we can find the set of +offsets $x_i$ that should be applied to each cell to ensure the +minimum differences between cells. + +Due to the known slope in some cells, the effect of this correction is +to align the cells into a single ramp, at the expense of the absolute +background level. However, as we subtract off a smooth background +model, this absolute level is unimportant. The fact that the final +ramp is smoother than it would be otherwise also allows for the +background subtracted image to more closely match the astronomical +sky, without over- and under-subtractions at cell edges. + +%% \section{Fringe correction} + +%% \czwdraft{Due to variations in the thickness of the detectors, we observe interference patterns at the infrared (red?) end of the filters, as the wavelength of the light becomes comparable to these variations. Visually inspecting the images shows that the fringing is most prevalent in the y-filter images, with minimal fringing in other bands. Because of this, we only apply a fringe correction to the y data.} + +%% \czwdraft{The fringe is constructed by randomly determining a set of boxes for each OTA cell, and measuring the sky subtracted median value in those boxes for a series of images. These samples are selected at the same location on each image, allowing the astronomical signal to be removed. A least squares fit to the data is then calculated, providing the model of the fringe strength at that location.} + +%% \czwdraft{Applying the fringe is done in the same way, with samples measured across the image to determine the relative strength of the fringing in this image. The solution derived from the detrend is then scaled to match that observed in the science image, and subtracted away.} + +%% \section{Background subtraction} + +%% \czwdraft{A background model is generated for each OTA, once all the individual cells have been mosaicked together. Super-pixels are then defined that divide the image into XxY subregions, and the mean calculated for each subregion. This grid is shifted by a half-width, and the means recalculated, to double the sampling frequency. A background model is then calculated by interpolating over this sampled grid.} + + \section{Discussion}