Index: trunk/doc/release.2015/ps1.detrend/Makefile =================================================================== --- trunk/doc/release.2015/ps1.detrend/Makefile (revision 39600) +++ trunk/doc/release.2015/ps1.detrend/Makefile (revision 39601) @@ -1,4 +1,4 @@ # $Id: Makefile,v 1.16 2006-01-16 01:11:40 eugene Exp $ - +PDFLATEX = env TEXINPUTS=.:..:inputs:./inputs:LaTeX:$(TEXINPUTS): pdflatex help: @echo "USAGE: make (target)" @@ -13,6 +13,6 @@ detrend.pdf: $(DETREND) - -detrend.ps: $(DETREND) + $(PDFLATEX) $< +#detrend.ps: $(DETREND) include ../Makefile.Common Index: trunk/doc/release.2015/ps1.detrend/detrend.tex =================================================================== --- trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 39600) +++ trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 39601) @@ -1,2 +1,3 @@ + %\documentclass[iop,floatfix]{emulateapj} @@ -31,4 +32,7 @@ } \newcommand{\erfcinv}{\mathop{\rm erfcinv}\nolimits} +\newcommand{\ippprog}[1]{\textbf{\texttt{#1}}} +\newcommand{\ippstage}[1]{\textsc{#1}} +\newcommand{\asinh}{\mathop{\rm asinh}\nolimits} @@ -140,25 +144,135 @@ \section{INTRODUCTION}\label{sec:intro} %% http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?2007ASPC..364..153M&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf -\section{Camera description} - -The Pan-STARRS 1 Science Survey uses the 1.4 giga-pixel GPC1 camera with the PS1 telescope on Haleakala Maui to image the sky north of $-30$ declination. The GPC1 camera is composed of 60 orthogonal transfer array (OTA) devices, each of with is an $8\times{}8$ grid of readout cells. This parallelizes the readout process, reducing the overhead in each exposure. However, as a consequence of this large number of individual detector readouts, there are a number of calibrations that need to be included to ensure the response is the same across the entire field of view. - -The Pan-STARRS image processing pipeline (IPP) is described elsewhere \citep{MagnierXXX}, but a short summary follows. The archive of raw exposures is stored on disk, with a database storing the metadata of exposure parameters. For the PV3 processing, large contiguous regions were defined, and the images for all exposures within that region lauched for the CHIP stage processing. This stage performs the image detrending (described below in section \ref{dead ref}), as well as the single epoch photometry \citep{MagnierXXY}. Following the CHIP stage is the CAMERA stage, in which the astrometry and photometry for entire exposure is calibrated against the reference catalog. This stage also performs masking updates based on the now-known positions and brightnesses of stars that create dynamic features (see \ref{dynamic_masks} below). The WARP stage is the next to operate on the data, transforming the detector oriented CHIP stage images into sky-oriented images that have common tesselations and sky projections (Section \ref{warping}). When all WARP stage processing is done for a region of the sky, STACK processing is performed (Section \ref{stacking}) to construct deeper, fully populated images from the set of WARP images that cover that region of the sky. Beyond the STACK stage, a series of addition stages are done that are described in other papers. Transient features are identified in the DIFF stage, which takes input WARP and/or STACK data and performs image differencing \citep{HuberXXX}. Further photometry is performed in the STATICSKY and SKYCAL stages, which add extended source fitting to the point source photometry of objects detected in the STACK images, and calibrate the results against the reference catalog. The FULLFORCE stage takes the catalog output of the SKYCAL stage, and uses the objects detected in that to perform forced photometry on the individual WARP stage images. The details of this photometry are provided in \citet{MagnierXXY}. - -The full detrend application and processing are described in detail in the the sections below, but a short summary follows. Once an exposure has been observed on the summit, it is transferred to the main IPP processing cluster at the MRTC-B and registered into the processing database. This triggers a new chip stage reduction for each of the 60 OTA images that detrends and mosaicks the individual readout cells before measuring the photometric properties of the astronomical objects detected therein. To begin the detrending, pre-determined static bad pixel masks are used to exclude detector regions that are known to be uncorrectable. Following this, persisitence trails related to the incomplete transfer of charge in the readout process are corrected. The image overscan is subtracted, and the known non-linearity of full readouts cells and the substantially worse issues on the edges of the cells are boosted to the expected levels. The temperature and exposure time dependent dark model is then applied, and the noisemap related to the correlated read noise that is not fully corrected by the dark model is then calculated. The flat field correction is applied next, and any fringe correction necessary for long wavelength data is subtracted. Finally, GPC1 specific ``pattern'' corrections are applied to attempt to reduce the cell to cell differences within a single OTA. - +\section{Introduction and Survey Description} + + +The Pan-STARRS 1 Science Survey uses the 1.4 giga-pixel GPC1 camera with the PS1 telescope on Haleakala Maui to image the sky north of $-30^\circ$ declination. The GPC1 camera is composed of 60 orthogonal transfer array (OTA) devices, each of with is an $8\times{}8$ grid of readout cells. This parallelizes the readout process, reducing the overhead in each exposure. However, as a consequence of this large number of individual detector readouts, there are a number of calibrations that need to be included to ensure the response is consistent across the entire field of view. + +The PV3 reduction represents the third full processing version of the Pan-STARRS archival data. The first two reductions were used internally for pipeline optimization and the development of the initial photometric and astrometric reference catalog. The products from these reductions were not publicly released, but have been used to produce a wide range of scientific papers from the Pan-STARRS 1 Science Consortium members. + +The Pan-STARRS image processing pipeline (IPP) is described elsewhere \citep{MagnierKaiserChambers2006}, but a short summary follows. The archive of raw exposures is stored on disk, with a database storing the metadata of exposure parameters. For the PV3 processing, large contiguous regions were defined, and the images for all exposures within that region lauched for the \ippstage{chip} stage processing. This stage performs the image detrending (described below in section \ref{sec:detrending}), as well as the single epoch photometry \citep{MagnierXXY}, in parallel on the individual OTA device data. Following the \ippstage{chip} stage is the \ippstage{camera} stage, in which the astrometry and photometry for the entire exposure is calibrated against the reference catalog. This stage also performs masking updates based on the now-known positions and brightnesses of stars that create dynamic features (see Section \ref{sec:dynamic_masks} below). The \ippstage{warp} stage is the next to operate on the data, transforming the detector oriented \ippstage{chip} stage images into sky oriented images that have common tesselations and sky projections (Section \ref{sec:warping}). When all \ippstage{warp} stage processing is done for the region of the sky, \ippstage{stack} processing is performed (Section \ref{sec:stacking}) to construct deeper, fully populated images from the set of \ippstage{warp} images that cover that region of the sky. Beyond the \ippstage{stack} stage, a series of additional stages are done that are more fully described in other papers. Transient features are identified in the \ippstage{diff} stage, which takes input \ippstage{warp} and/or \ippstage{stack} data and performs image differencing \citep{HuberXXX}. Further photometry is performed in the \ippstage{staticsky} and \ippstage{skycal} stages, which add extended source fitting to the point source photometry of objects detected in the \ippstage{stack} images, and calibrate the results against the reference catalog. The \ippstage{fullforce} stage takes the catalog output of the \ippstage{skycal} stage, and uses the objects detected in that to perform forced photometry on the individual \ippstage{warp} stage images. The details of these stages are provided in \citet{MagnierXXY}. + +The same reduction procedure described above is also performed in real time on new exposures as they are observed by the telescope. This process is largely automatic, with new exposures being downloaded from the summit to the main IPP processing cluster at the Maui Research and Technology Center in Kihei, and registered into the processing database. This triggers a new \ippstage{chip} stage reduction for science exposures, advancing processing upon completion through to the \ippstage{diff} stage. This allows the ongoing solar system moving object search to identify candidates for follow up observations within 24 hours of the initial set of observations \citep{WainscoatXXX}. + +\czwdraft{Should there be a discussion of any header keywords/OTA file formats?} + +Section \ref{sec:detrend construction} provides an overview of the detrend creation process for GPC1, with details of the application of those detrends to correct particular issues in Section \ref{sec:detrending}. The further image processing steps for \ippstage{warp} and \ippstage{stack} are given in Sections \ref{sec:warping} and \ref{sec:stacking} respectively. + +\czwdraft{An analysis of the algorithms used to complete the \ippstage{warp} (section \ref{sec:warping}) and \ippstage{stack} (section \ref{sec:stacking}) stage transformations of the image data to from the detector frame to a common sky frame, and the co-adding of those common sky frame images continues after the list of detrend steps. Finally, a discussion of the remaining issues and possible future development is presented in section \ref{sec:discussion}.} % Discuss 2-phase/3-phase device differnces -\section{Burntool / Persistence effect} - -Stars that are nearing saturation on GPC1 cause -persistance problems during the read out of the image, creating trails -of light are left on the image. During the read out process of an -image with a bright star above this threshold, some of the charge -associated with that object is not fully shifted toward the amplifier. -As a result, this charge remains in the starting cell, and is -partially collected in subsequent shifts, resulting in a ``burn +%\section{General Detrend Discussion} +%\label{sec:detrending} + +\section{GPC1 Detrend Construction} +\label{sec:detrend construction} + +The detrends for GPC1 are all constructed in similar ways. A series of appropriate exposures is selected from the database, and processed with the \ippprog{ppImage} program. The extent of this processing is dependent on the order in which the detrend is applied to science data. In general, the input exposures to the detrend have all stages of detrend processing applied. Table \ref{tab:detrend ppImage} summarizes stages applied the detrends we construct. + +Once the input data has been prepared, the \ippprog{ppMerge} program is used to construct some sort of ``average'' of the inputs. This step need not be a mathematical average, but is used to combine the signal from the individual exposures into a single output product. Table \ref{tab:detrend ppMerge} lists some of the properties of the process for the detrends, including how discrepant values are removed and the combination method used. The outputs from this step have the format of the detrend under construction, and after construction, are applied to the processed input data. This creates a set of residual files that can be checked to determine if the newly created detrend works correctly. + +The process of detrend construction and testing can be iterated, with individual exposures excluded if they are found to be contaminating the output. If the final detrend is considered sufficient, then the iterations are stopped and the detrend is finalized by selecting the date range to which it applies. This allows subsequent science processing to select the detrends needed based on the observation date. Table \ref{tab:detrend list} lists the set of detrends used in the PV3 processing. + +\begin{deluxetable}{lcccc} + \tablecolumns{5} + \tablewidth{0pc} + \tablecaption{Detrend Construction Processing} + \tablehead{\colhead{Detrend Type} & \colhead{Overscan Subtracted} & \colhead{Nonlinearity Correction} & \colhead{Dark Subtracted} & \colhead{Flat Applied} } + \startdata + LINEARITY & Y & & & \\ + DARKMASK & Y & Y & Y & \\ + FLATMASK & Y & Y & Y & Y \\ + CTEMASK & Y & Y & Y & Y \\ + DARK & Y & Y & & \\ + NOISEMAP & Y & Y & & \\ + FLAT & Y & Y & Y & \\ + FRINGE & Y & Y & Y & Y \\ + \enddata + \label{tab:detrend ppImage} +\end{deluxetable} + +\begin{deluxetable}{lcccc} + \tablecolumns{5} + \tablewidth{0pc} + \tablecaption{Detrend Merge Options} + \tablehead{\colhead{Detrend Type} & \colhead{Iterations} & \colhead{Rejection Threshold} & \colhead{Additional Clipping} & \colhead{Combination Method} } + \startdata + DARKMASK & 3 & $8\sigma$ & & Mask pixel if $>10\%$ rejected \\ + FLATMASK & 3 & $3\sigma$ & & Mask pixel if $>10\%$ rejected \\ + CTEMASK & 2 & $2\sigma$ & & Clipped mean; mask pixel if $\sigma^2/\langle I\rangle < 0.5$ \\ + DARK & 2 & $3\sigma$ & & Clipped mean \\ + NOISEMAP & 2 & $3\sigma$ & & Mean \\ + FLAT & 1 & $3\sigma$ & Exclude top $30\%$ and bottom $10\%$ & Mean \\ + FRINGE & 2 & $3\sigma$ & & Clipped mean \\ + \enddata + \label{tab:detrend ppMerge} +\end{deluxetable} + +\begin{deluxetable}{lclll} + \tablecolumns{5} + \tablewidth{0pc} + \tablecaption{PV3 Detrends} + \tablehead{\colhead{Detrend Type} & \colhead{Detrend ID} & \colhead{Start Date} & \colhead{End Date} & \colhead{Note} } + \startdata + LINEARITY & 421 & & & \\ + MASK & 945 & 2009-01-01 00:00:00 & & \\ + & 946 & 2009-12-09 00:00:00 & & \\ + & 947 & 2010-01-01 00:00:00 & & \\ + & 948 & 2011-01-06 00:00:00 & & \\ + & 949 & 2011-03-09 00:00:00 & 2011-03-10 23:59:59 & \\ + & 950 & 2011-08-02 00:00:00 & & \\ + & 1072 & 2015-12-17 00:00:00 & & Update OTA62 mask \\ + DARK & 223 & 2009-01-01 00:00:00 & 2009-12-09 00:00:00 & \\ + & 229 & 2009-12-09 00:00:00 & & \\ + & 863 & 2010-01-23 00:00:00 & 2011-05-01 00:00:00 & A-mode \\ + & 864 & 2011-05-01 00:00:00 & 2011-08-01 00:00:00 & \\ + & 865 & 2011-08-01 00:00:00 & 2011-11-01 00:00:00 & \\ + & 866 & 2011-11-01 00:00:00 & 2019-04-01 00:00:00 & \\ + & 869-935 & 2010-01-25 00:00:00* & 2011-04-25 23:59:59* & B-mode \\ + VIDEODARK & 976 & 2009-01-01 00:00:00 & 2009-12-09 00:00:00 & \\ + & 977 & 2009-12-09 00:00:00 & 2010-01-23 00:00:00 & \\ + & 978 & 2010-01-23 00:00:00 & 2011-05-01 00:00:00 & A-mode \\ + & 979 & 2011-05-01 00:00:00 & 2011-08-01 00:00:00 & \\ + & 980 & 2011-08-01 00:00:00 & 2011-11-01 00:00:00 & \\ + & 981 & 2011-11-01 00:00:00 & 2019-04-01 00:00:00 & \\ + & 982-1048 & 2010-01-25 00:00:00* & 2011-04-25 23:59:59* & B-mode \\ + & 1049 & 2010-09-12 00:00:00 & 2011-05-01 00:00:00 & A-mode with OTA47fix \\ + NOISEMAP & 963 & 2008-01-01 00:00:00 & 2010-09-01 00:00:00 & \\ + & 964 & 2010-09-01 00:00:00 & 2011-05-01 00:00:00 & \\ + & 965 & 2011-05-01 00:00:00 & & \\ + FLAT & 300 & 2009-12-09 00:00:00 & & g filter \\ + & 301 & 2009-12-09 00:00:00 & & r filter \\ + & 302 & 2009-12-09 00:00:00 & & i filter \\ + & 303 & 2009-12-09 00:00:00 & & z filter \\ + & 304 & 2009-12-09 00:00:00 & & y filter \\ + FRINGE & 296 & 2009-12-09 00:00:00 & & \\ + ASTROM & 1064 & 2008-05-06 00:00:00 & & \\ + \enddata + \label{tab:detrend list} +\end{deluxetable} + +\section{GPC1 Detrend Details} +\label{sec:detrending} + +Ensuring a consistent and uniform detector response across the three-degree diameter field of view of the GPC1 camera is essential to a well calibrated survey. Many standard image detrending steps are done for GPC1, with overscan subtraction removing the detector bias level, dark frame subtraction to remove temperature and exposure time dependent detector glows, and flat field correction to remove pixel to pixel response functions. We also construct fringe correction for the reddest data in the y filter, to remove the interference patterns that arise in that filter due to the variations in the thickness of the detector surface. + +These corrections, however, assume that the detector response is linear across the full range of values. This is not universally the case with GPC1, and this requires an additional set of detrending steps to remove these non-linear responses. The first of these is the \ippprog{burntool} correction, which removes the persistence trails caused by the incomplete transfer of charge along the readout columns. This bright-end nonlinearity is generally only evident for the brightest stars, as only pixels that are at or beyond the saturation point of the detector have this issue. More widespread is the non-linearity at the faint end of the pixel range. Some readout cells and some readout cell edge pixels experience a sag relative to linear at low illumination, such that faint pixels appear fainter than expected. The correction to this requires amplifying the pixel values in these regions to match the expected model. + +The final non-linear response issue has no good option for correction. Large regions of some OTA cells experience charge transfer issues, making them unusable to be used for science observations. These regions are therefore masked in processing, with these CTE regions making up the largest fraction of masked pixels on the detector. Other regions are masked for other regions, such as static bad pixel features or temporary readout masking caused by issues in the camera electronics that make these regions unreliable. These all contribute to the detector mask, which is augmented in each exposure for dynamic features that are masked based on the astronomical features within the field of view. + +For the PV3 processing, all detrending is done by the \ippprog{ppImage} program. This program applies the detrends to the individual cells, and then an OTA level mosaic is constructed for the science image, the mask image, and the variance map image. The single epoch photometry is done at this stage as well. The following subsections (\ref{sec:burntool} - \ref{sec:background}) detail these detrending steps, presented in the order in which they are applied to the individual OTA image data. + +\subsection{Burntool / Persistence effect} +\label{sec:burntool} + +Pixels that approach the saturation point on GPC1, which varies by +readout with common values around 60000 DN, cause persistance problems +on that and subsequent images. During the read out process of an image with such a +bright pixel, some of the charge associated with +it is not fully shifted down the detector column toward the +amplifier. As a result, this charge remains in the starting cell, and +is partially collected in subsequent shifts, resulting in a ``burn trail'' that extends from the center of the bright source away from the amplifier (vertically along the pixel columns toward the top of @@ -166,43 +280,52 @@ This incomplete charge shifting in nearly full wells continues as each -row is read out. This results in a remnant charge in the pixels that +row is read out. This results in a remnant charge being deposited in the pixels that the full well was shifted through. In following exposures, this remnant charge leaks out, resulting in a trail that extends from the initial location of the bright source on the previous image towards -the amplifier (vertically down along the pixel column). This charge +the amplifier (vertically down along the pixel column). This remnant charge can remain on the detector for up to thirty minutes, requiring the -locations of these ``burns'' needs to be retained between exposures. - -Both of these types of persistance trails are corrected via the -BURNTOOL program. This program does an initial scan of the images, -and identifies stars brighter than a given threshold of 30000 DN. The -trail from that star is fit with a one-dimensional power law in each pixel column, based on empirical evidence that this -is the functional form of this persistence effect. Once this fit is -done, the model is subtracted from the image, and the location of the -star is stored in a table along with the exposure PONTIME, which -denotes the number of seconds since the detector was last powered on. - -For a subsequent exposure, the table associated with the previous -image is read in, and after correcting trails from the stars on that -new image, it attempts to find remnant trails stored in the table. -These are fit and subtracted using a one-dimensional exponential -model, again based on empirical studies. If a significant model with -is determined \czwdraft{$\alpha$ < 4}, then this location is retained -in the image output table. If not, the old burn is allowed to -``expire.'' +locations of these ``burns'' be retained between exposures. + +Both of these types of persistance trails are detected and optionally repaired via the +\ippprog{burntool} program. This program does an initial scan of the images, +and identifies objects with pixel values brighter than a threshold of +30000 DN. The trail from that star is fit with a one-dimensional +power law in each pixel column above that threshold, based on +empirical evidence that this is the functional form of this +persistence effect. This also matches the expectation that + a constant fraction of charge is incompletely transfered at each + shift beyond the persistence threshold. Once this fit is done, the +model can subtracted from the image, and the location of the star is +stored in a table along with the exposure PONTIME, which denotes the +number of seconds since the detector was last powered on and provides +an internally consistent time scale. + +For subsequent exposures, the table associated with the previous image +is read in, and after correcting trails from the stars on the new +image, the positions of the bright stars from the table are used to +check for remnant trails on the image. These are fit and subtracted +using a one-dimensional exponential model, again based on empirical +studies. If a significant model with is determined, then this +location is retained in the image output table. If not, the old burn +is allowed to expire. An issue with this method of correcting the persistance trails is that -it is based on fits to the raw image data, which may have other -signals not determined by the persistence effect. The presence of -other stars or artifacts along the path of the burn can result in an -incorrect model to be determined, resulting in either an over- or -under-subtraction of the persistance burn. \czwdraft{However, it's - better than doing nothing.} - -Another issue is that the cores of very bright stars are deformed by -this process, as the burntool fitting preferentially subtracts flux -from one side of the star. As most stars that result in burns already -have the cores saturated, this does not significantly affect PSF -determination or photometry. \czwdraft{reference to photometry paper?} +it is based on fits to the raw image data, which may have other signal +sources not determined by the persistence effect. The presence of +other stars or artifacts along the path of the burn can result in a +poor model to be determined, resulting in either an over- or +under-subtraction of the persistance burn. For this reason, the image +mask is marked with a value indicating that this correction has been +applied. These pixels are not fully excluded, but they are marked as +suspect, which allows them to be excluded from consideration in +subsequent stages, such as image stacking. + +Another concern is that the cores of very bright stars are deformed by +this process, as the burntool fitting subtracts flux +from onlyl one side of the star. As most stars that result in burns already +have saturated cores, they are already ignored for the purpose of +PSF determination and are flagged as saturated by the photometry +reduction. \begin{figure} @@ -214,14 +337,15 @@ \end{figure} -\section{Masking} -\czwdraft{Technically, we mask the image prior to burntool application now.} - -\subsection{Static Masks} - -Due to the large size of the detector, it is to be expected that there -will be a number of pixel defects that do not have the detection sensitivity on par -with their neighbors. To remove these pixels, we have -constructed a static mask that identifies the known defects. This -mask is constructed in three phases. +\subsection{Masking} +\label{sec:masking} + +\subsubsection{Static Masks} +\label{sec:static_masks} + +Due to the large size of the detector, it is expected that there +are a number of pixel defects that do not have the detection +sensitivity on par with their neighbors. To remove these pixels, we +have constructed a static mask that identifies the known defects. +This mask is constructed in three phases. First, a CTEMASK is constructed to mask out regions in which the @@ -230,77 +354,54 @@ CTE issues, with this pattern showing up (to varying degrees) in roughly triangular patches on the OTA due to defects in the -semiconductor \czwdraft{doping}. To generate the mask for these -regions, a sample set of \czwdraft{N} evenly illuminated flat field -images were measured to produce a map of the image variance in 20x20 -pixel bins. As the flat image is expected to illuminate the image -uniformly, the expected variances in each bin should be Poissonian -distributed with the flux level. However, in regions with CTE issues, -adjacent pixels are not independent, allowing the charge in those -pixels to spread. This reduces the pixel-to-pixel differences, -resulting in a lower-than-expected variance. All regions with -variance \czwdraft{0.5} smaller than expected are added to the static -CTEMASK. +semiconductor manufacturing. To generate the mask for these regions, +a sample set of 26 evenly illuminated flat field images were measured +to produce a map of the image variance in 20x20 pixel bins. As the +flat image is expected to illuminate the image uniformly, the expected +variances in each bin should be Poissonian distributed with the flux +level. However, in regions with CTE issues, adjacent pixels are not +independent, as the charge in those pixels is more free to spread. +This reduces the pixel-to-pixel differences, resulting in a lower than +expected variance. All regions with variance less than half the +average image level are added to the static CTEMASK. The next step of mask construction is to examine the flat and dark models, and exclude pixels that appear to be poorly corrected by these -models. The darkmask process looks for pixels that are more than -\czwdraft{8} sigma discrepant in \czwdraft{10\%} of the -\czwdraft{test} images after those images have had the dark model -applied to them. These pixels are assumed to be unstable with respect -to the dark model, and have the DARK bit set in the static mask, -indicating that they are unreliable in scientific observing. -Similarly, the flatmask process looks for pixels that are \czwdraft{3} -sigma discrepant in the same fraction of \czwdraft{test} images after -both the dark and flat models have been applied. Those pixels that do -not follow the flat field model of the rest of image are assigned the -FLAT mask bit in the static mask, removing the pixels that cannot be -corrected to a linear response. +models. The DARKMASK process looks for pixels that are more than +$8\sigma$ discrepant in $10\%$ of the 100 input dark frame images +after those images have had the dark model applied to them. These +pixels are assumed to be unstable with respect to the dark model, and +have the DARK bit set in the static mask, indicating that they are +unreliable in scientific observing. Similarly, the FLATMASK process +looks for pixels that are $3\sigma$ discrepant in the same fraction of +16 input flat field images after both the dark and flat models have +been applied. Those pixels that do not follow the flat field model of +the rest of image are assigned the FLAT mask bit in the static mask, +removing the pixels that cannot be corrected to a linear response. The final step of mask construction is to examine the detector for bright columns and other static pixel issues. This is first done by -processing a set of \czwdraft{100 i filter} science images in the same -fashion as for the darktest. A median image is constructed from these -inputs along with the per-pixel variance. These images are used to -identify pixels that have unexpectedly low variation between all -inputs, as well as those that significantly deviate from the global -median value. Once this initial set of bad pixels is identified, a -$3\times{}3$ pixel triangular kernel is convolved with the initial -set, and any convolved pixel with value greater than \czwdraft{1.0} is -assigned to the static mask. This does an excellent job of removing -the majority of the problem pixels. A subsequent manual inspection -allows human interaction to identify other inconsistent pixels -including the vignetted regions around the edge of the detector. -\czwdraft{This might be a lie} As the size of the vignetted region -changes with filter, we have used the g filter to set the baseline -unvignetted region to define the static mask, resulting in the -smallest possible unvignetted region. +processing a set of 100 i filter science images in the same fashion as +for the darktest. A median image is constructed from these inputs +along with the per-pixel variance. These images are used to identify +pixels that have unexpectedly low variation between all inputs, as +well as those that significantly deviate from the global median value. +Once this initial set of bad pixels is identified, a $3\times{}3$ +pixel triangular kernel is convolved with the initial set, and any +convolved pixel with value greater than 1 is assigned to the static +mask. This does an excellent job of removing the majority of the +problem pixels. A subsequent manual inspection allows human +interaction to identify other inconsistent pixels including the +vignetted regions around the edge of the detector. + +Figure \ref{fig:static mask} shows an example of the static mask for the full GPC1 field of view. Table \ref{tab:mask_values} lists the bitmask values used for the different sources of masking. \begin{figure} - \caption{Image map of static mask. color coded based on mask reason? It won't be visible at true pixel scale.} + \begin{center} + \includegraphics[width=0.9\hsize,angle=0,clip]{images/gpc1_mask_indexed.png} + \label{fig:static mask} + \end{center} + + \caption{Image map of static mask. color coded based on mask reason? It won't be visible at true pixel scale.} \end{figure} - -\subsection{Dynamic masks} - -In addition to the static mask that removes the detector level -defects, we also generate a set of dynamic masks that change with the -astronomical features in the image. These masks are advisory in -nature, and do not completely exclude the pixel from further -processing consideration. The first of these dynamic masks indicates -the presence of a corrected burntool trail. These pixels are included -for phtometry, but are rejected more readily in the stacking and -difference image construction, as they are more likely to have small -residual contributions from the under or over subtraction of the -burntool correction. - -The remaining dynamic masks are not generated until the IPP camera -stage \czwdraft{IPP paper reference?}, at which point all object -photometry is complete, and an astrometric solution is known for the -exposure. This added information provides the positions of bright -sources based on the reference catalog, including those that fall -slightly out of the detector field of view or within the inter chip -gaps, where internal photometry may not have identified them. These -bright sources are the origin for many of the image artifacts that the -dynamic mask identifies and excludes. - \begin{deluxetable}{ccl} @@ -318,5 +419,5 @@ LOW & 0x0040 & The pixel has a lower value than expected. \\ SUSPECT & 0x0080 & The pixel is suspected of being bad. \\ - BURNTOOL & 0x0080 & The pixel may contain an uncorrected or over-corrected burntool streak. \\ + BURNTOOL & 0x0080 & The pixel contain an burntool repaired streak. \\ CR & 0x0100 & A cosmic ray is present. \\ SPIKE & 0x0200 & A diffraction spike is present. \\ @@ -330,25 +431,51 @@ \label{tab:mask_values} \end{deluxetable} - - -\subsubsection{Crosstalk ghosts} + +\subsubsection{Dynamic masks} +\label{sec:dynamic_masks} + +In addition to the static mask that removes the constant detector level +defects, we also generate a set of dynamic masks that change with the +astronomical features in the image. These masks are advisory in +nature, and do not completely exclude the pixel from further +processing consideration. The first of these dynamic masks is the burntool advisory mask mentioned above. These pixels are included +for photometry, but are rejected more readily in the stacking and +difference image construction, as they are more likely to have small +deviations due to imperfections in the burntool correction. + +The remaining dynamic masks are not generated until the IPP \ippstage{camera} +stage, at which point all object photometry is complete, and an +astrometric solution is known for the exposure. This added +information provides the positions of bright sources based on the +reference catalog, including those that fall slightly out of the +detector field of view or within the inter chip gaps, where internal +photometry may not have identified them. These bright sources are the +origin for many of the image artifacts that the dynamic mask +identifies and excludes. + +\subsubsection{Electronic crosstalk ghosts} +\label{sec:crosstalk} Due to electrical crosstalk between the flex cables connecting the -individual detector devices, ghost objects can be created on some OTAs -due to the presence of a bright source at a different position on the -camera. Table \ref{tab:crosstalk_rules} summarizes the list of known -crosstalk rules. In each of these cases, a source object brighter -than -14.47 magnitude (instrumental) creates a ghost object many -orders of magnitude fainter at the target location. The cell (x,y) +individual detector OTA devices, ghost objects can be created due to +the presence of a bright source at a different position on the camera. +Table \ref{tab:crosstalk_rules} summarizes the list of known crosstalk +rules, with an estimate of the magnitude difference between the source +and ghost. For all of the rules, any cell $v$ within the specified +column of cells on any of the OTAs in the specified column of OTAs $Y$ +creates the ghost in the same $v$ and $Y$ in the target column of +cells and OTAs. In each of these cases, a source object brighter than +-14.47 instrumental magnitude creates a ghost object many orders of +magnitude fainter at the target location. The cell (x,y) pixel coordinate is identical between source and ghost, as a result of the -transfer occurring as the devices are read. A circular mask is asdded +transfer occurring as the devices are read. A circular mask is added to the ghost location with radius $R = 3.44 \left(-14.47 - m_{source, instrumental}\right)$ pixels. Any objects in the photometric -catalog found at the location of the ghost mask have a \czwdraft{flag} -set, marking the object as a likely ghost. The majority of the +catalog found at the location of the ghost mask have the GHOST mask +bit set, marking the object as a likely ghost. The majority of the crosstalk rules are bi-directional, with a source in either position creating a ghost at the corresponding crosstalk target position. The -two faintest rules are uni-directional, likely due to differences in -the \czwdraft{magical properties of the electronics}. +two faintest rules are uni-directional, due to differences in the +electronic path for the crosstalk. For the very brightest sources ($m_{instrumental} < -15$), there can @@ -360,5 +487,5 @@ the bright source. The width of this box is a function of the source magnitude, with $W = 5 * \left(-15 - m_{source, instrumental}\right)$ - pixels. +pixels. \begin{deluxetable}{lllc} @@ -387,22 +514,60 @@ \subsubsection{Optical ghosts} +\label{sec:optical_ghosts} % http://arxiv.org/pdf/1207.2513v1.pdf -Due to imperfections in the anti-reflective coating, bright sources -can also result in large out of focus objects, particularly in the -g-filter data. These objects are the result of light reflecting back -off the surface of the detector, reflecting again off the lower -surfaces of the optics (particularly the L1 corrector lens), and then -back down onto the focal plane. Due to the extra travel distance, the -resulting source is out of focus and elongated along the radial -direction of the telescope. These optical ghosts can be modeled as a -bright star in location (X,Y) on the focal plane creates a reflection -ghost on the opposite side of the optical axis at (-X,-Y). The exact -location is fit as a third order polynomial in the focal plane x and y -directions. An elliptical annulus mask is constructed at the expected -ghost location, with the major and minor axes defined by linear -functions of the ghost distance from the optical axis, and oriented -along the radius of the detector. All stars brighter than a -filter-dependent threshold (listed in table -\ref{tab:ghost_magnitudes}) have such masks constructed. + +Due to imperfections in the anti-reflective coating on the optical +surfaces of GPC1, bright sources can also result in large out of focus +objects, particularly in the g-filter data. These objects are the +result of light reflecting back off the surface of the detector, +reflecting again off the lower surfaces of the optics (particularly +the L1 corrector lens), and then back down onto the focal plane. Due +to the extra travel distance, the resulting source is out of focus and +elongated along the radial direction of the camera focal plane. These +optical ghosts can be modeled in the focal plane coordinates (L,M) +which has its origin at the center of the focal plane. In this +system, a bright object at location (L,M) on the focal plane creates a +reflection ghost on the opposite side of the optical axis at (-L,-M). +The exact location is fit as a third order polynomial in the focal +plane L and M directions (as listed in Table \ref{tab:ghost_centers}). +An elliptical annulus mask is constructed at the expected ghost +location, with the major and minor axes defined by linear functions of +the ghost distance from the optical axis, and oriented with the +ellipse major axis is along the radial direction (Table +\ref{tab:ghost_radii}). All stars brighter than a filter-dependent +threshold (listed in Table \ref{tab:ghost_magnitudes}) have such masks +constructed. + +\begin{deluxetable}{lcc} + \tablecolumns{3} + \tablewidth{0pc} + \tablecaption{Optical Ghost Center Transformations} + \tablehead{\colhead{Polynomial Term}&\colhead{L center}&\colhead{M center}} + \startdata + $x^0 y^0$ & -1.215661e+02 & 2.422174e+01 \\ + $x^1 y^0$ & 1.321875e-02 & 4.170486e-04 \\ + $x^2 y^0$ & -4.017026e-09 & -1.934260e-08 \\ + $x^3 y^0$ & 1.148288e-10 & -1.173657e-12 \\ + $x^0 y^1$ & -1.908074e-03 & 1.189352e-02 \\ + $x^1 y^1$ & 8.479150e-08 & -9.256748e-08 \\ + $x^2 y^1$ & 1.635732e-11 & 1.140772e-10 \\ + $x^0 y^2$ & 2.625405e-08 & 8.123932e-08 \\ + $x^1 y^2$ & 1.125586e-10 & 1.328378e-11 \\ + $x^0 y^3$ & 2.912432e-12 & 1.170865e-10 \\ + \enddata + \label{tab:ghost_centers} +\end{deluxetable} + +\begin{deluxetable}{lcccc} + \tablecolumns{5} + \tablewidth{0pc} + \tablecaption{Optical Ghost Annulus Axis Length} + \tablehead{\colhead{Radial Order}&\colhead{Inner Major Axis}&\colhead{Inner Minor Axis}& \colhead{Outer Major Axis}&\colhead{Outer Minor Axis}} + \startdata + $r^0$ & 3.926693e+01 & 5.287548e+01 & 7.928722e+01 & 1.314265e+02 \\ + $r^1$ & 5.325759e-03 &-2.191669e-03 & 1.722181e-02 & -2.627153e-03 \\ + \enddata + \label{tab:ghost_radii} +\end{deluxetable} \begin{deluxetable}{lc} @@ -422,37 +587,4 @@ \end{deluxetable} -\czwdraft{include full polynomial forms? How best to do that?} - -\begin{deluxetable}{lcc} - \tablecolumns{3} - \tablewidth{0pc} - \tablecaption{Optical Ghost Center Transformations} - \tablehead{\colhead{Polynomial Term}&\colhead{X center}&\colhead{Y center}} - \startdata - $x^0 y^0$ & -1.215661e+02 & 2.422174e+01 \\ - $x^1 y^0$ & 1.321875e-02 & 4.170486e-04 \\ - $x^2 y^0$ & -4.017026e-09 & -1.934260e-08 \\ - $x^3 y^0$ & 1.148288e-10 & -1.173657e-12 \\ - $x^0 y^1$ & -1.908074e-03 & 1.189352e-02 \\ - $x^1 y^1$ & 8.479150e-08 & -9.256748e-08 \\ - $x^2 y^1$ & 1.635732e-11 & 1.140772e-10 \\ - $x^0 y^2$ & 2.625405e-08 & 8.123932e-08 \\ - $x^1 y^2$ & 1.125586e-10 & 1.328378e-11 \\ - $x^0 y^3$ & 2.912432e-12 & 1.170865e-10 \\ - \enddata - \label{tab:ghost_centers} -\end{deluxetable} - -\begin{deluxetable}{lcccc} - \tablecolumns{5} - \tablewidth{0pc} - \tablecaption{Optical Ghost Annulus Axis Length} - \tablehead{\colhead{Radial Order}&\colhead{Inner Major Axis}&\colhead{Inner Minor Axis}& \colhead{Outer Major Axis}&\colhead{Outer Minor Axis}} - \startdata - $r^0$ & 3.926693e+01 & 5.287548e+01 & 7.928722e+01 & 1.314265e+02 \\ - $r^1$ & 5.325759e-03 &-2.191669e-03 & 1.722181e-02 & -2.627153e-03 \\ - \enddata - \label{tab:ghost_radii} -\end{deluxetable} \begin{figure} @@ -460,17 +592,17 @@ \end{figure} -\subsubsection{Glints} - -\czwdraft{I thought we stopped this because of a hardware change? Is - that not true?} Prior to \czwdraft{DATE}, a reflective surface at -the edge of the camera aperture was open to light passing through the +\subsubsection{Optical glints} +\label{sec:glints} +Prior to \czwdraft{DATE}, a reflective surface at the edge of the +camera aperture was incompletely screened to light passing through the telescope. Sources brighter than $m = -20$ that fell on this reflective surface resulted in light being scattered across the detector surface in a long narrow glint. This surface was physically -masked on \czwdraft{DATE} \czwdraft{right?}, but data prior to that -have a dynamic mask constructed when a reference source falls on the -focal plane within \czwdraft{approximately} one degree of the detector -edge. This mask is 150 pixels wide, with length $L = 2500 \left(-20 - -m_{inst}\right)$. \czwdraft{Am I correct that this is basically a one-degree edge around the detector?} +masked on \czwdraft{DATE}, removing the possiblility of glints in +subsequent data, but that taken prior have a dynamic mask constructed +when a reference source falls on the focal plane within one degree of +the detector edge. This mask is 150 pixels wide, with length $L = +2500 \left(-20 - m_{inst}\right)$ pixels. \czwdraft{Am I correct that + this is basically a one-degree edge around the detector?} %% @@ -502,48 +634,81 @@ \end{figure} -\subsubsection{Diffraction spikes} - -Bright objects also form diffraction spikes that are dynamically +\subsubsection{Diffraction Spikes and Saturated Stars} +\label{sec:diffraction_spikes} + +Bright sources also form diffraction spikes that are dynamically masked. These are filter independent, and are modelled as rectangles -with length $L = 10^{0.096 * (7.35 - m)} - 200$ and width $W = 8 + (L -- 200) * 0.01$. These spikes are dependent on the camera rotation, -and are oriented at $\theta = n * \frac{\pi}{2} - \mathrm{ROTANGLE} + -0.798$. - -\subsubsection{Saturated stars} - -The cores of saturated stars are masked as well, with radius $r = 10.15 * (-15 - m_{inst})$. \czwdraft{good job here.} +with length $L = 10^{0.096 * (7.35 - m_{instrumental})} - 200$ and +width $W = 8 + (L - 200) * 0.01$, with negative values indicating no +mask is constructed, as the source is likely too faint to produce the +feature. These spikes are dependent on the camera rotation, and are +oriented at $\theta = n * \frac{\pi}{2} - \mathrm{ROTANGLE} + 0.798$, +based on the header keyword. + +%\subsubsection{Saturated stars} +%\label{sec:saturated_stars} + +The cores of stars that are saturated are masked as well, with a +circular maskradius $r = 10.15 * (-15 - m_{instrumental})$. An +example of a saturated star, with the masked regions for the +diffraction spikes and core saturation highlighted, is shown in Figure +\ref{fig:saturated star}. \begin{figure} \caption{Example of saturated star, which will also nicely show the diffraction spikes.} + \label{fig:saturated star} \end{figure} -\subsection{Video Mask} - -One aspect of the OTAs in GPC1 is that an individual cell can be read -off repeatedly while the other cells integrate, resulting in a video +\subsubsection{Video Mask} +\label{sec:video_masks} + +One aspect of the OTAs on GPC1 is that an individual cell can be read +repeatedly while the other cells integrate, resulting in a video signal from that cell. This data is used for telescope guiding purposes, and a single exposure is likely to have a number of these -video cells in different OTAs. However, reading these cells while -integrating on the others changes the characteristic dark model (see -below) experienced by the other cells on the OTA. The observed effect -of this is that the glow associated with the amplifiers in the corners -of the cells is depressed during the video readout, relative to the -nominal glow. Because of this, the standard dark model oversubtracts -this glow. Before the nature of this issue was fully understood, -these poorly constrained corners were masked with 25-pixel radius -quarter circles, centered on the (0,0) pixel nearest the cell -amplifier. The other corners of the cell were masked with a 15-pixel -radius quarter circle, as the amplifier location is off the edge of -the cell. - - -\subsection{Masking fraction} - -For the full field of view that falls on the sixty OTAs, 14.7\% \czwdraft{check this} of all pixels are masked. The majority of this masking is due to regions that fall within the vignetted region. Defining the radius of the unvignetted region to be 3 degrees, and excluding pixels that fall beyond this point reduces the static masking fraction to 9.7\%. - -Unfortunately, due to the design of the OTAs and readout cells, a non-negligible fraction of the field of view falls onto an area that does not have a detector pixel. For a given OTA mosaicked to a $4846\times{}4868$ pixel image, the 64 $590\times{}598$ pixel readout cells cover 95.7\% of the OTA area. - -For the inter-chip gap area loss, we use two field of view calculations. The reference field of view of GPC1 is 3 degrees, which at the nominal plate scale of 0.258 arcseconds per pixel, translates to a 20930 FPA pixel radius. However, based on the manual masking of the vignetted region, illuminated pixels are generally unvignetted out to 3.25 degrees, or a 22720 FPA pixel radius. Although these result in different coverage areas, summing the number of pixels on OTA mosaicked images ($4846\times{}4868$ pixels) within either field of view results in a inter-chip gap mask fraction of 7\%. +video cells active on different OTAs. For the 3PI survey, the median +exposure has 14 video cells being read, although this number ranges +from less than five to more than thirty, depending on the stellar +density and field pointing. Reading these cells while integrating on +the others changes the characteristic dark model (see Section +\ref{sec:video_darks} below) experienced by the other cells on the +OTA. The observed effect of this is that the glow associated with the +amplifiers in the corners of the cells is suppressed during the video +readout, relative to the nominal glow. The standard dark model +oversubtracts this glow, resulting in dark regions in the corners of +the cells on an OTA taking video data. Before the nature of this +issue was fully understood, these poorly constrained corners were +masked with 25-pixel radius quarter circles, centered on the (0,0) +pixel nearest the cell amplifier. The other corners of the cell were +masked with a 15-pixel radius quarter circle, as the amplifier +creating the glow is associated with another cell, separated by the +inter-cell spacing, diminishing the area affected. Due to the large +area that this masking would cover, the PV3 processing used a more +robust video dark model to correct this problem, as described in +section \ref{sec:video_darks} below. + + +\subsubsection{Masking Fraction} +\label{sec:masking_fraction} + +For the full field of view that falls on the sixty OTAs, 14.7\% +\czwdraft{check this} of all pixels are masked. The large fraction of +this masking is due to regions that fall within the vignetted region. +Defining the diameter of the unvignetted region to be 3 degrees, and +excluding pixels that fall beyond this point reduces the static +masking fraction to 9.7\%. + +Unfortunately, due to the design of the OTAs and readout cells, a +non-negligible fraction of the field of view falls onto an area that +does not have a detector pixel. For a given OTA mosaicked to a +$4846\times{}4868$ pixel image, the 64 $590\times{}598$ pixel readout +cells cover 95.7\% of the OTA area, providing an additional 4.3\% +masking in the unvignetted field of view due to the absense of a +detector pixel. + +For the inter-chip gap area loss, we use two field of view +calculations to estimate the masking fraction. The reference field of +view of GPC1 is 3 degrees, which at the nominal plate scale of 0.258 +arcseconds per pixel, translates to a 20930 FPA pixel radius. %% mysql> select filter,AVG(camProcessedExp.maskfrac_ref_static), AVG(camProcessedExp.maskfrac_ref_dynamic), AVG(camProcessedExp.maskfrac_ref_advisory), AVG(camProcessedExp.maskfrac_max_static),AVG(camProcessedExp.maskfrac_max_dynamic),AVG(camProcessedExp.maskfrac_max_advisory) from camRun join camProcessedExp USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.label = 'LAP.PV3.20140730.final' GROUP BY filter; @@ -563,110 +728,176 @@ %% | 0.21130344126869 | 0.00013634812877977 | 0.02163070300815 | -Summing mask fractions from these three contributions results in an average of $\sim 20\%$ masking fraction across the field of view. Dynamic masking adds an additional $2-3\%$, with advisory burntool masking contributing the largest component. - -\section{Overscan} - -Each cell on GPC1 has an overscan region that covers the -first\czwdraft{?} 34 columns of each row, and the last\czwdraft{?} 10 rows -of each column. No light lands on these pixels, so the image region -is trimmed to exclude them. Each row has an overscan value -subtracted, calculated by finding the median value of that row's -overscan pixels. These medians are then smoothed between rows with a -3-row wide boxcar. - -\section{Non-linearity Correction} - -The pixels of GPC1 are not perfectly linear at all flux levels. -Particularly, at low flux levels, some pixels have a tendency to sag +Summing mask fractions from these three contributions within the +unvignetted field of view results in an average of $\sim 20\%$ masking +fraction across the field of view. Dynamic masking adds an additional +$2-3\%$, with advisory burntool masking contributing the largest +single component. + +\subsection{Overscan} +\label{sec:overscan} + +Each cell on GPC1 has an overscan region that covers the first 34 +columns of each row, and the last 10 rows of each column. No light +lands on these pixels, so the image region is trimmed to exclude them. +Each row has an overscan value subtracted, calculated by finding the +median value of that row's overscan pixels and then smoothing between +rows with a three-row boxcar median. + +\subsection{Non-linearity Correction} +\label{sec:nonlinearity} +% check notebook, 2010-07/08 + +The pixels of GPC1 are not uniformly linear at all flux levels. In +particular, at low flux levels, some pixels have a tendency to sag relative to the expected linear value. This effect is most pronounced -along the edges of the detector, although some entire cells show +along the edges of the detector cells, although some entire cells show evidence of this effect. -To correct this sag, we study the flux behavior of a series of dark -frames with a ramp of exposure times. As the exposure time increases, -the flux on each pixel also increases in what is expected to be a -linear manner. Each of these dark exposures in this exposure time ramp is overscan -corrected, and then the median is calculated for each cell, as well as for -the rows and columns within ten pixels of the edge of the science -region. From these median values at each exposure time value, we can -construct the expected trend by fitting a linear model, $f_{region} = -gain * t_{exp} + bias_0$, to the median fluxes for darks with exposure -times between 3 and 12 seconds. This time interval was selected as it -avoids the non-linearity at low fluxes, as well as the possibility of -high-flux non-linearity effects. From this set of models for each -row, column, or full cell, we construct a table of correction values -by linear interpolating the row and column results to match the full -cell results in the center of the detector. - -This non-linearity effect appears to be stable in time, with little -evident change over the survey duration. - -\czwdraft{I have figures at http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearity that might be useful} +To correct this sag, we studied the flux behavior of a series of flat +frames for a ramp of exposure times with approximate logarithmically +equal spacing between 0.01s and 57.04s. As the exposure time +increases, the flux on each pixel also increases in what is expected +to be a linear manner. Each of these flat exposures in this ramp is +overscan corrected, and then the median is calculated for each cell, +as well as for the rows and columns within ten pixels of the edge of +the science region. From these median values at each exposure time +value, we can construct the expected trend by fitting a linear model, +$f_{region} = G * t_{exp} + B$, to determine the gain, $G$, and the +bias, $B$ for the region considered. This fitting was limited to only +the range of fluxes between 12000 and 38000 counts, as these ranges +were found to match the linear model well. This range avoids the +non-linearity at low fluxes, as well as the possibility of high-flux +non-linearity effects. + +We store the average flux measurement and deviation from the linear +fit for each exposure time for all regions on all detector cells in +the linearity detrend look up tables. When this is applied to science +data, these lookup tables are loaded, and a linear interpolation is +performed to determine the correction needed for the flux in that +pixel. This look up is performed for both the row and column of each +pixel, to allow the edge correction to be applied where applicable, +and the full cell correction elsewhere. The average of these two +values is then applied to the pixel value, reducing the effects of +pixel nonlinearity. + +This non-linearity effect appears to be stable in time for the +majority of the detector pixels, with little evident change over the +survey duration. However, as the non-linearity is most pronounced at +the edges of the detector cells, those are the regions where the +correction is most likely to be incomplete. Because of this fact, +most pixels in the static mask with either the DARKMASK or FLATMASK +bit set are found along these edges. As the non-linearity correction +is unable to reliably restore these pixels, they produce inconsistent +values after the dark and flat have been applied, and are therefore +rejected. + +%% exptime n_included/det_id = 372 +%% clearly this isn't the one used, as 3-12 spans three data points, poorly.x +%% 0.01 2 +%% 0.14 2 +%% 0.27 2 +%% 0.49 2 +%% 0.72 2 +%% 1.06 2 +%% 1.41 2 +%% 2.02 2 +%% 2.63 2 +%% 3.94 2 +%% 5.25 2 +%% 8.74 2 +%% 13.09 2 +%% 17.4 2 +%% 20.86 2 +%% 24.3 2 +%% 27.78 2 +%% 31.24 2 +%% 34.65 2 +%% 38.12 2 +%% 42.41 2 +%% 46.69 2 +%% 51.89 2 +%% 57.04 2 + + %http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearity_AllEdges %http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearityArchive \begin{figure} - \caption{Example plot of linearity as a function of incident brightness.} + \caption{Example plot of linearity as a function of incident brightness/exposure time.} \end{figure} -\section{Dark/Bias Subtraction} +\subsection{Dark/Bias Subtraction} +\label{sec:dark} % http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Dark_Model + The dark model we make for GPC1 considers each pixel individually, -independent of any neighbors. To create the dark model, we fit an multi-dimensional model to the array of input pixels -from a randomly selected set of 100-150 \czwdraft{overscan corrected} -dark frames chosen from a given date range. The model fits -each pixel as a function of the exposure time $t_{exposure}$ and the -detector temperature $T_{chip}$ such that $dark = a_0 + a_1 -t_{exposure} + a_2 T_{chip} t_{exposure} + a_3 T_{chip}^2 -t_{exposure}$. This fitting is performed over the sample of input pixels, -and the coefficients $a_i$ stored in the detrend image. The constant -$a_0$ term includes the bias signal, and as such, a separate bias -subtraction is not necessary. +independent of any neighbors. To create the dark model, we fit an +multi-dimensional model to the array of input pixels from a randomly +selected set of 100-150 overscan and non-linearity corrected dark +frames chosen from a given date range. The model fits each pixel as a +function of the exposure time $t_{exp}$ and the detector temperature +$T_{chip}$ of the input images such that $\mathrm{dark} = a_0 + a_1 +t_{exp} + a_2 T_{chip} t_{exp} + a_3 T_{chip}^2 t_{exp}$. This +fitting uses two iterations to produce a clipped fit, rejecting at the +$3\sigma$ level. The final coefficients $a_i$ for the dark model are +stored in the detrend image. The constant $a_0$ term includes the +residual bias signal after overscan subtraction, and as such, a +separate bias subtraction is not necessary. Applying the dark model is simply a matter of calculating the response -for the exposure time and detector temperature for the image to be +to the exposure time and detector temperature for the image to be corrected, and subtracting the resulting dark signal from the image. -\subsection{Time evolution} - -\czwdraft{The dark model is noticably unstable on time scales of months, and so we have generated a sequence in time to keep the effect of a missed correction low.} - -The dark model is not consistently stable over the full survey, with significant drift over the course of multiple months. Some of the changes in the dark can be -attributed to changes in the voltage settings of GPC1, but the -majority seem to be the result of some unknown parameter. We -can separate the dark model history of GPC1 into three epochs. The -first epoch covers all data taken prior to 2010-01-23. This epoch -used a different header keyword for the detector temperature, making -data from this epoch incompatible with later dark models. +\subsubsection{Time evolution} + +The dark model is not consistently stable over the full survey, with +significant drift over the course of multiple months. Some of the +changes in the dark can be attributed to changes in the voltage +settings of the GPC1 controller electronics, but the majority seem to +be the result of some unknown parameter. We can separate the dark +model history of GPC1 into three epochs. The first epoch covers all +data taken prior to 2010-01-23. This epoch used a different header +keyword for the detector temperature, making data from this epoch +incompatible with later dark models. The second epoch covers data between 2010-01-23 and 2011-05-01, and is characterized by a largely stable but oscillatory dark solution. -There appear to be two modes that the dark model switches between -apparently at random. No clear cause has been established for the -switching, but there are clear differences between the two modes -\czwdraft{figures?}. - -The evidence of these two modes comes from the discovery of a slight -gradient along the rows of certain cells. This is a result of a drift -in the bias level of the detector. Therefore, an appropriate dark -model should remove this gradient entirely. For these two modes, the -magnitude of this bias drift is different, so a single dark model over -corrects the low-magnitude mode, and undercorrects the high-magnitude -mode. Upon identifying this two-mode behavior, and determining the -switching points, two separate darks models were constructed from -appropriate ``A'' and ``B'' mode dark frames. Using the appropriate -dark minimizes the effect of this bias gradient in the dark corrected -data. Table \ref{tab:dark mode dates} lists the dates used for each dark mode. - -\czwdraft{The evidence of the mode switching can be visualized in Figure \ref{fig:dark switching}. This figure shows image profile along the x-pixel axis binned along the full y-axis of dark corrected images for OTA67. These images are from sequential days, and have been corrected with a dark model constructed from the full set of dark data within the second epoch. The opposite sign of the slopes of these profiles indicates that the average dark model does not correct these dates sufficiently, due to the contradictory dark signals between the two modes.} +There are two modes that the dark model switches between apparently at +random. No clear cause has been established for the switching, but +there are clear differences between the two modes that require the +observation dates to be split to use the model that is most +appropriate. + +The initial evidence of these two modes comes from the discovery of a +slight gradient along the rows of certain cells. This is a result of +a drift in the bias level of the detector as it is read out. An +appropriate dark model should remove this gradient entirely. For +these two modes, the direction of this bias drift is different, so a +single dark model generated from all dark images in the time range +over corrects the positive-gradient mode, and under corrects the +negative-gradient mode. Upon identifying this two-mode behavior, and +determining the dates each mode was dominant, two separate darks +models were constructed from appropriate ``A'' and ``B'' mode dark +frames. Using the appropriate dark minimizes the effect of this bias +gradient in the dark corrected data. + +The bias drift gradients of the mode switching can be visualized in +Figure \ref{fig:dark switching}. This figure shows image profile +along the x-pixel axis binned along the full y-axis of dark corrected +images for OTA67. These images are from sequential days, and have +been corrected with a dark model constructed from the full set of dark +data within the second epoch. The opposite sign of the slopes of +these profiles indicates that the average dark model does not correct +these dates sufficiently, due to the contradictory dark signals +between the two modes. \czwdraft{this paragraph dependent on that figure.} After 2011-05-01, the two-mode behavior of the dark disappears, and is -replaced with a slow dateobs-dependent drift in the magnitude of the -gradient. This drift is sufficiently slow that we have modeled it -using three dateobs-independent dark model for different date ranges. -These darks cover the range from 2011-05-01 to 2011-08-01, 2011-08-01 -to 2011-11-01, and 2011-11-01 and on. The reason for this time -evolution is unknown, but we seem to be able to model it with -reasonable accuracy by creating new dark models. +replaced with a slow observation date dependent drift in the magnitude +of the gradient. This drift is sufficiently slow that we have modeled +it using three dateobs-independent dark model for different date +ranges. These darks cover the range from 2011-05-01 to 2011-08-01, +2011-08-01 to 2011-11-01, and 2011-11-01 and on. The reason for this +time evolution is unknown, but as it is correctable with a small +number of dark models, this does not significantly impact detrending. \begin{figure} @@ -674,31 +905,113 @@ \end{figure} -\subsection{Video Dark} - -Dark signal is stronger in cell corners due to amplifier glow. Standard model corrects this. When OTA reads video cell, the dark model changes. The standard model oversubtracts the dark model. Make video darks from dark data that has had video signal running. Need two passes to cover all cells (shifting video cell between the two). Can construct the video dark and the standard dark simultaneously, by using OTAs that have video on and off. - -Video dark signal appears linear and stable, allowing archival data from prior to video dark data to be corrected by simply taking $VD_{2009} = D_{2009} - D_{Modern} + VD{Modern}$. - -\section{Noisemap} - -Based on a study of the positional dependence of detected sources, we have discovered that the cells in GPC1 do not have uniform noise characteristics. Instead, there is a gradient along the pixel rows, with the noise generally higher away from the read out amplifier. This is likely another effect of the row-by-row bias issue discussed below. This gradient has the effect that the read noise increases as the row is read out. As a result of this increased noise, more sources are detected when the readnoise is assumed constant across the readout. To mitigate this noise gradient, we construct a set of noisemap images by measuring the median variance on bias frames. The variance is calculated in boxes of 20x20 pixels, and then linearly interpolated to cover the full image. - -Unfortunately, due to correlations in the row-to-row offsets \czwdraft{in the noise?}, the variance measured from the bias images does not fully remove the positional dependence of objects that are detected. The reason for this is that the simple noisemap underestimates the noise observed when the image is filtered during the object detection process. This filtering convolves the background noise with a PSF, which has the effect of amplifying the correlated peaks in the noise. This amplification can therefore boost background fluctuations above the threshold used to select real objects, contaminating the final object catalogs. - -To resolve this issue, we chose a PSF with a typical FWHM, and used it to look for detections on a sample of bias images. As the bias has no real sources, all objects found are by definition false, which provides an idea of how much our noisemap estimation deviates from the ``true'' noise observed by the object detection process. For a region of area $X*Y$, if we find $k$ false detections above our signal-to-noise threshold $sigma_{thresh}$, then we can estimate how much the noise model deviates from what is observed. The observed noise threshold is defined as $\sigma_{observed} = \sqrt{2} * \erfcinv{2 * k A_{psf} / (X * Y * N_{exp})}$, where $A_{psf}$ is the footprint size of the PSF (taken as 16 pixels), and $N_{exp}$ is the number of exposures examined in this location. From this observed threshold, we scale the noisemap previously calculated by the boost factor $B = \sigma_{thresh} / \sigma_{observed}$. - -The row-to-row variations that contribute to the extra noise are related to the dark model, and because of this, as the dark model changes, the effective noise also changes. To ensure that the noisemap accurately matches the true noise level, we have created different noisemap models for the three major time ranges of the dark model. We do not see any evidence that the noisemaps have the A/B modes visible in the dark, and so we do not generate different models for each individual dark model. - -\section{Remnance?} - -\czwdraft{Despite the known persistence effects of the detectors, we do not do any remnance correction beyond what is discussed above in the burntool section. Therefore, I probably should just remove this section entirely.} - -\section{Shutter?} - -\czwdraft{I don't believe that we do a shutter correction either. So, again, probably shouldn't include it.} - -\section{Flat} - -\czwdraft{I don't know how the flat calibration code works. We start with flat field images of the sky, but due to the size of the detector, it is difficult to equally illuminate each pixel. Therefore, flat calibration.} +\begin{figure} + \caption{Example of the dark switching gradients} + \label{fig:dark switching} +\end{figure} + +\subsubsection{Video Dark} +\label{sec:video_darks} + +The dark signal is stronger in cell corners due to glow from the +read-out amplifiers. The standard dark model corrects this for most +observations. However, as mentioned above, when a cell is repeatedly +read in video mode, the dark model for the OTA containing it changes. +Surprisingly, added reads for the video cell do not amplify the +amplifier glow, but rather decrease the dark signal in these regions. +As a result, using the standard dark model on the data for these OTAs +results in oversubtraction of the corner glow. + +Video darks have been constructed to eliminate the effect this +observational change has on the final image quality. This was done by +running the standard dark construction process on a series of dark +frames that have had the video signal enabled for some cells. GPC1 +can only run video signals on a subset of the OTAs at a given time. +This requires two passes to enable the video signal across the full +set of OTAs that support video cells. This is beneficial to the +process of creating darks, as those OTAs that do not have video +signals enabled create standard dark models, while the video dark is +created for the other devices. + +This simultaneous construction of video and standard dark models is +useful, as it provides the ability to isolate the response on the +standard dark from the video signals. Isolating this response is +essential for attempting to create archival video darks. We only have +raw video dark frame data after 2012-05-16, when this problem was +initially identified, so any data prior to that can not be directly +corrected for the video dark signal. Isolating the video signal +response allows linear corrections to the pre-existing standard dark +models for archival data. Testing this shows that constructing a +video dark for older data simply as $VD_{2009} = D_{2009} - D_{Modern} ++ VD_{Modern}$ produces a satisfactory result that does not +oversubtract the amplifier glow. This is shown in figure +\ref{fig:video_darks}, which shows video cells from before and after +2012-05-16, corrected with both the standard and video darks, with the +early video dark constructed in such a manner. + +\begin{figure} + \caption{Example of dark/video dark application} + \label{fig:video_darks} +\end{figure} + +\subsection{Noisemap} +\label{sec:noisemap} + +Based on a study of the positional dependence of all detected sources, +we have discovered that the cells in GPC1 do not have uniform noise +characteristics. Instead, there is a gradient along the pixel rows, +with the noise generally higher away from the read out amplifier +(higher cell x pixel positions). This is likely an effect of the +row-by-row bias issue discussed below. This gradient causes the read +noise to increase as the row is read out. As a result of this +increased noise, more sources are detected in the higher noise regions +when the read noise is assumed constant across the readout. To +mitigate this noise gradient, we constructed an initial set of +noisemap images by measuring the median variance on bias frames. The +variance is calculated in boxes of 20x20 pixels, and then linearly +interpolated to cover the full image. + +Unfortunately, due to correlations within this noise, the variance +measured from the bias images does not fully remove the positional +dependence of objects that are detected. The reason for this is that +this simple noisemap underestimates the noise observed when the image +is filtered during the object detection process. This filtering +convolves the background noise with a PSF, which has the effect of +amplifying the correlated peaks in the noise. This amplification can +therefore boost background fluctuations above the threshold used to +select real objects, contaminating the final object catalogs. + +In the detection process, we expect false positives at a rate equal to +the one-tailed probability beyond the detection threshold. For these +tests, only detections measured at the $\sigma_{thresh} = 5\sigma$ +level are used, to match that used in the photometry on science data. +This probability can be converted into a number of false number by +considereing a given area. As the detections must be isolated to not +be detected as an extended object, this area must be reduced by the +area a given PSF occupies. Combining this, we find that we expecte a +probability $P = 1 - \Phi_{normal}(5) = \frac{1}{2} +\erfcinv\left(\frac{5}{\sqrt{2}}\right)$, and an area given $N$ +exposures of area $X\times Y$, $A = \frac{X \times Y \times + N}{A_{PSF}}$. For a typical $1"$ seeing, $A_{PSF}$ is approximately +16 pixels. Using this model for the false positives, we found that +the added read noise was insufficient to account for the observed +false positive rate. Inverting this relation, we can measure +$\sigma_{obs}$, the true threshold level based on the number of false +positives observed. This $\sigma_{obs}$ is the combined to form a +boost factor $B = \sigma_{thresh} / \sigma_{obs}$ that amplifies the + noisemap to match the observed false detection rate. + +The row-to-row variations that contribute to the extra noise are +related to the dark model, and because of this, as the dark model +changes, the effective noise also changes. To ensure that the +noisemap accurately matches the true noise level, we have created +different noisemap models for the three major time ranges of the dark +model. We do not see any strong evidence that the noisemaps have the +A/B modes visible in the dark, and so we do not generate different +models for each individual dark model. The additional pixel-to-pixel +variance from this noisemap is added to the Poissionian variance to +form the science variance image generated by the \ippstage{chip} +processing. + +\subsection{Flat} Determining a flat field correction for GPC1 is a challenging @@ -706,65 +1019,89 @@ uniformly illuminated image. Using a dome screen is not possible, as the variations in illumination and screen rigidity create unusably -large scatter between different images that are caused by the detector response function. Because of this, we use sky -flat images taken at twilight, which are more consistently illuminated -than screen flats. We calculate the mean of these images to determine -the starting flat model. - -From this initial flat model, we construct a correction to remove the -effect of the problems illuminating the large area. This is done by -dithering a series of science exposures across a given pointing. By -comparing the measured fluxes for a given star as a function of -position on the detector, we can determine the position dependent -scaling factors. These scale factors can then be used to correct the -initial flat field model to better represent the detector response. +large scatter between different images that are not caused by the +detector response function. Because of this, we use sky flat images +taken at twilight, which are more consistently illuminated than screen +flats. We calculate the mean of these images to determine the +initial flat model. + +From this starting model, we construct a correction to remove the +effect of the illumination differences over the detector surface. +This is done by dithering a series of science exposures with a given +pointing. By fully calibrating these exposures with the initial flat +model, and then comparing the measured fluxes for the same star as a +function of position on the detector, we can determine position +dependent scaling factors. From the set of scaling factors for the +full catalog of stars observed in the dithered sequence, we can +construct a model of the error in the initial flat model as a function +of detector position. Applying a correction that reduces the +amplitude of these errors produces a flat field model that better +represents the true detector response. The flat model appears stable with time, although directly measuring this is as difficult as originally constructing the model. However, -due to the photometric consistency observed in the catalog of GPC1 measurements, we -can be confident that the flat model is not as time dependent as the -dark correction. - - -\section{Pattern correction} - -Due to the row-by-row bias offsets that are not cleanly removed by the +due to the photometric consistency observed in the final catalog of +GPC1 measurements \citep{MagnierXXX}, we can be confident that the +flat model does not have a major time dependent component. + +\subsection{Pattern correction} +\label{sec:pattern} + +Due to detector specific issues that are not cleanly removed by the dark model, we have a set of ``pattern'' corrections that are applied -to some selection of the images. The PATTERN.ROW correction is used -to remove the remaining row-by-row variation, and the PATTERN.CELL and -PATTERN.CONTINUITY corrections attempt to ensure that the cells of a -given OTA are consistent with the other cells on that OTA. These corrections are -largely designed to fix issues that are not stable enough with time -for the dark model or flat field model to fully account for the -detector behavior. - -\subsection{Pattern Row} +to some selection of the OTAs in the camera. This is done to reduce +the effect that detector differences that are not stable enough to be +corrected with a global model have on the measured astronomical +signal. Because these are not stable features that can simply be +averaged over a large number of inputs, the pattern corrections +attempt to identify and correct the detector issues based on +appropriate filtering the individual science exposures. + +The PATTERN.ROW correction is used to remove any remaining row-by-row +bias variation, and the PATTERN.CELL and PATTERN.CONTINUITY +corrections attempt to ensure that the cells of a given OTA are +consistent with the other cells on that OTA. + +\subsubsection{Pattern Row} % http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Bias_Pattern_Study As discussed above in the dark and noisemap sections, certain -detectors have significant row-by-row bias offsets. The magnitude of -these offsets increases as the distance from the readout amplifier -increases, resulting in horizontal streaks that are more pronounced -along one edge of the cell. As the level of the offset is largely -random, the dark correction cannot fully remove this structure from -the images, and the noisemap level only indicates the level of the -variance added by these bias offsets. Therefore, we apply the -PATTERN.ROW correction in an attempt to mitigate the offsets. To -force the rows to agree, a \czwdraft{second} order polynomial is fit to -each row in the cell, and that trend subtracted from the data. The -median offset (corresponding to the background level) for each row is then fit by a first order polynomial, and that trend is then added -back to the image so that the sky level on the cell matches its neighbors during -background subtraction. +detectors have significant row-by-row bias offsets, caused by noise in +the camera control electronics. The magnitude of these offsets +increases as the distance from the readout amplifier increases, +resulting in horizontal streaks that are more pronounced along the +large x pixel edge of the cell. As the level of the offset is +apparently random between exposures, the dark correction cannot fully +remove this structure from the images, and the noisemap value only +indicates the level of the average variance added by these bias +offsets. Therefore, we apply the PATTERN.ROW correction in an attempt +to mitigate the offsets and correct the image values. To force the +rows to agree, a second order clipped polynomial is fit to each row in +the cell. Four fit iterations are run, and pixels $2.5\sigma$ deiant +are excluded from subsequent fits, to minimize the effect stars and +other astronomical signals have. The final trend is then subtracted +from the image. Simply doing this subtraction will also have the +effect of removing the background sky level. To prevent this, the +constant and linear terms for each row are stored, and linear fits are +made to these parameters as a function of row. This produces a plane +that is added back to the image to restore the background offset and +any linear ramp that exists in the sky. + This correction was required on all cells on all OTAs prior to -\czwdraft{2009-12-01}, at which point a modification of the camera -electronics resolved the row-by-row offsets for the majority of the -detectors. As a result, we only apply this correction where it is -necessary, as shown in Figure \ref{fig: pattern row required}. - -Although this correction does resolve the row-by-row offset issue in a -satifactory way, large and bright astronomical objects can bias the -fit significantly. This results in an oversubtraction of the offset -near these objects. As the offsets are calculated on the pixel rows, -this oversubtraction is not uniform around the object, but is -preferentially along the horizontal x axis of the object. +2009-12-01, at which point a modification of the camera electronics +reduced the scale of the row-by-row offsets for the majority of the +OTAs. As a result, we only apply this correction to the cells where +it is still necessary, as shown in Figure \ref{fig: pattern row + cells}. A list of these cells is listed in Table +\ref{tab:pattern_row_cells}. + +Although this correction does largely resolve the row-by-row offset +issue in a satifactory way, large and bright astronomical objects can +bias the fit significantly. This results in an oversubtraction of the +offset near these objects. As the offsets are calculated on the pixel +rows, this oversubtraction is not uniform around the object, but is +preferentially along the horizontal x axis of the object. Most +astronomical objects are not significantly distorted by this, with +this only becoming on issue for only bright objects comparable to the +size of the cell (598 pixels = 150"). %% \czwdraft{keep this?} This row-by-row offset is visible in similar @@ -778,18 +1115,44 @@ %% FFT component visible. +\begin{deluxetable}{lcccc} + \tablecolumns{3} + \tablewidth{0pc} + \tablecaption{Cells which have PATTERN.ROW correction applied} + \tablehead{\colhead{OTA} & \colhead{Cell columns} & \colhead{Additional cells}} + \startdata + OTA11 & & xy02, xy03, xy04, xy07 \\ + OTA14 & & xy23 \\ + OTA15 & 0 & \\ + OTA27 & 0, 1, 2, 3, 7 & \\ + OTA31 & 7 & \\ + OTA32 & 3, 7 & \\ + OTA45 & 3, 7 & \\ + OTA47 & 0, 3, 5, 7 & \\ + OTA57 & 0, 1, 2, 6, 7 & \\ + OTA60 & & xy55 \\ + OTA74 & 2, 7 & \\ + \enddata + \label{tab:pattern_row_cells} +\end{deluxetable} + +\begin{figure} + \caption{Diagram illustrating which cells on GPC1 still require the PATTERN.ROW correction to be applied.} + \label{fig: pattern row cells} +\end{figure} + \begin{figure} \caption{Example of pre/post pattern row application.} \end{figure} -\subsection{Pattern Cell} - -As the bias level of a given cell may not exactly match that of its -neighbors, fitting a smooth background model results in over and -under-subtraction of the sky level at the cell boundary -discontinuities. The PATTERN.CELL correction was the first attempt to -remove this effect on the worst cells, by forcing all the cells of an -OTA to the same level. Each cell has the median value measured, and -then each cell has an offset added that shifts the cell to match the -median of those medians. +\subsubsection{Pattern Cell} + +As the measured background level of a given cell may not exactly match +that of its neighbors, fitting a smooth background model over the full +OTA can result in over and under-subtraction of the sky level at the +cell boundary discontinuities. The PATTERN.CELL correction was an +initial attempt to remove this effect on the worst cells, by forcing +all the cells of an OTA to the same level. Each cell had the median +value measured, and then each cell had an offset added that shifts the +cell to match the median of those medians. This correction is reasonable when the astronomical signal is smooth, @@ -799,66 +1162,107 @@ this issue, we no longer apply this correction to any data. -\subsection{Pattern Continuity} - -As the PATTERN.CELL correction was clearly insufficient in many -situations, we designed a replacement correction that would lower the -distortion for large objects less. In addition, studies of the -background level illustrated that the row-by-row bias introduces -small background gradient variations along the rows of the cells that is not stable enough to be completely fit by the dark model. This results -in a ``sawtooth'' pattern horizontally across an OTA, and as the background model -assumes a smooth sky level, this induces over and under -subtraction at cell boundaries. As the PATTERN.CELL was designed to -correct mean changes between cells, it could not adequately resolve +\subsubsection{Pattern Continuity} + +As the PATTERN.CELL correction was insufficient in many situations, we +designed a replacement correction that would reduce the background +distortion for large objects. In addition, studies of the background +level illustrated that the row-by-row bias can introduce small +background gradient variations along the rows of the cells that is not +stable enough to be completely fit by the dark model. This common +feature across the columns of cells results in a ``sawtooth'' pattern +horizontally across an OTA, and as the background model fits a smooth +sky level, this induces over and under subtraction at the cell +boundaries. As the PATTERN.CELL was designed to correct changes only +in the median value between cells, it could not adequately resolve this higher order issue. -The replacment for PATTERN.CELL was the PATTERN.CONTINUITY correction, +The replacment for PATTERN.CELL is the PATTERN.CONTINUITY correction, which attempts to match the edges of a cell to those of its neighbors. -For each cell, a thin box \czwdraft{10} pixels wide on each edge is extracted and the median -value calculated for that box. These median values are then used to -construct a vector of differences $diff_i = \sum_{j,j'} Edge_{i,j} - -Edge)_{i',j'}$, along with a matrix of associations $A_{i,i'} = -\sum_{j,j'} \delta(j,j')$ denoting which cell boundaries touch -another. By solving the system $A x = diff$, we can find the set of -offsets $x_i$ that should be applied to each cell to ensure the -minimum differences between cells. - -Due to the known slope in some cells, the effect of this correction is -to align the cells into a single ramp, at the expense of the absolute -background level. However, as we subtract off a smooth background -model, the deviations from an absolute sky level are unimportant. The fact that the final -ramp is smoother than it would be otherwise also allows for the -background subtracted image to more closely match the astronomical -sky, without over- and under-subtractions at cell edges. +For each cell, a thin box 10 pixels wide on each edge is extracted and +the median value of unmasked values calculated for that box. These +median values are then used to construct a vector of differences +$\Delta_i = \sum_{j} Edge_{i} - Edge_{j}$, along with a matrix of +associations $A_{i,i'} = \sum_{j} \delta(i,j) \delta(j,i')$ denoting +which cell boundaries touch another. By solving the system $A x = +diff$, we find the set of offsets $x_i$ to be applied to each cell to +ensure the minimum differences between all cell edges and their +neighbors. + +For OTAs that initially show the sawtooth pattern, the effect of this +correction is to align the cells into a single ramp, at the expense of +the absolute background level. However, as we subtract off a smooth +background model prior to doing photometry, these deviations from an +absolute sky level are unimportant. The fact that the final ramp is +smoother than it would be otherwise also allows for the background +subtracted image to more closely match the astronomical sky, without +significant errors at cell boundaries. An example of the image before +and after this correction is shown in figure \ref{fig: continuity + example}. \begin{figure} \caption{Continuity example, with background issue.} + \label{fig: continuity example} \end{figure} -\section{Fringe correction} +\subsection{Fringe correction} +\label{sec:fringe} +% det_id 296 is the fringe we use. + +\czwdraft{This is still a mess} Due to variations in the thickness of the detectors, we observe -interference patterns at the infrared end of the filter set, as -the wavelength of the light becomes comparable to the thickness of these variations. -Visually inspecting the images shows that the fringing is most -prevalent in the y-filter images, with minimal fringing in other -bands. As a result of this, we only apply a fringe correction to the y filter -data. - -The fringe is constructed by randomly determining a set of boxes for -each OTA cell, and measuring the sky subtracted median value in those -boxes for a series of images. These samples are selected at the same -location on each image, allowing the astronomical signal to be -filtered out as an additional noise term. A least squares fit to the -data is then calculated, providing the model of the fringe strength at -that location. - -Applying the fringe is done in the same way, with samples measured -across the image to determine the relative strength of the fringing in -this image. The solution derived from the detrend is then scaled to -match that observed in the science image, and subtracted away. - -\section{Background subtraction} - -\czwdraft{A background model is generated for each OTA, once all the individual cells have been mosaicked together. Super-pixels are then defined that divide the image into XxY subregions, and the mean calculated for each subregion. This grid is shifted by a half-width, and the means recalculated, to double the sampling frequency. A background model is then calculated by interpolating over this sampled grid.} +interference patterns at the infrared end of the filter set, as the +wavelength of the light becomes comparable to the thickness of these +variations. Visually inspecting the images shows that the fringing is +most prevalent in the y-filter images, with negligible fringing in +other bands. As a result of this, we only apply a fringe correction +to the y filter data. + +The fringe used for PV3 processing was constructed from a set of 20 +120s science exposures. These exposures are overscan subtracted, and +corrected for non-linearity, and have the dark and flat models +applied. These images are smoothed with a Gaussian of $\sigma = 2$ +pixels to minimize pixel to pixel noise. The fringe image data is +then constructed by calculating the clipped mean of the input images +with two iteration of clipping at the $3\sigma$ level. + +A course background model is constructed by calculating the median on +a 3x3 grid (200x200 pixels each). A set of 1000 randomly selected +points are selected on \czwdraft{the final image} in each cell, and +median calculated for this position in a 10x10 pixel box, and the +background level subtracted. These sample locations provide scale +points to allow the amplitude of the measured fringe to be compared to +that found on science images. + +To apply the fringe, the same sample locations are measured on science +image to determine the relative strength of the fringing in that +particular image. A least squares fit between the fringe measurements +and the corresponding measurements on the science provides the scale +factor multiplied by the fringe before it is subtracted from the +science image. + +\begin{figure} + \caption{Example of y-filter fringe pattern, before and after correction.} + \label{fig: fringe example} +\end{figure} + +\subsection{Background subtraction} +\label{sec:background} + + +Once all other detrending is done, the pixels from each cell are +mosaicked into the full $4846\times{}4868$ pixel OTA image. A +background model for the full OTA is then determined prior to the +photometric analysis. The mosaicked image is binned into +$800\times{}800$ pixel bins, centered on the image center, and +overlapping by a factor of 2 in both axes. These bins have 10000 +random samples drawn, and a binned cumulative distribution function is +generated. These bins are interpolated to find the best mean value at +the $50\%$ level, as well as the distribution $\sigma$ by estimating +from the $32\%$ and $68\%$ levels. Repeating this across all bins +results in a $13\times{}13$ grid of background bins, which are +bilinearly interpolated to generate the background model to subtract. +Each object in the photometric catalog has a SKY and SKY\_SIGMA value +based on this model as well. %% * Magic @@ -876,15 +1280,18 @@ \section{Warping} - +\label{sec:warping} To provide a consistent and uniform set of images for co-added image -stacking and image differences, the individual mosaicked OTA images -are projected onto a common set of tangent plane projected regions. -These projection cells are $4\times{}4$ degree fields spaced onto set -of projection centers that fully cover the sky. These projection -cells are arranged into rings of constant declination, and allowed to -overlap as $|\delta|$ increases. Each projection cell is further -subdivided into \czwdraft{size} sky cells, which have constant overlap -regions of \czwdraft{overlap}. These skycells are the main image unit -used for processing image data beyond the initial chip stage. +stacking and differences, the individual mosaicked OTA images are +projected onto a common set of tangent plane projected regions called +projection cells. These projection cells are $4\times{}4$ degree +fields spaced onto set of centers that fully cover the sky. They are +arranged into rings of constant declination, and allowed to overlap as +$|\delta|$ increases. Each projection cell is further subdivided into +$10\times{}10$ sky cells with fixed $0.25"$ resolution pixels, with +constant overlap regions between adjacent skycells of $60"$. These +skycells are the main image unit used for processing image data beyond +the initial chip stage. The coordinate system used for these images +matches the parity of the sky, with north in the positive y direction +and east to the negative x direction. After the detrending and photometry, the detection catalog for the @@ -896,5 +1303,5 @@ Foreach output skycell, all overlapping OTAs and the calibrated -catalog are read into the \textbf{pswarp} program. Each input image +catalog are read into the \ippprog{pswarp} program. Each input image is examined in order, and the same transformation performed. This transformation breaks the output warp image into $128\times{}128$ @@ -914,7 +1321,10 @@ pixel. This process is repeated for all grid boxes, for all input images, and for each output image product: the science image, the -variance, and the mask. \czwdraft{The jacobian is multiplied to the - image value, and squared and multiplied to the variance. I don't - understand that.} +variance, and the mask. The image values are scaled by the absolute +value of the Jacobian determinant of the transformation. This +corrects the pixel values for the possible change in pixel area due to +the transformation. Similarly, the variance image is scaled by the +square of this value, again to correctly account for the pixel area +change. As the interpolation constructs the output pixels from more than one @@ -926,5 +1336,5 @@ An output catalog is also constructed from the full exposure input -catalog, including only those objects that fall on the warped image. +catalog, including only those objects that fall on the new warped image. These detections are transformed to match the new image location, and to scale the position errors based on the new orientation. @@ -958,42 +1368,50 @@ \section{Stacking} +\label{sec:stacking} Once individual exposures have been warped onto a common projection -system, they can then be combined without that added concern. In -order to obtain detections of faint images, and to provide a static -sky image without transient features, we coadd the individual warps -into a stacked image. Creating this stack also allows a complete -image to be constructed that does not have regions masked due to -falling between devices. +system, they can then be combined pixel-by-pixel regardless of their +original orientation. Creating a stacked image by coadding the +individual warps increases the signal to noise which allows objects +fainter than can be found on the individual inputs to be detected. +Creating this stack also allows a complete image to be constructed +that does not have regions masked due to the gaps between cells and +OTAs. This provides a fully populated static sky image that can +be used for subtraction to find transient sources. The stacked image is comprised of all warp frames for a given skycell -in a single filter. The source catalogs and images are loaded into -the \textbf{ppStack} program to do prepare the inputs and stack the -frames while rejecting bad pixels. +in a single filter. The source catalogs and image components are +loaded into the \ippprog{ppStack} program to prepare the inputs and +stack the frames. Once all files are ingested, the first step is to measure the size and shapes of the input image PSFs. We exclude images that have a PSF -FWHM greater than 10 pixels, as those images have the worst seeing and -would degrade the final output stack. A target PSF for the stack is -constructed from the envelope of all input PSFs, which sets the target -PSF at the largest value among the input PSFs for all radii. This PSF -is then circularized to prevent any of the input images from being -deconvolved when matched to the target. - -The input images also need to be normalized to prevent differences in -seeing and sky transparency from causing discrepancies during pixel -rejection. From the calibrated input catalogs, we have the -instrumental magnitudes of all sources, along with the airmass, image -exposure time, and zeropoint. All output stacks are calibrated to a -zeropoint of 25.0 in all filters, and to have an airmass of 1.0. The -output exposure time is set as the sum of the input exposure times. -With this information, we can determine the relative transparency for -each input image by comparing matched sources between the different -images. Each image then has a normalization factor defined, equal to -$norm_{image} = (ZP_{image} - ZP_{target}) - transparency_{image} - -2.5 * \log_{10} (t_{target} / t_{image}) - airmassTerm * -(airmass_{image} - airmass_{target})$. The input source catalog is -adjusted to reflect this normalization, which is also retained for -application when the pixels are combined. +FWHM greater than 10 pixels, as those images have the seeing far worse +than average, and would degrade the final output stack. For the PV3 +survey, this size represents a PSF larger than $97$th percentile in +all filters. A target PSF for the stack is constructed by finding the +maximum envelope of all input PSFs, which sets the target PSF to the +largest value among the input PSFs for a given position from the peak. +This PSF is then circularized to ensure azimuthal symmetry, which +prevents any of the input images from being deconvolved when matched +to the target. + +The input images also need to have their flux normalized to prevent +differences in seeing and sky transparency from causing discrepancies +during pixel rejection. From the calibrated input catalogs, we have +the instrumental magnitudes of all sources, along with the airmass, +image exposure time, and zeropoint. All output stacks are calibrated +to a zeropoint of 25.0 in all filters, and to have an airmass of 1.0. +The output exposure time is set to the sum of the input exposure +times. We can determine the relative transparency for each input +image by comparing the magnitudes of matched sources between the +different images. Each image then has a normalization factor defined, +equal to $norm_{i} = (ZP_{i} - ZP_{target}) - transparency_{i} - 2.5 * +\log_{10} (t_{target} / t_{i}) - airmassTerm * (airmass_{i} - +airmass_{target})$. \czwdraft{ZP.AIRMASS is zero for all filters. + Does this simply mean that we assume any airmass differences are + folded into the transparency differences? This would simplify this + discussion quite a bit if that's the case, as we can just say that + and skip all the extra airmass terms.} % PREPARE @@ -1030,11 +1448,12 @@ % // m_inst_o - m_inst_i = zp[i] - zpTarget - c1 * airmassTarget - 2.5log(sumExpTime) - trans_i -With the normalization factors and target PSF chosen, the convolution -kernels can be calculated for each image. ISIS kernels are used with -FWHM values of 1.5, 3.0, and 6.0 pixels and polynomial orders of 6, 4, -and 2. \czwdraft{Skipping this bit because I'm not completely sure I - understand it.} The image is then scaled by the normalization as -$renorm = 10^{-0.4 * norm_{image}} / norm_{convolution}$, and the - variance by the square of that value. +With the flux normalization factors and target PSF chosen, the +convolution kernels can be calculated for each image. ISIS kernels +are used with FWHM values of 1.5, 3.0, and 6.0 pixels and polynomial +orders of 6, 4, and 2. \czwdraft{Skipping this bit because I'm not + completely sure I understand it.} The image is then scaled by the +normalization as $renorm = 10^{-0.4 * norm_{image}} / +norm_{convolution}$, and the variance by the square of that value. + % MATCH @@ -1047,6 +1466,11 @@ Once the convolution kernels are defind for each image, they are used to convolve the image to match the target PSF. Any input image that -has a $\chi^2$ value larger than 4.0$\sigma$ larger than the median -value is rejected from the stack. +has a $\chi^2$ value greater than 4.0$\sigma$ larger than the median +value is rejected from the stack. Each image also has a weight +assigned, based on the image variance after convolution. For a given +image, the weight is equal to $W^{-1} = \langle Variance(x,y) \rangle +* f_{covariance}$, where the angle brackets denote a robust median of +the variance image, and the covariance factor $f_{covariance}$ is the +peak value of the covariance matrix of the convolution. % CONVOLVE @@ -1057,5 +1481,5 @@ % CovarianceFactor = covariance->kernel[0][0] -Following the convolution, and initial stack is constructed. For a +Following the convolution, an initial stack is constructed. For a given pixel coordinate, the values at that coordinate are extracted from all input images. Images that have a suspect mask bit (including @@ -1063,20 +1487,19 @@ values) are appended to a suspect pixel list for preferential exclusion. Following this, the pixel values are combined and tested -to attempt to identify discrepant values that should be excluded. +to attempt to identify discrepant input values that should be excluded. If only a single input is available, the initial stack contains the value from that single input. If there are only two inputs, the average of the two is used. These cases should occur only rarely in -the $3\Pi$ survey, as there are many input exposures that overlap any -particular point on the sky. The more common case for three or more -inputs constructs a weighted average from the inputs, with the weight -set as a single value for each input image, and defined as the inverse -of the median variance value from that image's associated variance -map. This weight is used for the image and the exposure weighted +the $3\Pi$ survey, as there are many input exposures that overlap each +point on the sky. For the more common case of three or more inputs, a +weighted average from the inputs is used, with the weight for each +image as defined above used for all pixels from that input image. +This weight is used for both the image and the exposure weighted image: \begin{eqnarray} - S_{value} &=& \sum_i\left(value_{i} * weight_i\right) / \sum_i\left(weight_i\right) \\ - S_{exp weight} &=& \sum_i \left(exptime_i * weight_i\right) / \sum_i\left(weight_i\right) \\ + S_{value} &=& \sum_i\left(value_{i} * W_i\right) / \sum_i\left(W_i\right) \\ + S_{exp weight} &=& \sum_i \left(exptime_i * W_i\right) / \sum_i\left(W_i\right) \\ \end{eqnarray} @@ -1140,74 +1563,71 @@ %% As described above. +Due to the various non-astronomical ghosts that can occur on GPC1, and +the fact that they may not be fully masked to ensure all bad pixels +are removed, it is expected that some of the inputs for a given stack +pixel are not in agreement with the others. In general, there is the +population of input pixel values around the correct astronomical +level, as well as possible populations at lower pixel value (such as +due to an over-subtracted burntool trail) and at higher pixel values +(such as that caused by an incompletely masked optical ghost). Due to +the observation strategy to image a given field twice to allow for +warp-warp difference images to be constructed to identify transient +detections, higher pixel values that come from sources like optical +ghosts depend on the telescope pointing will come in pairs as well. +The higher pixel value contaminants are also potentially problematic +as they may appear to be real sources, prompting photometry to be +performed on false objects. Because of the expectation that there are +more bright contaminants than faint ones, there is a slight preference +to reject higher pixel values than lower pixel values. + Following this initial combination, a ``testing'' loop iterates in an attempt to identify outlier points. Again, if only one input is available, that input is accepted. If there are two inputs, $A$ and -$B$, then a check is made to see if $(0.5 * (value_A - value_B))**2 > -rej**2 * (variance_A + variance_B + (sys * value_A)**2 + (sys * -value_B)**2)$, where $rej$ is the number of sigma deviant a point -needs to be to be excluded, set to 4.0 for the PV3 processing, and -$sys$ is an estimate of the systematic error, taken to be 0.1. - - -\czwdraft{This discussion seems out of place, but I'm not sure where a - better place is.} Due to the various non-astronomical ghosts that -can occur on GPC1, and the fact that they may not be masked -aggressively enough to ensure all bad pixels are removed, it is -expected that some of the inputs for a given stack pixel are not in -agreement with the others. In general, there is the population of -input pixel values around the correct astronomical level, as well as -possible populations at lower pixel value (such as due to an -over-subtracted burntool trail) and at higher pixel values (such as -that caused by an incompletely masked optical ghost). Due to the -observation strategy to image a given field twice to allow for -warp-warp difference images to be constructed to identify transient -detections, higher pixel values that come from sources like optical -ghosts that are a function of pointing will come in pairs as well. -The higher pixel value contaminants are also potentially problematic -as they may appear to be a real source, prompting photometry to be -performed on a false object. Because of these reasons, there is a -slight preference to reject higher pixel values than lower pixel -values. +$B$, then a check is made to see if $(0.5 * (value_A - value_B))^2 > +rej^2 * (variance_A + variance_B + (sys * value_A)^2 + (sys * +value_B)^2)$, where $rej$ is the number of sigma deviant a point needs +to be to be excluded, set to 4.0 for the PV3 processing, and $sys$ is +an estimate of the systematic error, taken to be 0.1. If the number of inputs is larger than 6, then a Gaussian mixture model analysis is run on the inputs to fit two sub populations, and determine an the likelihood that the distribution is best described by -an uni-modal model. If this probability is less than 0.05, then the +an uni-modal model. If this probability is less than $5\%$, then the mean is taken from the bimodal sub population with the largest fraction of inputs, as this should exclude any sub population comprised of high pixel value outliers. -If this is not the case (the distribution is likely unimodal) or if -there are insufficient inputs for the mixture model analysis, the +If this is not the case, and the distribution is likely unimodal, or +if there are insufficient inputs for this mixture model analysis, the input values are passed to an Olympic weighted mean calculation. We -set 0.2 as the fraction of the number of inputs to reject through this -process. This sets the number of bad inputs at $N_{bad} = 0.2 * -N_{input} + 0.5$, where the 0.5 term ensures at least one input is -rejected. This number is further separated into the number of low -values to exclude $N_{low} = N_{bad} / 2$, which will default to zero -if there are few inputs due to integer arithmatic, and $N_{high} = -N_{input} + N_{low} - N_{bad}$. After sorting the input values to -determine which values fall into the low and high groups, the -remaining input values have a weighted mean calculated as described -above. +reject $20\%$ of the number of inputs through this process. The +number of bad inputs is set to $N_{bad} = 0.2 * N_{input} + 0.5$, with +the 0.5 term ensuring at least one input is rejected. This number is +further separated into the number of low values to exclude $N_{low} = +N_{bad} / 2$, which will default to zero if there are few inputs, and +$N_{high} = N_{input} + N_{low} - N_{bad}$. After sorting the input +values to determine which values fall into the low and high groups, +the remaining input values are used in a weighted mean using the image +weights above. A systematic variance term is necessary to correctly scale how discrepant points can be from the ensemble mean. If the mixture model analysis was run, the Gaussian sigma from the largest sub population -is squared and used. If this is not available, a 0.1 scaling on the -input values is used. Each point then has a limit calculated: +is squared and used. If this is not available, a $10\%$ systematic +error on the input values is used. Each point then has a limit +calculated using a $4\sigma$ rejection \begin{eqnarray} - limit_{mixture_model} &=& rej**2 * (variance_i + \sigma_{MM}^2) \\ - limit_{default} &=& rej**2 * (variance_i + (0.1 * value_i)**2) + limit_{mixture model} &=& 4^2 * (variance_i + \sigma_{MM}^2) \\ + limit_{default} &=& 4^2 * (variance_i + (0.1 * value_i)^2) \end{eqnarray} -where $rej$ is the same factor of 4.0 used above. Each input pixel is -then compared against this limit, and the most discrepant pixel that -has $(value_i - mean)**2$ exceeding this limit is identified. If -there are suspect pixels in the set those pixels are marked for -rejection, otherwise this worst pixel is marked for rejection. -Following this, the combine and test loop is repeated for a total $0.5 -N_{input}$ iterations, or until no more pixels are rejected. +Each input pixel is then compared against this limit, and the most +discrepant pixel that has $(value_i - mean)^2$ exceeding this limit is +identified. If there are suspect pixels in the set those pixels are +marked for rejection, otherwise this worst pixel is marked for +rejection. Following this, the combine and test loop is repeated for +until no more pixels are rejected, up to a maximum number of +iterations equal to $50\%$ of the number of inputs. % combineTest @@ -1245,6 +1665,6 @@ is made by constructing an empty image that has the rejected pixels set to a value of 1.0. This image is then convolved with a 5 pixel -FWHM 0-order ISIS kernel. Any pixels that are above the threshold of -0.5 are marked as bad and will be rejected in the final convolution. +FWHM zeroth-order ISIS kernel. Any pixels that are above the threshold of +0.5 after this mask convolution are marked as bad and will be rejected in the final combination. If more than 10\% of all pixels from an input image are rejected, then that entire image is rejected as well. @@ -1263,19 +1683,12 @@ pixels. The ISIS kernel used in the previous step is used to determine the largest square box that contains under the limit of -$0.25 * \sum_{x,y} kernel**2$. This box is then convolved with the -rejected pixel mask to reject their neighbors. - -This final list of rejected pixels is passed to the final combination -pass, which does not iterate, and simply excludes the rejected -pixels. \czwdraft{This is a bad paragraph.} - -\czwdraft{We make the stacked image, the stacked variance, the stack - mask, the exposure time mask, the exp weight containing the weighted - exposure times, and a number image, containing the number of inputs - used for each pixel.} - - - - +$0.25 * \sum_{x,y} kernel^2$. This box is then convolved with the +rejected pixel mask to reject their neighbors. This final list of +rejected pixels is passed to the final combination, which creates the +final stack values from the weighted mean of the non-rejected pixels. +Six total images are constructed for this final stack: the image, its +variance, a mask, a map of the exposure time per pixel, that exposure +time map weighted by the input image weight, and a map of the number +of inputs per pixel. % FINAL COMBINE @@ -1294,6 +1707,5 @@ % combineSys, combineDiscard, useVariance, safe, nminpix, rejected)) { - -The convolved stack products are not retained, as the convolution +These convolved stack products are not retained, as the convolution reduces the resolution of the final image. Instead, we apply the normalizations and rejected pixel maps generated from the convolved @@ -1301,7 +1713,6 @@ an unconvolved stack that has the optimum image quality possible from the input images. Not convolving does mean that the PSF shape changes -somewhat across the image, as the different FWHM of the input images -print through in the different regions in which they have contributed -to the final image. +across the image, as the different PSF widths of the input images +print through in the different regions to which they have contributed. % UNCONVOLVED IMAGE @@ -1311,18 +1722,46 @@ % only retain unconvolved products. - -One benefit of producing the final stacked image from the weighted -mean of the unrejected input images is that faint sources do not have -their contribution removed as much as median filtering would allow. -\czwdraft{I did something to prove this once, but can't find it right - now. Comparing the ppStack output catalog to one constructed from a - simple median filtered stack shows that the ppStack catalog detects - sources up to 0.XX magnitudes fainter than the median stack. This - does increase the possibility of false positives.} - +%% Asinh compression + +Due to the expected large range of data values in the final stacked +image, saving them as compressed 16-bit integer images with linear +BSCALE and BZERO scaling values is likely to offer poor +reconstructions of the stacked image. This will lead either to +truncation of the extrema of the image, or quantized values that are +poorly spaced for the image histogram. Saving the images as 32-bit +floating point values would alleviate this quantization issue, at the +cost of a large increase in the disk space required for the stacked +images. + +Transforming the data prior to writing to disk by taking the logarithm +of the pixel values can resolve this, with the complication that all +data values must first be made positive, which then sets the highest +quantization sampling near the lowest values in the image. Following +techniques used by SDSS \citep{sdss}, we have instead opted to use the +inverse hyperbolic sine function to transform the data. The domain of +this function allows any input value to be converted. In addition, +the quantization sampling can be tuned by placing the zero of the +inverse hyperbolic sine function at a value where the highest sampling +is desired. + +Formally, prior to being written to disk, the pixel values are +transformed by $C = \alpha \asinh\left(\frac{L - \mathrm{BOFFSET}}{2.0 + \cdot \mathrm{BSOFTEN}}\right)$, where $L$ are the linear input +pixel values, $C$ the transformed values, $\alpha = 2.5 \log_{10}(e)$. +BOFFSET centers the transformed values, and the mean of the linear +input pixel values is used. BSOFTEN controls the stretch of the +transformation, and is set to $\sqrt{\alpha} \sigma_{L}$. These +parameters are saved to the output image header. The image is then +passed to the standard BSCALE and BZERO calculation and saved to disk. + +To reverse this process (on subsequent reads of the image, for example +in warp-stack difference calculations), the BOFFSET and BSOFTEN +parameters are read from the header and the transformation inverted, +such that: $L = \mathrm{BOFFSET} + \mathrm{BSOFTEN} \cdot \left(\exp(C +/ \alpha) - \exp(-C / \alpha)\right)$. \section{Discussion} - -\section{Conclusion} +\label{sec:discussion} + \end{document} @@ -1330,2 +1769,7 @@ % http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Detrend_Documentation +% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Detrend_Documentation#Currentdetrends +% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/stacking_coverage.20130307 +% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/staticsky.20120706_excess_detections +% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Stack_Rejection_Discussion +% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Stack_Algorithm