Index: trunk/doc/release.2015/ps1.detrend/detrend.tex =================================================================== --- trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 39849) +++ trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 39850) @@ -48,38 +48,42 @@ % list and (2) re-order the list at the bottom (and comment-out as needed) \def\IfA{1} -\def\CfA{2} -\def\MPIA{3} -\def\Princeton{3} -\def\USNO{4} -\def\JHU{1} +\def\Princeton{2} +\def\STSCI{3} +\def\Pitt{4} +%\def\CfA{2} +%\def\MPIA{3} +%\def\USNO{4} +%\def\JHU{1} % This example has a first author from UH: \author{ -C. Z. Waters,\altaffilmark{\IfA} -IPP Team, +C.~Z. Waters,\altaffilmark{\IfA} +E.~A. Magnier,\altaffilmark{\IfA} +P.~A. Price,\altaffilmark{\Princeton} +H.~A. Flewelling,\altaffilmark{\IfA} +M.~E. Huber,\altaffilmark{\IfA} +W.~E. Sweeney,\altaffilmark{\IfA} +J.~L. Tonry, \altaffilmark{\IfA} +K.~C. Chambers,\altaffilmark{\IfA} +R.~H. Lupton,\altaffilmark{\Princeton} +A. Rest,\altaffilmark{\STSCI} +W.~M. Wood-Vasey,\altaffilmark{\Pitt} +PS1 Builders %PS Builder List % W.~S. Burgett,\altaffilmark{\IfA} -% K.~C. Chambers,\altaffilmark{\IfA} % L. Denneau,\altaffilmark{\IfA} % P. Draper,\altaffilmark{\DUR} -% H.~A. Flewelling,\altaffilmark{\IfA} % T. Grav,\altaffilmark{\IfA} % J. N. Heasley,\altaffilmark{\IfA} % K. W. Hodapp,\altaffilmark{\IfA} -% M. E. Huber,\altaffilmark{\IfA} % R. Jedicke,\altaffilmark{\IfA} % N. Kaiser,\altaffilmark{\IfA} % R.-P. Kudritzki,\altaffilmark{\IfA} % G. A. Luppino,\altaffilmark{\IfA} -% R. H. Lupton,\altaffilmark{\Princeton} -% E. A. Magnier,\altaffilmark{\IfA} % N. Metcalfe,\altaffilmark{\DUH} % D. G. Monet,\altaffilmark{\USNO} % J.~S. Morgan,\altaffilmark{\IfA} % P. M. Onaka,\altaffilmark{\IfA} -% P.~A. Price,\altaffilmark{\Princeton} % C.~W. Stubbs,\altaffilmark{\CfA} -% W.~E. Sweeney,\altaffilmark{\IfA} -% J.~L. Tonry, \altaffilmark{\IfA} % R. J. Wainscoat,\altaffilmark{\IfA} and % C. Z. Waters,\altaffilmark{\IfA} @@ -89,5 +93,7 @@ \altaffiltext{\IfA}{Institute for Astronomy, University of Hawaii, 2680 Woodlawn Drive, Honolulu HI 96822} % \altaffiltext{\CfA}{Harvard-Smithsonian Center for Astrophysics, 60 Garden Street, Cambridge, MA 02138} -% \altaffiltext{\Princeton}{Department of Astrophysical Sciences, Princeton University, Princeton, NJ 08544, USA} +\altaffiltext{\Princeton}{Department of Astrophysical Sciences, Princeton University, Princeton, NJ 08544, USA} +\altaffiltext{\STSCI}{Space Telescope Science Institute, 3700 San Martin Drive, Baltimore, MD 21218, USA} +\altaffiltext{\Pitt}{Department of Physics and Astronomy, University of Pittsburgh, Pittsburgh, PA 15260, USA} % \altaffiltext{\USNO}{US Naval Observatory, Flagstaff Station, Flagstaff, AZ 86001, USA} % \altaffiltext{\JHU}{Department of Physics and Astronomy, Johns Hopkins University, 3400 North Charles Street, Baltimore, MD 21218, USA} @@ -95,13 +101,16 @@ \begin{abstract} -Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum -bibendum nisi id tristique posuere. Duis eu mollis nulla. Maecenas est -turpis, mattis tempor urna vitae, placerat rhoncus sem. Lorem ipsum -dolor sit amet, consectetur adipiscing elit. Sed quis velit -nisl. Aliquam erat volutpat. Cras lacinia, nisl tristique auctor -molestie, dolor nulla rhoncus purus, ac accumsan nunc nunc ac -nibh. Maecenas vitae mollis mauris. Ut sollicitudin pulvinar purus, -eget luctus lorem tincidunt vitae. Vestibulum eu mattis neque. Nulla -in tortor id urna dapibus gravida a vel leo. +the Pan-STARRS1 Science Consortium have carried out a set of imaging +surveys using the 1.4 giga-pixel GPC1 camera on the PS1 telescope. As +this camera is composed of many individual electronic readouts, and +covers a very large field of view, great care was taken to ensure that +the many instrumental effects were corrected to produce the most +uniform detector response possible. We present the image detrending +steps used as part of the processing of the data contained within the +public release of the Pan-STARRS1 Data Release 1 (DR1). In addition +to the single image processing, the methods used to transform the +375,573 individual exposures into a common sky-oriented grid are +discussed, as well as those used to produce both the image stack and +difference combination products. \end{abstract} @@ -131,13 +140,13 @@ Pan-STARRS 1 Science Consortium members. -\czwdraft{Nigel: you mention calibrating to the reference catalog without telling us -what this is composed of (maybe this is in a different section, but would be -nice to have some idea here).} - -\czwdraft{Can we get around this point by simply adding a reference to - the paper describing the reference catalog? It's not really part of - the detrending process, and is discussed here mostly to give an - overview of the stages, and later to find sources of ghosts for - masking.} +%% \czwdraft{Nigel: you mention calibrating to the reference catalog without telling us +%% what this is composed of (maybe this is in a different section, but would be +%% nice to have some idea here).} + +%% \czwdraft{Can we get around this point by simply adding a reference to +%% the paper describing the reference catalog? It's not really part of +%% the detrending process, and is discussed here mostly to give an +%% overview of the stages, and later to find sources of ghosts for +%% masking.} The Pan-STARRS image processing pipeline (IPP) is described elsewhere @@ -167,13 +176,13 @@ identified in the \ippstage{diff} stage, which takes input \ippstage{warp} and/or \ippstage{stack} data and performs image -differencing \citep{HuberXXX}. Further photometry is performed in the -\ippstage{staticsky} and \ippstage{skycal} stages, which add extended -source fitting to the point source photometry of objects detected in -the \ippstage{stack} images, and calibrate the results against the -reference catalog. The \ippstage{fullforce} stage takes the catalog -output of the \ippstage{skycal} stage, and uses the objects detected -in that to perform forced photometry on the individual \ippstage{warp} -stage images. The details of these stages are provided in -\citet{MagnierXXY}. +differencing (Section \ref{sec:diffs}). Further photometry is +performed in the \ippstage{staticsky} and \ippstage{skycal} stages, +which add extended source fitting to the point source photometry of +objects detected in the \ippstage{stack} images, and calibrate the +results against the reference catalog. The \ippstage{fullforce} stage +takes the catalog output of the \ippstage{skycal} stage, and uses the +objects detected in that to perform forced photometry on the +individual \ippstage{warp} stage images. The details of these stages +are provided in \citet{MagnierXXY}. The same reduction procedure described above is also performed in real @@ -192,11 +201,11 @@ details of the construction of those detrends in Section \ref{sec:detrend construction}. An analysis of the algorithms used to -complete the \ippstage{warp} (section \ref{sec:warping}) and -\ippstage{stack} (section \ref{sec:stacking}) stage transformations of -the image data to from the detector frame to a common sky frame, and -the co-adding of those common sky frame images continues after the -list of detrend steps. Finally, a discussion of the remaining issues -and possible future improvements is presented in section -\ref{sec:discussion}. +complete the \ippstage{warp} (section \ref{sec:warping}), +\ippstage{stack} (section \ref{sec:stacking}), and \ippstage{diff} +(section \ref{sec:diffs}) stage transformations of the image data to +from the detector frame to a common sky frame, and the co-adding of +those common sky frame images continues after the list of detrend +steps. Finally, a discussion of the remaining issues and possible +future improvements is presented in section \ref{sec:discussion}. Image products presented in figures have been @@ -218,4 +227,12 @@ and pixel $(1,1)$ to the top right of their position. +% Note taken verbatim from Ken's Paper 1. +\textit{Note: These papera are being placed on the arXiv.org to + provide crucial support information at the time of the public + release of Data Release 1 (DR1). We expect the arXiv versions to be + updated prior to submission and there could be significant + variations with the refereed papers. We apologize for the + inconvience.} + % Discuss 2-phase/3-phase device differnces @@ -227,8 +244,8 @@ \label{sec:detrending} -\czwdraft{Nigel: I forgot: when we are talking about the various bias corrections it might be -worth pointing out that we expect these to be more of an issue in the g-band -(and maybe r?) where read noise is a significant contributor. -} +%% \czwdraft{Nigel: I forgot: when we are talking about the various bias corrections it might be +%% worth pointing out that we expect these to be more of an issue in the g-band +%% (and maybe r?) where read noise is a significant contributor. +%% } Ensuring a consistent and uniform detector response across the @@ -727,10 +744,10 @@ better represents the true detector response. -\czwdraft{EAM: the flat-field construction part needs to make a clearer discussion of -the skyflat vs the photometric correction (photflat) built initially for -the survey vs the flat-field corrections determined in the database as part -of ubercal (for the latter, you should just mention the concept -- it will -also be mentioned in the calibration paper). The statement that the -flat-field response was stable is not true since we did need 5 'seasons'.} +%% \czwdraft{EAM: the flat-field construction part needs to make a clearer discussion of +%% the skyflat vs the photometric correction (photflat) built initially for +%% the survey vs the flat-field corrections determined in the database as part +%% of ubercal (for the latter, you should just mention the concept -- it will +%% also be mentioned in the calibration paper). The statement that the +%% flat-field response was stable is not true since we did need 5 'seasons'.} In addition to this flat field applied to the individual images, the @@ -740,5 +757,6 @@ survey, five separate ``seasons'' of database flat fields were needed to ensure proper calibration. This indicates that the flat field -response is not completely fixed in time. +response is not completely fixed in time. More details on this +process are contained in \citet{calibration}. \subsection{Pattern correction} @@ -954,5 +972,7 @@ \includegraphics[width=1.5\hsize,angle=0,clip]{images/o5220g0025o_XY53_fringe.png} \end{minipage} - \caption{Example of the \yps{} filter fringe pattern on exposure o5220g0025o OTA53 (\yps{} filter 30s). The left panel shows the OTA mosaic with all detrending except the fringe correction, while the right shows the same including the fringe correction. Both images have been smoothed with a Gaussian with $\sigma = 3$ pixels to highlight the faint and large scale fringe patterns. \czwdraft{See if there's a way to have mana produce images larger than the screen size.}} + \caption{Example of the \yps{} filter fringe pattern on exposure o5220g0025o OTA53 (\yps{} filter 30s). The left panel shows the OTA mosaic with all detrending except the fringe correction, while the right shows the same including the fringe correction. Both images have been smoothed with a Gaussian with $\sigma = 3$ pixels to highlight the faint and large scale fringe patterns. +%\czwdraft{See if there's a way to have mana produce images larger than the screen size.} +} \label{fig: fringe example} \end{figure} @@ -1372,16 +1392,16 @@ \label{sec:background} -\czwdraft{Nigel: 2.10 The background section is rather short, given all the fuss DRAVG made -about it. What is done to eliminate contamination by bright objects - isn't -there some sort of clipping? We also have a confusing number of ``bins'' in the -text (``These bins have 10000 .... a binned cumulative distribution is -generated. These bins are interpolated ... levels. Repeating this across all -bins ...''). There is no mention of the fact that this will subtract real -astrophysics backgrounds if they are on a suitably large scale, or of the fact -that the subtraction is not perfect (don't I remember that the stacks end up -with a non-zero background which scales with the number of input warps?). -} - -\czwdraft{Based on the wiki page on 2014-05-21 the stack background issue should be resolved.} +%% \czwdraft{Nigel: 2.10 The background section is rather short, given all the fuss DRAVG made +%% about it. What is done to eliminate contamination by bright objects - isn't +%% there some sort of clipping? We also have a confusing number of ``bins'' in the +%% text (``These bins have 10000 .... a binned cumulative distribution is +%% generated. These bins are interpolated ... levels. Repeating this across all +%% bins ...''). There is no mention of the fact that this will subtract real +%% astrophysics backgrounds if they are on a suitably large scale, or of the fact +%% that the subtraction is not perfect (don't I remember that the stacks end up +%% with a non-zero background which scales with the number of input warps?). +%% } + +%% \czwdraft{Based on the wiki page on 2014-05-21 the stack background issue should be resolved.} Once all other detrending is done, the pixels from each cell are @@ -1813,12 +1833,20 @@ assumed to be included in the zeropoint and transparency values. - -\czwdraft{Nigel: 5. ``The ouput exposure time is set to the sum of the input exposure times.'' -True, but we should note that as warps can be rejected later on in the -stacking process this output time is notional in some sense. -Calibration - for PV3 what photometric calibration has been used at this stage -for the input warps? Should we make it clear here that pixels are not subject -to the final (any?) ubercal? -} +The zeropoint calibration performed here uses the calibration of the +individual input exposures against the reference catalog. Upon the +conclusion of the survey, the entire set of detection catalogs is +further re-calibrated in the ``ubercal'' process \citep{ubercal}. +This produces a more consistent calibration of each exposure across +the entire region of the sky imaged. This further calibration is not +available at the time of stacking, and so there may be small residuals +in the transparency values as a result of this \citet{calibration}. + +%% \czwdraft{Nigel: 5. ``The ouput exposure time is set to the sum of the input exposure times.'' +%% True, but we should note that as warps can be rejected later on in the +%% stacking process this output time is notional in some sense. +%% Calibration - for PV3 what photometric calibration has been used at this stage +%% for the input warps? Should we make it clear here that pixels are not subject +%% to the final (any?) ubercal? +%% } % PREPARE @@ -2041,5 +2069,5 @@ \begin{eqnarray} - \mathrm{limit}_\mathrm{mixture model} &=& 4^2 * (\sigma^2_\mathrm{input} + \sigma_\mathrm{mixture model}^2) \\ + \mathrm{limit}_\mathrm{mixture\ model} &=& 4^2 * (\sigma^2_\mathrm{input} + \sigma_\mathrm{mixture\ model}^2) \\ \mathrm{limit}_\mathrm{default} &=& 4^2 * (\sigma^2_\mathrm{input} + (0.1 * \mathrm{value}_\mathrm{input})^2) \end{eqnarray} @@ -2269,7 +2297,59 @@ \end{figure} - - - +\section{Difference Images} +\label{sec:diffs} + +Constructing difference images is essentially the same as that used in +the stacking process. An image is chosen as a template, another image +as the input, and after matching sources to determine the scaling and +transparency, convolution kernels are defined that are used to +convolve one or both of the images to a target PSF. The images are +then subtracted, and as they should now share a common PSF, static +sources are largely subtracted (completely in an ideal case), whereas +sources that are not static between the two images leave a significant +remnant. More information on the difference image construction is +contained in \citet{pauls_diff_paper}. The follow section contains a +overview of the difference image construction used for the data in +DR2. + +The images used to construct difference images can be either +individual warp skycell frames or stacked images, with support for +either to be used as the template or input. In general, for +differences using stacks, the deepest stack (or the only stack in the +case of a warp-stack difference) is used as the template. The PV3 +processing used warp-stack differences of all input warps against the +stack that was constructed from those inputs. The same ISIS kernels +as were used in the stack image combination were again used to match +the stack PSF to the input warp PSF. After convolution of the image +products, the difference is constructed for both the positive (warp +minus stack) and inverse (stack minus warp) to allow for the +photometry of the difference image to detect sources that both rise +and fall relative to the stack. Note that the convolution process +grows the mask fraction of pixels relative to the warp (the largest +source of masked pixels in these warp stack differences). Any pixel +that after convolution has any contribution from a masked pixel is +masked as well, ensuring only fully unmasked pixels are used. + +For warp-warp differences, such as those used for the ongoing Solar +System moving object search in nightly observations \citep{MOPS}, the +warp that was taken first is used as the template. As there is less +certainty in which of the two input images will have better seeing, a +``dual'' convolution method is used. Both inputs are convolved to a +target PSF that is not identical to either input. This intermediate +target is essential for the case in which the PSFs of the two inputs +have been distorted in orthogonal directions. Simply convolving one +to match the other would require some degree of deconvolution along +one axis. As this convolution method by necessity uses more free +parameters, the ISIS kernels used are chosen to be simpler than those +used in the warp-stack differences. The ISIS widths are kept the same +(1.5, 3.0, 6.0 pixel FWHMs), but each Gaussian kernel is constrained +to only use a second-order polynomial. As with the warp-stack +differences, the mask fraction grows between the input warp and the +final difference image due to the convolution. For the warp-warp +differences, each image mask grows based on the appropriate +convolution kernel, so the final usable image area is highly dependent +on ensuring that the telescope pointings are as close to identical as +possible. The observing strategy to enable this is discussed in more +detail in \citet{paper1}. @@ -2297,6 +2377,7 @@ There is some evidence that we have not fully identified all of these crosstalk rules, based on a study of PV3 images. For example, -extremely bright stars \czwdraft{exp o5677g0123o has this rule, find a - magnitude} may be able to create crosstalk ghosts between the second +extremely bright stars %\czwdraft{exp o5677g0123o has this rule, find a +% magnitude} +may be able to create crosstalk ghosts between the second cell column of OTA01 and OTA21, with possibly fainter ghosts appearing on OTA11. Despite the symmetry observed in the main ghost rules, @@ -2329,12 +2410,12 @@ stacks if fewer pixels need to be rejected. +% \czwdraft{one, I believe} The fringe model used currently is based on only a limited number of -days of data \czwdraft{one, I believe}. This means that the model -calculated may not be fully sensitive to the exact spectrum of the -sky. This may make the model quality differ based on the date and -local time of observation. There is some evidence that the fringe -model does fit some dates better than others, and so improving this by -expanding the number of input exposures may improve a wider range of -dates. +days of data. This means that the model calculated may not be fully +sensitive to the exact spectrum of the sky. This may make the model +quality differ based on the date and local time of observation. There +is some evidence that the fringe model does fit some dates better than +others, and so improving this by expanding the number of input +exposures may improve a wider range of dates. % o5818g0349o is a good example of bad fringe correction. @@ -2349,15 +2430,16 @@ clip this peak to reduce the noise in the image space is not clear. + \section{Conclusion} -\czwdraft{Not happy with this.} +%\czwdraft{Not happy with this.} The Pan-STARRS1 PV3 processing has reduced an unprecidented volume of -image data, and has produced a catalog of \czwdraft{N} individual -measurements of \czwdraft{Y} astronomical objects. Accurately -calibrating and detrending is essential to ensuring the quality of -these results. The detrending process detailed here produces -consistent data, despite the many individual detectors and their -individual response functions. +image data, and has produced a catalog for the $3\Pi$ Survey +containing hundreds of billions of individual measurements of +\czwdraft{five billion} astronomical objects. Accurately calibrating +and detrending is essential to ensuring the quality of these results. +The detrending process detailed here produces consistent data, despite +the many individual detectors and their individual response functions. From these individual exposures, we are able to construct images on @@ -2367,19 +2449,19 @@ data set that is ideal for use as a template for image differences. -The Pan-STARRS1 Surveys (PS1) have been -made possible through contributions by the Institute for Astronomy, the -University of Hawaii, the Pan-STARRS Project Office, the Max-Planck -Society and its participating institutes, the Max Planck Institute for -Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial -Physics, Garching, The Johns Hopkins University, Durham University, -the University of Edinburgh, the Queen's University Belfast, the -Harvard-Smithsonian Center for Astrophysics, the Las -Cumbres Observatory Global Telescope Network Incorporated, the -National Central University of Taiwan, the Space Telescope Science Institute, and the National -Aeronautics and Space Administration under Grant No. NNX08AR22G issued -through the Planetary Science Division of the NASA Science Mission -Directorate, the National Science Foundation Grant No. AST-1238877, -the University of Maryland, Eotvos Lorand University (ELTE), -and the Los Alamos National Laboratory. +The Pan-STARRS1 Surveys (PS1) have been made possible through +contributions by the Institute for Astronomy, the University of +Hawaii, the Pan-STARRS Project Office, the Max-Planck Society and its +participating institutes, the Max Planck Institute for Astronomy, +Heidelberg and the Max Planck Institute for Extraterrestrial Physics, +Garching, The Johns Hopkins University, Durham University, the +University of Edinburgh, the Queen's University Belfast, the +Harvard-Smithsonian Center for Astrophysics, the Las Cumbres +Observatory Global Telescope Network Incorporated, the National +Central University of Taiwan, the Space Telescope Science Institute, +and the National Aeronautics and Space Administration under Grant +No. NNX08AR22G issued through the Planetary Science Division of the +NASA Science Mission Directorate, the National Science Foundation +Grant No. AST-1238877, the University of Maryland, Eotvos Lorand +University (ELTE), and the Los Alamos National Laboratory.