Index: trunk/doc/release.2015/ps1.detrend/detrend.tex =================================================================== --- trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 39861) +++ trunk/doc/release.2015/ps1.detrend/detrend.tex (revision 39862) @@ -14,6 +14,4 @@ %\usepackage{subcaption} %\usepackage{natbib} -%\bibliographystyle{apj} -%\bibliographystyle{plain} % online version may use color, but print version needs b/w @@ -123,5 +121,4 @@ \keywords{Surveys:\PSONE } -%% http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?2007ASPC..364..153M&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf \section{Introduction and Survey Description} @@ -192,14 +189,4 @@ Pan-STARRS 1 Science Consortium members. -%% \czwdraft{Nigel: you mention calibrating to the reference catalog without telling us -%% what this is composed of (maybe this is in a different section, but would be -%% nice to have some idea here).} - -%% \czwdraft{Can we get around this point by simply adding a reference to -%% the paper describing the reference catalog? It's not really part of -%% the detrending process, and is discussed here mostly to give an -%% overview of the stages, and later to find sources of ghosts for -%% masking.} - The Pan-STARRS image processing pipeline (IPP) is described elsewhere \citep{magnier2017a}, but a short summary follows. The @@ -290,15 +277,7 @@ % Discuss 2-phase/3-phase device differnces -%\section{General Detrend Discussion} -%\label{sec:detrending} - \section{GPC1 Detrend Details} \label{sec:detrending} - -%% \czwdraft{Nigel: I forgot: when we are talking about the various bias corrections it might be -%% worth pointing out that we expect these to be more of an issue in the g-band -%% (and maybe r?) where read noise is a significant contributor. -%% } Ensuring a consistent and uniform detector response across the @@ -472,5 +451,4 @@ \subsection{Non-linearity Correction} \label{sec:nonlinearity} -% check notebook, 2010-07/08 The pixels of GPC1 are not uniformly linear at all flux levels. In @@ -518,35 +496,4 @@ rejected. -%% exptime n_included/det_id = 372 -%% clearly this isn't the one used, as 3-12 spans three data points, poorly.x -%% 0.01 2 -%% 0.14 2 -%% 0.27 2 -%% 0.49 2 -%% 0.72 2 -%% 1.06 2 -%% 1.41 2 -%% 2.02 2 -%% 2.63 2 -%% 3.94 2 -%% 5.25 2 -%% 8.74 2 -%% 13.09 2 -%% 17.4 2 -%% 20.86 2 -%% 24.3 2 -%% 27.78 2 -%% 31.24 2 -%% 34.65 2 -%% 38.12 2 -%% 42.41 2 -%% 46.69 2 -%% 51.89 2 -%% 57.04 2 - - -%http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearity_AllEdges -%http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/DetectorLinearityArchive - \begin{figure} \centering @@ -557,5 +504,4 @@ \subsection{Dark/Bias Subtraction} \label{sec:dark} -% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/Background_Dark_Model The dark model we make for GPC1 considers each pixel individually, @@ -796,11 +742,4 @@ reduces the amplitude of these errors produces a flat field model that better represents the true detector response. - -%% \czwdraft{EAM: the flat-field construction part needs to make a clearer discussion of -%% the skyflat vs the photometric correction (photflat) built initially for -%% the survey vs the flat-field corrections determined in the database as part -%% of ubercal (for the latter, you should just mention the concept -- it will -%% also be mentioned in the calibration paper). The statement that the -%% flat-field response was stable is not true since we did need 5 'seasons'.} In addition to this flat field applied to the individual images, the @@ -887,14 +826,4 @@ size of the cell (598 pixels = 150"). -%% \czwdraft{keep this?} This row-by-row offset is visible in similar -%% camera designs, and has been removed by identifying the noise signal -%% in the pixel data stream. By taking the FFT of the pixels and a -%% reference signal, the frequency of this noise can be isolated and -%% removed, resulting in a much cleaner image. However, GPC1 does not -%% record the value of the reference signal, instead automatically -%% subtracting it from the data values. Without this comparison signal, -%% we have been unable to reproduce this method, as there is no obvious -%% FFT component visible. - \begin{deluxetable}{lcccc} \tablecolumns{3} @@ -976,9 +905,4 @@ this correction on an image profile is shown in Figure \ref{fig:dark switching}. -%% \begin{figure} -%% \centering -%% \caption{Continuity example, with background issue.} -%% \label{fig: continuity example} -%% \end{figure} \subsection{Fringe correction} @@ -1026,5 +950,4 @@ \end{minipage} \caption{Example of the \yps{} filter fringe pattern on exposure o5220g0025o OTA53 (\yps{} filter 30s). The left panel shows the OTA mosaic with all detrending except the fringe correction, while the right shows the same including the fringe correction. Both images have been smoothed with a Gaussian with $\sigma = 3$ pixels to highlight the faint and large scale fringe patterns. -%\czwdraft{See if there's a way to have mana produce images larger than the screen size.} } \label{fig: fringe example} @@ -1205,12 +1128,7 @@ \end{deluxetable} -%% \begin{figure} -%% \centering -%% \caption{Figure of crosstalk ghost and bright star source. Plot of cut across ghost to illustrate the flat-top shape.} -%% \end{figure} \subsubsubsection{Optical ghosts} \label{sec:optical_ghosts} -% http://arxiv.org/pdf/1207.2513v1.pdf Due to imperfections in the anti-reflective coating on the optical @@ -1294,34 +1212,4 @@ \label{sec:glints} -% I finally tracked it down: -%% > On 8/26/2010 9:24 AM, John Tonry wrote: -%% > -%% > Gene, -%% > -%% > This is a bit of a case of the dog that didn't bark, but the shutter mask -%% > went in on Tuesday. -%% > -%% > Can you can tell us whether -%% > -%% > a) it's helped the shutter glint problem and -%% > b) whether there's any discernable vignetting anywhere? -%% > -%% > - John - -%% On Thu, Aug 26, 2010 at 4:00 PM, Chris Waters wrote: - -%% > I'm not entirely sure why I'm not on the ps-ipp mailing list, but -%% > Heather forwarded this to me. I compared 240 exposures from -%% > 2010-08-22/ThreePi/y.00000 and 2010-08-25/ThreePi/y.00000. -%% > -%% > a) For the 22nd, I counted 28 star glints visible. For the 25th, I -%% > counted maybe 0-2 (I think the first is a conveniently placed satellite, -%% > and the other has a companion, so I think it's actually a moon glint). -%% > -%% > b) I was going to compare flat field images, but we don't have any -%% > from after the mask was applied. Blinking between a few pairs of the -%% > 240x2 exposures does not show any vignetting that I can detect from -%% > the IPP jpeg mosaics. - Prior to 2010-08-24, a reflective surface at the edge of the camera aperture was incompletely screened to light passing through the @@ -1339,28 +1227,4 @@ by 17 arcminutes, and extend outwards an additional degree. -%% -%% GLINT_MAX_MAG F32 -21.0 -%% GLINT.REGION MULTI - -%% GLINT.REGION METADATA -%% REGION STR [-38000:-24000,-20000:+20000] -%% GLINT.TYPE STR LEFT -%% END - -%% GLINT.REGION METADATA -%% REGION STR [+24000:+38000,-20000:+20000] -%% GLINT.TYPE STR RIGHT -%% END - -%% GLINT.REGION METADATA -%% REGION STR [-20000:+20000,+24000:+38000:] -%% GLINT.TYPE STR TOP -%% END - -%% GLINT.REGION METADATA -%% REGION STR [-20000:+20000,-38000:-24000] -%% GLINT.TYPE STR BOTTOM -%% END - \begin{figure} \centering @@ -1380,7 +1244,4 @@ oriented at $\theta = n * \frac{\pi}{2} - \mathrm{ROTANGLE} + 0.798$, based on the header keyword. - -%\subsubsection{Saturated stars} -%\label{sec:saturated_stars} The cores of stars that are saturated are masked as well, with a @@ -1442,39 +1303,6 @@ \end{deluxetable} - - - -%% mysql> select filter,AVG(camProcessedExp.maskfrac_ref_static), AVG(camProcessedExp.maskfrac_ref_dynamic), AVG(camProcessedExp.maskfrac_ref_advisory), AVG(camProcessedExp.maskfrac_max_static),AVG(camProcessedExp.maskfrac_max_dynamic),AVG(camProcessedExp.maskfrac_max_advisory) from camRun join camProcessedExp USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.label = 'LAP.PV3.20140730.final' GROUP BY filter; -%% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+ -%% | filter | AVG(camProcessedExp.maskfrac_ref_static) | AVG(camProcessedExp.maskfrac_ref_dynamic) | AVG(camProcessedExp.maskfrac_ref_advisory) | AVG(camProcessedExp.maskfrac_max_static) | AVG(camProcessedExp.maskfrac_max_dynamic) | AVG(camProcessedExp.maskfrac_max_advisory) | -%% +---------+------------------------------------------+-------------------------------------------+--------------------------------------------+------------------------------------------+-------------------------------------------+--------------------------------------------+ -%% static dynamic advisory -%% | g.00000 | 0.19642137972007 | 0.00010322263512709 | 0.026838445469766 -%% | 0.20949461794863 | 9.89200027293e-05 | 0.026431927734548 | -%% | r.00000 | 0.19675996201399 | 0.00025214447869606 | 0.032641054600788 -%% | 0.20989768279138 | 0.00023994155711801 | 0.032178525485201 | -%% | i.00000 | 0.19677587604327 | 0.00057470697316504 | 0.038096251937072 -%% | 0.21003570722292 | 0.00053987093278142 | 0.037471018638997 | -%% | z.00000 | 0.1974290315691 | 0.00024758901226967 | 0.03064123748973 -%% | 0.21055007930696 | 0.00023452690039757 | 0.030144453360769 | -%% | y.00000 | 0.19828990634315 | 0.00014523787521897 | 0.021984846417987 -%% | 0.21130344126869 | 0.00013634812877977 | 0.02163070300815 | - - \subsection{Background subtraction} \label{sec:background} - -%% \czwdraft{Nigel: 2.10 The background section is rather short, given all the fuss DRAVG made -%% about it. What is done to eliminate contamination by bright objects - isn't -%% there some sort of clipping? We also have a confusing number of ``bins'' in the -%% text (``These bins have 10000 .... a binned cumulative distribution is -%% generated. These bins are interpolated ... levels. Repeating this across all -%% bins ...''). There is no mention of the fact that this will subtract real -%% astrophysics backgrounds if they are on a suitably large scale, or of the fact -%% that the subtraction is not perfect (don't I remember that the stacks end up -%% with a non-zero background which scales with the number of input warps?). -%% } - -%% \czwdraft{Based on the wiki page on 2014-05-21 the stack background issue should be resolved.} Once all other detrending is done, the pixels from each cell are @@ -1588,16 +1416,4 @@ stage images, but this special processing was not used for the large scale $3\Pi$ PV3 reduction. - -%% * Magic -%% * Warping -%% * warping kernel -%% * linear-by-pieces -%% * Covariance -%% * def of skycells? -%% * Stacking -%% * pixel combination rules -%% * pixel rejections -%% * convolution for matching (success and failure) -%% * Difference Image analysis \section{GPC1 Detrend Construction} @@ -1835,25 +1651,4 @@ \end{figure} - -% Read all images and astrometry -% Check which input images overlap with output image. => 8007 when the inputs don't overlap. -% Loop over each image. -% Append detections from input into output detection list. -% Determine transform back from warp pixels to source image. -%% 2nd order polynomial in both x and y for this transformation. and save to header -% Break warp image into 128x128pixel locally linear areas -% Mask non finite pixels as saturated. -% Define Lanczos-3 interpolation over the input image. -% Iterate over warp pixel space (on each locally linear grid) and map interpolated input pixel positions onto warp. -% Warp pixel space is defined as center based, so that's where the intpolation point comes from. -% Covariance calculated based on the interpolation kernel at the center of the ll grid. -% image = interp_image * jacobian -% var = interp_var * jacobian**2 -% mask = interp_mask -% jacobian = abs(mapXx * mapYy - mapYx * mapXy) -% I don't understand that jacobian. -% - - \section{Stacking} \label{sec:stacking} @@ -1915,45 +1710,4 @@ in the transparency values as a result of this \citet{magnier2017c}. -%% \czwdraft{Nigel: 5. ``The ouput exposure time is set to the sum of the input exposure times.'' -%% True, but we should note that as warps can be rejected later on in the -%% stacking process this output time is notional in some sense. -%% Calibration - for PV3 what photometric calibration has been used at this stage -%% for the input warps? Should we make it clear here that pixels are not subject -%% to the final (any?) ubercal? -%% } - -% PREPARE -% load sources -% load psfs -% determine target as envelope of input psfs -% FWHM clipping at 10 -% measure seeing -% - // M_app = m_inst + zp + c1 * airmass + 2.5log(t) - transparency -% // EAM : the discussion here was not quite right (or at least sloppy). Here is a replacement explanation: -% // For any star, the observed instrumental magnitude on an image and the apparent magnitude are related by: -% // M_app = m_inst + zp + c1 * airmass + 2.5log(t) - transparency -% // NOTE the sign of 'transparency' this must agree with the definition in pmSourceMatch.c. see, eg, line 457 where -% // transparency = m_inst + zp + c1 * airmass + 2.5log(t) - M_app -% // we want to adjust the input images to be in a consistent flux system so that the -% // final stack can be generated with a specific target zero point. Any adjustment to -% // the flux scale of the image must be made in coordination with the resulting -% // zeropoint, exposure time, and airmass such that the above relationship yields the -% // same apparent magnitude for a given star: -% // m_inst_i : instrumental mags on input image (in) -% // m_inst_o : instrumental mags on re-normalized image (out) -% // m_inst_o + zp_o + c1 * airmass_o + 2.5log(t_o) - trans_o = m_inst_i + zp_i + c1 * airmass_i + 2.5log(t_i) - trans_i -% // m_inst_o = m_inst_i + (zp_i - zp_o) + c1 * (airmass_i - airmass_o) + 2.5log(t_i) - 2.5log(t_o) - trans_i + trans_o -% // zp_i, airmass_i, t_i, trans_i : reported or measured for input image -% // zp_o = zpTarget (from recipe) -% // airmass_o = airmassTarget (from recipe) -% // t_o = sumExpTime [sum of input exposure times: once images are scale to this time, they can be avereaged] -% // trans_o = 0.0 [obviously!] -% // we have 2 cases: (a) all reported ZPs are good or (b) some are bad: -% // (a) FPA.ZP = zp_i + c1 * airmass_i -% // --> zp[i] = zp_i + c1 * airmass_i + 2.5log(exptime_i) -% // (b) zp[i] = c1 * airmass_i + 2.5log(exptime_i) -% // NOTE: in case (b), the current code is equating the TARGET zp with the NOMINAL zp, which is wrong. -% // m_inst_o - m_inst_i = zp[i] - zpTarget - c1 * airmassTarget - 2.5log(sumExpTime) - trans_i - With the flux normalization factors and target PSF chosen, the convolution kernels can be calculated for each image. ISIS kernels @@ -1977,11 +1731,4 @@ the square of it, scaling all inputs to the common zeropoint. -% MATCH -% match to target PSF. -% use ISIS kernels to do matching/convolution -% Input sources used for psf matching. -% @ISIS.WIDTHS F32 1.5 3.0 6.0 # Gaussian kernel FWHM values -% @ISIS.ORDERS S32 6 4 2 # Polynomial orders for ISIS kernels - Once the convolution kernels are defined for each image, they are used to convolve the image to match the target PSF. Any input image that @@ -1994,11 +1741,4 @@ warping process). -% CONVOLVE -% Normalization to match target zeropoint/exptime -% Reject images with bad match chi^2 values. MATCH.REJ * rms + median threshold. -% Additional variance from the convolution chi^2 -% Calculate image weights based on variance: W_i = 1 / (ROBUST_MEDIAN(variance_image_i) * CovarianceFactor) -% CovarianceFactor = covariance->kernel[0][0] - Following the convolution, an initial stack is constructed. For a given pixel coordinate, the values at that coordinate are extracted @@ -2032,54 +1772,4 @@ The output mask value is taken to be zero (no masked bits), unless there were no valid inputs, in which case the BLANK mask bit is set. - -% INITIAL COMBINE -% Calculate weighted mean of input images -% mu = sum_i(f_i * W_i) / sum_i(W_i) -% sigma = 1 / sum_i(1 / W_i) -% nu = sum_i(m_i) -% // We're not using the input pixel variances to generate a weighted average for the pixel flux (because -% // that introduces systematic biases), so the variance of the output pixel value should simply be: -% // simga^2 = sum(weight_i^2 * sigma_i^2) / (sum(weight_i))^2 -% // This reduces, when the weights are all identically unity, to: -% // variance_combination = sum(variance_i) / N^2 -% // and if the variances are all equal: -% // variance_combination = variance_individual / N -% // which makes sense --- the standard deviation of the combination is reduced by a factor of sqrt(N). -% sumValueWeight = sum_i(values * weights) -% sumWeight = sum_i(weights) -% sumVarianceWeight == sum( 1 / variances) -% sumExp = sum_i(exptimes) -% sumExpWeight = sum_i(exptims * weights) -% mean = sumValueWeight / sumWeight -% var = 1 / sumVarianceWeight -% exp = sumExp -% expWeight = sumExpWeight - -% EXCEPT: if N = 1, accept it. if N = 2, take average. - -% if (!pmStackCombine(outRO, NULL, stack, maskBad, maskSuspect, maskBlank, kernelSize, iter, -% combineRej, combineSys, combineDiscard, useVariance, safe, nminpix, false)) { -%bool pmStackCombine( -% pmReadout *combined, // output stacked readout -% pmReadout *expmaps, // output exposure map information -% psArray *input, // input exposures -% psImageMaskType badMaskBits, // treat these bits as 'bad' -% psImageMaskType suspectMaskBits, // treat these bits as 'suspect' -% psImageMaskType blankMaskBits, // use this mask value for pixels missing input data (distinguish between Ninput = 0 and Ngood = 0?) -% int kernelSize, -% float iter, 0.5 -% float rej, 4.0 -% float sys, 0.1 -% float olympic, 0.2 -% bool useVariance, -% bool safe, -% int nminpix, -% bool rejection) -%{ - -% combineExtract -%% pixels with mask values as suspect are appended to suspect pixel list. -% combinePixels -%% As described above. Due to the various non-astronomical ghosts that can occur on GPC1, and @@ -2155,35 +1845,4 @@ number of inputs. -% combineTest -%% if (Ninput > 6) { use KMM } -%% KMM: -%% Calculate KMMmu KMMsigma KMMpi KMMPunimodal -%% SumWeights = sum(pixelWeights) -%% SysVar = KMMSigma**2 OR (sys * pixelData[i])**2 -%% pixelLimts[i] = rej**2 * (pixelVariances[i] + sysVar) -% Iterate 0.5 * Ninput times (at least once) -%% Ninput = 1 => accept -%% Ninput = 2 => if (0.5 * (A - B))**2 > rej**2 * (pixelVariance[A] + pixelVariance[B] + (sys * A)**2 + (sys * B)**2) -%% then if (suspect) mark reject else mark inspect -%% Else => if (useKMM and Punimodal < 0.05) median = KMMmean -%% => else median = combinationWeightedOlympic{} -%% => if (pixelData - median)**2 > pixelLimits[i] then find single worst deviant pixel value -%% then => if suspect (mark reject) else (mark reject worst deviant pixel value) - - -%% combinationWeightedOlympic => -%% numBad = frac * Ninput + 0.5 -%% low = numBad / 2, high = low + numGood - numBad -%% sort(values) => -%% if (i > low && i <= high) { sumValues = sum_i(values * weights); sumWeight = sum_i(weights) -%% return (sumValues / sumWeight) - -% obtain lists of inspect and reject pixels. - -% normalize:? -% float normalise = powf(10.0, -0.4 * norm->data.F32[i]); // Normalisation -% psBinaryOp(ro->image, ro->image, ``*'', psScalarAlloc(normalise, PS_TYPE_F32)); -% psBinaryOp(ro->variance, ro->variance, ``*'', psScalarAlloc(PS_SQR(normalise), PS_TYPE_F32)); - With the initial list of rejected pixels generated, a rejection mask is made for the input warp by constructing an empty image that has the @@ -2194,11 +1853,4 @@ more than 10\% of all pixels from an input image are rejected, then the entire image is rejected as it likely has some systematic issue. - -% PIXEL REJECTION -% Construct 15-pixel wide ISIS kernel with 5 pixel FWHM 0-order. -% Construct image of pixels to inspect and convolve with kernel (normalize out kernel power) -% Determine pixels are bad if they're larger than THRESHOLD.MASK = 0.5. -% If more than IMAGE.REJ = 0.1 fraction of pixels are rejected, the entire image is rejected. - Finally, a second pass at rejecting pixels is conducted, by growing the @@ -2215,19 +1867,4 @@ a map of the number of inputs per pixel. -% FINAL COMBINE -% Grow rejected pixels -%% set threshold of (POOR.FRACTION = 0.25) * sum(kernel)**2 -%% Choose the largest square box that contains just under that threshold. -%% Convolve that box with the rejected pixels to grow them. -% Run combination pass again, but without doing rejection, simply applying the rejection lists already calculated. -% :: -% if (!ppStackCombineFinal(stack, options->convCovars, options, config, false, true, true, true)) { -% iter = 0 -% combineRej = NAN -% combineSys = NAN -% combineDiscard = NAN -% if (!pmStackCombine(outRO, expRO, stack, maskBad, maskSuspect, maskBlank, 0, iter, combineRej, -% combineSys, combineDiscard, useVariance, safe, nminpix, rejected)) { - These convolved stack products are not retained, as the convolution reduces the resolution of the final image. Instead, we apply the @@ -2238,10 +1875,4 @@ across the image, as the different PSF widths of the input images print through in the different regions to which they have contributed. - -% UNCONVOLVED IMAGE -% if (!ppStackCombineFinal(stack, options->origCovars, options, config, false, true, false, true)) { -% no grow - -% only retain unconvolved products. %% Asinh compression @@ -2450,7 +2081,5 @@ There is some evidence that we have not fully identified all of these crosstalk rules, based on a study of PV3 images. For example, -extremely bright stars %\czwdraft{exp o5677g0123o has this rule, find a -% magnitude} -may be able to create crosstalk ghosts between the second +extremely bright stars may be able to create crosstalk ghosts between the second cell column of OTA01 and OTA21, with possibly fainter ghosts appearing on OTA11. Despite the symmetry observed in the main ghost rules, @@ -2483,5 +2112,4 @@ stacks if fewer pixels need to be rejected. -% \czwdraft{one, I believe} The fringe model used currently is based on only a limited number of days of data. This means that the model calculated may not be fully @@ -2491,5 +2119,4 @@ others, and so improving this by expanding the number of input exposures may improve a wider range of dates. -% o5818g0349o is a good example of bad fringe correction. Finally, a large number of issues arise due to the row-to-row bias @@ -2505,6 +2132,4 @@ \section{Conclusion} - -%\czwdraft{Not happy with this.} The Pan-STARRS1 PV3 processing has reduced an unprecedented volume of