IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 39799


Ignore:
Timestamp:
Nov 3, 2016, 12:14:23 PM (10 years ago)
Author:
watersc1
Message:

Updated version with all recent tweaks.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/release.2015/ps1.detrend/detrend.tex

    r39618 r39799  
    118118with the PS1 telescope on Haleakala Maui to image the sky north of
    119119$-30^\circ$ declination.  The GPC1 camera is composed of 60 orthogonal
    120 transfer array (OTA) devices, each of with is an $8\times{}8$ grid of
     120transfer array (OTA) devices, each of which is an $8\times{}8$ grid of
    121121readout cells.  This parallelizes the readout process, reducing the
    122122overhead in each exposure.  However, as a consequence of this large
    123 number of individual detector readouts, there are a number of
    124 calibrations that need to be included to ensure the response is
    125 consistent across the entire field of view.
    126 
    127 The PV3 reduction represents the third full processing version of the
    128 Pan-STARRS archival data.  The first two reductions were used
    129 internally for pipeline optimization and the development of the
    130 initial photometric and astrometric reference catalog.  The products
    131 from these reductions were not publicly released, but have been used
    132 to produce a wide range of scientific papers from the Pan-STARRS 1
    133 Science Consortium members.
     123number of individual detector readouts, many calibrations are needed
     124to ensure the response is consistent across the entire field of view.
     125
     126The Processing Version 3 (PV3) reduction represents the third full
     127reduction of the Pan-STARRS archival data.  The first two reductions
     128were used internally for pipeline optimization and the development of
     129the initial photometric and astrometric reference catalog.  The
     130products from these reductions were not publicly released, but have
     131been used to produce a wide range of scientific papers from the
     132Pan-STARRS 1 Science Consortium members.
    134133
    135134The Pan-STARRS image processing pipeline (IPP) is described elsewhere
    … …  
    144143Following the \ippstage{chip} stage is the \ippstage{camera} stage, in
    145144which the astrometry and photometry for the entire exposure is
    146 calibrated against the reference catalog.  This stage also performs
    147 masking updates based on the now-known positions and brightnesses of
    148 stars that create dynamic features (see Section
    149 \ref{sec:dynamic_masks} below).  The \ippstage{warp} stage is the next
    150 to operate on the data, transforming the detector oriented
    151 \ippstage{chip} stage images into sky oriented images that have common
    152 tessellations and sky projections (Section \ref{sec:warping}).  When
    153 all \ippstage{warp} stage processing is done for the region of the
    154 sky, \ippstage{stack} processing is performed (Section
    155 \ref{sec:stacking}) to construct deeper, fully populated images from
    156 the set of \ippstage{warp} images that cover that region of the sky.
    157 Beyond the \ippstage{stack} stage, a series of additional stages are
    158 done that are more fully described in other papers.  Transient
    159 features are identified in the \ippstage{diff} stage, which takes
    160 input \ippstage{warp} and/or \ippstage{stack} data and performs image
     145calibrated by matching the detections against the reference catalog.
     146This stage also performs masking updates based on the now-known
     147positions and brightnesses of stars that create dynamic features (see
     148Section \ref{sec:dynamic_masks} below).  The \ippstage{warp} stage is
     149the next to operate on the data, transforming the detector oriented
     150\ippstage{chip} stage images onto common sky oriented images that have
     151fixed sky projections (Section \ref{sec:warping}).  When all
     152\ippstage{warp} stage processing is done for the region of the sky,
     153\ippstage{stack} processing is performed (Section \ref{sec:stacking})
     154to construct deeper, fully populated images from the set of
     155\ippstage{warp} images that cover that region of the sky.  Beyond the
     156\ippstage{stack} stage, a series of additional stages are done that
     157are more fully described in other papers.  Transient features are
     158identified in the \ippstage{diff} stage, which takes input
     159\ippstage{warp} and/or \ippstage{stack} data and performs image
    161160differencing \citep{HuberXXX}.  Further photometry is performed in the
    162161\ippstage{staticsky} and \ippstage{skycal} stages, which add extended
    … …  
    196195\czwdraft{Is this a sufficient explanation?  Also, is this the right
    197196  place for it?}  Image products presented in figures have been
    198 mosaicked to arrange pixels in the following way.  Single cell images
    199 are arranged such that pixel $(1,1)$ is at the lower left corner.
    200 Images mosaicked to the OTA level have cell xy00 in the lower left
    201 corner, with cells xy10, xy20, etc. sequentially to the right, and
    202 cells xy01, xy02, etc. sequentially to the top of this cell.  Again,
    203 pixel $(1,1)$ of cell xy00 is located in the lower left corner of the
    204 image.  For mosaics of the full field of view, the OTAs are arranged
    205 as they see the sky.  The lower left corner is the empty location
    206 where OTA70 would exist.  Toward the right, the OTA labels decrease in
    207 $X$ label, with the empty OTA00 located in the lower right.  The OTA
    208 $Y$ labels increase upward in the mosaic.  The OTAs to the left of the
    209 midplane (OTA4Y-OTA7Y) are oriented with cell xy00 and pixel $(1,1)$
    210 to the lower left of their position.  Due to the electronic
    211 connections of the OTAs in the focal plane, the OTAs to the right of
    212 the midplane (OTA0Y-OTA3Y) oriented with cell xy00 and pixel $(1,1)$
    213 to the top right of their position, and have a negative parity to the
    214 mosaic in both x and y.
     197mosaicked to arrange pixels as follows.  Single cell images are
     198arranged such that pixel $(1,1)$ is at the lower left corner.  Images
     199mosaicked to the OTA level have cell xy00 in the lower left corner,
     200with cells xy10, xy20, etc. sequentially to the right, and cells xy01,
     201xy02, etc. sequentially to the top of this cell.  Again, pixel $(1,1)$
     202of cell xy00 is located in the lower left corner of the image.  For
     203mosaicks of the full field of view, the OTAs are arranged as they see
     204the sky.  The lower left corner is the empty location where OTA70
     205would exist.  Toward the right, the OTA labels decrease in $X$ label,
     206with the empty OTA00 located in the lower right.  The OTA $Y$ labels
     207increase upward in the mosaic.  The OTAs to the left of the midplane
     208(OTA4Y-OTA7Y) are oriented with cell xy00 and pixel $(1,1)$ to the
     209lower left of their position.  Due to the electronic connections of
     210the OTAs in the focal plane, the OTAs to the right of the midplane
     211(OTA0Y-OTA3Y) are rotated 180 degrees, and are oriented with cell xy00
     212and pixel $(1,1)$ to the top right of their position.
    215213
    216214% Discuss 2-phase/3-phase device differnces
    … …  
    222220\label{sec:detrend construction}
    223221
    224 The detrends for GPC1 are all constructed in similar ways.  A series
    225 of appropriate exposures is selected from the database, and processed
    226 with the \ippprog{ppImage} program.  This program is used for the
    227 \ippstage{chip} stage processing as well, and is designed to do image
    228 processing.  The extent of this processing is dependent on the order
    229 in which the detrend is applied to science data.  In general, the
    230 input exposures to the detrend have all prior stages of detrend
    231 processing applied.  Table \ref{tab:detrend ppImage} summarizes stages
    232 applied for the detrends we construct.
     222The various detrends for GPC1 are constructed in similar ways.  A
     223series of appropriate exposures is selected from the database, and
     224processed with the \ippprog{ppImage} program.  This program is used
     225for the \ippstage{chip} stage processing as well, and is designed to
     226do multiple image processing operations.  The extent of this
     227processing is dependent on the order in which the detrend to be
     228constructed is applied to science data.  In general, the input
     229exposures to the detrend have all prior stages of detrend processing
     230applied.  Table \ref{tab:detrend ppImage} summarizes stages applied
     231for the detrends we construct.
    233232
    234233Once the input data has been prepared, the \ippprog{ppMerge} program
    … …  
    241240format of the detrend under construction, and after construction, are
    242241applied to the processed input data.  This creates a set of residual
    243 files that can be checked to determine if the newly created detrend
    244 works correctly.
    245 
    246 The process of detrend construction and testing can be iterated, with
     242files that are checked to determine if the newly created detrend
     243correctly removes the detector dependent signal.
     244
     245This process of detrend construction and testing can be iterated, with
    247246individual exposures excluded if they are found to be contaminating
    248 the output.  If the final detrend is considered sufficient, then the
    249 iterations are stopped and the detrend is finalized by selecting the
    250 date range to which it applies.  This allows subsequent science
    251 processing to select the detrends needed based on the observation
    252 date.  Table \ref{tab:detrend list} lists the set of detrends used in
    253 the PV3 processing.
     247the output.  If the final detrend has sufficiently small residuals,
     248then the iterations are stopped and the detrend is finalized by
     249selecting the date range to which it applies.  This allows subsequent
     250science processing to select the detrends needed based on the
     251observation date.  Table \ref{tab:detrend list} lists the set of
     252detrends used in the PV3 processing.
    254253
    255254\begin{deluxetable}{lcccc}
    … …  
    363362
    364363The final non-linear response issue has no good option for correction.
    365 Large regions of some OTA cells experience charge transfer issues,
    366 making them unusable to be used for science observations.  These
    367 regions are therefore masked in processing, with these CTE regions
    368 making up the largest fraction of masked pixels on the detector.
    369 Other regions are masked for other regions, such as static bad pixel
    370 features or temporary readout masking caused by issues in the camera
    371 electronics that make these regions unreliable.  These all contribute
    372 to the detector mask, which is augmented in each exposure for dynamic
     364Large regions of some OTA cells experience significant charge transfer
     365issues, making them unusable for science observations.  These regions
     366are therefore masked in processing, with these CTE regions making up
     367the largest fraction of masked pixels on the detector.  Other regions
     368are masked for other regions, such as static bad pixel features or
     369temporary readout masking caused by issues in the camera electronics
     370that make these regions unreliable.  These all contribute to the
     371detector mask, which is augmented in each exposure for dynamic
    373372features that are masked based on the astronomical features within the
    374373field of view.
    … …  
    407406between exposures.
    408407
    409 Both of these types of persistence trails are detected and optionally
     408Both of these types of persistance trails are measured and optionally
    410409repaired via the \ippprog{burntool} program.  This program does an
    411410initial scan of the images, and identifies objects with pixel values
    412 brighter than a threshold of 30000 DN.  The trail from that star is
    413 fit with a one-dimensional power law in each pixel column above that
    414 threshold, based on empirical evidence that this is the functional
    415 form of this persistence effect.  This also matches the expectation
    416 that a constant fraction of charge is incompletely transferred at each
    417 shift beyond the persistence threshold.  Once this fit is done, the
    418 model can subtracted from the image, and the location of the star is
    419 stored in a table along with the exposure PONTIME, which denotes the
    420 number of seconds since the detector was last powered on and provides
    421 an internally consistent time scale.
     411brighter than a conservative threshold of 30000 DN.  The trail from
     412the peak of that object is fit with a one-dimensional power law in
     413each pixel column above the threshold, based on empirical evidence
     414that this is the functional form of this persistence effect.  This
     415also matches the expectation that a constant fraction of charge is
     416incompletely transfered at each shift beyond the persistence
     417threshold.  Once this fit is done, the model can be subtracted from
     418the image, and the location of the star is stored in a table along
     419with the exposure PONTIME, which denotes the number of seconds since
     420the detector was last powered on, and provides an internally consistent
     421time scale.
    422422
    423423For subsequent exposures, the table associated with the previous image
    … …  
    426426check for remnant trails on the image.  These are fit and subtracted
    427427using a one-dimensional exponential model, again based on empirical
    428 studies.  If a significant model with is determined, then this
    429 location is retained in the image output table.  If not, the old burn
    430 is allowed to expire.
    431 
    432 An issue with this method of correcting the persistence trails is that
    433 it is based on fits to the raw image data, which may have other signal
    434 sources not determined by the persistence effect.  The presence of
    435 other stars or artifacts along the path of the burn can result in a
    436 poor model to be determined, resulting in either an over- or
    437 under-subtraction of the persistence burn.  For this reason, the image
     428studies.  If a significant model is found, then this location is
     429retained in the image output table.  If not, the old burn is allowed
     430to expire.
     431
     432The main concern with this method of correcting the persistance trails
     433is that it is based on fits to the raw image data, which may have
     434other signal sources not determined by the persistence effect.  The
     435presence of other stars or artifacts along the path of the burn can
     436result in a poor model to be fit, resulting in either an over- or
     437under-subtraction of the persistance burn.  For this reason, the image
    438438mask is marked with a value indicating that this correction has been
    439439applied.  These pixels are not fully excluded, but they are marked as
    … …  
    462462  \end{minipage}
    463463
    464   \caption{Example of a profile cut along the y-axis through a bright star on exposure o5677g0123o OTA11 in cell xy60 (left panel) and on the subsequent exposure o5677g0124o (right panel).  In both figures, the green points show the image corrected with all appropriate detrending steps, but without burntool applied, illustrating the amplitude of the persistence trails.  The red points show the same data after the burntool correction, which reduce the impact of these features.  Both exposures are in the g-filter with exposure times of 43s}
     464  \caption{Example of a profile cut along the y-axis through a bright star on exposure o5677g0123o OTA11 in cell xy60 (left panel) and on the subsequent exposure o5677g0124o (right panel).  In both figures, the green points show the image corrected with all appropriate detrending steps, but without burntool applied, illustrating the amplitude of the persistence trails.  The red points show the same data after the burntool correction, which reduces the impact of these features.  Both exposures are in the g-filter with exposure times of 43s}
    465465\end{figure}
    466466
    … …  
    489489%  \end{subfigure}
    490490  \end{minipage}
    491   \caption{Example of OTA11 cell xy60 on exposures o5677g0123o (left) and o5677g0124o (right).  The top panels show the image with all appropriate detrending steps, but with burntool, and the bottom show the same with burntool applied.  There is some slight over subtraction in fitting the initial trail, but the impact of the trail is greatly reduced in both exposures.}
     491  \caption{Example of OTA11 cell xy60 on exposures o5677g0123o (left) and o5677g0124o (right).  The top panels show the image with all appropriate detrending steps, but without burntool, and the bottom show the same with burntool applied.  There is some slight over subtraction in fitting the initial trail, but the impact of the trail is greatly reduced in both exposures.}
    492492\end{figure}
    493493
    … …  
    507507charge transfer efficiency is low compared to the rest of the
    508508detector.  Twenty-five of the sixty OTAs in GPC1 show some evidence of
    509 CTE issues, with this pattern showing up (to varying degrees) in
     509CTE issues, with this pattern appearing (to varying degrees) in
    510510roughly triangular patches on the OTA due to defects in the
    511511semiconductor manufacturing.  To generate the mask for these regions,
    … …  
    558558  \label{fig:static mask}
    559559 
    560   \caption{Image map of static mask. color coded based on mask reason?  It won't be visible at true pixel scale.}
     560  \caption{Image map of the GPC1 static mask.  The CTE regions are clearly visible as roughly triangular patches covering the corners of some OTAs.  Some entire cells are masked, including an entire column of cells on OTA14.  Calcite cells remove large areas from OTA17 AND OTA76.}
    561561\end{figure}
    562562
    … …  
    592592
    593593In addition to the static mask that removes the constant detector
    594 level defects, we also generate a set of dynamic masks that change
    595 with the astronomical features in the image.  These masks are advisory
    596 in nature, and do not completely exclude the pixel from further
     594defects, we also generate a set of dynamic masks that change with the
     595astronomical features in the image.  These masks are advisory in
     596nature, and do not completely exclude the pixel from further
    597597processing consideration.  The first of these dynamic masks is the
    598598burntool advisory mask mentioned above.  These pixels are included for
    … …  
    601601deviations due to imperfections in the burntool correction.
    602602
    603 The remaining dynamic masks are not generated until the IPP \ippstage{camera}
    604 stage, at which point all object photometry is complete, and an
    605 astrometric solution is known for the exposure.  This added
    606 information provides the positions of bright sources based on the
    607 reference catalog, including those that fall slightly out of the
     603The remaining dynamic masks are not generated until the IPP
     604\ippstage{camera} stage, at which point all object photometry is
     605complete, and an astrometric solution is known for the exposure.  This
     606added information provides the positions of bright sources based on
     607the reference catalog, including those that fall slightly out of the
    608608detector field of view or within the inter chip gaps, where internal
    609 photometry may not have identified them.  These bright sources are the
    610 origin for many of the image artifacts that the dynamic mask
    611 identifies and excludes.
     609photometry may not identify them.  These bright sources are the origin
     610for many of the image artifacts that the dynamic mask identifies and
     611excludes.
    612612
    613613\subsubsection{Electronic crosstalk ghosts}
    … …  
    615615
    616616Due to electrical crosstalk between the flex cables connecting the
    617 individual detector OTA devices, ghost objects can be created due to
    618 the presence of a bright source at a different position on the camera.
     617individual detector OTA devices, ghost objects can be created by the
     618presence of a bright source at a different position on the camera.
    619619Table \ref{tab:crosstalk_rules} summarizes the list of known crosstalk
    620620rules, with an estimate of the magnitude difference between the source
    … …  
    622622column of cells on any of the OTAs in the specified column of OTAs $Y$
    623623creates the ghost in the same $v$ and $Y$ in the target column of
    624 cells and OTAs.  In each of these cases, a source object brighter than
    625 -14.47 instrumental magnitude creates a ghost object many orders of
    626 magnitude fainter at the target location.  The cell (x,y) pixel
    627 coordinate is identical between source and ghost, as a result of the
    628 transfer occurring as the devices are read.  A circular mask is added
    629 to the ghost location with radius $R = 3.44 \left(-14.47 - m_{source,
    630   instrumental}\right)$ pixels.  Any objects in the photometric
    631 catalog found at the location of the ghost mask have the GHOST mask
    632 bit set, marking the object as a likely ghost.  The majority of the
    633 crosstalk rules are bi-directional, with a source in either position
    634 creating a ghost at the corresponding crosstalk target position.  The
    635 two faintest rules are uni-directional, due to differences in the
    636 electronic path for the crosstalk.
     624cells and OTAs.  In each of these cases, a source object with an
     625instrumental magnitude brighter than -14.47 creates a ghost object
     626many orders of magnitude fainter at the target location.  The cell
     627(x,y) pixel coordinate is identical between source and ghost, as a
     628result of the transfer occurring as the devices are read.  A circular
     629mask is added to the ghost location with radius $R = 3.44 \left(-14.47
     630- m_{source, instrumental}\right)$ pixels.  Any objects in the
     631photometric catalog found at the location of the ghost mask have the
     632GHOST mask bit set, marking the object as a likely ghost.  The
     633majority of the crosstalk rules are bi-directional, with a source in
     634either position creating a ghost at the corresponding crosstalk target
     635position.  The two faintest rules are uni-directional, due to
     636differences in the electronic path for the crosstalk.
    637637
    638638For the very brightest sources ($m_{instrumental} < -15$), there can
    … …  
    843843the cells on an OTA taking video data.  Before the nature of this
    844844issue was fully understood, these poorly constrained corners were
    845 masked with 25-pixel radius quarter circles, centered on the (0,0)
     845masked with 25-pixel radius quarter circles, centered on the (1,1)
    846846pixel nearest the cell amplifier.  The other corners of the cell were
    847847masked with a 15-pixel radius quarter circle, as the amplifier
    848 creating the glow is associated with another cell, separated by the
    849 inter-cell spacing, diminishing the area affected.  Due to the large
     848creating the glow is associated with another cell and separated by the
     849inter-cell spacing, diminishing the area effected.  Due to the large
    850850area that this masking would cover, the PV3 processing used a more
    851851robust video dark model to correct this problem, as described in
    852852section \ref{sec:video_darks} below.
    853 
    854853
    855854\subsubsection{Masking Fraction}
    … …  
    873872calculations to estimate the masking fraction.  The reference field of
    874873view of GPC1 is 3 degrees, which at the nominal plate scale of 0.258
    875 arcseconds per pixel, translates to a 20930 FPA pixel radius.
     874arcseconds per pixel, translates to a 20930 FPA pixel radius. \czwdraft{I need a percentage here.}
    876875
    877876%% mysql> select filter,AVG(camProcessedExp.maskfrac_ref_static), AVG(camProcessedExp.maskfrac_ref_dynamic), AVG(camProcessedExp.maskfrac_ref_advisory), AVG(camProcessedExp.maskfrac_max_static),AVG(camProcessedExp.maskfrac_max_dynamic),AVG(camProcessedExp.maskfrac_max_advisory) from camRun join camProcessedExp USING(cam_id) JOIN chipRun USING(chip_id) JOIN rawExp USING(exp_id) WHERE camRun.label = 'LAP.PV3.20140730.final' GROUP BY filter;
    … …  
    894893unvignetted field of view results in an average of $\sim 20\%$ masking
    895894fraction across the field of view.  Dynamic masking adds an additional
    896 $2-3\%$, with advisory burntool masking contributing the largest
    897 single component.
     895$2-3\%$ on average, with advisory burntool masking contributing the
     896largest single component.
    898897
    899898\subsection{Overscan}
    … …  
    997996
    998997The dark model we make for GPC1 considers each pixel individually,
    999 independent of any neighbors.  To create the dark model, we fit an
     998independent of any neighbors.  To construct this model, we fit a
    1000999multi-dimensional model to the array of input pixels from a randomly
    10011000selected set of 100-150 overscan and non-linearity corrected dark
    … …  
    10551054these profiles indicates that the average dark model does not correct
    10561055these dates sufficiently, due to the contradictory dark signals
    1057 between the two modes. \czwdraft{this paragraph dependent on that figure.}
     1056between the two modes. \czwdraft{this paragraph dependent on that figure.  This doesn't quite match.}
    10581057
    10591058After 2011-05-01, the two-mode behavior of the dark disappears, and is
    … …  
    11091108can only run video signals on a subset of the OTAs at a given time.
    11101109This requires two passes to enable the video signal across the full
    1111 set of OTAs that support video cells.  This is beneficial to the
     1110set of OTAs that support video cells.  This is convenient for the
    11121111process of creating darks, as those OTAs that do not have video
    11131112signals enabled create standard dark models, while the video dark is
    1114 created for the other devices.
     1113created for those that do.
    11151114
    11161115This simultaneous construction of video and standard dark models is
    … …  
    11671166Unfortunately, due to correlations within this noise, the variance
    11681167measured from the bias images does not fully remove the positional
    1169 dependence of objects that are detected.  The reason for this is that
    1170 this simple noisemap underestimates the noise observed when the image
    1171 is filtered during the object detection process.  This filtering
    1172 convolves the background noise with a PSF, which has the effect of
    1173 amplifying the correlated peaks in the noise.  This amplification can
    1174 therefore boost background fluctuations above the threshold used to
    1175 select real objects, contaminating the final object catalogs.
     1168dependence of objects that are detected.  This simple noisemap
     1169underestimates the noise observed when the image is filtered during
     1170the object detection process.  This filtering convolves the background
     1171noise with a PSF, which has the effect of amplifying the correlated
     1172peaks in the noise.  This amplification can therefore boost background
     1173fluctuations above the threshold used to select real objects,
     1174contaminating the final object catalogs.
    11761175
    11771176In the detection process, we expect false positives at a rate equal to
    … …  
    12361235due to the photometric consistency observed in the final catalog of
    12371236GPC1 measurements \citep{MagnierXXX}, we can be confident that the
    1238 flat model does not have a major time dependent component.
     1237flat model does not have a significant time dependent component.
    12391238
    12401239\subsection{Pattern correction}
    … …  
    12441243dark model, we have a set of ``pattern'' corrections that are applied
    12451244to some selection of the OTAs in the camera.  This is done to reduce
    1246 the effect that detector differences that are not stable enough to be
    1247 corrected with a global model have on the measured astronomical
    1248 signal.  Because these are not stable features that can simply be
    1249 averaged over a large number of inputs, the pattern corrections
    1250 attempt to identify and correct the detector issues based on
    1251 appropriate filtering the individual science exposures.
     1245the effect that detector differences have on the measured astronomical
     1246signal that are not stable enough to be corrected with a static model.
     1247Because of this, the pattern corrections attempt to identify and
     1248correct the detector issues based on appropriate filtering the
     1249individual science exposures.
    12521250
    12531251The PATTERN.ROW correction is used to remove any remaining row-by-row
    … …  
    12591257% http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/GPC1_Bias_Pattern_Study
    12601258As discussed above in the dark and noisemap sections, certain
    1261 detectors have significant row-by-row bias offsets, caused by noise in
    1262 the camera control electronics.  The magnitude of these offsets
    1263 increases as the distance from the readout amplifier increases,
    1264 resulting in horizontal streaks that are more pronounced along the
    1265 large x pixel edge of the cell.  As the level of the offset is
    1266 apparently random between exposures, the dark correction cannot fully
    1267 remove this structure from the images, and the noisemap value only
    1268 indicates the level of the average variance added by these bias
     1259detectors have significant bias offsets between adjacent rows, caused
     1260by noise in the camera control electronics.  The magnitude of these
     1261offsets increases as the distance from the readout amplifier
     1262increases, resulting in horizontal streaks that are more pronounced
     1263along the large x pixel edge of the cell.  As the level of the offset
     1264is apparently random between exposures, the dark correction cannot
     1265fully remove this structure from the images, and the noisemap value
     1266only indicates the level of the average variance added by these bias
    12691267offsets.  Therefore, we apply the PATTERN.ROW correction in an attempt
    12701268to mitigate the offsets and correct the image values.  To force the
    … …  
    12721270the cell.  Four fit iterations are run, and pixels $2.5\sigma$ deviant
    12731271are excluded from subsequent fits, to minimize the effect stars and
    1274 other astronomical signals have.  The final trend is then subtracted
    1275 from the image.  Simply doing this subtraction will also have the
     1272other astronomical signals have.  This final trend is then subtracted
     1273from that row.  Simply doing this subtraction will also have the
    12761274effect of removing the background sky level.  To prevent this, the
    12771275constant and linear terms for each row are stored, and linear fits are
    1278 made to these parameters as a function of row.  This produces a plane
    1279 that is added back to the image to restore the background offset and
    1280 any linear ramp that exists in the sky.
    1281 
     1276made to these parameters as a function of row, perpendicular to the
     1277initial fits.  This produces a plane that is added back to the image
     1278to restore the background offset and any linear ramp that exists in
     1279the sky.
    12821280
    12831281This correction was required on all cells on all OTAs prior to
    … …  
    13501348%  \end{subfigure}
    13511349  \end{minipage}
    1352   \caption{Example of the PATTERN.ROW correction on exposure o5379g0103o OTA57 cell xy00 (i-filter 45s).  The left panel shows the cell with all appropriate detrending except the PATTERN.ROW, and the right shows the same cell with PATTERN.ROW applied.  The correction reduces the correlated noise on the right side, which is most distant from the read out amplifier.  There is a slight over subtraction along the rows near the bright star. \czwdraft{which I can't seem to find proper ranges to highlight.}}
     1350  \caption{Example of the PATTERN.ROW correction on exposure o5379g0103o OTA57 cell xy00 (i-filter 45s).  The left panel shows the cell with all appropriate detrending except the PATTERN.ROW, and the right shows the same cell with PATTERN.ROW applied.  The correction reduces the correlated noise on the right side, which is most distant from the read out amplifier.  There is a slight over subtraction along the rows near the bright star.}
    13531351\end{figure}
    13541352
    … …  
    13921390$\Delta_i = \sum_{j} Edge_{i} - Edge_{j}$, along with a matrix of
    13931391associations $A_{i,i'} = \sum_{j} \delta(i,j) \delta(j,i')$ denoting
    1394 which cell boundaries touch another.  By solving the system $A x =
     1392which cell boundaries are adjacent.  By solving the system $A x =
    13951393diff$, we find the set of offsets $x_i$ to be applied to each cell to
    13961394ensure the minimum differences between all cell edges and their
    … …  
    14211419Due to variations in the thickness of the detectors, we observe
    14221420interference patterns at the infrared end of the filter set, as the
    1423 wavelength of the light becomes comparable to the thickness of these
    1424 variations.  Visually inspecting the images shows that the fringing is
     1421wavelength of the light becomes comparable to the thickness of the
     1422detectors.  Visually inspecting the images shows that the fringing is
    14251423most prevalent in the y-filter images, with negligible fringing in
    14261424other bands.  As a result of this, we only apply a fringe correction
    … …  
    14361434
    14371435A course background model is constructed by calculating the median on
    1438 a 3x3 grid (200x200 pixels each).  A set of 1000 randomly selected
    1439 points are selected on \czwdraft{the final image} in each cell, and
    1440 median calculated for this position in a 10x10 pixel box, and the
    1441 background level subtracted.  These sample locations provide scale
    1442 points to allow the amplitude of the measured fringe to be compared to
    1443 that found on science images.
     1436a 3x3 grid (approximately 200x200 pixels each).  A set of 1000
     1437randomly selected points are selected on the fringe image in each
     1438cell, and a median calculated for this position in a 10x10 pixel box,
     1439with the background level subtracted.  These sample locations provide
     1440scale points to allow the amplitude of the measured fringe to be
     1441compared to that found on science images.
    14441442
    14451443To apply the fringe, the same sample locations are measured on science
    14461444image to determine the relative strength of the fringing in that
    14471445particular image.  A least squares fit between the fringe measurements
    1448 and the corresponding measurements on the science provides the scale
    1449 factor multiplied by the fringe before it is subtracted from the
     1446and the corresponding measurements on the science image provides the
     1447scale factor multiplied to the fringe before it is subtracted from the
    14501448science image.
    14511449
    … …  
    14701468\label{sec:background}
    14711469
    1472 
    14731470Once all other detrending is done, the pixels from each cell are
    14741471mosaicked into the full $4846\times{}4868$ pixel OTA image.  A
    … …  
    15051502projected onto a common set of tangent plane projected regions called
    15061503projection cells.  These projection cells are $4\times{}4$ degree
    1507 fields spaced onto set of centers that fully cover the sky.  They are
     1504fields spaced onto a set of centers that fully cover the sky.  They are
    15081505arranged into rings of constant declination, and allowed to overlap as
    15091506$|\delta|$ increases.  Each projection cell is further subdivided into
    1510 $10\times{}10$ sky cells with fixed $0.25"$ resolution pixels, with
     1507$10\times{}10$ sky cells with fixed $0.25"$ resolution pixels, and
    15111508constant overlap regions between adjacent skycells of $60"$.  These
    15121509skycells are the main image unit used for processing image data beyond
    … …  
    15931590system, they can then be combined pixel-by-pixel regardless of their
    15941591original orientation.  Creating a stacked image by coadding the
    1595 individual warps increases the signal to noise which allows objects
    1596 fainter than can be found on the individual inputs to be detected.
    1597 Creating this stack also allows a complete image to be constructed
    1598 that does not have regions masked due to the gaps between cells and
    1599 OTAs.  This provides a fully populated static sky image that can
    1600 be used for subtraction to find transient sources.
     1592individual warps increases the signal to noise, allowing objects
     1593fainter than the single image signal to noise threshold.  Creating
     1594this stack also allows a complete image to be constructed that does
     1595not have regions masked due to the gaps between cells and OTAs.  This
     1596fully populated static sky image can also be used as a template for
     1597subtraction to find transient sources.
    16011598
    16021599The stacked image is comprised of all warp frames for a given skycell
    … …  
    16701667With the flux normalization factors and target PSF chosen, the
    16711668convolution kernels can be calculated for each image.  ISIS kernels
    1672 are used with FWHM values of 1.5, 3.0, and 6.0 pixels and polynomial
    1673 orders of 6, 4, and 2.  \czwdraft{Skipping this bit because I'm not
    1674   completely sure I understand it.}  The image is then scaled by the
    1675 normalization as $renorm = 10^{-0.4 * norm_{image}} /
    1676 norm_{convolution}$, and the variance by the square of that value.
     1669\citep{ISIS_kernels} are used with FWHM values of 1.5, 3.0, and 6.0
     1670pixels and polynomial orders of 6, 4, and 2.  \czwdraft{Skipping this
     1671  bit because I'm not completely sure I understand it.}  The image is
     1672then scaled by the normalization as $renorm = 10^{-0.4 * norm_{image}}
     1673/ norm_{convolution}$, and the variance by the square of that value.
    16771674
    16781675
    … …  
    17941791warp-warp difference images to be constructed to identify transient
    17951792detections, higher pixel values that come from sources like optical
    1796 ghosts depend on the telescope pointing will come in pairs as well.
     1793ghosts that depend on the telescope pointing will come in pairs as well.
    17971794The higher pixel value contaminants are also potentially problematic
    17981795as they may appear to be real sources, prompting photometry to be
    … …  
    18061803$B$, then a check is made to see if $(0.5 * (value_A - value_B))^2 >
    18071804rej^2 * (variance_A + variance_B + (sys * value_A)^2 + (sys *
    1808 value_B)^2)$, where $rej$ is the number of sigma deviant a point needs
     1805value_B)^2)$, where $rej$ is the number of sigmas deviant a point needs
    18091806to be to be excluded, set to 4.0 for the PV3 processing, and $sys$ is
    18101807an estimate of the systematic error, taken to be 0.1.
    … …  
    19041901determine the largest square box that contains under the limit of
    19051902$0.25 * \sum_{x,y} kernel^2$.  This box is then convolved with the
    1906 rejected pixel mask to reject their neighbors.  This final list of
     1903rejected pixel mask to reject the neighboring pixels.  This final list of
    19071904rejected pixels is passed to the final combination, which creates the
    19081905final stack values from the weighted mean of the non-rejected pixels.
    … …  
    19841981\label{sec:discussion}
    19851982
    1986 \czwdraft{Although the detrending and image combination algorithms
    1987   work well to produce a consistent and calibrated images, having the
    1988   full PV3 data set allows issues to be identified and solutions
    1989   created for future improvements to the IPP pipeline.  In addition,
    1990   the existence of the final calibrated catalog can be used to look
    1991   for issues that appear dependent on focal plane position.}
     1983Although the detrending and image combination algorithms work well to
     1984produce a consistent and calibrated images, having the full PV3 data
     1985set allows issues to be identified and solutions created for future
     1986improvements to the IPP pipeline.  In addition, the existence of the
     1987final calibrated catalog can be used to look for issues that appear
     1988dependent on focal plane position.
    19921989
    19931990An obvious way to make use of the PV3 catalog is to do a statistical
    … …  
    20552052clip this peak to reduce the noise in the image space is not clear.
    20562053
    2057 
    2058 \czwdraft{I need a good concluding thing to say, so it doesn't end with, ``we should do better next time.''}
     2054\section{Conclusion}
     2055
     2056\czwdraft{Not happy with this.}
     2057
     2058The Pan-STARRS1 PV3 processing has reduced an unprecidented volume of
     2059image data, and has produced a catalog of \czwdraft{N} individual
     2060measurements of \czwdraft{Y} astronomical objects.  Accurately
     2061calibrating and detrending is essential to ensuring the quality of
     2062these results.  The detrending process detailed here produces
     2063consistent data, despite the many individual detectors and their
     2064individual response functions.
     2065
     2066From these individual exposures, we are able to construct images on
     2067common projections and orientations, further removing the particulars
     2068of any single exposure.  Furthermore, by created stacked images, we
     2069can determine an estimate of the true static sky, providing a deep
     2070data set that is ideal for use as a template for image differences.
    20592071
    20602072The Pan-STARRS1 Surveys (PS1) have been
Note: See TracChangeset for help on using the changeset viewer.