Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41204) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41206) @@ -288,5 +288,5 @@ \begin{figure*}[htbp] \begin{center} - \includegraphics[width=\hsize,clip]{PS1_Data_Analysis_System_Overview.pdf} + \includegraphics[width=\hsize,clip]{{flowchart.v1}.pdf} \caption{\label{fig:analysis.elements} Elements of the Pan-STARRS\,1 Data Analysis System. Rectangles represent data analysis steps; @@ -574,13 +574,19 @@ For GPC1, the \ippstage{registration} stage is also the stage at which the \ippprog{burntool} analysis is run. This analysis is more -completely described in Paper III. In brief, the -\ippprog{burntool} program identifies bright sources on the image, and -identifies persistence trails that result from the incomplete transfer -of charge. As this charge can leak out in subsequent exposures, the -burntool analysis is run sequentially on the exposures, based on the +completely described in Paper III. In brief, the \ippprog{burntool} +program identifies bright sources on the image, and identifies +persistence trails that result from the incomplete transfer of charge. +As this charge can leak out in subsequent exposures, the burntool +analysis is run sequentially on the exposures, based on the observation date and time listed in the headers, with the results stored on disk. As a result of the sequential nature of this analysis, the \ippstage{registration} of exposures is blocked until the \ippprog{burntool} has been run on the previous exposures. +\textadd{Because this stage is only run once per exposure, changes to + the burntool code require a semi-manual re-running of the analysis + outside of the regular processing sequence. Since this is a rare + event, a standardized pipeline infrastructure was not developed for + this circumstance. In a future re-organization, a standard + serialized pre-processing step may be needed in the pipeline. } Once the \ippstage{registration} process has finished, new science @@ -700,8 +706,19 @@ individual chips is performed, including a fit to a single model for the distortion introduced by the camera optics. The astrometric model -includes a set of 3rd order polynomials for the transformations from the chip -coordinate system to the camera focal plane coordinate system and a -single additional 3rd order polynomial transformation from the camera focal -plane coordinate system to the tangent plane of a tangent projection. +includes a set of 3rd order polynomials for the transformations from +the chip coordinate system to the camera focal plane coordinate system +and a single additional 3rd order polynomial transformation from the +camera focal plane coordinate system to the tangent plane of a tangent +projection. + +\textadd{As discussed in detail in Paper V, We find that, for the PS1 + images, small-scale structures are present in the astrometric + transformation. Some of these are due to ripples in the focal + surface, while others may be caused by the atmosphere. We find that + including higher-order terms in both the chip-to-focal plane and + focal-plane to sky are necessary to capture significant astrometric + signals. Some care must be taken in the fitting process to avoid + degeneracies between terms on different scales.} + For the $3\pi$ PV3 analysis, the typical astrometric residuals are in the range of 20 - 30 milliarcseconds, sufficient to match observations @@ -728,5 +745,8 @@ \ippstage{camera} stage also generates the dynamic masks for the images. These include masking for optical ghosts, glints, saturated -stars, diffraction spikes, and electronic crosstalk. The mask images +stars, diffraction spikes, and electronic crosstalk. \textadd{The mask +information is generated based on the reference star catalog, along +with models for the various effects. Note however that this analysis does not +go back to the pixels to validate the prediction.} The mask images generated by the \ippstage{chip} stage are updated with these dynamic masks and a new set of files are saved for the downstream analysis @@ -847,11 +867,11 @@ In the IPP processing, stacks may be made with various options for the input images. During nightly science processing, the 8 exposures per -filter for each Medium Deep field are combined into a set of stacks -for that field. These so-called ``nightly stacks'' are used by the -transient survey projects to detect faint supernovae, among other -transient events. For the PV3 $3\pi$ analysis, all images in each -filter from the observations for this survey were stacked together to -generate a single set of images with $\sim 10 - 20\times$ the exposure -of the individual survey exposures. +filter for each Medium Deep field are \textadd{automatically} combined +into a set of stacks for that field. These so-called ``nightly +stacks'' are used by the transient survey projects to detect faint +supernovae, among other transient events. For the PV3 $3\pi$ +analysis, all images in each filter from the observations for this +survey were stacked together to generate a single set of images with +$\sim 10 - 20\times$ the exposure of the individual survey exposures. For the PV3 processing of the Medium Deep fields, stacks have been @@ -868,5 +888,6 @@ When a given set of \ippstage{stack} stage processing is defined, exposures with existing \ippstage{warp} entries that match the filter, -position, and other criteria such as seeing are identified. An entry +position, and other criteria such as seeing are identified \textadd{(see +Section~\ref{sec:automation} to see how this is automated)}. An entry is then added for each skycell in the \ippdbtable{stackRun} table, with the \ippdbcolumn{warp_id} entries for the exposures added to the @@ -1103,21 +1124,38 @@ \subsection{Processing Failure Rates} -Table~\ref{tab:failure_rates} lists the unrecoverable failure rates +\textadd{Table~\ref{tab:failure_rates} lists the unrecoverable failure rates for several of the major IPP stages for both the regular nightly processing and the PV3 analysis of the $3\pi$ dataset. The table gives the rate per 100,000 of the item processed. In the case of the -\ippstage{camera} stage, the items correspond to complete exposures, -while for \ippstage{chip} and \ippstage{warp}, the items correspond to -individual chips and skycells, respectively. For \ippstage{stack}, -items are the full stack. For the \ippstage{camera} stage, the entire -exposure fails only in extreme cases. The astrometric calibration of -individual chips may fail if there are not enough stars in the image, -but the rest of the exposure may then still succeed. - -For the warp analysis, the apparent high failure rate is an artifact -of two features. First, - - -\begin{table*} +\ippstage{chip} and \ippstage{warp} stages, the items correspond to +individual chips and skycells, respectively, while for the +\ippstage{stack} stage, items are the stack skycells. For the +\ippstage{camera} stage, the items correspond to complete exposures. +The entire exposure fails for \ippstage{camera} only in extreme cases. +The astrometric calibration of individual chips may fail if there are +not enough stars in the image, but the rest of the exposure may then +still succeed. Chips which formally succeed in the astrometry +analysis but which have an astrometric calibration quality worse than +our specification will also be excluded from ingest into the DVO +database (see below). We list the astrometry failure rate for chips +based on their absence from the DVO database.} + +\textadd{For the warp analysis, the apparent high failure rate is something of +an artifact. Target output skycells are defined based on +conservatively generous boundaries for the corresponding chips. This +results in a number of skycells with only a small fraction of valid +pixels, for which there are likely few stars to measure the PSF. In +the processing, any warp skycell with less than 10\% of its pixels +unmasked in the output are automatically rejected. In addition, the +analysis will register a poor quality if too few stars are available +for the PSF modelling. To judge the rate at which the warp stage is +losing pixels, either due to this effect or veritable analysis +failures, we compare the total area of good (unmasked) pixels in the +warp skyfiles to the total number of expected unmasked pixels from the +corresponding input exposures using the masking fractions and total +detector areas reported in Paper III. The result is that roughly +3.9\% of the good input pixels are lost to the warp processing.} + +\begin{table} \begin{center} \caption{Processing Failure Rates per 100,000 Items\label{tab:failure_rates}} @@ -1129,6 +1167,7 @@ Chip & 48 & 34 \\ Camera & 262 & 280 \\ -Chip Astrometry & N/A & 307 \\ +~~~Chip Astrom & N/A & 307 \\ Warp & 14244 & 13835 \\ +~~~Warp Pixels & N/A & 3900 \\ Stack & N/A & 5 \\ \hline @@ -2762,6 +2801,6 @@ \bibliographystyle{apj} -%\bibliography{lib}{} -\input{datasystem.bbl} +\bibliography{lib}{} +%\input{datasystem.bbl} \end{document}