Index: /trunk/doc/release.2015/ps1.datasystem/Makefile =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/Makefile (revision 41205) +++ /trunk/doc/release.2015/ps1.datasystem/Makefile (revision 41206) @@ -2,5 +2,5 @@ DO_PDFLATEX = 1 -DO_BIBTEX = 0 +DO_BIBTEX = 1 help: @@ -21,5 +21,5 @@ ../inputs/code.sty \ ../inputs/apj.bst \ -PS1_Data_Analysis_System_Overview.pdf \ +flowchart.v1.pdf \ datasystem.tex Index: /trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41205) +++ /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41206) @@ -288,5 +288,5 @@ \begin{figure*}[htbp] \begin{center} - \includegraphics[width=\hsize,clip]{PS1_Data_Analysis_System_Overview.pdf} + \includegraphics[width=\hsize,clip]{{flowchart.v1}.pdf} \caption{\label{fig:analysis.elements} Elements of the Pan-STARRS\,1 Data Analysis System. Rectangles represent data analysis steps; @@ -574,13 +574,19 @@ For GPC1, the \ippstage{registration} stage is also the stage at which the \ippprog{burntool} analysis is run. This analysis is more -completely described in Paper III. In brief, the -\ippprog{burntool} program identifies bright sources on the image, and -identifies persistence trails that result from the incomplete transfer -of charge. As this charge can leak out in subsequent exposures, the -burntool analysis is run sequentially on the exposures, based on the +completely described in Paper III. In brief, the \ippprog{burntool} +program identifies bright sources on the image, and identifies +persistence trails that result from the incomplete transfer of charge. +As this charge can leak out in subsequent exposures, the burntool +analysis is run sequentially on the exposures, based on the observation date and time listed in the headers, with the results stored on disk. As a result of the sequential nature of this analysis, the \ippstage{registration} of exposures is blocked until the \ippprog{burntool} has been run on the previous exposures. +\textadd{Because this stage is only run once per exposure, changes to + the burntool code require a semi-manual re-running of the analysis + outside of the regular processing sequence. Since this is a rare + event, a standardized pipeline infrastructure was not developed for + this circumstance. In a future re-organization, a standard + serialized pre-processing step may be needed in the pipeline. } Once the \ippstage{registration} process has finished, new science @@ -700,8 +706,19 @@ individual chips is performed, including a fit to a single model for the distortion introduced by the camera optics. The astrometric model -includes a set of 3rd order polynomials for the transformations from the chip -coordinate system to the camera focal plane coordinate system and a -single additional 3rd order polynomial transformation from the camera focal -plane coordinate system to the tangent plane of a tangent projection. +includes a set of 3rd order polynomials for the transformations from +the chip coordinate system to the camera focal plane coordinate system +and a single additional 3rd order polynomial transformation from the +camera focal plane coordinate system to the tangent plane of a tangent +projection. + +\textadd{As discussed in detail in Paper V, We find that, for the PS1 + images, small-scale structures are present in the astrometric + transformation. Some of these are due to ripples in the focal + surface, while others may be caused by the atmosphere. We find that + including higher-order terms in both the chip-to-focal plane and + focal-plane to sky are necessary to capture significant astrometric + signals. Some care must be taken in the fitting process to avoid + degeneracies between terms on different scales.} + For the $3\pi$ PV3 analysis, the typical astrometric residuals are in the range of 20 - 30 milliarcseconds, sufficient to match observations @@ -728,5 +745,8 @@ \ippstage{camera} stage also generates the dynamic masks for the images. These include masking for optical ghosts, glints, saturated -stars, diffraction spikes, and electronic crosstalk. The mask images +stars, diffraction spikes, and electronic crosstalk. \textadd{The mask +information is generated based on the reference star catalog, along +with models for the various effects. Note however that this analysis does not +go back to the pixels to validate the prediction.} The mask images generated by the \ippstage{chip} stage are updated with these dynamic masks and a new set of files are saved for the downstream analysis @@ -847,11 +867,11 @@ In the IPP processing, stacks may be made with various options for the input images. During nightly science processing, the 8 exposures per -filter for each Medium Deep field are combined into a set of stacks -for that field. These so-called ``nightly stacks'' are used by the -transient survey projects to detect faint supernovae, among other -transient events. For the PV3 $3\pi$ analysis, all images in each -filter from the observations for this survey were stacked together to -generate a single set of images with $\sim 10 - 20\times$ the exposure -of the individual survey exposures. +filter for each Medium Deep field are \textadd{automatically} combined +into a set of stacks for that field. These so-called ``nightly +stacks'' are used by the transient survey projects to detect faint +supernovae, among other transient events. For the PV3 $3\pi$ +analysis, all images in each filter from the observations for this +survey were stacked together to generate a single set of images with +$\sim 10 - 20\times$ the exposure of the individual survey exposures. For the PV3 processing of the Medium Deep fields, stacks have been @@ -868,5 +888,6 @@ When a given set of \ippstage{stack} stage processing is defined, exposures with existing \ippstage{warp} entries that match the filter, -position, and other criteria such as seeing are identified. An entry +position, and other criteria such as seeing are identified \textadd{(see +Section~\ref{sec:automation} to see how this is automated)}. An entry is then added for each skycell in the \ippdbtable{stackRun} table, with the \ippdbcolumn{warp_id} entries for the exposures added to the @@ -1103,21 +1124,38 @@ \subsection{Processing Failure Rates} -Table~\ref{tab:failure_rates} lists the unrecoverable failure rates +\textadd{Table~\ref{tab:failure_rates} lists the unrecoverable failure rates for several of the major IPP stages for both the regular nightly processing and the PV3 analysis of the $3\pi$ dataset. The table gives the rate per 100,000 of the item processed. In the case of the -\ippstage{camera} stage, the items correspond to complete exposures, -while for \ippstage{chip} and \ippstage{warp}, the items correspond to -individual chips and skycells, respectively. For \ippstage{stack}, -items are the full stack. For the \ippstage{camera} stage, the entire -exposure fails only in extreme cases. The astrometric calibration of -individual chips may fail if there are not enough stars in the image, -but the rest of the exposure may then still succeed. - -For the warp analysis, the apparent high failure rate is an artifact -of two features. First, - - -\begin{table*} +\ippstage{chip} and \ippstage{warp} stages, the items correspond to +individual chips and skycells, respectively, while for the +\ippstage{stack} stage, items are the stack skycells. For the +\ippstage{camera} stage, the items correspond to complete exposures. +The entire exposure fails for \ippstage{camera} only in extreme cases. +The astrometric calibration of individual chips may fail if there are +not enough stars in the image, but the rest of the exposure may then +still succeed. Chips which formally succeed in the astrometry +analysis but which have an astrometric calibration quality worse than +our specification will also be excluded from ingest into the DVO +database (see below). We list the astrometry failure rate for chips +based on their absence from the DVO database.} + +\textadd{For the warp analysis, the apparent high failure rate is something of +an artifact. Target output skycells are defined based on +conservatively generous boundaries for the corresponding chips. This +results in a number of skycells with only a small fraction of valid +pixels, for which there are likely few stars to measure the PSF. In +the processing, any warp skycell with less than 10\% of its pixels +unmasked in the output are automatically rejected. In addition, the +analysis will register a poor quality if too few stars are available +for the PSF modelling. To judge the rate at which the warp stage is +losing pixels, either due to this effect or veritable analysis +failures, we compare the total area of good (unmasked) pixels in the +warp skyfiles to the total number of expected unmasked pixels from the +corresponding input exposures using the masking fractions and total +detector areas reported in Paper III. The result is that roughly +3.9\% of the good input pixels are lost to the warp processing.} + +\begin{table} \begin{center} \caption{Processing Failure Rates per 100,000 Items\label{tab:failure_rates}} @@ -1129,6 +1167,7 @@ Chip & 48 & 34 \\ Camera & 262 & 280 \\ -Chip Astrometry & N/A & 307 \\ +~~~Chip Astrom & N/A & 307 \\ Warp & 14244 & 13835 \\ +~~~Warp Pixels & N/A & 3900 \\ Stack & N/A & 5 \\ \hline @@ -2762,6 +2801,6 @@ \bibliographystyle{apj} -%\bibliography{lib}{} -\input{datasystem.bbl} +\bibliography{lib}{} +%\input{datasystem.bbl} \end{document} Index: /trunk/doc/release.2015/ps1.datasystem/response.v1.txt =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/response.v1.txt (revision 41205) +++ /trunk/doc/release.2015/ps1.datasystem/response.v1.txt (revision 41206) @@ -12,2 +12,79 @@ the bottom and added a line to show where the distribution and publication mechanisms interface to these customers. + +## Section 3.1 + +This is by no means necessary, but I'm curious to see a table or +discussion of what fraction of jobs of various types failed with bad +"quality". In other words, how much data could you not get through the +pipelines at all, and what was the most sensitive step? + +** We liked this suggestion and added a subsection 3.12 and a new + table (2) to discuss the failure rates. + +## Section 3.3 + +Running "Registration" only once for each exposure would seem to +prohibit re-running "burntool" after updating the algorithm for that - +and I'm guessing you didn't get that fully stabilized until after you +had already processed some images and learned from thr experience. How +did that work? + +** We added a couple of sentences to explain that we used a + semi-manual task to re-run just the burntool analysis during + development and if the code ever needs to be changed. + +## Section 3.5 + +Why a 3rd-order polynomial from chip to focal plane? Wouldn't an +affine transform have been sufficient (and more than that degenerate +with the focal plane to sky transform)? + +** We use the higher-order transformation for each chip to capture the + small-scale astrometric signal present in the data. One could use + an afine transformation for chip-to-focal plane and capture the + same signal in a much higher-order model for focal-plane to sky, + but that was not our development path. These would be equivalent + solutions. (Note that degeneracies exist in both cases). We + avoid the degeneracy of the chip positions in the focal plane + solution by fitting the local gradient to get the initial + distortion solution (and there are certain terms which are held + fixed for the focal plane.) We then limit the impact of the + degeneracy by fitting the two levels independently and fixing the + focal-plane solution after a few iterations. + + We have added some words to explain some of this, but leave the + details to Paper IV. + +What makes the masks generated in this step "dynamic"? Are they +generated wholly from the reference catalog (i.e. predicting where a +ghost will appear based on the position of a bright star)? It seems +like the CAMERA step does not utilize any of the pixel data (just the +pixel-level masks from CHIP). Is that correct? + + ** correct: the dynamic masks are generated from the reference + catalog and do not go back to the original pixels. We added a + paragraph to clarify. + + +## Section 3.8 + +Is the selection of which warped images go into a stack driven by +human operators, or are there automated systems to launch these jobs, +too? + + ** section 5.2 discusses how both the nightly stacks and + large-scale reprocessing campaign stacks are automatically + defined. We added some words to refer to this section in 3.8. + +## Section 3.10 + +How much of the PSF-convolved galaxy models do you re-fit in forced +photometry? If you're fitting more than just the amplitude at that +stage, and considering each exposure as independent, you're +potentially throwing away a lot of S/N (at least in the many-exposure +limit), even if you average later. If you're just fitting the +amplitude, the structural parameters are still going to be the ones +affected by poor PSFs in the stack. + + **