Changeset 41206

Timestamp:

Dec 16, 2019, 3:52:41 PM (7 years ago)

Author:

eugene

Message:

updates with referee responses

Location:

trunk/doc/release.2015/ps1.datasystem

Files:

• Makefile (modified) (2 diffs)
• datasystem.tex (modified) (9 diffs)
• response.v1.txt (modified) (1 diff)

trunk/doc/release.2015/ps1.datasystem/Makefile

@@ -2 +2 @@
 
 DO_PDFLATEX = 1
-DO_BIBTEX = 0
+DO_BIBTEX = 1
 
 help:
@@ -21 +21 @@
 ../inputs/code.sty \
 ../inputs/apj.bst \
-PS1_Data_Analysis_System_Overview.pdf \
+flowchart.v1.pdf \
 datasystem.tex
 

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

@@ -288 +288 @@
 \begin{figure*}[htbp]
   \begin{center}
- \includegraphics[width=\hsize,clip]{PS1_Data_Analysis_System_Overview.pdf}
+ \includegraphics[width=\hsize,clip]{{flowchart.v1}.pdf}
   \caption{\label{fig:analysis.elements} Elements of the Pan-STARRS\,1
     Data Analysis System.  Rectangles represent data analysis steps;
@@ -574 +574 @@
 For GPC1, the \ippstage{registration} stage is also the stage at which
 the \ippprog{burntool} analysis is run.  This analysis is more
-completely described in Paper III.  In brief, the
-\ippprog{burntool} program identifies bright sources on the image, and
-identifies persistence trails that result from the incomplete transfer
-of charge.  As this charge can leak out in subsequent exposures, the
-burntool analysis is run sequentially on the exposures, based on the
+completely described in Paper III.  In brief, the \ippprog{burntool}
+program identifies bright sources on the image, and identifies
+persistence trails that result from the incomplete transfer of charge.
+As this charge can leak out in subsequent exposures, the burntool
+analysis is run sequentially on the exposures, based on the
 observation date and time listed in the headers, with the results
 stored on disk.  As a result of the sequential nature of this
 analysis, the \ippstage{registration} of exposures is blocked until
 the \ippprog{burntool} has been run on the previous exposures.
+\textadd{Because this stage is only run once per exposure, changes to
+  the burntool code require a semi-manual re-running of the analysis
+  outside of the regular processing sequence.  Since this is a rare
+  event, a standardized pipeline infrastructure was not developed for
+  this circumstance.  In a future re-organization, a standard
+  serialized pre-processing step may be needed in the pipeline.  }
 
 Once the \ippstage{registration} process has finished, new science
@@ -700 +706 @@
 individual chips is performed, including a fit to a single model for
 the distortion introduced by the camera optics.  The astrometric model
-includes a set of 3rd order polynomials for the transformations from the chip
-coordinate system to the camera focal plane coordinate system and a
-single additional 3rd order polynomial transformation from the camera focal
-plane coordinate system to the tangent plane of a tangent projection.
+includes a set of 3rd order polynomials for the transformations from
+the chip coordinate system to the camera focal plane coordinate system
+and a single additional 3rd order polynomial transformation from the
+camera focal plane coordinate system to the tangent plane of a tangent
+projection.
+
+\textadd{As discussed in detail in Paper V, We find that, for the PS1
+  images, small-scale structures are present in the astrometric
+  transformation.  Some of these are due to ripples in the focal
+  surface, while others may be caused by the atmosphere.  We find that
+  including higher-order terms in both the chip-to-focal plane and
+  focal-plane to sky are necessary to capture significant astrometric
+  signals.  Some care must be taken in the fitting process to avoid
+  degeneracies between terms on different scales.}
+
 For the $3\pi$ PV3 analysis, the typical astrometric residuals are in
 the range of 20 - 30 milliarcseconds, sufficient to match observations
@@ -728 +745 @@
 \ippstage{camera} stage also generates the dynamic masks for the
 images.  These include masking for optical ghosts, glints, saturated
-stars, diffraction spikes, and electronic crosstalk.  The mask images
+stars, diffraction spikes, and electronic crosstalk.  \textadd{The mask
+information is generated based on the reference star catalog, along
+with models for the various effects.  Note however that this analysis does not
+go back to the pixels to validate the prediction.}  The mask images
 generated by the \ippstage{chip} stage are updated with these dynamic
 masks and a new set of files are saved for the downstream analysis
@@ -847 +867 @@
 In the IPP processing, stacks may be made with various options for the
 input images.  During nightly science processing, the 8 exposures per
-filter for each Medium Deep field are combined into a set of stacks
-for that field.  These so-called ``nightly stacks'' are used by the
-transient survey projects to detect faint supernovae, among other
-transient events.  For the PV3 $3\pi$ analysis, all images in each
-filter from the observations for this survey were stacked together to
-generate a single set of images with $\sim 10 - 20\times$ the exposure
-of the individual survey exposures.  
+filter for each Medium Deep field are \textadd{automatically} combined
+into a set of stacks for that field.  These so-called ``nightly
+stacks'' are used by the transient survey projects to detect faint
+supernovae, among other transient events.  For the PV3 $3\pi$
+analysis, all images in each filter from the observations for this
+survey were stacked together to generate a single set of images with
+$\sim 10 - 20\times$ the exposure of the individual survey exposures.
 
 For the PV3 processing of the Medium Deep fields, stacks have been
@@ -868 +888 @@
 When a given set of \ippstage{stack} stage processing is defined,
 exposures with existing \ippstage{warp} entries that match the filter,
-position, and other criteria such as seeing are identified.  An entry
+position, and other criteria such as seeing are identified \textadd{(see
+Section~\ref{sec:automation} to see how this is automated)}.  An entry
 is then added for each skycell in the \ippdbtable{stackRun} table,
 with the \ippdbcolumn{warp_id} entries for the exposures added to the
@@ -1103 +1124 @@
 \subsection{Processing Failure Rates}
 
-Table~\ref{tab:failure_rates} lists the unrecoverable failure rates
+\textadd{Table~\ref{tab:failure_rates} lists the unrecoverable failure rates
 for several of the major IPP stages for both the regular nightly
 processing and the PV3 analysis of the $3\pi$ dataset.  The table
 gives the rate per 100,000 of the item processed.  In the case of the
-\ippstage{camera} stage, the items correspond to complete exposures,
-while for \ippstage{chip} and \ippstage{warp}, the items correspond to
-individual chips and skycells, respectively.  For \ippstage{stack},
-items are the full stack.  For the \ippstage{camera} stage, the entire
-exposure fails only in extreme cases.  The astrometric calibration of
-individual chips may fail if there are not enough stars in the image,
-but the rest of the exposure may then still succeed.
-
-For the warp analysis, the apparent high failure rate is an artifact
-of two features.  First, 
-
-
-\begin{table*}
+\ippstage{chip} and \ippstage{warp} stages, the items correspond to
+individual chips and skycells, respectively, while for the
+\ippstage{stack} stage, items are the stack skycells.  For the
+\ippstage{camera} stage, the items correspond to complete exposures.
+The entire exposure fails for \ippstage{camera} only in extreme cases.
+The astrometric calibration of individual chips may fail if there are
+not enough stars in the image, but the rest of the exposure may then
+still succeed.  Chips which formally succeed in the astrometry
+analysis but which have an astrometric calibration quality worse than
+our specification will also be excluded from ingest into the DVO
+database (see below).  We list the astrometry failure rate for chips
+based on their absence from the DVO database.}
+
+\textadd{For the warp analysis, the apparent high failure rate is something of
+an artifact.  Target output skycells are defined based on 
+conservatively generous boundaries for the corresponding chips.  This
+results in a number of skycells with only a small fraction of valid
+pixels, for which there are likely few stars to measure the PSF.  In
+the processing, any warp skycell with less than 10\% of its pixels
+unmasked in the output are automatically rejected.  In addition, the
+analysis will register a poor quality if too few stars are available
+for the PSF modelling.  To judge the rate at which the warp stage is
+losing pixels, either due to this effect or veritable analysis
+failures, we compare the total area of good (unmasked) pixels in the
+warp skyfiles to the total number of expected unmasked pixels from the
+corresponding input exposures using the masking fractions and total
+detector areas reported in Paper III.  The result is that roughly
+3.9\% of the good input pixels are lost to the warp processing.}
+
+\begin{table}
 \begin{center}
 \caption{Processing Failure Rates per 100,000 Items\label{tab:failure_rates}}
@@ -1129 +1167 @@
 Chip & 48 & 34 \\
 Camera & 262 & 280 \\
-Chip Astrometry & N/A & 307 \\
+~~~Chip Astrom & N/A & 307 \\
 Warp & 14244 & 13835 \\
+~~~Warp Pixels & N/A & 3900 \\
 Stack & N/A & 5 \\
 \hline
@@ -2762 +2801 @@
 
 \bibliographystyle{apj}
-%\bibliography{lib}{}
-\input{datasystem.bbl}
+\bibliography{lib}{}
+%\input{datasystem.bbl}
 
 \end{document}

trunk/doc/release.2015/ps1.datasystem/response.v1.txt

@@ -12 +12 @@
    the bottom and added a line to show where the distribution and
    publication mechanisms interface to these customers.
+
+## Section 3.1
+
+This is by no means necessary, but I'm curious to see a table or
+discussion of what fraction of jobs of various types failed with bad
+"quality". In other words, how much data could you not get through the
+pipelines at all, and what was the most sensitive step?
+
+** We liked this suggestion and added a subsection 3.12 and a new
+   table (2) to discuss the failure rates.
+
+## Section 3.3
+
+Running "Registration" only once for each exposure would seem to
+prohibit re-running "burntool" after updating the algorithm for that -
+and I'm guessing you didn't get that fully stabilized until after you
+had already processed some images and learned from thr experience. How
+did that work?
+
+** We added a couple of sentences to explain that we used a
+   semi-manual task to re-run just the burntool analysis during
+   development and if the code ever needs to be changed.
+
+## Section 3.5
+
+Why a 3rd-order polynomial from chip to focal plane? Wouldn't an
+affine transform have been sufficient (and more than that degenerate
+with the focal plane to sky transform)?
+
+** We use the higher-order transformation for each chip to capture the
+   small-scale astrometric signal present in the data.  One could use
+   an afine transformation for chip-to-focal plane and capture the
+   same signal in a much higher-order model for focal-plane to sky,
+   but that was not our development path.  These would be equivalent
+   solutions.  (Note that degeneracies exist in both cases).   We
+   avoid the degeneracy of the chip positions in the focal plane
+   solution by fitting the local gradient to get the initial
+   distortion solution (and there are certain terms which are held
+   fixed for the focal plane.)  We then limit the impact of the
+   degeneracy by fitting the two levels independently and fixing the
+   focal-plane solution after a few iterations.
+
+   We have added some words to explain some of this, but leave the
+   details to Paper IV.
+
+What makes the masks generated in this step "dynamic"? Are they
+generated wholly from the reference catalog (i.e.  predicting where a
+ghost will appear based on the position of a bright star)? It seems
+like the CAMERA step does not utilize any of the pixel data (just the
+pixel-level masks from CHIP). Is that correct?
+
+  ** correct: the dynamic masks are generated from the reference
+     catalog and do not go back to the original pixels.  We added a
+     paragraph to clarify.
+
+
+## Section 3.8
+
+Is the selection of which warped images go into a stack driven by
+human operators, or are there automated systems to launch these jobs,
+too?
+
+  ** section 5.2 discusses how both the nightly stacks and
+     large-scale reprocessing campaign stacks are automatically
+     defined.  We added some words to refer to this section in 3.8.
+
+## Section 3.10
+
+How much of the PSF-convolved galaxy models do you re-fit in forced
+photometry? If you're fitting more than just the amplitude at that
+stage, and considering each exposure as independent, you're
+potentially throwing away a lot of S/N (at least in the many-exposure
+limit), even if you average later. If you're just fitting the
+amplitude, the structural parameters are still going to be the ones
+affected by poor PSFs in the stack.
+
+  **