Index: trunk/doc/release.2015/inputs/astro.sty =================================================================== --- trunk/doc/release.2015/inputs/astro.sty (revision 41207) +++ trunk/doc/release.2015/inputs/astro.sty (revision 41208) @@ -81,6 +81,9 @@ \newcommand\TBD[1]{\par\tbd{#1}\par} -\newcommand\textadd[1]{\textbf{\color{red}#1}} -\newcommand\textmod[1]{\textbf{\color{blue}#1}} +%\newcommand\textadd[1]{\textbf{\color{red}#1}} +%\newcommand\textmod[1]{\textbf{\color{blue}#1}} + +\newcommand\textadd[1]{\textbf{#1}} +\newcommand\textmod[1]{\textbf{#1}} \def\ippprog{\IPPprog} Index: trunk/doc/release.2015/inputs/code.sty =================================================================== --- trunk/doc/release.2015/inputs/code.sty (revision 41207) +++ trunk/doc/release.2015/inputs/code.sty (revision 41208) @@ -43,5 +43,5 @@ \def\IPPdbtable{\begingroup\setupc@de\IPPdbtableEND}% -\def\IPPdbcolumnEND#1{\textit{\textbf{#1}}\endgroup}% +\def\IPPdbcolumnEND#1{\textit{#1}\endgroup}% \def\IPPdbcolumn{\begingroup\setupc@de\IPPdbcolumnEND} Index: trunk/doc/release.2015/inputs/lib.bib =================================================================== --- trunk/doc/release.2015/inputs/lib.bib (revision 41207) +++ trunk/doc/release.2015/inputs/lib.bib (revision 41208) @@ -16845,2 +16845,29 @@ } +@ARTICLE{2008ApJ...674.1217P, + author = {{Padmanabhan}, Nikhil and {Schlegel}, David J. and + {Finkbeiner}, Douglas P. and {Barentine}, J.~C. and + {Blanton}, Michael R. and {Brewington}, Howard J. and {Gunn}, James E. and + {Harvanek}, Michael and {Hogg}, David W. and + {Ivezi{\'c}}, {\v{Z}}eljko and {Johnston}, David and + {Kent}, Stephen M. and {Kleinman}, S.~J. and {Knapp}, Gillian R. and + {Krzesinski}, Jurek and {Long}, Dan and {Neilsen}, Eric H., Jr. and + {Nitta}, Atsuko and {Loomis}, Craig and {Lupton}, Robert H. and + {Roweis}, Sam and {Snedden}, Stephanie A. and {Strauss}, Michael A. and + {Tucker}, Douglas L.}, + title = "{An Improved Photometric Calibration of the Sloan Digital Sky Survey Imaging Data}", + journal = {\apj}, + keywords = {techniques: photometric, Astrophysics}, + year = "2008", + month = "Feb", + volume = {674}, + number = {2}, + pages = {1217-1233}, + doi = {10.1086/524677}, +archivePrefix = {arXiv}, + eprint = {astro-ph/0703454}, + primaryClass = {astro-ph}, + adsurl = {https://ui.adsabs.harvard.edu/abs/2008ApJ...674.1217P}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} + Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41207) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41208) @@ -228,4 +228,13 @@ PV3 data release, with some details on the scale of computing needed to reduce this large number of exposures. + +In this article, we use the following type-faces to distinguish +different concepts: +\begin{itemize} +\item \ippstage{Small caps} for the analysis stages. +\item \ippdbtable{Italics} for database tables and columns. +\item \ippprog{Fixed-width} font for program names, variables, and + miscellaneous constants. +\end{itemize} \section{Overview of Pan-STARRS Data Processing} @@ -889,5 +898,5 @@ exposures with existing \ippstage{warp} entries that match the filter, position, and other criteria such as seeing are identified \textadd{(see -Section~\ref{sec:automation} to see how this is automated)}. An entry +Section~\ref{sec:automation} for details on how this is automated)}. An entry is then added for each skycell in the \ippdbtable{stackRun} table, with the \ippdbcolumn{warp_id} entries for the exposures added to the @@ -1060,50 +1069,24 @@ images.} In this analysis, the galaxy models determined by the \ippstage{staticsky} photometry analysis are used to seed the analysis -in the individual \ippstage{warp} images. - -The analysis tests a grid of galaxy model parameters in the vicinity -of the prior from the stack. For each warp image, each parameter set -is used to generate a model which is then convolved with the PSF for -that warp image and then compared to the observed data. The resulting -grid of $\chi^2$ values can then be for the - -For each object, a grid of galaxy model parameters is used compared tested on each -warp image and - -** we calculate a normalization and chisq for each warp grid point. -the chi square values can be summed across warps to give the solution -chi square dist. for a single warp, the error on Io goes like -sqrt(Ncnts). The average Io value is the weighted averages of the -inputs. The error on the weighted average is sqrt(1 / (sum(1/sigma^2))). - -S_t^2 = 1 / (1 / S_0^2 + 1 / S_1^2 + 1 / S_2^2) - -S_0^2 = N0, S_1^2 = N1, etc - -S_t^2 = 1 / (1/N0 + 1/N1 + 1/N2 ...) - -ideal: S_t^2 = N0 + N1 + N2 - -dI / Io = 1 / sqrt(No) - -The error on the - -\textmod{For each warp - image, the galaxy model (convolved with the PSF) is compared to the - observed pixels to calculate an element of the total model $\chi^2$ value - - a grid -of the galaxy model parameters are - -how does the error scale if I fit each Io for each warp vs a single -value? (sounds like I do kill the S/N...) - -%%%%% fix all of this... - -The purpose of this -analysis is the same as the \ippstage{fullforce} PSF photometry: the -PSF of the \ippstage{stack} image is poorly determined due to the -masking and PSF variations in the inputs. Without a good PSF model, -the PSF-convolved galaxy models are of limited accuracy. +in the individual \ippstage{warp} images. Galaxy models are {\em not} +fitted independently on each warp. Rather, the analysis tests a grid +of galaxy model parameters in the vicinity of the prior from the +stack. + +\textadd{For each warp image, each set of galaxy model parameter values is used +to generate a model which is then convolved with the PSF for that warp +image and then compared to the observed data. A normalization and +$\chi^2$ value is determied for each set of parameter values for each +warp image. For each set of parameter values, the normalizations and +$\chi^2$ values are combined across all warps to generate a single +grid of parameters. The best set of galaxy model parameters, along +with the corresponding normalizaiton and $\chi^2$ value is then +determined from the grid by interpolation. } + +The purpose of this galaxy model analysis is the same as the +\ippstage{fullforce} PSF photometry: the PSF of the \ippstage{stack} +image is poorly determined due to the masking and PSF variations in +the inputs. Without a good PSF model, the PSF-convolved galaxy models +are of limited accuracy. Upon completion of the forced photometry, an entry is added to the @@ -1120,5 +1103,5 @@ analysis measurements into a single value. The output catalogs listed in the \ippdbtable{fullForceResult} table are passed to the -\ippprog{psphotFullForceSummary} to calculate the averages of the +\ippprog{psphotFullForceSummary} program to calculate the averages of the individual warp measurements, weighted by the signal-to-noise of the flux measurements. When this analysis completes, an entry is added to @@ -1163,4 +1146,58 @@ eventual public released. +When a \ippstage{diff} processing is defined, an entry is added to the +\ippdbtable{diffRun} table, and the appropriate input images are added +to the \ippdbtable{diffInputSkyfile} table, with one entry for each +skycell that is covered by the images. For a \ippstage{diff} +generated from two \ippstage{warp} stage products, the input images +have their \ippdbcolumn{warp_id} values recorded in the +\ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that +overlaps. If two \ippstage{stack} stages are to be used in the +difference, their \ippdbcolumn{stack_id} entries are recorded in the +\ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each +\ippstage{stack} only covers a single skycell, the \ippstage{diff} is +usually defined indirectly, using other information from the +\ippdbtable{stackRun} table to select appropriate +\ippdbcolumn{stack_id} values. Similarly, \ippstage{diff} processing +is defined for the mixed case by creating entries that populate one of +\ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of +\ippdbcolumn{warp2} and \ippdbcolumn{stack2}. In all cases, the +minuend of the subtraction to be performed is the ``1'' entry, and the +subtrahend is the ``2'' entry. + +Jobs are created based on the entries of +\ippdbtable{diffInputSkyfile}, with the appropriate images and +catalogs passed to the \ippprog{ppSub} program. This does the +subtraction, as well as the photometry of any sources detected in the +\ippstage{diff} image. Sources may be detected as a positive source +(flux in the minuend is higher than the subtrahend) or as a negative +source (flux in the subtrahend is higher). The algorithm used for PSF +matching is described in Paper III. Upon completion of these +jobs, statistics about the processing are written to an entry in the +\ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the +completion of all of the components listed in +\ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun} +entry as such. + +\begin{table} +\begin{center} +\caption{Processing Failure Rates per 100,000 Items\label{tab:failure_rates}} +\begin{tabular}{lrr} +\hline +\hline +{\bf Stage} & {\bf Nightly} & {\bf $3\pi$} \\ + & {\bf Processing} & {\bf PV3} \\ +\hline +Chip & 48 & 34 \\ +Camera & 262 & 280 \\ +~~~Chip Astrom & N/A & 307 \\ +Warp & 14244 & 13835 \\ +~~~Warp Pixels & N/A & 3900 \\ +Stack & N/A & 5 \\ +\hline +\end{tabular} +\end{center} +\end{table} + \subsection{Processing Failure Rates} @@ -1197,23 +1234,4 @@ detector areas reported in Paper III. The result is that roughly 3.9\% of the good input pixels are lost to the warp processing.} - -\begin{table} -\begin{center} -\caption{Processing Failure Rates per 100,000 Items\label{tab:failure_rates}} -\begin{tabular}{lll} -\hline -\hline -{\bf Stage} & {\bf Nightly Processing} & {\bf $3\pi$ PV3} \\ -\hline -Chip & 48 & 34 \\ -Camera & 262 & 280 \\ -~~~Chip Astrom & N/A & 307 \\ -Warp & 14244 & 13835 \\ -~~~Warp Pixels & N/A & 3900 \\ -Stack & N/A & 5 \\ -\hline -\end{tabular} -\end{center} -\end{table} \begin{table*} @@ -1240,38 +1258,4 @@ \end{center} \end{table*} - -When a \ippstage{diff} processing is defined, an entry is added to the -\ippdbtable{diffRun} table, and the appropriate input images are added -to the \ippdbtable{diffInputSkyfile} table, with one entry for each -skycell that is covered by the images. For a \ippstage{diff} -generated from two \ippstage{warp} stage products, the input images -have their \ippdbcolumn{warp_id} values recorded in the -\ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that -overlaps. If two \ippstage{stack} stages are to be used in the -difference, their \ippdbcolumn{stack_id} entries are recorded in the -\ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each -\ippstage{stack} only covers a single skycell, the \ippstage{diff} is -usually defined indirectly, using other information from the -\ippdbtable{stackRun} table to select appropriate -\ippdbcolumn{stack_id} values. Similarly, \ippstage{diff} processing -is defined for the mixed case by creating entries that populate one of -\ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of -\ippdbcolumn{warp2} and \ippdbcolumn{stack2}. In all cases, the -minuend of the subtraction to be performed is the ``1'' entry, and the -subtrahend is the ``2'' entry. - -Jobs are created based on the entries of -\ippdbtable{diffInputSkyfile}, with the appropriate images and -catalogs passed to the \ippprog{ppSub} program. This does the -subtraction, as well as the photometry of any sources detected in the -\ippstage{diff} image. Sources may be detected as a positive source -(flux in the minuend is higher than the subtrahend) or as a negative -source (flux in the subtrahend is higher). The algorithm used for PSF -matching is described in Paper III. Upon completion of these -jobs, statistics about the processing are written to an entry in the -\ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the -completion of all of the components listed in -\ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun} -entry as such. \section{Database Ingest and Calibration} @@ -1676,5 +1660,5 @@ Within the PSPS, the \ippdbtable{Detection} table carries an ID which is unique for each measurement, equivalent to the DVO -\ippdbcolumn{det_id}, \ippdbcolumn{image_id} pair. In this case, the +\ippdbcolumn{detID}, \ippdbcolumn{imageID} pair. In this case, the PSPS \ippdbcolumn{detectID} is constructed from the MJD of the exposure, the number of the OTA (e.g., OTA64), and the detection @@ -1756,7 +1740,7 @@ The construction of the master DVO is performed in a hierarchical -fashion. The individual catalogs are added to a \ippmisc{minidvo}, +fashion. The individual catalogs are added to a mini-DVO, which is simply a DVO database defined over some subset of possible -inputs. These \ippmisc{minidvos} are then merged by stage into larger +inputs. These mini-DVOs are then merged by stage into larger databases to construct a single master DVO database. In the process, an intermediate master DVO for each stage is generated. The @@ -1770,7 +1754,7 @@ WISE telescope. -As of PV3, the process of merging \ippmisc{minidvos} is not highly +As of PV3, the process of merging mini-DVOs is not highly automated, requiring manual attention. The generation of the -\ippmisc{minidvos} is automated and managed by the \ippstage{addstar} +mini-DVOs is automated and managed by the \ippstage{addstar} stage. Each catalog that is to be added to DVO has an entry created in the \ippdbtable{addRun} database table. This entry notes which @@ -1781,5 +1765,5 @@ created, with the \ippdbcolumn{stage_extra1} field containing an index to the individual components. The catalog specified by the entry is -added to the target \ippmisc{minidvo} by the \ippprog{addstar} +added to the target mini-DVO by the \ippprog{addstar} program, updating the measurements in the appropriate DVO tables. When this completes, an entry containing the statistics of the job is @@ -1831,18 +1815,18 @@ exposures which were believed to be obtained in photometric conditions. This process, called ``\"ubercal'', is described in -detail by \cite{2012ApJ...756..158S} for the first (PV1) version -\note{add SDSS ref mentioned in Schlafly, also in cal paper}. In -brief, photometric periods, with time-scales of a large fraction of a -night, are identified by a combination of automatic analysis and -manual inspection. A single solution for all images in a given filter -is determined to minimize scatter for individual stars. The free -parameters in this solution consist of a single zero point and airmass -slope for each photometric period along with a collection of -flat-field offsets for several large time range (``flat-field -seasons''). For the PV3 \"ubercal analysis, the flat-field offsets -were determined on a $2\times2$ grid for each chip and 5 flat-field -seasons were identified. The boundaries of the flat-field seasons -were determined by independent inspection of the residuals observed in -the Medium Deep fields. +detail by \cite{2012ApJ...756..158S} for the first (PV1) version \textadd{and +is based on the process of the same name used for SDSS calibration +\citep{2008ApJ...674.1217P}}. In brief, photometric periods, with +time-scales of a large fraction of a night, are identified by a +combination of automatic analysis and manual inspection. A single +solution for all images in a given filter is determined to minimize +scatter for individual stars. The free parameters in this solution +consist of a single zero point and airmass slope for each photometric +period along with a collection of flat-field offsets for several large +time range (``flat-field seasons''). For the PV3 \"ubercal analysis, +the flat-field offsets were determined on a $2\times2$ grid for each +chip and 5 flat-field seasons were identified. The boundaries of the +flat-field seasons were determined by independent inspection of the +residuals observed in the Medium Deep fields. After the \"ubercal analysis of the photometric periods is completed, @@ -1874,9 +1858,18 @@ Telescopes (MAST). The underlying database at MAST is a copy of a database generated at the IfA by the Published Science Products -Subsystem (PSPS). The construction of the PSPS version of the PS1 -database starts once the PS1 photometry and astrometry measurements -have been calibrated within the DVO system. The construction takes -place in several stages, described in detail in Paper VI. -We summarize those steps here. +Subsystem (PSPS). \textadd{Both MAST and IfA versions of the PSPS are +implemented using a collection of Microsoft SQL Server instances as +the underlying database engine. Like in DVO, the tables holding the +large volume of measurements are distributed across the different +computers based on their location on the sky. Unlike DVO, the spatial +distribution uses slices which span all RA values for a narrow range +of Declinations on a single compter. The PSPS design and +implementation is described in some detail in Paper VI.} + +The construction of the PSPS version of the PS1 database starts once +the PS1 photometry and astrometry measurements have been calibrated +within the DVO system. The construction takes place in several +stages, described in detail in Paper VI. We summarize those steps +here. The first stage of constructing the PSPS database consists of the @@ -1982,5 +1975,5 @@ Within the \code{task.exec} macro, the command to be run is defined by -the script. Once the \code{task.exec} macro exits successfully, the +the script. Once the \code{task.exec} macro \mbox{exits} successfully, the defined command is then added to the list of jobs to be run within the UNIX environment. Jobs may be run in one of two ways: locally or via @@ -2139,5 +2132,5 @@ Most stages consist of two related tasks: a \ippmisc{load} task, which -is responsible to querying the processing database to identify entries +is responsible for querying the processing database to identify entries to be processed, and a \ippmisc{run} task, which is responsible for managing the processing of the individual entries. @@ -2234,6 +2227,4 @@ from other processing attempts. - - \subsection{Stage automation} \label{sec:automation} @@ -2250,15 +2241,16 @@ \ippmisc{ippScript}. These scripts have a well-defined and restricted set of goals: to ensure that difference images are generated for each -exposure (either by pairing together warps or pairs warps with +exposure (either by pairing together warps or pairing warps with pre-defined stacks), that nightly stacks are generated for MD fields, -and that the stacks are also differenced against an appropriate -reference. - -Pairing warps together is simplified by the observing strategy in -which the same pointing is observed multiple times in a night. By -limiting to warp-warp pairs from the same pointing, the problem is -significantly reduced from the arbitrary case. - -Queuing the diffs is done by first examining the set of all +and that the nightly stacks are also differenced against an appropriate +reference. + +\textmod{For the warp-warp difference images, pairing warps together is +simplified} by the observing strategy in which the same pointing is +observed multiple times in a night. By limiting to warp-warp pairs +from the same pointing, the problem is significantly reduced from the +arbitrary case. + +Queuing \textmod{these warp-warp difference images} is done by first examining the set of all exposures that have been taken at the summit on the current night of observing, and querying information from each stage up through @@ -2267,7 +2259,9 @@ identifier for each telescope pointing on the sky. Exposures in each group are then sorted by increasing observation date -(\ippdbcolumn{dateobs}). The database results for each stage -(\ippstage{chip}-\ippstage{warp}) are checked to ensure that the selected exposures have -been successfully processed for all stages through \ippstage{warp}. +(\ippdbcolumn{dateobs}). + +The database results for each stage (\ippstage{chip}-\ippstage{warp}) +are checked to ensure that the selected exposures have been +successfully processed for all stages through \ippstage{warp}. Exposure groups are ignored until all exposures have either been processed through warp or have failed with a bad quality, meaning the @@ -2276,5 +2270,8 @@ the final exposure ignored in the case of an odd number of accepted exposures. Exposures paired in this way are sent to the -ippstage{diff} analysis stage. +\ippstage{diff} analysis stage. \textadd{Nightly processing also + ensures that the difference image analysis is run using the warps in + comparison to the reference stack images generated for the full $3\pi$ + region.} Once observations have been completed for the night (signaled by the @@ -2299,6 +2296,8 @@ number of usable exposures. If no stack could be made for a given MD field with the minimum number of inputs by the time of the -end-of-night darks, stacks are generated using whatever -exposures are available. +end-of-night darks, stacks are generated using whatever exposures are +available. \textadd{Nightly processing also ensures that the + difference image analysis is run on these nightly stacks using a + pre-defined reference stack.} The automatic nightly processing ensures that data is processed as @@ -2306,19 +2305,22 @@ observation and the reduced data. -The other processing task that requires automation is the reprocessing -of the entire $3\pi$ survey, as the size of the dataset make it -challenging to do manually. To manage this, the ``large area -processing'' (LAP) task and script are used. The first stage of this -processing is generating an entry in the \ippdbtable{lapSequence} -table defining a new reprocessing. After this, individual -\ippdbtable{lapRun} entries can be queued that define a -\ippdbcolumn{filter} and a \ippdbcolumn{projection_cell} to be -considered. These projection cells match the tangent plane centers -used for the warp tessellation. A \ippdbcolumn{projection_cell} is a -unit of sky defined to be a square four degrees on each side which has -a single tangent plane projection (Paper III). -Once this -entry is defined, it is populated with all exposures (stored in the -\ippdbtable{lapExp} table in the database) that are located +{\bf The other processing task that requires automation is the reprocessing +of the entire $3\pi$ survey, as the size of the dataset makes it +challenging to organize the analysis manually. To manage large-scale +analyses, the ``large area processing'' (LAP) task and script are +used. The first stage of LAP generates an entry in the +\ippdbtable{lapSequence} table defining a new reprocessing. After +this, individual \ippdbtable{lapRun} entries can be queued that define +a \ippdbcolumn{filter} and a \ippdbcolumn{projection_cell} to be +considered. These projection cells corrrespond to the projections used +by the warp tessellation to define the skycells (see +Section~\ref{sec:warp}), which tangent plane centers matching those in +the warp tessellation. For the $3\pi$ survey analysis, a +\ippdbcolumn{projection_cell} is a unit of sky defined to be a square +four degrees on each side which has a single tangent plane projection +(Paper III).} + +Once this entry is defined, it is populated with all exposures (stored +in the \ippdbtable{lapExp} table in the database) that are located within 5 degrees of the center of the projection cell included. This radius ensures that any exposure that overlaps the projection cell @@ -2381,4 +2383,11 @@ Pan-STARRS cluster. +All of the IPP low-level C-based processing programs (e.g., +\ippprog{ppImage} and \ippprog{ppStack} interact with Nebulous to find +existing files and to create new files. The supporting Perl scripts +also interact with Nebulous to perform file instance duplication as +needed and to check for the existence of required input files and +expected output files. + \subsubsection{Implementation Details} @@ -2479,11 +2488,11 @@ impact processing. -The nebulous user APIs do not interact directly with the nebulous +The nebulous user APIs do not interact directly with the Nebulous mysql database. Instead, they interact with one of several computers with an Apache web server. Interactions with the Apache server are performed using the Simple Object Access Protocol (SOAP) interface, -while the Apache servers interact directly with the Mysql database +while the Apache servers interact directly with the mysql database server. This architecture avoids the overhead of setting up and -tearing down the Mysql connection for each Nebulous command, instead +tearing down the mysql connection for each Nebulous command, instead using only the low-latency SOAP communications. @@ -2679,5 +2688,8 @@ servers used as database replicants, which allow for quick switching from the main to backup servers in case of a hardware issue that -cannot be resolved immediately. +cannot be resolved immediately. \textadd{The IPP uses a set of three + computers to host the Nebulous mysql database and live back-ups. A + second set of computers are used to host the processing database and + backups.} \subsection{Los Alamos National Labs} @@ -2813,4 +2825,122 @@ \section{Conclusion} + +We began the development of the IPP in early 2004, soon after the +initial funding for the construction of the Pan-STARRS telescopes was +awarded to U.H. The landscape of the software and computing world has +changed in a number of ways. Some of the decisions we made at the +beginning have held up well while in other cases we would probably +make a different choice today. + +One choice we made early on was to develop new code for the data +analysis programs. This choice was driven partly by some of our +experiences with the existing major systems of the time. We were +advised by those with close experience with the SDSS data analysis +code base against attempting to modify that system for our purposes. +It was also our opinion that the IRAF suite of packages were not +well-suited to the large-scale automated pipeline needed for the +Pan-STARRS data. The Pan-STARRS data analysis rate was going to +surpass previous astronomical projects, and the cameras (with 60 +detectors each of 64 cells) would have an unprecedented level of +complexity. The original survey was intended to run for 10 years, so +long-term supportability was also a priority. With these design +constraints in mind, we decided to develop a new code base which would +be able to address the data rate and complexity. + +In our design, we have tried to make the analysis programs as generic +as possible, with all instrument-specific details addressed in the +configuration files. Our implementation has been generally successful +in this regard. The \ippprog{ppImage} program contains most of the +highly-specific detrending details, with much more limited +camera-specific features needed in the configuration files for +\ippprog{psastro} and \ippprog{pswarp}. This generalization of the +software has made it easy to run the full analysis pipeline on other +cameras, both for testing and for other science analysis projects. We +have used the full IPP analysis system for data from the CFHT Megacam +and CFH12K cameras as well as the Subaru Hypersuprime Camera. The +generalization made is relatively simple to add the second telescope +and camera (PS2 + GPC2) to the regular processing when they came +online for science operations in 2018. + +In retrospect, the additional design and coding effort needed to keep +the system general were worthwhile and have paid off. However, if we +were to start from scratch today, we would probably choose to adapt +the LSST pipeline for our use since it has been developed with some of +the same constraints. + +One early choice we made was to use standard C and to use Perl as a +wrapper language. We considered other language choices, including C++ +and Python. At the time, Python was fairly new and did not have the +wide-spread acceptance it has today. In retrospect, our choice of +Perl has not held up very well. The capabiliaties available within +the Python environment would have allowed us to include interesting +visualization and other high-level analysis options. It is also +easier to hire astronomers with good Python coding skills that Perl +coding skills. + +We also find that maintaining support for our Perl code has been a +challenge: changes to the Perl language syntax and changes in +externally supported Perl modules have required significant effort to +keep our code compatible with the changes. It is not obvious that +Python would obviate that particular problem, however. + +One important aspect of the design of the IPP is to use a single +database to manage the processing stages, with regular queries to the +database to choose the tasks which are ready to proceed. Other +choices were possible. In some pipelined processing systems, jobs +which complete trigger the next processing step. For example, +\ippprog{ppImage} or its wrapper (\ippprog{chip_imfile.pl}) could have +been responsible for launching the \ippprog{psastro} analysis. +Alternatively, a manager process could be responsible for launching +the next processing step when one step has completed. For example, +\ippprog{pantasks} could note when the \ippprog{ppImage} jobs were +complete and launch the \ippprog{psastro} analysis. Both of these +choices can potentially result in lower latency since the next step is +in principle run immediately when the previous step is completed. Our +choice has two important advantages: First, error and failure recovery +are trivial. If one of the many programs fails or is interrupted, the +system can easily notice and retry the job. In a triggered system, a +failure of one stage could mean the trigger never happens. Some +external cleanup system would need to be implemented to check for the +failures and re-launch. The second advantage of our design is that +each analysis stage is highly independent and can thus be flexibly run +in different ways. For example, alternative test systems can run in +parallel with the nightly operations system, using the outputs of the +nightly processing by simple changes to the queries used to select the +elements for an analysis stage. In addition, it is easy to add new +stages since they do not need to be injected into the standard +processing manager (\ippprog{pantasks}). + +The main challenge related to this database-managed design is that the +database can become a bottleneck. If the queries used to select the +processing items become too large and too slow, the whole system can +be slowed down. Care must the taken to avoid poorly implemented +queries, and in some cases the queries need to be restricted. For +example, if too many items are queued for processing at one time under +the same processing label, the associated queries can bog down. This +issue is one of the reasons we manage the large-scale processing with +the LAP system since it provides a method to automatically limit the +scale of the queries. In addition, it is critical that the database +hardware be sufficiently powerful to keep up with the demand from the +processing system. + +Finally, the choice to use Nebulous as a file management system is +ambiguous. When we began this project, the existing cluster file +systems did not seem to match the level of our project. Some were +will very much in an early development state (e.g., GFS from Red Hat), +while others seemed designed for much larger-scale systems, with very +expensive hardware requirements (e.g., Lustre). The requirements for +the filesystem for Pan-STARRS are somewhat different from the +large-scale computing clusters used by the national labs. Since the +data processing is very parallel, we do not have any strong +requirements on data access concurency. In theory, we could have +simply used NFS and made backup copies of the files using some simple +name-convention rules. We decided to implement the Nebulous system to +allow the targetted analysis and to automate the replication of the +data. In retrospect, the system has succeeded in these goals and has +behaved reliably. However, the support level has been somewhat high, +especially when we have needed to migrate large amounts of data within +the cluster. If we were to start from scratch today, we would +experiment with some of the existing cluster file systems. Since the Pan-STARRS\,1 telescope first came online in 2007, this Index: trunk/doc/release.2015/ps1.datasystem/response.v1.txt =================================================================== --- trunk/doc/release.2015/ps1.datasystem/response.v1.txt (revision 41207) +++ trunk/doc/release.2015/ps1.datasystem/response.v1.txt (revision 41208) @@ -88,6 +88,15 @@ affected by poor PSFs in the stack. - ** th - + ** the galaxy models are not fitted on each warp. rather we + calculate the normalizations and chi-square values for a grid of + galaxy model shape parameters for each warp image. The values + for each grid point are combined across all warps to generate a + total stack-equivalent grid. At this point, the best parameters + are determined from the grid (interpolating to the chi-square + minimum). This is mathematically equivalent to simultaneously + fitting (via a grid search) the pixels from all warps to a single + model, preserving the full signal-to-noise. We have updated the + text to add some detail to the description of what is being + measured to clarify this point. ## Section 3.11 @@ -107,6 +116,131 @@ aren't described until 4.1.3. + ** we added a sentence to 4.1.1 to note this point. + Missing punctuation in parenthetical HST GSC reference? ** fixed +## Section 4.1.4 + +There's some inconsistency here between "detID" and "det_id" (same for +"image"), both referring to measurement IDs in DVO. If those are +supposed to be meaningfully different, I'm confused. + + ** in the DVO section (and in the DVO schema), these should all be + 'detID' and 'imageID'. In the gpc1 database schema, the + underscored versions are used. we have fixed the erroneous + det_id and image_id entries in this section. + +## Section 4.2 + +I tend to associate the term "ubercal" specifically with the SDSS +version of the algorithm that coined the term, and think it probably +should be referenced here even if the actual algorithms used are only +vaguely similar. + + ** we agree and have added a sentence with reference. + +## Section 4.3 + +Is the PSPS database another spatially-shared, file-based database +using custom technology, a MySQL database like the Processing +Database, or something else? I assume the same system is used at both +IFA and MAST? + + ** PSPS is based on MS SQL Server. We have added a bit of + description to 4.3. + + +## Section 5.1.1 + +Apparent typo or missing text: macro ex- its job successfuly". + + ** this should have read 'macro exits successfully' ("exits" was + beign hyphenated). fixed. + +## Section 5.1.4 + +"responsible to" -> "responsible for" + + ** fixed + +## Section 5.2 + +> Pairing warps together is simplified by the observing strategy in +which the same pointing is observed multiple times in a night. By +limiting to warp-warp pairs from the same pointing, the problem is +significantly reduced from the arbitrary case. + +This (as well as the following paragraph) seems to imply that you +typically generate differences between images taken in the same night, +which of course limits you to detecting only very short-timescale +transients and fast-moving objects. I suspect that's just not what you +intended to imply, or is the nightly processing really not supposed to +find e.g. supernovae? + + ** the wording here was unclear that the nightly processing system + generates warp-warp difference images (for asteroids), warp-stack + difference images (for 3pi supernovae), and MD nightly stack - + reference stacks difference images (for deep MD supernovae). We + have updated the text to explain these differences. + +## Section 5.2 + +Are the `projection_cells` described here the same as or related to +the DVO partition cells of 4.1.3, or the RINGS.V3 skycells of 3.7? + + ** same as RINGS.V3. we have clarified this and also cleaned up the + wording of this paragraph. + +This is a more general concept, but it came to a head in this section: +I found the use of so many notation styles for different concepts more +distracting than helpful. I think I was able to infer that small caps +were used for processing stages and non-bold italics were used for +database tables, but it wasn't clear why some other stages were +written in fixed-width mixed case instead (were these scripts, rather +than stages?), or what the use of bold-italic meant (everything +eles?). I'd recommend either adding a notation legend paragraph early +in the paper or just cutting down on the number of styles used. + + ** We agree and have simplified the typography a bit (using only aa + single face for both db tables and db columns), eliminating the + use of boldface. We have also added a paragraph in the + introduction section to define the type faces. + +## Section 5.3 + +Was Nebulous just used by the orchestration levels like pantasks, or +was it used within the Perl scripts and C programs that constitute the +algorithmic steps as well? + + ** Nebulous is used by any level of the software that needs access + to a specific file. the c-based processing programs have direct + interfaces as do the Perl-based wrappers (ippScripts). We have + added a paragraph to explain this. + +Was the database used by Nebulous integrated with the Processing +Database at all (or even part of the same server)? + + ** these two databases are on separate machines and kept + independent. A sentence was added to the end of 6.1 to note + this. + +It's a bit strange to first encounter what seems like a core part of +the data access system this late in the description, given that it +would have needed to be updated by all of the processing steps +mentioned early. This would of course make more sense if Nebulous is +in fact used by the lowest levels of the pipeline and hence a Nebulous +database entry is created whenever a file is written to disk. + + ** our organizational scheme is meant to place the details closest + to the science analysis up front and leave the more general + systems toward the end, with only a few necessary broad concepts + introduced early on for context. Thus section 3 is about the + analysis steps and the related programs, section 4 is about the + science database and the calibration, section 5 is more generic + operations concepts, and section 6 is the computing hardware. + Within section 5, the processing organization comes first, while + nebulous is left to the end since it seems (to us) to be very + general and should not be driving the science decisions. +