Index: /trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41233) +++ /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41234) @@ -2826,12 +2826,12 @@ \section{Conclusion} -We began the development of the IPP in early 2004, soon after the +\textadd{We began the development of the IPP in early 2004, soon after the initial funding for the construction of the Pan-STARRS telescopes was awarded to U.H. The landscape of the software and computing world has changed in a number of ways. Some of the decisions we made at the beginning have held up well while in other cases we would probably -make a different choice today. - -One choice we made early on was to develop new code for the data +make a different choice today. } + +\textadd{One choice we made early on was to develop new code for the data analysis programs. This choice was driven partly by some of our experiences with the existing major systems of the time. We were @@ -2846,7 +2846,7 @@ long-term supportability was also a priority. With these design constraints in mind, we decided to develop a new code base which would -be able to address the data rate and complexity. - -In our design, we have tried to make the analysis programs as generic +be able to address the data rate and complexity.} + +\textadd{In our design, we have tried to make the analysis programs as generic as possible, with all instrument-specific details addressed in the configuration files. Our implementation has been generally successful @@ -2861,13 +2861,13 @@ generalization made is relatively simple to add the second telescope and camera (PS2 + GPC2) to the regular processing when they came -online for science operations in 2018. - -In retrospect, the additional design and coding effort needed to keep +online for science operations in 2018. } + +\textadd{In retrospect, the additional design and coding effort needed to keep the system general were worthwhile and have paid off. However, if we were to start from scratch today, we would probably choose to adapt the LSST pipeline for our use since it has been developed with some of -the same constraints. - -One early choice was to use standard C for analysis programs and to +the same constraints. } + +\textadd{One early choice was to use standard C for analysis programs and to use Perl as a wrapper language. We considered other language choices, including C++ and Python. Our choice of C over C++ has not held up @@ -2880,18 +2880,19 @@ visualization and other high-level analysis options. It is also easier to hire astronomers with good Python coding skills that Perl -coding skills. - -We also find that maintaining support for our Perl code has been a -challenge: changes to the Perl language syntax and changes in -externally supported Perl modules have required significant effort to -keep our code compatible with the changes. It is not obvious that -Python would obviate that particular problem, however. - -One important aspect of the design of the IPP is to use a single +coding skills.} + +\textadd{We also find that maintaining support for our Perl code has + been a challenge: changes to the Perl language syntax and changes in + externally supported Perl modules have required significant effort + to keep our code compatible with the changes. It is not obvious + that Python would obviate that particular problem, however.} + +%% the \textadd{} wrapper breaks the \ippprog{} ability to swallow underscores +\textadd{One important aspect of the design of the IPP is to use a single database to manage the processing stages, with regular queries to the database to choose the tasks which are ready to proceed. Other choices were possible. In some pipelined processing systems, completed jobs trigger the next processing step. For example, -\ippprog{ppImage} or its wrapper (\ippprog{chip_imfile.pl}) could have +\ippprog{ppImage} or its wrapper (\texttt{chip\_imfile.pl}) could have been responsible for launching the \ippprog{psastro} analysis, eliminating the \ippprog{pantasks} manager entirely. @@ -2901,21 +2902,22 @@ complete and launch the \ippprog{psastro} analysis. Both of these choices can potentially result in lower latency since the next step is -in principle run immediately when the previous step is completed. - -Our design choice has two important advantages: First, error and failure -recovery are trivial. If one of the many programs fails or is -interrupted, the system can easily notice and retry the job. In a -triggered system, a failure of one stage could mean the trigger never -happens. Some external cleanup system would need to be implemented to -check for the failures and re-launch. The second advantage of our -design is that each analysis stage is highly independent and can thus -be flexibly run in different ways. For example, alternative test -systems can run in parallel with the nightly operations system, using -the outputs of the nightly processing by simple changes to the queries -used to select the elements for an analysis stage. In addition, it is -easy to add new stages since they do not need to be injected into the -standard processing manager (\ippprog{pantasks}). - -The main challenge related to this database-managed design is that the +in principle run immediately when the previous step is completed.} + +\textadd{Our design choice has two important advantages: First, error + and failure recovery are trivial. If one of the many programs fails + or is interrupted, the system can easily notice and retry the job. + In a triggered system, a failure of one stage could mean the trigger + never happens. Some external cleanup system would need to be + implemented to check for the failures and re-launch. The second + advantage of our design is that each analysis stage is highly + independent and can thus be flexibly run in different ways. For + example, alternative test systems can run in parallel with the + nightly operations system, using the outputs of the nightly + processing by simple changes to the queries used to select the + elements for an analysis stage. In addition, it is easy to add new + stages since they do not need to be injected into the standard + processing manager (\ippprog{pantasks}).} + +\textadd{The main challenge related to this database-managed design is that the database can become a bottleneck. If the queries used to select the processing items become too large and too slow, the whole system can @@ -2928,7 +2930,7 @@ scale of the queries. In addition, it is critical that the database hardware be sufficiently powerful to keep up with the demand from the -processing system. - -Finally, the choice to use Nebulous as a file management system is +processing system.} + +\textadd{Finally, the choice to use Nebulous as a file management system is ambiguous. When we began this project, the existing cluster file systems did not seem to match the level of our project. Some were @@ -2948,5 +2950,5 @@ of data within the cluster. If we were to start from scratch today, it is possible that some of the existing cluster file systems would -address our needs within our budget. +address our needs within our budget.} Since the Pan-STARRS\,1 telescope first came online in 2007, this Index: unk/doc/release.2015/ps1.datasystem/response.txt =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/response.txt (revision 41233) +++ (revision ) @@ -1,8 +1,0 @@ - -## Section 2.4 - -Is the Distribution and Publication system mentioned in the text -supposed to be part of Figure 1, either as an umbrella term or a -(missing?) component? - -* the rounded circles on the bottom are all data distribution elements. Index: /trunk/doc/release.2015/ps1.datasystem/response.v1.txt =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/response.v1.txt (revision 41233) +++ /trunk/doc/release.2015/ps1.datasystem/response.v1.txt (revision 41234) @@ -1,3 +1,49 @@ +** We are resubmitting our article "The Pan-STARRS Data Processing + System" after addressing suggestions raised by the referee. We + thank the referee for detailed comments and suggestions. Below are + our responses to the referee's suggestions. (Our responses are + preceeded by "**") + +# General Notes + +This is a well-written and important technical paper that succeeds +admirably at what I consider the most important goal of any pipeline +paper: providing a decription of the processing steps that are +relevant for downstream science users (in this case, by providing the +big picture that ties together a number of more detailed papers). With +only a handful of minor cleanups (see detailed notes below), I think +the paper is ready for publication, and most of my comments represent +ideas for improvement that I hope the authors will consider (but +should not feel obliged to act on). + +My only general concern is that the paper often misses the opportunity +to pass on lessons learned to the developers of future pipelines, and +this makes much of the detailed description of how the PS1 systems +work (particularly in Section 5) feel like it belongs more in operator +documentation rather than an article like this one. I suspect a small +amounof additional historical context - how different systems evolved +over the course of the survey - and commentary on what worked well and +what was a regular pain point would go a long way. + +In particular, the described system seems to involve a both fair +amount of duplication (e.g. multiple databases, sky-tiling systems, +and task orchestration layers) and a number of in-house solutions to +what seem like fairly general problems (the DVO database and +especially the pantask/opihi system stand out in this regard). This is +not intended as criticism; I am quite aware that there are many good +reasons for both duplication and keeping central components in-house, +from deliberately keeping components loosely coupled to taking into +account the often-brief shelf-life of off-the-shelf solutions, +especially as compared to the duration of a major astronomical +survey. But describing *which* of many potential reasons actually +played a role in each of various design choices (and which, if any, +look less good in hindsight) would make the paper much more +interesting. + +** We have greatly expanded the Conclusion to address these questions, + and to identify choices we made which either turned out well or + which we would have done differently given changes to the software + landscape. # Detailed Notes