Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41209) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41234) @@ -2826,12 +2826,12 @@ \section{Conclusion} -We began the development of the IPP in early 2004, soon after the +\textadd{We began the development of the IPP in early 2004, soon after the initial funding for the construction of the Pan-STARRS telescopes was awarded to U.H. The landscape of the software and computing world has changed in a number of ways. Some of the decisions we made at the beginning have held up well while in other cases we would probably -make a different choice today. - -One choice we made early on was to develop new code for the data +make a different choice today. } + +\textadd{One choice we made early on was to develop new code for the data analysis programs. This choice was driven partly by some of our experiences with the existing major systems of the time. We were @@ -2846,7 +2846,7 @@ long-term supportability was also a priority. With these design constraints in mind, we decided to develop a new code base which would -be able to address the data rate and complexity. - -In our design, we have tried to make the analysis programs as generic +be able to address the data rate and complexity.} + +\textadd{In our design, we have tried to make the analysis programs as generic as possible, with all instrument-specific details addressed in the configuration files. Our implementation has been generally successful @@ -2861,13 +2861,13 @@ generalization made is relatively simple to add the second telescope and camera (PS2 + GPC2) to the regular processing when they came -online for science operations in 2018. - -In retrospect, the additional design and coding effort needed to keep +online for science operations in 2018. } + +\textadd{In retrospect, the additional design and coding effort needed to keep the system general were worthwhile and have paid off. However, if we were to start from scratch today, we would probably choose to adapt the LSST pipeline for our use since it has been developed with some of -the same constraints. - -One early choice was to use standard C for analysis programs and to +the same constraints. } + +\textadd{One early choice was to use standard C for analysis programs and to use Perl as a wrapper language. We considered other language choices, including C++ and Python. Our choice of C over C++ has not held up @@ -2880,18 +2880,19 @@ visualization and other high-level analysis options. It is also easier to hire astronomers with good Python coding skills that Perl -coding skills. - -We also find that maintaining support for our Perl code has been a -challenge: changes to the Perl language syntax and changes in -externally supported Perl modules have required significant effort to -keep our code compatible with the changes. It is not obvious that -Python would obviate that particular problem, however. - -One important aspect of the design of the IPP is to use a single +coding skills.} + +\textadd{We also find that maintaining support for our Perl code has + been a challenge: changes to the Perl language syntax and changes in + externally supported Perl modules have required significant effort + to keep our code compatible with the changes. It is not obvious + that Python would obviate that particular problem, however.} + +%% the \textadd{} wrapper breaks the \ippprog{} ability to swallow underscores +\textadd{One important aspect of the design of the IPP is to use a single database to manage the processing stages, with regular queries to the database to choose the tasks which are ready to proceed. Other choices were possible. In some pipelined processing systems, completed jobs trigger the next processing step. For example, -\ippprog{ppImage} or its wrapper (\ippprog{chip_imfile.pl}) could have +\ippprog{ppImage} or its wrapper (\texttt{chip\_imfile.pl}) could have been responsible for launching the \ippprog{psastro} analysis, eliminating the \ippprog{pantasks} manager entirely. @@ -2901,21 +2902,22 @@ complete and launch the \ippprog{psastro} analysis. Both of these choices can potentially result in lower latency since the next step is -in principle run immediately when the previous step is completed. - -Our design choice has two important advantages: First, error and failure -recovery are trivial. If one of the many programs fails or is -interrupted, the system can easily notice and retry the job. In a -triggered system, a failure of one stage could mean the trigger never -happens. Some external cleanup system would need to be implemented to -check for the failures and re-launch. The second advantage of our -design is that each analysis stage is highly independent and can thus -be flexibly run in different ways. For example, alternative test -systems can run in parallel with the nightly operations system, using -the outputs of the nightly processing by simple changes to the queries -used to select the elements for an analysis stage. In addition, it is -easy to add new stages since they do not need to be injected into the -standard processing manager (\ippprog{pantasks}). - -The main challenge related to this database-managed design is that the +in principle run immediately when the previous step is completed.} + +\textadd{Our design choice has two important advantages: First, error + and failure recovery are trivial. If one of the many programs fails + or is interrupted, the system can easily notice and retry the job. + In a triggered system, a failure of one stage could mean the trigger + never happens. Some external cleanup system would need to be + implemented to check for the failures and re-launch. The second + advantage of our design is that each analysis stage is highly + independent and can thus be flexibly run in different ways. For + example, alternative test systems can run in parallel with the + nightly operations system, using the outputs of the nightly + processing by simple changes to the queries used to select the + elements for an analysis stage. In addition, it is easy to add new + stages since they do not need to be injected into the standard + processing manager (\ippprog{pantasks}).} + +\textadd{The main challenge related to this database-managed design is that the database can become a bottleneck. If the queries used to select the processing items become too large and too slow, the whole system can @@ -2928,7 +2930,7 @@ scale of the queries. In addition, it is critical that the database hardware be sufficiently powerful to keep up with the demand from the -processing system. - -Finally, the choice to use Nebulous as a file management system is +processing system.} + +\textadd{Finally, the choice to use Nebulous as a file management system is ambiguous. When we began this project, the existing cluster file systems did not seem to match the level of our project. Some were @@ -2948,5 +2950,5 @@ of data within the cluster. If we were to start from scratch today, it is possible that some of the existing cluster file systems would -address our needs within our budget. +address our needs within our budget.} Since the Pan-STARRS\,1 telescope first came online in 2007, this