Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41208) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 41209) @@ -2869,10 +2869,13 @@ the same constraints. -One early choice we made was to use standard C and to use Perl as a -wrapper language. We considered other language choices, including C++ -and Python. At the time, Python was fairly new and did not have the -wide-spread acceptance it has today. In retrospect, our choice of -Perl has not held up very well. The capabiliaties available within -the Python environment would have allowed us to include interesting +One early choice was to use standard C for analysis programs and to +use Perl as a wrapper language. We considered other language choices, +including C++ and Python. Our choice of C over C++ has not held up +well: we would have done well to have the modern object-oriented +features of C++, some aspect of which we have imitated in our C coding +style. The choice of Perl over Python has also fared poorly. At the +time, Python was fairly new and did not have the wide-spread +acceptance it has today. The capabiliaties available within the +Python environment would have allowed us to include interesting visualization and other high-level analysis options. It is also easier to hire astronomers with good Python coding skills that Perl @@ -2888,8 +2891,9 @@ database to manage the processing stages, with regular queries to the database to choose the tasks which are ready to proceed. Other -choices were possible. In some pipelined processing systems, jobs -which complete trigger the next processing step. For example, +choices were possible. In some pipelined processing systems, +completed jobs trigger the next processing step. For example, \ippprog{ppImage} or its wrapper (\ippprog{chip_imfile.pl}) could have -been responsible for launching the \ippprog{psastro} analysis. +been responsible for launching the \ippprog{psastro} analysis, +eliminating the \ippprog{pantasks} manager entirely. Alternatively, a manager process could be responsible for launching the next processing step when one step has completed. For example, @@ -2897,18 +2901,19 @@ complete and launch the \ippprog{psastro} analysis. Both of these choices can potentially result in lower latency since the next step is -in principle run immediately when the previous step is completed. Our -choice has two important advantages: First, error and failure recovery -are trivial. If one of the many programs fails or is interrupted, the -system can easily notice and retry the job. In a triggered system, a -failure of one stage could mean the trigger never happens. Some -external cleanup system would need to be implemented to check for the -failures and re-launch. The second advantage of our design is that -each analysis stage is highly independent and can thus be flexibly run -in different ways. For example, alternative test systems can run in -parallel with the nightly operations system, using the outputs of the -nightly processing by simple changes to the queries used to select the -elements for an analysis stage. In addition, it is easy to add new -stages since they do not need to be injected into the standard -processing manager (\ippprog{pantasks}). +in principle run immediately when the previous step is completed. + +Our design choice has two important advantages: First, error and failure +recovery are trivial. If one of the many programs fails or is +interrupted, the system can easily notice and retry the job. In a +triggered system, a failure of one stage could mean the trigger never +happens. Some external cleanup system would need to be implemented to +check for the failures and re-launch. The second advantage of our +design is that each analysis stage is highly independent and can thus +be flexibly run in different ways. For example, alternative test +systems can run in parallel with the nightly operations system, using +the outputs of the nightly processing by simple changes to the queries +used to select the elements for an analysis stage. In addition, it is +easy to add new stages since they do not need to be injected into the +standard processing manager (\ippprog{pantasks}). The main challenge related to this database-managed design is that the @@ -2928,19 +2933,20 @@ ambiguous. When we began this project, the existing cluster file systems did not seem to match the level of our project. Some were -will very much in an early development state (e.g., GFS from Red Hat), -while others seemed designed for much larger-scale systems, with very -expensive hardware requirements (e.g., Lustre). The requirements for -the filesystem for Pan-STARRS are somewhat different from the -large-scale computing clusters used by the national labs. Since the -data processing is very parallel, we do not have any strong -requirements on data access concurency. In theory, we could have -simply used NFS and made backup copies of the files using some simple -name-convention rules. We decided to implement the Nebulous system to -allow the targetted analysis and to automate the replication of the -data. In retrospect, the system has succeeded in these goals and has -behaved reliably. However, the support level has been somewhat high, -especially when we have needed to migrate large amounts of data within -the cluster. If we were to start from scratch today, we would -experiment with some of the existing cluster file systems. +still very much in an early development state (e.g., GFS from Red +Hat), while others seemed designed for much larger-scale systems, with +very expensive hardware requirements (e.g., Lustre). The requirements +for the filesystem for Pan-STARRS are somewhat different from the +large-scale computing clusters such as those used by the U.S. national +labs. Since the data processing is very parallel, we do not have any +strong requirements on data access concurency. In theory, we could +have simply used NFS and made backup copies of the files using some +simple name-convention rules. We decided to implement the Nebulous +system to allow the targetted analysis and to automate the replication +of the data. In retrospect, the system has succeeded in these goals +and has behaved reliably. However, the support level has been +somewhat high, especially when we have needed to migrate large amounts +of data within the cluster. If we were to start from scratch today, +it is possible that some of the existing cluster file systems would +address our needs within our budget. Since the Pan-STARRS\,1 telescope first came online in 2007, this