Changeset 41209 for trunk

Timestamp:

Dec 21, 2019, 12:52:19 PM (7 years ago)

Author:

eugene

Message:

edits to lessons-learned

File:

• trunk/doc/release.2015/ps1.datasystem/datasystem.tex (modified) (4 diffs)

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

@@ -2869 +2869 @@
 the same constraints.  
 
-One early choice we made was to use standard C and to use Perl as a
-wrapper language.  We considered other language choices, including C++
-and Python.  At the time, Python was fairly new and did not have the
-wide-spread acceptance it has today.  In retrospect, our choice of
-Perl has not held up very well.  The capabiliaties available within
-the Python environment would have allowed us to include interesting
+One early choice was to use standard C for analysis programs and to
+use Perl as a wrapper language.  We considered other language choices,
+including C++ and Python.  Our choice of C over C++ has not held up
+well: we would have done well to have the modern object-oriented
+features of C++, some aspect of which we have imitated in our C coding
+style.  The choice of Perl over Python has also fared poorly.  At the
+time, Python was fairly new and did not have the wide-spread
+acceptance it has today.  The capabiliaties available within the
+Python environment would have allowed us to include interesting
 visualization and other high-level analysis options.  It is also
 easier to hire astronomers with good Python coding skills that Perl
@@ -2888 +2891 @@
 database to manage the processing stages, with regular queries to the
 database to choose the tasks which are ready to proceed.  Other
-choices were possible.  In some pipelined processing systems, jobs
-which complete trigger the next processing step.  For example,
+choices were possible.  In some pipelined processing systems,
+completed jobs trigger the next processing step.  For example,
 \ippprog{ppImage} or its wrapper (\ippprog{chip_imfile.pl}) could have
-been responsible for launching the \ippprog{psastro} analysis.
+been responsible for launching the \ippprog{psastro} analysis,
+eliminating the \ippprog{pantasks} manager entirely.
 Alternatively, a manager process could be responsible for launching
 the next processing step when one step has completed.  For example,
@@ -2897 +2901 @@
 complete and launch the \ippprog{psastro} analysis.  Both of these
 choices can potentially result in lower latency since the next step is
-in principle run immediately when the previous step is completed.  Our
-choice has two important advantages: First, error and failure recovery
-are trivial.  If one of the many programs fails or is interrupted, the
-system can easily notice and retry the job.  In a triggered system, a
-failure of one stage could mean the trigger never happens.  Some
-external cleanup system would need to be implemented to check for the
-failures and re-launch.  The second advantage of our design is that
-each analysis stage is highly independent and can thus be flexibly run
-in different ways.  For example, alternative test systems can run in
-parallel with the nightly operations system, using the outputs of the
-nightly processing by simple changes to the queries used to select the
-elements for an analysis stage.  In addition, it is easy to add new
-stages since they do not need to be injected into the standard
-processing manager (\ippprog{pantasks}).
+in principle run immediately when the previous step is completed.  
+
+Our design choice has two important advantages: First, error and failure
+recovery are trivial.  If one of the many programs fails or is
+interrupted, the system can easily notice and retry the job.  In a
+triggered system, a failure of one stage could mean the trigger never
+happens.  Some external cleanup system would need to be implemented to
+check for the failures and re-launch.  The second advantage of our
+design is that each analysis stage is highly independent and can thus
+be flexibly run in different ways.  For example, alternative test
+systems can run in parallel with the nightly operations system, using
+the outputs of the nightly processing by simple changes to the queries
+used to select the elements for an analysis stage.  In addition, it is
+easy to add new stages since they do not need to be injected into the
+standard processing manager (\ippprog{pantasks}).
 
 The main challenge related to this database-managed design is that the
@@ -2928 +2933 @@
 ambiguous.  When we began this project, the existing cluster file
 systems did not seem to match the level of our project.  Some were
-will very much in an early development state (e.g., GFS from Red Hat),
-while others seemed designed for much larger-scale systems, with very
-expensive hardware requirements (e.g., Lustre).  The requirements for
-the filesystem for Pan-STARRS are somewhat different from the
-large-scale computing clusters used by the national labs.  Since the
-data processing is very parallel, we do not have any strong
-requirements on data access concurency.  In theory, we could have
-simply used NFS and made backup copies of the files using some simple
-name-convention rules.  We decided to implement the Nebulous system to
-allow the targetted analysis and to automate the replication of the
-data.  In retrospect, the system has succeeded in these goals and has
-behaved reliably.  However, the support level has been somewhat high,
-especially when we have needed to migrate large amounts of data within
-the cluster.  If we were to start from scratch today, we would
-experiment with some of the existing cluster file systems.
+still very much in an early development state (e.g., GFS from Red
+Hat), while others seemed designed for much larger-scale systems, with
+very expensive hardware requirements (e.g., Lustre).  The requirements
+for the filesystem for Pan-STARRS are somewhat different from the
+large-scale computing clusters such as those used by the U.S. national
+labs.  Since the data processing is very parallel, we do not have any
+strong requirements on data access concurency.  In theory, we could
+have simply used NFS and made backup copies of the files using some
+simple name-convention rules.  We decided to implement the Nebulous
+system to allow the targetted analysis and to automate the replication
+of the data.  In retrospect, the system has succeeded in these goals
+and has behaved reliably.  However, the support level has been
+somewhat high, especially when we have needed to migrate large amounts
+of data within the cluster.  If we were to start from scratch today,
+it is possible that some of the existing cluster file systems would
+address our needs within our budget.
 
 Since the Pan-STARRS\,1 telescope first came online in 2007, this