Changeset 40020

Timestamp:

Apr 27, 2017, 2:49:02 AM (9 years ago)

Author:

eugene

Message:

various text updates

File:

• trunk/doc/release.2015/ps1.datasystem/datasystem.tex (modified) (10 diffs)

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

@@ -179 +179 @@
 \label{sec:overview}
 
+\subsection{Elements of the Pan-STARRS Data Processing System}
+
 The Pan-STARRS Data Analysis system contains many features to support
-the wide range of activities: archiving and management of the raw and
+a wide range of activities: archiving and management of the raw and
 processed image files; real-time nightly processing of images for
 transient and moving object science; large-scale re-processing and
@@ -194 +196 @@
 \item Summit : both the camera and observatory summit systems perform
   data analysis tasks needed to support the on-going observations.
-  In this article, we focus on those aspects used by the off-summit
+  In this article, we focus only on those aspects used by the off-summit
   analysis stages.
 \item Image Processing Pipeline (IPP) : this portion of the data
@@ -202 +204 @@
 \item Moving Object Processing System (MOPS) : this system is
   responsible for linking individual detections of solar-system
-  objects together and determining the orbits.
+  objects together and determining the orbits. \note{Denneau REF}
 \item PSPS : this system ingests the calibrated measurements from the
   IPP, MOPS, and others and generates a high-availability database
@@ -220 +222 @@
 Pan-STARRS community have implemented a variety of science-level
 analyses of their own to support their science goals (e.g., M31
-Cepheid search).
+Cepheid search).  This article discusses the analysis elements which
+take place at the IfA except as noted.
 
 Figure~\ref{fig:analysis.elements} illustrates the many elements of
@@ -228 +231 @@
 The MOPS is described in detail by \cite{MOPS}, while the summit
 systems are described by \note{REF?}.
+
+\subsection{Nightly Processing Analysis Stages}
 
 Data analysis to support nighly science operations is driven by two
@@ -241 +246 @@
 determined (camera), and finally images are geometric transformed to a
 common pixel representation (warp).  Warped images may either be added
-together (stack) or used in an image subtraction (diff).  For nightly
-science operations, images for certain fields such as the Medium Deep
+together (stack) or used in an image subtraction (diff).  As part of nightly
+science processing, images for certain fields such as the Medium Deep
 survey fields (see \cite{}), are stacked together in nightly chunks,
 providing deeper detection capability on short timescales.  Depending
@@ -252 +257 @@
 
 \note{need earlier mention of 3pi, MD, etc}
+
+\subsection{Re-processing Analysis Stages}
 
 Pan-STARRS has performed several large-scale reprocessings of both the
@@ -270 +277 @@
 analysis stages are ingested into the internal calibration database
 (DVO, the Desktop Virtual Observatory) and used for photometric and
-astrometric calibrations.
+astrometric calibrations (see Section~\ref{sec:dvo})
+
+\subsection{Data Access and Distribution}
 
 During the PS1 Science Consortium operations, data products were
@@ -281 +290 @@
 demand via the Postage Stamp server; access to large sets of
 pre-defined raw and reduced data products was provided via the
-Distribution and Publication systems.  The interal calibration DVO
+Distribution and Publication systems.  The internal calibration DVO
 databases were provided at several stages via a separate DVO
 distribution mechanism.  For the first two large-scale reprocessings
@@ -1415 +1424 @@
 commands and update the processing database as needed.  Within the
 Pan-STARRS IPP, the top-level management of these operations is
-performed with the program called \ippprog{pantasks}.  
+performed by the program called \ippprog{pantasks}.  
 
 The core capability of the \ippprog{pantasks} program is to take a
 collection of ``tasks'' which describe the concept of a command which
 might be run and to regularly generate new commands based on that
-concept.  These commands may the be run by the pantasks program
+concept.  The ``tasks'' are defined using the opihi scripting language
+(also shared by DVO and other user-interative programs  within the
+IPP).  
+
+Pantasks repeatedly checks each task in an attempt to generate a new
+command: we say pantasks attempts to 'execute' the task.  Tasks may
+specify the time between execution attempts, with a 1 second default.
+
+Each task must at a minimum define a command to generate.  Commands
+may be static or dynamic.  For a task with a static command, the
+command is explicity defined in the task block (see code example in
+Figure~\ref{fig:task_example1}) and is identical each time the task is
+execute.  For a task with a dynamic command, the command is defined
+within a special block of the task, called \code{task.exec}.  This
+block is a snipet of code (in the opihi language) which is run when
+the task is executed.  The \code{task.exec} code may refer to
+variables or other data structures defined by the opihi langage within
+the pantasks environment.  Within a single \ippprog{pantasks}
+instance, all opihi variables and data structures have global context
+(\ie, all are visible to all tasks).  Variables are by default global,
+but within the context of an opihi macro (equivalent of a function
+call), variables may be locally-scoped.  Other data structures (see
+below) are global and must be protected with name space choices. 
+
+Within the \ippprog{task.exec} macro, at some point the command to be
+run must be defined with the function 'command'.  Once the
+\ippprog{task.exec} macro exits successfully, the command is the added
+to the list of jobs to be run within the UNIX environment.  Jobs
+may be run in one of two ways: locally or via the parallel processing
+system.  The task, or the \ippprog{task.exec} macro, uses the 'host'
+command to define how to run the job.  If the host is set to 'local',
+then the job is run in the background by pantasks itself (using the
+C \code{execvp} function).  Otherwise, the job is sent to the parallel
+processing system to be run on another machine within the cluster.
+If the host is set to the special value 'anyhost', then the parallel
+processing system is allowed to choose the processing computer
+arbitrarily.  Any other value is taken to be the DNS name of the
+computer on which this job should run.  If the option \code{-required}
+is supplied to the \code{host} command, then the parallel processing
+system must ensure that the job only runs on the specifically named
+system.  Otherwise, the parallel processing system may choose to
+redirect the command to another computer (based on whatever rules are
+defined for the parallel processing system).
+
+When the \ippprog{task.exec} macro is run, the code may choose (e.g., based
+on tests of some global variables) to exit the macro with an error
+condition, e.g., with the 'break' command.  In this circumstance, no
+job is produced by the task.  The task will try again the next time it
+is executed.  This feature allows for the user to set processing
+blocks which depend on some external tests.  For example, some task
+may check external network connectivity and set a
+variable based on the network status; other tasks may then choose to
+wait until the network is available before attempting to run.
+
+Other task options discussed below exist to control the system
+behavior in detail.  Note that the options below may be dynamically
+reset by the \ippprog{task.exec} macro.   
+
+The option 'npending' may be used to limit the number of jobs which
+are simultaneously executed for a specific task.  For example, some
+classes of jobs should only be run one-at-a-time because they are not
+protected against collisions or they may overload a resource.  The use
+of 'npending' allows these situations to be handled cleanly within
+pantasks (avoiding cumbersome coding within with program or supporting
+script).
+
+The option 'nmax' limits the total number of jobs which a task
+generates.  This option may be useful in cases where
+\ippprog{pantasks} is used to perform a limited set of operations.
+\note{do we actually use this in IPP?}
+
+The option 'trange' allows the user to restrict the time period during
+which the specific tasks is executed.  This option is given with a
+start and an end time for the limiting time range.  These times may be
+of one of several forms: ``HH:MM:SS'' specifies a time within a day
+(in UT or local time?).  ``Day[@HH:MM:SS]'' specifies a time on a
+specific day, e.g., \code{trange Mon@13:00 Tue@09:00} says the task
+should be run from 1pm on Mondays to 9am on Tuesdays.  ``YYYY/MM/DD,HH:MM:SS''
+specifies a time on a specific date within the year.  The start and
+end times must be of the same class.  The \code{trange} command has
+some optional arguments as well.  The option \code{-nmax NNN} defines
+the maximum number of jobs which may be run in that time range.  The
+option \code{-exclude} specifies that the time range is a period when
+the task should {\em not} be executed.  An arbirary number of time
+ranges may be specified \note{how are they evaluated?}
+
+The option \code{nice} specifies the 'nice' level at which the job is
+run when it is executed.  The parallel processing system must respect
+this concept.
+
+The option \code{active} can be used to turn on and off a task for
+periods.  Since a user command or a macro run by pantasks can
+re-define task options, the \code{active} state may be changed
+independently of the task execute.  This is useful for keeping tasks
+defined by a \ippprog{pantasks} instance, but allowing the user to
+prevent them from running for some reason.
+
+\note{discuss pcontrol}
+
+Jobs which are generated by \ippprog{pantasks} may be run locally on
+the machine running \ippprog{pantasks} or they may be distributed
+across many machines in the computing cluster.  The parallel
+processing system used by \ippprog{pantasks} is an independent
+software system (and alternatives are possible).  The default parallel
+processing system is a program called \ippprog{pcontrol}.
+
+This program is based on the same \ippprog{opihi} shell language used
+by \ippprog{pantasks}.  The two programs communicate via a shared set
+of pipes: \ippprog{pantasks} sends commands to the standard input of
+the \ippproc{pcontrol}, and accepts back responses on the standard
+output and standard error.  
+
+\ippprog{pcontrol} maintains a list of jobs and a list of hosts,
+computers on which a job could be run.  Jobs may have one of several
+states: pending (ready to run), running, jobs which are
+running
+
+\note{discuss pclient}
+
+\begin{verbatim}
+* pantasks_client vs pantasks_server
+* opihi scripting language
+* task options (show example?)
+
+\end{verbatim}
+
+These commands may the be run by the pantasks program
 directly, or they may be passed to another program which 
 
 * task vs job
+
+\begin{figure}
+ \begin{center}
+\begin{verbatim}
+task       example.static.task
+  host     local
+  command  ls /tmp
+  periods  -exec 5.0
+  npending 1
+  stdout   NULL
+  stderr   /data/local/example.task
+end  
+\end{verbatim}
+
+ \caption{\label{fig:peaks} Illustration of peak finding and culling peaks within a
+    footprint.  Insignificant peaks within the footprint of a brighter
+    peak are ignored in further processing. }
+  \end{center}
+\end{figure}
+
+
 
 \note{this section needs to be re-written : pclient vs pcontrol vs pantasks}