Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40009) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40020) @@ -179,6 +179,8 @@ \label{sec:overview} +\subsection{Elements of the Pan-STARRS Data Processing System} + The Pan-STARRS Data Analysis system contains many features to support -the wide range of activities: archiving and management of the raw and +a wide range of activities: archiving and management of the raw and processed image files; real-time nightly processing of images for transient and moving object science; large-scale re-processing and @@ -194,5 +196,5 @@ \item Summit : both the camera and observatory summit systems perform data analysis tasks needed to support the on-going observations. - In this article, we focus on those aspects used by the off-summit + In this article, we focus only on those aspects used by the off-summit analysis stages. \item Image Processing Pipeline (IPP) : this portion of the data @@ -202,5 +204,5 @@ \item Moving Object Processing System (MOPS) : this system is responsible for linking individual detections of solar-system - objects together and determining the orbits. + objects together and determining the orbits. \note{Denneau REF} \item PSPS : this system ingests the calibrated measurements from the IPP, MOPS, and others and generates a high-availability database @@ -220,5 +222,6 @@ Pan-STARRS community have implemented a variety of science-level analyses of their own to support their science goals (e.g., M31 -Cepheid search). +Cepheid search). This article discusses the analysis elements which +take place at the IfA except as noted. Figure~\ref{fig:analysis.elements} illustrates the many elements of @@ -228,4 +231,6 @@ The MOPS is described in detail by \cite{MOPS}, while the summit systems are described by \note{REF?}. + +\subsection{Nightly Processing Analysis Stages} Data analysis to support nighly science operations is driven by two @@ -241,6 +246,6 @@ determined (camera), and finally images are geometric transformed to a common pixel representation (warp). Warped images may either be added -together (stack) or used in an image subtraction (diff). For nightly -science operations, images for certain fields such as the Medium Deep +together (stack) or used in an image subtraction (diff). As part of nightly +science processing, images for certain fields such as the Medium Deep survey fields (see \cite{}), are stacked together in nightly chunks, providing deeper detection capability on short timescales. Depending @@ -252,4 +257,6 @@ \note{need earlier mention of 3pi, MD, etc} + +\subsection{Re-processing Analysis Stages} Pan-STARRS has performed several large-scale reprocessings of both the @@ -270,5 +277,7 @@ analysis stages are ingested into the internal calibration database (DVO, the Desktop Virtual Observatory) and used for photometric and -astrometric calibrations. +astrometric calibrations (see Section~\ref{sec:dvo}) + +\subsection{Data Access and Distribution} During the PS1 Science Consortium operations, data products were @@ -281,5 +290,5 @@ demand via the Postage Stamp server; access to large sets of pre-defined raw and reduced data products was provided via the -Distribution and Publication systems. The interal calibration DVO +Distribution and Publication systems. The internal calibration DVO databases were provided at several stages via a separate DVO distribution mechanism. For the first two large-scale reprocessings @@ -1415,13 +1424,160 @@ commands and update the processing database as needed. Within the Pan-STARRS IPP, the top-level management of these operations is -performed with the program called \ippprog{pantasks}. +performed by the program called \ippprog{pantasks}. The core capability of the \ippprog{pantasks} program is to take a collection of ``tasks'' which describe the concept of a command which might be run and to regularly generate new commands based on that -concept. These commands may the be run by the pantasks program +concept. The ``tasks'' are defined using the opihi scripting language +(also shared by DVO and other user-interative programs within the +IPP). + +Pantasks repeatedly checks each task in an attempt to generate a new +command: we say pantasks attempts to 'execute' the task. Tasks may +specify the time between execution attempts, with a 1 second default. + +Each task must at a minimum define a command to generate. Commands +may be static or dynamic. For a task with a static command, the +command is explicity defined in the task block (see code example in +Figure~\ref{fig:task_example1}) and is identical each time the task is +execute. For a task with a dynamic command, the command is defined +within a special block of the task, called \code{task.exec}. This +block is a snipet of code (in the opihi language) which is run when +the task is executed. The \code{task.exec} code may refer to +variables or other data structures defined by the opihi langage within +the pantasks environment. Within a single \ippprog{pantasks} +instance, all opihi variables and data structures have global context +(\ie, all are visible to all tasks). Variables are by default global, +but within the context of an opihi macro (equivalent of a function +call), variables may be locally-scoped. Other data structures (see +below) are global and must be protected with name space choices. + +Within the \ippprog{task.exec} macro, at some point the command to be +run must be defined with the function 'command'. Once the +\ippprog{task.exec} macro exits successfully, the command is the added +to the list of jobs to be run within the UNIX environment. Jobs +may be run in one of two ways: locally or via the parallel processing +system. The task, or the \ippprog{task.exec} macro, uses the 'host' +command to define how to run the job. If the host is set to 'local', +then the job is run in the background by pantasks itself (using the +C \code{execvp} function). Otherwise, the job is sent to the parallel +processing system to be run on another machine within the cluster. +If the host is set to the special value 'anyhost', then the parallel +processing system is allowed to choose the processing computer +arbitrarily. Any other value is taken to be the DNS name of the +computer on which this job should run. If the option \code{-required} +is supplied to the \code{host} command, then the parallel processing +system must ensure that the job only runs on the specifically named +system. Otherwise, the parallel processing system may choose to +redirect the command to another computer (based on whatever rules are +defined for the parallel processing system). + +When the \ippprog{task.exec} macro is run, the code may choose (e.g., based +on tests of some global variables) to exit the macro with an error +condition, e.g., with the 'break' command. In this circumstance, no +job is produced by the task. The task will try again the next time it +is executed. This feature allows for the user to set processing +blocks which depend on some external tests. For example, some task +may check external network connectivity and set a +variable based on the network status; other tasks may then choose to +wait until the network is available before attempting to run. + +Other task options discussed below exist to control the system +behavior in detail. Note that the options below may be dynamically +reset by the \ippprog{task.exec} macro. + +The option 'npending' may be used to limit the number of jobs which +are simultaneously executed for a specific task. For example, some +classes of jobs should only be run one-at-a-time because they are not +protected against collisions or they may overload a resource. The use +of 'npending' allows these situations to be handled cleanly within +pantasks (avoiding cumbersome coding within with program or supporting +script). + +The option 'nmax' limits the total number of jobs which a task +generates. This option may be useful in cases where +\ippprog{pantasks} is used to perform a limited set of operations. +\note{do we actually use this in IPP?} + +The option 'trange' allows the user to restrict the time period during +which the specific tasks is executed. This option is given with a +start and an end time for the limiting time range. These times may be +of one of several forms: ``HH:MM:SS'' specifies a time within a day +(in UT or local time?). ``Day[@HH:MM:SS]'' specifies a time on a +specific day, e.g., \code{trange Mon@13:00 Tue@09:00} says the task +should be run from 1pm on Mondays to 9am on Tuesdays. ``YYYY/MM/DD,HH:MM:SS'' +specifies a time on a specific date within the year. The start and +end times must be of the same class. The \code{trange} command has +some optional arguments as well. The option \code{-nmax NNN} defines +the maximum number of jobs which may be run in that time range. The +option \code{-exclude} specifies that the time range is a period when +the task should {\em not} be executed. An arbirary number of time +ranges may be specified \note{how are they evaluated?} + +The option \code{nice} specifies the 'nice' level at which the job is +run when it is executed. The parallel processing system must respect +this concept. + +The option \code{active} can be used to turn on and off a task for +periods. Since a user command or a macro run by pantasks can +re-define task options, the \code{active} state may be changed +independently of the task execute. This is useful for keeping tasks +defined by a \ippprog{pantasks} instance, but allowing the user to +prevent them from running for some reason. + +\note{discuss pcontrol} + +Jobs which are generated by \ippprog{pantasks} may be run locally on +the machine running \ippprog{pantasks} or they may be distributed +across many machines in the computing cluster. The parallel +processing system used by \ippprog{pantasks} is an independent +software system (and alternatives are possible). The default parallel +processing system is a program called \ippprog{pcontrol}. + +This program is based on the same \ippprog{opihi} shell language used +by \ippprog{pantasks}. The two programs communicate via a shared set +of pipes: \ippprog{pantasks} sends commands to the standard input of +the \ippproc{pcontrol}, and accepts back responses on the standard +output and standard error. + +\ippprog{pcontrol} maintains a list of jobs and a list of hosts, +computers on which a job could be run. Jobs may have one of several +states: pending (ready to run), running, jobs which are +running + +\note{discuss pclient} + +\begin{verbatim} +* pantasks_client vs pantasks_server +* opihi scripting language +* task options (show example?) + +\end{verbatim} + +These commands may the be run by the pantasks program directly, or they may be passed to another program which * task vs job + +\begin{figure} + \begin{center} +\begin{verbatim} +task example.static.task + host local + command ls /tmp + periods -exec 5.0 + npending 1 + stdout NULL + stderr /data/local/example.task +end +\end{verbatim} + + \caption{\label{fig:peaks} Illustration of peak finding and culling peaks within a + footprint. Insignificant peaks within the footprint of a brighter + peak are ignored in further processing. } + \end{center} +\end{figure} + + \note{this section needs to be re-written : pclient vs pcontrol vs pantasks}