Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40020) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40021) @@ -311,5 +311,5 @@ stage is completed, summary information about the stage is written back to the database. In this way, the database records this history -of the processing, and also provides the information needed to +of the processing, and also provides the information needed by successive processing stages to begin their own tasks. @@ -319,5 +319,7 @@ This same database engine also has instances for other cameras processed by the IPP, e.g., GPC2, the test cameras TC1, TC3, and the -Imaging Sky Probe (ISP). +Imaging Sky Probe (ISP). In general, processing information for +different cameras is separate in differnt processing database; merging +of output products takes place in DVO. Within the processing database, the various processing stages are @@ -371,8 +373,15 @@ occurs, the system will not process an exposure through subsequent stages without the component that has failed temporarily. Since many -of the \ippdbcolumn{fault}s which occur are ephemeral, the processing -stages are set up to occasional clear and re-try the faulted entries. -Thus, automatic processing is able to keep the data flowing even in -the face of occasional network glitches or hardware crashes. +of the \ippdbcolumn{fault}s which occur are ephemeral due to current +conditions of the processing cluster, the processing stages are set up +to occasionally clear and re-try the faulted entries. Some faults +represent software bugs and in the early stages of processing were +accumulated until the corresponding software issue could be addressed; +since the start of the PS1 Science Consortium Surveys, these types of +faults have largely been eliminated. Thus, automatic processing is +able to keep the data flowing even in the face of occasional network +glitches or hardware crashes. + +\note{start of section needed a re-read} \subsection{Summit copy} @@ -641,4 +650,6 @@ \label{sec:warp} +\note{need to describe the RINGS.V3 tessellation and others} + The \ippstage{warp} stage moves the data from a given exposure beyond away from being camera specific and towards a uniform sky oriented @@ -752,4 +763,6 @@ entry, no \ippmisc{advance} job is required. +\note{end of section needed a re-read} + \subsection{Stack Photometry} \label{sec:staticsky} @@ -759,13 +772,14 @@ deferred to the \ippstage{staticsky} stage. This separation is maintained because the photometry analysis of the \ippstage{stack} -images is performed on all 5 filters simultaneously. By deferring -this analysis, the processing system may also decouple the generation -of the pixels from the source detection. This makes the sequencing of -analysis somewhat easier and less subject to blocks due to a failure -in the stacking analysis. Similar to the \ippstage{stack} stage, an -entry is created in the \ippdbtable{staticskyRun} table, linked to a -series of rows in the \ippdbtable{staticskyInput} table by a common -\ippdbcolumn{sky\_id}, each of which also contains the appropriate -\ippdbcolumn{stack\_id} entries for the skycell under consideration. +images, including convolved galaxy model fitting, is performed on all +5 filters simultaneously. By deferring this analysis, the processing +system may also decouple the generation of the pixels from the source +detection. This makes the sequencing of analysis somewhat easier and +less subject to blocks due to a failure in the stacking analysis. +Similar to the \ippstage{stack} stage, an entry is created in the +\ippdbtable{staticskyRun} table, linked to a series of rows in the +\ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky\_id}, +each of which also contains the appropriate \ippdbcolumn{stack\_id} +entries for the skycell under consideration. The input images are passed to the \ippprog{psphotStack} program, @@ -840,15 +854,16 @@ Any measurement which relies on a good knowledge of the PSF at the -location of an object either needs to determine the PSF variations -present in the \ippstage{stack} image, or the measurement will be -somewhat degraded. The highly textured PSF variations make this a -very challenging problem: not only would such a PSF model require an -unusually fine-grained PSF model, there would likely not be enough PSF -stars in a given \ippstage{stack} image to determine the model at the -resolution required. The IPP photometry analysis code uses a PSF -model with 2D variations using a grid of at most $6\times 6$ samples -per skycell, a number reasonably well-matched to the density of stars -at most moderate Galactic latitudes. This scale is far too large to -track the fine-grained changes apparent in the stack images. +location of an object needs to determine the PSF variations present in +the \ippstage{stack} image, or the measurement will be somewhat +degraded. The highly textured PSF variations make this a very +challenging problem: not only would such a PSF model need to be highly +fine-grained, there would likely not be enough PSF stars in a given +\ippstage{stack} image to determine the model at the resolution +required. The IPP photometry analysis code uses a PSF model with 2D +variations using a grid of at most $6\times 6$ samples per skycell, a +number reasonably well-matched to the density of stars at most +moderate Galactic latitudes for the PS1 3$\pi$ depths. This scale is +far too large to track the fine-grained changes apparent in the stack +images. Thus PSF photometry as well as convolved galaxy models in the stack @@ -865,12 +880,14 @@ individual warp images used to generate the stack. This \ippstage{fullforce} analysis is performed on all warps for a single -skycell and filter as a single unit, as this matches the arrangement -of the input source catalog from the \ippstage{skycal} stage. When -processing is queued for this stage, an entry is added to the -\ippdbtable{fullForceRun} primary database table linking to the -specific \ippdbcolumn{skycal\_id} entry that will be used as the -catalog for the photometry. The \ippdbcolumn{warp\_id} values for the -input \ippstage{warp} stage images that contributed to the -\ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are +skycell and filter as a single unit within the processing database, +while individual warps are processed individually in parallel as +separate processing jobs. + +When processing is queued for this stage, an entry is added to the +\ippdbtable{fullForceRun} primary database table with a reference to +the corresponding stack and \ippdbcolumn{skycal\_id} entry that is the +input source of detections to be measured. The \ippdbcolumn{warp\_id} +values for the input \ippstage{warp} stage images that contributed to +the \ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are then added to the \ippdbtable{fullForceInput} table, linked to the primary table by the \ippdbcolumn{ff\_id} identifier. The individual @@ -884,15 +901,16 @@ the same stars for all warps to the extent possible (PSF stars which are excessively masked on a particular image are not used to model the -PSF). \note{this doesn't seem correct, as each warp is run - independently. EAM: not true!} The PSF model is fitted to all of the known source -positions in the warp images. Aperture magnitudes, Kron magnitudes, -and moments are also measured at this stage for each warp. Note that -the flux measurement for a faint, but significant, source from the -stack image may be at a low significance (less than the $5\sigma$ -criterion used when the photometry is not run in this forced mode) in -any individual warp image; the flux may even be negative for specific -warps. When combined together, these low-significance measurements -will result in a signficant measurement as the signal-to-noise -increases by the square root of the number of measurements. +PSF). The PSF model is fitted to all of the known source positions in +the warp images. Aperture magnitudes, Kron magnitudes, and moments +are also measured at this stage for each warp. Note that the flux +measurement for a faint, but significant, source from the stack image +may be at a low significance (less than the $5\sigma$ criterion used +when the photometry is not run in this forced mode) in any individual +warp image; the flux may even be negative for specific warps. When +combined together, these low-significance measurements will result in +a signficant measurement as the signal-to-noise increases by the +square root of the number of measurements. \note{The individual warp +measurements are combined together to generate averages values within +DVO.} Upon completion of the forced photometry (for point sources as well as @@ -901,8 +919,7 @@ that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}. Once all of the entries in the \ippdbtable{fullForceInput} table have -finished, a summary operation is run to generate an appropriate -average value for each measurement, by combining the measurements from -each of the inputs. The output catalogs listed in the -\ippdbtable{fullForceResult} table are passed to the +finished, a summary operation is run to combine the galaxy photometry +analysis measurements into a single value. The output catalogs listed +in the \ippdbtable{fullForceResult} table are passed to the \ippprog{psphotFullForceSummary} to do this averaging. \note{describe what is done} When this completes, an entry is added to the @@ -1031,6 +1048,24 @@ entry as such. -\section{Post-Processing : Database Ingest \& Calibration} +\section{Post-Processing : Database Ingest and Calibration} \label{sec:postprocessing} + +\begin{verbatim} +DVO section outline or list of topics: + +* schema overview [ignoring sky partitioning] + * measurements -> objects + * images +* object definition +* tables in detail +* adding other data types (2mass, etc) +* storage details + * FITS + * compressed FITS +* sky partitioning +* parallelized DVO +* addstar / ingest process [stage -> this goes elsewhere] +* dvo shell description? +\end{verbatim} \subsection{DVO} @@ -1398,11 +1433,11 @@ \note{Default to pointing to Flewelling et al 2017?} -\subsection{PSPS Load \& Merge} +\subsection{PSPS Load and Merge} \label{sec:psps} \note{Default as well to pointing to Flewelling et al 2017?} -\section{Operations \& Automation} - -\subsection{Pantasks \& Parallel Processing} +\section{Operations and Automation} + +\subsection{Pantasks and Parallel Processing} \label{sec:pantasks} @@ -1414,12 +1449,12 @@ the logical links to relate the results of one analysis stage to another. In order to make a complete system which can run -automatically, it is necessary to have a process which can use the +automatically, it is necessary to have a software system which can use the contents of the processing database to generate the commands corresponding to the analysis stages. This system needs to (1) regularly examine the database to find items from stages which are -ready to be processed, (2) to have rules which define how to construct -the appropriate commands, (3) to cause those commands to be executed -within the processing system, (4) to monitor the active processing -jobs for completion, and (5) to check on the results of those +ready to be processed, (2) have rules which define how to construct +the appropriate commands, (3) cause those commands to be executed +within the processing system, (4) monitor the active processing +jobs for completion, and (5) check on the results of those commands and update the processing database as needed. Within the Pan-STARRS IPP, the top-level management of these operations is @@ -1430,10 +1465,11 @@ might be run and to regularly generate new commands based on that concept. The ``tasks'' are defined using the opihi scripting language -(also shared by DVO and other user-interative programs within the -IPP). +(also shared by DVO and other user-interative programs within the +IPP). Pantasks repeatedly checks each task in an attempt to generate a new -command: we say pantasks attempts to 'execute' the task. Tasks may -specify the time between execution attempts, with a 1 second default. +command: we say pantasks attempts to 'execute' the task in each of +these attempts. Tasks may specify the time between execution +attempts, with a 1 second default. Each task must at a minimum define a command to generate. Commands @@ -1441,43 +1477,43 @@ command is explicity defined in the task block (see code example in Figure~\ref{fig:task_example1}) and is identical each time the task is -execute. For a task with a dynamic command, the command is defined -within a special block of the task, called \code{task.exec}. This -block is a snipet of code (in the opihi language) which is run when -the task is executed. The \code{task.exec} code may refer to -variables or other data structures defined by the opihi langage within -the pantasks environment. Within a single \ippprog{pantasks} -instance, all opihi variables and data structures have global context -(\ie, all are visible to all tasks). Variables are by default global, -but within the context of an opihi macro (equivalent of a function -call), variables may be locally-scoped. Other data structures (see -below) are global and must be protected with name space choices. - -Within the \ippprog{task.exec} macro, at some point the command to be -run must be defined with the function 'command'. Once the -\ippprog{task.exec} macro exits successfully, the command is the added -to the list of jobs to be run within the UNIX environment. Jobs -may be run in one of two ways: locally or via the parallel processing -system. The task, or the \ippprog{task.exec} macro, uses the 'host' -command to define how to run the job. If the host is set to 'local', -then the job is run in the background by pantasks itself (using the -C \code{execvp} function). Otherwise, the job is sent to the parallel -processing system to be run on another machine within the cluster. -If the host is set to the special value 'anyhost', then the parallel -processing system is allowed to choose the processing computer -arbitrarily. Any other value is taken to be the DNS name of the -computer on which this job should run. If the option \code{-required} -is supplied to the \code{host} command, then the parallel processing -system must ensure that the job only runs on the specifically named -system. Otherwise, the parallel processing system may choose to -redirect the command to another computer (based on whatever rules are -defined for the parallel processing system). - -When the \ippprog{task.exec} macro is run, the code may choose (e.g., based -on tests of some global variables) to exit the macro with an error -condition, e.g., with the 'break' command. In this circumstance, no -job is produced by the task. The task will try again the next time it -is executed. This feature allows for the user to set processing -blocks which depend on some external tests. For example, some task -may check external network connectivity and set a +executed. A dynamic command is defined within a special block of the +task, called \code{task.exec}. This block is a snipet of code (in the +opihi language) which is run each time the task is executed. The +\code{task.exec} code may refer to variables or other data structures +defined by the opihi language within the pantasks environment. Within +a single \ippprog{pantasks} instance, all opihi variables and data +structures have global context (\ie, all are visible to all tasks). +Variables are by default global, but within the context of an opihi +macro (equivalent of a function call), variables may be +locally-scoped. Other data structures (see below) are global and must +be protected with name space choices. + +Within the \ippprog{task.exec} macro, the command to be run must be +defined with the function 'command'. Once the \ippprog{task.exec} +macro exits successfully, the defined command is the added to the list of jobs +to be run within the UNIX environment. Jobs may be run in one of two +ways: locally or via the parallel processing system. The task, or the +\ippprog{task.exec} macro, uses the 'host' command to define how to +run the job. If the host is set to 'local', then the job is run in +the background by pantasks itself (using the C \code{execvp} +function). Otherwise, the job is sent to the parallel processing +system to be run on another machine within the cluster. If the host +is set to the special value 'anyhost', then the parallel processing +system is allowed to choose the processing computer arbitrarily. Any +other value is taken to be the DNS name of the computer on which this +job should run. If the option \code{-required} is supplied to the +\code{host} command, then the parallel processing system must ensure +that the job only runs on the specifically named computer. Otherwise, +the parallel processing system may choose to redirect the command to +another computer (based on whatever rules are defined for the parallel +processing system). + +When the \ippprog{task.exec} macro is run, the code may choose (e.g., +based on tests of some global variables) to exit the macro with an +error condition, e.g., with the 'break' command. In this +circumstance, no job is produced by the task. The task will be tried +again the next time it is executed. This feature allows for the user +to set processing blocks which depend on some external tests. For +example, some task may check external network connectivity and set a variable based on the network status; other tasks may then choose to wait until the network is available before attempting to run. @@ -1486,4 +1522,7 @@ behavior in detail. Note that the options below may be dynamically reset by the \ippprog{task.exec} macro. + +\note{this section probably has too much detail; move this into an + online user guide?} The option 'npending' may be used to limit the number of jobs which @@ -1532,17 +1571,70 @@ across many machines in the computing cluster. The parallel processing system used by \ippprog{pantasks} is an independent -software system (and alternatives are possible). The default parallel -processing system is a program called \ippprog{pcontrol}. +software system. The default parallel processing system is a program +called \ippprog{pcontrol}\footnote{Alternatives are possible: e.g., + {\tt condor} has been experimentally integrated with + \ippprog{pantasks} for tests}. This program is based on the same \ippprog{opihi} shell language used by \ippprog{pantasks}. The two programs communicate via a shared set of pipes: \ippprog{pantasks} sends commands to the standard input of -the \ippproc{pcontrol}, and accepts back responses on the standard +the \ippprog{pcontrol}, and accepts back responses on the standard output and standard error. -\ippprog{pcontrol} maintains a list of jobs and a list of hosts, -computers on which a job could be run. Jobs may have one of several -states: pending (ready to run), running, jobs which are -running +\ippprog{pcontrol} maintains a list of jobs (commands to be run) and a +list of hosts (computers on which a job could be run). Jobs may have +one of several states: pending (ready to run), running (jobs which are +running), exit (job has completed), busy (job is being checked by +\ippprog{pcontrol}), crash (job has exited with a signal(?), normally +segv). + +Similarly, the hosts may also have one of several states: off, down, +busy, idle, etc. A single host can accept a single job at a time. +Multiple hosts instances corresponding to the same machine may be +specified allowing a single computer to run more than one simultaneous +job. + +During operation, pcontrol accepts new jobs from pantasks and adds +them to the list of jobs to execute. It also accepts from pantasks +the names of computers on which it is allowed to run those jobs. + +When pcontrol is provided with the name of a computer, it will attempt +to make an connection to that machine via ssh (or rsh?). When a +connection is made, the remote shell is used to run a special +interface program call \ippprog{pclient}. This program accepts +command lines from pcontrol and is responsible for executing the +individual commands in the local shell environment. A single ssh +connection to a remote host keeps a single pclient shell running for a +somewhat arbirarly long time, excuting many shell commands as needed. +This architecture avoids wasting overhead making the ssh connection to +the remote machine each time a command is executed, allowing for rapid +excution of many commands. As a result, a single job within the IPP +architecture is allowed to be very light and short running if needed. + +After pcontrol sends a job (commands) to a specific pclient, it checks +back occasionally to see if the command has been run and executed. If +it has finished, then pcontrol will query for the exit status, the +standard output and standard error streams from the command. (where +do these go, back to pantasks?), with the results associated with the +job statistics. At that point, the pclient on the remote machine is +ready to accept a new job from pcontrol. If any jobs are pending in +the list of jobs known to pcontrol, it will send those jobs to any +machines which are idle. + +While pcontrol interacts with the many remote machines, it +occasionally interacts with pantasks to report the results from the +jobs it has been monitoring. Pantasks occasionally requests a list of +the completed jobs. It then requests the status information for each +completed job, including the standard error and standard output. As +pantasks receives this completion information, the jobs are removed +from the list managed by pcontrol. Thus pcontrol maintains at most a +modest list of jobs which are 'in flight', leaving all interpretation +work to pantasks. + +At the pantasks level, the tasks define how pantasks should use the +exit status and output products from each job. For example, the +stderr and stdout may be specified to go to a file (with static name +or name dependent on the specific job). The task may define different +behavior depending on the exit code from the job. \note{discuss pclient} @@ -1554,9 +1646,4 @@ \end{verbatim} - -These commands may the be run by the pantasks program -directly, or they may be passed to another program which - -* task vs job \begin{figure} @@ -1572,12 +1659,8 @@ end \end{verbatim} - - \caption{\label{fig:peaks} Illustration of peak finding and culling peaks within a - footprint. Insignificant peaks within the footprint of a brighter - peak are ignored in further processing. } + \caption{\label{fig:simple.static.task} Example of a simple static + task in the opihi-based scripting language used by pantasks. } \end{center} \end{figure} - - \note{this section needs to be re-written : pclient vs pcontrol vs pantasks} @@ -1660,12 +1743,12 @@ \note{I'm not convinced this is the right place for it, but it felt like a natural extension of the ``advance''}. -\note{wording..} -Beyond the warp stage, there is no longer a single ``next'' stage into -which data can be queued. Because of this, more robust methods are -used to advance the data. For processing data that is actively being -observed at the summit, this is handled by a set of ``nightly -science'' tasks and an associated \ippmisc{ippScript}. The goal of -these tasks is to ensure that exposures are correctly paired into sets -of \ippstage{warp}-\ippstage{warp} difference images, and that nightly +\note{wording..} Beyond the warp stage, there is no longer a single +``next'' stage into which data can be queued. Because of this, more +sophisticated methods are used to advance the data. For processing +data that is actively being observed at the summit, this is handled by +a set of ``nightly science'' tasks and an associated +\ippmisc{ippScript}. The goal of these tasks is to ensure that +exposures are correctly paired into sets of +\ippstage{warp}-\ippstage{warp} difference images, and that nightly stacks are generated for MD fields. @@ -1742,14 +1825,16 @@ \subsection{Nebulous} \label{sec:nebulous} -Storing the large volume of data that is generated by the GPC1 camera -was recognized early in the Pan-STARRS project as a major concern. -The \ippprog{Nebulous} system was designed to organize this data. The -main components of this system are a database storing the locations of -the files, with a Simple Object Access Protocol (SOAP) interface -between the database and the other IPP programs \note{define / mention - http}. The actual files are stored on a collection of computers -with substantial disk partitions in the IPP cluster, shared within the -cluster via NFS. This distribution of files is useful to balance the -disk I/O, as this parallelizes the load. + +A major concern recognized early in the Pan-STARRS project is the +challenge of storing and managing the large volume of data that is +generated by the GPC1 camera. The \ippprog{Nebulous} system was +designed to organize this data. The main components of this system +are a database storing the locations of the files, with a Simple +Object Access Protocol (SOAP) interface between the database and the +other IPP programs \note{define / mention http}. The actual files are +stored on a collection of computers with substantial disk partitions +in the IPP cluster, shared within the cluster via NFS. This +distribution of files is useful to balance the disk I/O, as this +parallelizes the load. The original design of \ippprog{Nebulous} was intended to aid in the