IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 40023


Ignore:
Timestamp:
May 8, 2017, 5:47:48 AM (9 years ago)
Author:
eugene
Message:

updates to text; add figure

Location:
trunk/doc/release.2015/ps1.datasystem
Files:
1 added
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/release.2015/ps1.datasystem/datasystem.tex

    r40022 r40023  
    1 \documentclass[iop,floatfix]{emulateapj}
     1% \documentclass[iop,floatfix]{emulateapj}
    22% \documentclass[iop,floatfix,onecolumn]{emulateapj}
    33% \documentclass[12pt,preprint]{aastex}
     4\documentclass[10pt,preprint]{aastex}
    45% \pdfoutput=1
    56
    … …  
    78\RequirePackage{code}
    89\input{astro.sty}
     10
     11\usepackage[T1]{fontenc}% (2) specify encoding
    912
    1013% online version may use color, but print version needs b/w
    … …  
    8790\keywords{Surveys:\PSONE }
    8891
     92\begin{verbatim}
     93MAJOR TODO ITEMS:
     94* introduce and describe RINGS.V3 in or before warp section (refer to Waters if appropriate)
     95* re-read and trim details as needed (referring to the other papers)
     96* re-write the DVO description using topics list given
     97* write discussion of calibration operations (refer to cal paper)
     98* write IPP to PSPS summary
     99* write PSPS Load and Merge summary (use Flewelling paper for ref)
     100* add some specific numbers (data volume, processing times, etc)
     101\end{verbatim}
     102
    89103\section{Introduction}
    90104\label{sec:intro}
     105
     106The 1.8m Pan-STARRS\,1 telescope is located on the summit of Haleakala
     107on the Hawaiian island of Maui.  The wide-field optical design of the
     108telescope \citep{PS1.optics} produces a 3.3 degree field of view with
     109low distortion and minimal vignetting even at the edges of the
     110illuminated region.  The optics and natural seeing combine to yield
     111good image quality: 75\% of the images have full-width half-max values
     112less than (1.51, 1.39, 1.34, 1.27, 1.21) arcseconds for (\grizy), with
     113a floor of $\sim 0.7$ arcseconds.
     114
     115The \PSONE\ camera \citep{PS1.GPCA}, known as GPC1, consists of a
     116mosaic of 60 back-illuminated CCDs manufactured by Lincoln Laboratory.
     117The CCDs each consist of an $8\times8$ grid of $\sim 600\times 600$
     118pixel readout regions, yielding an effective $4800\times4800$
     119detector.  Initial performance assessments are presented in
     120\cite{PS1.GPCB}.  Routine observations are conducted remotely from the
     121Advanced Technology Research Center in Kula, the main facility of the
     122University of Hawaii's Institute for Astronomy operations on Maui.
     123
     124For nearly 4 years, from 2010 May through 2014 March, this telescope
     125was used to perform a collection of astronomical surveys under the
     126aegis of the Pan-STARRS Science Consortium.  The majority of the time
     127(56\%) was spent on surveying the $\frac{3}{4}$ of the sky north of
     128$-30$ Declination with \grizy\ filters in the so-called $3\pi$ Survey.
     129Another $\sim 25\%$ of the time was concentrated on repeated deep
     130observations of 10 specific fields in the Medium-Deep Survey.  The
     131rest of the time was used for several other surveys, including a
     132search for potentially hazardous asteroids in our solar system.  The
     133details of the telescope, surveys, and resulting science publications
     134are described by \cite{Chambers}.
    91135
    92136This is the second in a series of seven papers describing the
    … …  
    143187described in detail in \cite{2012ApJ...750...99T}.
    144188
    145 The Pan-STARRS Image Processing Pipeline consists of a suite of
    146 software programs and data systems that are designed to reduce
    147 astronomical images, with a focus on parallelization necessary to
    148 speed the processing of the large images produced by the GPC1 camera.
     189This paper presents a description of the Pan-STARRS data handling
     190systems, with an emphasis on the Image Processing Pipeline (IPP).  The
     191Pan-STARRS Image Processing Pipeline consists of a suite of software
     192programs and data systems that are designed to reduce astronomical
     193images, with the parallelization necessary to speed the processing of
     194the large images produced by the GPC1 camera. 
     195
    149196Part of this parallelization is derived from the fact that this camera
    150197consists of 60 independent orthogonal transfer array (OTA) devices,
    … …  
    153200majority of stages operate only on smaller segments of a full exposure
    154201to allow the processing tasks to be spread over the machines in the
    155 processing cluster.
    156 
    157 This paper presents a description of the IPP data handling system.
    158 Section \ref{sec:subsystems} describes the major IPP subsystems that
    159 underlie the main pipeline, providing a set of common interfaces and
    160 tools used at multiple stages.  The main processing stages of the
    161 pipeline are described in Section \ref{sec:stages}, although all
    162 exposures may not necessarily pass through each of these stages.  The
    163 hardware systems that have done the processing for the PV3 data
    164 release are listed in Section \ref{sec:hardware}, with some details
    165 on the scale of computing needed to reduce this large number of
    166 exposures.  Finally, Section \ref{sec:discussion} presents a
    167 discussion of some of the lessons learned in the creation of the IPP,
    168 and its utility in reducing data from other cameras and telescopes.
     202processing cluster. \note{move elsewhere?}
     203
     204Section~\ref{sec:overview} provides an overview of the full data
     205analysis system and breaks down the major elements of the Image
     206Processing Pipeline.  Section~\ref{sec:stages} discusses in some
     207detail each of the analysis steps which may be applied to the images
     208and resulting catalogs of detected sources.
     209Section~\ref{sec:postprocessing} discusses the calibration operations
     210and database used for calibration.  Section~\ref{sec:operations}
     211discusses the operational infrastructure of the IPP.
     212Section~\ref{sec:hardware} discusses the hardware systems used by the
     213IPP for regular nightly operations and for processing the PV3 data
     214release, with some details on the scale of computing needed to reduce
     215this large number of exposures.  Finally, Section~\ref{sec:discussion}
     216presents a discussion of some of the lessons learned in the creation
     217of the IPP, and its utility in reducing data from other cameras and
     218telescopes.
    169219
    170220{\color{red} {\em Note: These papers are being placed on arXiv.org to
    … …  
    186236transient and moving object science; large-scale re-processing and
    187237calibration to produce measurements for the science collaboration and
    188 the wider public; \note{manual/specialized} image processing to
    189 facilitate research and development of the analysis system itself;
    190 distribution of the resulting data products to various consumers in a
    191 variety of formats and modes.
     238the wider public; specialized image processing to facilitate research
     239and development of the analysis system itself; and distribution of the
     240resulting data products to various consumers in a variety of formats
     241and modes.
    192242
    193243The Pan-STARRS Data Analysis system is divided internally into several major
    … …  
    204254\item Moving Object Processing System (MOPS) : this system is
    205255  responsible for linking individual detections of solar-system
    206   objects together and determining the orbits. \note{Denneau REF}
     256  objects together and determining the orbits \citep[][]{2013PASP..125..357D}.
    207257\item PSPS : this system ingests the calibrated measurements from the
    208258  IPP, MOPS, and others and generates a high-availability database
    … …  
    229279analysis steps which occur within the Pan-STARRS observatory, with an
    230280emphasis on the analysis, calibration, and database ingest stages.
    231 The MOPS is described in detail by \cite{MOPS}, while the summit
    232 systems are described by \note{REF?}.
     281The MOPS is described in detail by \cite{2013PASP..125..357D}, while
     282the summit systems are described by \note{REF?}.
     283
     284\begin{figure*}[htbp]
     285  \begin{center}
     286 \includegraphics[width=\hsize,clip]{PS1_Data_Analysis_System_Overview.pdf}
     287  \caption{\label{fig:analysis.elements} Elements of the Pan-STARRS\,1
     288    Data Analysis System.  Rectangles represent data analysis steps;
     289    ellipses represent databases; rounded rectangles represent
     290    external groups (``customers'').  The arrows show a simplified representation
     291  of the major flow of data between the analysis stages and data
     292  processing elements.}
     293  \end{center}
     294\end{figure*}
    233295
    234296\subsection{Nightly Processing Analysis Stages}
    … …  
    256318appropriate part of the sky.
    257319
    258 \note{need earlier mention of 3pi, MD, etc}
    259 
    260320\subsection{Re-processing Analysis Stages}
    261321
    262322Pan-STARRS has performed several large-scale reprocessings of both the
    263 Medium Deep and 3pi Survey data.  For the 3pi Survey data, we identify
     323Medium Deep and $3\pi$ Survey data.  For the $3\pi$ Survey data, we identify
    264324these large-scale reprocessings as PV1, PV2, and PV3 (we also define
    265325the nightly science analysis of the data as PV0).  For these
    … …  
    277337analysis stages are ingested into the internal calibration database
    278338(DVO, the Desktop Virtual Observatory) and used for photometric and
    279 astrometric calibrations (see Section~\ref{sec:dvo})
     339astrometric calibrations (see Section~\ref{sec:DVO})
    280340
    281341\subsection{Data Access and Distribution}
    … …  
    383443glitches or hardware crashes.
    384444
    385 \note{start of section needed a re-read}
     445% \note{start of section needed a re-read}
    386446
    387447\subsection{Summit copy}
    … …  
    403463(\ippdbtable{summitExp}), indexed by an identifier that simply
    404464increments the number of exposures announced by the summit, the
    405 \ippdbcolumn{summit\_id}.  This tells the \ippstage{summitcopy} system
     465\ippdbcolumn{summit_id}.  This tells the \ippstage{summitcopy} system
    406466to look for the list of chips, which are then added to another table
    407467(\ippdbtable{summitImfile}).  This system then attempts to download
    … …  
    416476they are further entered into the \ippdbtable{newExp} and
    417477\ippdbtable{newImfile} tables, which index the exposures by
    418 \ippdbcolumn{exp\_id}.  This switch in index indicates that the
     478\ippdbcolumn{exp_id}.  This switch in index indicates that the
    419479exposure has successfully been copied from the summit to the IPP
    420480cluster, and that further processing is no longer dependent on outside
    … …  
    462522
    463523Once the registration process has finished, new science exposures that
    464 have an \ippdbcolumn{obs\_mode} value that indicates they are part of
     524have an \ippdbcolumn{obs_mode} value that indicates they are part of
    465525a particular science survey are automatically launched into the
    466526science analysis by defining entries for the \ippstage{chip}
    … …  
    539599also written to disk.  This metadata is used to populate a row in the
    540600\ippdbtable{chipProcessedImfile} table (linked to the
    541 \ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip\_id} value)
     601\ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip_id} value)
    542602to indicate that the processing of this OTA is complete.
    543603
    … …  
    549609If this condition is met, than all processing for that exposure is
    550610finished, and the \ippdbcolumn{state} field is set to ``full''.  If
    551 the \ippdbtable{chipRun}.\ippdbcolumn{end\_stage} field is set to
     611the \ippdbtable{chipRun}.\ippdbcolumn{end_stage} field is set to
    552612\ippstage{chip}, then no further action is taken.  However, this field
    553613is usually set to a subsequent stage (most often \ippstage{warp}),
    … …  
    572632%% data, with a \ippdbtable{chipRun} characterizing the processing of a
    573633%% single exposure, mapping to a set of \ippdbtable{chipProcessedImfile}
    574 %% entries for each OTA via a common \ippdbcolumn{chip\_id}. 
     634%% entries for each OTA via a common \ippdbcolumn{chip_id}. 
    575635
    576636\subsection{Camera Calibration}
    … …  
    623683\ippdbtable{camProcessedExp} database table.  As the full exposure is
    624684processed all at once, this update also updates the associated
    625 \ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam\_id}.  As
     685\ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam_id}.  As
    626686with the \ippstage{chip} stage, the
    627 \ippdbtable{camRun}.\ippdbcolumn{end\_stage} is for a subsequent
     687\ippdbtable{camRun}.\ippdbcolumn{end_stage} is for a subsequent
    628688stage, an appropriate entry is added to the \ippdbtable{fakeRun}
    629689table.
    … …  
    631691\subsection{Fake Analysis}
    632692\label{sec:fake}
    633 \note{drop}
     693% \note{drop}
    634694
    635695The \ippstage{fake} stage was originally designed to do false source
    … …  
    659719as M31 or other fields of particular interest that can be well
    660720described by a single tangent plane projection, or for larger regions
    661 which have multiple projection centers.  For the $3\Pi$ survey, the
     721which have multiple projection centers.  For the $3\pi$ survey, the
    662722\ippmisc{RINGS.V3} tessellation was used that used projection centers
    663723spaced every four degrees in both RA and DEC, with $0\farcs{}25$
    … …  
    696756When the jobs have completed, an entry for the skycell is added to the
    697757\ippdbtable{warpSkyfile} database table, linked to the
    698 \ippdbtable{warpRun} entry by a common \ippdbcolumn{warp\_id}.  An
     758\ippdbtable{warpRun} entry by a common \ippdbcolumn{warp_id}.  An
    699759\ippmisc{advance} task again checks that all potential skycells have
    700760been generated.  At this point, the direct promotion of exposures from
    … …  
    737797other criteria such as seeing are grouped by their skycell.  An entry
    738798is then added for each skycell in the \ippdbtable{stackRun} table,
    739 with the \ippdbcolumn{warp\_id} entries for the exposures added to the
     799with the \ippdbcolumn{warp_id} entries for the exposures added to the
    740800\ippdbtable{stackInputSkyfile} table, linked to the
    741 \ippdbtable{stackRun} entry by the \ippdbcolumn{stack\_id} field.
     801\ippdbtable{stackRun} entry by the \ippdbcolumn{stack_id} field.
    742802This defines the mapping for which exposures contribute to the
    743803\ippstage{stack}.  This breaks exposures into single skycells, but as
    … …  
    755815along with an exposure time map, and a weighted exposure time map that
    756816scales the exposure time based on the relative variance of each input.
    757 These images for the $3\Pi$ analysis are currently available from the
     817These images for the $3\pi$ analysis are currently available from the
    758818MAST image extraction tools at STSci.
    759819
    … …  
    763823entry, no \ippmisc{advance} job is required.
    764824
    765 \note{end of section needed a re-read}
     825% \note{end of section needed a re-read}
    766826
    767827\subsection{Stack Photometry}
    … …  
    779839Similar to the \ippstage{stack} stage, an entry is created in the
    780840\ippdbtable{staticskyRun} table, linked to a series of rows in the
    781 \ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky\_id},
    782 each of which also contains the appropriate \ippdbcolumn{stack\_id}
     841\ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky_id},
     842each of which also contains the appropriate \ippdbcolumn{stack_id}
    783843entries for the skycell under consideration.
    784844
    … …  
    815875\ippstage{skycal} stage, each skycell is processed independently.
    816876Because of this independence, when queued for processing, the entries
    817 in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky\_id}
    818 and \ippdbcolumn{stack\_id} entries of the parent data directly.  As
     877in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky_id}
     878and \ippdbcolumn{stack_id} entries of the parent data directly.  As
    819879in the \ippstage{camera} stage, the \ippprog{psastro} program reads in
    820880the stack photometry catalog, and produces a calibrated output.  A
    … …  
    886946When processing is queued for this stage, an entry is added to the
    887947\ippdbtable{fullForceRun} primary database table with a reference to
    888 the corresponding stack and \ippdbcolumn{skycal\_id} entry that is the
    889 input source of detections to be measured.  The \ippdbcolumn{warp\_id}
     948the corresponding stack and \ippdbcolumn{skycal_id} entry that is the
     949input source of detections to be measured.  The \ippdbcolumn{warp_id}
    890950values for the input \ippstage{warp} stage images that contributed to
    891 the \ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are
     951the \ippstage{stack} associated with that \ippdbcolumn{skycal_id} are
    892952then added to the \ippdbtable{fullForceInput} table, linked to the
    893 primary table by the \ippdbcolumn{ff\_id} identifier.  The individual
     953primary table by the \ippdbcolumn{ff_id} identifier.  The individual
    894954jobs for each warp are then run, which passes the \ippstage{warp}
    895955stage image products along with the \ippstage{skycal} catalog to the
    … …  
    917977galaxies, discussed below), an entry is added to the
    918978\ippdbtable{fullForceResult} table with the processing statistics for
    919 that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}.
     979that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}.
    920980Once all of the entries in the \ippdbtable{fullForceInput} table have
    921981finished, a summary operation is run to combine the galaxy photometry
    … …  
    10211081skycell that are covered by the images.  For a \ippstage{diff}
    10221082generated from two \ippstage{warp} stage products, the input images
    1023 have their \ippdbcolumn{warp\_id} values recorded in the
     1083have their \ippdbcolumn{warp_id} values recorded in the
    10241084\ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that
    10251085overlaps.  If two \ippstage{stack} stages are to be used in the
    1026 difference, their \ippdbcolumn{stack\_id} entries are recorded in the
     1086difference, their \ippdbcolumn{stack_id} entries are recorded in the
    10271087\ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields.  As each
    10281088\ippstage{stack} only covers a single skycell, the \ippstage{diff} is
    10291089usually defined indirectly, using other information from the
    10301090\ippdbtable{stackRun} table to select appropriate
    1031 \ippdbcolumn{stack\_id} values.  Similarly, \ippstage{diff} processing
     1091\ippdbcolumn{stack_id} values.  Similarly, \ippstage{diff} processing
    10321092is defined for the mixed case by creating entries that populate one of
    10331093\ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of
    … …  
    11641224one of the supporting tables, \ippdbtable{SkyTable}.  This table
    11651225contains the definitions of the boundaries for each sky region
    1166 (\ippdbcolumn{R\_MIN}, \ippdbcolumn{R\_MAX}, \ippdbcolumn{D\_MIN},
    1167 \ippdbcolumn{D\_MAX}), the name of the sky region, an ID
     1226(\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN},
     1227\ippdbcolumn{D_MAX}), the name of the sky region, an ID
    11681228(\ippdbcolumn{INDEX}, equal to the sequence number of the region in
    11691229the table), and index entries to enable navigation within the table.
    … …  
    14161476\ippdbtable{addRun} database table.  This entry notes which
    14171477\ippdbcolumn{stage} is the source of the catalog, and links to the
    1418 appropriate database table with the \ippdbcolumn{stage\_id} field.  As
     1478appropriate database table with the \ippdbcolumn{stage_id} field.  As
    14191479some stages, such as the \ippstage{diff} stage, create more than a
    1420 single catalog, multiple entries with the \ippdbcolumn{stage\_id} are
    1421 created, with the \ippdbcolumn{stage\_extra1} field containing an
     1480single catalog, multiple entries with the \ippdbcolumn{stage_id} are
     1481created, with the \ippdbcolumn{stage_extra1} field containing an
    14221482index to the individual components.  The catalog specified by the
    14231483entry is added to the target \ippmisc{minidvo} by the
    … …  
    14381498
    14391499\section{Operations and Automation}
    1440 
     1500\label{sec:operations}
    14411501
    14421502\subsection{Pantasks and Parallel Processing}
    … …  
    14451505\subsubsection{Pantasks}
    14461506
    1447 Sections~\ref{sec:subsystesm} \& \ref{sec:postprocessing} describe the
     1507Sections~\ref{sec:stages} \& \ref{sec:postprocessing} describe the
    14481508analysis steps which take place in the Pan-STARRS data analysis
    14491509systems.  Individually, these steps appear as commands which could be
    … …  
    14791539may be static or dynamic.  For a task with a static command, the
    14801540command is explicity defined in the task block (see code example in
    1481 Figure~\ref{fig:task_example1}) and is identical each time the task is
     1541Figure~\ref{fig:task_example}) and is identical each time the task is
    14821542executed.  A dynamic command is defined within a special block of the
    14831543task, called \code{task.exec}.  This block is a snipet of code (in the
    … …  
    16671727end 
    16681728\end{verbatim}
    1669  \caption{\label{fig:simple.static.task} Example of a simple static
     1729 \caption{\label{fig:task_example} Example of a simple static
    16701730   task in the opihi-based scripting language used by pantasks.  In
    16711731   this example, pantasks would run a single instance of the command
    1672    \code{ls /tmp} every 5 seconds, sending the stdout and stderr to
     1732   ({\tt ls /tmp}) every 5 seconds, sending the stdout and stderr to
    16731733   the listed files. }
    16741734  \end{center}
    16751735\end{figure}
     1736
     1737%\code{ls /tmp}
    16761738
    16771739\subsubsection{Pantasks scripts: ippTasks}
    … …  
    18691931table defining a new reprocessing.  After this, individual
    18701932\ippdbtable{lapRun} entries can be queued that define a
    1871 \ippdbcolumn{filter} and a \ippdbcolumn{projection\_cell} to be
    1872 considered.  A \ippdbcolumn{projection\_cell} is a unit of sky defined
     1933\ippdbcolumn{filter} and a \ippdbcolumn{projection_cell} to be
     1934considered.  A \ippdbcolumn{projection_cell} is a unit of sky defined
    18731935to be a square four degrees on each side which has a single tangent
    18741936plane projection \citep[][see]{waters2017}.  \note{does waters2017
    … …  
    18801942exposures have been added, the other exposures within the same
    18811943sequence are checked to see if a \ippstage{chip} stage entry has been
    1882 generated, and if so, the \ippdbcolumn{chip\_id} for that entry is
     1944generated, and if so, the \ippdbcolumn{chip_id} for that entry is
    18831945saved into the \ippdbtable{lapExp} as well.  This linkage ensures that
    18841946each exposure is only processed once.  If no entry is found, a new
    … …  
    18871949and if they have all completed the \ippstage{warp} stage, then a
    18881950\ippstage{stack} is queued for each skycell contained within the
    1889 \ippdbcolumn{projection\_cell}.
     1951\ippdbcolumn{projection_cell}.
    18901952
    18911953
    … …  
    19522014string with the form of a UNIX file path: e.g. a/b/c/file.  When a
    19532015program creates a new file in \ippprog{Nebulous}, it supplies a URI of
    1954 the form \code{neb://HOST.VOLUME/PATH/FILE}.  The host and volume
    1955 specifiers are optional, and allow a file to be created on a specific
    1956 node.  The path and filename portions become the identifier and are
    1957 recorded in the \ippmisc{storage_object} table in the
    1958 \ippmisc{extern_id} field.  A storage object entry is then created in
    1959 the database for this id, and an instance of the file created on the
    1960 specified node (or at random from available nodes if left empty).
     2016the form \code{neb://HOST.VOL/PATH/FILE}.  The HOST and VOL(ume)
     2017specifiers are optional, allowing a file to be created on a specific
     2018computer (HOST) and disk (VOL).  The path and filename portions become
     2019the identifier and are recorded in the \ippmisc{storage_object} table
     2020in the \ippmisc{extern_id} field.  A storage object entry is then
     2021created in the database for this id, and an instance of the file
     2022created on the specified node (or at random from available nodes if
     2023left empty).
    19612024
    19622025Files are stored on specific computers in a \ippprog{Nebulous}
    … …  
    19812044can represent a file in the UNIX filesystem.  For the example URI
    19822045above, this results in a file located on disk in a location like
    1983 \code{/data/HOST.VOLUME/nebulous/d5/d8/9876.PATH:FILE}.
     2046\code{/data/HOST.VOL/nebulous/d5/d8/42.PATH:FILE}.
    19842047This file naming structure has the benefit of providing redundancy
    19852048between the filename on disk and the instance in the database.
    … …  
    19992062the user. 
    20002063
    2001 \note{we care about the concepts here, but not the numbers. reword}
    20022064Another column, \ippdbcolumn{xattr}, is used to control the behavior
    20032065of this volume, with specific values used to denote desired behavior.
    … …  
    20652127the provided links.
    20662128
     2129\note{add a discussion of gpc1 filenames?}
     2130
    20672131The IPP also uses datastores to provide access to its own data
    20682132products.  The detections identified in the \ippstage{diff} stage
    … …  
    20792143
    20802144% \section{IPP Software Subsystems}
    2081 % \label{sec:subsystems}
    20822145
    20832146The IPP relies on a number of common libraries and programs to handle
    … …  
    22132276products.  These nodes are also used to do processing, and have jobs
    22142277targeted to them in an effort to reduce the network I/O demands
    2215 (see~\ref{chip section} for more on this process).
     2278(see~\ref{sec:chip} for more on this process).
    22162279
    22172280These storage nodes are not fully capable of completing all processing
    … …  
    22352298\label{sec:LANL}
    22362299
    2237 In order to increase the processing rate for the $3\Pi$ PV3 data, we
     2300In order to increase the processing rate for the $3\pi$ PV3 data, we
    22382301partnered with Los Alamos National Lab to gain access to the Mustang
    22392302supercomputer.  The supercomputer is comprised of 3088 processing
    … …  
    24072470  \tablehead{\colhead{Stage} & \colhead{Primary Table} & \colhead{Secondary Table} & \colhead{Key} & \colhead{Notes}}
    24082471  \startdata
    2409   \ippstage{addstar}      & \ippdbtable{addRun}       & \ippdbtable{addProcessedExp}     & \ippdbcolumn{add\_id} & \\
    2410   \ippstage{camera}       & \ippdbtable{camRun}       & \ippdbtable{camProcessedExp}     & \ippdbcolumn{cam\_id} & \\
    2411   \ippstage{chip}         & \ippdbtable{chipRun}      & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip\_id} & \\
    2412   \ippstage{detrend}      & \ippdbtable{detRun}       & \ippdbtable{detRunSummary}       & \ippdbcolumn{det\_id} & \\
     2472  \ippstage{addstar}      & \ippdbtable{addRun}       & \ippdbtable{addProcessedExp}     & \ippdbcolumn{add_id} & \\
     2473  \ippstage{camera}       & \ippdbtable{camRun}       & \ippdbtable{camProcessedExp}     & \ippdbcolumn{cam_id} & \\
     2474  \ippstage{chip}         & \ippdbtable{chipRun}      & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} & \\
     2475  \ippstage{detrend}      & \ippdbtable{detRun}       & \ippdbtable{detRunSummary}       & \ippdbcolumn{det_id} & \\
    24132476                          &                           & \ippdbtable{detInputExp}         & & \\
    24142477                          &                           & \ippdbtable{detRegisteredImfile} & & Information about detrends produced externally.\\
    … …  
    24172480                          & \ippdbtable{detResidExp}  & \ippdbtable{detResidImfile}      & & \\
    24182481                          & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & & \\
    2419   \ippstage{diff}         & \ippdbtable{diffRun}      & \ippdbtable{diffSkyfile}         & \ippdbcolumn{diff\_id} & \\
     2482  \ippstage{diff}         & \ippdbtable{diffRun}      & \ippdbtable{diffSkyfile}         & \ippdbcolumn{diff_id} & \\
    24202483                          &                           & \ippdbtable{diffInputSkyfile}    & & \\
    2421   \ippstage{distribution} & \ippdbtable{distRun}      & \ippdbtable{distComponent}       & \ippdbcolumn{dist\_id} & \\
     2484  \ippstage{distribution} & \ippdbtable{distRun}      & \ippdbtable{distComponent}       & \ippdbcolumn{dist_id} & \\
    24222485                          &                           & \ippdbtable{distTarget}          & & \\
    2423   \ippstage{fake}         & \ippdbtable{fakeRun}      & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake\_id} & \\
    2424   \ippstage{fullforce}    & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput}      & \ippdbcolumn{ff\_id} & \\
     2486  \ippstage{fake}         & \ippdbtable{fakeRun}      & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} & \\
     2487  \ippstage{fullforce}    & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput}      & \ippdbcolumn{ff_id} & \\
    24252488                          &                           & \ippdbtable{fullForceResult}     & & \\
    24262489                          &                           & \ippdbtable{fullForceSummary}    & & Properties about average parameters from all results.\\
    2427   \ippstage{lap}          & \ippdbtable{lapSequence}  & \ippdbtable{lapRun}              & \ippdbcolumn{seq\_id} & Sequence of full reprocessing\\
    2428                           & \ippdbtable{lapRun}       & \ippdbtable{lapExp}              & \ippdbcolumn{lap\_id} & \\
    2429   \ippstage{publish}      & \ippdbtable{publishRun}   & \ippdbtable{publishDone}         & \ippdbcolumn{pub\_id} & \\
     2490  \ippstage{lap}          & \ippdbtable{lapSequence}  & \ippdbtable{lapRun}              & \ippdbcolumn{seq_id} & Sequence of full reprocessing\\
     2491                          & \ippdbtable{lapRun}       & \ippdbtable{lapExp}              & \ippdbcolumn{lap_id} & \\
     2492  \ippstage{publish}      & \ippdbtable{publishRun}   & \ippdbtable{publishDone}         & \ippdbcolumn{pub_id} & \\
    24302493                          &                           & \ippdbtable{publishClient}       & & \\
    24312494  \ippstage{summitcopy}   & \ippdbtable{pzDataStore}  &                                  & & Lists locations to check for new exposures.\\
    2432                           & \ippdbtable{summitExp}    & \ippdbtable{summitImfile}        & \ippdbcolumn{summit\_id} & Exposures available at the telescope.\\
     2495                          & \ippdbtable{summitExp}    & \ippdbtable{summitImfile}        & \ippdbcolumn{summit_id} & Exposures available at the telescope.\\
    24332496                          & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile}    & & Exposures that are being downloaded.\\
    2434                           & \ippdbtable{newExp}       & \ippdbtable{newImfile}           & \ippdbcolumn{exp\_id} & Exposures that have been saved to IPP cluster.\\
    2435 
    2436   \ippstage{registration} & \ippdbtable{rawExp}       & \ippdbtable{rawImfile}           & \ippdbcolumn{exp\_id} & \\
    2437   \ippstage{remote}       & \ippdbtable{remoteRun}    & \ippdbtable{remoteComponent}     & \ippdbcolumn{remote\_id} & \\
    2438   \ippstage{skycal}       & \ippdbtable{skycalRun}    & \ippdbtable{skycalResult}        & \ippdbcolumn{skycal\_id} & \\
    2439   \ippstage{stack}        & \ippdbtable{stackRun}     & \ippdbtable{stackInputSkyfile}   & \ippdbcolumn{stack\_id} & \\
     2497                          & \ippdbtable{newExp}       & \ippdbtable{newImfile}           & \ippdbcolumn{exp_id} & Exposures that have been saved to IPP cluster.\\
     2498
     2499  \ippstage{registration} & \ippdbtable{rawExp}       & \ippdbtable{rawImfile}           & \ippdbcolumn{exp_id} & \\
     2500  \ippstage{remote}       & \ippdbtable{remoteRun}    & \ippdbtable{remoteComponent}     & \ippdbcolumn{remote_id} & \\
     2501  \ippstage{skycal}       & \ippdbtable{skycalRun}    & \ippdbtable{skycalResult}        & \ippdbcolumn{skycal_id} & \\
     2502  \ippstage{stack}        & \ippdbtable{stackRun}     & \ippdbtable{stackInputSkyfile}   & \ippdbcolumn{stack_id} & \\
    24402503                          &                           & \ippdbtable{stackSumSkyfile}     & & \\
    2441   \ippstage{staticsky}    & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput}      & \ippdbcolumn{sky\_id} & \\
     2504  \ippstage{staticsky}    & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput}      & \ippdbcolumn{sky_id} & \\
    24422505                          &                           & \ippdbtable{staticskyResult}     & & \\
    2443   \ippstage{warp}         & \ippdbtable{warpRun}      & \ippdbtable{warpImfile}          & \ippdbcolumn{warp\_id} & \\
     2506  \ippstage{warp}         & \ippdbtable{warpRun}      & \ippdbtable{warpImfile}          & \ippdbcolumn{warp_id} & \\
    24442507                          &                           & \ippdbtable{warpSkyCellMap}      & & Mapping of input chips to projection skycells.\\
    24452508                          &                           & \ippdbtable{warpSkyfile}         & & \\
Note: See TracChangeset for help on using the changeset viewer.