Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40022) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40023) @@ -1,5 +1,6 @@ -\documentclass[iop,floatfix]{emulateapj} +% \documentclass[iop,floatfix]{emulateapj} % \documentclass[iop,floatfix,onecolumn]{emulateapj} % \documentclass[12pt,preprint]{aastex} +\documentclass[10pt,preprint]{aastex} % \pdfoutput=1 @@ -7,4 +8,6 @@ \RequirePackage{code} \input{astro.sty} + +\usepackage[T1]{fontenc}% (2) specify encoding % online version may use color, but print version needs b/w @@ -87,6 +90,47 @@ \keywords{Surveys:\PSONE } +\begin{verbatim} +MAJOR TODO ITEMS: +* introduce and describe RINGS.V3 in or before warp section (refer to Waters if appropriate) +* re-read and trim details as needed (referring to the other papers) +* re-write the DVO description using topics list given +* write discussion of calibration operations (refer to cal paper) +* write IPP to PSPS summary +* write PSPS Load and Merge summary (use Flewelling paper for ref) +* add some specific numbers (data volume, processing times, etc) +\end{verbatim} + \section{Introduction} \label{sec:intro} + +The 1.8m Pan-STARRS\,1 telescope is located on the summit of Haleakala +on the Hawaiian island of Maui. The wide-field optical design of the +telescope \citep{PS1.optics} produces a 3.3 degree field of view with +low distortion and minimal vignetting even at the edges of the +illuminated region. The optics and natural seeing combine to yield +good image quality: 75\% of the images have full-width half-max values +less than (1.51, 1.39, 1.34, 1.27, 1.21) arcseconds for (\grizy), with +a floor of $\sim 0.7$ arcseconds. + +The \PSONE\ camera \citep{PS1.GPCA}, known as GPC1, consists of a +mosaic of 60 back-illuminated CCDs manufactured by Lincoln Laboratory. +The CCDs each consist of an $8\times8$ grid of $\sim 600\times 600$ +pixel readout regions, yielding an effective $4800\times4800$ +detector. Initial performance assessments are presented in +\cite{PS1.GPCB}. Routine observations are conducted remotely from the +Advanced Technology Research Center in Kula, the main facility of the +University of Hawaii's Institute for Astronomy operations on Maui. + +For nearly 4 years, from 2010 May through 2014 March, this telescope +was used to perform a collection of astronomical surveys under the +aegis of the Pan-STARRS Science Consortium. The majority of the time +(56\%) was spent on surveying the $\frac{3}{4}$ of the sky north of +$-30$ Declination with \grizy\ filters in the so-called $3\pi$ Survey. +Another $\sim 25\%$ of the time was concentrated on repeated deep +observations of 10 specific fields in the Medium-Deep Survey. The +rest of the time was used for several other surveys, including a +search for potentially hazardous asteroids in our solar system. The +details of the telescope, surveys, and resulting science publications +are described by \cite{Chambers}. This is the second in a series of seven papers describing the @@ -143,8 +187,11 @@ described in detail in \cite{2012ApJ...750...99T}. -The Pan-STARRS Image Processing Pipeline consists of a suite of -software programs and data systems that are designed to reduce -astronomical images, with a focus on parallelization necessary to -speed the processing of the large images produced by the GPC1 camera. +This paper presents a description of the Pan-STARRS data handling +systems, with an emphasis on the Image Processing Pipeline (IPP). The +Pan-STARRS Image Processing Pipeline consists of a suite of software +programs and data systems that are designed to reduce astronomical +images, with the parallelization necessary to speed the processing of +the large images produced by the GPC1 camera. + Part of this parallelization is derived from the fact that this camera consists of 60 independent orthogonal transfer array (OTA) devices, @@ -153,18 +200,21 @@ majority of stages operate only on smaller segments of a full exposure to allow the processing tasks to be spread over the machines in the -processing cluster. - -This paper presents a description of the IPP data handling system. -Section \ref{sec:subsystems} describes the major IPP subsystems that -underlie the main pipeline, providing a set of common interfaces and -tools used at multiple stages. The main processing stages of the -pipeline are described in Section \ref{sec:stages}, although all -exposures may not necessarily pass through each of these stages. The -hardware systems that have done the processing for the PV3 data -release are listed in Section \ref{sec:hardware}, with some details -on the scale of computing needed to reduce this large number of -exposures. Finally, Section \ref{sec:discussion} presents a -discussion of some of the lessons learned in the creation of the IPP, -and its utility in reducing data from other cameras and telescopes. +processing cluster. \note{move elsewhere?} + +Section~\ref{sec:overview} provides an overview of the full data +analysis system and breaks down the major elements of the Image +Processing Pipeline. Section~\ref{sec:stages} discusses in some +detail each of the analysis steps which may be applied to the images +and resulting catalogs of detected sources. +Section~\ref{sec:postprocessing} discusses the calibration operations +and database used for calibration. Section~\ref{sec:operations} +discusses the operational infrastructure of the IPP. +Section~\ref{sec:hardware} discusses the hardware systems used by the +IPP for regular nightly operations and for processing the PV3 data +release, with some details on the scale of computing needed to reduce +this large number of exposures. Finally, Section~\ref{sec:discussion} +presents a discussion of some of the lessons learned in the creation +of the IPP, and its utility in reducing data from other cameras and +telescopes. {\color{red} {\em Note: These papers are being placed on arXiv.org to @@ -186,8 +236,8 @@ transient and moving object science; large-scale re-processing and calibration to produce measurements for the science collaboration and -the wider public; \note{manual/specialized} image processing to -facilitate research and development of the analysis system itself; -distribution of the resulting data products to various consumers in a -variety of formats and modes. +the wider public; specialized image processing to facilitate research +and development of the analysis system itself; and distribution of the +resulting data products to various consumers in a variety of formats +and modes. The Pan-STARRS Data Analysis system is divided internally into several major @@ -204,5 +254,5 @@ \item Moving Object Processing System (MOPS) : this system is responsible for linking individual detections of solar-system - objects together and determining the orbits. \note{Denneau REF} + objects together and determining the orbits \citep[][]{2013PASP..125..357D}. \item PSPS : this system ingests the calibrated measurements from the IPP, MOPS, and others and generates a high-availability database @@ -229,6 +279,18 @@ analysis steps which occur within the Pan-STARRS observatory, with an emphasis on the analysis, calibration, and database ingest stages. -The MOPS is described in detail by \cite{MOPS}, while the summit -systems are described by \note{REF?}. +The MOPS is described in detail by \cite{2013PASP..125..357D}, while +the summit systems are described by \note{REF?}. + +\begin{figure*}[htbp] + \begin{center} + \includegraphics[width=\hsize,clip]{PS1_Data_Analysis_System_Overview.pdf} + \caption{\label{fig:analysis.elements} Elements of the Pan-STARRS\,1 + Data Analysis System. Rectangles represent data analysis steps; + ellipses represent databases; rounded rectangles represent + external groups (``customers''). The arrows show a simplified representation + of the major flow of data between the analysis stages and data + processing elements.} + \end{center} +\end{figure*} \subsection{Nightly Processing Analysis Stages} @@ -256,10 +318,8 @@ appropriate part of the sky. -\note{need earlier mention of 3pi, MD, etc} - \subsection{Re-processing Analysis Stages} Pan-STARRS has performed several large-scale reprocessings of both the -Medium Deep and 3pi Survey data. For the 3pi Survey data, we identify +Medium Deep and $3\pi$ Survey data. For the $3\pi$ Survey data, we identify these large-scale reprocessings as PV1, PV2, and PV3 (we also define the nightly science analysis of the data as PV0). For these @@ -277,5 +337,5 @@ analysis stages are ingested into the internal calibration database (DVO, the Desktop Virtual Observatory) and used for photometric and -astrometric calibrations (see Section~\ref{sec:dvo}) +astrometric calibrations (see Section~\ref{sec:DVO}) \subsection{Data Access and Distribution} @@ -383,5 +443,5 @@ glitches or hardware crashes. -\note{start of section needed a re-read} +% \note{start of section needed a re-read} \subsection{Summit copy} @@ -403,5 +463,5 @@ (\ippdbtable{summitExp}), indexed by an identifier that simply increments the number of exposures announced by the summit, the -\ippdbcolumn{summit\_id}. This tells the \ippstage{summitcopy} system +\ippdbcolumn{summit_id}. This tells the \ippstage{summitcopy} system to look for the list of chips, which are then added to another table (\ippdbtable{summitImfile}). This system then attempts to download @@ -416,5 +476,5 @@ they are further entered into the \ippdbtable{newExp} and \ippdbtable{newImfile} tables, which index the exposures by -\ippdbcolumn{exp\_id}. This switch in index indicates that the +\ippdbcolumn{exp_id}. This switch in index indicates that the exposure has successfully been copied from the summit to the IPP cluster, and that further processing is no longer dependent on outside @@ -462,5 +522,5 @@ Once the registration process has finished, new science exposures that -have an \ippdbcolumn{obs\_mode} value that indicates they are part of +have an \ippdbcolumn{obs_mode} value that indicates they are part of a particular science survey are automatically launched into the science analysis by defining entries for the \ippstage{chip} @@ -539,5 +599,5 @@ also written to disk. This metadata is used to populate a row in the \ippdbtable{chipProcessedImfile} table (linked to the -\ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip\_id} value) +\ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip_id} value) to indicate that the processing of this OTA is complete. @@ -549,5 +609,5 @@ If this condition is met, than all processing for that exposure is finished, and the \ippdbcolumn{state} field is set to ``full''. If -the \ippdbtable{chipRun}.\ippdbcolumn{end\_stage} field is set to +the \ippdbtable{chipRun}.\ippdbcolumn{end_stage} field is set to \ippstage{chip}, then no further action is taken. However, this field is usually set to a subsequent stage (most often \ippstage{warp}), @@ -572,5 +632,5 @@ %% data, with a \ippdbtable{chipRun} characterizing the processing of a %% single exposure, mapping to a set of \ippdbtable{chipProcessedImfile} -%% entries for each OTA via a common \ippdbcolumn{chip\_id}. +%% entries for each OTA via a common \ippdbcolumn{chip_id}. \subsection{Camera Calibration} @@ -623,7 +683,7 @@ \ippdbtable{camProcessedExp} database table. As the full exposure is processed all at once, this update also updates the associated -\ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam\_id}. As +\ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam_id}. As with the \ippstage{chip} stage, the -\ippdbtable{camRun}.\ippdbcolumn{end\_stage} is for a subsequent +\ippdbtable{camRun}.\ippdbcolumn{end_stage} is for a subsequent stage, an appropriate entry is added to the \ippdbtable{fakeRun} table. @@ -631,5 +691,5 @@ \subsection{Fake Analysis} \label{sec:fake} -\note{drop} +% \note{drop} The \ippstage{fake} stage was originally designed to do false source @@ -659,5 +719,5 @@ as M31 or other fields of particular interest that can be well described by a single tangent plane projection, or for larger regions -which have multiple projection centers. For the $3\Pi$ survey, the +which have multiple projection centers. For the $3\pi$ survey, the \ippmisc{RINGS.V3} tessellation was used that used projection centers spaced every four degrees in both RA and DEC, with $0\farcs{}25$ @@ -696,5 +756,5 @@ When the jobs have completed, an entry for the skycell is added to the \ippdbtable{warpSkyfile} database table, linked to the -\ippdbtable{warpRun} entry by a common \ippdbcolumn{warp\_id}. An +\ippdbtable{warpRun} entry by a common \ippdbcolumn{warp_id}. An \ippmisc{advance} task again checks that all potential skycells have been generated. At this point, the direct promotion of exposures from @@ -737,7 +797,7 @@ other criteria such as seeing are grouped by their skycell. An entry is then added for each skycell in the \ippdbtable{stackRun} table, -with the \ippdbcolumn{warp\_id} entries for the exposures added to the +with the \ippdbcolumn{warp_id} entries for the exposures added to the \ippdbtable{stackInputSkyfile} table, linked to the -\ippdbtable{stackRun} entry by the \ippdbcolumn{stack\_id} field. +\ippdbtable{stackRun} entry by the \ippdbcolumn{stack_id} field. This defines the mapping for which exposures contribute to the \ippstage{stack}. This breaks exposures into single skycells, but as @@ -755,5 +815,5 @@ along with an exposure time map, and a weighted exposure time map that scales the exposure time based on the relative variance of each input. -These images for the $3\Pi$ analysis are currently available from the +These images for the $3\pi$ analysis are currently available from the MAST image extraction tools at STSci. @@ -763,5 +823,5 @@ entry, no \ippmisc{advance} job is required. -\note{end of section needed a re-read} +% \note{end of section needed a re-read} \subsection{Stack Photometry} @@ -779,6 +839,6 @@ Similar to the \ippstage{stack} stage, an entry is created in the \ippdbtable{staticskyRun} table, linked to a series of rows in the -\ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky\_id}, -each of which also contains the appropriate \ippdbcolumn{stack\_id} +\ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky_id}, +each of which also contains the appropriate \ippdbcolumn{stack_id} entries for the skycell under consideration. @@ -815,6 +875,6 @@ \ippstage{skycal} stage, each skycell is processed independently. Because of this independence, when queued for processing, the entries -in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky\_id} -and \ippdbcolumn{stack\_id} entries of the parent data directly. As +in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky_id} +and \ippdbcolumn{stack_id} entries of the parent data directly. As in the \ippstage{camera} stage, the \ippprog{psastro} program reads in the stack photometry catalog, and produces a calibrated output. A @@ -886,10 +946,10 @@ When processing is queued for this stage, an entry is added to the \ippdbtable{fullForceRun} primary database table with a reference to -the corresponding stack and \ippdbcolumn{skycal\_id} entry that is the -input source of detections to be measured. The \ippdbcolumn{warp\_id} +the corresponding stack and \ippdbcolumn{skycal_id} entry that is the +input source of detections to be measured. The \ippdbcolumn{warp_id} values for the input \ippstage{warp} stage images that contributed to -the \ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are +the \ippstage{stack} associated with that \ippdbcolumn{skycal_id} are then added to the \ippdbtable{fullForceInput} table, linked to the -primary table by the \ippdbcolumn{ff\_id} identifier. The individual +primary table by the \ippdbcolumn{ff_id} identifier. The individual jobs for each warp are then run, which passes the \ippstage{warp} stage image products along with the \ippstage{skycal} catalog to the @@ -917,5 +977,5 @@ galaxies, discussed below), an entry is added to the \ippdbtable{fullForceResult} table with the processing statistics for -that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}. +that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}. Once all of the entries in the \ippdbtable{fullForceInput} table have finished, a summary operation is run to combine the galaxy photometry @@ -1021,13 +1081,13 @@ skycell that are covered by the images. For a \ippstage{diff} generated from two \ippstage{warp} stage products, the input images -have their \ippdbcolumn{warp\_id} values recorded in the +have their \ippdbcolumn{warp_id} values recorded in the \ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that overlaps. If two \ippstage{stack} stages are to be used in the -difference, their \ippdbcolumn{stack\_id} entries are recorded in the +difference, their \ippdbcolumn{stack_id} entries are recorded in the \ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each \ippstage{stack} only covers a single skycell, the \ippstage{diff} is usually defined indirectly, using other information from the \ippdbtable{stackRun} table to select appropriate -\ippdbcolumn{stack\_id} values. Similarly, \ippstage{diff} processing +\ippdbcolumn{stack_id} values. Similarly, \ippstage{diff} processing is defined for the mixed case by creating entries that populate one of \ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of @@ -1164,6 +1224,6 @@ one of the supporting tables, \ippdbtable{SkyTable}. This table contains the definitions of the boundaries for each sky region -(\ippdbcolumn{R\_MIN}, \ippdbcolumn{R\_MAX}, \ippdbcolumn{D\_MIN}, -\ippdbcolumn{D\_MAX}), the name of the sky region, an ID +(\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, +\ippdbcolumn{D_MAX}), the name of the sky region, an ID (\ippdbcolumn{INDEX}, equal to the sequence number of the region in the table), and index entries to enable navigation within the table. @@ -1416,8 +1476,8 @@ \ippdbtable{addRun} database table. This entry notes which \ippdbcolumn{stage} is the source of the catalog, and links to the -appropriate database table with the \ippdbcolumn{stage\_id} field. As +appropriate database table with the \ippdbcolumn{stage_id} field. As some stages, such as the \ippstage{diff} stage, create more than a -single catalog, multiple entries with the \ippdbcolumn{stage\_id} are -created, with the \ippdbcolumn{stage\_extra1} field containing an +single catalog, multiple entries with the \ippdbcolumn{stage_id} are +created, with the \ippdbcolumn{stage_extra1} field containing an index to the individual components. The catalog specified by the entry is added to the target \ippmisc{minidvo} by the @@ -1438,5 +1498,5 @@ \section{Operations and Automation} - +\label{sec:operations} \subsection{Pantasks and Parallel Processing} @@ -1445,5 +1505,5 @@ \subsubsection{Pantasks} -Sections~\ref{sec:subsystesm} \& \ref{sec:postprocessing} describe the +Sections~\ref{sec:stages} \& \ref{sec:postprocessing} describe the analysis steps which take place in the Pan-STARRS data analysis systems. Individually, these steps appear as commands which could be @@ -1479,5 +1539,5 @@ may be static or dynamic. For a task with a static command, the command is explicity defined in the task block (see code example in -Figure~\ref{fig:task_example1}) and is identical each time the task is +Figure~\ref{fig:task_example}) and is identical each time the task is executed. A dynamic command is defined within a special block of the task, called \code{task.exec}. This block is a snipet of code (in the @@ -1667,11 +1727,13 @@ end \end{verbatim} - \caption{\label{fig:simple.static.task} Example of a simple static + \caption{\label{fig:task_example} Example of a simple static task in the opihi-based scripting language used by pantasks. In this example, pantasks would run a single instance of the command - \code{ls /tmp} every 5 seconds, sending the stdout and stderr to + ({\tt ls /tmp}) every 5 seconds, sending the stdout and stderr to the listed files. } \end{center} \end{figure} + +%\code{ls /tmp} \subsubsection{Pantasks scripts: ippTasks} @@ -1869,6 +1931,6 @@ table defining a new reprocessing. After this, individual \ippdbtable{lapRun} entries can be queued that define a -\ippdbcolumn{filter} and a \ippdbcolumn{projection\_cell} to be -considered. A \ippdbcolumn{projection\_cell} is a unit of sky defined +\ippdbcolumn{filter} and a \ippdbcolumn{projection_cell} to be +considered. A \ippdbcolumn{projection_cell} is a unit of sky defined to be a square four degrees on each side which has a single tangent plane projection \citep[][see]{waters2017}. \note{does waters2017 @@ -1880,5 +1942,5 @@ exposures have been added, the other exposures within the same sequence are checked to see if a \ippstage{chip} stage entry has been -generated, and if so, the \ippdbcolumn{chip\_id} for that entry is +generated, and if so, the \ippdbcolumn{chip_id} for that entry is saved into the \ippdbtable{lapExp} as well. This linkage ensures that each exposure is only processed once. If no entry is found, a new @@ -1887,5 +1949,5 @@ and if they have all completed the \ippstage{warp} stage, then a \ippstage{stack} is queued for each skycell contained within the -\ippdbcolumn{projection\_cell}. +\ippdbcolumn{projection_cell}. @@ -1952,11 +2014,12 @@ string with the form of a UNIX file path: e.g. a/b/c/file. When a program creates a new file in \ippprog{Nebulous}, it supplies a URI of -the form \code{neb://HOST.VOLUME/PATH/FILE}. The host and volume -specifiers are optional, and allow a file to be created on a specific -node. The path and filename portions become the identifier and are -recorded in the \ippmisc{storage_object} table in the -\ippmisc{extern_id} field. A storage object entry is then created in -the database for this id, and an instance of the file created on the -specified node (or at random from available nodes if left empty). +the form \code{neb://HOST.VOL/PATH/FILE}. The HOST and VOL(ume) +specifiers are optional, allowing a file to be created on a specific +computer (HOST) and disk (VOL). The path and filename portions become +the identifier and are recorded in the \ippmisc{storage_object} table +in the \ippmisc{extern_id} field. A storage object entry is then +created in the database for this id, and an instance of the file +created on the specified node (or at random from available nodes if +left empty). Files are stored on specific computers in a \ippprog{Nebulous} @@ -1981,5 +2044,5 @@ can represent a file in the UNIX filesystem. For the example URI above, this results in a file located on disk in a location like -\code{/data/HOST.VOLUME/nebulous/d5/d8/9876.PATH:FILE}. +\code{/data/HOST.VOL/nebulous/d5/d8/42.PATH:FILE}. This file naming structure has the benefit of providing redundancy between the filename on disk and the instance in the database. @@ -1999,5 +2062,4 @@ the user. -\note{we care about the concepts here, but not the numbers. reword} Another column, \ippdbcolumn{xattr}, is used to control the behavior of this volume, with specific values used to denote desired behavior. @@ -2065,4 +2127,6 @@ the provided links. +\note{add a discussion of gpc1 filenames?} + The IPP also uses datastores to provide access to its own data products. The detections identified in the \ippstage{diff} stage @@ -2079,5 +2143,4 @@ % \section{IPP Software Subsystems} -% \label{sec:subsystems} The IPP relies on a number of common libraries and programs to handle @@ -2213,5 +2276,5 @@ products. These nodes are also used to do processing, and have jobs targeted to them in an effort to reduce the network I/O demands -(see~\ref{chip section} for more on this process). +(see~\ref{sec:chip} for more on this process). These storage nodes are not fully capable of completing all processing @@ -2235,5 +2298,5 @@ \label{sec:LANL} -In order to increase the processing rate for the $3\Pi$ PV3 data, we +In order to increase the processing rate for the $3\pi$ PV3 data, we partnered with Los Alamos National Lab to gain access to the Mustang supercomputer. The supercomputer is comprised of 3088 processing @@ -2407,8 +2470,8 @@ \tablehead{\colhead{Stage} & \colhead{Primary Table} & \colhead{Secondary Table} & \colhead{Key} & \colhead{Notes}} \startdata - \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add\_id} & \\ - \ippstage{camera} & \ippdbtable{camRun} & \ippdbtable{camProcessedExp} & \ippdbcolumn{cam\_id} & \\ - \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip\_id} & \\ - \ippstage{detrend} & \ippdbtable{detRun} & \ippdbtable{detRunSummary} & \ippdbcolumn{det\_id} & \\ + \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add_id} & \\ + \ippstage{camera} & \ippdbtable{camRun} & \ippdbtable{camProcessedExp} & \ippdbcolumn{cam_id} & \\ + \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} & \\ + \ippstage{detrend} & \ippdbtable{detRun} & \ippdbtable{detRunSummary} & \ippdbcolumn{det_id} & \\ & & \ippdbtable{detInputExp} & & \\ & & \ippdbtable{detRegisteredImfile} & & Information about detrends produced externally.\\ @@ -2417,29 +2480,29 @@ & \ippdbtable{detResidExp} & \ippdbtable{detResidImfile} & & \\ & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & & \\ - \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff\_id} & \\ + \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff_id} & \\ & & \ippdbtable{diffInputSkyfile} & & \\ - \ippstage{distribution} & \ippdbtable{distRun} & \ippdbtable{distComponent} & \ippdbcolumn{dist\_id} & \\ + \ippstage{distribution} & \ippdbtable{distRun} & \ippdbtable{distComponent} & \ippdbcolumn{dist_id} & \\ & & \ippdbtable{distTarget} & & \\ - \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake\_id} & \\ - \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff\_id} & \\ + \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} & \\ + \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff_id} & \\ & & \ippdbtable{fullForceResult} & & \\ & & \ippdbtable{fullForceSummary} & & Properties about average parameters from all results.\\ - \ippstage{lap} & \ippdbtable{lapSequence} & \ippdbtable{lapRun} & \ippdbcolumn{seq\_id} & Sequence of full reprocessing\\ - & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap\_id} & \\ - \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub\_id} & \\ + \ippstage{lap} & \ippdbtable{lapSequence} & \ippdbtable{lapRun} & \ippdbcolumn{seq_id} & Sequence of full reprocessing\\ + & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap_id} & \\ + \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub_id} & \\ & & \ippdbtable{publishClient} & & \\ \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & & Lists locations to check for new exposures.\\ - & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit\_id} & Exposures available at the telescope.\\ + & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} & Exposures available at the telescope.\\ & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile} & & Exposures that are being downloaded.\\ - & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp\_id} & Exposures that have been saved to IPP cluster.\\ - - \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp\_id} & \\ - \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote\_id} & \\ - \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal\_id} & \\ - \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack\_id} & \\ + & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp_id} & Exposures that have been saved to IPP cluster.\\ + + \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp_id} & \\ + \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote_id} & \\ + \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal_id} & \\ + \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack_id} & \\ & & \ippdbtable{stackSumSkyfile} & & \\ - \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky\_id} & \\ + \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky_id} & \\ & & \ippdbtable{staticskyResult} & & \\ - \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp\_id} & \\ + \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp_id} & \\ & & \ippdbtable{warpSkyCellMap} & & Mapping of input chips to projection skycells.\\ & & \ippdbtable{warpSkyfile} & & \\