Index: /trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40129) +++ /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40130) @@ -93,4 +93,6 @@ \label{sec:intro} +\note{missing figures: analysis elements, DVO schema} + The 1.8m Pan-STARRS\,1 telescope is located on the summit of Haleakala on the Hawaiian island of Maui. The wide-field optical design of the @@ -104,10 +106,10 @@ The \PSONE\ camera \citep{2009amos.confE..40T}, known as GPC1, consists of a mosaic of 60 back-illuminated CCDs manufactured by Lincoln Laboratory. -The CCDs each consist of an $8\times8$ grid of $\sim 600\times 600$ -pixel readout regions, yielding an effective $4800\times4800$ +The CCDs each consist of an $8\times8$ grid of $590 \times 598$ +pixel readout regions, yielding an effective $4846 \times 4868$ detector. Initial performance assessments are presented in \cite{2008SPIE.7014E..0DO}. Routine observations are conducted remotely from the Advanced Technology Research Center in Kula, the main facility of the -University of Hawaii's Institute for Astronomy operations on Maui. +University of Hawaii's Institute for Astronomy (IfA) operations on Maui. The Pan-STARRS1 filters and photometric system have already been described in detail in \cite{2012ApJ...750...99T}. @@ -167,5 +169,5 @@ %Pan-STARRS Pixel Analysis : Source Detection \citet[][Paper IV]{magnier2017.analysis} -describes the details of the source detection and photometry, including point-spread-function and extended source fitting models, and the techniques for ``forced" photometry measurements. +describes the details of the source detection and photometry, including point-spread-function and extended source fitting models, and the techniques for ``forced'' photometry measurements. %Magnier et al. 2017 (Paper V) @@ -202,35 +204,4 @@ reducing data from other cameras and telescopes. -\note{overview discussion of Pan-STARRS: the telescope, survey time - period, surveys. 2 paragraphs.} - -The Pan-STARRS Image Processing Pipeline consists of a suite of -software programs and data systems that are designed to reduce -astronomical images, with a focus on parallelization necessary to -speed the processing of the large images produced by the GPC1 camera. -Part of this parallelization is derived from the fact that this camera -consists of 60 independent orthogonal transfer array (OTA) devices, -and can therefore be processed simultaneously. Although there are -multiple stages that operate on an entire exposure at once, the -majority of stages operate only on smaller segments of a full exposure -to allow the processing tasks to be spread over the machines in the -processing cluster. - - -\note{fix this summary once outline is solidified} - -This paper presents a description of the IPP data handling system. -Section \ref{sec:subsystems} describes the major IPP subsystems that -underlie the main pipeline, providing a set of common interfaces and -tools used at multiple stages. The main processing stages of the -pipeline are described in Section \ref{sec:stages}, although all -exposures may not necessarily pass through each of these stages. The -hardware systems that have done the processing for the PV3 data -release are listed in Section \ref{sec:hardware}, with some details -on the scale of computing needed to reduce this large number of -exposures. Finally, Section \ref{sec:discussion} presents a -discussion of some of the lessons learned in the creation of the IPP, -and its utility in reducing data from other cameras and telescopes. - {\color{red} {\em Note: These papers are being placed on arXiv.org to provide crucial support information at the time of the public @@ -244,15 +215,17 @@ \label{sec:overview} -The Pan-STARRS Data Analysis system consists of many elements to -support the wide range of activities: archiving and management of the +\subsection{Elements of the Pan-STARRS Data Processing System} + +The Pan-STARRS data analysis system consists of many elements to +support a wide range of activities: archiving and management of the raw and processed image files; real-time nightly processing of images for transient and moving object science; large-scale re-processing and calibration to produce measurements for the science collaboration and -the wider public; specialized image processing tasks to facilitate -research and development of the analysis system itself; distribution -of the resulting data products to various consumers in a variety of -formats and modes. - -The Pan-STARRS Data Analysis system is divided internally into several major +the wider public; specialized image processing to facilitate research +and development of the analysis system itself; and distribution of the +resulting data products to various consumers in a variety of formats +and modes. + +The Pan-STARRS data analysis system is divided internally into several major components: \begin{itemize} @@ -260,5 +233,5 @@ data analysis tasks needed to support the on-going observations. In this article, we focus only on those aspects used by the off-summit - analysis stages. \note{is summit processing discussed anywhere?} + analysis stages. \item Image Processing Pipeline (IPP) : this portion of the data analysis system takes the data from raw pixels on the summit @@ -295,4 +268,18 @@ the summit systems are described by \note{REF?}. +\begin{figure*}[htbp] + \begin{center} + \includegraphics[width=\hsize,clip]{PS1_Data_Analysis_System_Overview.pdf} + \caption{\label{fig:analysis.elements} Elements of the Pan-STARRS\,1 + Data Analysis System. Rectangles represent data analysis steps; + ellipses represent databases; rounded rectangles represent + external groups (``customers''). The arrows show a simplified representation + of the major flow of data between the analysis stages and data + processing elements.} + \end{center} +\end{figure*} + +\subsection{Nightly Processing Analysis Stages} + Data analysis to support nightly science operations is driven by two main goals: 1) rapid detection of the moving and transient sources to @@ -309,33 +296,36 @@ (\IPPstage{warp}). Warped images may either be added together (\IPPstage{stack}) or used in an image subtraction (\IPPstage{diff}). -For nightly science operations, images for certain fields such as the -Medium Deep survey fields \citep[see][]{MDref}, are stacked together -in nightly chunks, providing deeper detection capability on 1-day -timescales. Depending on the survey mode, difference images are -generated for the nightly stack images (vs a deep stack template) or -for individual warp images. In the later case, the warp images may be -difference against another warp from the same night or against a +As part of nightly science processing, images for certain fields such +as the Medium Deep survey fields \citep[see][]{MDref}, are stacked +together in nightly chunks, providing deeper detection capability on +1-day timescales. Depending on the survey mode, difference images are +generated for the nightly stack images (using a deep stack template) +or for individual warp images. In the later case, the warp images may +be differenced against another warp from the same night or against a reference stack from the appropriate part of the sky. +\subsection{Re-processing Analysis Stages} + Pan-STARRS has performed several large-scale reprocessings of both the -Medium Deep and 3pi Survey data for internal consumption. For the 3pi -Survey data, we identify these large-scale reprocessings as PV1, PV2, -and PV3, with PV3 the analysis used for the first public data release, -DR1. We also refer to the nightly science analysis of the data as -PV0. For these reprocessing stages, the standard steps of chip -through warp, plus stack and diff are performed, starting from raw -data, usually using a single homogenous version of the data analysis -procedures. PV2 was a special case in which we started from the -camera level products of PV1 to speed up the turn-around to the -community. In addition to the analysis stages listed above which are -shared with the nightly processing, these large-scale reprocessing -analyses include additional processing. A more detailed photometric -analysis is performed on the stacks, including morphological analysis -appropriate to galaxies. The results of the stack photometry analysis -are used to drive a forced-photometry analysis of the warp images. -The data products from the camera, stack photometry, and forced-warp -photometry analysis stages are ingested into the internal calibration -database (DVO, the Desktop Virtual Observatory) and used for -photometric and astrometric calibrations. +Medium Deep and $3\pi$ Survey data for internal consumption. For the +$3\pi$ Survey data, we identify these large-scale reprocessings as +PV1, PV2, and PV3, with PV3 the analysis used for the first public +data release, DR1. We also refer to the nightly science analysis of +the data as PV0. For these reprocessing stages, the standard steps of +\ippstage{chip} through \ippstage{warp}, plus \ippstage{stack} and +\ippstage{diff} are performed, starting from raw data, usually using a +single homogenous version of the data analysis procedures. PV2 was a +special case in which we started from the camera level products of PV1 +to speed up the turn-around to the community. In addition to the +analysis stages listed above which are shared with the nightly +processing, these large-scale reprocessing analyses include additional +processing. A more detailed photometric analysis is performed on the +stacks, including morphological analysis appropriate to galaxies. The +results of the stack photometry analysis are used to drive a +forced-photometry analysis of the warp images. The data products from +the camera, stack photometry, and forced-warp photometry analysis +stages are ingested into the internal calibration database (DVO, the +Desktop Virtual Observatory) and used for photometric and astrometric +calibrations (see Section~\ref{sec:DVO}). \subsection{Data Access and Distribution} @@ -371,7 +361,26 @@ {\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} & {\bf Notes} \\ \hline - \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add_id} & \\ + \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & & Lists locations to check for new exposures.\\ + & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} & Exposures available at the telescope.\\ + & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile} & & Exposures that are being downloaded.\\ + & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp_id} & Exposures that have been saved to IPP cluster.\\ + + \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp_id} & \\ + \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} & \\ \ippstage{camera} & \ippdbtable{camRun} & \ippdbtable{camProcessedExp} & \ippdbcolumn{cam_id} & \\ - \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} & \\ + \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} & \\ + \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp_id} & \\ + & & \ippdbtable{warpSkyCellMap} & & Mapping of input chips to projection skycells.\\ + & & \ippdbtable{warpSkyfile} & & \\ + \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack_id} & \\ + & & \ippdbtable{stackSumSkyfile} & & \\ + \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky_id} & \\ + & & \ippdbtable{staticskyResult} & & \\ + \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal_id} & \\ + \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff_id} & \\ + & & \ippdbtable{fullForceResult} & & \\ + & & \ippdbtable{fullForceSummary} & & Properties about average parameters from all results.\\ + \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff_id} & \\ + & & \ippdbtable{diffInputSkyfile} & & \\ \ippstage{detrend} & \ippdbtable{detRun} & \ippdbtable{detRunSummary} & \ippdbcolumn{det_id} & \\ & & \ippdbtable{detInputExp} & & \\ @@ -381,31 +390,12 @@ & \ippdbtable{detResidExp} & \ippdbtable{detResidImfile} & & \\ & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & & \\ - \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff_id} & \\ - & & \ippdbtable{diffInputSkyfile} & & \\ + \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add_id} & \\ \ippstage{distribution} & \ippdbtable{distRun} & \ippdbtable{distComponent} & \ippdbcolumn{dist_id} & \\ & & \ippdbtable{distTarget} & & \\ - \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} & \\ - \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff_id} & \\ - & & \ippdbtable{fullForceResult} & & \\ - & & \ippdbtable{fullForceSummary} & & Properties about average parameters from all results.\\ + \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub_id} & \\ + & & \ippdbtable{publishClient} & & \\ \ippstage{lap} & \ippdbtable{lapSequence} & \ippdbtable{lapRun} & \ippdbcolumn{seq_id} & Sequence of full reprocessing\\ & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap_id} & \\ - \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub_id} & \\ - & & \ippdbtable{publishClient} & & \\ - \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & & Lists locations to check for new exposures.\\ - & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} & Exposures available at the telescope.\\ - & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile} & & Exposures that are being downloaded.\\ - & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp_id} & Exposures that have been saved to IPP cluster.\\ - - \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp_id} & \\ \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote_id} & \\ - \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal_id} & \\ - \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack_id} & \\ - & & \ippdbtable{stackSumSkyfile} & & \\ - \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky_id} & \\ - & & \ippdbtable{staticskyResult} & & \\ - \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp_id} & \\ - & & \ippdbtable{warpSkyCellMap} & & Mapping of input chips to projection skycells.\\ - & & \ippdbtable{warpSkyfile} & & \\ \hline \end{tabular} @@ -424,10 +414,12 @@ successive processing stages to begin their own tasks. -The processing database is colloquially referred to as the `gpc1' +The processing database is colloquially referred to as the ``gpc1'' database, since a single instance of the database is used to track the processing of images and data products related to the PS1 GPC1 camera. This same database engine also has instances (same schema, different data) for other cameras processed by the IPP, e.g., GPC2, the test -cameras TC1, TC3, and the Imaging Sky Probe (ISP). +cameras TC1, TC3, and the Imaging Sky Probe (ISP). In general, +processing information for different cameras is separate in different +processing database; merging of output products takes place in DVO. Within the processing database, the various processing stages are @@ -435,24 +427,25 @@ primary table which defines the conceptual list of processing items either to be done, in progress, or completed. An associated secondary -table (or set of tables) lists the details of elements which have been -processed. Table \ref{tab: database schema} contains an outline of -the database schema, showing the relations between tables organized by -processing stage. As an example, one critical stage is the -\ippstage{chip} processing stage (see \S\ref{sec:chip}) in which the -individual chips from an exposure are detrended and sources are -detected. Within the gpc1 database, the primary table is called -\ippdbtable{chipRun} in which each exposure has a single entry. -Associated with this table is the \ippdbtable{chipProcessedImfile} -table, which contains one row for each of the chips -associated with the exposure (up to 60 for gpc1). The primary tables, such as -\ippdbtable{chipRun}, are populated once the system has decided that a -specific item (e.g., an exposure) should be processed at that stage. -Initially, the entry is given a state of ``run'', denoting that the -exposure is ready to be processed. The low-level table entries, such -as the \ippdbtable{chipProcessedImfile} entries, are only populated -once the element (e.g., the chip) has been processed by the analysis -system. Once all elements for a given stage, e.g., chips in this -case, are completed, then the status of the top-level table entry -(\ippdbtable{chipRun}) are switched from ``run'' to ``full''. +table (or set of tables) lists the details of component elements which +have been processed for each top-level item. Table \ref{tab: database + schema} contains an outline of the database schema, showing the +relations between tables organized by processing stage. As an +example, one critical stage is the \ippstage{chip} processing stage +(see \S\ref{sec:chip}) in which the individual chips from an exposure +are detrended and sources are detected. Within the gpc1 database, the +primary table is called \ippdbtable{chipRun} in which each exposure +has a single entry. Associated with this table is the +\ippdbtable{chipProcessedImfile} table, which contains one row for +each of the chips associated with the exposure (up to 60 for gpc1). +The primary tables, such as \ippdbtable{chipRun}, are populated once +the system has decided that a specific item (e.g., an exposure) should +be processed at that stage. Initially, the entry is given a state of +``run'', denoting that the exposure is ready to be processed. The +low-level table entries, such as the \ippdbtable{chipProcessedImfile} +entries, are only populated once the element (e.g., the chip) has been +processed by the analysis system. Once all elements for a given +stage, e.g., chips in this case, are completed, then the status of the +top-level table entry (\ippdbtable{chipRun}) are switched from ``run'' +to ``full''. If the analysis of an element (e.g., the individual OTA chip) @@ -467,5 +460,5 @@ other hand, if the analysis failed because of a problem with the input data, this is noted by setting a non-zero value in a different table -field, \ippdbcolumn{quality}. For example, if the chip analysis +field, \ippdbcolumn{quality}. For example, if the \ippstage{chip} analysis failed to discover any stars because the image was completely saturated, the analysis can complete successfully (\ippdbcolumn{fault} @@ -483,9 +476,9 @@ of the \ippdbcolumn{fault}s which occur are ephemeral due to current conditions of the processing cluster, the processing stages are set up -to occasionally clear and re-try the faulted entries. Some faults +to occasionally clear and re-try the faulted entries. Some \ippdbcolumn{fault}s represent software bugs and in the early stages of processing were accumulated until the corresponding software issue could be addressed; since the start of the PS1 Science Consortium Surveys, these types of -faults have largely been eliminated. Thus, automatic processing is +\ippdbcolumn{fault}s have largely been eliminated. Thus, automatic processing is able to keep the data flowing even in the face of occasional network glitches or hardware crashes. @@ -496,8 +489,10 @@ As exposures are taken by the PS1 telescope \& GPC1 camera system, the data from the 60 OTA devices are read out by the camera software -wsystem and written to disk on a collection of computers at the summit +system and written to disk on a collection of computers at the summit in the PS1 facility called ``pixel servers.'' After the images are written to disk, a summary listing of the information about the -exposure and the chip images are added to the summit datastore. +exposure and the chip images are added to the summit datastore (an +internal http-based data sharing tool, see +Section~\ref{sec:datastore}). During night-time operations, while the summit datastore is being @@ -531,12 +526,12 @@ Once the chips for an exposure have all been downloaded, the exposure -is ready to be registered. In this context, `registration' refers to +is ready to be registered. In this context, ``registration'' refers to the process of adding them to the database listing of known, raw -exposures (not to be confused with `registration' in the sense of -pixel re-alignment). The result of the registration analysis is an +exposures (not to be confused with ``registration'' in the sense of +pixel re-alignment). The result of the \ippstage{registration} analysis is an entry for each exposure in the \ippdbtable{rawExp} table, and one for each chip in the \ippdbtable{rawImfile} table. These tables are critical for downstream processing to identify what exposures are -available for processing in any other stage. At the registration +available for processing in any other stage. At the \ippstage{registration} stage, a large amount of descriptive metadata for each chip is added to the \ippdbtable{rawImfile} table, the majority of which is @@ -552,8 +547,8 @@ Unlike much of the rest of the IPP stage, the raw exposures may only -have a single entry in the registration tables of the processing +have a single entry in the \ippstage{registration} tables of the processing database tables (\ippdbtable{rawExp} and \ippdbtable{rawImfile}). -For GPC1, the image registration stage is also the stage at which the +For GPC1, the \ippstage{registration} stage is also the stage at which the \ippprog{burntool} analysis is run. This analysis is more completely described in \citet{waters2017}. In brief, the \ippprog{burntool} @@ -564,16 +559,16 @@ observation date and time listed in the headers, with the results stored in an text table. As a result of the sequential nature of this -analysis, the registration of exposures is blocked until the +analysis, the \ippstage{registration} of exposures is blocked until the \ippprog{burntool} has been run on the previous exposures. -Once the registration process has finished, new science exposures that -have an \ippdbcolumn{obs_mode} value that indicates they are part of -a particular science survey are automatically launched into the -science analysis by defining entries for the \ippstage{chip} -processing stage, as described above. This analysis can be relaunched -multiple times, such as for the large scale PV3 reprocessing. -However, this automatic process ensures the shortest time between -observation and analysis, which is particularly important in the -search for transient sources. +Once the \ippstage{registration} process has finished, new science +exposures that have an \ippdbcolumn{obs_mode} value that indicates +they are part of a particular science survey are automatically +launched into the science analysis by defining entries for the +\ippstage{chip} processing stage, as described above. The science +analysis of a given exposure can be relaunched multiple times, such as +for the large scale PV3 reprocessing. The automatically-launched +analysis process ensures the shortest time between observation and +analysis, particularly important in the search for transient sources. \subsection{Chip Processing} @@ -619,5 +614,5 @@ %% attempts to target the processing for each OTA to the machine on which %% the data for that detector is stored. The output products are then -%% primarily saved back to the same machine. This `targetted' processing +%% primarily saved back to the same machine. This ``targetted'' processing %% was an early design choice to minimize the system wide network load %% during processing. In practice, as computer disks filled up at @@ -647,26 +642,26 @@ The results of the image processing are then written to disk, -including the science, mask, and variance images, the background model -subtracted, the PSF model used in the photometry process, and a FITS -catalog of detected sources. Additional binned images of the full OTA -are also saved, providing $16\times{}16$ and $256\times{}256$ pixel -binning scales for quick visualization. The processing log and a -selection of summary metadata describing the processing results are -also written to disk. This metadata is used to populate a row in the -\ippdbtable{chipProcessedImfile} table (linked to the -\ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip_id} value) -to indicate that the processing of this OTA is complete. +including the science, mask, and variance images, the binned +background model subtracted, the PSF model used in the photometry +process, and a FITS catalog of detected sources. Additional binned +images of the full OTA are also saved, using $16\times{}16$ and +$256\times{}256$ pixel binning scales for quick visualization. The +processing log and a selection of summary metadata describing the +processing results are also written to disk. This metadata is used to +populate a row in the \ippdbtable{chipProcessedImfile} table to +indicate that the processing of this OTA is complete. As each OTA is processed independently of the others across a number -of computers, the \ippprog{pantasks} managing the jobs periodically -runs an \ippmisc{advance} task that checks that the number of rows in -\ippdbtable{chipProcessedImfile} with \ippdbcolumn{fault} equal to -zero matches the associated number of rows in \ippdbtable{rawImfile}. -If this condition is met, than all processing for that exposure is -finished, and the \ippdbcolumn{state} field is set to ``full''. If -the \ippdbtable{chipRun}.\ippdbcolumn{end_stage} field is set to +of computers, the \ippprog{pantasks} server managing the jobs +periodically runs an \ippmisc{advance} task that checks that the +number of rows in \ippdbtable{chipProcessedImfile} with +\ippdbcolumn{fault} equal to zero matches the associated number of +rows in \ippdbtable{rawImfile}. If this condition is met, than all +processing for that exposure is finished, and the \ippdbcolumn{state} +field is set to ``full''. If the +\ippdbtable{chipRun}.\ippdbcolumn{end_stage} field is set to \ippstage{chip}, then no further action is taken. However, this field is usually set to a subsequent stage (most often \ippstage{warp}), -then an entry for this exposure is added to the \ippdbtable{camRun} +in which case an entry for this exposure is added to the \ippdbtable{camRun} table, and processing continues. @@ -710,6 +705,8 @@ to help guarantee a solution in the case of a modest pointing error. The guess astrometry is used to match the reference catalog to the -observed stellar positions in the focal plane coordinate system. Once -an acceptable match is found, the astrometric calibration of the +observed stellar positions in the focal plane coordinate system +\citep[see][]{magnier2017.calibration}). + +Once an acceptable match is found, the astrometric calibration of the individual chips is performed, including a fit to a single model for the distortion introduced by the camera optics. After the astrometic @@ -720,7 +717,7 @@ used to generate synthetic w-band photometry for areas where no PS1-based calibrated w-band photometry is available. For more -details, see \cite{magnier2017.calibration}. The result of these calibrations is -stored as a single multi-extension FITS table containing the results -from each OTA as a separate extension. +details, see \cite{magnier2017.calibration}. The result of these +calibrations is stored as a single multi-extension FITS table +containing the results from each OTA as a separate extension. In addition to the astrometric and photometric calibrations, the @@ -740,25 +737,25 @@ processed all at once, this update also updates the associated \ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam_id}. As -with the \ippstage{chip} stage, the +with the \ippstage{chip} stage, if the \ippdbtable{camRun}.\ippdbcolumn{end_stage} is for a subsequent stage, an appropriate entry is added to the \ippdbtable{fakeRun} -table. - -%% \subsection{Fake Analysis} -%% \label{sec:fake} -%% -%% The \ippstage{fake} stage was originally designed to do false source -%% injection and recovery, in order to determine the detection efficiency -%% of sources on the exposure. However, early in the design of the IPP, -%% this task was moved to the rest of the photometry analysis done at the -%% \ippstage{chip} stage. Removing the stage would require significant -%% changes to the database schema. As a result, this conveniently named -%% stage generally does no actual data processing, and consists mainly of -%% database operations to move the exposure on to the \ippstage{warp} -%% stage. The operations mimic the \ippstage{chip} stage, with -%% individual jobs run for each OTA that update rows in the -%% \ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that -%% updates the \ippdbtable{fakeRun} table and promotes the exposure to -%% the next stage by adding a row to the \ippdbtable{warpRun} table. +table. + +\subsection{Fake Analysis} +\label{sec:fake} + +The \ippstage{fake} stage was originally designed to do false source +injection and recovery, in order to determine the detection efficiency +of sources on the exposure. However, early in the design of the IPP, +this task was moved to the rest of the photometry analysis done at the +\ippstage{chip} stage. Removing the stage would require significant +changes to the database schema. As a result, this conveniently named +stage generally does no actual data processing, and consists mainly of +database operations to move the exposure on to the \ippstage{warp} +stage. The operations mimic the \ippstage{chip} stage, with +individual jobs run for each OTA that update rows in the +\ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that +updates the \ippdbtable{fakeRun} table and promotes the exposure to +the next stage by adding a row to the \ippdbtable{warpRun} table. \subsection{Image Warping} @@ -776,5 +773,5 @@ described by a single tangent plane projection, or for larger regions which have multiple projection centers. For the $3\pi$ survey, the -\ippmisc{RINGS.V3} tessellation was used that used projection centers +\ippmisc{RINGS.V3} tessellation was used that arrange projection centers spaced every four degrees in both RA and DEC, with $0\farcs{}25$ pixels. These projections are further broken down into ``skycells'' @@ -822,6 +819,6 @@ \label{sec:stack} -The skycell images generated by the \ippstage{warp} process are added -together to make deeper, higher signal-to-noise images in the +The skycell images generated by the \ippstage{warp} process can be +added together to make deeper, higher signal-to-noise images in the \ippstage{stack} stage. These stacked images also fill in coverage gaps between different exposures, resulting in an image of the sky @@ -831,5 +828,5 @@ input images. During nightly science processing, the 8 exposures per filter for each Medium Deep field are combined into a set of stacks -for that field. These so-called `nightly stacks' are used by the +for that field. These so-called ``nightly stacks'' are used by the transient survey projects to detect faint supernovae, among other transient events. For the PV3 $3\pi$ analysis, all images in each @@ -840,8 +837,8 @@ For the PV3 processing of the Medium Deep fields, stacks have been generated for the nightly groups and for the full depth using all -exposures, producing ``deep stacks''. In addition, a `best seeing' +exposures, producing ``deep stacks''. In addition, a ``best seeing'' set of stacks have been produced \note{using image quality cuts to be described: need input from MEH}. We have also generated -out-of-season stacks for the Medium Deep fields, in which all image +out-of-season stacks for the Medium Deep fields, in which all images not from a particular observing season for a field are combined into a stack. These later stacks are useful as deep templates when studying @@ -850,16 +847,14 @@ season. -When a given set of \ippstage{stack} stage are defined, exposures with -existing \ippstage{warp} entries that match the filter, position, and -other criteria such as seeing are grouped by their skycell. An entry +When a given set of \ippstage{stack} stage processing is defined, +exposures with existing \ippstage{warp} entries that match the filter, +position, and other criteria such as seeing are identified. An entry is then added for each skycell in the \ippdbtable{stackRun} table, with the \ippdbcolumn{warp_id} entries for the exposures added to the \ippdbtable{stackInputSkyfile} table, linked to the -\ippdbtable{stackRun} entry by the \ippdbcolumn{stack_id} field. -This defines the mapping for which exposures contribute to the -\ippstage{stack}. This breaks exposures into single skycells, but as -adjacent \ippstage{stack} skycells may contain inputs from different -exposures, there is no simple way to group the processing at the -\ippstage{stack} stage into exposures. +\ippdbtable{stackRun} entry by the \ippdbcolumn{stack_id} field. This +defines the mapping for which exposures contribute to the +\ippstage{stack}. The \ippstage{stack} stage processing is performed +at the skycell level. The \ippstage{stack} jobs pass the information about the input images @@ -867,5 +862,5 @@ image combinations. See~\cite{waters2017} for details on the stack combination algorithm. In addition to the standard image, mask, and -variance produced at other stage, additional images are constructed +variance produced at other stages, additional images are constructed with information about the contributions to each pixel. A number image contains the number of input exposures used for each pixel, @@ -887,13 +882,14 @@ deferred to the \ippstage{staticsky} stage. This separation is maintained because the photometry analysis of the \ippstage{stack} -images is performed on all 5 filters simultaneously. By deferring -this analysis, the processing system may also decouple the generation -of the pixels from the source detection. This makes the sequencing of -analysis somewhat easier and less subject to blocks due to a failure -in the stacking analysis. Similar to the \ippstage{stack} stage, an -entry is created in the \ippdbtable{staticskyRun} table, linked to a -series of rows in the \ippdbtable{staticskyInput} table by a common -\ippdbcolumn{sky_id}, each of which also contains the appropriate -\ippdbcolumn{stack_id} entries for the skycell under consideration. +images, including convolved galaxy model fitting, is performed on all +5 filters simultaneously. By deferring this analysis, the processing +system may also decouple the generation of the pixels from the source +detection. This makes the sequencing of analysis somewhat easier and +less subject to blocks due to a failure in the stacking analysis. +Similar to the \ippstage{stack} stage, an entry is created in the +\ippdbtable{staticskyRun} table, linked to a series of rows in the +\ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky_id}, +each of which also contains the appropriate \ippdbcolumn{stack_id} +entries for the skycell under consideration. The input images are passed to the \ippprog{psphotStack} program, @@ -927,16 +923,24 @@ The stack photometry output catalogs are re-calibrated for both photometry and astrometry in a process very similar to the -\ippstage{camera} calibration stage. In the case of this -\ippstage{skycal} stage, each skycell is processed independently. -Because of this independence, when queued for processing, the entries -in the \ippdbtable{skycalRun} table contain the \IPPdbcolumn{sky_id} -and \ippdbcolumn{stack_id} entries of the parent data directly. As -in the \ippstage{camera} stage, the \ippprog{psastro} program reads in -the stack photometry catalog, and produces a calibrated output, with -format matching the input. A different processing recipe is supplied -to \ippprog{psastro}, which controls for the different data. The same -reference catalog is used for the \ippstage{camera} and -\ippstage{stack} calibration stages. Upon completion, the analysis -statistics are written to the \ippdbtable{skycalResult} table. +\ippstage{camera} calibration stage. Although the individual warps +which go into the stack are calibrated based on the \ippstage{camera} +stage analysis, there was some concern that these calibrations might +not be sufficiently well-defined for some of the input warps, biasing +the photometry of the stack. By re-calibrating the stacks, we can be +sure that the stack photometry as measured is tied to the photometric +reference system. + +In the case of this \ippstage{skycal} stage, each skycell is processed +independently. Because of this independence, when queued for +processing, the entries in the \ippdbtable{skycalRun} table contain +the \ippdbcolumn{sky_id} and \ippdbcolumn{stack_id} entries of the +parent data directly. As in the \ippstage{camera} stage, the +\ippprog{psastro} program reads in the stack photometry catalog, and +produces a calibrated output, with format matching the input. A +different processing recipe is supplied to \ippprog{psastro}, which +controls for the different data. The same reference catalog is used +for the \ippstage{camera} and \ippstage{stack} calibration stages. +Upon completion, the analysis statistics are written to the +\ippdbtable{skycalResult} table. \subsection{Forced Warp Photometry} @@ -995,12 +999,14 @@ individual warp images used to generate the stack. This \ippstage{fullforce} analysis is performed on all warps for a single -skycell and filter as a single unit, as this matches the arrangement -of the input source catalog from the \ippstage{skycal} stage. When -processing is queued for this stage, an entry is added to the -\ippdbtable{fullForceRun} primary database table linking to the -specific \ippdbcolumn{skycal_id} entry that will be used as the -catalog for the photometry. The \ippdbcolumn{warp_id} values for the -input \ippstage{warp} stage images that contributed to the -\ippstage{stack} associated with that \ippdbcolumn{skycal_id} are +skycell and filter as a single unit within the processing database, +while individual warps are processed individually in parallel as +separate processing jobs. + +When processing is queued for this stage, an entry is added to the +\ippdbtable{fullForceRun} primary database table with a reference to +the corresponding stack and \ippdbcolumn{skycal_id} entry that is the +input source of detections to be measured. The \ippdbcolumn{warp_id} +values for the input \ippstage{warp} stage images that contributed to +the \ippstage{stack} associated with that \ippdbcolumn{skycal_id} are then added to the \ippdbtable{fullForceInput} table, linked to the primary table by the \ippdbcolumn{ff_id} identifier. The individual @@ -1008,4 +1014,22 @@ stage image products along with the \ippstage{skycal} catalog to the \ippprog{psphotFullForce} program. + +%% In this program, the positions of sources are loaded from the input +%% catalog. PSF stars are pre-identified from the stack image and a PSF +%% model generated for each \ippstage{warp} image based on those stars, +%% using the same stars for all warps to the extent possible (PSF stars +%% which are excessively masked on a particular image are not used to +%% model the PSF). The PSF model is fitted to all of the known source +%% positions in the warp images. Aperture magnitudes, Kron magnitudes, +%% and moments are also measured at this stage for each warp. Note that +%% the flux measurement for a faint, but significant, source from the +%% stack image may be at a low significance (less than the $5\sigma$ +%% criterion used when the photometry is not run in this forced mode) in +%% any individual warp image; the flux may even be negative for specific +%% warps. When combined together, these low-significance measurements +%% will result in a signficant measurement as the signal-to-noise +%% increases by the square root of the number of measurements. The +%% individual warp measurements are combined together to generate +%% averages values within DVO. The convolved galaxy models are also re-measured on the @@ -1053,10 +1077,10 @@ images are matched. \note{discuss Alard-Lupton}. -In the \ippstage{diff} stage, the IPP generates diffferece images for +In the \ippstage{diff} stage, the IPP generates difference images for appropriately specified pairs of images. It is possible for the difference image to be generated from a pair of \ippstage{warp} stage images, from a \ippstage{warp} and a \ippstage{stack} of some variety, or from a pair of \ippstage{stack} stage images. During the PS1 -survey, pairs of exposures, call TTI pairs (see~\note{Survey +survey, pairs of exposures, called TTI pairs (see~\note{Survey Strategy in Chambers et al}), were obtained for each pointing within a $\approx$ 1 hour period in the same filter, and to the extent possible with the @@ -1074,5 +1098,5 @@ \ippdbtable{diffRun} table, and the appropriate input images are added to the \ippdbtable{diffInputSkyfile} table, with one entry for each -skycell that are covered by the images. For a \ippstage{diff} +skycell that is covered by the images. For a \ippstage{diff} generated from two \ippstage{warp} stage products, the input images have their \ippdbcolumn{warp_id} values recorded in the @@ -1095,7 +1119,9 @@ catalogs passed to the \ippprog{ppSub} program. This does the subtraction, as well as the photometry of any sources detected in the -\ippstage{diff} image. The algorithm used for PSF matching is -described in \citet{waters2017}. Upon completion of these jobs, -statistics about the processing are written to an entry in the +\ippstage{diff} image. Sources may be detected as a positive source +(flux in the minuend is higher than the subtrahend) or as a negative +source (flux in the subtrahend is higher). The algorithm used for PSF +matching is described in \citet{waters2017}. Upon completion of these +jobs, statistics about the processing are written to an entry in the \ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the completion of all of the components listed in @@ -1111,5 +1137,6 @@ \begin{table}[hb] \begin{center} -\caption{DVO Database Tables\label{tab:DVO_schema}} +\caption{DVO Database Tables\label{tab:DVO_schema} \note{fix order, + drop invalid tables}} \begin{tabular}{ll} \hline @@ -1155,5 +1182,5 @@ DVO tracks three main classes of information: 1) average properties of astronomical objects; 2) measurements of those objects (from which the -average properties are derived); 3) properties of image which provided +average properties are derived); 3) properties of the images which provided some or all of the measuements. Figure~\ref{fig:DVO_schema} illustrates the schematic relationship between these types of @@ -1182,14 +1209,4 @@ measurements; those which store information about the images; those which store supporting information (metadata). - -\subsubsubsection{Photcodes} - -% photcodes -DVO has a special metadata table called \ippdbcolumn{photcode} which -identifies the photometry filter systems. Entries in this table are -used to identify the source of measurements and images. Each row in -the \ippdbcolumn{photcode} table includes a \ippdbcolumn{photcode} -name, a unique numerical ID, and information about that photometry -system. DVO includes two major classes of database tables: those containing @@ -1208,4 +1225,25 @@ levels each containing a finer mesh of regions covering the sky. +\subsubsubsection{Photcodes} + +% photcodes +DVO has a special metadata table called \ippdbtable{photcode} which +identifies the photometry filter systems. Entries in this table are +used to identify the source of measurements and images. Each row in +the \ippdbtable{photcode} table includes a \ippdbtable{photcode} +name, a unique numerical ID, and information about that photometry +system. + +There are 3 classes of photcodes defined within the DVO system. One +class of photcodes define the filter systems for the average +photometry measurements; these are called \ippmisc{SEC} photcodes. A +second class of photcode is associated with measurements from a +specific camera for which image metadata is available are called +\ippmisc{DEP} photcodes. There are also those measurements which come +from external data sources for which DVO does not have any information +to determine a calibration (e.g., instrumental magnitudes and detector +coordinates). These are measurements are reference values and are +assigned \ippmisc{REF} photcodes. + The names for \ippmisc{SEC} photcodes are the names of filter systems, such as $g,r,i$ or $J,H,K$. For \ippmisc{DEP} and \ippmisc{REF} @@ -1229,5 +1267,5 @@ properties derived from multiple measurements, and for which the measurement-to-image relationship is not provided. Ingests methods -have been defined for example for 2MASS, WISE, Gaia, USNO-B. In each +have been defined, for example, for 2MASS, WISE, Gaia, USNO-B. In each of these cases, the astrometric and photometric measurements are stored in the \ippdbtable{Measure} table, with the data source @@ -1258,6 +1296,6 @@ discussed below) and the astrometrically calibrated position. Astrometric offsets for several systematic corrections discussed below -are also defined for each measurement. Photometry from chip, warp, -and stack are all placed in the same table with photcodes +are also defined for each measurement. Photometry from \ippstage{chip}, \ippstage{warp}, +and \ippstage{stack} are all placed in the same table with photcodes distinguishing the source \note{show example of stack and warp photcodes}. Since stacks and forced warp fluxes may have @@ -1269,5 +1307,5 @@ For the warp images, we also measure the weak lensing KSB parameters related to the shear and smear tensors \citep{1995ApJ...449..460K}. -These measurements are stored in the \ippdbcolumn{Lensing} table, +These measurements are stored in the \ippdbtable{Lensing} table, along with the radial aperture fluxes for radii numbers 5, 6, \& 7 (respectively 3.0, 4.63, and 7.43 arcsec). This table contains one @@ -1281,4 +1319,6 @@ sorted \ippdbtable{Lensing} table entries. \note{discuss failure of the Lensing to Measure indexing} + +\note{Average used above but defined below} \subsubsubsection{Object Tables} @@ -1359,5 +1399,5 @@ these photometric distance modulus measurements are not extremely precise (see below), they provide a constraint on the distance is used -in our analysis of the astrometry \citep[][see]{magnier2017.calibration}. +in our analysis of the astrometry \citep[see][]{magnier2017.calibration}. In the \ippdbtable{Measure} table, there are three fields which @@ -1416,9 +1456,10 @@ determined by the photometry calibration analysis and the astrometric flat-field corrections determined by the astrometry calibration -analysis \citep[][see]{magnier2017.calibration}. +analysis \citep[see][]{magnier2017.calibration}. +\note{use names and match DVO schema table} \subsubsection{Sky Partition} -DVO includes two major classes of database tables: those containing +\note{re-word this sentence} DVO includes two major classes of database tables: those containing information about astronomical objects in the sky and those containing other supporting information. The object-related tables are @@ -1438,6 +1479,6 @@ on the one used by the Hubble Space Telescope Guide Star Catalog files. \note{add figure} Level 0 is a single region covering the full -sky. Level 1 divides the sky in Declination into bands -7.5\degree\ high. Level 2 subdivides these Declination bands in the +sky. Level 1 divides the sky in declination into bands +7.5\degree\ high. Level 2 subdivides these declination bands in the RA direction, with spacing related to the stellar density. Level 3 divides these RA chunks into 4 - 8 smaller partitions. This level @@ -1459,5 +1500,5 @@ astronomical objects in the database files, with an associated maximum of \approx 30 million measurements in these files. With the compression -scheme described above, the largest database files are \approx +scheme described below, the largest database files are \approx 3GB, which can be loaded into memory in 30 seconds on the processing machines that contain partition data. @@ -1499,5 +1540,5 @@ tables are compressed using the (to date) experimental FITS binary table compression strategy outlined by \note{REF}. Table compression -is in general an option in DVO; for the PV3 database, the large data +is an option in DVO; for the PV3 database, the large data volume (70TB compressed) drove the decision to compress the tables. @@ -1505,5 +1546,5 @@ The FITS binary table compression scheme uses a strategy similar to that used for FITS image compression (\note{REF}). The binary tabular -data is compressed and stored in the `HEAP' section of the FITS table +data is compressed and stored in the ``HEAP'' section of the FITS table extension, with pointers to the compressed data stored in the regular data section. Each column in the FITS table is compressed as one (or @@ -1511,5 +1552,5 @@ column format (e.g., TFORM1) are replaced with keywords which describe the location and size of the compressed data in the HEAP section; the -information about the uncompressed data is moved to a keyword with `Z' +information about the uncompressed data is moved to a keyword with ``Z'' prepended (e.g., ZFORM1) and an additional field is added to define the compression algorithm (e.g., ZCTYP1). The column names (e.g., @@ -1533,5 +1574,5 @@ in the tables. In practice, we have chosen a default in which floating point numbers use \code{GZIP_2}, character strings use -\code{GZIP_1}, integers use \code{RICE}. +\code{GZIP_1}, and integers use \code{RICE}. \subsubsection{Addstar : DVO Ingest} @@ -1540,18 +1581,18 @@ Upon completion of the processing of each stage, the results of the photometry analysis are stored in a large number of individual catalog -files as described in~\ref{XXX}. The data from these files are loaded -into a DVO database to define the astronomical objects and to allow -for calibration analysis. The program which loads the data into the -DVO database is called \ippprog{addstar}, and is associated with the -the \ippstage{addstar} processing stage. The measurement catalogs -generated by the \ippstage{camera}, \ippstage{staticsky}, -\ippstage{skycal}, \ippstage{fullforce}, and \ippstage{diff} stages -are processed loaded into DVOs in this fashion, although not every -measurement in each catalog are included in the master DVO that is -constructed. For a particular re-processing version, a single master -DVO is constructed for the positive image stages (\ippstage{camera}, -\ippstage{staticsky}, \ippstage{skycal}, \ippstage{fullforce}) and a -separate one is constructed for the difference image analysis stage -results. +files as described in \cite{magnier2017.analysis}. The data from +these files are loaded into a DVO database to define the astronomical +objects and to allow for calibration analysis. The program which +loads the data into the DVO database is called \ippprog{addstar}, and +is associated with the the \ippstage{addstar} processing stage. The +measurement catalogs generated by the \ippstage{camera}, +\ippstage{staticsky}, \ippstage{skycal}, \ippstage{fullforce}, and +\ippstage{diff} stages are processed loaded into DVOs in this fashion, +although not every measurement in each catalog are included in the +master DVO that is constructed. For a particular re-processing +version, a single master DVO is constructed for the positive image +stages (\ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal}, +\ippstage{fullforce}) and a separate one is constructed for the +difference image analysis stage results. The construction of the master DVO is performed in a hierarchical @@ -1564,5 +1605,5 @@ databases together. In the merge, astronomical objects are joined together using essentially the same rules as those used to associated -detections into objects. One exception: the match radius may be +detections into objects with one exception: the match radius may be chosen to be a different size depending on the data source. For example, when WISE data is merged with PS1 data, as discussed below, a @@ -1612,6 +1653,6 @@ a function of position in the camera (essentially an astrometric flat-field correction), as a function of the brightness of the star -(the so-called Koppenh\"offer effect, see~\ref{magnier2017.calibration}), and as -a function of airmass and color (Differential chromatic refraction). +(the so-called Koppenh\"offer effect, see~\citealt{magnier2017.calibration}), and as +a function of airmass and color (differential chromatic refraction). Once the systematic errors have been measured, they are applied back to the measurements in the database. Within the DVO @@ -1624,11 +1665,13 @@ astrometry is again performed this time using the corrected positions. +\note{have eddie suggest wording here?} + Photometric calibration consists of determination of zero points for each exposure along with corrections for systematic effects. In this case, we rely on efforts of our external collaborators for the initial zero point determination. The team at CfA downloaded the per-exposure -catalog files (`smf files') and determined the zero points of those +catalog files (``smf files'') and determined the zero points of those exposures which were believed to be obtained in photometric -conditions. This process, called `\"ubercal', is described in detail +conditions. This process, called ``\"ubercal'', is described in detail by \cite{2012ApJ...756..158S} for the first (PV1) version. In brief, photometric periods, with time-scales of at least \note{half of a night}, are @@ -1638,6 +1681,6 @@ parameters in this solution consist of a single zero point and airmass slope for each photometric period along with a collection of -flat-field offsets for several large time range (`flat-field -seasons'). For the PV3 \"ubercal analysis, the flat-field offsets +flat-field offsets for several large time range (``flat-field +seasons''). For the PV3 \"ubercal analysis, the flat-field offsets were determined on a $2\times2$ grid for each chip and 5 flat-field seasons were chosen (listed in Table~\ref{tab:flat-field-seasons}). @@ -1673,5 +1716,5 @@ Telescope Sciences Institute through their Mikulski Archive for Space Telescopes (MAST). The underying database at MAST is a copy of a -database generated at the Institute for Astronomy by the subsystem +database generated at the IfA by the subsystem called PSPS : the \note{define PSPS}. The construction of the PSPS version of the PS1 database starts once the PS1 photometry and @@ -1681,5 +1724,5 @@ The first stage of constructing the PSPS database consists of the -generation of small files called `batches' which contain a complete +generation of small files called ``batches'' which contain a complete set of measurements for a small chunk of the database tables. The program which is responsible for the construction of these batches is @@ -1690,5 +1733,5 @@ One type of batch consists of measurements from the individual exposures. These batches are generated based on the output catalog -files generated at the \ippstage{camera} stage (`smf files'). The +files generated at the \ippstage{camera} stage (``smf files''). The \ippprog{ipptopsps} program loads the complete set of measurements and metadata from the smf catalog file, then queries the DVO database for @@ -1757,9 +1800,9 @@ might be run and to regularly generate new commands based on that concept. The ``tasks'' are defined using the opihi scripting language -(also shared by DVO and other user-interative programs within the +(also shared by DVO and other user-interactive programs within the IPP). -Pantasks repeatedly checks each task in an attempt to generate a new -command: we say pantasks attempts to `execute' the task in each of +\ippprog{Pantasks} repeatedly checks each task in an attempt to generate a new +command: we say \ippprog{pantasks} attempts to ``execute'' the task in each of these attempts. Tasks may specify the time between execution attempts, with a 1 second default. @@ -1773,5 +1816,5 @@ opihi language) which is run each time the task is executed. The \code{task.exec} code may refer to variables or other data structures -defined by the opihi language within the pantasks environment. Within +defined by the opihi language within the \ippprog{pantasks} environment. Within a single \ippprog{pantasks} instance, all opihi variables and data structures have global context (\ie, all are visible to all tasks). @@ -1782,14 +1825,14 @@ Within the \ippprog{task.exec} macro, the command to be run must be -defined with the function `command'. Once the \ippprog{task.exec} -macro exits successfully, the defined command is the added to the list of jobs +defined with the function ``command''. Once the \ippprog{task.exec} +macro exits successfully, the defined command is then added to the list of jobs to be run within the UNIX environment. Jobs may be run in one of two ways: locally or via the parallel processing system. The task, or the -\ippprog{task.exec} macro, uses the `host' command to define how to -run the job. If the host is set to `local', then the job is run in -the background by pantasks itself (using the C \code{execvp} +\ippprog{task.exec} macro, uses the ``host'; command to define how to +run the job. If the host is set to ``local'', then the job is run in +the background by \ippprog{pantasks} itself (using the C \code{execvp} function). Otherwise, the job is sent to the parallel processing system to be run on another machine within the cluster. If the host -is set to the special value `anyhost', then the parallel processing +is set to the special value ``anyhost'', then the parallel processing system is allowed to choose the processing computer arbitrarily. Any other value is taken to be the DNS name of the computer on which this @@ -1798,10 +1841,10 @@ that the job only runs on the specifically named computer. Otherwise, the parallel processing system may choose to redirect the command to -another computer (based on whatever rules are defined for the parallel -processing system). +another computer using its own rules, e.g. to balance processing load +across the cluster. When the \ippprog{task.exec} macro is run, the code may choose (e.g., based on tests of some global variables) to exit the macro with an -error condition, e.g., with the `break' command. In this +error condition, e.g., with the ``break'' command. In this circumstance, no job is produced by the task. The task will be tried again the next time it is executed. This feature allows for the user @@ -1818,18 +1861,18 @@ online user guide?} -The option `npending' may be used to limit the number of jobs which +The option ``npending'' may be used to limit the number of jobs which are simultaneously executed for a specific task. For example, some classes of jobs should only be run one-at-a-time because they are not protected against collisions or they may overload a resource. The use -of `npending' allows these situations to be handled cleanly within -pantasks (avoiding cumbersome coding within with program or supporting +of ``npending'' allows these situations to be handled cleanly within +\ippprog{pantasks} (avoiding cumbersome coding within with program or supporting script). -The option `nmax' limits the total number of jobs which a task +The option ``nmax'' limits the total number of jobs which a task generates. This option may be useful in cases where \ippprog{pantasks} is used to perform a limited set of operations. \note{do we actually use this in IPP?} -The option `trange' allows the user to restrict the time period during +The option ``trange'' allows the user to restrict the time period during which the specific tasks is executed. This option is given with a start and an end time for the limiting time range. These times may be @@ -1846,10 +1889,10 @@ ranges may be specified \note{how are they evaluated?} -The option \code{nice} specifies the `nice' level at which the job is +The option \code{nice} specifies the ``nice'' level at which the job is run when it is executed. The parallel processing system must respect this concept. The option \code{active} can be used to turn on and off a task for -periods. Since a user command or a macro run by pantasks can +periods. Since a user command or a macro run by \ippprog{pantasks} can re-define task options, the \code{active} state may be changed independently of the task execute. This is useful for keeping tasks @@ -1857,5 +1900,5 @@ prevent them from running for some reason. -\subsubsection{pantasks passes jobs to pcontrol} +\subsubsection{pcontrol} Jobs which are generated by \ippprog{pantasks} may be run locally on @@ -1883,21 +1926,21 @@ Similarly, the hosts may also have one of several states: off, down, busy, idle, etc. A single host can accept a single job at a time. -Multiple hosts instances corresponding to the same machine may be +Multiple host instances corresponding to the same machine may be specified allowing a single computer to run more than one simultaneous job. -During operation, pcontrol accepts new jobs from pantasks and adds -them to the list of jobs to execute. It also accepts from pantasks +During operation, \ippprog{pcontrol} accepts new jobs from \ippprog{pantasks} and adds +them to the list of jobs to execute. It also accepts from \ippprog{pantasks} the names of computers on which it is allowed to run those jobs. -\subsubsection{pcontrol passes jobs to pclient} - -When pcontrol is provided with the name of a computer, it will attempt +\subsubsection{pclient} + +When \ippprog{pcontrol} is provided with the name of a computer, it will attempt to make an connection to that machine via ssh (or rsh?). When a connection is made, the remote shell is used to run a special interface program call \ippprog{pclient}. This program accepts -command lines from pcontrol and is responsible for executing the +command lines from \ippprog{pcontrol} and is responsible for executing the individual commands in the local shell environment. A single ssh -connection to a remote host keeps a single pclient shell running for a +connection to a remote host keeps a single \ippprog{pclient} shell running for a somewhat arbirarly long time, excuting many shell commands as needed. This architecture avoids wasting overhead making the ssh connection to @@ -1906,25 +1949,25 @@ architecture is allowed to be very light and short running if needed. -After pcontrol sends a job (commands) to a specific pclient, it checks +After \ippprog{pcontrol} sends a job (commands) to a specific \ippprog{pclient}, it checks back occasionally to see if the command has been run and executed. If -it has finished, then pcontrol will query for the exit status, the +it has finished, then \ippprog{pcontrol} will query for the exit status, the standard output and standard error streams from the command. (where -do these go, back to pantasks?), with the results associated with the -job statistics. At that point, the pclient on the remote machine is -ready to accept a new job from pcontrol. If any jobs are pending in -the list of jobs known to pcontrol, it will send those jobs to any +do these go, back to \ippprog{pantasks}?), with the results associated with the +job statistics. At that point, the \ippprog{pclient} on the remote machine is +ready to accept a new job from \ippprog{pcontrol}. If any jobs are pending in +the list of jobs known to \ippprog{pcontrol}, it will send those jobs to any machines which are idle. -While pcontrol interacts with the many remote machines, it -occasionally interacts with pantasks to report the results from the -jobs it has been monitoring. Pantasks occasionally requests a list of +While \ippprog{pcontrol} interacts with the many remote machines, it +occasionally interacts with \ippprog{pantasks} to report the results from the +jobs it has been monitoring. \ippprog{Pantasks} occasionally requests a list of the completed jobs. It then requests the status information for each completed job, including the standard error and standard output. As -pantasks receives this completion information, the jobs are removed -from the list managed by pcontrol. Thus pcontrol maintains at most a -modest list of jobs which are `in flight', leaving all interpretation -work to pantasks. - -At the pantasks level, the tasks define how pantasks should use the +\ippprog{pantasks} receives this completion information, the jobs are removed +from the list managed by \ippprog{pcontrol}. Thus \ippprog{pcontrol} maintains at most a +modest list of jobs which are ``in flight'' , leaving all interpretation +work to \ippprog{pantasks}. + +At the \ippprog{pantasks} level, the tasks define how \ippprog{pantasks} should use the exit status and output products from each job. For example, the stderr and stdout may be specified to go to a file (with static name @@ -1936,10 +1979,10 @@ started. This mode is useful for testing as all errors are reported back to the opihi shell. However, when the user exits the shell, the -pantasks instance exits, shutting down pcontrol and all remote client -connections. In standard operations, pantasks is run in a client +\ippprog{pantasks} instance exits, shutting down \ippprog{pcontrol} and all remote client +connections. In standard operations, \ippprog{pantasks} is run in a client server mode. The server runs continuously in the background and multiple users may connect via the \ippprog{pantasks_client} program. Users can the send commands to the server to load scripts, add -parallel hosts, check status, and start or stop the pantasks +parallel hosts, check status, and start or stop the \ippprog{pantasks} operations. @@ -1956,9 +1999,9 @@ end \end{verbatim} - \caption{\label{fig:task_example} Example of a simple static - task in the opihi-based scripting language used by pantasks. In - this example, pantasks would run a single instance of the command - ({\tt ls /tmp}) every 5 seconds, sending the stdout and stderr to - the listed files. } +\caption{\label{fig:task_example} Example of a simple static + task in the opihi-based scripting language used by ippprog{pantasks}. In + this example, ippprog{pantasks} would run a single instance of the command + ({\tt ls /tmp}) every 5 seconds, sending the stdout and stderr to + the listed files. } \end{center} \end{figure} @@ -1968,8 +2011,8 @@ \subsubsection{Pantasks scripts: ippTasks} -Pantasks provides an environment in which commands can be generated +\ippprog{Pantasks} provides an environment in which commands can be generated and extensive parallel processing managed. The details of how to implement the different stages of IPP processing are captured in a -collection of scripts written for pantasks in the \code{opihi} +collection of scripts written for \ippprog{pantasks} in the \code{opihi} language. In general, each stage is defined by an associated script collected together under the \ippmisc{ippTasks} collection. While @@ -2001,7 +2044,7 @@ row in the result set, each column in the row is stored as a separate line on the \ippmisc{page}, identified by the database column name. An -additional line, the \ippdbcolumn{pantasksState}, is added so pantasks +additional line, the \ippdbcolumn{pantasksState}, is added so \ippprog{pantasks} can manage the processing of the job which will be generated by this -page. When the page is first generate, the +page. When the page is first generated, the \ippdbcolumn{pantasksState} is set to \ippmisc{INIT}, indicating that this \ippmisc{page} is a new addition to the \ippmisc{book}. Once all @@ -2018,7 +2061,7 @@ construct the appropriate command-line (e.g., lines in the page may include input file names and output file names for the specific item -in the database). The resulting command becomes a job in the pantasks +in the database). The resulting command becomes a job in the \ippprog{pantasks} collection of jobs. Most IPP analysis stages specify that the jobs -are then sent to pcontrol for parallel process. Before task generates +are then sent to \ippprog{pcontrol} for parallel process. Before task generates the job, the \ippdbcolumn{pantasksState} is set to \ippmisc{RUN} so a future execution of the task will not attempt to re-run this specific job. @@ -2029,9 +2072,9 @@ this responsibility is left to the program which ran the analysis. IPP analysis steps normally consist of two main elements: a C-language -program to do the data analysis work and a supporting perl script +program to do the data analysis work and a supporting Perl script which performs the database update upon completion. Upon completion, -the pantasks \ippmisc{RUN} tasks is responsible for updating the +the \ippprog{pantasks} \ippmisc{RUN} tasks is responsible for updating the status within the book, but not within the processing database. This -split keeps the interactions at the pantasks level relatively light, +split keeps the interactions at the \ippprog{pantasks} level relatively light, leaving the overhead of the database interaction within the job running on one of the computing machines in the cluster. @@ -2042,6 +2085,6 @@ clear jobs which have failed with one of the ephemeral failure modes (see the discussion in Section~\ref{sec:processing.database}). This -step allows these failures to be cleared from the system, and schedule -those jobs again for a retry. +step allows these failures to be cleared from the system, allowing +those jobs to be scheduled again. Similarly, some stages have \ippmisc{advance} tasks that update the @@ -2066,8 +2109,8 @@ discussed above, the query to the processing database for new items is restricted to a set of user-defined labels. A given instance of -pantasks will be supplied a set of labels which are then applied to -all tasks managed by that pantasks. For example, the pantasks which +\ippprog{pantasks} will be supplied a set of labels which are then applied to +all tasks managed by that \ippprog{pantasks}. For example, the \ippprog{pantasks} which manages the nightly processing of the basic science analysis stages -(chip - warp, stack, diff) is supplied with several labels which +(\ippstage{chip} - \ippstage{warp}, \ippstage{stack}, \ippstage{diff}) is supplied with several labels which correspond to the different kinds of observations being performed. In this way, the analysis of the nightly observations is kept separate @@ -2083,5 +2126,5 @@ \note{then discuss the addstar sequences with manual triggering} -Outside of the basic sequence of chip to warp, there is no single +Outside of the basic sequence of \ippstage{chip} to \ippstage{warp}, there is no single natural next step. For example: a stack can be generated with any number of input warps; a difference image can be generated between a @@ -2103,5 +2146,5 @@ significantly reduced from the arbitrary case. -{\em Queuing the diffs} is done by first examining the set of all +Queuing the diffs is done by first examining the set of all exposures that have been taken at the summit on the current night of observing, and querying information from each stage up through @@ -2111,5 +2154,5 @@ group are then sorted by increasing observation date (\ippdbcolumn{dateobs}). The database results for each stage -(chip-warp) are checked to ensure that the selected exposures have +(\ippstage{chip}-\ippstage{warp}) are checked to ensure that the selected exposures have been successfully processed for all stages through \ippstage{warp}. Exposure groups are ignored until all exposures have either been @@ -2129,5 +2172,5 @@ that were excluded due to an odd number of exposures to be paired with the exposure closest in time (with the exposure that was previously -first ignored). Exposure pairs in which at least one exposures does +first ignored). Exposure pairs in which at least one exposure does not have a pre-existing difference image are queued for difference image analysis. @@ -2138,9 +2181,9 @@ exposures, as this is the number of exposures taken for each field. Once this number was reached, no more exposures are expected, so -\ippstage{stack} database entries can be queued with the +\ippstage{stack} database entries can be queued from the \ippstage{warp} entries. Again, failures and weather can reduce the number of usable exposures. If no stack could be made for a given MD field with the minimum number of inputs by the time of the -end-of-night darks, stacks are generated using using whatever +end-of-night darks, stacks are generated using whatever exposures are available. @@ -2161,21 +2204,23 @@ \ippdbtable{lapRun} entries can be queued that define a \ippdbcolumn{filter} and a \ippdbcolumn{projection_cell} to be -considered. A \ippdbcolumn{projection_cell} is a unit of sky defined -to be a square four degrees on each side which has a single tangent -plane projection \citep[][see]{waters2017}. \note{does waters2017 - discuss RINGS.V3? if not, where?} Once this entry is defined, is is -populated with exposures (stored in the \ippdbtable{lapExp} table in -the database), with any exposure located within 5 degrees of the -center of the projection cell included. This radius ensures that any -exposure that overlaps the projection cell will be included. Once the -exposures have been added, the other exposures within the same -sequence are checked to see if a \ippstage{chip} stage entry has been -generated, and if so, the \ippdbcolumn{chip_id} for that entry is -saved into the \ippdbtable{lapExp} as well. This linkage ensures that -each exposure is only processed once. If no entry is found, a new -\ippstage{chip} entry is queued for processing. The task periodically -checks the status of the exposures in each \ippdbtable{lapRun} entry, -and if they have all completed the \ippstage{warp} stage, then a -\ippstage{stack} is queued for each skycell contained within the +considered. These projection cells match the tangent plane centers +used for the warp tessellation. A \ippdbcolumn{projection_cell} is a +unit of sky defined to be a square four degrees on each side which has +a single tangent plane projection \citep[][see]{waters2017}. +\note{does waters2017 discuss RINGS.V3? if not, where?} Once this +entry is defined, it is populated with all exposures (stored in the +\ippdbtable{lapExp} table in the database) that are located +within 5 degrees of the center of the projection cell included. This +radius ensures that any exposure that overlaps the projection cell +will be included. Once the exposures have been added, the other +exposures within the same sequence are checked to see if a +\ippstage{chip} stage entry has been generated, and if so, the +\ippdbcolumn{chip_id} for that entry is saved into the +\ippdbtable{lapExp} as well. This linkage ensures that each exposure +is only processed once. If no entry is found, a new \ippstage{chip} +entry is queued for processing. The task periodically checks the +status of the exposures in each \ippdbtable{lapRun} entry, and if they +have all completed the \ippstage{warp} stage, then a \ippstage{stack} +is queued for each skycell contained within the \ippdbcolumn{projection_cell}. @@ -2192,6 +2237,6 @@ system per-se, but only method of tracking the locations of files within the file system, and of tracking duplicate copies of the same -file. The core of \ippprog{Nebulous} is a dedicated database engine -which tracks ``storage objects'', the concept of a file exists in the +file. The core of \ippprog{Nebulous} is a mysql database which tracks +``storage objects'', the equivalent concept of a file within the system. Each storage object may be associated with a number of copies of the actual files on the disks in the storage system (called @@ -2213,6 +2258,6 @@ stored on a specific computer (for at least one of the instances). All of the analysis stages which interact with that chip could then be -preferentially targetted to be run on that computer. The localization -in \ippprog{Nebulous} and the host targetted processing in pantasks +preferentially targeted to be run on that computer. The localization +in \ippprog{Nebulous} and the host targeted processing in \ippprog{pantasks} can therefore work together to encourage processing to require only local disk access, reducing the I/O local on the network @@ -2221,6 +2266,6 @@ practice, the as-built IPP has had sufficient network bandwidth that this targetting was not required. In practice, due to the timing of -hardware aquisition, occasional hardware failures, and other -organizational details, targetted processing has only been used to a +hardware acquisition, occasional hardware failures, and other +organizational details, targeted processing has only been used to a moderate degree within the Pan-STARRS cluster. \note{can we get a number here?} @@ -2229,14 +2274,26 @@ The user interfaces to Nebulous consist of command-line programs as -well as APIs in both C and Perl. The basic user commands to interact -with Nebulous are to 1) create a new storage object and associated -instance; 2) add a new instance to an existing storage object; 3) -remove (cull) an instance; 4) delete a storage object; and 5) find a -file associated with a given storage objects. Note that these user -commands do not affect the files on disk \note{true for cull?} -(exception: the create function will create an empty file if one does -not exist). They only change the state of the Nebulous database; it -is the responsibility of the user program to read and write data to a -file and to create the copies, etc. +well as APIs in both C and Perl. + +"The basic user commands to interact with Nebulous are to 1) query the +database for an existing storage object, and find a valid file +instance associated with that object; 2) create a new storage object, +which instantiates an empty file that can be opened for writing; 3) +replicate an existing storage object to create more file instances; 4) +cull a single file instance of storage object from the cluster; and 5) +remove a storage object, and ensure that all file instances are +removed. The filehandles returned for newly created instances can +then be opened for reading and writing data to that instance. + +% The basic user commands to interact +% with Nebulous are to 1) create a new storage object and associated +% instance; 2) add a new instance to an existing storage object; 3) +% remove (cull) an instance; 4) delete a storage object; and 5) find a +% file associated with a given storage objects. Note that these user +% commands do not affect the files on disk \note{true for cull?} +% (exception: the create function will create an empty file if one does +% not exist). They only change the state of the Nebulous database; it +% is the responsibility of the user program to read and write data to a +% file and to create the copies, etc. For the Nebulous users, the identifier of a storage object is a unique @@ -2247,8 +2304,8 @@ computer (HOST) and disk (VOL). The path and filename portions become the identifier and are recorded in the \ippmisc{storage_object} table -in the \ippmisc{extern_id} field. A storage object entry is then -created in the database for this id, and an instance of the file -created on the specified node (or at random from available nodes if -left empty). +in the \ippmisc{ext_id} field. A storage object entry is then created +in the database for this id, and an instance of the file created on +the specified node. If the host is unspecified, or if the specified +volume is full, then a host is chosen at random from available nodes. Files are stored on specific computers in a \ippprog{Nebulous} @@ -2258,9 +2315,9 @@ \code{nebulous}. Beneath the top-level directory are 256 subdirectories with names of the form 00 - ff (i.e., 2 digit -hexadecimate number). Each subdirectory again as 256 subdirectories -with the same naming scheme. +hexadecimal number). Each subdirectory has 256 subdirectories with +the same naming scheme. The filename of an instance in Nebulous is deterministic and derived -from the \ippmisc{extern_id}: the \ippmisc{extern_id} is hashed using +from the \ippmisc{ext_id}: the \ippmisc{ext_id} is hashed using the SHA-1 function, and the first four hexadecimal digits of this hash are separated into two two-digit strings and used as the top and @@ -2269,5 +2326,5 @@ provide a unique SQL ID for each instance. Under the subdirectory identified above, the disk file name is by appending the database -instance id with a string derived from the \code{extern_id}: forward +instance id with a string derived from the \code{ext_id}: forward slash characters are replaced in the name with colons so the string can represent a file in the UNIX filesystem. For the example URI @@ -2333,6 +2390,16 @@ using only the low-latency SOAP communications. -\note{need a paragraph or two on stats: how many objects, how many - instances?} +The Nebulous database currently (2017 July) contains information about +5,560,533,654 file instances for 3,543,240,981 storage objects. All +raw data, along with permanent products such as catalogs and the +current versions of full-sky stacks, are replicated to ensure at least +two copies exist in case of hardware failure. Based on the most +recent database ID values (which are unique and never reused), this +corresponds to roughly half of all the storage objects and file +instances ever created, due to the transient nature of many pipeline +products. + +% those numbers are so_id 6758205602 ins_id 9971666505, with ratios +% 0.5242, 0.5576) \subsection{Datastore repositories} @@ -2343,5 +2410,5 @@ that exposes data in a common form. \note{add Isani / Hoblitt reference?} One of the main datastores used by the IPP is the one -located at the summit. This datastore exposes, a list of the +located at the summit. This datastore exposes a list of the exposures obtained since the start of the PS1 operations. Requests to this server may restrict to the latest by time. Each row in the @@ -2353,5 +2420,5 @@ associated with that exposure. This listing includes a link to the individual chip FITS files as well as an md5 checksum. Systems which -are allowed access may download chip FITS files via http requests to +are allowed access may download the raw chip FITS files via http requests to the provided links. @@ -2509,10 +2576,10 @@ These storage nodes are not fully capable of completing all processing on the short timescale necessary for each night's worth of data. To -increase the processing capability, we have a large number -\note{actual number?} of ``compute'' nodes, that have small amounts of -local storage, but are able to add processing power. In addition to -the direct processing of image data, these nodes are also used to -manage the \ippprog{Nebulous} file interface, as well as controlling -the job scheduling for the processing. +increase the processing capability, we have 212 ``compute'' nodes that +have small amounts of local storage, but are able to provide +additional processing power. In addition to the direct processing of +image data, these nodes are also used to manage the \ippprog{Nebulous} +file interface, as well as controlling the job scheduling for the +processing. The final type of computer in the cluster are the database servers. @@ -2631,18 +2698,18 @@ products are present. -Approximately half of the chip through warp processing for the PV3 -reduction was performed on Mustang, with 201,040 / 375,573 of the -\ippstage{camera} stage products reduced there. Only processing -through the \ippstage{stack} stage was attempted, although with a -smaller fraction of the total compared to the \ippstage{camera} stage, -with 290,257 / 998,886 being produced at Los Alamos. One reason for -this decrease is that due to the memory constraints on the Mustang -processing nodes, we were unable to run stacks with more than 25 -inputs there. Stacks with this larger number of inputs overflow the -memory of the processing node, and as they do not have disk space -available for use as virtual memory, cause the machine to hang until -the job time limit is reached. These stacks were instead processed on -the regular IPP cluster, where hosts with sufficent memory were -available. +Approximately half of the \ippstage{chip} through \ippstage{warp} +processing for the PV3 reduction was performed on Mustang, with +201,040 / 375,573 of the \ippstage{camera} stage products reduced +there. Only processing through the \ippstage{stack} stage was +attempted, although with a smaller fraction of the total compared to +the \ippstage{camera} stage, with 290,257 / 998,886 being produced at +Los Alamos. One reason for this decrease is that due to the memory +constraints on the Mustang processing nodes, we were unable to run +stacks with more than 25 inputs there. Stacks with larger numbers of +inputs overflow the memory of the processing node, and as they do not +have disk space available for use as virtual memory, cause the machine +to hang until the job time limit is reached. These stacks were +instead processed on the regular IPP cluster, where hosts with +sufficent memory were available. \subsection{UH Cray Cluster}