Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40065) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40071) @@ -202,4 +202,35 @@ reducing data from other cameras and telescopes. +\note{overview discussion of Pan-STARRS: the telescope, survey time + period, surveys. 2 paragraphs.} + +The Pan-STARRS Image Processing Pipeline consists of a suite of +software programs and data systems that are designed to reduce +astronomical images, with a focus on parallelization necessary to +speed the processing of the large images produced by the GPC1 camera. +Part of this parallelization is derived from the fact that this camera +consists of 60 independent orthogonal transfer array (OTA) devices, +and can therefore be processed simultaneously. Although there are +multiple stages that operate on an entire exposure at once, the +majority of stages operate only on smaller segments of a full exposure +to allow the processing tasks to be spread over the machines in the +processing cluster. + + +\note{fix this summary once outline is solidified} + +This paper presents a description of the IPP data handling system. +Section \ref{sec:subsystems} describes the major IPP subsystems that +underlie the main pipeline, providing a set of common interfaces and +tools used at multiple stages. The main processing stages of the +pipeline are described in Section \ref{sec:stages}, although all +exposures may not necessarily pass through each of these stages. The +hardware systems that have done the processing for the PV3 data +release are listed in Section \ref{sec:hardware}, with some details +on the scale of computing needed to reduce this large number of +exposures. Finally, Section \ref{sec:discussion} presents a +discussion of some of the lessons learned in the creation of the IPP, +and its utility in reducing data from other cameras and telescopes. + {\color{red} {\em Note: These papers are being placed on arXiv.org to provide crucial support information at the time of the public @@ -213,23 +244,21 @@ \label{sec:overview} -\subsection{Elements of the Pan-STARRS Data Processing System} - -The Pan-STARRS Data Analysis system contains many features to support -a wide range of activities: archiving and management of the raw and -processed image files; real-time nightly processing of images for -transient and moving object science; large-scale re-processing and +The Pan-STARRS Data Analysis system consists of many elements to +support the wide range of activities: archiving and management of the +raw and processed image files; real-time nightly processing of images +for transient and moving object science; large-scale re-processing and calibration to produce measurements for the science collaboration and -the wider public; specialized image processing to facilitate research -and development of the analysis system itself; and distribution of the -resulting data products to various consumers in a variety of formats -and modes. +the wider public; specialized image processing tasks to facilitate +research and development of the analysis system itself; distribution +of the resulting data products to various consumers in a variety of +formats and modes. The Pan-STARRS Data Analysis system is divided internally into several major components: \begin{itemize} -\item Summit : both the camera and observatory summit systems perform +\item Summit Processing : both the camera and observatory summit systems perform data analysis tasks needed to support the on-going observations. In this article, we focus only on those aspects used by the off-summit - analysis stages. + analysis stages. \note{is summit processing discussed anywhere?} \item Image Processing Pipeline (IPP) : this portion of the data analysis system takes the data from raw pixels on the summit @@ -244,18 +273,18 @@ \end{itemize} The above set of analysis stages take place at the IfA within the -scope of responsibility of the Pan-STARRS Observatory. Within the +scope of responsibility of the Pan-STARRS Observatory. Across the wider Pan-STARRS colloboration(s), additional data analysis operations are performed to support science results. These collaboration-wide analysis operations range from those which are tightly-coupled to the Pan-STARRS Observatory system, such as the analysis of the transient -discovery teams and the public archive database at MAST, to those -which perform offline analysis for eventual ingest back into the -Pan-STARRS databases and archive. The latter category includes the -ubercal photometric analysis, the photo-z analysis, and the QSO / RR -Lyra search efforts. In addition, collaborations within the wider +search teams and the public archive database at MAST, to those which +perform offline analysis for eventual ingest back into the Pan-STARRS +databases and archive. The latter category includes the ubercal +photometric analysis \citep{ubercal}, the photo-z analysis +\citep{photoz}, and the QSO / RR Lyra search efforts +\citep{hernitschek2016}. In addition, collaborations within the wider Pan-STARRS community have implemented a variety of science-level -analyses of their own to support their science goals (e.g., M31 -Cepheid search). This article discusses the analysis elements which -take place at the IfA except as noted. +analyses of their own to support their science goals \citep[e.g., M31 + variable search][]{M31.REF}. Figure~\ref{fig:analysis.elements} illustrates the many elements of @@ -266,19 +295,5 @@ the summit systems are described by \note{REF?}. -\begin{figure*}[htbp] - \begin{center} - \includegraphics[width=\hsize,clip]{PS1_Data_Analysis_System_Overview.pdf} - \caption{\label{fig:analysis.elements} Elements of the Pan-STARRS\,1 - Data Analysis System. Rectangles represent data analysis steps; - ellipses represent databases; rounded rectangles represent - external groups (``customers''). The arrows show a simplified representation - of the major flow of data between the analysis stages and data - processing elements.} - \end{center} -\end{figure*} - -\subsection{Nightly Processing Analysis Stages} - -Data analysis to support nighly science operations is driven by two +Data analysis to support nightly science operations is driven by two main goals: 1) rapid detection of the moving and transient sources to enable recovery or follow-up with other telescopes. 2) regular @@ -289,37 +304,38 @@ detail below. In short, each image is processed independently to correct for instrumental signatures and to detect the astronomical -sources (chip); astrometric and photometric calibrations are -determined (camera), and finally images are geometric transformed to a -common pixel representation (warp). Warped images may either be added -together (stack) or used in an image subtraction (diff). As part of nightly -science processing, images for certain fields such as the Medium Deep -survey fields (see \cite{}), are stacked together in nightly chunks, -providing deeper detection capability on short timescales. Depending -on the survey mode, difference images are generated for the nightly -stack images (vs a deep stack template) or for individual warp images. -In the later case, the warp images may be difference against another -warp from the same night or against a reference stack from the -appropriate part of the sky. - -\subsection{Re-processing Analysis Stages} +sources (\IPPstage{chip}); astrometric and photometric calibrations +are determined (\IPPstage{camera}), and finally images are +geometrically transformed to a common pixel representation +(\IPPstage{warp}). Warped images may either be added together +(\IPPstage{stack}) or used in an image subtraction (\IPPstage{diff}). +For nightly science operations, images for certain fields such as the +Medium Deep survey fields \citep[see][]{MDref}, are stacked together +in nightly chunks, providing deeper detection capability on 1-day +timescales. Depending on the survey mode, difference images are +generated for the nightly stack images (vs a deep stack template) or +for individual warp images. In the later case, the warp images may be +difference against another warp from the same night or against a +reference stack from the appropriate part of the sky. Pan-STARRS has performed several large-scale reprocessings of both the -Medium Deep and $3\pi$ Survey data. For the $3\pi$ Survey data, we identify -these large-scale reprocessings as PV1, PV2, and PV3 (we also define -the nightly science analysis of the data as PV0). For these -reprocessing stages, the standard steps of chip through warp, plus -stack and diff are performed, starting from raw data, using a single -homogenous version of the data analysis procedures. (PV2 was a -special case in which we started from the camera level products of -PV1). In addition to the analysis stages which are common with the -nightly processing, these large-scale reprocessing stages include -additional processing: a more detailed photometric analysis is -performed on the stacks, including morphological analysis appropriate -to galaxies. The results of the stack photometry analysis are used to -drive a forced-photometry analysis of the warp images. The data -products from the camera, stack photometry, and forced-warp photometry -analysis stages are ingested into the internal calibration database -(DVO, the Desktop Virtual Observatory) and used for photometric and -astrometric calibrations (see Section~\ref{sec:DVO}) +Medium Deep and 3pi Survey data for internal consumption. For the 3pi +Survey data, we identify these large-scale reprocessings as PV1, PV2, +and PV3, with PV3 the analysis used for the first public data release, +DR1. We also refer to the nightly science analysis of the data as +PV0. For these reprocessing stages, the standard steps of chip +through warp, plus stack and diff are performed, starting from raw +data, usually using a single homogenous version of the data analysis +procedures. PV2 was a special case in which we started from the +camera level products of PV1 to speed up the turn-around to the +community. In addition to the analysis stages listed above which are +shared with the nightly processing, these large-scale reprocessing +analyses include additional processing. A more detailed photometric +analysis is performed on the stacks, including morphological analysis +appropriate to galaxies. The results of the stack photometry analysis +are used to drive a forced-photometry analysis of the warp images. +The data products from the camera, stack photometry, and forced-warp +photometry analysis stages are ingested into the internal calibration +database (DVO, the Desktop Virtual Observatory) and used for +photometric and astrometric calibrations. \subsection{Data Access and Distribution} @@ -347,4 +363,54 @@ \label{sec:processing.database} +\begin{table*} +\caption{\label{tab:database_schema} GPC1 Database Schema Outline}\vspace{-0.5cm} +\begin{center} +\begin{tabular}{lllll} +\hline +\hline +{\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} & {\bf Notes} \\ +\hline + \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add_id} & \\ + \ippstage{camera} & \ippdbtable{camRun} & \ippdbtable{camProcessedExp} & \ippdbcolumn{cam_id} & \\ + \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} & \\ + \ippstage{detrend} & \ippdbtable{detRun} & \ippdbtable{detRunSummary} & \ippdbcolumn{det_id} & \\ + & & \ippdbtable{detInputExp} & & \\ + & & \ippdbtable{detRegisteredImfile} & & Information about detrends produced externally.\\ + & & \ippdbtable{detStackedImfile} & & \\ + & \ippdbtable{detProcessedExp} & \ippdbtable{detProcessedImfile} & & \\ + & \ippdbtable{detResidExp} & \ippdbtable{detResidImfile} & & \\ + & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & & \\ + \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff_id} & \\ + & & \ippdbtable{diffInputSkyfile} & & \\ + \ippstage{distribution} & \ippdbtable{distRun} & \ippdbtable{distComponent} & \ippdbcolumn{dist_id} & \\ + & & \ippdbtable{distTarget} & & \\ + \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} & \\ + \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff_id} & \\ + & & \ippdbtable{fullForceResult} & & \\ + & & \ippdbtable{fullForceSummary} & & Properties about average parameters from all results.\\ + \ippstage{lap} & \ippdbtable{lapSequence} & \ippdbtable{lapRun} & \ippdbcolumn{seq_id} & Sequence of full reprocessing\\ + & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap_id} & \\ + \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub_id} & \\ + & & \ippdbtable{publishClient} & & \\ + \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & & Lists locations to check for new exposures.\\ + & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} & Exposures available at the telescope.\\ + & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile} & & Exposures that are being downloaded.\\ + & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp_id} & Exposures that have been saved to IPP cluster.\\ + + \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp_id} & \\ + \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote_id} & \\ + \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal_id} & \\ + \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack_id} & \\ + & & \ippdbtable{stackSumSkyfile} & & \\ + \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky_id} & \\ + & & \ippdbtable{staticskyResult} & & \\ + \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp_id} & \\ + & & \ippdbtable{warpSkyCellMap} & & Mapping of input chips to projection skycells.\\ + & & \ippdbtable{warpSkyfile} & & \\ +\hline +\end{tabular} +\end{center} +\end{table*} + A critical element in the Pan-STARRS IPP infrastructure is the processing database. This database, using the mysql database engine, @@ -361,9 +427,7 @@ database, since a single instance of the database is used to track the processing of images and data products related to the PS1 GPC1 camera. -This same database engine also has instances for other cameras -processed by the IPP, e.g., GPC2, the test cameras TC1, TC3, and the -Imaging Sky Probe (ISP). In general, processing information for -different cameras is separate in differnt processing database; merging -of output products takes place in DVO. +This same database engine also has instances (same schema, different +data) for other cameras processed by the IPP, e.g., GPC2, the test +cameras TC1, TC3, and the Imaging Sky Probe (ISP). Within the processing database, the various processing stages are @@ -681,21 +745,20 @@ table. -\subsection{Fake Analysis} -\label{sec:fake} -% \note{drop} - -The \ippstage{fake} stage was originally designed to do false source -injection and recovery, in order to determine the detection efficiency -of sources on the exposure. However, early in the design of the IPP, -this task was moved to the rest of the photometry analysis done at the -\ippstage{chip} stage. Removing the stage would require significant -changes to the database schema. As a result, this conveniently named -stage generally does no actual data processing, and consists mainly of -database operations to move the exposure on to the \ippstage{warp} -stage. The operations mimic the \ippstage{chip} stage, with -individual jobs run for each OTA that update rows in the -\ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that -updates the \ippdbtable{fakeRun} table and promotes the exposure to -the next stage by adding a row to the \ippdbtable{warpRun} table. +%% \subsection{Fake Analysis} +%% \label{sec:fake} +%% +%% The \ippstage{fake} stage was originally designed to do false source +%% injection and recovery, in order to determine the detection efficiency +%% of sources on the exposure. However, early in the design of the IPP, +%% this task was moved to the rest of the photometry analysis done at the +%% \ippstage{chip} stage. Removing the stage would require significant +%% changes to the database schema. As a result, this conveniently named +%% stage generally does no actual data processing, and consists mainly of +%% database operations to move the exposure on to the \ippstage{warp} +%% stage. The operations mimic the \ippstage{chip} stage, with +%% individual jobs run for each OTA that update rows in the +%% \ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that +%% updates the \ippdbtable{fakeRun} table and promotes the exposure to +%% the next stage by adding a row to the \ippdbtable{warpRun} table. \subsection{Image Warping} @@ -779,10 +842,11 @@ exposures, producing ``deep stacks''. In addition, a `best seeing' set of stacks have been produced \note{using image quality cuts to be - described}. We have also generated out-of-season stacks for the -Medium Deep fields, in which all image not from a particular observing -season for a field are combined into a stack. These later stacks are -useful as deep templates when studying long-term transient events in -the Medium Deep fields as they are not (or less) contaminated by the -flux of the transients from a given season. + described: need input from MEH}. We have also generated +out-of-season stacks for the Medium Deep fields, in which all image +not from a particular observing season for a field are combined into a +stack. These later stacks are useful as deep templates when studying +long-term transient events in the Medium Deep fields as they are not +(or less) contaminated by the flux of the transients from a given +season. When a given set of \ippstage{stack} stage are defined, exposures with @@ -823,14 +887,13 @@ deferred to the \ippstage{staticsky} stage. This separation is maintained because the photometry analysis of the \ippstage{stack} -images, including convolved galaxy model fitting, is performed on all -5 filters simultaneously. By deferring this analysis, the processing -system may also decouple the generation of the pixels from the source -detection. This makes the sequencing of analysis somewhat easier and -less subject to blocks due to a failure in the stacking analysis. -Similar to the \ippstage{stack} stage, an entry is created in the -\ippdbtable{staticskyRun} table, linked to a series of rows in the -\ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky_id}, -each of which also contains the appropriate \ippdbcolumn{stack_id} -entries for the skycell under consideration. +images is performed on all 5 filters simultaneously. By deferring +this analysis, the processing system may also decouple the generation +of the pixels from the source detection. This makes the sequencing of +analysis somewhat easier and less subject to blocks due to a failure +in the stacking analysis. Similar to the \ippstage{stack} stage, an +entry is created in the \ippdbtable{staticskyRun} table, linked to a +series of rows in the \ippdbtable{staticskyInput} table by a common +\ippdbcolumn{sky_id}, each of which also contains the appropriate +\ippdbcolumn{stack_id} entries for the skycell under consideration. The input images are passed to the \ippprog{psphotStack} program, @@ -853,10 +916,12 @@ The stack photometry output files consist of a set of FITS table catalogs, with one file for each filter. Within these files, there -are multiple table extensions that include: the measurements of -sources based on the PSF model; aperture like parameters such as the -Petrosian flux and radius; the convolved galaxy model fits; and the -radial aperture measurements. Once the photometry is complete, a row -is added to the \ippdbtable{staticskyResult} table with basic -statistics from the analysis. +are multiple table extensions, with different classes of measurements +saved in the different extensions. The extensions include a table of +the measurements of sources based on the PSF model; a table of +aperture-like parameters such as the Petrosian flux and radius; a +table of the convolved galaxy model fits; and a table of the radial +aperture measurements. Once the photometry is complete, a row is +added to the \ippdbtable{staticskyResult} table with basic statistics +from the analysis. The stack photometry output catalogs are re-calibrated for both @@ -865,13 +930,13 @@ \ippstage{skycal} stage, each skycell is processed independently. Because of this independence, when queued for processing, the entries -in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky_id} +in the \ippdbtable{skycalRun} table contain the \IPPdbcolumn{sky_id} and \ippdbcolumn{stack_id} entries of the parent data directly. As in the \ippstage{camera} stage, the \ippprog{psastro} program reads in -the stack photometry catalog, and produces a calibrated output. A -different processing recipe is supplied to \ippprog{psastro}, which -controls for the different data. The same reference catalog is used -for the \ippstage{camera} and \ippstage{stack} calibration stages. -Upon completion, the analysis statistics are written to the -\ippdbtable{skycalResult} table. +the stack photometry catalog, and produces a calibrated output, with +format matching the input. A different processing recipe is supplied +to \ippprog{psastro}, which controls for the different data. The same +reference catalog is used for the \ippstage{camera} and +\ippstage{stack} calibration stages. Upon completion, the analysis +statistics are written to the \ippdbtable{skycalResult} table. \subsection{Forced Warp Photometry} @@ -930,14 +995,12 @@ individual warp images used to generate the stack. This \ippstage{fullforce} analysis is performed on all warps for a single -skycell and filter as a single unit within the processing database, -while individual warps are processed individually in parallel as -separate processing jobs. - -When processing is queued for this stage, an entry is added to the -\ippdbtable{fullForceRun} primary database table with a reference to -the corresponding stack and \ippdbcolumn{skycal_id} entry that is the -input source of detections to be measured. The \ippdbcolumn{warp_id} -values for the input \ippstage{warp} stage images that contributed to -the \ippstage{stack} associated with that \ippdbcolumn{skycal_id} are +skycell and filter as a single unit, as this matches the arrangement +of the input source catalog from the \ippstage{skycal} stage. When +processing is queued for this stage, an entry is added to the +\ippdbtable{fullForceRun} primary database table linking to the +specific \ippdbcolumn{skycal_id} entry that will be used as the +catalog for the photometry. The \ippdbcolumn{warp_id} values for the +input \ippstage{warp} stage images that contributed to the +\ippstage{stack} associated with that \ippdbcolumn{skycal_id} are then added to the \ippdbtable{fullForceInput} table, linked to the primary table by the \ippdbcolumn{ff_id} identifier. The individual @@ -945,37 +1008,4 @@ stage image products along with the \ippstage{skycal} catalog to the \ippprog{psphotFullForce} program. - -In this program, the positions of sources are loaded from the input -catalog. PSF stars are pre-identified from the stack image and a PSF -model generated for each \ippstage{warp} image based on those stars, -using the same stars for all warps to the extent possible (PSF stars -which are excessively masked on a particular image are not used to -model the PSF). The PSF model is fitted to all of the known source -positions in the warp images. Aperture magnitudes, Kron magnitudes, -and moments are also measured at this stage for each warp. Note that -the flux measurement for a faint, but significant, source from the -stack image may be at a low significance (less than the $5\sigma$ -criterion used when the photometry is not run in this forced mode) in -any individual warp image; the flux may even be negative for specific -warps. When combined together, these low-significance measurements -will result in a signficant measurement as the signal-to-noise -increases by the square root of the number of measurements. The -individual warp measurements are combined together to generate -averages values within DVO. - -Upon completion of the forced photometry (for point sources as well as -galaxies, discussed below), an entry is added to the -\ippdbtable{fullForceResult} table with the processing statistics for -that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}. -Once all of the entries in the \ippdbtable{fullForceInput} table have -finished, a summary operation is run to combine the galaxy photometry -analysis measurements into a single value. The output catalogs listed -in the \ippdbtable{fullForceResult} table are passed to the -\ippprog{psphotFullForceSummary} to do this averaging. When this -completes, an entry is added to the \ippdbtable{fullForceSummary}, and -the \ippdbtable{fullForceRun} entry is marked as completed. - -\subsubsection{Forced Galaxy Models} -\note{too much detail in this section; balance relative to psphot} The convolved galaxy models are also re-measured on the @@ -989,48 +1019,25 @@ the PSF-convolved galaxy models are of limited accuracy. -In the \ippstage{fullforce} galaxy model analysis, we assume that the -galaxy position and position angle, along with the Sersic index if -appropriate, have been sufficiently well determined in the -\ippstage{staticsky} analysis. In this case, the goal is to determine -the best values for the major and minor axis of the elliptical contour -and at the same time the best normalization corresponding to the best -elliptical shape, and thus the best galaxy magnitude value. - -For each \ippstage{warp} image, the \ippstage{staticsky} value for the -major and minor axis are used as the center of a $7\times{} 7$ grid -search of the major and minor axis parameter values. The grid spacing -is defined as a function of the signal-to-noise of the galaxy in the -stack image so that bright galaxies are measured with a much finer -grid spacing that faint galaxies \note{need to quantify this}. For -each grid point, the major and minor axis values at that point are -determined for the model. The model is then generated and convolved -with the PSF model for the \ippstage{warp} image at that point. The -resulting model is then compared to the \ippstage{warp} pixel data -values and the best fit normalization value is defined. The -normalization and the $\chi^2$ value for each grid point is recorded. - -For a given galaxy, the result is a collection of $\chi^2$ values for -each of the grid points spanning all \ippstage{warp} images. A single -$\chi^2$ grid can then be made by combining each grid point across the -inputs. The combined $\chi^2$ for a single grid point is simply the -sum of all $\chi^2$ values at that point. If, for a single -\ippstage{warp} image, the galaxy model is excessively masked, then -that image will be dropped for all grid points for that galaxy. The -reduced $\chi^2$ values can be determined by tracking the total number -of pixels used across all inputs to generate the combined $\chi^2$ -values. From the combined grid of $\chi^2$ values, the point in the -grid with the minimum $\chi^2$ is found. Quadratic interpolation is -used to determine the major, minor axis values for the interpolated -minimum $\chi^2$ value. The errors on these two parameters is then -found by determining the contour at which the $\chi^2$ increases by 1. - -Thus the \ippstage{fullforce} galaxy analysis uses the PSF information -from each \ippstage{warp} to determine a best set of convovled galaxy -models for each object in the \ippstage{skycal} catalog. - -\note{discuss the subset of galaxy models and objects}. +Upon completion of the forced photometry (for point sources as well as +galaxies, discussed below), an entry is added to the +\ippdbtable{fullForceResult} table with the processing statistics for +that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}. +The individual warp measurements are combined together to produce an +average warp photometry value for each object within the context of +the DVO object database system, including re-calibration of each warp +based on the tie to the average photometry of the objects measured in +the \ippstage{camera} stage. + +Once all of the entries in the \ippdbtable{fullForceInput} table have +finished, a summary operation is run to combine the galaxy photometry +analysis measurements into a single value. The output catalogs listed +in the \ippdbtable{fullForceResult} table are passed to the +\ippprog{psphotFullForceSummary} to do this averaging. When this +completes, an entry is added to the \ippdbtable{fullForceSummary}, and +the \ippdbtable{fullForceRun} entry is marked as completed. \subsection{Difference Images} \label{sec:diff} + Two of the primary science drivers for the Pan-STARRS system are the search hazardous asteroids and the search for Type Ia supernovae to @@ -1052,5 +1059,5 @@ or from a pair of \ippstage{stack} stage images. During the PS1 survey, pairs of exposures, call TTI pairs (see~\note{Survey - Strategy}), were obtained for each pointing within a $\approx$ 1 + Strategy in Chambers et al}), were obtained for each pointing within a $\approx$ 1 hour period in the same filter, and to the extent possible with the same orientation and boresite position. The standard PS1 nightly @@ -1186,14 +1193,18 @@ system. -There are 3 classes of photcodes defined within the DVO system. One -class of photcodes define the filter systems for the average -photometry measurements; these are called \ippmisc{SEC} photcodes. A -second class of photcode is associated with measurements from a -specific camera for which image metadata is available are called -\ippmisc{DEP} photcodes. There are also those measurements which come -from external data sources for which DVO does not have any information -to determine a calibration (e.g., instrumental magnitudes and detector -coordinates). These are measurements are reference values and are -assigned \ippmisc{REF} photcodes. +DVO includes two major classes of database tables: those containing +information about astronomical objects in the sky and those containing +other supporting information. The object-related tables are +partitioned on the basis of position in the sky: objects within a +region bounded by lines of constant RA,DEC are contained in a specific +file. The boundaries and the associated partition names are stored in +one of the supporting tables, \ippdbtable{SkyTable}. This table +contains the definitions of the boundaries for each sky region +(\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, +\ippdbcolumn{D_MAX}), the name of the sky region, an ID +(\ippdbcolumn{INDEX}, equal to the sequence number of the region in +the table), and index entries to enable navigation within the table. +The regions are defined in a hierarchical sense, with a series of +levels each containing a finer mesh of regions covering the sky. The names for \ippmisc{SEC} photcodes are the names of filter systems, @@ -1567,10 +1578,9 @@ appropriate database table with the \ippdbcolumn{stage_id} field. As some stages, such as the \ippstage{diff} stage, create more than a -single catalog for a single exposure, multiple entries with the -\ippdbcolumn{stage_id} are created, with the -\ippdbcolumn{stage_extra1} field containing an index to the individual -components. The catalog specified by the entry is added to the target -\ippmisc{minidvo} by the \ippprog{addstar} program, with object -constructed as described above (\S~\ref{sec:object}). When this +single catalog, multiple entries with the \ippdbcolumn{stage_id} are +created, with the \ippdbcolumn{stage_extra1} field containing an +index to the individual components. The catalog specified by the +entry is added to the target \ippmisc{minidvo} by the +\ippprog{addstar} program, \note{describe what's done?}. When this completes, an entry containing the statistics of the job is added to the \ippdbtable{addProcessedExp} table. @@ -2566,10 +2576,12 @@ values used for the various IPP processing stages. -\begin{deluxetable}{lcc} - \tablecolumns{3} - \tablewidth{0pc} - \tablecaption{Cost values for remote processing} - \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}} - \startdata +\begin{table} +\caption{\label{tab:SC_processing_parameters} Cost values for remote processing}\vspace{-0.5cm} +\begin{center} +\begin{tabular}{lcc} +\hline +\hline +{\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ +\hline \ippstage{chip} & 150 & 2 \\ \ippstage{camera} & 1700 & 2 \\ @@ -2578,8 +2590,26 @@ \ippstage{staticsky} & 7200 & 6 \\ % \ippstage{diff} & 300 & 2 \\ - \ippstage{fullforce} & 300 & 2 - \enddata - \label{tab:SC processing parameters} -\end{deluxetable} + \ippstage{fullforce} & 300 & 2 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +%% \begin{deluxetable}{lcc} +%% \tablecolumns{3} +%% \tablewidth{0pc} +%% \tablecaption{Cost values for remote processing} +%% \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}} +%% \startdata +%% \ippstage{chip} & 150 & 2 \\ +%% \ippstage{camera} & 1700 & 2 \\ +%% \ippstage{warp} & 110 & 2 \\ +%% \ippstage{stack} & 1500 & 6 \\ +%% \ippstage{staticsky} & 7200 & 6 \\ +%% % \ippstage{diff} & 300 & 2 \\ +%% \ippstage{fullforce} & 300 & 2 +%% \enddata +%% \label{tab:SC processing parameters} +%% \end{deluxetable} Once the preparation for the job is complete, the input and output @@ -2682,4 +2712,9 @@ \note{logical or alphabetical sequence?} +\end{document} + +Figures needed for this document: + +* \begin{center} \begin{deluxetable}{lllll} @@ -2730,15 +2765,3 @@ \end{deluxetable} \end{center} - - -\begin{verbatim} -MAJOR TODO ITEMS: -* add figure showing DVO schema relationships -* re-read and trim details as needed (referring to the other papers) -* add some specific numbers (data volume, processing times, etc) -* where is the smf/cmf format defined? psphot? -* where is the GPC1 naming convention discussed? -* where are the flat-field seasons listed (magnier2017.calibration?) -\end{verbatim} - -\end{document} +