Index: /trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40029) +++ /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40030) @@ -92,10 +92,5 @@ \begin{verbatim} MAJOR TODO ITEMS: -* introduce and describe RINGS.V3 in or before warp section (refer to Waters if appropriate) * re-read and trim details as needed (referring to the other papers) -* re-write the DVO description using topics list given -* write discussion of calibration operations (refer to cal paper) -* write IPP to PSPS summary -* write PSPS Load and Merge summary (use Flewelling paper for ref) * add some specific numbers (data volume, processing times, etc) \end{verbatim} @@ -121,4 +116,6 @@ Advanced Technology Research Center in Kula, the main facility of the University of Hawaii's Institute for Astronomy operations on Maui. +The Pan-STARRS1 filters and photometric system have already been +described in detail in \cite{2012ApJ...750...99T}. For nearly 4 years, from 2010 May through 2014 March, this telescope @@ -141,4 +138,14 @@ database which is a critical element in the IPP infrastructure. +This paper (Paper II) presents a description of the Pan-STARRS data handling +systems, with an emphasis on the Image Processing Pipeline (IPP). The +Pan-STARRS Image Processing Pipeline consists of a suite of software +programs and data systems that are designed to reduce astronomical +images, measure astronomical sources on the images, perform the +calibration, and distribute the results to various users. The +processing system includes extensive parallelization across a large +cluster of computers in order to process the large amount of data +generated by the Pan-STARRS\,1 telescope. + %Chambers et al. 2017 (Paper I) %The Pan-STARRS\,1 Surveys @@ -182,23 +189,4 @@ %Huber et al. 2017 (Paper VII) describes the Medium Deep Survey in detail, including the unique issues and data products specific to that survey. The Medium Deep Survey is not part of Data Release 1. (DR1) - -% -The Pan-STARRS1 filters and photometric system have already been -described in detail in \cite{2012ApJ...750...99T}. - -This paper presents a description of the Pan-STARRS data handling -systems, with an emphasis on the Image Processing Pipeline (IPP). The -Pan-STARRS Image Processing Pipeline consists of a suite of software -programs and data systems that are designed to reduce astronomical -images, with the parallelization necessary to speed the processing of -the large images produced by the GPC1 camera. - -Part of this parallelization is derived from the fact that this camera -consists of 60 independent orthogonal transfer array (OTA) devices, -and can therefore be processed simultaneously. Although there are -multiple stages that operate on an entire exposure at once, the -majority of stages operate only on smaller segments of a full exposure -to allow the processing tasks to be spread over the machines in the -processing cluster. \note{move elsewhere?} Section~\ref{sec:overview} provides an overview of the full data @@ -552,4 +540,14 @@ originally expected. +\note{keep this paragraph?} + +Part of this parallelization is derived from the fact that this camera +consists of 60 independent orthogonal transfer array (OTA) devices, +and can therefore be processed simultaneously. Although there are +multiple stages that operate on an entire exposure at once, the +majority of stages operate only on smaller segments of a full exposure +to allow the processing tasks to be spread over the machines in the +processing cluster. + %% In the \ippstage{chip} stage, %% the individual OTA image files are processed independently in parallel @@ -710,5 +708,6 @@ \label{sec:warp} -\note{need to describe the RINGS.V3 tessellation and others} +\note{re-read and improve the text: better description of RINGS.V3 and + other related tessellations.} The \ippstage{warp} stage moves the data from a given exposure beyond @@ -1110,4 +1109,7 @@ \section{Post-Processing : Database Ingest and Calibration} \label{sec:postprocessing} + +\note{introduction to this section: data ingested into DVO database, + database gets calibrated, data ingested into PSPS via IPP to PSPS} \begin{table}[hb] @@ -1626,27 +1628,105 @@ astrometry is again performed this time using the corrected positions. -Photometric calibration involved the efforts of external collaborative -analysis. - -\begin{verbatim} -* data goes to harvard -* eddie determines the zero points for photometric data -* zero points are returned to ifa -* zero points are applied to the DVO -* systematic errors are measured (high-resolution flat-field) -* applied back to DVO -* relative photometry measured for non-photometric data -\end{verbatim} - -\subsection{IPP to PSPS} +Photometric calibration consists of determination of zero points for +each exposure along with corrections for systematic effects. In this +case, we rely on efforts of our external collaborators for the initial +zero point determination. The team at CfA downloaded the per-exposure +catalog files (`smf files') and determined the zero points of those +exposures which were believed to be obtained in photometric +conditions. This process, called `\"ubercal', is described in detail +by \cite{ubercal} for the first (PV1) version. In brief, photometric +periods, with time-scales of at least \note{half of a night}, are +identified by a combination of automatic analysis and manual +inspection. A single solution for all images in a given filter is +determined to minimize scatter for individual stars. The free +parameters in this solution consist of a single zero point and airmass +slope for each photometric period along with a collection of +flat-field offsets for several large time range (`flat-field +seasons'). For the PV3 \"ubercal analysis, the flat-field offsets +were determined on a $2\times2$ grid for each chip and 5 flat-field +seasons were chosen (listed in Table~\ref{tab:flat-field-seasons}). +The boundaries of the flat-field seasons were determined by +independent inspection of the residuals observed in the Medium Deep +fields. + +After the \"ubercal analysis of the photometric periods is completed, +the determined zero points, airmass corrections, and flat-field terms +are transmitted back to the IfA IPP team. These values are then +ingested into the master DVO database. An initial relative photometry +analysis is performed to tie the images without \"ubercal zero points +to the \"ubercal system. Zero points from the \"ubercal analysis are +not allowed to change, but zero points of the rest of the exposures +are determined to minimize the photometric scatter for bright stars. +These zero points are determined uniquely for each image. After an +initial relative photometry analysis, the photometric residuals are +used to determine a systematic correction as function of position in +the camera. This correction is equivalent to the flat-field +corrections determined as part of the \"ubercal analysis, but are much +higher spatial resolution ($40\times40$ corrections per chip) and are +determined for only the full time range of PV3. This high-resolution +flat-field correction addresses photometric variations due to spatial +variations in the PSF due to the optics and low-level effects on the +chips \citep[see][]{magnier2017c}. After the systematic corrections +have been determined and applied back to the database, a final +relative photometry analysis pass is performed. + +\subsection{Construction of the PSPS database} \label{sec:ipp2psps} -\note{Default to pointing to Flewelling et al 2017?} - -\begin{verbatim} -\end{verbatim} - -\subsection{PSPS Load and Merge} -\label{sec:psps} -\note{Default as well to pointing to Flewelling et al 2017?} + +The publically-visible Pan-STARRS database is hosted by the Space +Telescope Sciences Institute through their Mikulski Archive for Space +Telescopes (MAST). The underying database at MAST is a copy of a +database generated at the Institute for Astronomy by the subsystem +called PSPS : the \note{define PSPS}. The construction of the PSPS +version of the PS1 database starts once the PS1 photometry and +astrometry measurements have been calibrated within the DVO system. +The construction takes place in several stages, described in detail by +\cite{flewelling2017}. We summarize those steps here. + +The first stage of constructing the PSPS database consists of the +generation of small files called `batches' which contain a complete +set of measurements for a small chunk of the database tables. The +program which is responsible for the construction of these batches is +called \ippprog{ipptopsps}. Several different types of batches are +generated, relating to the different types of tables in PSPS. The +details of the batch construction depend on the batch type. + +One type of batch consists of measurements from the individual +exposures. These batches are generated based on the output catalog +files generated at the \ippstage{camera} stage (`smf files'). The +\ippprog{ipptopsps} program loads the complete set of measurements and +metadata from the smf catalog file, then queries the DVO database for +calibration parameters related to that smf file. The batch is +constructed by applying the photometric calibrations to the raw flux +measurements in the smf file. + +A second type of batch file consists of the measurements related to +the stack images. Again, \ippprog{ipptopsps} starts with the output +catalog files, selects the appropriate calibration information from +the DVO, and applies the calibration data to the raw measurements in +the stack catalog files. + +A third type of batch file consists of average properties of the +astronomical objects in the DVO database. Unlike the other two batch +types, this operation is performed solely via queries to the DVO +database. The complete set of average measurements for objects in a +single DVO spatial partition are loaded by \ippprog{ipptopsps} and +used to generate the batch file. + +As the batch files above are generated, the PSPS system can run in +parallel to ingest the measurements from these batch files. PSPS +downloads in sequence the batch files as they are generated and +unpacks the data. The data are then loaded into a small-scale version +of the PSPS database, using the full schema. After a large chunk of +batches have been loaded, the resulting tables are then merged into +the master PSPS database. After another large chunk of data has been +merged into the master PSPS database, a large-scale copy of the +database is made internally to provide a long-term backup and to aid +in error recovery. + +Once the full PSPS database has been loaded, or a complete set of +batches for a given batch type, the entire database is copied to +STScI where it can then be made visible either to the Pan-STARRS +Science Consortium or to the wider public. \section{Operations and Automation}