Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40064) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40065) @@ -95,5 +95,5 @@ The 1.8m Pan-STARRS\,1 telescope is located on the summit of Haleakala on the Hawaiian island of Maui. The wide-field optical design of the -telescope \citep{PS1.optics} produces a 3.3 degree field of view with +telescope \citep{2004SPIE.5489..667H} produces a 3.3 degree field of view with low distortion and minimal vignetting even at the edges of the illuminated region. The optics and natural seeing combine to yield @@ -102,10 +102,10 @@ a floor of $\sim 0.7$ arcseconds. -The \PSONE\ camera \citep{PS1.GPCA}, known as GPC1, consists of a +The \PSONE\ camera \citep{2009amos.confE..40T}, known as GPC1, consists of a mosaic of 60 back-illuminated CCDs manufactured by Lincoln Laboratory. The CCDs each consist of an $8\times8$ grid of $\sim 600\times 600$ pixel readout regions, yielding an effective $4800\times4800$ detector. Initial performance assessments are presented in -\cite{PS1.GPCB}. Routine observations are conducted remotely from the +\cite{2008SPIE.7014E..0DO}. Routine observations are conducted remotely from the Advanced Technology Research Center in Kula, the main facility of the University of Hawaii's Institute for Astronomy operations on Maui. @@ -123,5 +123,5 @@ search for potentially hazardous asteroids in our solar system. The details of the telescope, surveys, and resulting science publications -are described by \cite{Chambers}. +are described by \cite{chambers2017}. This is the second in a series of seven papers describing the @@ -153,5 +153,5 @@ %Magnier et al. 2017 (Paper II) %Pan-STARRS Data Processing Stages -%\citet[][Paper II]{magnier2017c} +%\citet[][Paper II]{magnier2017.datasystem} %describes how the various data processing stages are organised and implemented %in the Imaging Processing Pipeline (IPP), including details of the @@ -166,10 +166,10 @@ %Magnier et al. 2017 (Paper IV) %Pan-STARRS Pixel Analysis : Source Detection -\citet[][Paper IV]{magnier2017a} +\citet[][Paper IV]{magnier2017.analysis} describes the details of the source detection and photometry, including point-spread-function and extended source fitting models, and the techniques for ``forced" photometry measurements. %Magnier et al. 2017 (Paper V) %Pan-STARRS Photometric and Astrometric Calibration -\citet[][Paper V]{magnier2017b} +\citet[][Paper V]{magnier2017.calibration} describes the final calibration process, and the resulting photometric and astrometric quality. @@ -190,14 +190,15 @@ detail each of the analysis steps which may be applied to the images and resulting catalogs of detected sources. -Section~\ref{sec:postprocessing} discusses the calibration operations -and database used for calibration. Section~\ref{sec:operations} -discusses the operational infrastructure of the IPP. -Section~\ref{sec:hardware} discusses the hardware systems used by the -IPP for regular nightly operations and for processing the PV3 data -release, with some details on the scale of computing needed to reduce -this large number of exposures. Finally, Section~\ref{sec:discussion} -presents a discussion of some of the lessons learned in the creation -of the IPP, and its utility in reducing data from other cameras and -telescopes. +Section~\ref{sec:postprocessing} discusses the databasing system used +for calibration, the calibration operations, and summarizes the +construction of the public release database. +Section~\ref{sec:operations} discusses the operational infrastructure +of the IPP. Section~\ref{sec:hardware} discusses the hardware systems +used by the IPP for regular nightly operations and for processing the +PV3 data release, with some details on the scale of computing needed +to reduce this large number of exposures. Finally, +Section~\ref{sec:discussion} presents a discussion of some of the +lessons learned in the creation of the IPP, and its utility in +reducing data from other cameras and telescopes. {\color{red} {\em Note: These papers are being placed on arXiv.org to @@ -426,6 +427,4 @@ glitches or hardware crashes. -% \note{start of section needed a re-read} - \subsection{Summit copy} \label{sec:summitcopy} @@ -470,5 +469,5 @@ is ready to be registered. In this context, `registration' refers to the process of adding them to the database listing of known, raw -exposures (not to be confused with 'registration' in the sense of +exposures (not to be confused with `registration' in the sense of pixel re-alignment). The result of the registration analysis is an entry for each exposure in the \ippdbtable{rawExp} table, and one for @@ -525,23 +524,23 @@ (with the \ippdbcolumn{state} column indicating it needs processing), and the associated information listed in the \ippdbtable{rawImfile}, -jobs can be spawned for each component OTA. The \ippprog{pantasks} -environment managing the jobs attempts to target the processing host -to one that should host the OTA, to reduce number of operations done -on remote data. In practice, this targeted processing has not had as -large of an effect as was originally intended, as the data volume has +jobs can be spawned for each component OTA. + +The \ippstage{chip} stage is naturally parallelized by processing data +from each of the 60 OTAs independently. Several stages in the IPP +analysis are parallelized in a similar fashion; although there are +multiple stages that operate on an entire exposure at once, the +majority of stages operate on smaller segments of a full exposure, +allowing the processing tasks to be spread over the machines in the +processing cluster. The \ippprog{pantasks} environment, which manages +the jobs, attempts to target the processing to a computer which is +assigned to host data for the particular OTA. This capability is +implemented to reduce the network I/O load by minimizing the number of +operations done on non-local data. In practice, this targeted +processing has not had as large of an impact as was originally +intended: the data volume and operational details of the hardware has reduced the ability of any one node to reliably contain a particular OTA. The targeted processing has probably reduced the network load somewhat but it has not been as critical of a requirement as originally expected. - -\note{keep this paragraph?} - -Part of this parallelization is derived from the fact that this camera -consists of 60 independent orthogonal transfer array (OTA) devices, -and can therefore be processed simultaneously. Although there are -multiple stages that operate on an entire exposure at once, the -majority of stages operate only on smaller segments of a full exposure -to allow the processing tasks to be spread over the machines in the -processing cluster. %% In the \ippstage{chip} stage, @@ -581,5 +580,5 @@ this analysis, removing the need to write out and re-read the image data. The details of the detection and characterization of the -sources in the image are provided in \citet{magnier2017b}. +sources in the image are provided in \citet{magnier2017.analysis}. The results of the image processing are then written to disk, @@ -657,5 +656,5 @@ used to generate synthetic w-band photometry for areas where no PS1-based calibrated w-band photometry is available. For more -details, see \cite{magnier2017c}. The result of these calibrations is +details, see \cite{magnier2017.calibration}. The result of these calibrations is stored as a single multi-extension FITS table containing the results from each OTA as a separate extension. @@ -718,5 +717,5 @@ pixels. These projections are further broken down into ``skycells'' that form a $10\times{}10$ grid within the projection, with an overlap -region of 60" between adjacent skycells to ensure that objects are not +region of 60\arcsec\ between adjacent skycells to ensure that objects are not split on all images. @@ -778,5 +777,5 @@ For the PV3 processing of the Medium Deep fields, stacks have been generated for the nightly groups and for the full depth using all -exposures, producing ``deep stacks''. In addition, a 'best seeing' +exposures, producing ``deep stacks''. In addition, a `best seeing' set of stacks have been produced \note{using image quality cuts to be described}. We have also generated out-of-season stacks for the @@ -816,6 +815,4 @@ \ippstage{stack} processing. As this completes all processing for the entry, no \ippmisc{advance} job is required. - -% \note{end of section needed a re-read} \subsection{Stack Photometry} @@ -839,5 +836,5 @@ The input images are passed to the \ippprog{psphotStack} program, which does the analysis. The stack photometry algorithms are -described in detail in \cite{magnier2017b}. In short, sources are +described in detail in \cite{magnier2017.analysis}. In short, sources are detected in all 5 filter images down to the $5\sigma$ significance. The collection of detected sources is merged into a single master @@ -859,8 +856,7 @@ sources based on the PSF model; aperture like parameters such as the Petrosian flux and radius; the convolved galaxy model fits; and the -radial aperture measurements. \note{is this list complete?} Once the -photometry is complete, a row is added to the -\ippdbtable{staticskyResult} table with basic statistics from the -analysis. +radial aperture measurements. Once the photometry is complete, a row +is added to the \ippdbtable{staticskyResult} table with basic +statistics from the analysis. The stack photometry output catalogs are re-calibrated for both @@ -877,5 +873,5 @@ for the \ippstage{camera} and \ippstage{stack} calibration stages. Upon completion, the analysis statistics are written to the -\ippdbtable{skycalResult} table. \note{Any difference in output formats?} +\ippdbtable{skycalResult} table. \subsection{Forced Warp Photometry} @@ -951,20 +947,20 @@ In this program, the positions of sources are loaded from the input -catalog. PSF stars are pre-identified \note{how?} and a PSF model -generated for each \ippstage{warp} image based on those stars, using -the same stars for all warps to the extent possible (PSF stars which -are excessively masked on a particular image are not used to model the -PSF). The PSF model is fitted to all of the known source positions in -the warp images. Aperture magnitudes, Kron magnitudes, and moments -are also measured at this stage for each warp. Note that the flux -measurement for a faint, but significant, source from the stack image -may be at a low significance (less than the $5\sigma$ criterion used -when the photometry is not run in this forced mode) in any individual -warp image; the flux may even be negative for specific warps. When -combined together, these low-significance measurements will result in -a signficant measurement as the signal-to-noise increases by the -square root of the number of measurements. \note{The individual warp -measurements are combined together to generate averages values within -DVO.} +catalog. PSF stars are pre-identified from the stack image and a PSF +model generated for each \ippstage{warp} image based on those stars, +using the same stars for all warps to the extent possible (PSF stars +which are excessively masked on a particular image are not used to +model the PSF). The PSF model is fitted to all of the known source +positions in the warp images. Aperture magnitudes, Kron magnitudes, +and moments are also measured at this stage for each warp. Note that +the flux measurement for a faint, but significant, source from the +stack image may be at a low significance (less than the $5\sigma$ +criterion used when the photometry is not run in this forced mode) in +any individual warp image; the flux may even be negative for specific +warps. When combined together, these low-significance measurements +will result in a signficant measurement as the signal-to-noise +increases by the square root of the number of measurements. The +individual warp measurements are combined together to generate +averages values within DVO. Upon completion of the forced photometry (for point sources as well as @@ -976,11 +972,9 @@ analysis measurements into a single value. The output catalogs listed in the \ippdbtable{fullForceResult} table are passed to the -\ippprog{psphotFullForceSummary} to do this averaging. \note{describe - what is done} When this completes, an entry is added to the -\ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} entry -is marked as completed. +\ippprog{psphotFullForceSummary} to do this averaging. When this +completes, an entry is added to the \ippdbtable{fullForceSummary}, and +the \ippdbtable{fullForceRun} entry is marked as completed. \subsubsection{Forced Galaxy Models} -\note{CZW: is this the appropriate place for this section?} \note{too much detail in this section; balance relative to psphot} @@ -1020,19 +1014,19 @@ $\chi^2$ grid can then be made by combining each grid point across the inputs. The combined $\chi^2$ for a single grid point is simply the -sum of all $\chi^2$ values at that point. If, for a single \ippstage{warp} -image, the galaxy model is excessively masked, then that image will be -dropped for all grid points for that galaxy. The reduced $\chi^2$ -values can be determined by tracking the total number of pixels -used across all inputs to generate the combined $\chi^2$ values. From -the combined grid of $\chi^2$ values, the point in the grid with the -minimum $\chi^2$ is found. Quadratic interpolation is used to -determine the major, minor axis values for the interpolated minimum -$\chi^2$ value. The errors on these two parameters is then found by -determining the contour at which the \note{reduced?} $\chi^2$ -increases by 1. +sum of all $\chi^2$ values at that point. If, for a single +\ippstage{warp} image, the galaxy model is excessively masked, then +that image will be dropped for all grid points for that galaxy. The +reduced $\chi^2$ values can be determined by tracking the total number +of pixels used across all inputs to generate the combined $\chi^2$ +values. From the combined grid of $\chi^2$ values, the point in the +grid with the minimum $\chi^2$ is found. Quadratic interpolation is +used to determine the major, minor axis values for the interpolated +minimum $\chi^2$ value. The errors on these two parameters is then +found by determining the contour at which the $\chi^2$ increases by 1. Thus the \ippstage{fullforce} galaxy analysis uses the PSF information from each \ippstage{warp} to determine a best set of convovled galaxy models for each object in the \ippstage{skycal} catalog. + \note{discuss the subset of galaxy models and objects}. @@ -1110,5 +1104,5 @@ \begin{table}[hb] \begin{center} -\caption{DVO Database Tables\label{tab:DVOtables}} +\caption{DVO Database Tables\label{tab:DVO_schema}} \begin{tabular}{ll} \hline @@ -1238,5 +1232,5 @@ processed by the IPP may also be included similarly in a DVO database. Measurements from other sources, such as SDSS, 2MASS, or WISE, can -also be included in this table (see \S\ref{sec:other.photometry}. +also be included in this table. The \ippdbtable{Measure} table includes the instrumental magnitudes @@ -1278,4 +1272,5 @@ \subsubsubsection{Object Tables} +\label{sec:object} % object -> detection @@ -1353,5 +1348,5 @@ these photometric distance modulus measurements are not extremely precise (see below), they provide a constraint on the distance is used -in our analysis of the astrometry \citep[][see]{magnier2017a}. +in our analysis of the astrometry \citep[][see]{magnier2017.calibration}. In the \ippdbtable{Measure} table, there are three fields which @@ -1410,5 +1405,5 @@ determined by the photometry calibration analysis and the astrometric flat-field corrections determined by the astrometry calibration -analysis \citep[][see]{magnier2017a}. +analysis \citep[][see]{magnier2017.calibration}. \subsubsection{Sky Partition} @@ -1457,5 +1452,5 @@ machines that contain partition data. -\note{is the use of the term 'partition host' consistent in this paper +\note{is the use of the term `partition host' consistent in this paper and the calibration paper?} @@ -1499,5 +1494,5 @@ The FITS binary table compression scheme uses a strategy similar to that used for FITS image compression (\note{REF}). The binary tabular -data is compressed and stored in the 'HEAP' section of the FITS table +data is compressed and stored in the `HEAP' section of the FITS table extension, with pointers to the compressed data stored in the regular data section. Each column in the FITS table is compressed as one (or @@ -1505,5 +1500,5 @@ column format (e.g., TFORM1) are replaced with keywords which describe the location and size of the compressed data in the HEAP section; the -information about the uncompressed data is moved to a keyword with 'Z' +information about the uncompressed data is moved to a keyword with `Z' prepended (e.g., ZFORM1) and an additional field is added to define the compression algorithm (e.g., ZCTYP1). The column names (e.g., @@ -1594,5 +1589,5 @@ astrometric and photometric calibrations can be calculated. The details of the calibration analysis are discussed in -\cite{magnier2017c}. We present a brief summary here. +\cite{magnier2017.calibration}. We present a brief summary here. Astrometric calibration consists of measuring and correcting @@ -1607,5 +1602,5 @@ a function of position in the camera (essentially an astrometric flat-field correction), as a function of the brightness of the star -(the so-called Koppenh\"offer effect, see~\ref{magnier2017c}), and as +(the so-called Koppenh\"offer effect, see~\ref{magnier2017.calibration}), and as a function of airmass and color (Differential chromatic refraction). Once the systematic errors have been measured, they are applied back @@ -1626,5 +1621,5 @@ exposures which were believed to be obtained in photometric conditions. This process, called `\"ubercal', is described in detail -by \cite{ubercal} for the first (PV1) version. In brief, photometric +by \cite{2012ApJ...756..158S} for the first (PV1) version. In brief, photometric periods, with time-scales of at least \note{half of a night}, are identified by a combination of automatic analysis and manual @@ -1658,5 +1653,5 @@ flat-field correction addresses photometric variations due to spatial variations in the PSF due to the optics and low-level effects on the -chips \citep[see][]{magnier2017c}. After the systematic corrections +chips \citep[see][]{magnier2017.calibration}. After the systematic corrections have been determined and applied back to the database, a final relative photometry analysis pass is performed. @@ -1756,5 +1751,5 @@ Pantasks repeatedly checks each task in an attempt to generate a new -command: we say pantasks attempts to 'execute' the task in each of +command: we say pantasks attempts to `execute' the task in each of these attempts. Tasks may specify the time between execution attempts, with a 1 second default. @@ -1777,14 +1772,14 @@ Within the \ippprog{task.exec} macro, the command to be run must be -defined with the function 'command'. Once the \ippprog{task.exec} +defined with the function `command'. Once the \ippprog{task.exec} macro exits successfully, the defined command is the added to the list of jobs to be run within the UNIX environment. Jobs may be run in one of two ways: locally or via the parallel processing system. The task, or the -\ippprog{task.exec} macro, uses the 'host' command to define how to -run the job. If the host is set to 'local', then the job is run in +\ippprog{task.exec} macro, uses the `host' command to define how to +run the job. If the host is set to `local', then the job is run in the background by pantasks itself (using the C \code{execvp} function). Otherwise, the job is sent to the parallel processing system to be run on another machine within the cluster. If the host -is set to the special value 'anyhost', then the parallel processing +is set to the special value `anyhost', then the parallel processing system is allowed to choose the processing computer arbitrarily. Any other value is taken to be the DNS name of the computer on which this @@ -1798,5 +1793,5 @@ When the \ippprog{task.exec} macro is run, the code may choose (e.g., based on tests of some global variables) to exit the macro with an -error condition, e.g., with the 'break' command. In this +error condition, e.g., with the `break' command. In this circumstance, no job is produced by the task. The task will be tried again the next time it is executed. This feature allows for the user @@ -1813,18 +1808,18 @@ online user guide?} -The option 'npending' may be used to limit the number of jobs which +The option `npending' may be used to limit the number of jobs which are simultaneously executed for a specific task. For example, some classes of jobs should only be run one-at-a-time because they are not protected against collisions or they may overload a resource. The use -of 'npending' allows these situations to be handled cleanly within +of `npending' allows these situations to be handled cleanly within pantasks (avoiding cumbersome coding within with program or supporting script). -The option 'nmax' limits the total number of jobs which a task +The option `nmax' limits the total number of jobs which a task generates. This option may be useful in cases where \ippprog{pantasks} is used to perform a limited set of operations. \note{do we actually use this in IPP?} -The option 'trange' allows the user to restrict the time period during +The option `trange' allows the user to restrict the time period during which the specific tasks is executed. This option is given with a start and an end time for the limiting time range. These times may be @@ -1841,5 +1836,5 @@ ranges may be specified \note{how are they evaluated?} -The option \code{nice} specifies the 'nice' level at which the job is +The option \code{nice} specifies the `nice' level at which the job is run when it is executed. The parallel processing system must respect this concept. @@ -1918,5 +1913,5 @@ pantasks receives this completion information, the jobs are removed from the list managed by pcontrol. Thus pcontrol maintains at most a -modest list of jobs which are 'in flight', leaving all interpretation +modest list of jobs which are `in flight', leaving all interpretation work to pantasks. @@ -2461,5 +2456,5 @@ isolation of source objects is included, providing the organization of detections that is used in the \ippprog{psphot} photometry analysis -\citep{magnier2017c}. The PSF matching required for \ippstage{stack} +\citep{magnier2017.analysis}. The PSF matching required for \ippstage{stack} and \ippstage{diff} stage image combinations is as well. The unification of configuration options between config files on disk and @@ -2685,5 +2680,5 @@ column that link the tables together. -\note{logical or alphabetical sequence? alignment is broken} +\note{logical or alphabetical sequence?} \begin{center} @@ -2739,11 +2734,11 @@ \begin{verbatim} MAJOR TODO ITEMS: +* add figure showing DVO schema relationships * re-read and trim details as needed (referring to the other papers) * add some specific numbers (data volume, processing times, etc) +* where is the smf/cmf format defined? psphot? +* where is the GPC1 naming convention discussed? +* where are the flat-field seasons listed (magnier2017.calibration?) \end{verbatim} \end{document} - -Figures needed for this document: - -*