IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Ignore:
Timestamp:
Jul 5, 2017, 5:11:07 PM (9 years ago)
Author:
eugene
Message:

text updates from May

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/release.2015/ps1.datasystem/datasystem.tex

    r40065 r40071  
    202202reducing data from other cameras and telescopes.
    203203
     204\note{overview discussion of Pan-STARRS: the telescope, survey time
     205  period, surveys.  2 paragraphs.}
     206
     207The Pan-STARRS Image Processing Pipeline consists of a suite of
     208software programs and data systems that are designed to reduce
     209astronomical images, with a focus on parallelization necessary to
     210speed the processing of the large images produced by the GPC1 camera.
     211Part of this parallelization is derived from the fact that this camera
     212consists of 60 independent orthogonal transfer array (OTA) devices,
     213and can therefore be processed simultaneously.  Although there are
     214multiple stages that operate on an entire exposure at once, the
     215majority of stages operate only on smaller segments of a full exposure
     216to allow the processing tasks to be spread over the machines in the
     217processing cluster.
     218
     219
     220\note{fix this summary once outline is solidified}
     221
     222This paper presents a description of the IPP data handling system.
     223Section \ref{sec:subsystems} describes the major IPP subsystems that
     224underlie the main pipeline, providing a set of common interfaces and
     225tools used at multiple stages.  The main processing stages of the
     226pipeline are described in Section \ref{sec:stages}, although all
     227exposures may not necessarily pass through each of these stages.  The
     228hardware systems that have done the processing for the PV3 data
     229release are listed in Section \ref{sec:hardware}, with some details
     230on the scale of computing needed to reduce this large number of
     231exposures.  Finally, Section \ref{sec:discussion} presents a
     232discussion of some of the lessons learned in the creation of the IPP,
     233and its utility in reducing data from other cameras and telescopes.
     234
    204235{\color{red} {\em Note: These papers are being placed on arXiv.org to
    205236    provide crucial support information at the time of the public
    … …  
    213244\label{sec:overview}
    214245
    215 \subsection{Elements of the Pan-STARRS Data Processing System}
    216 
    217 The Pan-STARRS Data Analysis system contains many features to support
    218 a wide range of activities: archiving and management of the raw and
    219 processed image files; real-time nightly processing of images for
    220 transient and moving object science; large-scale re-processing and
     246The Pan-STARRS Data Analysis system consists of many elements to
     247support the wide range of activities: archiving and management of the
     248raw and processed image files; real-time nightly processing of images
     249for transient and moving object science; large-scale re-processing and
    221250calibration to produce measurements for the science collaboration and
    222 the wider public; specialized image processing to facilitate research
    223 and development of the analysis system itself; and distribution of the
    224 resulting data products to various consumers in a variety of formats
    225 and modes.
     251the wider public; specialized image processing tasks to facilitate
     252research and development of the analysis system itself; distribution
     253of the resulting data products to various consumers in a variety of
     254formats and modes.
    226255
    227256The Pan-STARRS Data Analysis system is divided internally into several major
    228257components:
    229258\begin{itemize}
    230 \item Summit : both the camera and observatory summit systems perform
     259\item Summit Processing : both the camera and observatory summit systems perform
    231260  data analysis tasks needed to support the on-going observations.
    232261  In this article, we focus only on those aspects used by the off-summit
    233   analysis stages.
     262  analysis stages.  \note{is summit processing discussed anywhere?}
    234263\item Image Processing Pipeline (IPP) : this portion of the data
    235264  analysis system takes the data from raw pixels on the summit
    … …  
    244273\end{itemize}
    245274The above set of analysis stages take place at the IfA within the
    246 scope of responsibility of the Pan-STARRS Observatory.  Within the
     275scope of responsibility of the Pan-STARRS Observatory.  Across the
    247276wider Pan-STARRS colloboration(s), additional data analysis operations
    248277are performed to support science results.  These collaboration-wide
    249278analysis operations range from those which are tightly-coupled to the
    250279Pan-STARRS Observatory system, such as the analysis of the transient
    251 discovery teams and the public archive database at MAST, to those
    252 which perform offline analysis for eventual ingest back into the
    253 Pan-STARRS databases and archive.  The latter category includes the
    254 ubercal photometric analysis, the photo-z analysis, and the QSO / RR
    255 Lyra search efforts.  In addition, collaborations within the wider
     280search teams and the public archive database at MAST, to those which
     281perform offline analysis for eventual ingest back into the Pan-STARRS
     282databases and archive.  The latter category includes the ubercal
     283photometric analysis \citep{ubercal}, the photo-z analysis
     284\citep{photoz}, and the QSO / RR Lyra search efforts
     285\citep{hernitschek2016}.  In addition, collaborations within the wider
    256286Pan-STARRS community have implemented a variety of science-level
    257 analyses of their own to support their science goals (e.g., M31
    258 Cepheid search).  This article discusses the analysis elements which
    259 take place at the IfA except as noted.
     287analyses of their own to support their science goals \citep[e.g., M31
     288  variable search][]{M31.REF}.
    260289
    261290Figure~\ref{fig:analysis.elements} illustrates the many elements of
    … …  
    266295the summit systems are described by \note{REF?}.
    267296
    268 \begin{figure*}[htbp]
    269   \begin{center}
    270  \includegraphics[width=\hsize,clip]{PS1_Data_Analysis_System_Overview.pdf}
    271   \caption{\label{fig:analysis.elements} Elements of the Pan-STARRS\,1
    272     Data Analysis System.  Rectangles represent data analysis steps;
    273     ellipses represent databases; rounded rectangles represent
    274     external groups (``customers'').  The arrows show a simplified representation
    275   of the major flow of data between the analysis stages and data
    276   processing elements.}
    277   \end{center}
    278 \end{figure*}
    279 
    280 \subsection{Nightly Processing Analysis Stages}
    281 
    282 Data analysis to support nighly science operations is driven by two
     297Data analysis to support nightly science operations is driven by two
    283298main goals: 1) rapid detection of the moving and transient sources to
    284299enable recovery or follow-up with other telescopes. 2) regular
    … …  
    289304detail below.  In short, each image is processed independently to
    290305correct for instrumental signatures and to detect the astronomical
    291 sources (chip); astrometric and photometric calibrations are
    292 determined (camera), and finally images are geometric transformed to a
    293 common pixel representation (warp).  Warped images may either be added
    294 together (stack) or used in an image subtraction (diff).  As part of nightly
    295 science processing, images for certain fields such as the Medium Deep
    296 survey fields (see \cite{}), are stacked together in nightly chunks,
    297 providing deeper detection capability on short timescales.  Depending
    298 on the survey mode, difference images are generated for the nightly
    299 stack images (vs a deep stack template) or for individual warp images.
    300 In the later case, the warp images may be difference against another
    301 warp from the same night or against a reference stack from the
    302 appropriate part of the sky.
    303 
    304 \subsection{Re-processing Analysis Stages}
     306sources (\IPPstage{chip}); astrometric and photometric calibrations
     307are determined (\IPPstage{camera}), and finally images are
     308geometrically transformed to a common pixel representation
     309(\IPPstage{warp}).  Warped images may either be added together
     310(\IPPstage{stack}) or used in an image subtraction (\IPPstage{diff}).
     311For nightly science operations, images for certain fields such as the
     312Medium Deep survey fields \citep[see][]{MDref}, are stacked together
     313in nightly chunks, providing deeper detection capability on 1-day
     314timescales.  Depending on the survey mode, difference images are
     315generated for the nightly stack images (vs a deep stack template) or
     316for individual warp images.  In the later case, the warp images may be
     317difference against another warp from the same night or against a
     318reference stack from the appropriate part of the sky.
    305319
    306320Pan-STARRS has performed several large-scale reprocessings of both the
    307 Medium Deep and $3\pi$ Survey data.  For the $3\pi$ Survey data, we identify
    308 these large-scale reprocessings as PV1, PV2, and PV3 (we also define
    309 the nightly science analysis of the data as PV0).  For these
    310 reprocessing stages, the standard steps of chip through warp, plus
    311 stack and diff are performed, starting from raw data, using a single
    312 homogenous version of the data analysis procedures.  (PV2 was a
    313 special case in which we started from the camera level products of
    314 PV1).  In addition to the analysis stages which are common with the
    315 nightly processing, these large-scale reprocessing stages include
    316 additional processing: a more detailed photometric analysis is
    317 performed on the stacks, including morphological analysis appropriate
    318 to galaxies.  The results of the stack photometry analysis are used to
    319 drive a forced-photometry analysis of the warp images.  The data
    320 products from the camera, stack photometry, and forced-warp photometry
    321 analysis stages are ingested into the internal calibration database
    322 (DVO, the Desktop Virtual Observatory) and used for photometric and
    323 astrometric calibrations (see Section~\ref{sec:DVO})
     321Medium Deep and 3pi Survey data for internal consumption.  For the 3pi
     322Survey data, we identify these large-scale reprocessings as PV1, PV2,
     323and PV3, with PV3 the analysis used for the first public data release,
     324DR1.  We also refer to the nightly science analysis of the data as
     325PV0.  For these reprocessing stages, the standard steps of chip
     326through warp, plus stack and diff are performed, starting from raw
     327data, usually using a single homogenous version of the data analysis
     328procedures.  PV2 was a special case in which we started from the
     329camera level products of PV1 to speed up the turn-around to the
     330community.  In addition to the analysis stages listed above which are
     331shared with the nightly processing, these large-scale reprocessing
     332analyses include additional processing.  A more detailed photometric
     333analysis is performed on the stacks, including morphological analysis
     334appropriate to galaxies.  The results of the stack photometry analysis
     335are used to drive a forced-photometry analysis of the warp images.
     336The data products from the camera, stack photometry, and forced-warp
     337photometry analysis stages are ingested into the internal calibration
     338database (DVO, the Desktop Virtual Observatory) and used for
     339photometric and astrometric calibrations.
    324340
    325341\subsection{Data Access and Distribution}
    … …  
    347363\label{sec:processing.database}
    348364
     365\begin{table*}
     366\caption{\label{tab:database_schema} GPC1 Database Schema Outline}\vspace{-0.5cm}
     367\begin{center}
     368\begin{tabular}{lllll}
     369\hline
     370\hline
     371{\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} & {\bf Notes} \\
     372\hline
     373  \ippstage{addstar}      & \ippdbtable{addRun}       & \ippdbtable{addProcessedExp}     & \ippdbcolumn{add_id} & \\
     374  \ippstage{camera}       & \ippdbtable{camRun}       & \ippdbtable{camProcessedExp}     & \ippdbcolumn{cam_id} & \\
     375  \ippstage{chip}         & \ippdbtable{chipRun}      & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} & \\
     376  \ippstage{detrend}      & \ippdbtable{detRun}       & \ippdbtable{detRunSummary}       & \ippdbcolumn{det_id} & \\
     377                          &                           & \ippdbtable{detInputExp}         & & \\
     378                          &                           & \ippdbtable{detRegisteredImfile} & & Information about detrends produced externally.\\
     379                          &                           & \ippdbtable{detStackedImfile}    & & \\
     380                          & \ippdbtable{detProcessedExp} & \ippdbtable{detProcessedImfile}  & & \\
     381                          & \ippdbtable{detResidExp}  & \ippdbtable{detResidImfile}      & & \\
     382                          & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & & \\
     383  \ippstage{diff}         & \ippdbtable{diffRun}      & \ippdbtable{diffSkyfile}         & \ippdbcolumn{diff_id} & \\
     384                          &                           & \ippdbtable{diffInputSkyfile}    & & \\
     385  \ippstage{distribution} & \ippdbtable{distRun}      & \ippdbtable{distComponent}       & \ippdbcolumn{dist_id} & \\
     386                          &                           & \ippdbtable{distTarget}          & & \\
     387  \ippstage{fake}         & \ippdbtable{fakeRun}      & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} & \\
     388  \ippstage{fullforce}    & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput}      & \ippdbcolumn{ff_id} & \\
     389                          &                           & \ippdbtable{fullForceResult}     & & \\
     390                          &                           & \ippdbtable{fullForceSummary}    & & Properties about average parameters from all results.\\
     391  \ippstage{lap}          & \ippdbtable{lapSequence}  & \ippdbtable{lapRun}              & \ippdbcolumn{seq_id} & Sequence of full reprocessing\\
     392                          & \ippdbtable{lapRun}       & \ippdbtable{lapExp}              & \ippdbcolumn{lap_id} & \\
     393  \ippstage{publish}      & \ippdbtable{publishRun}   & \ippdbtable{publishDone}         & \ippdbcolumn{pub_id} & \\
     394                          &                           & \ippdbtable{publishClient}       & & \\
     395  \ippstage{summitcopy}   & \ippdbtable{pzDataStore}  &                                  & & Lists locations to check for new exposures.\\
     396                          & \ippdbtable{summitExp}    & \ippdbtable{summitImfile}        & \ippdbcolumn{summit_id} & Exposures available at the telescope.\\
     397                          & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile}    & & Exposures that are being downloaded.\\
     398                          & \ippdbtable{newExp}       & \ippdbtable{newImfile}           & \ippdbcolumn{exp_id} & Exposures that have been saved to IPP cluster.\\
     399
     400  \ippstage{registration} & \ippdbtable{rawExp}       & \ippdbtable{rawImfile}           & \ippdbcolumn{exp_id} & \\
     401  \ippstage{remote}       & \ippdbtable{remoteRun}    & \ippdbtable{remoteComponent}     & \ippdbcolumn{remote_id} & \\
     402  \ippstage{skycal}       & \ippdbtable{skycalRun}    & \ippdbtable{skycalResult}        & \ippdbcolumn{skycal_id} & \\
     403  \ippstage{stack}        & \ippdbtable{stackRun}     & \ippdbtable{stackInputSkyfile}   & \ippdbcolumn{stack_id} & \\
     404                          &                           & \ippdbtable{stackSumSkyfile}     & & \\
     405  \ippstage{staticsky}    & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput}      & \ippdbcolumn{sky_id} & \\
     406                          &                           & \ippdbtable{staticskyResult}     & & \\
     407  \ippstage{warp}         & \ippdbtable{warpRun}      & \ippdbtable{warpImfile}          & \ippdbcolumn{warp_id} & \\
     408                          &                           & \ippdbtable{warpSkyCellMap}      & & Mapping of input chips to projection skycells.\\
     409                          &                           & \ippdbtable{warpSkyfile}         & & \\
     410\hline
     411\end{tabular}
     412\end{center}
     413\end{table*}
     414
    349415A critical element in the Pan-STARRS IPP infrastructure is the
    350416processing database.  This database, using the mysql database engine,
    … …  
    361427database, since a single instance of the database is used to track the
    362428processing of images and data products related to the PS1 GPC1 camera.
    363 This same database engine also has instances for other cameras
    364 processed by the IPP, e.g., GPC2, the test cameras TC1, TC3, and the
    365 Imaging Sky Probe (ISP).  In general, processing information for
    366 different cameras is separate in differnt processing database; merging
    367 of output products takes place in DVO.
     429This same database engine also has instances (same schema, different
     430data) for other cameras processed by the IPP, e.g., GPC2, the test
     431cameras TC1, TC3, and the Imaging Sky Probe (ISP).
    368432
    369433Within the processing database, the various processing stages are
    … …  
    681745table.
    682746
    683 \subsection{Fake Analysis}
    684 \label{sec:fake}
    685 % \note{drop}
    686 
    687 The \ippstage{fake} stage was originally designed to do false source
    688 injection and recovery, in order to determine the detection efficiency
    689 of sources on the exposure.  However, early in the design of the IPP,
    690 this task was moved to the rest of the photometry analysis done at the
    691 \ippstage{chip} stage.  Removing the stage would require significant
    692 changes to the database schema.  As a result, this conveniently named
    693 stage generally does no actual data processing, and consists mainly of
    694 database operations to move the exposure on to the \ippstage{warp}
    695 stage.  The operations mimic the \ippstage{chip} stage, with
    696 individual jobs run for each OTA that update rows in the
    697 \ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that
    698 updates the \ippdbtable{fakeRun} table and promotes the exposure to
    699 the next stage by adding a row to the \ippdbtable{warpRun} table.
     747%% \subsection{Fake Analysis}
     748%% \label{sec:fake}
     749%%
     750%% The \ippstage{fake} stage was originally designed to do false source
     751%% injection and recovery, in order to determine the detection efficiency
     752%% of sources on the exposure.  However, early in the design of the IPP,
     753%% this task was moved to the rest of the photometry analysis done at the
     754%% \ippstage{chip} stage.  Removing the stage would require significant
     755%% changes to the database schema.  As a result, this conveniently named
     756%% stage generally does no actual data processing, and consists mainly of
     757%% database operations to move the exposure on to the \ippstage{warp}
     758%% stage.  The operations mimic the \ippstage{chip} stage, with
     759%% individual jobs run for each OTA that update rows in the
     760%% \ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that
     761%% updates the \ippdbtable{fakeRun} table and promotes the exposure to
     762%% the next stage by adding a row to the \ippdbtable{warpRun} table.
    700763
    701764\subsection{Image Warping}
    … …  
    779842exposures, producing ``deep stacks''.  In addition, a `best seeing'
    780843set of stacks have been produced \note{using image quality cuts to be
    781   described}.  We have also generated out-of-season stacks for the
    782 Medium Deep fields, in which all image not from a particular observing
    783 season for a field are combined into a stack.  These later stacks are
    784 useful as deep templates when studying long-term transient events in
    785 the Medium Deep fields as they are not (or less) contaminated by the
    786 flux of the transients from a given season.
     844  described: need input from MEH}.  We have also generated
     845out-of-season stacks for the Medium Deep fields, in which all image
     846not from a particular observing season for a field are combined into a
     847stack.  These later stacks are useful as deep templates when studying
     848long-term transient events in the Medium Deep fields as they are not
     849(or less) contaminated by the flux of the transients from a given
     850season.
    787851
    788852When a given set of \ippstage{stack} stage are defined, exposures with
    … …  
    823887deferred to the \ippstage{staticsky} stage.  This separation is
    824888maintained because the photometry analysis of the \ippstage{stack}
    825 images, including convolved galaxy model fitting, is performed on all
    826 5 filters simultaneously.  By deferring this analysis, the processing
    827 system may also decouple the generation of the pixels from the source
    828 detection.  This makes the sequencing of analysis somewhat easier and
    829 less subject to blocks due to a failure in the stacking analysis.
    830 Similar to the \ippstage{stack} stage, an entry is created in the
    831 \ippdbtable{staticskyRun} table, linked to a series of rows in the
    832 \ippdbtable{staticskyInput} table by a common \ippdbcolumn{sky_id},
    833 each of which also contains the appropriate \ippdbcolumn{stack_id}
    834 entries for the skycell under consideration.
     889images is performed on all 5 filters simultaneously.  By deferring
     890this analysis, the processing system may also decouple the generation
     891of the pixels from the source detection.  This makes the sequencing of
     892analysis somewhat easier and less subject to blocks due to a failure
     893in the stacking analysis.  Similar to the \ippstage{stack} stage, an
     894entry is created in the \ippdbtable{staticskyRun} table, linked to a
     895series of rows in the \ippdbtable{staticskyInput} table by a common
     896\ippdbcolumn{sky_id}, each of which also contains the appropriate
     897\ippdbcolumn{stack_id} entries for the skycell under consideration.
    835898
    836899The input images are passed to the \ippprog{psphotStack} program,
    … …  
    853916The stack photometry output files consist of a set of FITS table
    854917catalogs, with one file for each filter.  Within these files, there
    855 are multiple table extensions that include: the measurements of
    856 sources based on the PSF model; aperture like parameters such as the
    857 Petrosian flux and radius; the convolved galaxy model fits; and the
    858 radial aperture measurements.  Once the photometry is complete, a row
    859 is added to the \ippdbtable{staticskyResult} table with basic
    860 statistics from the analysis.
     918are multiple table extensions, with different classes of measurements
     919saved in the different extensions.  The extensions include a table of
     920the measurements of sources based on the PSF model; a table of
     921aperture-like parameters such as the Petrosian flux and radius; a
     922table of the convolved galaxy model fits; and a table of the radial
     923aperture measurements.  Once the photometry is complete, a row is
     924added to the \ippdbtable{staticskyResult} table with basic statistics
     925from the analysis.
    861926
    862927The stack photometry output catalogs are re-calibrated for both
    … …  
    865930\ippstage{skycal} stage, each skycell is processed independently.
    866931Because of this independence, when queued for processing, the entries
    867 in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky_id}
     932in the \ippdbtable{skycalRun} table contain the \IPPdbcolumn{sky_id}
    868933and \ippdbcolumn{stack_id} entries of the parent data directly.  As
    869934in the \ippstage{camera} stage, the \ippprog{psastro} program reads in
    870 the stack photometry catalog, and produces a calibrated output.  A
    871 different processing recipe is supplied to \ippprog{psastro}, which
    872 controls for the different data.  The same reference catalog is used
    873 for the \ippstage{camera} and \ippstage{stack} calibration stages.
    874 Upon completion, the analysis statistics are written to the
    875 \ippdbtable{skycalResult} table.
     935the stack photometry catalog, and produces a calibrated output, with
     936format matching the input.  A different processing recipe is supplied
     937to \ippprog{psastro}, which controls for the different data.  The same
     938reference catalog is used for the \ippstage{camera} and
     939\ippstage{stack} calibration stages.  Upon completion, the analysis
     940statistics are written to the \ippdbtable{skycalResult} table.
    876941
    877942\subsection{Forced Warp Photometry}
    … …  
    930995individual warp images used to generate the stack.  This
    931996\ippstage{fullforce} analysis is performed on all warps for a single
    932 skycell and filter as a single unit within the processing database,
    933 while individual warps are processed individually in parallel as
    934 separate processing jobs. 
    935 
    936 When processing is queued for this stage, an entry is added to the
    937 \ippdbtable{fullForceRun} primary database table with a reference to
    938 the corresponding stack and \ippdbcolumn{skycal_id} entry that is the
    939 input source of detections to be measured.  The \ippdbcolumn{warp_id}
    940 values for the input \ippstage{warp} stage images that contributed to
    941 the \ippstage{stack} associated with that \ippdbcolumn{skycal_id} are
     997skycell and filter as a single unit, as this matches the arrangement
     998of the input source catalog from the \ippstage{skycal} stage.  When
     999processing is queued for this stage, an entry is added to the
     1000\ippdbtable{fullForceRun} primary database table linking to the
     1001specific \ippdbcolumn{skycal_id} entry that will be used as the
     1002catalog for the photometry.  The \ippdbcolumn{warp_id} values for the
     1003input \ippstage{warp} stage images that contributed to the
     1004\ippstage{stack} associated with that \ippdbcolumn{skycal_id} are
    9421005then added to the \ippdbtable{fullForceInput} table, linked to the
    9431006primary table by the \ippdbcolumn{ff_id} identifier.  The individual
    … …  
    9451008stage image products along with the \ippstage{skycal} catalog to the
    9461009\ippprog{psphotFullForce} program.
    947 
    948 In this program, the positions of sources are loaded from the input
    949 catalog.  PSF stars are pre-identified from the stack image and a PSF
    950 model generated for each \ippstage{warp} image based on those stars,
    951 using the same stars for all warps to the extent possible (PSF stars
    952 which are excessively masked on a particular image are not used to
    953 model the PSF).  The PSF model is fitted to all of the known source
    954 positions in the warp images.  Aperture magnitudes, Kron magnitudes,
    955 and moments are also measured at this stage for each warp.  Note that
    956 the flux measurement for a faint, but significant, source from the
    957 stack image may be at a low significance (less than the $5\sigma$
    958 criterion used when the photometry is not run in this forced mode) in
    959 any individual warp image; the flux may even be negative for specific
    960 warps.  When combined together, these low-significance measurements
    961 will result in a signficant measurement as the signal-to-noise
    962 increases by the square root of the number of measurements.  The
    963 individual warp measurements are combined together to generate
    964 averages values within DVO.
    965 
    966 Upon completion of the forced photometry (for point sources as well as
    967 galaxies, discussed below), an entry is added to the
    968 \ippdbtable{fullForceResult} table with the processing statistics for
    969 that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}.
    970 Once all of the entries in the \ippdbtable{fullForceInput} table have
    971 finished, a summary operation is run to combine the galaxy photometry
    972 analysis measurements into a single value.  The output catalogs listed
    973 in the \ippdbtable{fullForceResult} table are passed to the
    974 \ippprog{psphotFullForceSummary} to do this averaging.  When this
    975 completes, an entry is added to the \ippdbtable{fullForceSummary}, and
    976 the \ippdbtable{fullForceRun} entry is marked as completed.
    977 
    978 \subsubsection{Forced Galaxy Models}
    979 \note{too much detail in this section; balance relative to psphot}
    9801010
    9811011The convolved galaxy models are also re-measured on the
    … …  
    9891019the PSF-convolved galaxy models are of limited accuracy.
    9901020
    991 In the \ippstage{fullforce} galaxy model analysis, we assume that the
    992 galaxy position and position angle, along with the Sersic index if
    993 appropriate, have been sufficiently well determined in the
    994 \ippstage{staticsky} analysis.  In this case, the goal is to determine
    995 the best values for the major and minor axis of the elliptical contour
    996 and at the same time the best normalization corresponding to the best
    997 elliptical shape, and thus the best galaxy magnitude value.
    998 
    999 For each \ippstage{warp} image, the \ippstage{staticsky} value for the
    1000 major and minor axis are used as the center of a $7\times{} 7$ grid
    1001 search of the major and minor axis parameter values.  The grid spacing
    1002 is defined as a function of the signal-to-noise of the galaxy in the
    1003 stack image so that bright galaxies are measured with a much finer
    1004 grid spacing that faint galaxies \note{need to quantify this}.  For
    1005 each grid point, the major and minor axis values at that point are
    1006 determined for the model.  The model is then generated and convolved
    1007 with the PSF model for the \ippstage{warp} image at that point.  The
    1008 resulting model is then compared to the \ippstage{warp} pixel data
    1009 values and the best fit normalization value is defined.  The
    1010 normalization and the $\chi^2$ value for each grid point is recorded.
    1011 
    1012 For a given galaxy, the result is a collection of $\chi^2$ values for
    1013 each of the grid points spanning all \ippstage{warp} images.  A single
    1014 $\chi^2$ grid can then be made by combining each grid point across the
    1015 inputs.  The combined $\chi^2$ for a single grid point is simply the
    1016 sum of all $\chi^2$ values at that point.  If, for a single
    1017 \ippstage{warp} image, the galaxy model is excessively masked, then
    1018 that image will be dropped for all grid points for that galaxy.  The
    1019 reduced $\chi^2$ values can be determined by tracking the total number
    1020 of pixels used across all inputs to generate the combined $\chi^2$
    1021 values.  From the combined grid of $\chi^2$ values, the point in the
    1022 grid with the minimum $\chi^2$ is found.  Quadratic interpolation is
    1023 used to determine the major, minor axis values for the interpolated
    1024 minimum $\chi^2$ value.  The errors on these two parameters is then
    1025 found by determining the contour at which the $\chi^2$ increases by 1.
    1026 
    1027 Thus the \ippstage{fullforce} galaxy analysis uses the PSF information
    1028 from each \ippstage{warp} to determine a best set of convovled galaxy
    1029 models for each object in the \ippstage{skycal} catalog.
    1030 
    1031 \note{discuss the subset of galaxy models and objects}.
     1021Upon completion of the forced photometry (for point sources as well as
     1022galaxies, discussed below), an entry is added to the
     1023\ippdbtable{fullForceResult} table with the processing statistics for
     1024that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}.
     1025The individual warp measurements are combined together to produce an
     1026average warp photometry value for each object within the context of
     1027the DVO object database system, including re-calibration of each warp
     1028based on the tie to the average photometry of the objects measured in
     1029the \ippstage{camera} stage.
     1030
     1031Once all of the entries in the \ippdbtable{fullForceInput} table have
     1032finished, a summary operation is run to combine the galaxy photometry
     1033analysis measurements into a single value.  The output catalogs listed
     1034in the \ippdbtable{fullForceResult} table are passed to the
     1035\ippprog{psphotFullForceSummary} to do this averaging.  When this
     1036completes, an entry is added to the \ippdbtable{fullForceSummary}, and
     1037the \ippdbtable{fullForceRun} entry is marked as completed.
    10321038
    10331039\subsection{Difference Images}
    10341040\label{sec:diff}
     1041
    10351042Two of the primary science drivers for the Pan-STARRS system are the
    10361043search hazardous asteroids and the search for Type Ia supernovae to
    … …  
    10521059or from a pair of \ippstage{stack} stage images.  During the PS1
    10531060survey, pairs of exposures, call TTI pairs (see~\note{Survey
    1054   Strategy}), were obtained for each pointing within a $\approx$ 1
     1061  Strategy in Chambers et al}), were obtained for each pointing within a $\approx$ 1
    10551062hour period in the same filter, and to the extent possible with the
    10561063same orientation and boresite position.  The standard PS1 nightly
    … …  
    11861193system. 
    11871194
    1188 There are 3 classes of photcodes defined within the DVO system.  One
    1189 class of photcodes define the filter systems for the average
    1190 photometry measurements; these are called \ippmisc{SEC} photcodes.  A
    1191 second class of photcode is associated with measurements from a
    1192 specific camera for which image metadata is available are called
    1193 \ippmisc{DEP} photcodes.  There are also those measurements which come
    1194 from external data sources for which DVO does not have any information
    1195 to determine a calibration (e.g., instrumental magnitudes and detector
    1196 coordinates).  These are measurements are reference values and are
    1197 assigned \ippmisc{REF} photcodes.
     1195DVO includes two major classes of database tables: those containing
     1196information about astronomical objects in the sky and those containing
     1197other supporting information.  The object-related tables are
     1198partitioned on the basis of position in the sky: objects within a
     1199region bounded by lines of constant RA,DEC are contained in a specific
     1200file.  The boundaries and the associated partition names are stored in
     1201one of the supporting tables, \ippdbtable{SkyTable}.  This table
     1202contains the definitions of the boundaries for each sky region
     1203(\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN},
     1204\ippdbcolumn{D_MAX}), the name of the sky region, an ID
     1205(\ippdbcolumn{INDEX}, equal to the sequence number of the region in
     1206the table), and index entries to enable navigation within the table.
     1207The regions are defined in a hierarchical sense, with a series of
     1208levels each containing a finer mesh of regions covering the sky.
    11981209
    11991210The names for \ippmisc{SEC} photcodes are the names of filter systems,
    … …  
    15671578appropriate database table with the \ippdbcolumn{stage_id} field.  As
    15681579some stages, such as the \ippstage{diff} stage, create more than a
    1569 single catalog for a single exposure, multiple entries with the
    1570 \ippdbcolumn{stage_id} are created, with the
    1571 \ippdbcolumn{stage_extra1} field containing an index to the individual
    1572 components.  The catalog specified by the entry is added to the target
    1573 \ippmisc{minidvo} by the \ippprog{addstar} program, with object
    1574 constructed as described above (\S~\ref{sec:object}).  When this
     1580single catalog, multiple entries with the \ippdbcolumn{stage_id} are
     1581created, with the \ippdbcolumn{stage_extra1} field containing an
     1582index to the individual components.  The catalog specified by the
     1583entry is added to the target \ippmisc{minidvo} by the
     1584\ippprog{addstar} program, \note{describe what's done?}.  When this
    15751585completes, an entry containing the statistics of the job is added to
    15761586the \ippdbtable{addProcessedExp} table.
    … …  
    25662576values used for the various IPP processing stages.
    25672577
    2568 \begin{deluxetable}{lcc}
    2569   \tablecolumns{3}
    2570   \tablewidth{0pc}
    2571   \tablecaption{Cost values for remote processing}
    2572   \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}}
    2573   \startdata
     2578\begin{table}
     2579\caption{\label{tab:SC_processing_parameters} Cost values for remote processing}\vspace{-0.5cm}
     2580\begin{center}
     2581\begin{tabular}{lcc}
     2582\hline
     2583\hline
     2584{\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\
     2585\hline
    25742586  \ippstage{chip} & 150 & 2 \\
    25752587  \ippstage{camera} & 1700 & 2 \\
    … …  
    25782590  \ippstage{staticsky} & 7200 & 6 \\
    25792591%  \ippstage{diff} & 300 & 2 \\
    2580   \ippstage{fullforce} & 300 & 2
    2581   \enddata
    2582   \label{tab:SC processing parameters}
    2583 \end{deluxetable}
     2592  \ippstage{fullforce} & 300 & 2 \\
     2593\hline
     2594\end{tabular}
     2595\end{center}
     2596\end{table}
     2597
     2598%% \begin{deluxetable}{lcc}
     2599%%   \tablecolumns{3}
     2600%%   \tablewidth{0pc}
     2601%%   \tablecaption{Cost values for remote processing}
     2602%%   \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}}
     2603%%   \startdata
     2604%%   \ippstage{chip} & 150 & 2 \\
     2605%%   \ippstage{camera} & 1700 & 2 \\
     2606%%   \ippstage{warp} & 110 & 2 \\
     2607%%   \ippstage{stack} & 1500 & 6 \\
     2608%%   \ippstage{staticsky} & 7200 & 6 \\
     2609%% %  \ippstage{diff} & 300 & 2 \\
     2610%%   \ippstage{fullforce} & 300 & 2
     2611%%   \enddata
     2612%%   \label{tab:SC processing parameters}
     2613%% \end{deluxetable}
    25842614
    25852615Once the preparation for the job is complete, the input and output
    … …  
    26822712\note{logical or alphabetical sequence?}
    26832713
     2714\end{document}
     2715
     2716Figures needed for this document:
     2717
     2718*
    26842719\begin{center}
    26852720\begin{deluxetable}{lllll}
    … …  
    27302765\end{deluxetable}
    27312766\end{center}
    2732  
    2733 
    2734 \begin{verbatim}
    2735 MAJOR TODO ITEMS:
    2736 * add figure showing DVO schema relationships
    2737 * re-read and trim details as needed (referring to the other papers)
    2738 * add some specific numbers (data volume, processing times, etc)
    2739 * where is the smf/cmf format defined?  psphot?
    2740 * where is the GPC1 naming convention discussed?
    2741 * where are the flat-field seasons listed (magnier2017.calibration?)
    2742 \end{verbatim}
    2743 
    2744 \end{document}
     2767
Note: See TracChangeset for help on using the changeset viewer.