Changeset 40559 for trunk/doc/release.2015/ps1.datasystem/datasystem.tex
- Timestamp:
- Nov 18, 2018, 10:38:23 AM (8 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/release.2015/ps1.datasystem/datasystem.tex
r40298 r40559 1 1 % \documentclass[iop,floatfix]{emulateapj} 2 2 % \documentclass[iop,floatfix,onecolumn]{emulateapj} 3 \documentclass[12pt,preprint]{aastex}4 %\documentclass[10pt,preprint]{aastex}3 % \documentclass[12pt,preprint]{aastex} 4 \documentclass[10pt,preprint]{aastex} 5 5 % \pdfoutput=1 6 6 7 %\RequirePackage{deluxetable} -- included by aastex? 8 \RequirePackage{nsfprop} 7 9 \RequirePackage{color} 8 10 \RequirePackage{code} … … 93 95 \label{sec:intro} 94 96 95 \note{missing figures: analysis elements, DVO schema}96 97 97 The 1.8m Pan-STARRS\,1 telescope is located on the summit of Haleakala 98 98 on the Hawaiian island of Maui. The wide-field optical design of the … … 185 185 \citet[][Paper VII]{huber2017} 186 186 %Huber et al. 2017 (Paper VII) 187 describes the Medium Deep Survey in detail, including the unique issues and data products specific to that survey. The Medium Deep Survey is not part of Data Release 1 . (DR1)187 describes the Medium Deep Survey in detail, including the unique issues and data products specific to that survey. The Medium Deep Survey is not part of Data Release 1 or 2. 188 188 189 189 Section~\ref{sec:overview} provides an overview of the full data … … 204 204 reducing data from other cameras and telescopes. 205 205 206 {\color{red} {\em Note: These papers are being placed on arXiv.org to207 provide crucial support information at the time of the public208 release of Data Release 1 (DR1). We expect the arXiv versions to209 be updated prior to submission to the Astrophysical Journal in210 January 2017. Feedback and suggestions for additional information211 from early users of the data products are welcome during the212 submission and refereeing process.}}206 %% {\color{red} {\em Note: These papers are being placed on arXiv.org to 207 %% provide crucial support information at the time of the public 208 %% release of Data Release 1 (DR1). We expect the arXiv versions to 209 %% be updated prior to submission to the Astrophysical Journal in 210 %% January 2017. Feedback and suggestions for additional information 211 %% from early users of the data products are welcome during the 212 %% submission and refereeing process.}} 213 213 214 214 \section{Overview of Pan-STARRS Data Processing} … … 243 243 \item PSPS : this system ingests the calibrated measurements from the 244 244 IPP, MOPS, and others and generates a high-availability database 245 with web-based interactions for public consumption. 245 with web-based interactions for public consumption \citet[][]{flewelling2017}. 246 246 247 \end{itemize} 247 The above set of analysis stages take place at the IfA within the 248 scope of responsibility of the Pan-STARRS Observatory. Across the 249 wider Pan-STARRS colloboration(s), additional data analysis operations250 are performed to support science results. These collaboration-wide251 analysis operations range from those which are tightly-coupled to the252 Pan-STARRS Observatory system, such as the analysis of the transient 253 search teams and the public archive database at MAST, to those which 254 perform offline analysis for eventual ingest back into the Pan-STARRS 255 databases and archive. The latter category includes the ubercal 256 photometric analysis \citep{ubercal}, the photo-z analysis 257 \citep{photoz}, and the QSO / RR Lyra search efforts248 Management of the above set of analysis stages takes place at the IfA 249 within the scope of responsibility of the Pan-STARRS Observatory. 250 Across the wider Pan-STARRS colloboration(s), additional data analysis 251 operations are performed to support science results. These 252 collaboration-wide analysis operations range from those which are 253 tightly coupled to the Pan-STARRS Observatory system, such as the 254 analysis of the transient search teams and the public archive database 255 at MAST, to those which perform offline analysis for eventual ingest 256 back into the Pan-STARRS databases and archive. The latter category 257 includes the ubercal photometric analysis \citep{ubercal}, the photo-z 258 analysis \citep{photoz}, and the QSO / RR Lyra search efforts 258 259 \citep{hernitschek2016}. In addition, collaborations within the wider 259 260 Pan-STARRS community have implemented a variety of science-level … … 263 264 Figure~\ref{fig:analysis.elements} illustrates the many elements of 264 265 the Pan-STARRS data analysis system. This figure focuses on the data 265 analysis steps which occur within the Pan-STARRS observatory, with an266 analysis steps which occur within the Pan-STARRS Observatory, with an 266 267 emphasis on the analysis, calibration, and database ingest stages. 267 268 The MOPS is described in detail by \cite{2013PASP..125..357D}, while … … 276 277 external groups (``customers''). The arrows show a simplified representation 277 278 of the major flow of data between the analysis stages and data 278 processing elements. }279 processing elements. \note{arrow types are unclear for on-demand vs DVO}} 279 280 \end{center} 280 281 \end{figure*} … … 320 321 analysis stages listed above which are shared with the nightly 321 322 processing, these large-scale reprocessing analyses include additional 322 processing. A more detailed photometric analysis is performed on the 323 stacks, including morphological analysis appropriate to galaxies. The 324 results of the stack photometry analysis are used to drive a 325 forced-photometry analysis of the warp images. The data products from 326 the camera, stack photometry, and forced-warp photometry analysis 327 stages are ingested into the internal calibration database (DVO, the 328 Desktop Virtual Observatory) and used for photometric and astrometric 329 calibrations (see Section~\ref{sec:DVO}). 323 processing steps. A more detailed photometric analysis is performed 324 on the stacks, including morphological analysis appropriate to 325 galaxies. The results of the stack photometry analysis are used to 326 drive a forced-photometry analysis of the warp images. These analysis 327 steps are discussed in detail by 328 \citet[][]{magnier2017.analysis}. The data products from the 329 camera, stack photometry, and forced-warp photometry analysis stages 330 are ingested into the internal calibration database (DVO, the Desktop 331 Virtual Observatory) and used for photometric and astrometric 332 calibrations \citet[see Section~\ref{sec:DVO} and][]{magnier2017.calibration}. 330 333 331 334 \subsection{Data Access and Distribution} … … 345 348 (PV1 \& PV2), the data were ingested into the PSPS database system and 346 349 made available to the PS1SC community through a web portal based at 347 the IfA as well as the MAST portal. 350 the IfA as well as the MAST portal \citep[see][for full 351 details]{flewelling2017}. 348 352 349 353 \section{IPP Data Processing Stages} … … 354 358 355 359 \begin{table*} 356 \caption{ \label{tab:database_schema} GPC1 Database Schema Outline}\vspace{-0.5cm}360 \caption{GPC1 Database Schema Outline} %\vspace{-0.5cm} 357 361 \begin{center} 358 \begin{tabular}{llll l}362 \begin{tabular}{llll} 359 363 \hline 360 364 \hline 361 {\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} & {\bf Notes} \\ 362 \hline 363 \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & & Lists locations to check for new exposures.\\ 364 & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} & Exposures available at the telescope.\\ 365 & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile} & & Exposures that are being downloaded.\\ 366 & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp_id} & Exposures that have been saved to IPP cluster.\\ 367 368 \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp_id} & \\ 369 \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} & \\ 370 \ippstage{camera} & \ippdbtable{camRun} & \ippdbtable{camProcessedExp} & \ippdbcolumn{cam_id} & \\ 371 \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} & \\ 372 \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp_id} & \\ 373 & & \ippdbtable{warpSkyCellMap} & & Mapping of input chips to projection skycells.\\ 374 & & \ippdbtable{warpSkyfile} & & \\ 375 \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack_id} & \\ 376 & & \ippdbtable{stackSumSkyfile} & & \\ 377 \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky_id} & \\ 378 & & \ippdbtable{staticskyResult} & & \\ 379 \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal_id} & \\ 380 \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff_id} & \\ 381 & & \ippdbtable{fullForceResult} & & \\ 382 & & \ippdbtable{fullForceSummary} & & Properties about average parameters from all results.\\ 383 \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff_id} & \\ 384 & & \ippdbtable{diffInputSkyfile} & & \\ 385 \ippstage{detrend} & \ippdbtable{detRun} & \ippdbtable{detRunSummary} & \ippdbcolumn{det_id} & \\ 386 & & \ippdbtable{detInputExp} & & \\ 387 & & \ippdbtable{detRegisteredImfile} & & Information about detrends produced externally.\\ 388 & & \ippdbtable{detStackedImfile} & & \\ 389 & \ippdbtable{detProcessedExp} & \ippdbtable{detProcessedImfile} & & \\ 390 & \ippdbtable{detResidExp} & \ippdbtable{detResidImfile} & & \\ 391 & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & & \\ 392 \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add_id} & \\ 393 \ippstage{distribution} & \ippdbtable{distRun} & \ippdbtable{distComponent} & \ippdbcolumn{dist_id} & \\ 394 & & \ippdbtable{distTarget} & & \\ 395 \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub_id} & \\ 396 & & \ippdbtable{publishClient} & & \\ 397 \ippstage{lap} & \ippdbtable{lapSequence} & \ippdbtable{lapRun} & \ippdbcolumn{seq_id} & Sequence of full reprocessing\\ 398 & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap_id} & \\ 399 \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote_id} & \\ 365 {\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} \\% & {\bf Notes} \\ 366 %%D \begin{deluxetable}{llll} 367 \footnotesize 368 %%D \tablecolumns{5} 369 %%D \tablewidth{0pc} 370 %%D \tablecaption{GPC1 Database Schema Outline} 371 %%D \tablehead{\colhead{Stage} & \colhead{Primary Table} & \colhead{Secondary Table} & \colhead{Key}} % & \colhead{Notes}} 372 %%D \startdata 373 %\hline 374 \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & \\% & Lists locations to check for new exposures.\\ 375 & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} \\% & Exposures available at the telescope.\\ 376 & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile} & \\% & Exposures that are being downloaded.\\ 377 & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp_id} \\% & Exposures that have been saved to IPP cluster.\\ 378 379 \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp_id} \\% & \\ 380 \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} \\% & \\ 381 \ippstage{camera} & \ippdbtable{camRun} & \ippdbtable{camProcessedExp} & \ippdbcolumn{cam_id} \\% & \\ 382 \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} \\% & \\ 383 \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp_id} \\% & \\ 384 & & \ippdbtable{warpSkyCellMap} & \\% & Mapping of input chips to projection skycells.\\ 385 & & \ippdbtable{warpSkyfile} & \\% & \\ 386 \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack_id} \\% & \\ 387 & & \ippdbtable{stackSumSkyfile} & \\% & \\ 388 \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky_id} \\% & \\ 389 & & \ippdbtable{staticskyResult} & \\% & \\ 390 \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal_id} \\% & \\ 391 \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff_id} \\% & \\ 392 & & \ippdbtable{fullForceResult} & \\% & \\ 393 & & \ippdbtable{fullForceSummary} & \\% & Properties about average parameters from all results.\\ 394 \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff_id} \\% & \\ 395 & & \ippdbtable{diffInputSkyfile} & \\% & \\ 396 \ippstage{detrend} & \ippdbtable{detRun} & \ippdbtable{detRunSummary} & \ippdbcolumn{det_id} \\% & \\ 397 & & \ippdbtable{detInputExp} & \\% & \\ 398 & & \ippdbtable{detRegisteredImfile} & \\% & Information about detrends produced externally.\\ 399 & & \ippdbtable{detStackedImfile} & \\% & \\ 400 & \ippdbtable{detProcessedExp} & \ippdbtable{detProcessedImfile} & \\% & \\ 401 & \ippdbtable{detResidExp} & \ippdbtable{detResidImfile} & \\% & \\ 402 & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & \\% & \\ 403 \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add_id} \\% & \\ 404 \ippstage{distribution} & \ippdbtable{distRun} & \ippdbtable{distComponent} & \ippdbcolumn{dist_id} \\% & \\ 405 & & \ippdbtable{distTarget} & \\% & \\ 406 \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub_id} \\% & \\ 407 & & \ippdbtable{publishClient} & \\% & \\ 408 \ippstage{lap} & \ippdbtable{lapSequence} & \ippdbtable{lapRun} & \ippdbcolumn{seq_id} \\% & Sequence of full reprocessing\\ 409 & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap_id} \\% & \\ 410 \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote_id} \\% & \\ 411 %%D \enddata 400 412 \hline 401 413 \end{tabular} 414 \label{tab:database_schema} 415 %%D \end{deluxetable} 402 416 \end{center} 403 417 \end{table*} … … 428 442 either to be done, in progress, or completed. An associated secondary 429 443 table (or set of tables) lists the details of component elements which 430 have been processed for each top-level item. Table \ref{tab: database431 schema} contains an outline of the database schema, showing the 432 relations between tables organized by processing stage. As an 433 example, one critical stage is the \ippstage{chip} processing stage 434 (see \S\ref{sec:chip}) in which the individual chips from an exposure 435 are detrended and sources are detected. Within the gpc1 database, the 436 primary table is called \ippdbtable{chipRun} in which each exposure 437 has a single entry. Associated with this table is the444 have been processed for each top-level item. Table 445 \ref{tab:database_schema} contains an outline of the database schema, 446 showing the relations between tables organized by processing stage. 447 As an example, one critical stage is the \ippstage{chip} processing 448 stage (see \S\ref{sec:chip}) in which the individual chips from an 449 exposure are detrended and sources are detected. Within the gpc1 450 database, the primary table is called \ippdbtable{chipRun} in which 451 each exposure has a single entry. Associated with this table is the 438 452 \ippdbtable{chipProcessedImfile} table, which contains one row for 439 453 each of the chips associated with the exposure (up to 60 for gpc1). … … 550 564 database tables (\ippdbtable{rawExp} and \ippdbtable{rawImfile}). 551 565 552 For GPC1, the \ippstage{registration} stage is also the stage at which the553 \ippprog{burntool} analysis is run. This analysis is more completely 554 described in \citet{waters2017}. In brief, the \ippprog{burntool} 555 program identifies bright sources on the image, and identifies 556 persistence trails that result from the incomplete transfer of charge. 557 As this charge can leak out in subsequent exposures, the burntool 558 analysis is run sequentially on the exposures, based on the566 For GPC1, the \ippstage{registration} stage is also the stage at which 567 the \ippprog{burntool} analysis is run. This analysis is more 568 completely described in \citet{waters2017}. In brief, the 569 \ippprog{burntool} program identifies bright sources on the image, and 570 identifies persistence trails that result from the incomplete transfer 571 of charge. As this charge can leak out in subsequent exposures, the 572 burntool analysis is run sequentially on the exposures, based on the 559 573 observation date and time listed in the headers, with the results 560 stored in an text table. As a result of the sequential nature of this561 analysis, the \ippstage{registration} of exposures is blocked until the562 \ippprog{burntool} has been run on the previous exposures.574 stored on disk. As a result of the sequential nature of this 575 analysis, the \ippstage{registration} of exposures is blocked until 576 the \ippprog{burntool} has been run on the previous exposures. 563 577 564 578 Once the \ippstage{registration} process has finished, new science … … 591 605 majority of stages operate on smaller segments of a full exposure, 592 606 allowing the processing tasks to be spread over the machines in the 593 processing cluster. The \ippprog{pantasks} environment , which manages594 the jobs, attempts to target the processing to a computer which is 595 a ssigned to host data for the particular OTA. This capability is596 implemented to reduce the network I/O load by minimizing the number of 597 operations done on non-local data. In practice, this targeted 598 processing has not had as large of an impact as was originally 599 intended: the data volume and operational details of the hardware has 600 reduced the ability of any one node to reliably contain a particular 601 OTA. The targeted processing has probably reduced the network load602 somewhat but it has not been as critical of a requirementas603 originally expected.607 processing cluster. The \ippprog{pantasks} environment (the system 608 which manages the processing jobs, see Section~\ref{sec:pantasks}) 609 attempts to target the processing to a computer which is assigned to 610 host data for the particular OTA. This capability is implemented to 611 reduce the network I/O load by minimizing the number of operations 612 done on non-local data. In practice, this targeted processing has not 613 had as large of an impact as was originally intended: the data volume 614 and operational details of the hardware has reduced the ability of any 615 one node to reliably contain a particular OTA. The targeted 616 processing has probably reduced the network load somewhat but it has 617 not been as critical of a requirement as originally expected. 604 618 605 619 %% In the \ippstage{chip} stage, … … 623 637 program. This program reads the raw data into memory and applies the 624 638 detrend corrections \citep[see][]{waters2017} to each cell in the OTA 625 ( which are stored as different extensions in the FITS file format),626 and then mosaics the cells into a single contiguous \ippstage{chip} 627 stage image. This step also creates in memory additional images to 628 hold the mask data, which indicates which pixels may not be valid, and 629 the variance image, constructed as the Poissonian noise on the number 630 of electrons detected based on the original pixel value and the 631 detector gain. A background model is then fit across the image and632 subtractedto remove the expected contribution from the sky639 (stored as different extensions in the FITS file format), and then 640 mosaics the cells into a single contiguous \ippstage{chip} stage 641 image. This step also creates in memory additional images to hold the 642 mask data, which indicates which pixels may not be valid, and the 643 variance image, constructed as the Poissonian noise on the number of 644 electrons detected based on the original pixel value and the detector 645 gain. A background model is then fit across the image and subtracted 646 to remove the expected contribution from the sky 633 647 \citep[see][]{waters2017} for details. 634 648 … … 706 720 The guess astrometry is used to match the reference catalog to the 707 721 observed stellar positions in the focal plane coordinate system 708 \citep[see][]{magnier2017.calibration} ).722 \citep[see][]{magnier2017.calibration}. 709 723 710 724 Once an acceptable match is found, the astrometric calibration of the … … 838 852 generated for the nightly groups and for the full depth using all 839 853 exposures, producing ``deep stacks''. In addition, a ``best seeing'' 840 set of stacks have been produced \note{using image quality cuts to be841 described: need input from MEH}. We have also generated 842 out-of-season stacks for the Medium Deep fields, in which all images 843 not from a particular observing season for a field are combined into a 844 stack.These later stacks are useful as deep templates when studying854 set of stacks have been produced using image quality cuts described by 855 \citet[][Paper VII]{huber2017}. We have also generated out-of-season 856 stacks for the Medium Deep fields, in which all images {\em not} from a 857 particular observing season for a field are combined into a stack. 858 These later stacks are useful as deep templates when studying 845 859 long-term transient events in the Medium Deep fields as they are not 846 860 (or less) contaminated by the flux of the transients from a given … … 879 893 880 894 Although images are generated in the \ippstage{stack} stage of the 881 IPP, the source detection and extraction analysis of those images is882 deferred to the \ippstage{staticsky} stage. This separation is 883 maintained because the photometry analysis of the \ippstage{stack} 884 images, including convolved galaxy model fitting, is performed on all 885 5 filters simultaneously. By deferring this analysis, the processing 886 system may also decouple the generation of the pixels from the source 887 detection. This makes the sequencing of analysis somewhat easier and 888 less subject to blocks due to a failure in thestacking analysis.895 IPP, the source detection and analysis of those images is deferred to 896 the \ippstage{staticsky} stage. This separation is maintained because 897 the photometry analysis of the \ippstage{stack} images, including 898 convolved galaxy model fitting, is performed on all 5 filters 899 simultaneously. By deferring this analysis, the processing system may 900 also decouple the generation of the pixels from the source detection. 901 This makes the sequencing of analysis somewhat easier and less subject 902 to blocks due to a failure in the long-running stacking analysis. 889 903 Similar to the \ippstage{stack} stage, an entry is created in the 890 904 \ippdbtable{staticskyRun} table, linked to a series of rows in the … … 893 907 entries for the skycell under consideration. 894 908 895 The input images are passed to the \ippprog{psphotStack} program, 896 which does the analysis. The stack photometry algorithms are 897 described in detail in \cite{magnier2017.analysis}. In short, sources are 898 detected in all 5 filter images down to the $5\sigma$ significance. 899 The collection of detected sources is merged into a single master 900 list. If a source is detected in at least two bands, or only in 901 \yps{} band, then a PSF model is fitted to the pixels of the other 902 bands in which the source was not detected. This forced photometry 903 results in lower significance measurements of the flux at the 904 positions of objects which are thought to be real sources, by virtue 905 of triggering a detection in at least two bands. The relaxed limit 906 for \yps{} band is included to allow for searches of \yps{} dropout 907 objects: it is known that faint, high-redshift quasars may be detected 908 in \yps{} band only. Sources detected only in \yps{} band are 909 therefore more likely to have a higher false-positive rate than the 910 other stack sources. 909 The input images are passed to the \ippprog{psphotStack} program which 910 does the analysis. The stack photometry algorithms are described in 911 detail in \cite{magnier2017.analysis}. In short, sources are detected 912 in all 5 filter images down to the $5\sigma$ significance. The 913 collection of detected sources is merged into a single master list. 914 If a source is detected in at least two bands, or only in \yps{} band, 915 then a PSF model is fitted to the pixels of the other bands in which 916 the source was not detected. This forced photometry results in lower 917 significance measurements of the flux at the positions of objects 918 which are thought to be real sources, by virtue of triggering a 919 detection in at least two bands. The relaxed limit for \yps{} band is 920 included to allow for searches of \yps{} dropout objects: it is known 921 that faint, high-redshift quasars may be detected in \yps{} band only. 922 Sources detected only in \yps{} band are therefore more likely to have 923 a higher false-positive rate than the other stack sources. 911 924 912 925 The stack photometry output files consist of a set of FITS table … … 946 959 \subsection{Forced Warp Photometry} 947 960 \label{sec:fullforce} 948 949 \note{too much detail in this section; balance relative to psphot}950 961 951 962 Traditionally, projects which use multiple exposures to increase the … … 977 988 degraded. The highly textured PSF variations make this a very 978 989 challenging problem: not only would such a PSF model need to be highly 979 fine-grained, there would likely not be enough PSFstars in a given990 fine-grained, there would likely not be enough stars in a given 980 991 \ippstage{stack} image to determine the model at the resolution 981 992 required. The IPP photometry analysis code uses a PSF model with 2D … … 986 997 images. 987 998 988 Thus PSF photometry a s well as convolved galaxy models in the stack999 Thus PSF photometry and convolved galaxy model analysis in the stack 989 1000 are degraded by the PSF variations. Aperture-like measurements are in 990 1001 general not as affected by the PSF variations, as long as the aperture … … 1043 1054 the PSF-convolved galaxy models are of limited accuracy. 1044 1055 1045 Upon completion of the forced photometry (for point sources as well as 1046 galaxies, discussed below), an entry is added to the 1056 Upon completion of the forced photometry, an entry is added to the 1047 1057 \ippdbtable{fullForceResult} table with the processing statistics for 1048 1058 that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}. … … 1075 1085 epoch. The quality of such a difference image can be enhanced by 1076 1086 convolving one or both of the images so that the PSFs in the two 1077 images are matched . \note{discuss Alard-Lupton}.1087 images are matched \citep[e.g.,][]{AlardLupton}. 1078 1088 1079 1089 In the \ippstage{diff} stage, the IPP generates difference images for … … 1082 1092 images, from a \ippstage{warp} and a \ippstage{stack} of some variety, 1083 1093 or from a pair of \ippstage{stack} stage images. During the PS1 1084 survey, pairs of exposures, called TTI pairs (see~\note{Survey 1085 Strategy in Chambers et al}), were obtained for each pointing within a $\approx$ 1 1086 hour period in the same filter, and to the extent possible with the 1087 same orientation and boresite position. The standard PS1 nightly 1088 processing generated difference images from the resulting pairs of 1089 \ippstage{warp} images. The nightly processing generated 1090 \ippstage{stack} images for the Medium Deep fields, and these were 1091 combined with a template reference \ippstage{stack} image to generate 1092 ``stack-stack diffs'' each night they were observed. For the PV3 1093 $3\pi$ processing, the entire collection of \ippstage{warp} stage 1094 images for the survey were combined with images generated by the 1095 \ippstage{stack} processing to generate ``warp-stack diffs''. 1094 survey, pairs of exposures, called TTI pairs \citep[see Survey 1095 Strategy in][]{chambers2017}, were obtained for each pointing within 1096 a $\approx$ 1 hour period in the same filter, and to the extent 1097 possible with the same orientation and boresite position. The 1098 standard PS1 nightly processing generated difference images from the 1099 resulting pairs of \ippstage{warp} images. The nightly processing 1100 generated \ippstage{stack} images for the Medium Deep fields, and 1101 these were combined with a template reference \ippstage{stack} image 1102 to generate ``stack-stack diffs'' each night they were observed. For 1103 the PV3 $3\pi$ processing, the entire collection of \ippstage{warp} 1104 stage images for the survey were combined with images generated by the 1105 \ippstage{stack} processing to generate ``warp-stack diffs'', for 1106 eventual public released. 1096 1107 1097 1108 When a \ippstage{diff} processing is defined, an entry is added to the … … 1132 1143 \label{sec:postprocessing} 1133 1144 1134 \note{introduction to this section: data ingested into DVO database,1135 database gets calibrated, data ingested into PSPS via IPP to PSPS}1136 1137 1145 \begin{table}[hb] 1138 1146 \begin{center} 1139 \caption{DVO Database Tables\label{tab:DVO_schema} \note{fix order, 1140 drop invalid tables}} 1147 \caption{DVO Database Tables\label{tab:DVO_schema} \note{fix names, include missing}} 1141 1148 \begin{tabular}{ll} 1142 1149 \hline … … 1145 1152 \hline 1146 1153 Images & The images that have objects in the DB. \\ 1147 Image Overlaps & Image regions which are touched by specific images. \\1154 % Image Overlaps & Image regions which are touched by specific images. \\ 1148 1155 Objects & The objects --- average properties of multiple detections of the same object. \\ 1149 Average Magnitudes& Average photometry in multiple filters \\1150 Solar System Objects & Identification of solar system objects \\1151 M atched Detections& Detections of sources in an image identified with an Object. \\1152 Orphaned Detections & Detections of sources in an image not identified with an Object. \\1153 Non-detections & Non-detections of objects in an image. \\1156 Average & Average photometry in multiple filters \\ 1157 % Solar System Objects & Identification of solar system objects \\ 1158 Measure & Detections of sources in an image identified with an Object. \\ 1159 % Orphaned Detections & Detections of sources in an image not identified with an Object. \\ 1160 % Non-detections & Non-detections of objects in an image. \\ 1154 1161 SkyRegions & spatial distribution of tables \\ 1155 Filters & Filters understood by the system. \\1162 % Filters & Filters understood by the system. \\ 1156 1163 Photcodes & Transformations between different photometric systems \\ 1157 Zero Points & History of Zero-point \& Airmass terms \\1158 Distortion Models & History of Optical Distortion terms \\1159 Database Hosts& computers used to store the tables \\1164 % Zero Points & History of Zero-point \& Airmass terms \\ 1165 % Distortion Models & History of Optical Distortion terms \\ 1166 Hosts & computers used to store the tables \\ 1160 1167 \hline 1161 1168 \end{tabular} … … 1210 1217 which store supporting information (metadata). 1211 1218 1212 DVO includes two major classes of database tables: those containing1213 information about astronomical objects in the sky and those containing1214 other supporting information. The object-related tables are1215 partitioned on the basis of position in the sky: objects within a1216 region bounded by lines of constant RA,DEC are contained in a specific1217 file. The boundaries and the associated partition names are stored in1218 one of the supporting tables, \ippdbtable{SkyTable}. This table1219 contains the definitions of the boundaries for each sky region1220 (\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN},1221 \ippdbcolumn{D_MAX}), the name of the sky region, an ID1222 (\ippdbcolumn{INDEX}, equal to the sequence number of the region in1223 the table), and index entries to enable navigation within the table.1224 The regions are defined in a hierarchical sense, with a series of1225 levels each containing a finer mesh of regions covering the sky.1219 %% DVO includes two major classes of database tables: those containing 1220 %% information about astronomical objects in the sky and those containing 1221 %% other supporting information. The object-related tables are 1222 %% partitioned on the basis of position in the sky: objects within a 1223 %% region bounded by lines of constant RA,DEC are contained in a specific 1224 %% file. The boundaries and the associated partition names are stored in 1225 %% one of the supporting tables, \ippdbtable{SkyTable}. This table 1226 %% contains the definitions of the boundaries for each sky region 1227 %% (\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, 1228 %% \ippdbcolumn{D_MAX}), the name of the sky region, an ID 1229 %% (\ippdbcolumn{INDEX}, equal to the sequence number of the region in 1230 %% the table), and index entries to enable navigation within the table. 1231 %% The regions are defined in a hierarchical sense, with a series of 1232 %% levels each containing a finer mesh of regions covering the sky. 1226 1233 1227 1234 \subsubsubsection{Photcodes} … … 1257 1264 transform a measurement in the specific photcode to a common system. 1258 1265 For example, a \ippmisc{DEP} photcode GPC1.g.X01 would have the 1259 nominal zero point (2 5.XX) and airmass term (0.14). The structures1266 nominal zero point (24.563) and airmass term (0.147). The structures 1260 1267 allow for individual chips to have different color terms to bring them 1261 to a common filter system. 1268 to a common filter system. 1262 1269 1263 1270 Beyond the basic use, DVO has the ability to accept data from other … … 1281 1288 processed by the IPP may also be included similarly in a DVO database. 1282 1289 Measurements from other sources, such as SDSS, 2MASS, or WISE, can 1283 also be included in this table. 1290 also be included in this table, distinguished by their different 1291 photcodes. 1284 1292 1285 1293 The \ippdbtable{Measure} table includes the instrumental magnitudes … … 1296 1304 discussed below) and the astrometrically calibrated position. 1297 1305 Astrometric offsets for several systematic corrections discussed below 1298 are also defined for each measurement. Photometry from \ippstage{chip}, \ippstage{warp},1299 and \ippstage{stack} are all placed in the same table with photcodes 1300 distinguishing the source \note{show example of stack and warp 1301 photcodes}. Since stacks and forced warp fluxes may have1302 non-significant values, the table is somewhat de-normalized: it also 1303 carries both magnitudes as well as instrumental flux values for the 1304 PSF, aperture, and Kron photometry. In this case, we have chosen to 1305 trade storage space forcomputing time.1306 are also defined for each measurement. Photometry from 1307 \ippstage{chip}, \ippstage{warp}, and \ippstage{stack} are all placed 1308 in the same table with photcodes distinguishing the source. Since 1309 stacks and forced warp fluxes may have non-significant values, the 1310 table is somewhat de-normalized: it also carries both magnitudes as 1311 well as instrumental flux values for the PSF, aperture, and Kron 1312 photometry. In this case, we have chosen to trade storage space for 1313 computing time. 1306 1314 1307 1315 For the warp images, we also measure the weak lensing KSB parameters … … 1310 1318 along with the radial aperture fluxes for radii numbers 5, 6, \& 7 1311 1319 (respectively 3.0, 4.63, and 7.43 arcsec). This table contains one 1312 row for every warp row. \note{warp row hasn't been defined anywhere.} 1313 Similarly to the \ippdbtable{Measure} table, the fields 1314 \ippdbcolumn{objID}, \ippdbcolumn{catID}, and \ippdbcolumn{averef} 1315 define links from the \ippdbtable{Lensing} table to the 1316 \ippdbtable{Average} table. In a similar fashion, the fields 1317 \ippdbtable{Average}.\ippdbcolumn{lensingOffset} and 1318 \ippdbtable{Average}.\ippdbcolumn{Nlensing} are the index into the 1319 sorted \ippdbtable{Lensing} table entries. \note{discuss failure of 1320 the Lensing to Measure indexing} 1321 1322 \note{Average used above but defined below} 1320 row for every warp image on which the object was measured. 1321 1322 %% Similarly to the \ippdbtable{Measure} table, the fields 1323 %% \ippdbcolumn{objID}, \ippdbcolumn{catID}, and \ippdbcolumn{averef} 1324 %% define links from the \ippdbtable{Lensing} table to the 1325 %% \ippdbtable{Average} table. In a similar fashion, the fields 1326 %% \ippdbtable{Average}.\ippdbcolumn{lensingOffset} and 1327 %% \ippdbtable{Average}.\ippdbcolumn{Nlensing} are the index into the 1328 %% sorted \ippdbtable{Lensing} table entries. \note{discuss failure of 1329 %% the Lensing to Measure indexing} 1330 1331 % \note{Average used above but defined below} 1323 1332 1324 1333 \subsubsubsection{Object Tables} … … 1332 1341 new detections are loaded, they are compared to the objects already 1333 1342 stored in the database. If an object is already found in the database 1334 within the match radius of \note{one arcsecond}, the new detection is 1335 assigned to that object. If more than one object exists within the 1336 database, the detection is associated with the closest object. 1343 within the match radius, the new detection is assigned to that 1344 object. If more than one object exists within the database, the 1345 detection is associated with the closest object. For most data 1346 sources, a match radius of 1.0 arcsecond is used, but this may be 1347 adjusted in special cases. 1337 1348 1338 1349 Two tables carry the most important information about the astronomical … … 1343 1354 \pi$) and associated errors, data quality flags for each object, links 1344 1355 to the other tables, and a number of IDs, with one row for each 1345 astronomical object. \note{go into complete detail here on the IDs?}.1356 astronomical object. 1346 1357 The \ippdbtable{SecFilt} table\footnote{The name \ippdbtable{SecFilt} 1347 1358 is a bit of a historical misnomer: originally, DVO was designed for … … 1391 1402 The \ippdbtable{Starpar} table carries measurements provide by Greg 1392 1403 Green \& Eddie Schlafly from their analysis of the SED of objects in 1393 the PS1 $3\pi$ data, using the \note{PV1?}version of the analysis1404 the PS1 $3\pi$ data, using the PV1 version of the analysis 1394 1405 \citep{2015ApJ...810...25G}. In this work, the goal was a 3D model of 1395 1406 the dust in the Galaxy based on Pan-STARRS and 2MASS photometry. As … … 1399 1410 these photometric distance modulus measurements are not extremely 1400 1411 precise (see below), they provide a constraint on the distance is used 1401 in our analysis of the astrometry \citep[see][]{magnier2017.calibration}. 1412 in our analysis of the astrometry 1413 \citep[see][]{magnier2017.calibration}. 1402 1414 1403 1415 In the \ippdbtable{Measure} table, there are three fields which 1404 1416 provide two independent links from the specific measurement to the 1405 1417 associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies 1406 the spatial partition to which the measurement belongs; 1418 the spatial partition to which the measurement belongs (see 1419 Section~\ref{sec:SkyPartition} below); 1407 1420 \ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in 1408 1421 the \ippdbtable{Average} table the measurement belongs. These two 32 1409 1422 bit fields can thus be combined into a single 64 bit ID unique for all 1410 objects in the database. \note{PSPS IDs}In addition, the field1423 objects in the database. In addition, the field 1411 1424 \ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in 1412 1425 the \ippdbtable{Average} table of the associated object. The … … 1421 1434 field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from 1422 1435 the measurement to the image which supplied the measurement. 1436 1437 \note{Discuss PSPS IDs} 1423 1438 1424 1439 \subsubsubsection{Image Tables} … … 1460 1475 1461 1476 \subsubsection{Sky Partition} 1462 1463 \note{re-word this sentence} DVO includes two major classes of database tables: those containing 1464 information about astronomical objects in the sky and those containing 1465 other supporting information. The object-related tables are 1466 partitioned on the basis of position in the sky: objects within a 1477 \label{sec:SkyPartition} 1478 1479 Tables within DVO containing information about astronomical objects 1480 are partitioned on the basis of position in the sky: objects within a 1467 1481 region bounded by lines of constant RA,DEC are contained in a specific 1468 1482 file. The boundaries and the associated partition names are stored in … … 1478 1492 In the default used by the PV3 DVO, the partitioning scheme is based 1479 1493 on the one used by the Hubble Space Telescope Guide Star Catalog 1480 files. \note{add figure} Level 0 is a single region covering the full1481 sky. Level 1 divides the sky in declination into bands 1482 7.5\degree\ high. Level 2 subdivides these declination bands in the 1483 RAdirection, with spacing related to the stellar density. Level 31494 files. Level 0 is a single region covering the full sky. Level 1 1495 divides the sky in declination into bands 7.5\degree\ high, as defined 1496 by the HST GSC. Level 2 subdivides these declination bands in the RA 1497 direction, with spacing related to the stellar density. Level 3 1484 1498 divides these RA chunks into 4 - 8 smaller partitions. This level 1485 1499 exactly matches the HST GSC layout, and uses the same naming 1486 convention to identify the partitions: \code{n0000/0000}, 1487 etc. \note{more on the names?}. Level 4 further divides these regions 1488 by a factor of 16. In the \ippdbtable{SkyTable}, a region at one 1489 level has a pointer to its parent region (the one which contains it) 1490 and a sequence pointing to its children (regions it contains). The 1491 \ippdbtable{SkyTable} enables fast lookups of the on-disk partitions 1492 which map to a specific coordinate on the sky. In general, a single 1493 DVO will have the full sky represented with tables at a single 1494 level. Although it is possible for mixed levels to be used, this mode 1495 is not well tested and is avoided in the PV3 DVO database. For the 1496 PV3 master database, the partitioning at the \note{should this be 1497 4th?} 5th level results in \approx 150,000 regions to cover the full 1498 sky, of which \approx 110,000 are used for the PV3 $3\pi$ data. The 1499 densest portions of the bulge contain at most \approx 300,000 1500 astronomical objects in the database files, with an associated maximum 1501 of \approx 30 million measurements in these files. With the compression 1502 scheme described below, the largest database files are \approx 1503 3GB, which can be loaded into memory in 30 seconds on the processing 1504 machines that contain partition data. 1505 1506 \note{is the use of the term `partition host' consistent in this paper 1507 and the calibration paper?} 1500 convention to identify the partitions: \code{n0000/0000}, etc. Level 4 1501 further divides these regions by a factor of 16. In the 1502 \ippdbtable{SkyTable}, a region at one level has a pointer to its 1503 parent region (the one which contains it) and a sequence pointing to 1504 its children (regions it contains). The \ippdbtable{SkyTable} enables 1505 fast lookups of the on-disk partitions which map to a specific 1506 coordinate on the sky. In general, a single DVO will have the full 1507 sky represented with tables at a single level, although it is possible 1508 for mixed levels to be used. For the PV3 master database, the 1509 partitioning is at Level 4, resulting in \approx 150,000 regions to 1510 cover the full sky, of which \approx 110,000 are used for the PV3 1511 $3\pi$ data. The densest portions of the bulge contain at most 1512 \approx 300,000 astronomical objects in the database files, with an 1513 associated maximum of \approx 30 million measurements in these files. 1514 With the compression scheme described below, the largest database 1515 files are \approx 3GB, which can be loaded into memory in 30 seconds 1516 on the processing machines that contain partition data. 1508 1517 1509 1518 % parallel partitions 1510 1519 The DVO software system allows the tables which are partitioned across 1511 1520 the sky to also be distributed across multiple computers, which we 1512 call partition hosts. A single file defines the names of these1513 partition hosts and the location of the database partition on the 1514 disks of that machine. The \ippdbtable{SkyTable} contains elements to 1515 define by ID the parition host to which a partitioned set of tables 1516 has been assigned. Operations which query the database, or perform 1517 other operations on the database, are aware of the partitioning scheme 1518 and will launch their operations as remote processes on the machines 1519 which contain the data they need. For example, a query for data from 1520 a small region will launch sub-query operations on the machines which 1521 contain the data overlapping the region of interest. These remote 1522 query operations will select the database information which matches 1523 the query request (i.e., applying restrictions as defined) and return 1524 the results to the master process. The results from the various 1525 partition hosts are then merged into a single result by the master 1526 process. When the parallel partitioning for a DVO instance is 1527 defined, the tables are randomly assigned to the partition hosts. As 1528 a result, queries which span more than a single parition are likely to 1529 spread the I/O load across a large number of machines. This 1530 parallelization is critical to querying and manipulating the enormous 1531 database on areasonable timescale.1521 call partition hosts. A single file identifies these partition hosts 1522 and the location of the database partition on the disks of that 1523 machine. The \ippdbtable{SkyTable} contains elements to define by ID 1524 the parition host to which a set of tables has been assigned. 1525 Operations which query the database, or perform other operations on 1526 the database, are aware of the partitioning scheme and will launch 1527 their operations as remote processes on the machines which contain the 1528 data they need. For example, a query for data from a small region 1529 will launch sub-query operations on the machines which contain the 1530 data overlapping the region of interest. These remote query 1531 operations will select the database information which matches the 1532 query request (i.e., applying restrictions as defined) and return the 1533 results to the master process. The results from the various partition 1534 hosts are then merged into a single result by the master process. 1535 When the parallel partitioning for a DVO instance is defined, the 1536 tables are randomly assigned to the partition hosts. As a result, 1537 queries which span more than a single parition are likely to spread 1538 the I/O load across a large number of machines. This parallelization 1539 is critical to querying and manipulating the enormous database on a 1540 reasonable timescale. 1532 1541 1533 1542 \subsubsection{DVO Data Storage} … … 1537 1546 the database tables are stored on disk using binary FITS tables. Each 1538 1547 type of database table is stored as a separate file, or a collection 1539 of files for table which are spatially partitioned. The binary FITS1548 of files for tables which are spatially partitioned. The binary FITS 1540 1549 tables are compressed using the (to date) experimental FITS binary 1541 table compression strategy outlined by \ note{REF}. Table compression1550 table compression strategy outlined by \citet{RickWhite}. Table compression 1542 1551 is an option in DVO; for the PV3 database, the large data 1543 1552 volume (70TB compressed) drove the decision to compress the tables. … … 1587 1596 is associated with the the \ippstage{addstar} processing stage. The 1588 1597 measurement catalogs generated by the \ippstage{camera}, 1589 \ippstage{staticsky}, \ippstage{skycal}, \ippstage{fullforce}, and 1590 \ippstage{diff} stages are processed loaded into DVOs in this fashion, 1591 although not every measurement in each catalog are included in the 1592 master DVO that is constructed. For a particular re-processing 1593 version, a single master DVO is constructed for the positive image 1594 stages (\ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal}, 1595 \ippstage{fullforce}) and a separate one is constructed for the 1596 difference image analysis stage results. 1598 \ippstage{skycal}, \ippstage{fullforce}, and \ippstage{diff} stages 1599 are loaded into DVOs in this fashion, although not every measurement 1600 in each catalog are included in the master DVO that is constructed. 1601 For a particular re-processing version, a single master DVO is 1602 constructed for the positive image stages (\ippstage{camera}, 1603 \ippstage{skycal}, \ippstage{fullforce}) and a separate one is 1604 constructed for the difference image analysis stage results. 1597 1605 1598 1606 The construction of the master DVO is performed in a hierarchical … … 1620 1628 some stages, such as the \ippstage{diff} stage, create more than a 1621 1629 single catalog, multiple entries with the \ippdbcolumn{stage_id} are 1622 created, with the \ippdbcolumn{stage_extra1} field containing an 1623 index to the individual components. The catalog specified by the 1624 entry is added to the target \ippmisc{minidvo} by the 1625 \ippprog{addstar} program, \note{describe what's done?}. When this 1626 completes, an entry containing the statistics of the job is added to 1627 the \ippdbtable{addProcessedExp} table.1630 created, with the \ippdbcolumn{stage_extra1} field containing an index 1631 to the individual components. The catalog specified by the entry is 1632 added to the target \ippmisc{minidvo} by the \ippprog{addstar} 1633 program, updating the measurements in the appropriate DVO tables. 1634 When this completes, an entry containing the statistics of the job is 1635 added to the \ippdbtable{addProcessedExp} table. 1628 1636 1629 1637 After the master DVO is contructed containing the PS1 data, data from 1630 1638 other sources are also added to the database. For the PV3 DVO 1631 database, data was added from 2MASS, WISE, Gaia , and Tycho. These1639 database, data was added from 2MASS, WISE, Gaia DR1, and Tycho. These 1632 1640 external data sources are added by first generating a DVO database 1633 1641 containing just the particular data source, then using the same DVO … … 1665 1673 astrometry is again performed this time using the corrected positions. 1666 1674 1667 \note{have eddie suggest wording here?}1668 1669 1675 Photometric calibration consists of determination of zero points for 1670 1676 each exposure along with corrections for systematic effects. In this … … 2122 2128 \label{sec:automation} 2123 2129 2124 \note{start with a discussion of the standard sequencing (end-stage)}2125 2126 \note{then discuss the addstar sequences with manual triggering}2127 2128 2130 Outside of the basic sequence of \ippstage{chip} to \ippstage{warp}, there is no single 2129 2131 natural next step. For example: a stack can be generated with any … … 2136 2138 \ippmisc{ippScript}. These scripts have a well-defined and restricted 2137 2139 set of goals: to ensure that difference images are generated for each 2138 exposure s(either by pairing together warps or pairs warps with2140 exposure (either by pairing together warps or pairs warps with 2139 2141 pre-defined stacks), that nightly stacks are generated for MD fields, 2140 2142 and that the stacks are also differenced against an appropriate … … 2190 2192 The automatic nightly processing ensures that data is processed as 2191 2193 soon as it is downloaded from the summit, reducing the lag between an 2192 observation and the reduced data. \note{some numbers here about 2193 completion times and such? Words about getting data to MOPS and SN 2194 transient folks} 2195 2196 \note{re-read paragraph below and cleanup} 2194 observation and the reduced data. 2197 2195 2198 2196 The other processing task that requires automation is the reprocessing … … 2208 2206 unit of sky defined to be a square four degrees on each side which has 2209 2207 a single tangent plane projection \citep[][see]{waters2017}. 2210 \note{does waters2017 discuss RINGS.V3? if not, where?}Once this2208 Once this 2211 2209 entry is defined, it is populated with all exposures (stored in the 2212 2210 \ippdbtable{lapExp} table in the database) that are located … … 2268 2266 hardware acquisition, occasional hardware failures, and other 2269 2267 organizational details, targeted processing has only been used to a 2270 moderate degree within the Pan-STARRS cluster. \note{can we get a 2271 number here?} 2268 moderate degree within the Pan-STARRS cluster. 2272 2269 2273 2270 \subsubsection{Implementation Details} … … 2276 2273 well as APIs in both C and Perl. 2277 2274 2278 "The basic user commands to interact with Nebulous are to 1) query the2275 The basic user commands to interact with Nebulous are to 1) query the 2279 2276 database for an existing storage object, and find a valid file 2280 2277 instance associated with that object; 2) create a new storage object, … … 2408 2405 Transferring data between the IPP and other parts of the Pan-STARRS 2409 2406 system is generally accomplished via a ``datastore'', an http service 2410 that exposes data in a common form. \note{add Isani / Hoblitt 2411 reference?} One of the main datastores used by the IPP is the one 2412 located at the summit. This datastore exposes a list of the 2413 exposures obtained since the start of the PS1 operations. Requests to 2414 this server may restrict to the latest by time. Each row in the 2415 listing includes basic information about the exposure: an exposure 2416 identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for details), the 2417 date and time of the exposure, the telescope commanded pointing, the 2418 filter and exposure time, and the observation comment for that 2419 exposure. The row also provides a link to a listing of the chips 2420 associated with that exposure. This listing includes a link to the 2421 individual chip FITS files as well as an md5 checksum. Systems which 2422 are allowed access may download the raw chip FITS files via http requests to 2423 the provided links. 2424 2425 \note{add a discussion of gpc1 filenames?} 2407 that exposes data in a common form. One of the main datastores used 2408 by the IPP is the one located at the summit. This datastore exposes a 2409 list of the exposures obtained since the start of the PS1 operations. 2410 Requests to this server may restrict to the latest by time. Each row 2411 in the listing includes basic information about the exposure: an 2412 exposure identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for 2413 details), the date and time of the exposure, the telescope commanded 2414 pointing, the filter and exposure time, and the observation comment 2415 for that exposure. The row also provides a link to a listing of the 2416 chips associated with that exposure. This listing includes a link to 2417 the individual chip FITS files as well as an md5 checksum. Systems 2418 which are allowed access may download the raw chip FITS files via http 2419 requests to the provided links. 2420 2421 % \note{add a discussion of gpc1 filenames?} 2426 2422 2427 2423 The IPP also uses datastores to provide access to its own data … … 2543 2539 library. 2544 2540 2545 \note{This likely needs cleaning up and more information.}2546 2547 2541 \section{IPP Hardware Systems} 2548 2542 \label{sec:hardware} 2549 2550 \note{what about psps hardware? mops hardware?}2551 2543 2552 2544 \subsection{Kihei Processing Cluster} … … 2567 2559 with a variety of individual specifications due to the cluster being 2568 2560 assembled from multiple generations of purchases. The data storage 2569 nodes contain \note{check, as this contains B nodes} approximately 10 2570 petabytes of storage space that are used to store both the raw 2571 exposure data downloaded from the telescope as well as processed data 2572 products. These nodes are also used to do processing, and have jobs 2573 targeted to them in an effort to reduce the network I/O demands 2574 (see~\ref{sec:chip} for more on this process). 2561 nodes contain several petabytes of storage space that are used to 2562 store both the raw exposure data downloaded from the telescope as well 2563 as processed data products. These nodes are also used to do 2564 processing, and have jobs targeted to them in an effort to reduce the 2565 network I/O demands (see~\ref{sec:chip} for more on this process). 2575 2566 2576 2567 These storage nodes are not fully capable of completing all processing … … 2584 2575 2585 2576 The final type of computer in the cluster are the database servers. 2586 These special purpose computers \note{have lots of memory and disk 2587 space? Is that it?} are used to store and manage both the IPP gpc1 2588 and \ippprog{Nebulous} databases. In addition to the main master 2589 servers, we have duplicate servers used as database replicants, which 2590 allow for quick switching from the main to backup servers in case of a 2591 hardware issue that cannot be resolved immediately. 2577 These computers have large memory capacity and high-speed disk access 2578 (originally fast spindle spinning disks, now migrated to SSDs) are 2579 used to store and manage both the IPP gpc1 and \ippprog{Nebulous} 2580 databases. In addition to the main master servers, we have duplicate 2581 servers used as database replicants, which allow for quick switching 2582 from the main to backup servers in case of a hardware issue that 2583 cannot be resolved immediately. 2592 2584 2593 2585 \subsection{Los Alamos National Labs} … … 2643 2635 values used for the various IPP processing stages. 2644 2636 2645 \begin{table} 2646 \caption{\label{tab:SC_processing_parameters} Cost values for remote processing}\vspace{-0.5cm} 2647 \begin{center} 2648 \begin{tabular}{lcc} 2649 \hline 2650 \hline 2651 {\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ 2652 \hline 2637 %% \begin{table} 2638 %% \caption{\label{tab:SC_processing_parameters} Cost values for remote processing}\vspace{-0.5cm} 2639 %% \begin{center} 2640 %% \begin{tabular}{lcc} 2641 %% \hline 2642 %% \hline 2643 %% {\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ 2644 %% \hline 2645 %% \ippstage{chip} & 150 & 2 \\ 2646 %% \ippstage{camera} & 1700 & 2 \\ 2647 %% \ippstage{warp} & 110 & 2 \\ 2648 %% \ippstage{stack} & 1500 & 6 \\ 2649 %% \ippstage{staticsky} & 7200 & 6 \\ 2650 %% % \ippstage{diff} & 300 & 2 \\ 2651 %% \ippstage{fullforce} & 300 & 2 \\ 2652 %% \hline 2653 %% \end{tabular} 2654 %% \end{center} 2655 %% \end{table} 2656 2657 \begin{deluxetable}{lcc} 2658 \tablecolumns{3} 2659 \tablewidth{0pc} 2660 \tablecaption{Cost values for remote processing} 2661 \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}} 2662 \startdata 2653 2663 \ippstage{chip} & 150 & 2 \\ 2654 2664 \ippstage{camera} & 1700 & 2 \\ … … 2657 2667 \ippstage{staticsky} & 7200 & 6 \\ 2658 2668 % \ippstage{diff} & 300 & 2 \\ 2659 \ippstage{fullforce} & 300 & 2 \\ 2660 \hline 2661 \end{tabular} 2662 \end{center} 2663 \end{table} 2664 2665 %% \begin{deluxetable}{lcc} 2666 %% \tablecolumns{3} 2667 %% \tablewidth{0pc} 2668 %% \tablecaption{Cost values for remote processing} 2669 %% \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}} 2670 %% \startdata 2671 %% \ippstage{chip} & 150 & 2 \\ 2672 %% \ippstage{camera} & 1700 & 2 \\ 2673 %% \ippstage{warp} & 110 & 2 \\ 2674 %% \ippstage{stack} & 1500 & 6 \\ 2675 %% \ippstage{staticsky} & 7200 & 6 \\ 2676 %% % \ippstage{diff} & 300 & 2 \\ 2677 %% \ippstage{fullforce} & 300 & 2 2678 %% \enddata 2679 %% \label{tab:SC processing parameters} 2680 %% \end{deluxetable} 2669 \ippstage{fullforce} & 300 & 2 2670 \enddata 2671 \label{tab:SC processing parameters} 2672 \end{deluxetable} 2681 2673 2682 2674 Once the preparation for the job is complete, the input and output … … 2764 2756 %\input{datasystem.bbl} 2765 2757 2766 \appendix 2767 2768 \section{GPC1 Database Schema Outline} 2769 \label{sec:database.schema} 2770 2771 Table \ref{tab: database schema} provides a list of a majority of the 2772 tables in the GPC1 database schema. Tables that have been excluded 2773 are either no longer used in IPP processing, or are used for rare 2774 reductions that were not used for the PV3 data release. The tables 2775 are grouped into stages, with the primary table and any secondary 2776 tables for that stage listed together, along with the primary key 2777 column that link the tables together. 2778 2779 \note{logical or alphabetical sequence?} 2758 % \appendix 2759 2760 % Table \ref{tab: database schema} provides a list of a majority of the 2761 % tables in the GPC1 database schema. Tables that have been excluded 2762 % are either no longer used in IPP processing, or are used for rare 2763 % reductions that were not used for the PV3 data release. The tables 2764 % are grouped into stages, with the primary table and any secondary 2765 % tables for that stage listed together, along with the primary key 2766 % column that link the tables together. 2780 2767 2781 2768 \end{document} 2782 2769 2783 Figures needed for this document:2784 2785 *2786 2770 \begin{center} 2787 2771 \begin{deluxetable}{lllll}
Note:
See TracChangeset
for help on using the changeset viewer.
