Changeset 40065 for trunk/doc/release.2015/ps1.datasystem/datasystem.tex
- Timestamp:
- Jun 22, 2017, 3:35:25 PM (9 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/release.2015/ps1.datasystem/datasystem.tex
r40032 r40065 95 95 The 1.8m Pan-STARRS\,1 telescope is located on the summit of Haleakala 96 96 on the Hawaiian island of Maui. The wide-field optical design of the 97 telescope \citep{ PS1.optics} produces a 3.3 degree field of view with97 telescope \citep{2004SPIE.5489..667H} produces a 3.3 degree field of view with 98 98 low distortion and minimal vignetting even at the edges of the 99 99 illuminated region. The optics and natural seeing combine to yield … … 102 102 a floor of $\sim 0.7$ arcseconds. 103 103 104 The \PSONE\ camera \citep{ PS1.GPCA}, known as GPC1, consists of a104 The \PSONE\ camera \citep{2009amos.confE..40T}, known as GPC1, consists of a 105 105 mosaic of 60 back-illuminated CCDs manufactured by Lincoln Laboratory. 106 106 The CCDs each consist of an $8\times8$ grid of $\sim 600\times 600$ 107 107 pixel readout regions, yielding an effective $4800\times4800$ 108 108 detector. Initial performance assessments are presented in 109 \cite{ PS1.GPCB}. Routine observations are conducted remotely from the109 \cite{2008SPIE.7014E..0DO}. Routine observations are conducted remotely from the 110 110 Advanced Technology Research Center in Kula, the main facility of the 111 111 University of Hawaii's Institute for Astronomy operations on Maui. … … 123 123 search for potentially hazardous asteroids in our solar system. The 124 124 details of the telescope, surveys, and resulting science publications 125 are described by \cite{ Chambers}.125 are described by \cite{chambers2017}. 126 126 127 127 This is the second in a series of seven papers describing the … … 153 153 %Magnier et al. 2017 (Paper II) 154 154 %Pan-STARRS Data Processing Stages 155 %\citet[][Paper II]{magnier2017 c}155 %\citet[][Paper II]{magnier2017.datasystem} 156 156 %describes how the various data processing stages are organised and implemented 157 157 %in the Imaging Processing Pipeline (IPP), including details of the … … 166 166 %Magnier et al. 2017 (Paper IV) 167 167 %Pan-STARRS Pixel Analysis : Source Detection 168 \citet[][Paper IV]{magnier2017 a}168 \citet[][Paper IV]{magnier2017.analysis} 169 169 describes the details of the source detection and photometry, including point-spread-function and extended source fitting models, and the techniques for ``forced" photometry measurements. 170 170 171 171 %Magnier et al. 2017 (Paper V) 172 172 %Pan-STARRS Photometric and Astrometric Calibration 173 \citet[][Paper V]{magnier2017 b}173 \citet[][Paper V]{magnier2017.calibration} 174 174 describes the final calibration process, and the resulting photometric and astrometric quality. 175 175 … … 190 190 detail each of the analysis steps which may be applied to the images 191 191 and resulting catalogs of detected sources. 192 Section~\ref{sec:postprocessing} discusses the calibration operations 193 and database used for calibration. Section~\ref{sec:operations} 194 discusses the operational infrastructure of the IPP. 195 Section~\ref{sec:hardware} discusses the hardware systems used by the 196 IPP for regular nightly operations and for processing the PV3 data 197 release, with some details on the scale of computing needed to reduce 198 this large number of exposures. Finally, Section~\ref{sec:discussion} 199 presents a discussion of some of the lessons learned in the creation 200 of the IPP, and its utility in reducing data from other cameras and 201 telescopes. 192 Section~\ref{sec:postprocessing} discusses the databasing system used 193 for calibration, the calibration operations, and summarizes the 194 construction of the public release database. 195 Section~\ref{sec:operations} discusses the operational infrastructure 196 of the IPP. Section~\ref{sec:hardware} discusses the hardware systems 197 used by the IPP for regular nightly operations and for processing the 198 PV3 data release, with some details on the scale of computing needed 199 to reduce this large number of exposures. Finally, 200 Section~\ref{sec:discussion} presents a discussion of some of the 201 lessons learned in the creation of the IPP, and its utility in 202 reducing data from other cameras and telescopes. 202 203 203 204 {\color{red} {\em Note: These papers are being placed on arXiv.org to … … 426 427 glitches or hardware crashes. 427 428 428 % \note{start of section needed a re-read}429 430 429 \subsection{Summit copy} 431 430 \label{sec:summitcopy} … … 470 469 is ready to be registered. In this context, `registration' refers to 471 470 the process of adding them to the database listing of known, raw 472 exposures (not to be confused with 'registration' in the sense of471 exposures (not to be confused with `registration' in the sense of 473 472 pixel re-alignment). The result of the registration analysis is an 474 473 entry for each exposure in the \ippdbtable{rawExp} table, and one for … … 525 524 (with the \ippdbcolumn{state} column indicating it needs processing), 526 525 and the associated information listed in the \ippdbtable{rawImfile}, 527 jobs can be spawned for each component OTA. The \ippprog{pantasks} 528 environment managing the jobs attempts to target the processing host 529 to one that should host the OTA, to reduce number of operations done 530 on remote data. In practice, this targeted processing has not had as 531 large of an effect as was originally intended, as the data volume has 526 jobs can be spawned for each component OTA. 527 528 The \ippstage{chip} stage is naturally parallelized by processing data 529 from each of the 60 OTAs independently. Several stages in the IPP 530 analysis are parallelized in a similar fashion; although there are 531 multiple stages that operate on an entire exposure at once, the 532 majority of stages operate on smaller segments of a full exposure, 533 allowing the processing tasks to be spread over the machines in the 534 processing cluster. The \ippprog{pantasks} environment, which manages 535 the jobs, attempts to target the processing to a computer which is 536 assigned to host data for the particular OTA. This capability is 537 implemented to reduce the network I/O load by minimizing the number of 538 operations done on non-local data. In practice, this targeted 539 processing has not had as large of an impact as was originally 540 intended: the data volume and operational details of the hardware has 532 541 reduced the ability of any one node to reliably contain a particular 533 542 OTA. The targeted processing has probably reduced the network load 534 543 somewhat but it has not been as critical of a requirement as 535 544 originally expected. 536 537 \note{keep this paragraph?}538 539 Part of this parallelization is derived from the fact that this camera540 consists of 60 independent orthogonal transfer array (OTA) devices,541 and can therefore be processed simultaneously. Although there are542 multiple stages that operate on an entire exposure at once, the543 majority of stages operate only on smaller segments of a full exposure544 to allow the processing tasks to be spread over the machines in the545 processing cluster.546 545 547 546 %% In the \ippstage{chip} stage, … … 581 580 this analysis, removing the need to write out and re-read the image 582 581 data. The details of the detection and characterization of the 583 sources in the image are provided in \citet{magnier2017 b}.582 sources in the image are provided in \citet{magnier2017.analysis}. 584 583 585 584 The results of the image processing are then written to disk, … … 657 656 used to generate synthetic w-band photometry for areas where no 658 657 PS1-based calibrated w-band photometry is available. For more 659 details, see \cite{magnier2017 c}. The result of these calibrations is658 details, see \cite{magnier2017.calibration}. The result of these calibrations is 660 659 stored as a single multi-extension FITS table containing the results 661 660 from each OTA as a separate extension. … … 718 717 pixels. These projections are further broken down into ``skycells'' 719 718 that form a $10\times{}10$ grid within the projection, with an overlap 720 region of 60 "between adjacent skycells to ensure that objects are not719 region of 60\arcsec\ between adjacent skycells to ensure that objects are not 721 720 split on all images. 722 721 … … 778 777 For the PV3 processing of the Medium Deep fields, stacks have been 779 778 generated for the nightly groups and for the full depth using all 780 exposures, producing ``deep stacks''. In addition, a 'best seeing'779 exposures, producing ``deep stacks''. In addition, a `best seeing' 781 780 set of stacks have been produced \note{using image quality cuts to be 782 781 described}. We have also generated out-of-season stacks for the … … 816 815 \ippstage{stack} processing. As this completes all processing for the 817 816 entry, no \ippmisc{advance} job is required. 818 819 % \note{end of section needed a re-read}820 817 821 818 \subsection{Stack Photometry} … … 839 836 The input images are passed to the \ippprog{psphotStack} program, 840 837 which does the analysis. The stack photometry algorithms are 841 described in detail in \cite{magnier2017 b}. In short, sources are838 described in detail in \cite{magnier2017.analysis}. In short, sources are 842 839 detected in all 5 filter images down to the $5\sigma$ significance. 843 840 The collection of detected sources is merged into a single master … … 859 856 sources based on the PSF model; aperture like parameters such as the 860 857 Petrosian flux and radius; the convolved galaxy model fits; and the 861 radial aperture measurements. \note{is this list complete?} Once the 862 photometry is complete, a row is added to the 863 \ippdbtable{staticskyResult} table with basic statistics from the 864 analysis. 858 radial aperture measurements. Once the photometry is complete, a row 859 is added to the \ippdbtable{staticskyResult} table with basic 860 statistics from the analysis. 865 861 866 862 The stack photometry output catalogs are re-calibrated for both … … 877 873 for the \ippstage{camera} and \ippstage{stack} calibration stages. 878 874 Upon completion, the analysis statistics are written to the 879 \ippdbtable{skycalResult} table. \note{Any difference in output formats?}875 \ippdbtable{skycalResult} table. 880 876 881 877 \subsection{Forced Warp Photometry} … … 951 947 952 948 In this program, the positions of sources are loaded from the input 953 catalog. PSF stars are pre-identified \note{how?} and a PSF model954 generated for each \ippstage{warp} image based on those stars, using 955 the same stars for all warps to the extent possible (PSF stars which 956 are excessively masked on a particular image are not used to model the 957 PSF). The PSF model is fitted to all of the known source positions in 958 the warp images. Aperture magnitudes, Kron magnitudes, and moments 959 a re also measured at this stage for each warp. Note that the flux960 measurement for a faint, but significant, source from the stack image961 may be at a low significance (less than the $5\sigma$ criterion used 962 when the photometry is not run in this forced mode) in any individual 963 warp image; the flux may even be negative for specific warps. When 964 combined together, these low-significance measurements will result in 965 a signficant measurement as the signal-to-noise increases by the966 square root of the number of measurements. \note{The individual warp 967 measurements are combined together to generate averages values within 968 DVO.} 949 catalog. PSF stars are pre-identified from the stack image and a PSF 950 model generated for each \ippstage{warp} image based on those stars, 951 using the same stars for all warps to the extent possible (PSF stars 952 which are excessively masked on a particular image are not used to 953 model the PSF). The PSF model is fitted to all of the known source 954 positions in the warp images. Aperture magnitudes, Kron magnitudes, 955 and moments are also measured at this stage for each warp. Note that 956 the flux measurement for a faint, but significant, source from the 957 stack image may be at a low significance (less than the $5\sigma$ 958 criterion used when the photometry is not run in this forced mode) in 959 any individual warp image; the flux may even be negative for specific 960 warps. When combined together, these low-significance measurements 961 will result in a signficant measurement as the signal-to-noise 962 increases by the square root of the number of measurements. The 963 individual warp measurements are combined together to generate 964 averages values within DVO. 969 965 970 966 Upon completion of the forced photometry (for point sources as well as … … 976 972 analysis measurements into a single value. The output catalogs listed 977 973 in the \ippdbtable{fullForceResult} table are passed to the 978 \ippprog{psphotFullForceSummary} to do this averaging. \note{describe 979 what is done} When this completes, an entry is added to the 980 \ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} entry 981 is marked as completed. 974 \ippprog{psphotFullForceSummary} to do this averaging. When this 975 completes, an entry is added to the \ippdbtable{fullForceSummary}, and 976 the \ippdbtable{fullForceRun} entry is marked as completed. 982 977 983 978 \subsubsection{Forced Galaxy Models} 984 \note{CZW: is this the appropriate place for this section?}985 979 \note{too much detail in this section; balance relative to psphot} 986 980 … … 1020 1014 $\chi^2$ grid can then be made by combining each grid point across the 1021 1015 inputs. The combined $\chi^2$ for a single grid point is simply the 1022 sum of all $\chi^2$ values at that point. If, for a single \ippstage{warp} 1023 image, the galaxy model is excessively masked, then that image will be 1024 dropped for all grid points for that galaxy. The reduced $\chi^2$ 1025 values can be determined by tracking the total number of pixels 1026 used across all inputs to generate the combined $\chi^2$ values. From 1027 the combined grid of $\chi^2$ values, the point in the grid with the 1028 minimum $\chi^2$ is found. Quadratic interpolation is used to 1029 determine the major, minor axis values for the interpolated minimum 1030 $\chi^2$ value. The errors on these two parameters is then found by 1031 determining the contour at which the \note{reduced?} $\chi^2$ 1032 increases by 1. 1016 sum of all $\chi^2$ values at that point. If, for a single 1017 \ippstage{warp} image, the galaxy model is excessively masked, then 1018 that image will be dropped for all grid points for that galaxy. The 1019 reduced $\chi^2$ values can be determined by tracking the total number 1020 of pixels used across all inputs to generate the combined $\chi^2$ 1021 values. From the combined grid of $\chi^2$ values, the point in the 1022 grid with the minimum $\chi^2$ is found. Quadratic interpolation is 1023 used to determine the major, minor axis values for the interpolated 1024 minimum $\chi^2$ value. The errors on these two parameters is then 1025 found by determining the contour at which the $\chi^2$ increases by 1. 1033 1026 1034 1027 Thus the \ippstage{fullforce} galaxy analysis uses the PSF information 1035 1028 from each \ippstage{warp} to determine a best set of convovled galaxy 1036 1029 models for each object in the \ippstage{skycal} catalog. 1030 1037 1031 \note{discuss the subset of galaxy models and objects}. 1038 1032 … … 1110 1104 \begin{table}[hb] 1111 1105 \begin{center} 1112 \caption{DVO Database Tables\label{tab:DVO tables}}1106 \caption{DVO Database Tables\label{tab:DVO_schema}} 1113 1107 \begin{tabular}{ll} 1114 1108 \hline … … 1238 1232 processed by the IPP may also be included similarly in a DVO database. 1239 1233 Measurements from other sources, such as SDSS, 2MASS, or WISE, can 1240 also be included in this table (see \S\ref{sec:other.photometry}.1234 also be included in this table. 1241 1235 1242 1236 The \ippdbtable{Measure} table includes the instrumental magnitudes … … 1278 1272 1279 1273 \subsubsubsection{Object Tables} 1274 \label{sec:object} 1280 1275 1281 1276 % object -> detection … … 1353 1348 these photometric distance modulus measurements are not extremely 1354 1349 precise (see below), they provide a constraint on the distance is used 1355 in our analysis of the astrometry \citep[][see]{magnier2017 a}.1350 in our analysis of the astrometry \citep[][see]{magnier2017.calibration}. 1356 1351 1357 1352 In the \ippdbtable{Measure} table, there are three fields which … … 1410 1405 determined by the photometry calibration analysis and the astrometric 1411 1406 flat-field corrections determined by the astrometry calibration 1412 analysis \citep[][see]{magnier2017 a}.1407 analysis \citep[][see]{magnier2017.calibration}. 1413 1408 1414 1409 \subsubsection{Sky Partition} … … 1457 1452 machines that contain partition data. 1458 1453 1459 \note{is the use of the term 'partition host' consistent in this paper1454 \note{is the use of the term `partition host' consistent in this paper 1460 1455 and the calibration paper?} 1461 1456 … … 1499 1494 The FITS binary table compression scheme uses a strategy similar to 1500 1495 that used for FITS image compression (\note{REF}). The binary tabular 1501 data is compressed and stored in the 'HEAP' section of the FITS table1496 data is compressed and stored in the `HEAP' section of the FITS table 1502 1497 extension, with pointers to the compressed data stored in the regular 1503 1498 data section. Each column in the FITS table is compressed as one (or … … 1505 1500 column format (e.g., TFORM1) are replaced with keywords which describe 1506 1501 the location and size of the compressed data in the HEAP section; the 1507 information about the uncompressed data is moved to a keyword with 'Z'1502 information about the uncompressed data is moved to a keyword with `Z' 1508 1503 prepended (e.g., ZFORM1) and an additional field is added to define 1509 1504 the compression algorithm (e.g., ZCTYP1). The column names (e.g., … … 1594 1589 astrometric and photometric calibrations can be calculated. The 1595 1590 details of the calibration analysis are discussed in 1596 \cite{magnier2017 c}. We present a brief summary here.1591 \cite{magnier2017.calibration}. We present a brief summary here. 1597 1592 1598 1593 Astrometric calibration consists of measuring and correcting … … 1607 1602 a function of position in the camera (essentially an astrometric 1608 1603 flat-field correction), as a function of the brightness of the star 1609 (the so-called Koppenh\"offer effect, see~\ref{magnier2017 c}), and as1604 (the so-called Koppenh\"offer effect, see~\ref{magnier2017.calibration}), and as 1610 1605 a function of airmass and color (Differential chromatic refraction). 1611 1606 Once the systematic errors have been measured, they are applied back … … 1626 1621 exposures which were believed to be obtained in photometric 1627 1622 conditions. This process, called `\"ubercal', is described in detail 1628 by \cite{ ubercal} for the first (PV1) version. In brief, photometric1623 by \cite{2012ApJ...756..158S} for the first (PV1) version. In brief, photometric 1629 1624 periods, with time-scales of at least \note{half of a night}, are 1630 1625 identified by a combination of automatic analysis and manual … … 1658 1653 flat-field correction addresses photometric variations due to spatial 1659 1654 variations in the PSF due to the optics and low-level effects on the 1660 chips \citep[see][]{magnier2017 c}. After the systematic corrections1655 chips \citep[see][]{magnier2017.calibration}. After the systematic corrections 1661 1656 have been determined and applied back to the database, a final 1662 1657 relative photometry analysis pass is performed. … … 1756 1751 1757 1752 Pantasks repeatedly checks each task in an attempt to generate a new 1758 command: we say pantasks attempts to 'execute' the task in each of1753 command: we say pantasks attempts to `execute' the task in each of 1759 1754 these attempts. Tasks may specify the time between execution 1760 1755 attempts, with a 1 second default. … … 1777 1772 1778 1773 Within the \ippprog{task.exec} macro, the command to be run must be 1779 defined with the function 'command'. Once the \ippprog{task.exec}1774 defined with the function `command'. Once the \ippprog{task.exec} 1780 1775 macro exits successfully, the defined command is the added to the list of jobs 1781 1776 to be run within the UNIX environment. Jobs may be run in one of two 1782 1777 ways: locally or via the parallel processing system. The task, or the 1783 \ippprog{task.exec} macro, uses the 'host' command to define how to1784 run the job. If the host is set to 'local', then the job is run in1778 \ippprog{task.exec} macro, uses the `host' command to define how to 1779 run the job. If the host is set to `local', then the job is run in 1785 1780 the background by pantasks itself (using the C \code{execvp} 1786 1781 function). Otherwise, the job is sent to the parallel processing 1787 1782 system to be run on another machine within the cluster. If the host 1788 is set to the special value 'anyhost', then the parallel processing1783 is set to the special value `anyhost', then the parallel processing 1789 1784 system is allowed to choose the processing computer arbitrarily. Any 1790 1785 other value is taken to be the DNS name of the computer on which this … … 1798 1793 When the \ippprog{task.exec} macro is run, the code may choose (e.g., 1799 1794 based on tests of some global variables) to exit the macro with an 1800 error condition, e.g., with the 'break' command. In this1795 error condition, e.g., with the `break' command. In this 1801 1796 circumstance, no job is produced by the task. The task will be tried 1802 1797 again the next time it is executed. This feature allows for the user … … 1813 1808 online user guide?} 1814 1809 1815 The option 'npending' may be used to limit the number of jobs which1810 The option `npending' may be used to limit the number of jobs which 1816 1811 are simultaneously executed for a specific task. For example, some 1817 1812 classes of jobs should only be run one-at-a-time because they are not 1818 1813 protected against collisions or they may overload a resource. The use 1819 of 'npending' allows these situations to be handled cleanly within1814 of `npending' allows these situations to be handled cleanly within 1820 1815 pantasks (avoiding cumbersome coding within with program or supporting 1821 1816 script). 1822 1817 1823 The option 'nmax' limits the total number of jobs which a task1818 The option `nmax' limits the total number of jobs which a task 1824 1819 generates. This option may be useful in cases where 1825 1820 \ippprog{pantasks} is used to perform a limited set of operations. 1826 1821 \note{do we actually use this in IPP?} 1827 1822 1828 The option 'trange' allows the user to restrict the time period during1823 The option `trange' allows the user to restrict the time period during 1829 1824 which the specific tasks is executed. This option is given with a 1830 1825 start and an end time for the limiting time range. These times may be … … 1841 1836 ranges may be specified \note{how are they evaluated?} 1842 1837 1843 The option \code{nice} specifies the 'nice' level at which the job is1838 The option \code{nice} specifies the `nice' level at which the job is 1844 1839 run when it is executed. The parallel processing system must respect 1845 1840 this concept. … … 1918 1913 pantasks receives this completion information, the jobs are removed 1919 1914 from the list managed by pcontrol. Thus pcontrol maintains at most a 1920 modest list of jobs which are 'in flight', leaving all interpretation1915 modest list of jobs which are `in flight', leaving all interpretation 1921 1916 work to pantasks. 1922 1917 … … 2461 2456 isolation of source objects is included, providing the organization of 2462 2457 detections that is used in the \ippprog{psphot} photometry analysis 2463 \citep{magnier2017 c}. The PSF matching required for \ippstage{stack}2458 \citep{magnier2017.analysis}. The PSF matching required for \ippstage{stack} 2464 2459 and \ippstage{diff} stage image combinations is as well. The 2465 2460 unification of configuration options between config files on disk and … … 2685 2680 column that link the tables together. 2686 2681 2687 \note{logical or alphabetical sequence? alignment is broken}2682 \note{logical or alphabetical sequence?} 2688 2683 2689 2684 \begin{center} … … 2739 2734 \begin{verbatim} 2740 2735 MAJOR TODO ITEMS: 2736 * add figure showing DVO schema relationships 2741 2737 * re-read and trim details as needed (referring to the other papers) 2742 2738 * add some specific numbers (data volume, processing times, etc) 2739 * where is the smf/cmf format defined? psphot? 2740 * where is the GPC1 naming convention discussed? 2741 * where are the flat-field seasons listed (magnier2017.calibration?) 2743 2742 \end{verbatim} 2744 2743 2745 2744 \end{document} 2746 2747 Figures needed for this document:2748 2749 *
Note:
See TracChangeset
for help on using the changeset viewer.
