- Timestamp:
- Jan 26, 2019, 11:47:09 AM (7 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/release.2015/ps1.datasystem/datasystem.tex
r40599 r40612 82 82 \begin{abstract} 83 83 84 The Pan-STARRS Image Processing Pipeline performs the processing84 The Pan-STARRS Data Processing System is responsible for the steps 85 85 needed to downloaded, archive, and process all images obtained by the 86 Pan-STARRS telescopes. This article describes the overall data 87 analysis system. 86 Pan-STARRS telescopes, including real-time detection of transient 87 sources such as supernovae and moving objects including potentially 88 hazardous asteroids. With a nightly data volume of up to 4 terabytes 89 and an archive of over 4 petabytes of raw imagery, Pan-STARRS is 90 solidly in the realm of Big Data astronomy. The full data processing 91 system consists of several subsystems covering the wide range of 92 necessary capabilities. This article describes the Image Processing 93 Pipeline and its connections to both the summit data systems and the 94 outward-facing systems downstream. The latter include the Moving 95 Object Processing System (MOPS) \& the public database: the Published 96 Science Products Subsystem (PSPS). 88 97 89 98 \end{abstract} … … 130 139 Release 1 (DR1) on 16 December 2016. DR1 contains the results of the 131 140 third full reduction of the Pan-STARRS $3\pi$ Survey archival data, 132 identified as PV3. Previous reductions \citep[PV0, PV1, PV2;133 see][]{magnier2017.datasystem} were used internally for pipeline134 optimization and the development of the initial photometric and 135 astrometric reference catalog \citep{magnier2017.calibration}. The 136 products from these reductions were not publicly released, but have137 been used to produce a wide range of scientific papers from the 138 Pan-STARRS 1 Science Consortium members \citep{chambers2017}. DR1 139 contained only average information resulting from the many individual 140 images obtained by the $3\pi$ Survey observations. A second data 141 release, DR2, was made available \note{20 January 2019}. DR2 provides 142 measurements from all of the individual exposures, and include an 143 improved calibration of the PV3 processingof that dataset.141 identified as PV3. Previous reductions (PV0, PV1, PV2) were used 142 internally for pipeline optimization and the development of the 143 initial photometric and astrometric reference catalog 144 \citep{magnier2017.calibration}. The products from these reductions 145 were not publicly released, but have been used to produce a wide range 146 of scientific papers from the Pan-STARRS 1 Science Consortium members 147 \citep{chambers2017}. DR1 contained only average information 148 resulting from the many individual images obtained by the $3\pi$ 149 Survey observations. A second data release, DR2, was made available 150 28 January 2019. DR2 provides measurements from all of the individual 151 exposures, and include an improved calibration of the PV3 processing 152 of that dataset. 144 153 145 154 This is the second in a series of seven papers describing the 146 Pan-STARRS1 Surveys, the data reduction tech iques and the resulting155 Pan-STARRS1 Surveys, the data reduction techniques and the resulting 147 156 data products. This paper (Paper II) presents a description of the 148 157 Pan-STARRS data handling systems, with an emphasis on the Image … … 198 207 %Pan-STARRS 1 Database and Data Products 199 208 \citet[][Paper VI]{flewelling2017} 200 describe the details of the resulting catalog data and its organization in the Pan-STARRS database. 209 describe the details of the resulting catalog data and its 210 organization in the Pan-STARRS database. 201 211 202 212 %Huber et al. 2017 (Paper VII) … … 218 228 used by the IPP for regular nightly operations and for processing the 219 229 PV3 data release, with some details on the scale of computing needed 220 to reduce this large number of exposures. Finally, 221 Section~\ref{sec:discussion} presents a discussion of some of the 222 lessons learned in the creation of the IPP, and its utility in 223 reducing data from other cameras and telescopes. 230 to reduce this large number of exposures. 231 232 % Finally, 233 % Section~\ref{sec:discussion} presents a discussion of some of the 234 % lessons learned in the creation of the IPP, and its utility in 235 % reducing data from other cameras and telescopes. 224 236 225 237 %% {\color{red} {\em Note: These papers are being placed on arXiv.org to … … 263 275 ingests the calibrated measurements from the IPP, MOPS, and others 264 276 and generates a high-availability database with web-based 265 interactions for public consumption \cite t[][]{flewelling2017}.277 interactions for public consumption \citep[][]{flewelling2017}. 266 278 267 279 \end{itemize} 268 280 Management of the above set of analysis stages takes place at the IfA 269 281 within the scope of responsibility of the Pan-STARRS Observatory. 270 Across the wider Pan-STARRS coll oboration(s), additional data analysis282 Across the wider Pan-STARRS collaboration(s), additional data analysis 271 283 operations are performed to support science results. These 272 284 collaboration-wide analysis operations range from those which are … … 331 343 Pan-STARRS has performed several large-scale reprocessings of both the 332 344 Medium Deep and $3\pi$ Survey data for internal consumption. For the 333 $3\pi$ Survey data, we identify these large-scale reprocessings as 334 PV1, PV2, and PV3, with PV3 the analysis used for the first public 335 data release, DR1. We also refer to the nightly science analysis of 336 the data as PV0. For these reprocessing stages, the standard steps of 337 \ippstage{chip} through \ippstage{warp}, plus \ippstage{stack} and 338 \ippstage{diff} are performed, starting from raw data, usually using a 339 single homogenous version of the data analysis procedures. PV2 was a 340 special case in which we started from the camera level products of PV1 341 to speed up the turn-around to the community. In addition to the 342 analysis stages listed above which are shared with the nightly 343 processing, these large-scale reprocessing analyses include additional 344 processing steps. A more detailed photometric analysis is performed 345 on the stacks, including morphological analysis appropriate to 346 galaxies. The results of the stack photometry analysis are used to 347 drive a forced-photometry analysis of the warp images. These analysis 348 steps are discussed in detail by 349 \citet[][]{magnier2017.analysis}. The data products from the 350 camera, stack, and forced-warp photometry analysis stages 351 are ingested into the internal calibration database (DVO, the Desktop 352 Virtual Observatory) and used for photometric and astrometric 353 calibrations \citet[see Section~\ref{sec:DVO} and][]{magnier2017.calibration}. 345 $3\pi$ Survey data, we identify these large-scale reprocessings as PV1 346 (Processing Version 1), PV2, and PV3, with PV3 the analysis used for 347 the first public data release, DR1. We also refer to the nightly 348 science analysis of the data as PV0. For these reprocessing stages, 349 the standard steps of \ippstage{chip} through \ippstage{warp}, plus 350 \ippstage{stack} and \ippstage{diff} are performed, starting from raw 351 data, usually using a single homogeneous version of the data analysis 352 procedures. PV2 was a special case in which we started from the 353 camera level products of PV1 to speed up the turn-around to the 354 community. In addition to the analysis stages listed above which are 355 shared with the nightly processing, these large-scale reprocessing 356 analyses include additional processing steps. A more detailed 357 photometric analysis is performed on the stacks, including 358 morphological analysis appropriate to galaxies (model fits, Kron and 359 Petrosian aperture photometry, etc). The results of the stack 360 photometry analysis are used to drive a forced-photometry analysis of 361 the warp images. These analysis steps are discussed in detail by 362 \citet[][]{magnier2017.analysis}. The data products from the camera, 363 stack, and forced-warp photometry analysis stages are ingested into 364 the internal calibration database (DVO, the Desktop Virtual 365 Observatory) and used for photometric and astrometric calibrations 366 \citet[see Section~\ref{sec:DVO} and][]{magnier2017.calibration}. 354 367 355 368 \subsection{Data Access and Distribution} … … 379 392 380 393 \begin{table*} 381 \caption{GPC1 Database Schema Outline} %\vspace{-0.5cm}394 \caption{GPC1 Database Schema Outline} 382 395 \begin{center} 396 \footnotesize 383 397 \begin{tabular}{llll} 384 398 \hline … … 386 400 {\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} \\% & {\bf Notes} \\ 387 401 %%D \begin{deluxetable}{llll} 388 \footnotesize389 402 %%D \tablecolumns{5} 390 403 %%D \tablewidth{0pc} … … 741 754 The guess astrometry is used to match the reference catalog to the 742 755 observed stellar positions in the focal plane coordinate system 743 \citep[see][]{magnier2017.calibration}. 756 \citep[see][]{magnier2017.calibration}. Early on in the PS1SC 757 mission, the nightly processing (PV0) used a reference catalog based 758 on a combination of external catalogs (2MASS, Tycho, USNO). Later, 759 reference catalogs based on Pan-STARRS data was used. For the $3\pi$ PV3 analysis, 760 the reference catalog was based on Pan-STARRS data from the PV2 761 analysis \citep[see][for more details]{magnier2017.calibration}. 744 762 745 763 Once an acceptable match is found, the astrometric calibration of the 746 764 individual chips is performed, including a fit to a single model for 747 the distortion introduced by the camera optics. After the astrometic 748 analysis is completed, the photometric calibration is determined using 749 the final match to the reference catalog. At this stage, 750 pre-determined color terms may be included to convert the reference 751 photometry to an appropriate photometric system. For PS1, this is 752 used to generate synthetic w-band photometry for areas where no 753 PS1-based calibrated w-band photometry is available. For more 754 details, see \cite{magnier2017.calibration}. The result of these 755 calibrations is stored as a single multi-extension FITS table 756 containing the results from each OTA as a separate extension. 765 the distortion introduced by the camera optics. The astrometric model 766 includes a set of 3rd order polynomials for the transformations from the chip 767 coordinate system to the camera focal plane coordinate system and a 768 single additional 3rd order polynomial transformation from the camera focal 769 plane coordinate system to the tangent plane of a tangent projection. 770 For the $3\pi$ PV3 analysis, the typical astrometric residuals are in 771 the range of 20 - 30 milliarcseconds, sufficient to match observations 772 of the same objects between different exposures. There are, however, 773 inevitable outliers. Certain chips occasionally have systematically worse 774 astrometry, with OTA XY17 notably poor in this respect. 775 776 After the astrometic analysis is completed, the photometric 777 calibration is determined using the final match to the reference 778 catalog. A single photometric zero point is determined for each 779 exposure, with the airmass term fixed to the nominal linear slope for 780 each filter. No color terms are measured between the observed 781 photometry and the reference photometry. However, at this stage, 782 pre-determined color terms may be used to transform the reference 783 photometry to an appropriate photometric system. For the PS1 nightly 784 processing, the reference catalog does not include \wps\ photometry, 785 so a fixed color transformation is used to generate synthetic w-band 786 photometry from the \rps\ \& \ips\ photometry. For more details, see 787 \cite{magnier2017.calibration}. The result of these calibrations is 788 stored as a single multi-extension FITS table containing the results 789 from each OTA as a separate extension. 757 790 758 791 In addition to the astrometric and photometric calibrations, the … … 842 875 \ippdbtable{warpSkyCellMap} table in the database, which contains a 843 876 row for each skycell and OTA that overlap. Each skycell may contain 844 contributions from multiple OTAs. 877 contributions from multiple OTAs; since they are similar in size, in a 878 typical situation the warp is constructed from 4-6 neighboring OTAs. 845 879 846 880 Once this mapping has been defined, jobs to warp the pixels onto each … … 914 948 The \ippstage{stack} jobs pass the information about the input images 915 949 and catalogs to the \ippprog{ppStack} program, which performs the 916 image combinations. See~\cite{waters2017} for details on the stack 917 combination algorithm. In addition to the standard image, mask, and 918 variance produced at other stages, additional images are constructed 919 with information about the contributions to each pixel. A number 920 image contains the number of input exposures used for each pixel, 921 along with an exposure time map, and a weighted exposure time map that 922 scales the exposure time based on the relative variance of each input. 923 These images for the $3\pi$ analysis are currently available from the 924 MAST image extraction tools at STSci. 950 image combinations. Input warps are combined based on a weighting 951 defined by the median variance for each image; see~\cite{waters2017} 952 for details on the stack combination algorithm. In addition to the 953 standard image, mask, and variance produced at other stages, 954 additional images are constructed with information about the 955 contributions to each pixel. A number image contains the number of 956 input exposures used for each pixel, along with an exposure time map, 957 and a weighted exposure time map that scales the exposure time based 958 on the relative variance of each input. These images for the $3\pi$ 959 analysis are currently available from the MAST image extraction tools 960 at STScI. 925 961 926 962 Upon completing the generation of these images, a row is added to the … … 961 997 that faint, high-redshift quasars may be detected in \yps{} band only. 962 998 Sources detected only in \yps{} band are therefore more likely to have 963 a higher false-positive rate than the other stack sources. 999 a higher false-positive rate than the other stack sources. The 1000 parameters of the PSF model are allowed to vary with position in the 1001 skycell. The PSF model is also used to convolve the analytical galaxy 1002 models, which are the fitted to the observed flux distributions. 1003 Galaxy models include S\'ersic, DeVaucouleur, and Exponential 1004 profiles. 964 1005 965 1006 The stack photometry output files consist of a set of FITS table … … 1010 1051 particularly severe for the Pan-STARRS $3\pi$ survey stacks due to the 1011 1052 combination of the substantial mask fraction of the individual input 1012 exposures, the large in strinsic image quality variations within a1053 exposures, the large intrinsic image quality variations within a 1013 1054 single exposure, and the wide range of image quality conditions under 1014 1055 which data were obtained and used to generate the $3\pi$ PV3 stacks. … … 1052 1093 skycell and filter as a single unit within the processing database, 1053 1094 while individual warps are processed individually in parallel as 1054 separate processing jobs. 1095 separate processing jobs. A separate PSF model is determine for each 1096 of the warp images so that the combined measurement is reliable. 1055 1097 1056 1098 When processing is queued for this stage, an entry is added to the … … 1107 1149 analysis measurements into a single value. The output catalogs listed 1108 1150 in the \ippdbtable{fullForceResult} table are passed to the 1109 \ippprog{psphotFullForceSummary} to do this averaging. When this 1110 completes, an entry is added to the \ippdbtable{fullForceSummary}, and 1111 the \ippdbtable{fullForceRun} entry is marked as completed. 1151 \ippprog{psphotFullForceSummary} to calculate the averages of the 1152 individual warp measurements, weighted by the signal-to-noise of the 1153 flux measurements. When this analysis completes, an entry is added to 1154 the \ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} 1155 entry is marked as completed. 1156 1157 % flux averaging takes place in psphotFullForceSummaryReadout.c:409 1112 1158 1113 1159 \subsection{Difference Images} … … 1115 1161 1116 1162 Two of the primary science drivers for the Pan-STARRS system are the 1117 search hazardous asteroids and the search for Type Ia supernovae to1163 search for hazardous asteroids and the search for Type Ia supernovae to 1118 1164 measure the history of the expansion of the universe. Both of these 1119 1165 projects require the discovery of faint, transient source in the … … 1214 1260 % intro 1215 1261 The Pan-STARRS IPP uses an internal database system, distinct from the 1216 public ally visible database system, to determine the association1262 publicly visible database system, to determine the association 1217 1263 between multiple detections of the same astronomical object and as 1218 1264 part of the astrometric and photometric calibration process. This … … 1228 1274 astronomical objects; 2) measurements of those objects (from which the 1229 1275 average properties are derived); 3) properties of the images which 1230 provided some or all of the measu ements. In addition, certain1276 provided some or all of the measurements. In addition, certain 1231 1277 metadata tables define general features of the database. 1232 1278 Table~\ref{tab:DVO_schema} lists the full collection of database … … 1236 1282 %illustrates the schematic relationship between these types of 1237 1283 %measurements. 1238 1239 \begin{figure*}[htbp]1240 \begin{center}1241 \includegraphics[width=\hsize,clip]{skypartition.png}1242 \caption{\label{fig:sky.partition} Level 3 sky paritioning. The1243 blue grid shows the outlines of the different regions assigned to1244 separate tables in the sky partitioning scheme. The Galactic1245 plane is shown as a solid red line while the ecliptic is shown in1246 green. This organization of the sky duplicates that used by the1247 HST Guide Star Catalog \citep{1988IAUS..133..239J}.1248 }1249 \end{center}1250 \end{figure*}1251 1284 1252 1285 In the most basic implementation, a collection of measurements for … … 1376 1409 The \ippdbtable{Galphot} table stores the results of the forced galaxy 1377 1410 fitting analysis for each object that has been measured. This table 1378 contains one row per filter and model type (S ersic, Exponential, or1411 contains one row per filter and model type (S\'ersic, Exponential, or 1379 1412 DeVaucouleur) if forced galaxy models have been calculate for the 1380 1413 object. … … 1514 1547 \label{sec:SkyPartition} 1515 1548 1549 \begin{figure*}[htbp] 1550 \begin{center} 1551 \includegraphics[width=\hsize,clip]{skypartition.png} 1552 \caption{\label{fig:sky.partition} Level 3 sky paritioning. The 1553 blue grid shows the outlines of the different regions assigned to 1554 separate tables in the sky partitioning scheme. The Galactic 1555 plane is shown as a solid red line while the ecliptic is shown in 1556 green. This organization of the sky duplicates that used by the 1557 HST Guide Star Catalog \citep{1988IAUS..133..239J}. 1558 } 1559 \end{center} 1560 \end{figure*} 1561 1516 1562 Tables within DVO containing information about astronomical objects 1517 1563 are partitioned on the basis of position in the sky: objects within a … … 1535 1581 subdivides these declination bands in the RA direction, with spacing 1536 1582 related to the stellar density. Level 3 divides these RA chunks into 1537 4 - 8 smaller partitions . This level exactly matches the HST GSC1538 l ayout, and uses the same naming convention to identify the1539 partitions: \code{n0000/0000}, etc. Level 4 further divides these 1540 regions by a factor of 16. In the \ippdbtable{SkyTable}, a region at 1541 one level has a pointer to its parent region (the one which contains1542 it) and a sequence pointing to its children (regions it contains). 1543 The \ippdbtable{SkyTable} enables fast lookups of the on-disk 1544 partitions which map to a specific coordinate on the sky. In general, 1545 a single DVO will have the full sky represented with tables at a 1546 s ingle level, although it is possible for mixed levels to be used.1547 For the PV3 master database, the partitioning is at Level 4, resulting 1548 in \approx 150,000 regions to cover the full sky, of which \approx 1549 110,000 are used for the PV3 $3\pi$ data. The densest portions of the 1550 bulge contain at most \approx 300,000 astronomical objects in the 1551 database files, with an associated maximum of \approx 30 million1552 measurements in these files. With the compression scheme described 1553 below, the largest database files are \approx 3GB, which can be loaded 1554 into memory in 30 seconds on the processing machines that contain 1555 partition data.1583 4 - 8 smaller partitions (see Figure~\ref{fig:sky.partition}). This 1584 level exactly matches the HST GSC layout, and uses the same naming 1585 convention to identify the partitions: \code{n0000/0000}, etc. Level 4 1586 further divides these regions by a factor of 16. In the 1587 \ippdbtable{SkyTable}, a region at one level has a pointer to its 1588 parent region (the one which contains it) and a sequence pointing to 1589 its children (regions it contains). The \ippdbtable{SkyTable} enables 1590 fast lookups of the on-disk partitions which map to a specific 1591 coordinate on the sky. In general, a single DVO will have the full 1592 sky represented with tables at a single level, although it is possible 1593 for mixed levels to be used. For the PV3 master database, the 1594 partitioning is at Level 4, resulting in \approx 150,000 regions to 1595 cover the full sky, of which \approx 110,000 are used for the PV3 1596 $3\pi$ data. The densest portions of the bulge contain at most 1597 \approx 300,000 astronomical objects in the database files, with an 1598 associated maximum of \approx 30 million measurements in these files. 1599 With the compression scheme described below, the largest database 1600 files are \approx 3GB, which can be loaded into memory in 30 seconds 1601 on the processing machines that contain partition data. 1556 1602 1557 1603 % parallel partitions … … 1561 1607 and the location of the database partition on the disks of that 1562 1608 machine. The \ippdbtable{SkyTable} contains elements to define by ID 1563 the par ition host to which a set of tables has been assigned.1609 the partition host to which a set of tables has been assigned. 1564 1610 Operations which query the database, or perform other operations on 1565 1611 the database, are aware of the partitioning scheme and will launch … … 1574 1620 When the parallel partitioning for a DVO instance is defined, the 1575 1621 tables are randomly assigned to the partition hosts. As a result, 1576 queries which span more than a single par ition are likely to spread1622 queries which span more than a single partition are likely to spread 1577 1623 the I/O load across a large number of machines. This parallelization 1578 1624 is critical to querying and manipulating the enormous database on a … … 1605 1651 The \ippdbtable{Measure} table, containing the detections of objects 1606 1652 from individual exposures or stack, or the (potentially 1607 non-sign ficant) measurements from a warp, uses the 32-bit integer1653 non-significant) measurements from a warp, uses the 32-bit integer 1608 1654 fields \ippdbcolumn{detID} and \ippdbcolumn{imageID} to uniquely 1609 1655 identify each entry. The \ippdbcolumn{imageID} is the running … … 1653 1699 sequence within the image to form a single unique 64-bit integer value. 1654 1700 For detections from the stack images, the MJD is not unique, so a 1655 different rubric kis used to define IDs for those detections. The1701 different rubric is used to define IDs for those detections. The 1656 1702 field \ippdbcolumn{XstackDetectID} (where '\ippdbcolumn{X}' is one of 1657 1703 g,r,i,z,y) is constructed from the GPC1 stack ID … … 1684 1730 describe the location and size of the compressed data in the HEAP 1685 1731 section; the information about the uncompressed data is moved to a 1686 keyword with ``Z'' pre pended (e.g., ZFORM1) and an additional field is1732 keyword with ``Z'' pre-pended (e.g., ZFORM1) and an additional field is 1687 1733 added to define the compression algorithm (e.g., ZCTYP1). The column 1688 1734 names (e.g., TTYPE1) and units (e.g., TUNIT1) are retained in their … … 1694 1740 compressed in turn (this must be the same for all columns). 1695 1741 Additional header information is added to describe the block sizes and 1696 info mation needed to describe the HEAP data section. The compression1742 information needed to describe the HEAP data section. The compression 1697 1743 algorithms currently defined consist of the GZIP, RICE, PLIO, and 1698 1744 HCOMPRESS (REFS). For GZIP, the compression algorithm may transpose 1699 1745 the byte order before compression: for floating point data of a 1700 simil iar dynamic range, this choice may allow for better compression1746 similar dynamic range, this choice may allow for better compression 1701 1747 as each byte in the 4 or 8 byte floating point value is more likely to 1702 1748 be similar to the same byte in other rows than to the other bytes of … … 1758 1804 added to the \ippdbtable{addProcessedExp} table. 1759 1805 1760 After the master DVO is con tructed containing the PS1 data, data from1806 After the master DVO is constructed containing the PS1 data, data from 1761 1807 other sources are also added to the database. For the PV3 DVO 1762 1808 database, data was added from 2MASS, WISE, Gaia DR1, and Tycho. These … … 1843 1889 \label{sec:ipp2psps} 1844 1890 1845 The public ally-visible Pan-STARRS database is hosted by the Space1891 The publicly-visible Pan-STARRS database is hosted by the Space 1846 1892 Telescope Sciences Institute through their Mikulski Archive for Space 1847 Telescopes (MAST). The under ying database at MAST is a copy of a1893 Telescopes (MAST). The underlying database at MAST is a copy of a 1848 1894 database generated at the IfA by the Published Science Products 1849 1895 Subsystem (PSPS). The construction of the PSPS version of the PS1 … … 1940 1986 Each task must at a minimum define a command to generate. Commands 1941 1987 may be static or dynamic. For a task with a static command, the 1942 command is explicit y defined in the task block (see code example in1988 command is explicitly defined in the task block (see code example in 1943 1989 Figure~\ref{fig:task_example}) and is identical each time the task is 1944 1990 executed. A dynamic command is defined within a special block of the 1945 task, called \code{task.exec}. This block is a snip et of code (in the1991 task, called \code{task.exec}. This block is a snippet of code (in the 1946 1992 \ippprog{opihi} language) which is run each time the task is executed. The 1947 1993 \code{task.exec} code may refer to variables or other data structures … … 1987 2033 These options may be dynamically reset by the \code{task.exec} macro. 1988 2034 Some options control the number of jobs, such as limiting the number 1989 of currently-outs anding jobs for a given task, or limiting the total2035 of currently-outstanding jobs for a given task, or limiting the total 1990 2036 number generated. Other options can be used to control the time when 1991 2037 jobs of a certain task are allowed to run. It is also possible to … … 2030 2076 2031 2077 When \ippprog{pcontrol} is provided with the name of a computer, it will attempt 2032 to make an connection to that machine via ssh (or rsh?). When a2078 to make an connection to that machine via ssh. When a 2033 2079 connection is made, the remote shell is used to run a special 2034 2080 interface program call \ippprog{pclient}. This program accepts … … 2036 2082 individual commands in the local shell environment. A single ssh 2037 2083 connection to a remote host keeps a single \ippprog{pclient} shell running for a 2038 somewhat arbi rarly long time, excuting many shell commands as needed.2084 somewhat arbitrarly long time, executing many shell commands as needed. 2039 2085 This architecture avoids wasting overhead making the ssh connection to 2040 2086 the remote machine each time a command is executed, allowing for rapid 2041 ex cution of many commands. As a result, a single job within the IPP2087 execution of many commands. As a result, a single job within the IPP 2042 2088 architecture is allowed to be very light and short running if needed. 2043 2089 … … 2255 2301 ippstage{diff} analysis stage. 2256 2302 2257 Once observations have been completed for the night (signal led by the2303 Once observations have been completed for the night (signaled by the 2258 2304 end-of-night dark exposures that are taken each morning after the 2259 2305 telescope closes), and the script has generated all \ippstage{diff} … … 2321 2367 challenge of storing and managing the large volume of data that is 2322 2368 generated by the GPC1 camera. The \ippprog{Nebulous} system was 2323 designed to aid in thi eprocess. \ippprog{Nebulous} is not a file2369 designed to aid in this process. \ippprog{Nebulous} is not a file 2324 2370 system per-se, but only a method of tracking the locations of files 2325 2371 within the file system, and of tracking duplicate copies of the same … … 2622 2668 unification of configuration options between config files on disk and 2623 2669 the options specified on the command line is handled by 2624 \ippmisc{psModules} functions, as is the con truction of data2670 \ippmisc{psModules} functions, as is the construction of data 2625 2671 structures in memory to represent the astronomical camera based on the 2626 2672 header information in the input file. The functions to generate and … … 2792 2838 to hang until the job time limit is reached. These stacks were 2793 2839 instead processed on the regular IPP cluster, where hosts with 2794 suffic ent memory were available.2840 sufficient memory were available. 2795 2841 2796 2842 \subsection{UH Cray Cluster} … … 2820 2866 994,890 runs processed there. 2821 2867 2822 \section{Discussion} 2823 \label{sec:discussion} 2868 \section{Conclusion} 2869 2870 Since the Pan-STARRS\,1 telescope first came online in 2007, this 2871 telescope has obtained 1.43 million exposures with GPC1, amounting to 2872 a raw data volume of 4.32 petabytes. The Pan-STARRS Image Processing 2873 Pipeline has archived and processed these images on-the-fly to produce 2874 discoveries of transient events and hazardous asteroids in real-time. 2875 The IPP has been used to perform several re-processings of large 2876 fractions of the science exposures to produce a well-calibrated data 2877 release of the $3\pi$ Survey data. To date, and including repeated 2878 analysis, the IPP has processed 2.1 million exposures, detecting 900 billion 2879 sources in those exposures (real and otherwise!). The Pan-STARRS data 2880 processing system represents a real example of astronomy data 2881 processing on the very large scale. 2824 2882 2825 2883 \acknowledgments
Note:
See TracChangeset
for help on using the changeset viewer.
