- Timestamp:
- Apr 16, 2019, 8:21:43 AM (7 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/release.2015/ps1.datasystem/datasystem.tex
r40613 r40696 1 % \documentclass[preprint2]{emulateapj} % works for 2-column2 1 \documentclass[iop,floatfix]{emulateapj} 3 % \documentclass[iop,floatfix,onecolumn]{emulateapj}4 % \documentclass[12pt,preprint]{aastex}5 2 % \documentclass[10pt,preprint]{aastex} % use for 1-column 6 % \documentclass[preprint]{aastex}7 3 % \pdfoutput=1 8 4 9 5 %\RequirePackage{deluxetable} -- included by aastex? 10 %\RequirePackage{nsfprop} % defines \subsubsubsection but breaks 2-col11 6 \RequirePackage{color} 12 7 \RequirePackage{code} … … 15 10 \usepackage[T1]{fontenc}% (2) specify encoding 16 11 12 % these options allow the code to swap between figure types & versions: 13 17 14 % online version may use color, but print version needs b/w 18 15 \def\plotmode{col} … … 22 19 \def\plotext{ps} 23 20 24 %\def\picdir{/home/eugene/chipresid.20140404} 25 \def\picdir{/data/pikake.2/eugene/chipresid.20140404} 21 \def\picdir{figures} 26 22 27 23 % Pick a terse version of the title here; … … 232 228 to reduce this large number of exposures. 233 229 234 % Finally,235 % Section~\ref{sec:discussion} presents a discussion of some of the236 % lessons learned in the creation of the IPP, and its utility in237 % reducing data from other cameras and telescopes.238 239 %% {\color{red} {\em Note: These papers are being placed on arXiv.org to240 %% provide crucial support information at the time of the public241 %% release of Data Release 1 (DR1). We expect the arXiv versions to242 %% be updated prior to submission to the Astrophysical Journal in243 %% January 2017. Feedback and suggestions for additional information244 %% from early users of the data products are welcome during the245 %% submission and refereeing process.}}246 247 230 \section{Overview of Pan-STARRS Data Processing} 248 231 \label{sec:overview} … … 277 260 ingests the calibrated measurements from the IPP, MOPS, and others 278 261 and generates a high-availability database with web-based 279 interactions for public consumption \citep[][]{flewelling2017}.262 interactions for public consumption (Paper VI). 280 263 281 264 \end{itemize} … … 301 284 emphasis on the analysis, calibration, and database ingest stages. 302 285 The MOPS is described in detail by \cite{2013PASP..125..357D}. 303 304 % the summit systems are described by \note{REF?}.305 286 306 287 \begin{figure*}[htbp] … … 361 342 Petrosian aperture photometry, etc). The results of the stack 362 343 photometry analysis are used to drive a forced-photometry analysis of 363 the warp images. These analysis steps are discussed in detail by364 \citet[][]{magnier2017.analysis}. The data products from the camera, 365 stack, and forced-warp photometry analysis stages are ingested into 366 the internal calibration database (DVO, the Desktop Virtual 367 Observatory) and used for photometric and astrometric calibrations 368 \citet[see Section~\ref{sec:DVO} and][]{magnier2017.calibration}.344 the warp images. These analysis steps are discussed in detail in 345 Paper IV. The data products from the camera, stack, and forced-warp 346 photometry analysis stages are ingested into the internal calibration 347 database (DVO, the Desktop Virtual Observatory) and used for 348 photometric and astrometric calibrations (see Section~\ref{sec:DVO} 349 and Paper V). 369 350 370 351 \subsection{Data Access and Distribution} … … 384 365 (PV1 \& PV2), the data were ingested into the PSPS database system and 385 366 made available to the PS1SC community through a web portal based at 386 the IfA as well as the MAST portal \citep[see][for full 387 details]{flewelling2017}. 367 the IfA as well as the MAST portal (see Paper VI for full details). 388 368 389 369 \section{IPP Data Processing Stages} … … 401 381 \hline 402 382 {\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} \\% & {\bf Notes} \\ 403 %%D \begin{deluxetable}{llll}404 %%D \tablecolumns{5}405 %%D \tablewidth{0pc}406 %%D \tablecaption{GPC1 Database Schema Outline}407 %%D \tablehead{\colhead{Stage} & \colhead{Primary Table} & \colhead{Secondary Table} & \colhead{Key}} % & \colhead{Notes}}408 %%D \startdata409 %\hline410 383 \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & \\% & Lists locations to check for new exposures.\\ 411 384 & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} \\% & Exposures available at the telescope.\\ … … 445 418 & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap_id} \\% & \\ 446 419 \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote_id} \\% & \\ 447 %%D \enddata448 420 \hline 449 421 \end{tabular} 450 422 \label{tab:database_schema} 451 %%D \end{deluxetable}452 423 \end{center} 453 424 \end{table*} … … 602 573 For GPC1, the \ippstage{registration} stage is also the stage at which 603 574 the \ippprog{burntool} analysis is run. This analysis is more 604 completely described in \citet{waters2017}. In brief, the575 completely described in Paper III. In brief, the 605 576 \ippprog{burntool} program identifies bright sources on the image, and 606 577 identifies persistence trails that result from the incomplete transfer … … 653 624 not been as critical of a requirement as originally expected. 654 625 655 %% In the \ippstage{chip} stage,656 %% the individual OTA image files are processed independently in parallel657 %% within the data processing cluster. \note{move this to kihei658 %% discussion?} Within the processing computer cluster, most of the659 %% data storage resources are in the form of computers with large raids660 %% as well as substantial processing capability. The processing system661 %% attempts to locate one copy of specific raw registered data on662 %% pre-defined computers that have been set as storage targets for that663 %% OTA. The processing system is aware of this data localization and664 %% attempts to target the processing for each OTA to the machine on which665 %% the data for that detector is stored. The output products are then666 %% primarily saved back to the same machine. This ``targetted'' processing667 %% was an early design choice to minimize the system wide network load668 %% during processing. In practice, as computer disks filled up at669 %% different rates, the data has not been localized to a very high670 %% degree.671 672 626 The actual image processing is performed by the \ippprog{ppImage} 673 627 program. This program reads the raw data into memory and applies the 674 detrend corrections \citep[see][]{waters2017} to each cell in the OTA 675 (stored as different extensions in the FITS file format), and then 676 mosaics the cells into a single contiguous \ippstage{chip} stage 677 image. This step also creates in memory additional images to hold the 678 mask data, which indicates which pixels may not be valid, and the 679 variance image, constructed as the Poissonian noise on the number of 680 electrons detected based on the original pixel value and the detector 681 gain. A background model is then fit across the image and subtracted 682 to remove the expected contribution from the sky 683 \citep[see][]{waters2017} for details. 628 detrend corrections (see Paper III) to each cell in the OTA (stored as 629 different extensions in the FITS file format), and then mosaics the 630 cells into a single contiguous \ippstage{chip} stage image. This step 631 also creates in memory additional images to hold the mask data, which 632 indicates which pixels may not be valid, and the variance image, 633 constructed as the Poissonian noise on the number of electrons 634 detected based on the original pixel value and the detector gain. A 635 background model is then fit across the image and subtracted to remove 636 the expected contribution from the sky (see Paper III for details). 684 637 685 638 With the image calibration procedure finished, object identification … … 689 642 this analysis, removing the need to write out and re-read the image 690 643 data. The details of the detection and characterization of the 691 sources in the image are provided in \citet{magnier2017.analysis}.644 sources in the image are provided in Paper IV. 692 645 693 646 The results of the image processing are then written to disk, … … 715 668 in which case an entry for this exposure is added to the \ippdbtable{camRun} 716 669 table, and processing continues. 717 718 %% The \ippstage{chip} processing stage consists of: reading the raw image into719 %% memory, applying the detrending steps \citep[see][]{waters2017},720 %% stiching the individual OTA cells into a single chip image, detection721 %% and characterization of the sources in the image722 %% \citep[see][]{magnier2017b}, and output of the various data products.723 %% These include the detrended chip image, variance image, and mask724 %% image, as well as the FITS catalog of detected sources. The PSF model725 %% and background model are also saved, along with a processing log. A726 %% selection of summary metadata describing the processing results are727 %% saved and written to the processing database along with the completion728 %% status of the process. Finally, binned chip images are generated (on729 %% two scales, binned by 16 and 256 pixels) for use in the visualization730 %% system of the processing monitor tool. \note{describe elsewhere?}731 732 %% The database structure for the \stage{chip} stage mimics that of raw733 %% data, with a \ippdbtable{chipRun} characterizing the processing of a734 %% single exposure, mapping to a set of \ippdbtable{chipProcessedImfile}735 %% entries for each OTA via a common \ippdbcolumn{chip_id}.736 670 737 671 \subsection{Camera Calibration} … … 755 689 to help guarantee a solution in the case of a modest pointing error. 756 690 The guess astrometry is used to match the reference catalog to the 757 observed stellar positions in the focal plane coordinate system 758 \citep[see][]{magnier2017.calibration}. Early on in the PS1SC 759 mission, the nightly processing (PV0) used a reference catalog based 760 on a combination of external catalogs (2MASS, Tycho, USNO). Later, 761 reference catalogs based on Pan-STARRS data was used. For the $3\pi$ PV3 analysis, 762 the reference catalog was based on Pan-STARRS data from the PV2 763 analysis \citep[see][for more details]{magnier2017.calibration}. 691 observed stellar positions in the focal plane coordinate system. 692 Early on in the PS1SC mission, the nightly processing (PV0) used a 693 reference catalog based on a combination of external catalogs (2MASS, 694 Tycho, USNO). Later, reference catalogs based on Pan-STARRS data was 695 used. For the $3\pi$ PV3 analysis, the reference catalog was based on 696 Pan-STARRS data from the PV2 analysis (see Paper V for more details). 764 697 765 698 Once an acceptable match is found, the astrometric calibration of the … … 787 720 so a fixed color transformation is used to generate synthetic w-band 788 721 photometry from the \rps\ \& \ips\ photometry. For more details, see 789 \cite{magnier2017.calibration}. The result of these calibrations is 790 stored as a single multi-extension FITS table containing the results 791 from each OTA as aseparate extension.722 Paper V. The result of these calibrations is stored as a single 723 multi-extension FITS table containing the results from each OTA as a 724 separate extension. 792 725 793 726 In addition to the astrometric and photometric calibrations, the … … 884 817 \ippstage{chip} stage images (including the variance images and the 885 818 updated masks) to the \ippprog{pswarp} program. For details on the 886 warping algorithm, see \cite{waters2017}. The outputs of this program819 warping algorithm, see Paper III. The outputs of this program 887 820 are the geometrically transformed images (signal, variance, and mask) 888 821 containing all input pixels warped to the common skycell pixel grid, … … 892 825 extraction tools at the MAST archive at STScI as part of the DR2 data 893 826 release. 894 895 %% A catalog is896 %% also generated containing the locations of sources from the input897 %% catalog that fall within area of the \ippstage{warp}.898 827 899 828 When the \ippstage{warp} jobs have completed, an entry for the skycell … … 928 857 generated for the nightly groups and for the full depth using all 929 858 exposures, producing ``deep stacks''. In addition, a ``best seeing'' 930 set of stacks have been produced using image quality cuts described by 931 \citet[][Paper VII]{huber2017}. We have also generated out-of-season 932 stacks for the Medium Deep fields, in which all images {\em not} from a 933 particular observing season for a field are combined into a stack. 934 These later stacks are useful as deep templates when studying 935 long-term transient events in the Medium Deep fields as they are not 936 (or less) contaminated by the flux of the transients from a given 937 season. 859 set of stacks have been produced using image quality cuts described in 860 Paper VII. We have also generated out-of-season stacks for the Medium 861 Deep fields, in which all images {\em not} from a particular observing 862 season for a field are combined into a stack. These later stacks are 863 useful as deep templates when studying long-term transient events in 864 the Medium Deep fields as they are not (or less) contaminated by the 865 flux of the transients from a given season. 938 866 939 867 When a given set of \ippstage{stack} stage processing is defined, … … 951 879 and catalogs to the \ippprog{ppStack} program, which performs the 952 880 image combinations. Input warps are combined based on a weighting 953 defined by the median variance for each image; see~ \cite{waters2017}881 defined by the median variance for each image; see~Paper III 954 882 for details on the stack combination algorithm. In addition to the 955 883 standard image, mask, and variance produced at other stages, … … 987 915 The input images are passed to the \ippprog{psphotStack} program which 988 916 does the analysis. The stack photometry algorithms are described in 989 detail in \cite{magnier2017.analysis}. In short, sources are detected990 i n all 5 filter images down to the $5\sigma$ significance. The991 collection of detected sources is merged into a single master list. 992 If a source is detected in at least two bands, or only in \yps{} band, 993 then a PSF model is fitted to the pixels of the other bands in which 994 the source was not detected. This forced photometry results in lower 995 significance measurements of the flux at the positions of objects 996 which are thought to be real sources, by virtue of triggering a 997 detection in at least two bands. The relaxed limit for \yps{} band is 998 included to allow for searches of \yps{} dropout objects: it is known 999 that faint, high-redshift quasars may be detected in \yps{} band only. 1000 Sources detected only in \yps{} band are therefore more likely to have 1001 ahigher false-positive rate than the other stack sources. The917 detail in Paper IV. In short, sources are detected in all 5 filter 918 images down to the $5\sigma$ significance. The collection of detected 919 sources is merged into a single master list. If a source is detected 920 in at least two bands, or only in \yps{} band, then a PSF model is 921 fitted to the pixels of the other bands in which the source was not 922 detected. This forced photometry results in lower significance 923 measurements of the flux at the positions of objects which are thought 924 to be real sources, by virtue of triggering a detection in at least 925 two bands. The relaxed limit for \yps{} band is included to allow for 926 searches of \yps{} dropout objects: it is known that faint, 927 high-redshift quasars may be detected in \yps{} band only. Sources 928 detected only in \yps{} band are therefore more likely to have a 929 higher false-positive rate than the other stack sources. The 1002 930 parameters of the PSF model are allowed to vary with position in the 1003 931 skycell. The PSF model is also used to convolve the analytical galaxy … … 1085 1013 in question is large compared to the FWHM of the PSF. 1086 1014 1087 %% The IPP team initially explored the option of convolving each input1088 %% warp to a single target PSF chosen to match the worst of the input1089 %% images for a given stack.1090 1091 1015 The IPP analysis solves this problem by using the sources 1092 1016 detected in the stack images and performing forced photometry on the … … 1109 1033 stage image products along with the \ippstage{skycal} catalog to the 1110 1034 \ippprog{psphotFullForce} program. 1111 1112 %% In this program, the positions of sources are loaded from the input1113 %% catalog. PSF stars are pre-identified from the stack image and a PSF1114 %% model generated for each \ippstage{warp} image based on those stars,1115 %% using the same stars for all warps to the extent possible (PSF stars1116 %% which are excessively masked on a particular image are not used to1117 %% model the PSF). The PSF model is fitted to all of the known source1118 %% positions in the warp images. Aperture magnitudes, Kron magnitudes,1119 %% and moments are also measured at this stage for each warp. Note that1120 %% the flux measurement for a faint, but significant, source from the1121 %% stack image may be at a low significance (less than the $5\sigma$1122 %% criterion used when the photometry is not run in this forced mode) in1123 %% any individual warp image; the flux may even be negative for specific1124 %% warps. When combined together, these low-significance measurements1125 %% will result in a signficant measurement as the signal-to-noise1126 %% increases by the square root of the number of measurements. The1127 %% individual warp measurements are combined together to generate1128 %% averages values within DVO.1129 1035 1130 1036 The convolved galaxy models are also re-measured on the … … 1180 1086 images, from a \ippstage{warp} and a \ippstage{stack} of some variety, 1181 1087 or from a pair of \ippstage{stack} stage images. During the PS1 1182 survey, pairs of exposures, called TTI pairs \citep[see Survey1183 Strategy in][]{chambers2017}, were obtained for each pointing within 1184 a $\approx$ 1 hour period in the same filter, and to the extent 1185 possible with the same orientation and boresite position. The 1186 standard PS1 nightly processing generated difference images from the 1187 resulting pairs of \ippstage{warp} images. The nightly processing 1188 generated \ippstage{stack} images for the Medium Deep fields, and 1189 these were combined with a template reference \ippstage{stack} image1190 to generate ``stack-stack diffs'' each night they were observed. For 1191 the PV3 $3\pi$ processing, the entire collection of \ippstage{warp} 1192 stageimages for the survey were combined with images generated by the1088 survey, pairs of exposures, called TTI pairs (see Survey Strategy in 1089 Paper I), were obtained for each pointing within a $\approx$ 1 hour 1090 period in the same filter, and to the extent possible with the same 1091 orientation and boresite position. The standard PS1 nightly 1092 processing generated difference images from the resulting pairs of 1093 \ippstage{warp} images. The nightly processing generated 1094 \ippstage{stack} images for the Medium Deep fields, and these were 1095 combined with a template reference \ippstage{stack} image to generate 1096 ``stack-stack diffs'' each night they were observed. For the PV3 1097 $3\pi$ processing, the entire collection of \ippstage{warp} stage 1098 images for the survey were combined with images generated by the 1193 1099 \ippstage{stack} processing to generate ``warp-stack diffs'', for 1194 1100 eventual public released. … … 1221 1127 (flux in the minuend is higher than the subtrahend) or as a negative 1222 1128 source (flux in the subtrahend is higher). The algorithm used for PSF 1223 matching is described in \citet{waters2017}. Upon completion of these1129 matching is described in Paper III. Upon completion of these 1224 1130 jobs, statistics about the processing are written to an entry in the 1225 1131 \ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the … … 1256 1162 Table~\ref{tab:DVO_schema} lists the full collection of database 1257 1163 tables used by DVO. 1258 1259 %Figure~\ref{fig:DVO_schema}1260 %illustrates the schematic relationship between these types of1261 %measurements.1262 1164 1263 1165 In the most basic implementation, a collection of measurements for … … 1274 1176 measurements and a many-to-one relationship between the measurements 1275 1177 and the derived astronomical objects. 1276 1277 %1278 %% These tables fall into one of several classes:1279 %% those which store information about the average properties of1280 %% astronomical objects; those which store information about individual1281 %% measurements; those which store information about the images; those1282 %% which store supporting information (metadata).1283 1284 %% DVO includes two major classes of database tables: those containing1285 %% information about astronomical objects in the sky and those containing1286 %% other supporting information. The object-related tables are1287 %% partitioned on the basis of position in the sky: objects within a1288 %% region bounded by lines of constant RA,DEC are contained in a specific1289 %% file. The boundaries and the associated partition names are stored in1290 %% one of the supporting tables, \ippdbtable{SkyTable}. This table1291 %% contains the definitions of the boundaries for each sky region1292 %% (\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN},1293 %% \ippdbcolumn{D_MAX}), the name of the sky region, an ID1294 %% (\ippdbcolumn{INDEX}, equal to the sequence number of the region in1295 %% the table), and index entries to enable navigation within the table.1296 %% The regions are defined in a hierarchical sense, with a series of1297 %% levels each containing a finer mesh of regions covering the sky.1298 1178 1299 1179 \subsubsection{DVO Schema} … … 1426 1306 magnitude. While these photometric distance modulus measurements are 1427 1307 not extremely precise, they provide a constraint on the distance which 1428 is used in our analysis of the astrometry 1429 \citep[see][]{magnier2017.calibration}. 1430 1431 %% Similarly to the \ippdbtable{Measure} table, the fields 1432 %% \ippdbcolumn{objID}, \ippdbcolumn{catID}, and \ippdbcolumn{averef} 1433 %% define links from the \ippdbtable{Lensing} table to the 1434 %% \ippdbtable{Average} table. In a similar fashion, the fields 1435 %% \ippdbtable{Average}.\ippdbcolumn{lensingOffset} and 1436 %% \ippdbtable{Average}.\ippdbcolumn{Nlensing} are the index into the 1437 %% sorted \ippdbtable{Lensing} table entries. \note{discuss failure of 1438 %% the Lensing to Measure indexing} 1439 1440 % \note{Average used above but defined below} 1308 is used in our analysis of the astrometry (see Paper V). 1441 1309 1442 1310 \paragraph{Object Tables} … … 1530 1398 across the different IPP stages. 1531 1399 1532 %% Data from GPC1 (and other cameras processed by the IPP) are loaded1533 %% into DVO in units \code{smf} files generated by the \ippstage{camera}1534 %% calibration stage (see section \ref{sec:camera} below). As1535 %% described above, these files contain all measurements from a complete1536 %% exposure, with measurements from each chip grouped into separate FITS1537 %% tables. When these measurements are loaded into the1538 %% \ippdbtable{Measure} and similar tables,1539 1540 1400 \paragraph{Other Tables} 1541 1401 … … 1544 1404 determined by the photometry calibration analysis and the astrometric 1545 1405 flat-field corrections determined by the astrometry calibration 1546 analysis \citep[see][]{magnier2017.calibration}.1406 analysis (see Paper V). 1547 1407 1548 1408 \subsubsection{Sky Partition} … … 1686 1546 allows valid joins between tables to select the different kinds of 1687 1547 attributes of the same astronomical objects. This 64-bit integer ID 1688 is constructed based on the coordinates of the object, as described by1689 \cite[][]{flewelling2017}. In short, the digits of the right1548 is constructed based on the coordinates of the object, as described in 1549 Paper VI. In short, the digits of the right 1690 1550 ascension and declination coordinates are used to define a single 1691 1551 64-bit integer with spatial resolution of roughly 3 milliarcseconds. … … 1761 1621 Upon completion of the processing of each stage, the results of the 1762 1622 photometry analysis are stored in a large number of individual catalog 1763 files as described in \cite{magnier2017.analysis}. The data from1764 these files are loaded into a DVO database to define the astronomical 1765 objects and to allow for calibration analysis. The program which 1766 loads the data into the DVO database is called \ippprog{addstar}, and 1767 is associated with the the \ippstage{addstar} processing stage. The 1768 measurement catalogs generated by the \ippstage{camera},1769 \ippstage{ skycal}, \ippstage{fullforce}, and \ippstage{diff} stages1770 are loaded into DVOs in this fashion, although not every measurement 1771 in each catalog are included in the master DVO that is constructed.1772 For a particular re-processing version, a single master DVO is 1773 constructed for the positive image stages (\ippstage{camera},1774 \ippstage{ skycal}, \ippstage{fullforce}) and a separate one is1775 constructed for thedifference image analysis stage results.1623 files as described in Paper IV. The data from these files are loaded 1624 into a DVO database to define the astronomical objects and to allow 1625 for calibration analysis. The program which loads the data into the 1626 DVO database is called \ippprog{addstar}, and is associated with the 1627 the \ippstage{addstar} processing stage. The measurement catalogs 1628 generated by the \ippstage{camera}, \ippstage{skycal}, 1629 \ippstage{fullforce}, and \ippstage{diff} stages are loaded into DVOs 1630 in this fashion, although not every measurement in each catalog are 1631 included in the master DVO that is constructed. For a particular 1632 re-processing version, a single master DVO is constructed for the 1633 positive image stages (\ippstage{camera}, \ippstage{skycal}, 1634 \ippstage{fullforce}) and a separate one is constructed for the 1635 difference image analysis stage results. 1776 1636 1777 1637 The construction of the master DVO is performed in a hierarchical … … 1818 1678 Once the master DVO database has been constructed, high-quality 1819 1679 astrometric and photometric calibrations can be calculated. The 1820 details of the calibration analysis are discussed in 1821 \cite{magnier2017.calibration}. Wepresent a brief summary here.1680 details of the calibration analysis are discussed in Paper V. We 1681 present a brief summary here. 1822 1682 1823 1683 Astrometric calibration consists of measuring and correcting … … 1832 1692 a function of position in the camera (essentially an astrometric 1833 1693 flat-field correction), as a function of the brightness of the star 1834 (the so-called Koppenh\"of fer effect, see~\citealt{magnier2017.calibration}), and as1694 (the so-called Koppenh\"ofer effect, see~Paper V), and as 1835 1695 a function of airmass and color (differential chromatic refraction). 1836 1696 Once the systematic errors have been measured, they are applied back … … 1865 1725 the Medium Deep fields. 1866 1726 1867 %% (listed in Table~\ref{tab:flat-field-seasons}) XXX add this table1868 1869 1727 After the \"ubercal analysis of the photometric periods is completed, 1870 1728 the determined zero points, airmass corrections, and flat-field terms … … 1884 1742 flat-field correction addresses photometric variations due to spatial 1885 1743 variations in the PSF due to the optics and low-level effects on the 1886 chips \citep[see][]{magnier2017.calibration}. After the systematic corrections1744 chips (see Paper V). After the systematic corrections 1887 1745 have been determined and applied back to the database, a final 1888 1746 relative photometry analysis pass is performed. … … 1898 1756 database starts once the PS1 photometry and astrometry measurements 1899 1757 have been calibrated within the DVO system. The construction takes 1900 place in several stages, described in detail by \cite{flewelling2017}.1758 place in several stages, described in detail in Paper VI. 1901 1759 We summarize those steps here. 1902 1760 … … 2148 2006 \end{figure} 2149 2007 2150 %\code{ls /tmp}2151 2152 2008 \subsubsection{Pantasks scripts: ippTasks} 2153 2009 … … 2194 2050 \ippmisc{DONE}, and removes them from the book, as these represent 2195 2051 jobs that have finished. 2196 2197 % \note{the manipulation above takes place in the task.exit subscript}2198 2052 2199 2053 The associated \ippmisc{run} task generates jobs constructed from the … … 2342 2196 used for the warp tessellation. A \ippdbcolumn{projection_cell} is a 2343 2197 unit of sky defined to be a square four degrees on each side which has 2344 a single tangent plane projection \citep[][see]{waters2017}.2198 a single tangent plane projection (Paper III). 2345 2199 Once this 2346 2200 entry is defined, it is populated with all exposures (stored in the … … 2420 2274 data to that instance. 2421 2275 2422 % The basic user commands to interact2423 % with Nebulous are to 1) create a new storage object and associated2424 % instance; 2) add a new instance to an existing storage object; 3)2425 % remove (cull) an instance; 4) delete a storage object; and 5) find a2426 % file associated with a given storage objects. Note that these user2427 % commands do not affect the files on disk \note{true for cull?}2428 % (exception: the create function will create an empty file if one does2429 % not exist). They only change the state of the Nebulous database; it2430 % is the responsibility of the user program to read and write data to a2431 % file and to create the copies, etc.2432 2433 2276 For the Nebulous users, the identifier of a storage object is a unique 2434 2277 string with the form of a UNIX file path: e.g. a/b/c/file. When a … … 2547 2390 Requests to this server may restrict to the latest by time. Each row 2548 2391 in the listing includes basic information about the exposure: an 2549 exposure identifier \citep[e.g., o5432g0123o; see][for 2550 details]{chambers2017}, the date and time of the exposure, the 2551 telescope commanded pointing, the filter and exposure time, and the 2552 observation comment for that exposure. The row also provides a link 2553 to a listing of the chips associated with that exposure. This listing 2554 includes a link to the individual chip FITS files as well as an md5 2555 checksum. Systems which are allowed access may download the raw chip 2556 FITS files via http requests to the provided links. 2557 2558 % \note{add a discussion of gpc1 filenames?} 2392 exposure identifier (e.g., o5432g0123o; see Paper I for details), the 2393 date and time of the exposure, the telescope commanded pointing, the 2394 filter and exposure time, and the observation comment for that 2395 exposure. The row also provides a link to a listing of the chips 2396 associated with that exposure. This listing includes a link to the 2397 individual chip FITS files as well as an md5 checksum. Systems which 2398 are allowed access may download the raw chip FITS files via http 2399 requests to the provided links. 2559 2400 2560 2401 The IPP also uses datastores to provide access to its own data … … 2666 2507 isolation of source objects is included, providing the organization of 2667 2508 detections that is used in the \ippprog{psphot} photometry analysis 2668 \citep{magnier2017.analysis}. The PSF matching required for \ippstage{stack} 2669 and \ippstage{diff} stage image combinations is as well. The 2670 unification of configuration options between config files on disk and 2671 the options specified on the command line is handled by 2672 \ippmisc{psModules} functions, as is the construction of data 2673 structures in memory to represent the astronomical camera based on the 2674 header information in the input file. The functions to generate and 2675 apply the detrend corrections to the data are also provided by this 2676 library. 2509 (Paper IV). The PSF matching required for \ippstage{stack} and 2510 \ippstage{diff} stage image combinations is as well. The unification 2511 of configuration options between config files on disk and the options 2512 specified on the command line is handled by \ippmisc{psModules} 2513 functions, as is the construction of data structures in memory to 2514 represent the astronomical camera based on the header information in 2515 the input file. The functions to generate and apply the detrend 2516 corrections to the data are also provided by this library. 2677 2517 2678 2518 \section{IPP Hardware Systems} … … 2688 2528 by the University of Hawaii. This site was chosen based on the 2689 2529 original development funding provided by the Air Force Research Labs 2690 \citep[see][for more details]{chambers2017}. Once the Air Force 2691 funding stopped being a significant driver for Pan-STARRS, the cluster was 2692 physically moved from the MHPCC to a similar nearby computing center 2693 located atthe Maui Research and Technology Center.2530 (see Paper I for more details). Once the Air Force funding stopped 2531 being a significant driver for Pan-STARRS, the cluster was physically 2532 moved from the MHPCC to a similar nearby computing center located at 2533 the Maui Research and Technology Center. 2694 2534 2695 2535 The computing cluster is comprised of three main types of computers, … … 2792 2632 \end{table*} 2793 2633 2794 %%\begin{deluxetable}{lcc}2795 %% \tablecolumns{3}2796 %% \tablewidth{0pc}2797 %% \tablecaption{Cost values for remote processing}2798 %% \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}}2799 %% \startdata2800 %% \ippstage{chip} & 150 & 2 \\2801 %% \ippstage{camera} & 1700 & 2 \\2802 %% \ippstage{warp} & 110 & 2 \\2803 %% \ippstage{stack} & 1500 & 6 \\2804 %% \ippstage{staticsky} & 7200 & 6 \\2805 %%% \ippstage{diff} & 300 & 2 \\2806 %% \ippstage{fullforce} & 300 & 22807 %% \enddata2808 %% \label{tab:SC processing parameters}2809 %%\end{deluxetable}2810 2811 2634 Once the preparation for the job is complete, the input and output 2812 2635 file lists, the task list, and the job control file are transferred … … 2868 2691 994,890 runs processed there. 2869 2692 2693 %% add a discussion of lessons-learned? 2694 2870 2695 \section{Conclusion} 2871 2696 … … 2905 2730 \input{datasystem.bbl} 2906 2731 2907 % \appendix2908 2909 % Table \ref{tab: database schema} provides a list of a majority of the2910 % tables in the GPC1 database schema. Tables that have been excluded2911 % are either no longer used in IPP processing, or are used for rare2912 % reductions that were not used for the PV3 data release. The tables2913 % are grouped into stages, with the primary table and any secondary2914 % tables for that stage listed together, along with the primary key2915 % column that link the tables together.2916 2917 2732 \end{document} 2918 2733 2734 % this is a 'deluxetable' version of table 1 2919 2735 \begin{center} 2920 2736 \begin{deluxetable}{lllll}
Note:
See TracChangeset
for help on using the changeset viewer.
