Changeset 41208 for trunk/doc/release.2015/ps1.datasystem/datasystem.tex
- Timestamp:
- Dec 21, 2019, 11:52:17 AM (7 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/release.2015/ps1.datasystem/datasystem.tex
r41207 r41208 228 228 PV3 data release, with some details on the scale of computing needed 229 229 to reduce this large number of exposures. 230 231 In this article, we use the following type-faces to distinguish 232 different concepts: 233 \begin{itemize} 234 \item \ippstage{Small caps} for the analysis stages. 235 \item \ippdbtable{Italics} for database tables and columns. 236 \item \ippprog{Fixed-width} font for program names, variables, and 237 miscellaneous constants. 238 \end{itemize} 230 239 231 240 \section{Overview of Pan-STARRS Data Processing} … … 889 898 exposures with existing \ippstage{warp} entries that match the filter, 890 899 position, and other criteria such as seeing are identified \textadd{(see 891 Section~\ref{sec:automation} to seehow this is automated)}. An entry900 Section~\ref{sec:automation} for details on how this is automated)}. An entry 892 901 is then added for each skycell in the \ippdbtable{stackRun} table, 893 902 with the \ippdbcolumn{warp_id} entries for the exposures added to the … … 1060 1069 images.} In this analysis, the galaxy models determined by the 1061 1070 \ippstage{staticsky} photometry analysis are used to seed the analysis 1062 in the individual \ippstage{warp} images. 1063 1064 The analysis tests a grid of galaxy model parameters in the vicinity 1065 of the prior from the stack. For each warp image, each parameter set 1066 is used to generate a model which is then convolved with the PSF for 1067 that warp image and then compared to the observed data. The resulting 1068 grid of $\chi^2$ values can then be for the 1069 1070 For each object, a grid of galaxy model parameters is used compared tested on each 1071 warp image and 1072 1073 ** we calculate a normalization and chisq for each warp grid point. 1074 the chi square values can be summed across warps to give the solution 1075 chi square dist. for a single warp, the error on Io goes like 1076 sqrt(Ncnts). The average Io value is the weighted averages of the 1077 inputs. The error on the weighted average is sqrt(1 / (sum(1/sigma^2))). 1078 1079 S_t^2 = 1 / (1 / S_0^2 + 1 / S_1^2 + 1 / S_2^2) 1080 1081 S_0^2 = N0, S_1^2 = N1, etc 1082 1083 S_t^2 = 1 / (1/N0 + 1/N1 + 1/N2 ...) 1084 1085 ideal: S_t^2 = N0 + N1 + N2 1086 1087 dI / Io = 1 / sqrt(No) 1088 1089 The error on the 1090 1091 \textmod{For each warp 1092 image, the galaxy model (convolved with the PSF) is compared to the 1093 observed pixels to calculate an element of the total model $\chi^2$ value 1094 1095 a grid 1096 of the galaxy model parameters are 1097 1098 how does the error scale if I fit each Io for each warp vs a single 1099 value? (sounds like I do kill the S/N...) 1100 1101 %%%%% fix all of this... 1102 1103 The purpose of this 1104 analysis is the same as the \ippstage{fullforce} PSF photometry: the 1105 PSF of the \ippstage{stack} image is poorly determined due to the 1106 masking and PSF variations in the inputs. Without a good PSF model, 1107 the PSF-convolved galaxy models are of limited accuracy. 1071 in the individual \ippstage{warp} images. Galaxy models are {\em not} 1072 fitted independently on each warp. Rather, the analysis tests a grid 1073 of galaxy model parameters in the vicinity of the prior from the 1074 stack. 1075 1076 \textadd{For each warp image, each set of galaxy model parameter values is used 1077 to generate a model which is then convolved with the PSF for that warp 1078 image and then compared to the observed data. A normalization and 1079 $\chi^2$ value is determied for each set of parameter values for each 1080 warp image. For each set of parameter values, the normalizations and 1081 $\chi^2$ values are combined across all warps to generate a single 1082 grid of parameters. The best set of galaxy model parameters, along 1083 with the corresponding normalizaiton and $\chi^2$ value is then 1084 determined from the grid by interpolation. } 1085 1086 The purpose of this galaxy model analysis is the same as the 1087 \ippstage{fullforce} PSF photometry: the PSF of the \ippstage{stack} 1088 image is poorly determined due to the masking and PSF variations in 1089 the inputs. Without a good PSF model, the PSF-convolved galaxy models 1090 are of limited accuracy. 1108 1091 1109 1092 Upon completion of the forced photometry, an entry is added to the … … 1120 1103 analysis measurements into a single value. The output catalogs listed 1121 1104 in the \ippdbtable{fullForceResult} table are passed to the 1122 \ippprog{psphotFullForceSummary} to calculate the averages of the1105 \ippprog{psphotFullForceSummary} program to calculate the averages of the 1123 1106 individual warp measurements, weighted by the signal-to-noise of the 1124 1107 flux measurements. When this analysis completes, an entry is added to … … 1163 1146 eventual public released. 1164 1147 1148 When a \ippstage{diff} processing is defined, an entry is added to the 1149 \ippdbtable{diffRun} table, and the appropriate input images are added 1150 to the \ippdbtable{diffInputSkyfile} table, with one entry for each 1151 skycell that is covered by the images. For a \ippstage{diff} 1152 generated from two \ippstage{warp} stage products, the input images 1153 have their \ippdbcolumn{warp_id} values recorded in the 1154 \ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that 1155 overlaps. If two \ippstage{stack} stages are to be used in the 1156 difference, their \ippdbcolumn{stack_id} entries are recorded in the 1157 \ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each 1158 \ippstage{stack} only covers a single skycell, the \ippstage{diff} is 1159 usually defined indirectly, using other information from the 1160 \ippdbtable{stackRun} table to select appropriate 1161 \ippdbcolumn{stack_id} values. Similarly, \ippstage{diff} processing 1162 is defined for the mixed case by creating entries that populate one of 1163 \ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of 1164 \ippdbcolumn{warp2} and \ippdbcolumn{stack2}. In all cases, the 1165 minuend of the subtraction to be performed is the ``1'' entry, and the 1166 subtrahend is the ``2'' entry. 1167 1168 Jobs are created based on the entries of 1169 \ippdbtable{diffInputSkyfile}, with the appropriate images and 1170 catalogs passed to the \ippprog{ppSub} program. This does the 1171 subtraction, as well as the photometry of any sources detected in the 1172 \ippstage{diff} image. Sources may be detected as a positive source 1173 (flux in the minuend is higher than the subtrahend) or as a negative 1174 source (flux in the subtrahend is higher). The algorithm used for PSF 1175 matching is described in Paper III. Upon completion of these 1176 jobs, statistics about the processing are written to an entry in the 1177 \ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the 1178 completion of all of the components listed in 1179 \ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun} 1180 entry as such. 1181 1182 \begin{table} 1183 \begin{center} 1184 \caption{Processing Failure Rates per 100,000 Items\label{tab:failure_rates}} 1185 \begin{tabular}{lrr} 1186 \hline 1187 \hline 1188 {\bf Stage} & {\bf Nightly} & {\bf $3\pi$} \\ 1189 & {\bf Processing} & {\bf PV3} \\ 1190 \hline 1191 Chip & 48 & 34 \\ 1192 Camera & 262 & 280 \\ 1193 ~~~Chip Astrom & N/A & 307 \\ 1194 Warp & 14244 & 13835 \\ 1195 ~~~Warp Pixels & N/A & 3900 \\ 1196 Stack & N/A & 5 \\ 1197 \hline 1198 \end{tabular} 1199 \end{center} 1200 \end{table} 1201 1165 1202 \subsection{Processing Failure Rates} 1166 1203 … … 1197 1234 detector areas reported in Paper III. The result is that roughly 1198 1235 3.9\% of the good input pixels are lost to the warp processing.} 1199 1200 \begin{table}1201 \begin{center}1202 \caption{Processing Failure Rates per 100,000 Items\label{tab:failure_rates}}1203 \begin{tabular}{lll}1204 \hline1205 \hline1206 {\bf Stage} & {\bf Nightly Processing} & {\bf $3\pi$ PV3} \\1207 \hline1208 Chip & 48 & 34 \\1209 Camera & 262 & 280 \\1210 ~~~Chip Astrom & N/A & 307 \\1211 Warp & 14244 & 13835 \\1212 ~~~Warp Pixels & N/A & 3900 \\1213 Stack & N/A & 5 \\1214 \hline1215 \end{tabular}1216 \end{center}1217 \end{table}1218 1236 1219 1237 \begin{table*} … … 1240 1258 \end{center} 1241 1259 \end{table*} 1242 1243 When a \ippstage{diff} processing is defined, an entry is added to the1244 \ippdbtable{diffRun} table, and the appropriate input images are added1245 to the \ippdbtable{diffInputSkyfile} table, with one entry for each1246 skycell that is covered by the images. For a \ippstage{diff}1247 generated from two \ippstage{warp} stage products, the input images1248 have their \ippdbcolumn{warp_id} values recorded in the1249 \ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that1250 overlaps. If two \ippstage{stack} stages are to be used in the1251 difference, their \ippdbcolumn{stack_id} entries are recorded in the1252 \ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each1253 \ippstage{stack} only covers a single skycell, the \ippstage{diff} is1254 usually defined indirectly, using other information from the1255 \ippdbtable{stackRun} table to select appropriate1256 \ippdbcolumn{stack_id} values. Similarly, \ippstage{diff} processing1257 is defined for the mixed case by creating entries that populate one of1258 \ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of1259 \ippdbcolumn{warp2} and \ippdbcolumn{stack2}. In all cases, the1260 minuend of the subtraction to be performed is the ``1'' entry, and the1261 subtrahend is the ``2'' entry.1262 1263 Jobs are created based on the entries of1264 \ippdbtable{diffInputSkyfile}, with the appropriate images and1265 catalogs passed to the \ippprog{ppSub} program. This does the1266 subtraction, as well as the photometry of any sources detected in the1267 \ippstage{diff} image. Sources may be detected as a positive source1268 (flux in the minuend is higher than the subtrahend) or as a negative1269 source (flux in the subtrahend is higher). The algorithm used for PSF1270 matching is described in Paper III. Upon completion of these1271 jobs, statistics about the processing are written to an entry in the1272 \ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the1273 completion of all of the components listed in1274 \ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun}1275 entry as such.1276 1260 1277 1261 \section{Database Ingest and Calibration} … … 1676 1660 Within the PSPS, the \ippdbtable{Detection} table carries an ID which 1677 1661 is unique for each measurement, equivalent to the DVO 1678 \ippdbcolumn{det _id}, \ippdbcolumn{image_id} pair. In this case, the1662 \ippdbcolumn{detID}, \ippdbcolumn{imageID} pair. In this case, the 1679 1663 PSPS \ippdbcolumn{detectID} is constructed from the MJD of the 1680 1664 exposure, the number of the OTA (e.g., OTA64), and the detection … … 1756 1740 1757 1741 The construction of the master DVO is performed in a hierarchical 1758 fashion. The individual catalogs are added to a \ippmisc{minidvo},1742 fashion. The individual catalogs are added to a mini-DVO, 1759 1743 which is simply a DVO database defined over some subset of possible 1760 inputs. These \ippmisc{minidvos}are then merged by stage into larger1744 inputs. These mini-DVOs are then merged by stage into larger 1761 1745 databases to construct a single master DVO database. In the process, 1762 1746 an intermediate master DVO for each stage is generated. The … … 1770 1754 WISE telescope. 1771 1755 1772 As of PV3, the process of merging \ippmisc{minidvos}is not highly1756 As of PV3, the process of merging mini-DVOs is not highly 1773 1757 automated, requiring manual attention. The generation of the 1774 \ippmisc{minidvos}is automated and managed by the \ippstage{addstar}1758 mini-DVOs is automated and managed by the \ippstage{addstar} 1775 1759 stage. Each catalog that is to be added to DVO has an entry created 1776 1760 in the \ippdbtable{addRun} database table. This entry notes which … … 1781 1765 created, with the \ippdbcolumn{stage_extra1} field containing an index 1782 1766 to the individual components. The catalog specified by the entry is 1783 added to the target \ippmisc{minidvo}by the \ippprog{addstar}1767 added to the target mini-DVO by the \ippprog{addstar} 1784 1768 program, updating the measurements in the appropriate DVO tables. 1785 1769 When this completes, an entry containing the statistics of the job is … … 1831 1815 exposures which were believed to be obtained in photometric 1832 1816 conditions. This process, called ``\"ubercal'', is described in 1833 detail by \cite{2012ApJ...756..158S} for the first (PV1) version 1834 \note{add SDSS ref mentioned in Schlafly, also in cal paper}. In1835 brief, photometric periods, with time-scales of a large fraction of a 1836 night, are identified by a combination of automatic analysis and 1837 manual inspection. A single solution for all images in a given filter 1838 is determined to minimize scatter for individual stars. The free1839 parameters in this solution consist of a single zero point and airmass 1840 slope for each photometric period along with a collection of 1841 flat-field offsets for several large time range (``flat-field 1842 seasons''). For the PV3 \"ubercal analysis, the flat-field offsets 1843 were determined on a $2\times2$ grid for each chip and 5 flat-field 1844 seasons were identified. The boundaries of the flat-field seasons 1845 were determined by independent inspection of the residuals observed in 1846 the Medium Deep fields.1817 detail by \cite{2012ApJ...756..158S} for the first (PV1) version \textadd{and 1818 is based on the process of the same name used for SDSS calibration 1819 \citep{2008ApJ...674.1217P}}. In brief, photometric periods, with 1820 time-scales of a large fraction of a night, are identified by a 1821 combination of automatic analysis and manual inspection. A single 1822 solution for all images in a given filter is determined to minimize 1823 scatter for individual stars. The free parameters in this solution 1824 consist of a single zero point and airmass slope for each photometric 1825 period along with a collection of flat-field offsets for several large 1826 time range (``flat-field seasons''). For the PV3 \"ubercal analysis, 1827 the flat-field offsets were determined on a $2\times2$ grid for each 1828 chip and 5 flat-field seasons were identified. The boundaries of the 1829 flat-field seasons were determined by independent inspection of the 1830 residuals observed in the Medium Deep fields. 1847 1831 1848 1832 After the \"ubercal analysis of the photometric periods is completed, … … 1874 1858 Telescopes (MAST). The underlying database at MAST is a copy of a 1875 1859 database generated at the IfA by the Published Science Products 1876 Subsystem (PSPS). The construction of the PSPS version of the PS1 1877 database starts once the PS1 photometry and astrometry measurements 1878 have been calibrated within the DVO system. The construction takes 1879 place in several stages, described in detail in Paper VI. 1880 We summarize those steps here. 1860 Subsystem (PSPS). \textadd{Both MAST and IfA versions of the PSPS are 1861 implemented using a collection of Microsoft SQL Server instances as 1862 the underlying database engine. Like in DVO, the tables holding the 1863 large volume of measurements are distributed across the different 1864 computers based on their location on the sky. Unlike DVO, the spatial 1865 distribution uses slices which span all RA values for a narrow range 1866 of Declinations on a single compter. The PSPS design and 1867 implementation is described in some detail in Paper VI.} 1868 1869 The construction of the PSPS version of the PS1 database starts once 1870 the PS1 photometry and astrometry measurements have been calibrated 1871 within the DVO system. The construction takes place in several 1872 stages, described in detail in Paper VI. We summarize those steps 1873 here. 1881 1874 1882 1875 The first stage of constructing the PSPS database consists of the … … 1982 1975 1983 1976 Within the \code{task.exec} macro, the command to be run is defined by 1984 the script. Once the \code{task.exec} macro exitssuccessfully, the1977 the script. Once the \code{task.exec} macro \mbox{exits} successfully, the 1985 1978 defined command is then added to the list of jobs to be run within the 1986 1979 UNIX environment. Jobs may be run in one of two ways: locally or via … … 2139 2132 2140 2133 Most stages consist of two related tasks: a \ippmisc{load} task, which 2141 is responsible toquerying the processing database to identify entries2134 is responsible for querying the processing database to identify entries 2142 2135 to be processed, and a \ippmisc{run} task, which is responsible for 2143 2136 managing the processing of the individual entries. … … 2234 2227 from other processing attempts. 2235 2228 2236 2237 2238 2229 \subsection{Stage automation} 2239 2230 \label{sec:automation} … … 2250 2241 \ippmisc{ippScript}. These scripts have a well-defined and restricted 2251 2242 set of goals: to ensure that difference images are generated for each 2252 exposure (either by pairing together warps or pair swarps with2243 exposure (either by pairing together warps or pairing warps with 2253 2244 pre-defined stacks), that nightly stacks are generated for MD fields, 2254 and that the stacks are also differenced against an appropriate 2255 reference. 2256 2257 Pairing warps together is simplified by the observing strategy in 2258 which the same pointing is observed multiple times in a night. By 2259 limiting to warp-warp pairs from the same pointing, the problem is 2260 significantly reduced from the arbitrary case. 2261 2262 Queuing the diffs is done by first examining the set of all 2245 and that the nightly stacks are also differenced against an appropriate 2246 reference. 2247 2248 \textmod{For the warp-warp difference images, pairing warps together is 2249 simplified} by the observing strategy in which the same pointing is 2250 observed multiple times in a night. By limiting to warp-warp pairs 2251 from the same pointing, the problem is significantly reduced from the 2252 arbitrary case. 2253 2254 Queuing \textmod{these warp-warp difference images} is done by first examining the set of all 2263 2255 exposures that have been taken at the summit on the current night of 2264 2256 observing, and querying information from each stage up through … … 2267 2259 identifier for each telescope pointing on the sky. Exposures in each 2268 2260 group are then sorted by increasing observation date 2269 (\ippdbcolumn{dateobs}). The database results for each stage 2270 (\ippstage{chip}-\ippstage{warp}) are checked to ensure that the selected exposures have 2271 been successfully processed for all stages through \ippstage{warp}. 2261 (\ippdbcolumn{dateobs}). 2262 2263 The database results for each stage (\ippstage{chip}-\ippstage{warp}) 2264 are checked to ensure that the selected exposures have been 2265 successfully processed for all stages through \ippstage{warp}. 2272 2266 Exposure groups are ignored until all exposures have either been 2273 2267 processed through warp or have failed with a bad quality, meaning the … … 2276 2270 the final exposure ignored in the case of an odd number of accepted 2277 2271 exposures. Exposures paired in this way are sent to the 2278 ippstage{diff} analysis stage. 2272 \ippstage{diff} analysis stage. \textadd{Nightly processing also 2273 ensures that the difference image analysis is run using the warps in 2274 comparison to the reference stack images generated for the full $3\pi$ 2275 region.} 2279 2276 2280 2277 Once observations have been completed for the night (signaled by the … … 2299 2296 number of usable exposures. If no stack could be made for a given MD 2300 2297 field with the minimum number of inputs by the time of the 2301 end-of-night darks, stacks are generated using whatever 2302 exposures are available. 2298 end-of-night darks, stacks are generated using whatever exposures are 2299 available. \textadd{Nightly processing also ensures that the 2300 difference image analysis is run on these nightly stacks using a 2301 pre-defined reference stack.} 2303 2302 2304 2303 The automatic nightly processing ensures that data is processed as … … 2306 2305 observation and the reduced data. 2307 2306 2308 The other processing task that requires automation is the reprocessing 2309 of the entire $3\pi$ survey, as the size of the dataset make it 2310 challenging to do manually. To manage this, the ``large area 2311 processing'' (LAP) task and script are used. The first stage of this 2312 processing is generating an entry in the \ippdbtable{lapSequence} 2313 table defining a new reprocessing. After this, individual 2314 \ippdbtable{lapRun} entries can be queued that define a 2315 \ippdbcolumn{filter} and a \ippdbcolumn{projection_cell} to be 2316 considered. These projection cells match the tangent plane centers 2317 used for the warp tessellation. A \ippdbcolumn{projection_cell} is a 2318 unit of sky defined to be a square four degrees on each side which has 2319 a single tangent plane projection (Paper III). 2320 Once this 2321 entry is defined, it is populated with all exposures (stored in the 2322 \ippdbtable{lapExp} table in the database) that are located 2307 {\bf The other processing task that requires automation is the reprocessing 2308 of the entire $3\pi$ survey, as the size of the dataset makes it 2309 challenging to organize the analysis manually. To manage large-scale 2310 analyses, the ``large area processing'' (LAP) task and script are 2311 used. The first stage of LAP generates an entry in the 2312 \ippdbtable{lapSequence} table defining a new reprocessing. After 2313 this, individual \ippdbtable{lapRun} entries can be queued that define 2314 a \ippdbcolumn{filter} and a \ippdbcolumn{projection_cell} to be 2315 considered. These projection cells corrrespond to the projections used 2316 by the warp tessellation to define the skycells (see 2317 Section~\ref{sec:warp}), which tangent plane centers matching those in 2318 the warp tessellation. For the $3\pi$ survey analysis, a 2319 \ippdbcolumn{projection_cell} is a unit of sky defined to be a square 2320 four degrees on each side which has a single tangent plane projection 2321 (Paper III).} 2322 2323 Once this entry is defined, it is populated with all exposures (stored 2324 in the \ippdbtable{lapExp} table in the database) that are located 2323 2325 within 5 degrees of the center of the projection cell included. This 2324 2326 radius ensures that any exposure that overlaps the projection cell … … 2381 2383 Pan-STARRS cluster. 2382 2384 2385 All of the IPP low-level C-based processing programs (e.g., 2386 \ippprog{ppImage} and \ippprog{ppStack} interact with Nebulous to find 2387 existing files and to create new files. The supporting Perl scripts 2388 also interact with Nebulous to perform file instance duplication as 2389 needed and to check for the existence of required input files and 2390 expected output files. 2391 2383 2392 \subsubsection{Implementation Details} 2384 2393 … … 2479 2488 impact processing. 2480 2489 2481 The nebulous user APIs do not interact directly with the nebulous2490 The nebulous user APIs do not interact directly with the Nebulous 2482 2491 mysql database. Instead, they interact with one of several computers 2483 2492 with an Apache web server. Interactions with the Apache server are 2484 2493 performed using the Simple Object Access Protocol (SOAP) interface, 2485 while the Apache servers interact directly with the Mysql database2494 while the Apache servers interact directly with the mysql database 2486 2495 server. This architecture avoids the overhead of setting up and 2487 tearing down the Mysql connection for each Nebulous command, instead2496 tearing down the mysql connection for each Nebulous command, instead 2488 2497 using only the low-latency SOAP communications. 2489 2498 … … 2679 2688 servers used as database replicants, which allow for quick switching 2680 2689 from the main to backup servers in case of a hardware issue that 2681 cannot be resolved immediately. 2690 cannot be resolved immediately. \textadd{The IPP uses a set of three 2691 computers to host the Nebulous mysql database and live back-ups. A 2692 second set of computers are used to host the processing database and 2693 backups.} 2682 2694 2683 2695 \subsection{Los Alamos National Labs} … … 2813 2825 2814 2826 \section{Conclusion} 2827 2828 We began the development of the IPP in early 2004, soon after the 2829 initial funding for the construction of the Pan-STARRS telescopes was 2830 awarded to U.H. The landscape of the software and computing world has 2831 changed in a number of ways. Some of the decisions we made at the 2832 beginning have held up well while in other cases we would probably 2833 make a different choice today. 2834 2835 One choice we made early on was to develop new code for the data 2836 analysis programs. This choice was driven partly by some of our 2837 experiences with the existing major systems of the time. We were 2838 advised by those with close experience with the SDSS data analysis 2839 code base against attempting to modify that system for our purposes. 2840 It was also our opinion that the IRAF suite of packages were not 2841 well-suited to the large-scale automated pipeline needed for the 2842 Pan-STARRS data. The Pan-STARRS data analysis rate was going to 2843 surpass previous astronomical projects, and the cameras (with 60 2844 detectors each of 64 cells) would have an unprecedented level of 2845 complexity. The original survey was intended to run for 10 years, so 2846 long-term supportability was also a priority. With these design 2847 constraints in mind, we decided to develop a new code base which would 2848 be able to address the data rate and complexity. 2849 2850 In our design, we have tried to make the analysis programs as generic 2851 as possible, with all instrument-specific details addressed in the 2852 configuration files. Our implementation has been generally successful 2853 in this regard. The \ippprog{ppImage} program contains most of the 2854 highly-specific detrending details, with much more limited 2855 camera-specific features needed in the configuration files for 2856 \ippprog{psastro} and \ippprog{pswarp}. This generalization of the 2857 software has made it easy to run the full analysis pipeline on other 2858 cameras, both for testing and for other science analysis projects. We 2859 have used the full IPP analysis system for data from the CFHT Megacam 2860 and CFH12K cameras as well as the Subaru Hypersuprime Camera. The 2861 generalization made is relatively simple to add the second telescope 2862 and camera (PS2 + GPC2) to the regular processing when they came 2863 online for science operations in 2018. 2864 2865 In retrospect, the additional design and coding effort needed to keep 2866 the system general were worthwhile and have paid off. However, if we 2867 were to start from scratch today, we would probably choose to adapt 2868 the LSST pipeline for our use since it has been developed with some of 2869 the same constraints. 2870 2871 One early choice we made was to use standard C and to use Perl as a 2872 wrapper language. We considered other language choices, including C++ 2873 and Python. At the time, Python was fairly new and did not have the 2874 wide-spread acceptance it has today. In retrospect, our choice of 2875 Perl has not held up very well. The capabiliaties available within 2876 the Python environment would have allowed us to include interesting 2877 visualization and other high-level analysis options. It is also 2878 easier to hire astronomers with good Python coding skills that Perl 2879 coding skills. 2880 2881 We also find that maintaining support for our Perl code has been a 2882 challenge: changes to the Perl language syntax and changes in 2883 externally supported Perl modules have required significant effort to 2884 keep our code compatible with the changes. It is not obvious that 2885 Python would obviate that particular problem, however. 2886 2887 One important aspect of the design of the IPP is to use a single 2888 database to manage the processing stages, with regular queries to the 2889 database to choose the tasks which are ready to proceed. Other 2890 choices were possible. In some pipelined processing systems, jobs 2891 which complete trigger the next processing step. For example, 2892 \ippprog{ppImage} or its wrapper (\ippprog{chip_imfile.pl}) could have 2893 been responsible for launching the \ippprog{psastro} analysis. 2894 Alternatively, a manager process could be responsible for launching 2895 the next processing step when one step has completed. For example, 2896 \ippprog{pantasks} could note when the \ippprog{ppImage} jobs were 2897 complete and launch the \ippprog{psastro} analysis. Both of these 2898 choices can potentially result in lower latency since the next step is 2899 in principle run immediately when the previous step is completed. Our 2900 choice has two important advantages: First, error and failure recovery 2901 are trivial. If one of the many programs fails or is interrupted, the 2902 system can easily notice and retry the job. In a triggered system, a 2903 failure of one stage could mean the trigger never happens. Some 2904 external cleanup system would need to be implemented to check for the 2905 failures and re-launch. The second advantage of our design is that 2906 each analysis stage is highly independent and can thus be flexibly run 2907 in different ways. For example, alternative test systems can run in 2908 parallel with the nightly operations system, using the outputs of the 2909 nightly processing by simple changes to the queries used to select the 2910 elements for an analysis stage. In addition, it is easy to add new 2911 stages since they do not need to be injected into the standard 2912 processing manager (\ippprog{pantasks}). 2913 2914 The main challenge related to this database-managed design is that the 2915 database can become a bottleneck. If the queries used to select the 2916 processing items become too large and too slow, the whole system can 2917 be slowed down. Care must the taken to avoid poorly implemented 2918 queries, and in some cases the queries need to be restricted. For 2919 example, if too many items are queued for processing at one time under 2920 the same processing label, the associated queries can bog down. This 2921 issue is one of the reasons we manage the large-scale processing with 2922 the LAP system since it provides a method to automatically limit the 2923 scale of the queries. In addition, it is critical that the database 2924 hardware be sufficiently powerful to keep up with the demand from the 2925 processing system. 2926 2927 Finally, the choice to use Nebulous as a file management system is 2928 ambiguous. When we began this project, the existing cluster file 2929 systems did not seem to match the level of our project. Some were 2930 will very much in an early development state (e.g., GFS from Red Hat), 2931 while others seemed designed for much larger-scale systems, with very 2932 expensive hardware requirements (e.g., Lustre). The requirements for 2933 the filesystem for Pan-STARRS are somewhat different from the 2934 large-scale computing clusters used by the national labs. Since the 2935 data processing is very parallel, we do not have any strong 2936 requirements on data access concurency. In theory, we could have 2937 simply used NFS and made backup copies of the files using some simple 2938 name-convention rules. We decided to implement the Nebulous system to 2939 allow the targetted analysis and to automate the replication of the 2940 data. In retrospect, the system has succeeded in these goals and has 2941 behaved reliably. However, the support level has been somewhat high, 2942 especially when we have needed to migrate large amounts of data within 2943 the cluster. If we were to start from scratch today, we would 2944 experiment with some of the existing cluster file systems. 2815 2945 2816 2946 Since the Pan-STARRS\,1 telescope first came online in 2007, this
Note:
See TracChangeset
for help on using the changeset viewer.
