Index: trunk/doc/design/ippCDRresponse.tex =================================================================== --- trunk/doc/design/ippCDRresponse.tex (revision 8140) +++ trunk/doc/design/ippCDRresponse.tex (revision 8703) @@ -20,8 +20,8 @@ % -- Revision History -- -\RevisionsStart +%\RevisionsStart % version Date Description -DR.01 & 2006.07.28 & First Draft \\ \hline -\RevisionsEnd +%00 & 2006.08.13 & Initial Release \\ \hline +%\RevisionsEnd \tableofcontents @@ -104,9 +104,9 @@ This question appears to be principally concerned with perceived overlap between the IPP DVO system and the PSPS Object Data Manager -(ODM). Where there is superficial similarity between the systems, it +(ODM). While there is superficial similarity between the systems, it is our belief that the development of two complementary systems is not -only worthwhile, but extremely important for a variety of reasons. - -The IPP operational requirments include the analysis of images to +only worthwhile, but extremely important for a variety of reasons. + +The IPP operational requirements include the analysis of images to produce the P2, P4$\Delta$, and P4$\Sigma$ detections, the Static Sky images, the Astrometric and Photometric Reference catalog, and the @@ -128,13 +128,13 @@ and objects, along with the photometric and astrometric calibration information. There are several motivations for defining a separate -entity within the IPP specifially for this task: +entity within the IPP specifically for this task: \begin{itemize} -\item the PSPS system is foreseen as the definitive view on the object +\item The PSPS system is foreseen as the definitive view on the object database problem, while the IPP requires the ability to manipulate and update the astrometric and photometric calibrations. -\item the PSPS system requires a high-level of sophistication in the +\item The PSPS system requires a high-level of sophistication in the scientific queries which may be performed. While critical, this - sophistication will result in a much longer development timeline for + sophistication will result in a much longer development time line for the PSPS. \item By performing the analysis of the astrometric and photometric @@ -146,9 +146,9 @@ trade-off allows the IPP DVO system to operate at a much higher throughput than is possible for the PSPS, and perform more - reprocessing operations. + re-processing operations. \end{itemize} It is useful to ask if the PSPS could have been ready within the -necessary timeframe, would the IPP use that instead of the DVO system +necessary time frame, would the IPP use that instead of the DVO system for support of photometric and astrometric calibrations. The answer would be 'yes' only if the PSPS requirements were modified to allow it @@ -157,5 +157,5 @@ subsystems after they have both matured? My opinion, and I believe, that of the project management, is that this choice would result in -development of a system which would be substantially overengineered +development of a system which would be substantially over-engineered for its role in either of the two subsystems, with concomitant increase in the cost of development and support. By dividing the @@ -187,5 +187,5 @@ psphot and ppImage upgrades. \item {\bf Release Capabilities} Image warping, Image differencing, - Image stacking, object modelling for difference images (loading PSF + Image stacking, object modeling for difference images (loading PSF from an external source, fitting positive and negative sources), analysis of the OTA guide kernel, detrend images convolved with the @@ -220,7 +220,7 @@ We have worked with the other PS1 teams to define complete ICDs between all of the interacting subsystems. The IPP interfaces with 4 -existing subsystems, and will interact with several other science -clients. We have completed the definitions of the ICDs for the -Camera-IPP (PSDC-940-003), OTIS-IPP (PSDC-940-004), IPP-MOPS +existing subsystems, and will interact with other as-yet-undefined +science clients. We have completed the definitions of the ICDs for +the Camera-IPP (PSDC-940-003), OTIS-IPP (PSDC-940-004), IPP-MOPS (PSDC-940-005), and IPP-PSPS (PSDC-940-006). These are now part of the PSDC document tree. @@ -277,5 +277,5 @@ The CDR Review Committee requests a plan for finalizing the analysis algorithms within the IPP. There are three stages to freezing the -analysis algorithims within the IPP: +analysis algorithms within the IPP: \begin{itemize} \item Algorithm conceptual design. @@ -287,18 +287,18 @@ The IPP SSDD defines the conceptual details of nearly all of the analysis algorithms in Sections 5, 6, and 7. Of these descriptions, -only Sections 5.5.5, describing the stacking of the Static Sky, -and Section 7, defining the Static Sky photometry analysis, are -insufficient in detail for the analysis to be well understood. Also -missing from version 00 of the document is a discussion of the -creation of the astrometric and photometric reference catalogs. A new -version of the SSDD with these three sections updated \note{will be -posted by DATE}. +only Sections 5.5.5, describing the stacking of the Static Sky, and +Section 7.2, defining the Static Sky photometry analysis, are +insufficient in detail for the analysis to be well understood. The +discussion of the analysis used to generate the astrometric and +photometric reference catalog are included in Section 4 on the design +of DVO, and are somewhat sparse. A new version of the SSDD with these +three sections updated will be posted by Aug 30. Table~\ref{algorithms} lists the IPP algorithms, the relevant program in the IPP tree, the development state of the program with respect to the identified analysis, and the data needed to set the algorithm -parameters. In a separate document (REF), we present a detailed -discussion of the choices to be made and guidelines on making those -choices. +parameters. In a separate document (`How to Define IPP Analysis +Parameters'), we present a detailed discussion of the choices to be +made and guidelines on making those choices. There are a handful of decisions which need to be made which have an @@ -332,11 +332,15 @@ analysis be minimized since it is impossible to afford the I/O load demanded by a large number of input fringe images. A related - question is that of how to subselect the night-time fringe images + question is that of how to sub-select the night-time fringe images for best effect, if sky fringe images are used. Based on experience from CFHT/Megacam, it may be possible to use fringe images selected on the basis of the time of night, but this must be tested for - Haleakala. It seems unlikely at this time that a spectral skyprobe - will be available for the start of PS1, so we cannot rely on such a - device to guide our choices. + Haleakala. An alternative strategy, in which master fringe images + are combined based on their consistency with the science image, has + also been implemented for the IPP. The decision between these + options will be guided by further testing of Megacam images and real + Haleakala data in a range of conditions. It seems unlikely at this + time that a spectral skyprobe will be available for the start of + PS1, so we cannot rely on such a device to guide our choices. \item {\bf Static Sky Cell definition} What is the layout of the @@ -376,5 +380,5 @@ range of field angles into the static sky. These will have a range of image qualities. We will need to set the cuts to trade-off - between degredation of the final image quality versus degredation of + between degradation of the final image quality versus degradation of the signal-to-noise in the final image. In general, our guidance for the Static Sky is to maximize our ability to measure accurate @@ -391,5 +395,5 @@ common seeing (for a stable PSF across the image for difference image)? Is it possible to measure the weak lensing parameters - sufficiently accuractly from the stacked image with knowledge of the + sufficiently accurately from the stacked image with knowledge of the PSF? To what level of detail is the PSF model required? Is it necessary to perform the weak lensing analysis (and other galaxy @@ -410,29 +414,31 @@ steps. The functional flow of these different analysis steps can be seen in IPP SSDD Figures 22-30. The use of the Q/A measurements is -not summarized very clearly within the text of version 00; \note{this -will be updated in the new SSDD release}. Within the IPP, the -analysis stages use these measurements to mark input images as -accepted or rejected. These assessments are passed back to the OTIS -subsystem, along with the image statistics measured for the input -images. OTIS has the option of setting more stringent filters on the -input images and re-observing images on the basis of the IPP feedback, -even if the IPP accepted images which OTIS re-observes. There is also -a system-wide plan in place to use feedback from the IPP and from OTIS -to guide the project's choices for survey strategies and science -goals. +not summarized very clearly within the text of version 00; this will +be updated in the new SSDD release. The types of Q/A measurements are +also discussed below. + +Within the IPP, the analysis stages use these measurements to mark +input images as accepted or rejected. These assessments are passed +back to the OTIS subsystem, along with the image statistics measured +for the input images. OTIS has the option of setting more stringent +filters on the input images and re-observing images on the basis of +the IPP feedback, even if the IPP accepted images which OTIS +re-observes. There is also a system-wide plan in place to use +feedback from the IPP and from OTIS to guide the project's choices for +survey strategies and science goals. \subsection{Detrend Images} -The input detrend images all have their pixel count levels measured -for each chip. Input images which have counts or fluxes outside of a +The input detrend images have their pixel count levels measured for +each chip. Input images which have counts or fluxes outside of a defined range will be flagged and excluded from any detrend analysis. For example, the flat-field images should never use input images which are saturated, nor should the dark image analysis use input images with flux levels wildly outside of the nominal range. Both conditions -are evidence that the observing process was performed inappropriately. +are evidence that the data were incorrectly obtained. In addition to raw pixel values, the input detrend images are -contronted with the resulting master detrend images. In general, the -effect corrected by the master detrend image should adaquately correct +confronted with the resulting master detrend images. In general, the +effect corrected by the master detrend image should adequately correct each of the input raw detrend images. The residual scatter of the detrended raw images should be small. As part of the detrend creation @@ -483,5 +489,5 @@ scatter, and substantial photometric offsets are all evidence of problems with the observing conditions. These may be the presence - of clouds and/or haze, degredation of the optics, and/or extreme + of clouds and/or haze, degradation of the optics, and/or extreme image-quality problems. \end{itemize} @@ -500,5 +506,5 @@ result in large deviations between the components of an image stack, and will also result in large numbers of difference image detections. -Similarly, bright stars with larger than expected halos or saturation +Similarly, bright stars with larger-than-expected halos or saturation regions will result in excess difference image detections. @@ -519,22 +525,22 @@ hardware to meet the processing and I/O requirements. -That analysis is based largely on prototype tests of our processing -algorithms, and is somewhat limited by being focused on the steady -state operations. We present here new numbers for the processing -timeline based on the current baseline software on our existing -baseline cluster hardware. - -For PS1, there is a significant processing challenge in the first 6 - -9 months when only a fraction of the IPP storage hardware will be -available. This period is further complicated by the budgetary -constraints placed on the IPP to limit the hardware purchase to a bare -minimum. An important area for clarification by the project is the -processing requirement in the beginning of the project. If the IPP is -required to perform a complete Static Sky analysis on every image as -it becomes available, then the total hardware required in the first -6-9 months for processing must be increased. If it is only necessary -to stack sets of, for example, 4 images as they are available, then -the requirements are somewhat reduced. A trade-off must be made by -the project to choose between these options. +That analysis was based largely on prototype tests of our processing +algorithms and has been made out-of-date by changes to the survey +plans. We present here new numbers for the processing time line based +on the current baseline software on our existing baseline cluster +hardware. + +% For PS1, there is a significant processing challenge in the first 6 - +% 9 months when only a fraction of the IPP storage hardware will be +% available. This period is further complicated by the budgetary +% constraints placed on the IPP to limit the hardware purchase to a bare +% minimum. An important area for clarification by the project is the +% processing requirement in the beginning of the project. If the IPP is +% required to perform a complete Static Sky analysis on every image as +% it becomes available, then the total hardware required in the first +% 6-9 months for processing must be increased. If it is only necessary +% to stack sets of, for example, 4 images as they are available, then +% the requirements are somewhat reduced. A trade-off must be made by +% the project to choose between these options. The data storage requirements are determined from the design reference @@ -546,7 +552,7 @@ images per year. Combining these two, we find that the total number of raw image data is roughly 1.7PB (555,000 images). In addition, we -have a requirement for Static Sky storage (using 0.2 arcsec pixels) of -roughly 300TB, and miscellaneous additional storage of nearly 100 TB. -Our hardware purchase plan has a minimum total storage of 2.4PB, +have a requirement for Static Sky storage (assuming 0.2 arcsec pixels) +of roughly 300TB, and miscellaneous additional storage of nearly 100 +TB. Our hardware purchase plan has a minimum total storage of 2.4PB, giving us a margin of about 10\%. Our plan is to purchase the hardware in 5 stages of 16 computers each, for a total system cost of @@ -558,5 +564,6 @@ required to buy all of the machines up front), we would require roughly 145 machines, increasing the cost of the cluster to a total of -roughly \$2.2M. We judge this to be a very low risk. +roughly \$2.2M. We judge this to be a very low risk as hard disk +capacities continue to grow. We have performed timing test of the current versions of the IPP tools @@ -572,13 +579,14 @@ speeds of the CPU cores, but increasing numbers of cores per socket. By the end of this year, quad-core processors are expected. If we -stagger the purchase of the computers as planned, and make reasonable -estimates for the number of cores available, we expect the final -cluster configuration to have between 400 and 800 cores. We will use -600 as an estimate. Note that it is possible to supplement the -processing power of the cluster by buying 1U boxes with processors but -no storage. Each of these boxes cost roughly 15\% of a storage node -and add an equal number of processor cores. Such an option can be -taken at any time, though it is not needed in our current development -plan. +stagger the purchase of the computers as planned (5 sets of 16 +machines), and make reasonable estimates for the number of cores +available, we expect the final cluster configuration to have between +400 and 800 cores. We will use 600 as an estimate. Note that it is +possible to supplement the processing power of the cluster by buying +1U boxes with processors but minimal storage. Each of these boxes +cost roughly 15\% of a storage node and add an equal number of +processor cores. Such an option can be taken at any time to +supplement the raw processing power, though it is not needed in our +current development plan. There are two potentially dominant analysis steps in the process: the @@ -589,11 +597,11 @@ takes roughly 16 seconds for a Megacam chip (single core), equivalent to roughly 38 seconds on a full GPC-1 chip. Most other steps of the -analysis scale are constant per image, and contribute only a few -seconds relative to the 38 seconds. We use 50 seconds per chip per -core to judge the total processing power for the portion which scales -by the number of images. +analysis require a constant amount of time per image, and contribute +only a few seconds relative to the 38 seconds. We use 50 seconds per +chip per core to judge the total processing power for the portion +which scales by the number of images. A useful statistic to judge the capability of the processing system is -the time required to reprocess all images at the end of the survey. +the time required to re-process all images at the end of the survey. Given the total number of images above (555,000), the per-image analysis portion of the processing would require a total of $\sim 1.8 @@ -617,8 +625,9 @@ predict the Pan-STARRS magnitudes of stars, then extrapolated the source counts to our magnitudes limits. We find 50,000 objects per -square degree above our threshold in this region. If every image -required the non-linear fitting for this density of objects, and we -accept the 10ms time, this analysis would require a total of $\sim 2.0 -\times 10^9$ CPU core-seconds, or about 39 days on the 600 cores. +square degree above our threshold in this region. If we make the +conservative assumption that every image required the non-linear +fitting for this density of objects, and that each object requies +10ms, this analysis would require a total of $\sim 2.0 \times 10^9$ +CPU core-seconds, or about 39 days on the 600 cores. In conclusion, given the assumptions above, the processing power of @@ -650,24 +659,26 @@ requirements of the IPP for object databasing. Some effort has been needed to make DVO completely suitable for its role within the IPP. -Regardless of what object databasing system was chosen, a certain -level of effort would have been required. In this case, we were clear -just how much would be required, and it was not large. +Regardless of what object databasing system was chosen, however, a +certain level of effort would have been required. In the case of DVO, +we were clear what effort was required, and it was judged to be +reasonable. Of that effort, only the ability to support older table formats was -required to maintain CFHT Elixir compatibility. In fact, this is a -feature which we would have added even if we did not want to maintain -compatibility with CFHT's DVO installation. We have found in our -experience with the Elixir system that having a rigidly defined schema -hindered the usability and extensibility of the DVO system. The fixed -tables made it difficult to add new elements to the database, and -required multiple versions to support previously defined tables. The -new design allows us to be more flexible about changes without fear -that this will break database instances which already exist. One of -the best ways we have found to test the DVO object databasing system -is to engage students in science projects using DVO. These projects -explore the user interface and highlight problems and areas for -possible improvement. Such projects would not be possible if the -users feared that their DVO instances would be unusable in the future -because of lack of backwards compatibility. +required to maintain compatibility with the existing CFHT DVO +databases. In fact, this is a feature which we would have added even +if we did not want to maintain compatibility with CFHT's DVO +installation. We have found in our experience with the Elixir system +that having a rigidly defined schema hindered the usability and +extensibility of the DVO system. The fixed tables made it difficult +to add new elements to the database, and required multiple versions to +support previously defined tables. The new design allows us to be +more flexible about changes without fear that this will break database +instances which already exist. One of the best ways we have found to +test the DVO object databasing system is to engage students in science +projects using DVO. These projects explore the user interface and +highlight problems and areas for possible improvement. Such projects +would not be possible if the users feared that their DVO instances +would be unusable in the future because of lack of backwards +compatibility. The PanTasks system used the existing Opihi command-line interface @@ -684,5 +695,5 @@ of a large set of real images obtained by the CFHT engineering staff over several years. We also gain by discussions with our Elixir -collegues about details of the analysis and possible sources of errors +colleagues about details of the analysis and possible sources of errors observed in the CFHT dataset. The only cost to the IPP is in preventing excessive forking of the DVO databasing system, something