Changeset 8703 for trunk/doc/design/ippCDRresponse.tex
- Timestamp:
- Aug 29, 2006, 6:19:30 PM (20 years ago)
- File:
-
- 1 edited
-
trunk/doc/design/ippCDRresponse.tex (modified) (23 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/design/ippCDRresponse.tex
r8140 r8703 20 20 21 21 % -- Revision History -- 22 \RevisionsStart22 %\RevisionsStart 23 23 % version Date Description 24 DR.01 & 2006.07.28 & First Draft\\ \hline25 \RevisionsEnd24 %00 & 2006.08.13 & Initial Release \\ \hline 25 %\RevisionsEnd 26 26 27 27 \tableofcontents … … 104 104 This question appears to be principally concerned with perceived 105 105 overlap between the IPP DVO system and the PSPS Object Data Manager 106 (ODM). Wh ere there is superficial similarity between the systems, it106 (ODM). While there is superficial similarity between the systems, it 107 107 is our belief that the development of two complementary systems is not 108 only worthwhile, but extremely important for a variety of reasons. 109 110 The IPP operational requir ments include the analysis of images to108 only worthwhile, but extremely important for a variety of reasons. 109 110 The IPP operational requirements include the analysis of images to 111 111 produce the P2, P4$\Delta$, and P4$\Sigma$ detections, the Static Sky 112 112 images, the Astrometric and Photometric Reference catalog, and the … … 128 128 and objects, along with the photometric and astrometric calibration 129 129 information. There are several motivations for defining a separate 130 entity within the IPP specifi ally for this task:130 entity within the IPP specifically for this task: 131 131 132 132 \begin{itemize} 133 \item the PSPS system is foreseen as the definitive view on the object133 \item The PSPS system is foreseen as the definitive view on the object 134 134 database problem, while the IPP requires the ability to manipulate 135 135 and update the astrometric and photometric calibrations. 136 \item the PSPS system requires a high-level of sophistication in the136 \item The PSPS system requires a high-level of sophistication in the 137 137 scientific queries which may be performed. While critical, this 138 sophistication will result in a much longer development time line for138 sophistication will result in a much longer development time line for 139 139 the PSPS. 140 140 \item By performing the analysis of the astrometric and photometric … … 146 146 trade-off allows the IPP DVO system to operate at a much higher 147 147 throughput than is possible for the PSPS, and perform more 148 re processing operations.148 re-processing operations. 149 149 \end{itemize} 150 150 151 151 It is useful to ask if the PSPS could have been ready within the 152 necessary time frame, would the IPP use that instead of the DVO system152 necessary time frame, would the IPP use that instead of the DVO system 153 153 for support of photometric and astrometric calibrations. The answer 154 154 would be 'yes' only if the PSPS requirements were modified to allow it … … 157 157 subsystems after they have both matured? My opinion, and I believe, 158 158 that of the project management, is that this choice would result in 159 development of a system which would be substantially over engineered159 development of a system which would be substantially over-engineered 160 160 for its role in either of the two subsystems, with concomitant 161 161 increase in the cost of development and support. By dividing the … … 187 187 psphot and ppImage upgrades. 188 188 \item {\bf Release Capabilities} Image warping, Image differencing, 189 Image stacking, object model ling for difference images (loading PSF189 Image stacking, object modeling for difference images (loading PSF 190 190 from an external source, fitting positive and negative sources), 191 191 analysis of the OTA guide kernel, detrend images convolved with the … … 220 220 We have worked with the other PS1 teams to define complete ICDs 221 221 between all of the interacting subsystems. The IPP interfaces with 4 222 existing subsystems, and will interact with several other science223 clients. We have completed the definitions of the ICDs for the 224 Camera-IPP (PSDC-940-003), OTIS-IPP (PSDC-940-004), IPP-MOPS222 existing subsystems, and will interact with other as-yet-undefined 223 science clients. We have completed the definitions of the ICDs for 224 the Camera-IPP (PSDC-940-003), OTIS-IPP (PSDC-940-004), IPP-MOPS 225 225 (PSDC-940-005), and IPP-PSPS (PSDC-940-006). These are now part of 226 226 the PSDC document tree. … … 277 277 The CDR Review Committee requests a plan for finalizing the analysis 278 278 algorithms within the IPP. There are three stages to freezing the 279 analysis algorith ims within the IPP:279 analysis algorithms within the IPP: 280 280 \begin{itemize} 281 281 \item Algorithm conceptual design. … … 287 287 The IPP SSDD defines the conceptual details of nearly all of the 288 288 analysis algorithms in Sections 5, 6, and 7. Of these descriptions, 289 only Sections 5.5.5, describing the stacking of the Static Sky, 290 and Section 7, defining the Static Sky photometry analysis, are291 insufficient in detail for the analysis to be well understood. Also292 missing from version 00 of the document is a discussion of the 293 creation of the astrometric and photometric reference catalogs. A new 294 version of the SSDD with these three sections updated \note{will be295 posted by DATE}.289 only Sections 5.5.5, describing the stacking of the Static Sky, and 290 Section 7.2, defining the Static Sky photometry analysis, are 291 insufficient in detail for the analysis to be well understood. The 292 discussion of the analysis used to generate the astrometric and 293 photometric reference catalog are included in Section 4 on the design 294 of DVO, and are somewhat sparse. A new version of the SSDD with these 295 three sections updated will be posted by Aug 30. 296 296 297 297 Table~\ref{algorithms} lists the IPP algorithms, the relevant program 298 298 in the IPP tree, the development state of the program with respect to 299 299 the identified analysis, and the data needed to set the algorithm 300 parameters. In a separate document ( REF), we present a detailed301 discussion of the choices to be made and guidelines on making those302 choices.300 parameters. In a separate document (`How to Define IPP Analysis 301 Parameters'), we present a detailed discussion of the choices to be 302 made and guidelines on making those choices. 303 303 304 304 There are a handful of decisions which need to be made which have an … … 332 332 analysis be minimized since it is impossible to afford the I/O load 333 333 demanded by a large number of input fringe images. A related 334 question is that of how to sub select the night-time fringe images334 question is that of how to sub-select the night-time fringe images 335 335 for best effect, if sky fringe images are used. Based on experience 336 336 from CFHT/Megacam, it may be possible to use fringe images selected 337 337 on the basis of the time of night, but this must be tested for 338 Haleakala. It seems unlikely at this time that a spectral skyprobe 339 will be available for the start of PS1, so we cannot rely on such a 340 device to guide our choices. 338 Haleakala. An alternative strategy, in which master fringe images 339 are combined based on their consistency with the science image, has 340 also been implemented for the IPP. The decision between these 341 options will be guided by further testing of Megacam images and real 342 Haleakala data in a range of conditions. It seems unlikely at this 343 time that a spectral skyprobe will be available for the start of 344 PS1, so we cannot rely on such a device to guide our choices. 341 345 342 346 \item {\bf Static Sky Cell definition} What is the layout of the … … 376 380 range of field angles into the static sky. These will have a range 377 381 of image qualities. We will need to set the cuts to trade-off 378 between degr edation of the final image quality versus degredation of382 between degradation of the final image quality versus degradation of 379 383 the signal-to-noise in the final image. In general, our guidance 380 384 for the Static Sky is to maximize our ability to measure accurate … … 391 395 common seeing (for a stable PSF across the image for difference 392 396 image)? Is it possible to measure the weak lensing parameters 393 sufficiently accura ctly from the stacked image with knowledge of the397 sufficiently accurately from the stacked image with knowledge of the 394 398 PSF? To what level of detail is the PSF model required? Is it 395 399 necessary to perform the weak lensing analysis (and other galaxy … … 410 414 steps. The functional flow of these different analysis steps can be 411 415 seen in IPP SSDD Figures 22-30. The use of the Q/A measurements is 412 not summarized very clearly within the text of version 00; \note{this 413 will be updated in the new SSDD release}. Within the IPP, the 414 analysis stages use these measurements to mark input images as 415 accepted or rejected. These assessments are passed back to the OTIS 416 subsystem, along with the image statistics measured for the input 417 images. OTIS has the option of setting more stringent filters on the 418 input images and re-observing images on the basis of the IPP feedback, 419 even if the IPP accepted images which OTIS re-observes. There is also 420 a system-wide plan in place to use feedback from the IPP and from OTIS 421 to guide the project's choices for survey strategies and science 422 goals. 416 not summarized very clearly within the text of version 00; this will 417 be updated in the new SSDD release. The types of Q/A measurements are 418 also discussed below. 419 420 Within the IPP, the analysis stages use these measurements to mark 421 input images as accepted or rejected. These assessments are passed 422 back to the OTIS subsystem, along with the image statistics measured 423 for the input images. OTIS has the option of setting more stringent 424 filters on the input images and re-observing images on the basis of 425 the IPP feedback, even if the IPP accepted images which OTIS 426 re-observes. There is also a system-wide plan in place to use 427 feedback from the IPP and from OTIS to guide the project's choices for 428 survey strategies and science goals. 423 429 424 430 \subsection{Detrend Images} 425 431 426 The input detrend images all have their pixel count levels measured427 foreach chip. Input images which have counts or fluxes outside of a432 The input detrend images have their pixel count levels measured for 433 each chip. Input images which have counts or fluxes outside of a 428 434 defined range will be flagged and excluded from any detrend analysis. 429 435 For example, the flat-field images should never use input images which 430 436 are saturated, nor should the dark image analysis use input images 431 437 with flux levels wildly outside of the nominal range. Both conditions 432 are evidence that the observing process was performed inappropriately.438 are evidence that the data were incorrectly obtained. 433 439 434 440 In addition to raw pixel values, the input detrend images are 435 con tronted with the resulting master detrend images. In general, the436 effect corrected by the master detrend image should ad aquately correct441 confronted with the resulting master detrend images. In general, the 442 effect corrected by the master detrend image should adequately correct 437 443 each of the input raw detrend images. The residual scatter of the 438 444 detrended raw images should be small. As part of the detrend creation … … 483 489 scatter, and substantial photometric offsets are all evidence of 484 490 problems with the observing conditions. These may be the presence 485 of clouds and/or haze, degr edation of the optics, and/or extreme491 of clouds and/or haze, degradation of the optics, and/or extreme 486 492 image-quality problems. 487 493 \end{itemize} … … 500 506 result in large deviations between the components of an image stack, 501 507 and will also result in large numbers of difference image detections. 502 Similarly, bright stars with larger thanexpected halos or saturation508 Similarly, bright stars with larger-than-expected halos or saturation 503 509 regions will result in excess difference image detections. 504 510 … … 519 525 hardware to meet the processing and I/O requirements. 520 526 521 That analysis is based largely on prototype tests of our processing522 algorithms , and is somewhat limited by being focused on the steady523 state operations. We present here new numbers for the processing 524 timeline based on the current baseline software on our existing 525 baseline clusterhardware.526 527 For PS1, there is a significant processing challenge in the first 6 -528 9 months when only a fraction of the IPP storage hardware will be529 available. This period is further complicated by the budgetary530 constraints placed on the IPP to limit the hardware purchase to a bare531 minimum. An important area for clarification by the project is the532 processing requirement in the beginning of the project. If the IPP is533 required to perform a complete Static Sky analysis on every image as534 it becomes available, then the total hardware required in the first535 6-9 months for processing must be increased. If it is only necessary536 to stack sets of, for example, 4 images as they are available, then537 the requirements are somewhat reduced. A trade-off must be made by538 the project to choose between these options.527 That analysis was based largely on prototype tests of our processing 528 algorithms and has been made out-of-date by changes to the survey 529 plans. We present here new numbers for the processing time line based 530 on the current baseline software on our existing baseline cluster 531 hardware. 532 533 % For PS1, there is a significant processing challenge in the first 6 - 534 % 9 months when only a fraction of the IPP storage hardware will be 535 % available. This period is further complicated by the budgetary 536 % constraints placed on the IPP to limit the hardware purchase to a bare 537 % minimum. An important area for clarification by the project is the 538 % processing requirement in the beginning of the project. If the IPP is 539 % required to perform a complete Static Sky analysis on every image as 540 % it becomes available, then the total hardware required in the first 541 % 6-9 months for processing must be increased. If it is only necessary 542 % to stack sets of, for example, 4 images as they are available, then 543 % the requirements are somewhat reduced. A trade-off must be made by 544 % the project to choose between these options. 539 545 540 546 The data storage requirements are determined from the design reference … … 546 552 images per year. Combining these two, we find that the total number 547 553 of raw image data is roughly 1.7PB (555,000 images). In addition, we 548 have a requirement for Static Sky storage ( using 0.2 arcsec pixels) of549 roughly 300TB, and miscellaneous additional storage of nearly 100 TB. 550 Our hardware purchase plan has a minimum total storage of 2.4PB,554 have a requirement for Static Sky storage (assuming 0.2 arcsec pixels) 555 of roughly 300TB, and miscellaneous additional storage of nearly 100 556 TB. Our hardware purchase plan has a minimum total storage of 2.4PB, 551 557 giving us a margin of about 10\%. Our plan is to purchase the 552 558 hardware in 5 stages of 16 computers each, for a total system cost of … … 558 564 required to buy all of the machines up front), we would require 559 565 roughly 145 machines, increasing the cost of the cluster to a total of 560 roughly \$2.2M. We judge this to be a very low risk. 566 roughly \$2.2M. We judge this to be a very low risk as hard disk 567 capacities continue to grow. 561 568 562 569 We have performed timing test of the current versions of the IPP tools … … 572 579 speeds of the CPU cores, but increasing numbers of cores per socket. 573 580 By the end of this year, quad-core processors are expected. If we 574 stagger the purchase of the computers as planned, and make reasonable 575 estimates for the number of cores available, we expect the final 576 cluster configuration to have between 400 and 800 cores. We will use 577 600 as an estimate. Note that it is possible to supplement the 578 processing power of the cluster by buying 1U boxes with processors but 579 no storage. Each of these boxes cost roughly 15\% of a storage node 580 and add an equal number of processor cores. Such an option can be 581 taken at any time, though it is not needed in our current development 582 plan. 581 stagger the purchase of the computers as planned (5 sets of 16 582 machines), and make reasonable estimates for the number of cores 583 available, we expect the final cluster configuration to have between 584 400 and 800 cores. We will use 600 as an estimate. Note that it is 585 possible to supplement the processing power of the cluster by buying 586 1U boxes with processors but minimal storage. Each of these boxes 587 cost roughly 15\% of a storage node and add an equal number of 588 processor cores. Such an option can be taken at any time to 589 supplement the raw processing power, though it is not needed in our 590 current development plan. 583 591 584 592 There are two potentially dominant analysis steps in the process: the … … 589 597 takes roughly 16 seconds for a Megacam chip (single core), equivalent 590 598 to roughly 38 seconds on a full GPC-1 chip. Most other steps of the 591 analysis scale are constant per image, and contribute only a few592 seconds relative to the 38 seconds. We use 50 seconds per chipper593 c ore to judge the total processing power for the portion which scales594 by the number of images. 599 analysis require a constant amount of time per image, and contribute 600 only a few seconds relative to the 38 seconds. We use 50 seconds per 601 chip per core to judge the total processing power for the portion 602 which scales by the number of images. 595 603 596 604 A useful statistic to judge the capability of the processing system is 597 the time required to re process all images at the end of the survey.605 the time required to re-process all images at the end of the survey. 598 606 Given the total number of images above (555,000), the per-image 599 607 analysis portion of the processing would require a total of $\sim 1.8 … … 617 625 predict the Pan-STARRS magnitudes of stars, then extrapolated the 618 626 source counts to our magnitudes limits. We find 50,000 objects per 619 square degree above our threshold in this region. If every image 620 required the non-linear fitting for this density of objects, and we 621 accept the 10ms time, this analysis would require a total of $\sim 2.0 622 \times 10^9$ CPU core-seconds, or about 39 days on the 600 cores. 627 square degree above our threshold in this region. If we make the 628 conservative assumption that every image required the non-linear 629 fitting for this density of objects, and that each object requies 630 10ms, this analysis would require a total of $\sim 2.0 \times 10^9$ 631 CPU core-seconds, or about 39 days on the 600 cores. 623 632 624 633 In conclusion, given the assumptions above, the processing power of … … 650 659 requirements of the IPP for object databasing. Some effort has been 651 660 needed to make DVO completely suitable for its role within the IPP. 652 Regardless of what object databasing system was chosen, a certain 653 level of effort would have been required. In this case, we were clear 654 just how much would be required, and it was not large. 661 Regardless of what object databasing system was chosen, however, a 662 certain level of effort would have been required. In the case of DVO, 663 we were clear what effort was required, and it was judged to be 664 reasonable. 655 665 656 666 Of that effort, only the ability to support older table formats was 657 required to maintain CFHT Elixir compatibility. In fact, this is a 658 feature which we would have added even if we did not want to maintain 659 compatibility with CFHT's DVO installation. We have found in our 660 experience with the Elixir system that having a rigidly defined schema 661 hindered the usability and extensibility of the DVO system. The fixed 662 tables made it difficult to add new elements to the database, and 663 required multiple versions to support previously defined tables. The 664 new design allows us to be more flexible about changes without fear 665 that this will break database instances which already exist. One of 666 the best ways we have found to test the DVO object databasing system 667 is to engage students in science projects using DVO. These projects 668 explore the user interface and highlight problems and areas for 669 possible improvement. Such projects would not be possible if the 670 users feared that their DVO instances would be unusable in the future 671 because of lack of backwards compatibility. 667 required to maintain compatibility with the existing CFHT DVO 668 databases. In fact, this is a feature which we would have added even 669 if we did not want to maintain compatibility with CFHT's DVO 670 installation. We have found in our experience with the Elixir system 671 that having a rigidly defined schema hindered the usability and 672 extensibility of the DVO system. The fixed tables made it difficult 673 to add new elements to the database, and required multiple versions to 674 support previously defined tables. The new design allows us to be 675 more flexible about changes without fear that this will break database 676 instances which already exist. One of the best ways we have found to 677 test the DVO object databasing system is to engage students in science 678 projects using DVO. These projects explore the user interface and 679 highlight problems and areas for possible improvement. Such projects 680 would not be possible if the users feared that their DVO instances 681 would be unusable in the future because of lack of backwards 682 compatibility. 672 683 673 684 The PanTasks system used the existing Opihi command-line interface … … 684 695 of a large set of real images obtained by the CFHT engineering staff 685 696 over several years. We also gain by discussions with our Elixir 686 colle gues about details of the analysis and possible sources of errors697 colleagues about details of the analysis and possible sources of errors 687 698 observed in the CFHT dataset. The only cost to the IPP is in 688 699 preventing excessive forking of the DVO databasing system, something
Note:
See TracChangeset
for help on using the changeset viewer.
