Index: /trunk/doc/design/Makefile =================================================================== --- /trunk/doc/design/Makefile (revision 8139) +++ /trunk/doc/design/Makefile (revision 8140) @@ -1,3 +1,3 @@ -# $Id: Makefile,v 1.16 2006-02-16 18:57:50 eugene Exp $ +# $Id: Makefile,v 1.17 2006-08-04 12:50:49 eugene Exp $ PDFLATEX = env TEXINPUTS=../../latex/inputs:$(TEXINPUTS):.: pdflatex @@ -6,9 +6,11 @@ help: @echo "USAGE: make (target)" - @echo " targets: srs ssdd scd all" + @echo " targets: srs ssdd scd all cdrresp pars" scd: ippSCD.pdf srs: ippSRS.pdf ssdd: ippSSDD.pdf +cdrresp: ippCDRresponse.pdf +pars: ippParameters.pdf scd-draft: ippSCDdraft.pdf Index: /trunk/doc/design/ippCDRresponse.tex =================================================================== --- /trunk/doc/design/ippCDRresponse.tex (revision 8140) +++ /trunk/doc/design/ippCDRresponse.tex (revision 8140) @@ -0,0 +1,749 @@ +\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{Response to the IPP CDR Committee Report} +\subtitle{} +\shorttitle{IPP CDD Report Response} +\author{Eugene A. Magnier} +\audience{PMO} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{DR} +\docnumber{PSDC-440-004} + +% allow paragraphs to be listed in TOC for now +\setcounter{tocdepth}{3} + +\begin{document} +\maketitle + +% -- Revision History -- +\RevisionsStart +% version Date Description +DR.01 & 2006.07.28 & First Draft \\ \hline +\RevisionsEnd + +\tableofcontents +\pagebreak +\pagenumbering{arabic} + +\section{Document Overview} + +The IPP team appreciates the care, detail, and effort which went into +the IPP CDR Committee Report. The guidance and suggestions provided +by this review are extremely valuable and will help us to refine the +IPP design and improve the overall reliability and usability. + +This document is meant to address concerns raised by the CDR report +and offer a fuller explanation of certain issues. Some of the +questions raised are addressed in separate documents (IPP-related +ICDs, Updates to the IPP SSDD, the IPP Commissioning Plan, and the IPP +Analysis Parameters Guide). In particular, the CDR Review Committee +listed the following Action Items and requested a report on items 3-10 +from the IPP team. + +\begin{enumerate} +\item Work with other Pan-STARRS project members to complete a system +level requirements specification. Not having such a document at this +late stage of PS1 development is a significant risk, and it is +incumbent upon all relevant parties to address this as soon as +possible. (Note: Although recommending system level actions is +strictly outside the scope of the charge to this committee, it is +obvious that additional system level and operational requirements need +to be specified/clarified in the immediate future.) + +\item Currently, no requirements on completeness and reliability have +been specified for the PS1 Design Reference Mission. The project +should develop internal goals for at least completeness to help IPP +development set limits on how hard to work on difficult objects. + +\item Identify more explicitly the operational requirements of the IPP +(what is the IPP required to do as part of the complete PS1 +system?). In particular, compare the functionality of certain IPP +components relative to the PSPS. Then ensure that any functional +redundancy is intentionally present to increase system robustness, and +does not represent a needless duplication of effort. + +\item Major version deliveries of the IPP should be defined in the +development schedule. Each major version should be associated with +specific tasks and added functionality. It is a good thing to have +major milestones not just to track progress, but also to help focus +the development work by the team. Making these major deliveries can +also give a sense of accomplishment to the team, helping with morale. + +\item All the external and internal interfaces to/within the IPP +should be specified in detail. For the external interfaces, this +includes documenting the requirements in Interface Requirements +Specifications (IRSs) or as part of the PS1 System/Subsystem +Specification (SSS). The interface implementation should be presented +in Interface Control Documents (ICDs). For future reference and +project reviews maintain a matrix showing the status of each, and +describe the ICD maintenance and update process. + +\item Present the plan for finalizing the analysis algorithms. The +plan should include a complete list of analysis algorithms and +functions, the status/completeness of each, what data is required to +complete those not yet fully implemented or designed, and (if known) +what choices are available to be traded. + +\item Establish more clearly the data validation process within the +PS1 operations plan and budget. + +\item Provide the IPP commissioning plan. + +\item Provide additional information as to how the proposed hardware +configuration allows IPP to meet its requirements. + +\item Explain more fully design philosophy and benefits of maintaining +CFHT compatibility. +\end{enumerate} + +\section{Action Item 3: Operational Requirements of the IPP} + +This question appears to be principally concerned with perceived +overlap between the IPP DVO system and the PSPS Object Data Manager +(ODM). Where there is superficial similarity between the systems, it +is our belief that the development of two complementary systems is not +only worthwhile, but extremely important for a variety of reasons. + +The IPP operational requirments include the analysis of images to +produce the P2, P4$\Delta$, and P4$\Sigma$ detections, the Static Sky +images, the Astrometric and Photometric Reference catalog, and the +astrometric and photometric calibration of the data products (images +and detections). These requirements are highly coupled: in order to +produce the Static Sky at the level required, the IPP must perform +astrometric and photometric calibrations of the data stream at a high +level of accuracy. In order to perform these calibrations, the IPP +must construct an improved astrometric and photometric reference +catalog to supplant the existing references, at least within the PS1 +pass-bands. + +The DVO system is designed to meet the twin requirements of performing +the astrometric and photometric calibrations and to perform the +construction of the astrometric and photometric reference catalog. +The IPP DVO system and the PSPS thus have overlapping capabilities in +terms of linking detections together into objects on the sky and in +terms of tracking the relationships between the detections, images, +and objects, along with the photometric and astrometric calibration +information. There are several motivations for defining a separate +entity within the IPP specifially for this task: + +\begin{itemize} +\item the PSPS system is foreseen as the definitive view on the object + database problem, while the IPP requires the ability to manipulate + and update the astrometric and photometric calibrations. +\item the PSPS system requires a high-level of sophistication in the + scientific queries which may be performed. While critical, this + sophistication will result in a much longer development timeline for + the PSPS. +\item By performing the analysis of the astrometric and photometric + calibration within the DVO system, and avoiding these requirements + within the PSPS, the PSPS design may be focused on the database + management issues of supplying the data to users. +\item The IPP requirements for object databasing are fairly limited in + terms of the ability to perform detection/object matching. This + trade-off allows the IPP DVO system to operate at a much higher + throughput than is possible for the PSPS, and perform more + reprocessing operations. +\end{itemize} + +It is useful to ask if the PSPS could have been ready within the +necessary timeframe, would the IPP use that instead of the DVO system +for support of photometric and astrometric calibrations. The answer +would be 'yes' only if the PSPS requirements were modified to allow it +to perform the same operations as required by the IPP DVO. Would it +be worthwhile to the project to merge the functionality of the two +subsystems after they have both matured? My opinion, and I believe, +that of the project management, is that this choice would result in +development of a system which would be substantially overengineered +for its role in either of the two subsystems, with concomitant +increase in the cost of development and support. By dividing the +tasks served by these two systems in a logical way, the project as a +whole is better served. + +\section{Action Item 4: Major version deliveries of the IPP} + +\subsection{IPP Release 1 (Aug 30, 2006) : Phase 1-3, Detrend Creation, Infrastructure, ISP Support} + +\begin{itemize} +\item {\bf Release Contents} Analysis programs for Phases 1-3 and the + Detrend Creation analysis, PanTasks (the controller/scheduler), + PanTasks scripts for Phase 1-3 and Detrend Creation, the Metadata + Database tools (ippTools) needed to track the process flow, + Nebulous, and DVO. +\item {\bf Release Capabilities} Detrend image creation (bias, dark, + flat, fringe), image detrending (excluding convolved guide kernels), + single-image positive object detection and classification, + astrometric calibration, object detection databasing, basic + zero-point analysis. Real-time ISP transparency measurements. +\end{itemize} + +\subsection{IPP Release 2 (Oct 30, 2006) : Phase 4, static sky pixel definitions} + +\begin{itemize} +\item {\bf Release Contents} Phase 4 analysis programs, Phase 4 + PanTasks scripts, Static Sky pixel layout, interface to Magic, + psphot and ppImage upgrades. +\item {\bf Release Capabilities} Image warping, Image differencing, + Image stacking, object modelling for difference images (loading PSF + from an external source, fitting positive and negative sources), + analysis of the OTA guide kernel, detrend images convolved with the + guiding kernel. +\end{itemize} + +\subsection{IPP Release 3 (Dec 30, 2006) : Static Sky, AP Catalog tools, output interfaces} + +\begin{itemize} +\item {\bf Release Contents} Updates to psphot; DVO crawler tools + (uniphot, relphot, relastro); external interfaces. +\item {\bf Release Capabilities} Static sky version of the photometry + analysis, with more-complete characterization of galaxy parameters + and the simultaneous multi-filter photometry; tools to perform the + astrometric and photometric calibration of the all-sky survey data; + the interfaces to provide data to the MOPS, PSPS, and other science + clients as defined. +\end{itemize} + +\subsection{IPP Release 4 (Mar 30, 2007) : Code Freeze for ORR and the Survey Start} + +\begin{itemize} +\item {\bf Release Contents} Updates to programs identified in the + commissioning. +\item {\bf Release Capabilities} Improvements determined on the basis + of the commissioning with GPC-1 (possibly new PSPhot PSF models, + temperature dependent selects of input detrend data as needed, etc). +\end{itemize} + +\section{Action Item 5: Internal \& External Interfaces} + +We have worked with the other PS1 teams to define complete ICDs +between all of the interacting subsystems. The IPP interfaces with 4 +existing subsystems, and will interact with several other science +clients. We have completed the definitions of the ICDs for the +Camera-IPP (PSDC-940-003), OTIS-IPP (PSDC-940-004), IPP-MOPS +(PSDC-940-005), and IPP-PSPS (PSDC-940-006). These are now part of +the PSDC document tree. + +We have clarified the internal interfaces of the IPP in the SSDD. We +have improved the discussion of the types of interfaces, created a +table listing the major interfaces, and identified the data format for +each of the interfaces. The detailed contents of those interfaces are +specified in the software design description of the relevant software +components. For example, the SDD for PSPhot defines the formats of +output object data tables, while the SDD for DVO/addstar references +this document. + +\section{Action Item 6 : Plan for Finalizing the Analysis Algorithms} + +\begin{table}[ht] +\begin{center} +\caption{IPP Analysis Algorithms\label{algorithms}} +\begin{tabular}{llll} +\hline +\hline +{\bf Analysis} & {\bf IPP program} & {\bf Status} & {\bf Data Needed} \\ +\hline +Bias Creation & \code{ppMerge} & completed & raw bias images (range of conditions) \\ +Dark Creation & \code{ppMerge} & completed & raw dark images (range of conditions) \\ +Flat Creation & \code{ppMerge} & completed & raw flat images (dome and sky flats) \\ +Fringe Creation & \code{ppMerge} & 90\% & raw fringe image (dome and night-sky images) \\ +\hline +Overscan Correction & \code{ppImage} & completed & raw bias images (range of conditions) \\ +OTA Convolution & \code{ppImage} & partial & raw science images \\ +Non-Linear Correction & \code{ppImage} & partial & correction functions (from camera group) \\ +Shutter Correction & \code{ppImage} & missing & correction functions (from camera group) \\ +Domeflats or skyflats? & \code{ppImage} & completed & photometry dither images \\ +Domefringe or skyfringes? & \code{ppImage} & completed & night-time images (range of sky conditions) \\ +\hline +PSF model & \code{psphot} & completed$^1$ & science images (range of seeing) \\ +Sky model grid size & \code{psphot} & completed & science images (range of seeing) \\ +Other PSPhot parameters & \code{psphot} & completed & science images (range of seeing) \\ +\hline +Astrometry search radius & \code{psastro} & completed & telescope pointing measurements \\ +Astrometry model order & \code{psastro} & completed & science images (range of focus) \\ +\hline +Interpolation kernel & \code{stac} & partial & science images \\ +Difference image kernel & \code{poisub} & partial & science images \\ +Static Sky cells & \code{stac} & partial & simulations \\ +Static Sky combination & \code{stac} & partial & science images \\ +\hline +\multicolumn{4}{l}{$^1$Only analytical models are implemented. If more +complex models are needed they will have to be implemented.} +\end{tabular} +\end{center} +\end{table} + +The CDR Review Committee requests a plan for finalizing the analysis +algorithms within the IPP. There are three stages to freezing the +analysis algorithims within the IPP: +\begin{itemize} +\item Algorithm conceptual design. +\item Development of the software used to implement the algorithm. +\item Definition of parameters which modify the details of an + algorithm. +\end{itemize}. + +The IPP SSDD defines the conceptual details of nearly all of the +analysis algorithms in Sections 5, 6, and 7. Of these descriptions, +only Sections 5.5.5, describing the stacking of the Static Sky, +and Section 7, defining the Static Sky photometry analysis, are +insufficient in detail for the analysis to be well understood. Also +missing from version 00 of the document is a discussion of the +creation of the astrometric and photometric reference catalogs. A new +version of the SSDD with these three sections updated \note{will be +posted by DATE}. + +Table~\ref{algorithms} lists the IPP algorithms, the relevant program +in the IPP tree, the development state of the program with respect to +the identified analysis, and the data needed to set the algorithm +parameters. In a separate document (REF), we present a detailed +discussion of the choices to be made and guidelines on making those +choices. + +There are a handful of decisions which need to be made which have an +important impact on the way the analysis is performed. Most of these +will have a major impact on the results, but have little or no impact +on the IPP {\em design}. The one exception is the question of exactly +how the object analysis is performed on the Static Sky, as there are +several possible choices here. The major analysis issues to address +are: + +\begin{itemize} +\item {\bf Skyflats or Domeflats?} We will need to choose between + skyflats and domeflats. This choice will require a comparison of the + photometric accuracy achieved by both methods. Also important in + making this decision are the questions of how efficiently raw flats + may be obtained using both methods, how stable the two versions are, + and what kind of operations load they represent. These are + questions beyond the scope of IPP and are largely the responsibility + of OTIS. It is also an open question at this time if the planned + calibration screen will be available at the start of the Survey. + This is an operational requirement/decision that does not affect the + IPP design, though feedback from the IPP will be critical in making + the decision. + +\item {\bf Skyfringe or Domefringe?} This decision is similar to the + question of domeflats vs skyflats. Here the operational concerns + for the domefringe images are more acute. An important trade which + must be made is how many raw fringe images are needed to generate + the fringe images to be used by IPP. It is also critical that the + number of input fringe images required by IPP for any Phase 2 + analysis be minimized since it is impossible to afford the I/O load + demanded by a large number of input fringe images. A related + question is that of how to subselect the night-time fringe images + for best effect, if sky fringe images are used. Based on experience + from CFHT/Megacam, it may be possible to use fringe images selected + on the basis of the time of night, but this must be tested for + Haleakala. It seems unlikely at this time that a spectral skyprobe + will be available for the start of PS1, so we cannot rely on such a + device to guide our choices. + +\item {\bf Static Sky Cell definition} What is the layout of the + Static Sky cells, and how does this depend on the different surveys? + In particular, what are the projection centers and the size of + individual files? This question can be answered before we begin + collecting data by testing several of the competing options. The + software is not tied to any specific choice of static sky cell + arrangement. The simulations required to perform this decision are + being deferred until the IPP Release 1 can be completed. + +\item {\bf How is the image difference PSF-matching kernel + represented?} The software currently implements the Alard-Lupton + kernel (multiple Gaussians) and the POIS kernel (array of delta + functions). The software is easily modified to accommodate other + equivalent kernels such as Airy or Bessel-function decompositions. + It will be necessary to perform tests with real GPC-1 images to + determine which of these kernels achieves the best results. + +\item {\bf What interpolation kernel is used for image warping?} The + choice of interpolation kernel has two important impacts on the + resulting data product. First, there is a trade-off between the + accuracy of the kernel and the processing time required for the + analysis. Second, there is the question of aliasing of + high-frequency features. The first trade-off can be easily + performed with speed tests of the algorithms on the test hardware. + The second part of this question requires examples of the range of + PSFs and noise properties in real PS1 images. + +\item {\bf How are the input images to the static sky combined?} Once + images are warped, there are several choices to be made in how the + data from multiple images are combined into a single static sky + image. How do we select the input images as their PSFs change? How + are images with different PSFs weighted? What smoothing kernel (if + any) is applied to the input images before combination? The 3$\pi$ + survey will necessarily require the combination of pixels from a + range of field angles into the static sky. These will have a range + of image qualities. We will need to set the cuts to trade-off + between degredation of the final image quality versus degredation of + the signal-to-noise in the final image. In general, our guidance + for the Static Sky is to maximize our ability to measure accurate + morphology on the images (to meet the Weak Lensing requirements) and + to provide the best reference for the image differencing algorithms. + Astrometric and photometric accuracy of the static sky measurements + is not as critical a driving requirement. + +\item {\bf How are the Static Sky images processed?}. This particular + issue is critical to finalizing several aspects of the IPP design. + First, is it possible to meet the requirements for the Static Sky + (weak lensing accuracy and difference image accuracy) with a single + combination? Is it necessary to degrade all images to a single, + common seeing (for a stable PSF across the image for difference + image)? Is it possible to measure the weak lensing parameters + sufficiently accuractly from the stacked image with knowledge of the + PSF? To what level of detail is the PSF model required? Is it + necessary to perform the weak lensing analysis (and other galaxy + shape parameter measurements) from the individual images rather than + from the Static Sky? + +\end{itemize} + +\section{Action Item 7: Data Validation Process} + +Throughout the IPP, the analysis steps have been defined with data +validation as an integral step. As the images are processed through +the detrend creation stage, the Phase 1-3, and Phase 4, the analysis +performed includes tests of the quality of the input images. These +measurements are included in the stream of metadata produced by the +IPP (e.g., Tables 23, 25, 37 IPP SSDD). In the discussion below, we +identify the data validation measurements performed for the different +steps. The functional flow of these different analysis steps can be +seen in IPP SSDD Figures 22-30. The use of the Q/A measurements is +not summarized very clearly within the text of version 00; \note{this +will be updated in the new SSDD release}. Within the IPP, the +analysis stages use these measurements to mark input images as +accepted or rejected. These assessments are passed back to the OTIS +subsystem, along with the image statistics measured for the input +images. OTIS has the option of setting more stringent filters on the +input images and re-observing images on the basis of the IPP feedback, +even if the IPP accepted images which OTIS re-observes. There is also +a system-wide plan in place to use feedback from the IPP and from OTIS +to guide the project's choices for survey strategies and science +goals. + +\subsection{Detrend Images} + +The input detrend images all have their pixel count levels measured +for each chip. Input images which have counts or fluxes outside of a +defined range will be flagged and excluded from any detrend analysis. +For example, the flat-field images should never use input images which +are saturated, nor should the dark image analysis use input images +with flux levels wildly outside of the nominal range. Both conditions +are evidence that the observing process was performed inappropriately. + +In addition to raw pixel values, the input detrend images are +contronted with the resulting master detrend images. In general, the +effect corrected by the master detrend image should adaquately correct +each of the input raw detrend images. The residual scatter of the +detrended raw images should be small. As part of the detrend creation +process, input images with excessive residual scatter are +automatically rejected from the input stack. These images are also +flagged in the database. + +The master detrend images are also evaluated by using the statistics +of the (surviving) input residual images. The global statistics of +the residuals from the input set of images can be used to judge if the +master detrend images has sufficient signal-to-noise or consistency to +meet the requirements of the detrending process. + +\subsection{Science Images : Single-Image phases} + +As the science images are processed, a number of measurements are +performed to tests the quality of these input images. I discuss here +the measurements which will be performed. + +\begin{itemize} +\item median sky background : this can be used to trigger on images + taken in very poor conditions (eg, too early, dome lights on, too + close to the moon). +\item sky background statistics after detrending : this can be used to + detect images taken under poor conditions (close to the moon, many + bright stars). In the early stages, until we have a model for the + static sky surface brightness in each of the PS filters, these + measurements will only provide a reference and cannot be used as a + rigid identification of good/bad images. +\item fringe residual statistics : after the fringe correction has + been performed, a measurement of the variations correlated the + fringe pattern can be used to measure the quality of the fringe + correction. Images which fall outside of the nominal range may be + indicative of sky emission line conditions not expected in the + initial master fringe sample, and may indicate a way towards further + improvements to the fringe correction process. +\item image quality statistics : the PSF as a function of field + position can be used to detect images taken in degraded focus or + seeing conditions. Images outside defined ranges can be excluded + from further analysis. +\item Astrometric calibration \& scatter : failure of the astrometric + analysis to converge, or elevated scatter of the astrometric + solution are both evidence of images taken in poor conditions. + These may include telescope mis-pointing, tracking failures, or + focus failures. +\item Photometric calibration \& scatter : Excessive variations of the + zero-point as a function of mosaic position, generally elevated + scatter, and substantial photometric offsets are all evidence of + problems with the observing conditions. These may be the presence + of clouds and/or haze, degredation of the optics, and/or extreme + image-quality problems. +\end{itemize} + + +\subsection{Science Images : Image Stacking and Image Difference Phases} + +During the image stacking (P4$\Sigma$) and image differencing +(P4$\Delta$) stages, data validation is based on measurements of +consistency of the input images with the output stack and on the +number of peaks detected in the difference image. Images which have +passed the single-image quality tests should in general succeed on the +image stacking and difference stages. However, large deviations can +be expected in the event of unusual features within the images. For +example, uncorrected background variations from image to image will +result in large deviations between the components of an image stack, +and will also result in large numbers of difference image detections. +Similarly, bright stars with larger than expected halos or saturation +regions will result in excess difference image detections. + +\section{Action Item 8 : IPP Commissioning Plan} + +The IPP commissioning plan is supplied in the accompanying document, +IPP Commissioning Plan. + +\section{Action Item 9 : Hardware Configuration and Requirements} + +The proposed IPP hardware configuration is driven by three major +requirements: storage, processing power, I/O bandwidth. We have +performed a detailed analysis of these issues, which we have presented +in Section 12 of the SSDD. We concluded in our analysis that the +storage component of the requirements completely dominated our +hardware design considerations: by specifying reasonable hardware to +meet the storage requirements, we were supplying more than enough +hardware to meet the processing and I/O requirements. + +That analysis is based largely on prototype tests of our processing +algorithms, and is somewhat limited by being focused on the steady +state operations. We present here new numbers for the processing +timeline based on the current baseline software on our existing +baseline cluster hardware. + +For PS1, there is a significant processing challenge in the first 6 - +9 months when only a fraction of the IPP storage hardware will be +available. This period is further complicated by the budgetary +constraints placed on the IPP to limit the hardware purchase to a bare +minimum. An important area for clarification by the project is the +processing requirement in the beginning of the project. If the IPP is +required to perform a complete Static Sky analysis on every image as +it becomes available, then the total hardware required in the first +6-9 months for processing must be increased. If it is only necessary +to stack sets of, for example, 4 images as they are available, then +the requirements are somewhat reduced. A trade-off must be made by +the project to choose between these options. + +The data storage requirements are determined from the design reference +mission. The current plan for the design reference mission has +increased somewhat the number of images obtained by the survey. The +current plan consists of an All-Sky Survey re-visiting each point in +the sky 60 times over the 3 years (12 visits per filter). There is +also a Medium Deep survey which is expected to generate roughly 75000 +images per year. Combining these two, we find that the total number +of raw image data is roughly 1.7PB (555,000 images). In addition, we +have a requirement for Static Sky storage (using 0.2 arcsec pixels) of +roughly 300TB, and miscellaneous additional storage of nearly 100 TB. +Our hardware purchase plan has a minimum total storage of 2.4PB, +giving us a margin of about 10\%. Our plan is to purchase the +hardware in 5 stages of 16 computers each, for a total system cost of +roughly \$1.25M (each purchase set is expected to cost roughly +\$250k). Making reasonable estimates of disk capacity growth, we +expect these 80 machines to meet the storage capacity requirements. +Currently, we are able to buy 16.5TB per machine. In the worst case +that the hard disks stopped increasing in storage capacity (or we were +required to buy all of the machines up front), we would require +roughly 145 machines, increasing the cost of the cluster to a total of +roughly \$2.2M. We judge this to be a very low risk. + +We have performed timing test of the current versions of the IPP tools +for the different stages of the analysis using MegaCam images. Four +factors are critical in understanding if the cluster will be capable +of performing the processing needed: 1) processing power of the +cluster, 2) speed of the dominant analysis steps, 3) total number of +object for which a full non-linear model fit is performed, 4) total +number of images for which the warping and image differencing is +performed. + +Current CPU development roadmaps project a slow rise in the clock +speeds of the CPU cores, but increasing numbers of cores per socket. +By the end of this year, quad-core processors are expected. If we +stagger the purchase of the computers as planned, and make reasonable +estimates for the number of cores available, we expect the final +cluster configuration to have between 400 and 800 cores. We will use +600 as an estimate. Note that it is possible to supplement the +processing power of the cluster by buying 1U boxes with processors but +no storage. Each of these boxes cost roughly 15\% of a storage node +and add an equal number of processor cores. Such an option can be +taken at any time, though it is not needed in our current development +plan. + +There are two potentially dominant analysis steps in the process: the +warping and PSF-matching for a single image (ie, the Phase 4 steps) +and the non-linear model fitting performed on the brighter sources. +The processing time for the warping and PSF-matching is essentially +constant for each image. On our demonstration hardware, this step +takes roughly 16 seconds for a Megacam chip (single core), equivalent +to roughly 38 seconds on a full GPC-1 chip. Most other steps of the +analysis scale are constant per image, and contribute only a few +seconds relative to the 38 seconds. We use 50 seconds per chip per +core to judge the total processing power for the portion which scales +by the number of images. + +A useful statistic to judge the capability of the processing system is +the time required to reprocess all images at the end of the survey. +Given the total number of images above (555,000), the per-image +analysis portion of the processing would require a total of $\sim 1.8 +\times 10^9$ CPU core-seconds, or about 35 days on the 600 cores. + +The non-linear fitting speed is a bit more uncertain: the speed will +depend on the number of parameters in the model and the number of +pixels needed for most objects. We have run a variety of example +Megacam images and measured the speed per object for a several +possible combinations of models and object footprints. We find that +the speed ranges from 2 - 10 milliseconds per object. This number can +be confronted with the total number of objects in images for which the +non-linear fitting is performed. This number depends on both our +choice of a threshold for the non-linear fitting and the actual sky +density of sources at that depth. Our current proposal is to perform +the non-linear fitting for all objects above a S/N limit of 20. The +total number of sources is dominated by the sources in the Galactic +Plane. As a rough estimate of the total number of sources above our +threshold, we have examined the 2MASS catalog in the Galactic Plane +anti-center region. We have used the 2MASS and USNO-B colors to +predict the Pan-STARRS magnitudes of stars, then extrapolated the +source counts to our magnitudes limits. We find 50,000 objects per +square degree above our threshold in this region. If every image +required the non-linear fitting for this density of objects, and we +accept the 10ms time, this analysis would require a total of $\sim 2.0 +\times 10^9$ CPU core-seconds, or about 39 days on the 600 cores. + +In conclusion, given the assumptions above, the processing power of +the proposed hardware will be sufficient to allow for a complete +re-processing of all survey data within less than three months after +the survey is completed. We recognize, however, that there are still +some large potential uncertainties in this analysis, particularly in +the number of sources for which the analysis is performed. To get a +better understanding of the sky density of sources, we are extending +our analysis of the 2MASS and USNO-B source densities across the full +sky. However, given the fact that the processing power of the cluster +can be extended significantly without a large fractional cost, and the +fact that the re-processing time above is well below our requirement, +we do not believe there is a large risk in the hardware capabilities +of the IPP design. + +\section{Action Item 10 : CFHT compatibility} + +The concern that the IPP is driven by CFHT compatibility issues is +understandable, but easily addressed. The IPP design is not locked +into any specific design choices of the CFHT Elixir system. The IPP +uses one major subsystem, DVO, which is also used by the Elixir +system. It also is using code shared by certain Elixir systems in the +IPP scheduler/controller system (PanTasks). + +The IPP is using the DVO system developed at CFHT for the object +databasing. This is a natural decision because the DVO system 1) +existed, 2) was extensively tested, and 3) met most of the basic +requirements of the IPP for object databasing. Some effort has been +needed to make DVO completely suitable for its role within the IPP. +Regardless of what object databasing system was chosen, a certain +level of effort would have been required. In this case, we were clear +just how much would be required, and it was not large. + +Of that effort, only the ability to support older table formats was +required to maintain CFHT Elixir compatibility. In fact, this is a +feature which we would have added even if we did not want to maintain +compatibility with CFHT's DVO installation. We have found in our +experience with the Elixir system that having a rigidly defined schema +hindered the usability and extensibility of the DVO system. The fixed +tables made it difficult to add new elements to the database, and +required multiple versions to support previously defined tables. The +new design allows us to be more flexible about changes without fear +that this will break database instances which already exist. One of +the best ways we have found to test the DVO object databasing system +is to engage students in science projects using DVO. These projects +explore the user interface and highlight problems and areas for +possible improvement. Such projects would not be possible if the +users feared that their DVO instances would be unusable in the future +because of lack of backwards compatibility. + +The PanTasks system used the existing Opihi command-line interface +system which is also used by elements of DVO among other programs. +The choice to use this existing framework made the implementation of +the PanTasks system quick and efficient. + +The IPP owes a substantial intellectual heritage to the CFHT Elixir +pipeline. What the IPP gains by maintaining personal relationships +with the CFHT Elixir team is access to an unparalleled set of test +data and ongoing experience with a live running system facing the same +concerns the IPP will soon face. For example, only by virtue of this +relationship are we able to test the IPP flat-field correction scheme +of a large set of real images obtained by the CFHT engineering staff +over several years. We also gain by discussions with our Elixir +collegues about details of the analysis and possible sources of errors +observed in the CFHT dataset. The only cost to the IPP is in +preventing excessive forking of the DVO databasing system, something +we would attempt to avoid in any case. + +\end{document} + +----------------------------------------------------- + +* things we need to do within IPP + + - clarify the rules of MAGIC + - do we run a version MAGIC on P2 images? + - do we use the imclean algorithm to remove detection lines? + - better list of all data products with details + - better list of all metadata database tables + - identify the Q/A elements + - explain authority chain: + - image headers have the primary authority + - non-image tables from the summit have the primary authority + - metadata database table elements derived from + + - completeness + - static sky images + weight maps specify if a region has been + observed at all + - DVO image query specifies which images (generally) overlapped an + area. + - queries via IPP mask tool specify if a location could have been + observed (PSF * mask). + - queries via IPP object tool could be used to force an object + measurement on a given location (PSF or aperture). + - IPP is willing to support up to 10 / min / machine (~200 - 800 + per minute) during the daytime. one request may specify many + objects per image (up to 100?) + + - clarify DVO / MySQL relationship. + - DVO needs to use the FITS tables internally for speed + - DVO will export the FITS tables to MySQL for external use + + - better definition / discussion of the Phase 4 and Static Sky + components. + + - commissioning plan + - deliverables and schedule + + +--- + +hardware test points: + +ppImage (no psphot) on Megacam : 40.650u 31.489s 2:46.41 43.3% 0+0k 0+0io 30pf+0w + + * using alala I/O with 50 MB/sec read (and write?) + * total I/O was 4*708 MB (read) + 2778MB (write) = 112 sec + + * processing time = 40 sec (actual) | 54 sec (clock) + * Npix = 353Mpix + * speed = 0.15 sec / Mpix + + ** for 1 OTA : 3.5 sec processing time + 2-3 sec I/O (200 MB/sec) + ** for 1 ISP : 0.7 sec processing time + I/O + +psphot : 45 sec + Index: /trunk/doc/design/ippParameters.tex =================================================================== --- /trunk/doc/design/ippParameters.tex (revision 8140) +++ /trunk/doc/design/ippParameters.tex (revision 8140) @@ -0,0 +1,298 @@ +\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{How to Define IPP Analysis Parameters} +\subtitle{A User's Guide} +\shorttitle{Defining IPP Parameters} +\author{Eugene A. Magnier} +\audience{IPP Users} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{XX} +\docnumber{PSDC-XXX-XXX} + +% allow paragraphs to be listed in TOC for now +\setcounter{tocdepth}{3} + +\begin{document} +\maketitle + +% -- Revision History -- +\RevisionsStart +% version Date Description +DR.01 & 2006.07.28 & First Draft \\ \hline +\RevisionsEnd + +\tableofcontents +\pagebreak + +% \listoffigures +% \pagebreak +\pagenumbering{arabic} + +\section{Overview} + +This Pan-STARRS IPP is a complex software system with many components. +It is very flexible, and may be configured to handle data from a wide +variety of sources. The analysis programs have user-defined +parameters which need to be set appropriately for a given set of data. +Each camera, and in some cases each chip, may require slightly +different settings. This document outlines the process by which these +parameters may be set and guidelines for which parts of the analysis +programs may require tweaks. + +This document does not cover the setup of the IPP infrastructure. The +way in which the analysis programs are interrelated and how to define +the tables used to track the data through the IPP is discussed in a +separated document (TBD). + +The IPP performs a wide range of tasks, and consists of several +top-level programs. Some of these programs (eg, ppImage) are used in +different contexts, and may require different parameter settings +depending on the context. The recipe system allows these contexts to +be well-defined; please consult the relevant software design +descriptions for information on managing different recipes. The +analysis program relevant to this document are listed in +Table~\ref{programs}. Notice that certain programs (i.e., +\code{psphot} and \code{psastro}) are available as stand-alone +programs and also a library calls within other programs. The recipes +defined for one of these contexts may also be used by the other. + +\begin{table}[ht] +\begin{center} +\caption{IPP Analysis Programs\label{programs}} +\begin{tabular}{lll} +\hline +\hline +{\bf IPP program} & {\bf Status} & {\bf Function} \\ +\hline +\code{ppMerge} & completed & Combine images for detrend creation (no warping) \\ +\code{ppImage} & completed & General single-image analysis program \\ +\code{ppStats} & completed & Report image statistics (from header and pixels) \\ +\code{ppNorm} & completed & Perform normalization on detrend images \\ +\code{psphot} & completed & Perform object detection and charaterization \\ +\code{psastro} & completed & Perform astrometric calibration \\ +\code{stac} & prototype & Warp and combine science images \\ +\code{poisub} & prototype & Perform image differencing \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsection{Detrend Creation} + +The Detrend Creation portion of the IPP depends heavily on +\code{ppMerge} to build stacks of individual images. In this section, +we discuss the issues which must be examined for each of the types of +detrend data to be created. + +\subsubsection{Bias Creation} + +The bias creation tool (\code{ppMerge}) combines raw bias images into +a master stack. The tool may use several possible statistics to +determine the value of an output pixel: mean and median, clipped +versions, and robust version. The value of the clipping parameters, +if used must be set. Decisions are primarily based on constructing +bias images from a set of raw biases and applying them back to the raw +bias imaes. For any camera, following decisions must be taken: + +\begin{itemize} +\item Is a bias image needed? It can be skipped if a dark image is + used and the bias is the same for all dark exposures. It may also + be skipped if the overscan subtraction is sufficient and + sufficiently stable. +\item Which combination statistic must be used? The available options + may be tried and the best one selected. +\item Is it necessary to select the bias image on the basis of other + camera parameters (detector temperature? readout mode?). Note that + these type of selections may be made by the detrend creation and + application tools, but they are not provided by default: they will + only be added to the tool if needed or if time permits before hand. +\item Is it necessary to reject raw bias images from the stack on the + basis of other camera parameters (dome open/closed? detector + temperature?). Note that these type of selections may be made by + the detrend creation tool, but they are not provided by default: + they will only be added to the tool if needed or if time permits + before hand. +\end{itemize} + +\subsection{Dark Creation} + +The dark creation tool (\code{ppMerge}) combines raw dark images into +a master stack. The tool may use several possible statistics to +determine the value of an output pixel: mean and median, clipped +versions, and robust version. The value of the clipping parameters, +if used must be set. Decisions are primarily based on constructing +dark images from a set of raw darks and applying them back to the raw +dark images. For any camera, following decisions must be taken: + +\begin{itemize} +\item Is a dark image needed? It can be skipped if there is + insufficient dark current. +\item Which combination statistic must be used? The available options + may be tried and the best one selected. +\item Is it necessary to select the dark image on the basis of other + camera parameters (detector temperature? readout mode?). Note that + these type of selections may be made by the detrend creation and + application tools, but they are not provided by default: they will + only be added to the tool if needed or if time permits before hand. +\item Is it necessary to reject raw dark images from the stack on the + basis of other camera parameters (dome open/closed? detector + temperature?). Note that these type of selections may be made by + the detrend creation tool, but they are not provided by default: + they will only be added to the tool if needed or if time permits + before hand. +\item What is the scaling of a dark image by dark exposure time? Is + it necessary to have well-matched darks, or can a set of exposure + times be chosen and interpolation used between these exposure times? +\end{itemize} + +\subsection{Flat Creation} + +The flat creation tool (\code{ppMerge}) combines raw flat images into +a master stack. The tool may use several possible statistics to +determine the value of an output pixel: mean and median, clipped +versions, and robust version. The value of the clipping parameters, +if used must be set. Decisions are primarily based on constructing +flat images from a set of raw flats and applying them back to the raw +flat images. For any camera, following decisions must be taken: + +\begin{itemize} +\item Are skyflats or domeflats better? Both must be generated and + photometric analysis of observations of stars used to select between + these options. Both the spatial consistency and the source + stability should be considered in making the decision. Thus the + residual images should be used to judge the flat-field quality as + well. +\item Which combination statistic must be used? The available options + may be tried and the best one selected. +\item For skyflats, what restrictions should be placed on the input + image? (eg, time-from-sunset, sky flux level, mean image counts). +\item Is it necessary to reject raw flat images from the stack on the + basis of other camera parameters (dome open/closed? detector + temperature?). Note that these type of selections may be made by + the detrend creation tool, but they are not provided by default: + they will only be added to the tool if needed or if time permits + before hand. +\end{itemize} + +\subsection{Fringe Creation} + +The fringe creation tool (\code{ppMerge}) combines raw fringe images into +a master stack. The tool may use several possible statistics to +determine the value of an output pixel: mean and median, clipped +versions, and robust version. The value of the clipping parameters, +if used must be set. Decisions are primarily based on constructing +fringe images from a set of raw fringes and applying them back to the raw +fringe images. For any camera, following decisions must be taken: + +\begin{itemize} +\item Are skyfringes or domefringes better? Both must be generated and + photometric analysis of observations of stars used to select between + these options. Both the spatial consistency and the source + stability should be considered in making the decision. Thus the + residual images should be used to judge the fringe-field quality as + well. +\item if skyfringes are used, how many reference images are needed? +\item Which combination statistic must be used? The available options + may be tried and the best one selected. +\item For skyfringes, what restrictions should be placed on the input + image? (eg, time-of-night, sky flux level, mean image counts). +\item Is it necessary to reject raw fringe images from the stack on + the basis of other camera parameters (dome open/closed? detector + temperature?). Time relative to sunset is a default option, but + other parameters must be added on an as-needed basis. +\end{itemize} + +\section{Detrend Analysis Recipes} + +A critical operation performed by the IPP is the application of the +master detrend images to the individual science images. This is +performed by \code{ppImage} and is the core of the Phase 2 analysis +stage. Many of the issues important for the application of the +detrend images have already been addressed in the detrend creation +section above. In this section, we discuss the steps of the analysis +which may require inputs beyond those already discussed. + +\subsection{Overscan correction} + +For science images, we can correct the basic electronic offset (bias) +by measuring the value of the overscan region. The overscan may be +used to determine a single value, a row-by-row vector, a spline fit, or a +polynomial fit. Clipping ranges must also be chosen. These decisions +are made by performing the overscan correction on bias images with +different options and determining which results in a better +correction. + +\subsection{OTA Convolution of Detrend Images} + +For the OTA guided images, it is necessary to convolve the OTA kernel +with some of the detrend images. It is necessary to decide which of +the images should be convolved. The likely candidates are the +flat-field and the fringe images. The bias and dark images do not +need to be convolved. The flat images do. The fringe images are less +clear. If the fringe patterns consist of lower spatial frequencies +than the OTA kernel, then they may not need to be convolved. The +convolution method (direct vs fft) must be tested as well; the choice +will depend on which is faster, which in-turn will depend on the +typical size of the convolution kerne. Night-time science images are +needed to make this judgement. + +\subsection{Non-Linear Correction} + +The non-linearity of the devices must be measured. This is the +responsibility of the Camera team. The resulting function or lookup +table must supplied to the IPP and will be applied to the data. The +choice to make here is how the non-linear correction is represented: +lookup table or interpolation function? + +\section{Single-Image Photometry} + +\subsection{PSF model function} + +The PSF model function must be selected from the available models, or +a new model which better represents the available data supplied. The +current design of PSPhot supplies a range of analytical models. The +choice of which model to use, and the depends on the optical +parameters of the camera + +\subsection{Sky Model Grid Scale} + +PSPhot constructs a global model for the large-scale variations in the +background to subtract before performing source detection and +characterization. This model is built by measuring the median for +image pixels within large superpixels. The size of the superpixels is +configurable and must be chosen based on the maximum allowed size of +the PSF and on the size of the smallest features which should be +subtracted. + +\subsection{Other PSPhot parameters} + +A variety of minor PSPhot parameters must be tuned to meet both the +speed and the photometric accuracy requirements. For example, the S/N +per pixel of the pixels included in a source fit, or the maximum +number of stars included in the measurement of the PSF model. A +number of images from the camera representing the expected or allowed +range of seeing are required to choose these parameters. + +\section{Astrometry} + +\subsection{Initial Search Radius} + +The astrometry analysis requires a search radius within which it will +search for matches between the reference stars and the observed +stars. The size of this search radius will depend on the +repeatability of the telescope pointing. The search radius should be +as small as possible yet still guarantee that the match will be made +for all images. + +\subsection{Order of Fitting Functions} + +The order of the polynomials used to represent the astrometry model +needs to be set based on the observed data. The smallest order which +achieves the astrometric accuracy requirements should be used. This +value will depend on the real properties of the distortion in the +images. + +\end{document} Index: /trunk/doc/design/ippSSDD.tex =================================================================== --- /trunk/doc/design/ippSSDD.tex (revision 8139) +++ /trunk/doc/design/ippSSDD.tex (revision 8140) @@ -649,5 +649,5 @@ \begin{table}[ht] \begin{center} -\caption{Nebulos Database Tables\label{tab:ImageServerTables}} +\caption{Nebulous Database Tables\label{tab:ImageServerTables}} \begin{tabular}{ll} \hline @@ -4747,6 +4747,8 @@ FITS Images will be used to transport images between the components of -the IPP. Non-standard FITS images representing triangular collections -of pixels may be used to store the static sky images. +the IPP. The Static Sky FITS Images will use standard rectangular +FITS images and mask values to identify pixels which are included in a +specific image. Astrometric transformations will be stored in the +image headers. FITS Tables will be used to store and transport tabular data, @@ -4755,20 +4757,29 @@ different table interactions. -XML files will be used to store and transport data which is not -well-suited to the rectangular form of FITS Tables. Hierarchical data -concepts and variable-length structures fall in this class. Examples -include mosaic astrometry description information and configuration -information. - SQL queries and C wrappers of SQL queries will be used as the direct interface to the databases. -Within IPP and Pan-STARRS in general, process-to-process communication -will be defined through auto-coded APIs which support a limited and -validated communication protocol. The APIs will be coded based on a -table which defines the allowed command set and the grammar to be -used. This mechanism will allow a single code block to define -inter-process communication methods for many Pan-STARRS subsystems, -including, within the IPP, the Scheduler-Controller communications. +\begin{tabular}{lll} +{\bf Interface} & {\bf Data} & {\bf Format} \\ +ippTools : PanTasks & image state info & psMetadataConfig \\ +PanTasks : ippTools & image state updates & ippTools CLI, psMetadataConfig \\ +PanTasks : ppImage & image operations & ppImage CLI, recipe files \\ +ppImage : psphot & detrend images & pmReadout in pmFPA structure \\ +ppImage : psastro & detected objects & pmSources in pmFPA structure \\ +DVO/getstar : psastro & ref object coords & FITS Tables \\ +psphot : DVO/addstar & detection tables & FITS Tables \\ +psastro : DVO/addstar & astrometry & FITS Header \\ +ippTools : ppImage & detrend images & FITS Images, psMetadataConfig \\ +PanTasks : ppMerge & image operations & ppMerge CLI, recipe files \\ +PanTasks : ppStats & image info & ppStats CLI, recipe files \\ +PanTasks : ppNorm & image results & ppNorm CLI, recipe files \\ +DVO : relphot & DVO database & FITS Tables \\ +DVO : relastro & DVO database & FITS Tables \\ +psastro : stac & astrometry & FITS Header \\ +psphot : poisub & psf model & psMetadataConfig \\ +psphot : poisub & detected stars & FITS Table \\ +Static Sky : stac & astrometry & FITS Header \\ + +\end{tabular} \subsection{External Interfaces} @@ -4776,29 +4787,39 @@ This subsection describes the interfaces between the IPP and other Pan-STARRS systems and the external clients. The interfaces are -illustrated in Figure~\ref{fig:overview}. +illustrated in Figure~\ref{fig:overview}. The details of the IPP +interfaces with other external subsystems are specified in the ICDs +between those subsystems. + +\subsubsection{Camera} + +Images are pulled from the Summit Pixel Server, part of the Camera +team's purview, via the Data Store mechanism. The locations of the +files are specified by the Data Store (see the Data Store SDD, +PSDC-940-010). IPP grabs these via {\tt http}. This interface is +described in detail by the Camera-IPP ICD (PSDC-940-003). \subsubsection{OTIS} -The IPP Scheduler may query OTIS for a list of new images and -metadata. The locations of those images in the Summit Pixel Server is -sent back as a table, and all metadata may be sent to the IPP as a -collection of FITS Tables. The IPP also may send quality assessment -information for each FPA and major frame by writing out FITS tables -and notifying OTIS of the presence of the new tables. - -\subsubsection{Camera} - -Images are pulled from the Summit Pixel Server, part of the Camera -team's purview. The locations of the files are sent by OTIS. IPP may -grab these via {\tt http} commands or via {\tt NFS} or another network -file exchange protocol. The IPP notifies OTIS (and Camera) when each -image has been received. +The IPP Scheduler receives metadata from OTIS as a collection of FITS +Tables. The IPP sends quality assessment information for each FPA and +major frame as FITS tables. These metadata tables are exchanged in +both directions using the Data Store mechanism (PSDC-940-010). The +interface is described in detail by the OTIS-IPP ICD (PSDC-940-004). + +\subsubsection{MOPS} + +Data will be sent to MOPS from the IPP as part of the Phase 4 +analysis. The data to be transfered include: +\begin{itemize} +\item Image Metadata tables - to be transferred as FITS tables +\item Transient Detections - to be transferred as FITS tables +\end{itemize} +This interface uses the Data Store mechanism (PSDC-940-010), and is +described in detail by the IPP-MOPS ICD (PSDC-940-005). \subsubsection{PSPS} -Data will be sent to PSPS from the IPP as part of a daily or weekly -analysis process on the Static Sky. The data will be pushed from the -IPP to PSPS when they are available. The data to be transfered -include: +Data is sent to PSPS from the IPP on a regular basis. The data to be +transfered include: \begin{itemize} \item Static Sky images - to be transferred as FITS images or @@ -4808,14 +4829,6 @@ \item Detections \& Object associations - to be transferred as FITS tables. \end{itemize} - -\subsubsection{MOPS} - -Data will be sent to MOPS from the IPP as part of the Phase 4 -analysis. The data will be pushed from the IPP to MOPS when they are -available. The data to be transfered include: -\begin{itemize} -\item Image Metadata tables - to be transferred as FITS tables -\item Orphaned Detections - to be transferred as FITS tables -\end{itemize} +This interface uses the Data Store mechanism (PSDC-940-010), and is +described in detail by the IPP-PSPS ICD (PSDC-940-006). \subsubsection{Other Preferred Client Science Pipelines}