Index: /trunk/doc/design/ippCDRresponse.tex =================================================================== --- /trunk/doc/design/ippCDRresponse.tex (revision 8702) +++ /trunk/doc/design/ippCDRresponse.tex (revision 8703) @@ -20,8 +20,8 @@ % -- Revision History -- -\RevisionsStart +%\RevisionsStart % version Date Description -DR.01 & 2006.07.28 & First Draft \\ \hline -\RevisionsEnd +%00 & 2006.08.13 & Initial Release \\ \hline +%\RevisionsEnd \tableofcontents @@ -104,9 +104,9 @@ This question appears to be principally concerned with perceived overlap between the IPP DVO system and the PSPS Object Data Manager -(ODM). Where there is superficial similarity between the systems, it +(ODM). While there is superficial similarity between the systems, it is our belief that the development of two complementary systems is not -only worthwhile, but extremely important for a variety of reasons. - -The IPP operational requirments include the analysis of images to +only worthwhile, but extremely important for a variety of reasons. + +The IPP operational requirements include the analysis of images to produce the P2, P4$\Delta$, and P4$\Sigma$ detections, the Static Sky images, the Astrometric and Photometric Reference catalog, and the @@ -128,13 +128,13 @@ and objects, along with the photometric and astrometric calibration information. There are several motivations for defining a separate -entity within the IPP specifially for this task: +entity within the IPP specifically for this task: \begin{itemize} -\item the PSPS system is foreseen as the definitive view on the object +\item The PSPS system is foreseen as the definitive view on the object database problem, while the IPP requires the ability to manipulate and update the astrometric and photometric calibrations. -\item the PSPS system requires a high-level of sophistication in the +\item The PSPS system requires a high-level of sophistication in the scientific queries which may be performed. While critical, this - sophistication will result in a much longer development timeline for + sophistication will result in a much longer development time line for the PSPS. \item By performing the analysis of the astrometric and photometric @@ -146,9 +146,9 @@ trade-off allows the IPP DVO system to operate at a much higher throughput than is possible for the PSPS, and perform more - reprocessing operations. + re-processing operations. \end{itemize} It is useful to ask if the PSPS could have been ready within the -necessary timeframe, would the IPP use that instead of the DVO system +necessary time frame, would the IPP use that instead of the DVO system for support of photometric and astrometric calibrations. The answer would be 'yes' only if the PSPS requirements were modified to allow it @@ -157,5 +157,5 @@ subsystems after they have both matured? My opinion, and I believe, that of the project management, is that this choice would result in -development of a system which would be substantially overengineered +development of a system which would be substantially over-engineered for its role in either of the two subsystems, with concomitant increase in the cost of development and support. By dividing the @@ -187,5 +187,5 @@ psphot and ppImage upgrades. \item {\bf Release Capabilities} Image warping, Image differencing, - Image stacking, object modelling for difference images (loading PSF + Image stacking, object modeling for difference images (loading PSF from an external source, fitting positive and negative sources), analysis of the OTA guide kernel, detrend images convolved with the @@ -220,7 +220,7 @@ We have worked with the other PS1 teams to define complete ICDs between all of the interacting subsystems. The IPP interfaces with 4 -existing subsystems, and will interact with several other science -clients. We have completed the definitions of the ICDs for the -Camera-IPP (PSDC-940-003), OTIS-IPP (PSDC-940-004), IPP-MOPS +existing subsystems, and will interact with other as-yet-undefined +science clients. We have completed the definitions of the ICDs for +the Camera-IPP (PSDC-940-003), OTIS-IPP (PSDC-940-004), IPP-MOPS (PSDC-940-005), and IPP-PSPS (PSDC-940-006). These are now part of the PSDC document tree. @@ -277,5 +277,5 @@ The CDR Review Committee requests a plan for finalizing the analysis algorithms within the IPP. There are three stages to freezing the -analysis algorithims within the IPP: +analysis algorithms within the IPP: \begin{itemize} \item Algorithm conceptual design. @@ -287,18 +287,18 @@ The IPP SSDD defines the conceptual details of nearly all of the analysis algorithms in Sections 5, 6, and 7. Of these descriptions, -only Sections 5.5.5, describing the stacking of the Static Sky, -and Section 7, defining the Static Sky photometry analysis, are -insufficient in detail for the analysis to be well understood. Also -missing from version 00 of the document is a discussion of the -creation of the astrometric and photometric reference catalogs. A new -version of the SSDD with these three sections updated \note{will be -posted by DATE}. +only Sections 5.5.5, describing the stacking of the Static Sky, and +Section 7.2, defining the Static Sky photometry analysis, are +insufficient in detail for the analysis to be well understood. The +discussion of the analysis used to generate the astrometric and +photometric reference catalog are included in Section 4 on the design +of DVO, and are somewhat sparse. A new version of the SSDD with these +three sections updated will be posted by Aug 30. Table~\ref{algorithms} lists the IPP algorithms, the relevant program in the IPP tree, the development state of the program with respect to the identified analysis, and the data needed to set the algorithm -parameters. In a separate document (REF), we present a detailed -discussion of the choices to be made and guidelines on making those -choices. +parameters. In a separate document (`How to Define IPP Analysis +Parameters'), we present a detailed discussion of the choices to be +made and guidelines on making those choices. There are a handful of decisions which need to be made which have an @@ -332,11 +332,15 @@ analysis be minimized since it is impossible to afford the I/O load demanded by a large number of input fringe images. A related - question is that of how to subselect the night-time fringe images + question is that of how to sub-select the night-time fringe images for best effect, if sky fringe images are used. Based on experience from CFHT/Megacam, it may be possible to use fringe images selected on the basis of the time of night, but this must be tested for - Haleakala. It seems unlikely at this time that a spectral skyprobe - will be available for the start of PS1, so we cannot rely on such a - device to guide our choices. + Haleakala. An alternative strategy, in which master fringe images + are combined based on their consistency with the science image, has + also been implemented for the IPP. The decision between these + options will be guided by further testing of Megacam images and real + Haleakala data in a range of conditions. It seems unlikely at this + time that a spectral skyprobe will be available for the start of + PS1, so we cannot rely on such a device to guide our choices. \item {\bf Static Sky Cell definition} What is the layout of the @@ -376,5 +380,5 @@ range of field angles into the static sky. These will have a range of image qualities. We will need to set the cuts to trade-off - between degredation of the final image quality versus degredation of + between degradation of the final image quality versus degradation of the signal-to-noise in the final image. In general, our guidance for the Static Sky is to maximize our ability to measure accurate @@ -391,5 +395,5 @@ common seeing (for a stable PSF across the image for difference image)? Is it possible to measure the weak lensing parameters - sufficiently accuractly from the stacked image with knowledge of the + sufficiently accurately from the stacked image with knowledge of the PSF? To what level of detail is the PSF model required? Is it necessary to perform the weak lensing analysis (and other galaxy @@ -410,29 +414,31 @@ steps. The functional flow of these different analysis steps can be seen in IPP SSDD Figures 22-30. The use of the Q/A measurements is -not summarized very clearly within the text of version 00; \note{this -will be updated in the new SSDD release}. Within the IPP, the -analysis stages use these measurements to mark input images as -accepted or rejected. These assessments are passed back to the OTIS -subsystem, along with the image statistics measured for the input -images. OTIS has the option of setting more stringent filters on the -input images and re-observing images on the basis of the IPP feedback, -even if the IPP accepted images which OTIS re-observes. There is also -a system-wide plan in place to use feedback from the IPP and from OTIS -to guide the project's choices for survey strategies and science -goals. +not summarized very clearly within the text of version 00; this will +be updated in the new SSDD release. The types of Q/A measurements are +also discussed below. + +Within the IPP, the analysis stages use these measurements to mark +input images as accepted or rejected. These assessments are passed +back to the OTIS subsystem, along with the image statistics measured +for the input images. OTIS has the option of setting more stringent +filters on the input images and re-observing images on the basis of +the IPP feedback, even if the IPP accepted images which OTIS +re-observes. There is also a system-wide plan in place to use +feedback from the IPP and from OTIS to guide the project's choices for +survey strategies and science goals. \subsection{Detrend Images} -The input detrend images all have their pixel count levels measured -for each chip. Input images which have counts or fluxes outside of a +The input detrend images have their pixel count levels measured for +each chip. Input images which have counts or fluxes outside of a defined range will be flagged and excluded from any detrend analysis. For example, the flat-field images should never use input images which are saturated, nor should the dark image analysis use input images with flux levels wildly outside of the nominal range. Both conditions -are evidence that the observing process was performed inappropriately. +are evidence that the data were incorrectly obtained. In addition to raw pixel values, the input detrend images are -contronted with the resulting master detrend images. In general, the -effect corrected by the master detrend image should adaquately correct +confronted with the resulting master detrend images. In general, the +effect corrected by the master detrend image should adequately correct each of the input raw detrend images. The residual scatter of the detrended raw images should be small. As part of the detrend creation @@ -483,5 +489,5 @@ scatter, and substantial photometric offsets are all evidence of problems with the observing conditions. These may be the presence - of clouds and/or haze, degredation of the optics, and/or extreme + of clouds and/or haze, degradation of the optics, and/or extreme image-quality problems. \end{itemize} @@ -500,5 +506,5 @@ result in large deviations between the components of an image stack, and will also result in large numbers of difference image detections. -Similarly, bright stars with larger than expected halos or saturation +Similarly, bright stars with larger-than-expected halos or saturation regions will result in excess difference image detections. @@ -519,22 +525,22 @@ hardware to meet the processing and I/O requirements. -That analysis is based largely on prototype tests of our processing -algorithms, and is somewhat limited by being focused on the steady -state operations. We present here new numbers for the processing -timeline based on the current baseline software on our existing -baseline cluster hardware. - -For PS1, there is a significant processing challenge in the first 6 - -9 months when only a fraction of the IPP storage hardware will be -available. This period is further complicated by the budgetary -constraints placed on the IPP to limit the hardware purchase to a bare -minimum. An important area for clarification by the project is the -processing requirement in the beginning of the project. If the IPP is -required to perform a complete Static Sky analysis on every image as -it becomes available, then the total hardware required in the first -6-9 months for processing must be increased. If it is only necessary -to stack sets of, for example, 4 images as they are available, then -the requirements are somewhat reduced. A trade-off must be made by -the project to choose between these options. +That analysis was based largely on prototype tests of our processing +algorithms and has been made out-of-date by changes to the survey +plans. We present here new numbers for the processing time line based +on the current baseline software on our existing baseline cluster +hardware. + +% For PS1, there is a significant processing challenge in the first 6 - +% 9 months when only a fraction of the IPP storage hardware will be +% available. This period is further complicated by the budgetary +% constraints placed on the IPP to limit the hardware purchase to a bare +% minimum. An important area for clarification by the project is the +% processing requirement in the beginning of the project. If the IPP is +% required to perform a complete Static Sky analysis on every image as +% it becomes available, then the total hardware required in the first +% 6-9 months for processing must be increased. If it is only necessary +% to stack sets of, for example, 4 images as they are available, then +% the requirements are somewhat reduced. A trade-off must be made by +% the project to choose between these options. The data storage requirements are determined from the design reference @@ -546,7 +552,7 @@ images per year. Combining these two, we find that the total number of raw image data is roughly 1.7PB (555,000 images). In addition, we -have a requirement for Static Sky storage (using 0.2 arcsec pixels) of -roughly 300TB, and miscellaneous additional storage of nearly 100 TB. -Our hardware purchase plan has a minimum total storage of 2.4PB, +have a requirement for Static Sky storage (assuming 0.2 arcsec pixels) +of roughly 300TB, and miscellaneous additional storage of nearly 100 +TB. Our hardware purchase plan has a minimum total storage of 2.4PB, giving us a margin of about 10\%. Our plan is to purchase the hardware in 5 stages of 16 computers each, for a total system cost of @@ -558,5 +564,6 @@ required to buy all of the machines up front), we would require roughly 145 machines, increasing the cost of the cluster to a total of -roughly \$2.2M. We judge this to be a very low risk. +roughly \$2.2M. We judge this to be a very low risk as hard disk +capacities continue to grow. We have performed timing test of the current versions of the IPP tools @@ -572,13 +579,14 @@ speeds of the CPU cores, but increasing numbers of cores per socket. By the end of this year, quad-core processors are expected. If we -stagger the purchase of the computers as planned, and make reasonable -estimates for the number of cores available, we expect the final -cluster configuration to have between 400 and 800 cores. We will use -600 as an estimate. Note that it is possible to supplement the -processing power of the cluster by buying 1U boxes with processors but -no storage. Each of these boxes cost roughly 15\% of a storage node -and add an equal number of processor cores. Such an option can be -taken at any time, though it is not needed in our current development -plan. +stagger the purchase of the computers as planned (5 sets of 16 +machines), and make reasonable estimates for the number of cores +available, we expect the final cluster configuration to have between +400 and 800 cores. We will use 600 as an estimate. Note that it is +possible to supplement the processing power of the cluster by buying +1U boxes with processors but minimal storage. Each of these boxes +cost roughly 15\% of a storage node and add an equal number of +processor cores. Such an option can be taken at any time to +supplement the raw processing power, though it is not needed in our +current development plan. There are two potentially dominant analysis steps in the process: the @@ -589,11 +597,11 @@ takes roughly 16 seconds for a Megacam chip (single core), equivalent to roughly 38 seconds on a full GPC-1 chip. Most other steps of the -analysis scale are constant per image, and contribute only a few -seconds relative to the 38 seconds. We use 50 seconds per chip per -core to judge the total processing power for the portion which scales -by the number of images. +analysis require a constant amount of time per image, and contribute +only a few seconds relative to the 38 seconds. We use 50 seconds per +chip per core to judge the total processing power for the portion +which scales by the number of images. A useful statistic to judge the capability of the processing system is -the time required to reprocess all images at the end of the survey. +the time required to re-process all images at the end of the survey. Given the total number of images above (555,000), the per-image analysis portion of the processing would require a total of $\sim 1.8 @@ -617,8 +625,9 @@ predict the Pan-STARRS magnitudes of stars, then extrapolated the source counts to our magnitudes limits. We find 50,000 objects per -square degree above our threshold in this region. If every image -required the non-linear fitting for this density of objects, and we -accept the 10ms time, this analysis would require a total of $\sim 2.0 -\times 10^9$ CPU core-seconds, or about 39 days on the 600 cores. +square degree above our threshold in this region. If we make the +conservative assumption that every image required the non-linear +fitting for this density of objects, and that each object requies +10ms, this analysis would require a total of $\sim 2.0 \times 10^9$ +CPU core-seconds, or about 39 days on the 600 cores. In conclusion, given the assumptions above, the processing power of @@ -650,24 +659,26 @@ requirements of the IPP for object databasing. Some effort has been needed to make DVO completely suitable for its role within the IPP. -Regardless of what object databasing system was chosen, a certain -level of effort would have been required. In this case, we were clear -just how much would be required, and it was not large. +Regardless of what object databasing system was chosen, however, a +certain level of effort would have been required. In the case of DVO, +we were clear what effort was required, and it was judged to be +reasonable. Of that effort, only the ability to support older table formats was -required to maintain CFHT Elixir compatibility. In fact, this is a -feature which we would have added even if we did not want to maintain -compatibility with CFHT's DVO installation. We have found in our -experience with the Elixir system that having a rigidly defined schema -hindered the usability and extensibility of the DVO system. The fixed -tables made it difficult to add new elements to the database, and -required multiple versions to support previously defined tables. The -new design allows us to be more flexible about changes without fear -that this will break database instances which already exist. One of -the best ways we have found to test the DVO object databasing system -is to engage students in science projects using DVO. These projects -explore the user interface and highlight problems and areas for -possible improvement. Such projects would not be possible if the -users feared that their DVO instances would be unusable in the future -because of lack of backwards compatibility. +required to maintain compatibility with the existing CFHT DVO +databases. In fact, this is a feature which we would have added even +if we did not want to maintain compatibility with CFHT's DVO +installation. We have found in our experience with the Elixir system +that having a rigidly defined schema hindered the usability and +extensibility of the DVO system. The fixed tables made it difficult +to add new elements to the database, and required multiple versions to +support previously defined tables. The new design allows us to be +more flexible about changes without fear that this will break database +instances which already exist. One of the best ways we have found to +test the DVO object databasing system is to engage students in science +projects using DVO. These projects explore the user interface and +highlight problems and areas for possible improvement. Such projects +would not be possible if the users feared that their DVO instances +would be unusable in the future because of lack of backwards +compatibility. The PanTasks system used the existing Opihi command-line interface @@ -684,5 +695,5 @@ of a large set of real images obtained by the CFHT engineering staff over several years. We also gain by discussions with our Elixir -collegues about details of the analysis and possible sources of errors +colleagues about details of the analysis and possible sources of errors observed in the CFHT dataset. The only cost to the IPP is in preventing excessive forking of the DVO databasing system, something Index: /trunk/doc/design/ippSSDD.tex =================================================================== --- /trunk/doc/design/ippSSDD.tex (revision 8702) +++ /trunk/doc/design/ippSSDD.tex (revision 8703) @@ -613,12 +613,13 @@ \paragraph{House keeping} -\subparagraph{Lock sweeping} In the event that a Storage Object operation fails to complete successfully -stale locks will have to be identified and removed from the IPP Pixel -Data Server Database. This should be done periodically by comparing -the entries in the Lock table to the list of active nodes maintained -by the IPP Controller. It should also happen as soon as possible -after a node goes offline (triggered by the IPP Controller marking a -node as offline?). A sweep must be /completed/ before an offline node -can be marked on-line. +\subparagraph{Lock sweeping} +In the event that a Storage Object operation fails to complete +successfully stale locks will have to be identified and removed from +the IPP Pixel Data Server Database. This should be done periodically +by comparing the entries in the Lock table to the list of active nodes +maintained by the IPP Controller. It should also happen as soon as +possible after a node goes offline (triggered by the IPP Controller +marking a node as offline?). A sweep must be /completed/ before an +offline node can be marked on-line. Once a node is determined to be offline all entries in the Lock table @@ -628,8 +629,9 @@ table. -\subparagraph{Consistency sweeping} Periodically the IPP Pixel Data Server meta-data and Storage Object will need -to be checked for sanity. This would be similar to running fsck on a -modern filesystem. Consistency sweeping should include Lock sweeping -and should be considered a super-set. +\subparagraph{Consistency sweeping} +Periodically the IPP Pixel Data Server meta-data and Storage Object +will need to be checked for sanity. This would be similar to running +fsck on a modern filesystem. Consistency sweeping should include Lock +sweeping and should be considered a super-set. \subsubsection{Nebulous Database} @@ -1491,4 +1493,7 @@ photometrically corrected flats (-grid option). +\tbd{fill out this discussion in the analysis section on the +astrometric and photometric reference catalog}. + \subsubsection{Uniphot : Zero Point Analysis} @@ -1496,5 +1501,5 @@ points for images and the spatial overlap information to determine a best set of image zero points which have a specific time scale for the -atmospheric stability. This analysis would be used after relative +atmospheric stability. This analysis is used after relative photometry has been determined for data in DVO. This analysis currently is defined to unify the zero points of a collection of @@ -1503,11 +1508,17 @@ photometry corrections for a collection of images distributed over a large range in space and time, but still with significant -overlap. distritions with subustanailaccount for the c - -\subsubsection{Global Astrometry Analysis} - -This operation uses the reference and image detections to determine an -optical distortion model for the camera and static astrometry model -components. The astrometry model includes: (1) field distortion +overlap. + +\tbd{fill out this discussion in the analysis section on the +astrometric and photometric reference catalog}. + +\subsubsection{relastro : Global Astrometry Analysis} + +This operation uses the reference and image detections to improve the +astrometric reference catalog. It determines an improved optical +distortion model for the camera and static astrometry model +components, and then applies the improved astrometric solutions to the +observations to yield high-quality astrometry for the average object +positions. The astrometry model includes: (1) field distortion introduced by the telescope optics, which is a smoothly-varying function of the field position relative to the center of the telescope @@ -1516,5 +1527,8 @@ along with chip-dependent plate-scale modifications needed to represent tilts or warps of the individual detectors relative to the -ideal flat focal plane. . +ideal flat focal plane. + +\tbd{fill out this discussion in the analysis section on the +astrometric and photometric reference catalog}. \subsubsection{DVO shell} @@ -2835,4 +2849,9 @@ to an error upstream in the processing). +\tbd{add discussion of the choices to be made in generating the + static sky image stacks: interpolation methods, selection of input + images by IQ, smoothing of input images by their PSF, weighting and + clipping of input pixels} + Object analysis of the static sky images is {\em not} a part of the Phase 4 analysis. This processing is envisioned to take place @@ -2840,4 +2859,7 @@ scheduled as a separate analysis task, probably run during the day at a time when the computing infrastructure is not under significant load. + +\tbd{add discussion of the multiple image analysis and object + analysis without the static sky (ie, on all input images at once)} \subsubsection{Magic and Phase 4 Modifications} @@ -3161,5 +3183,5 @@ parameter $\nu$, and a collection of annular aperture flux measurements, all of which are also measured for the P4$\Sigma$ -images. In addition, the galaxy-shape parameters $Gamma_1 \& +images. In addition, the galaxy-shape parameters $\Gamma_1 \& \Gamma_2$, along with the complete `polarization' terms are measured, as well as a collection of annular aperture flux and variance @@ -3173,5 +3195,21 @@ per second), it is only necessary to process the complete sky in a year, or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the -object analysis in the other analysis stages. +object analysis in the other analysis stages. These operations are +all functions which will be performed within the PSPhot program using +recipe options. + +\subsection{Astrometric and Photometric Reference Catalog} + +The IPP is responsible for generating the Astrometric and Photometric +(AP) Reference Catalog. The IPP provides several tools for performing +this analysis. The DVO programs \code{relphot}, \code{uniphot}, and +\code{relastro} perform most of the operations required to generate +the AP Reference Catalog. These include the determination of the +image zero-points, the identification of objects with significant +variability, the detection of individual outlier measurements, the +detection of objects with substantial astrometric error, the analysis +of parallax and proper-motions, etc. In addition, the DVO shell +program will be used to generate the color transformations from the +observed data and to perform other tests of the catalog quality. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -3191,6 +3229,5 @@ functions in the operational system, the IPP will make use of Perl as the scripting language to provide the required flow-control to tie the -modules together. \tbd{note that we use C only, not perl for -scripting}. +modules together. This approach satisfies the requirement that complicated low-level @@ -3551,9 +3588,9 @@ from the selected reference catalog. The observed sources are matched to the reference sources, using either a two-point grid search or -optionally a \tbd{triangle match}. Once an approximate match is -found, a linear fit between detector coordinates an projected -celestial coordinates is attempted. The projected coordinate system -may optionally make use of the default telescope distortion model, if -it is known. The radius of the match between observed and reference +optionally a triangle match. Once an approximate match is found, a +linear fit between detector coordinates an projected celestial +coordinates is attempted. The projected coordinate system may +optionally make use of the default telescope distortion model, if it +is known. The radius of the match between observed and reference sources is reduced to improve the statistics of the match. This anaysis mode is used in the Phase 2 processing. @@ -3610,6 +3647,6 @@ \subsection{poisub} -Poisub is the image difference analysis program. \tbd{Paul: please - flesh this out!}. +Poisub is the image difference analysis program. \tbd{finish this +discussion}. \subsection{stac} @@ -3618,5 +3655,5 @@ same region of the sky. It consists of two major stages: the warping stage and the image combination stage with robust outlier rejection. -\tbd{Paul: flesh this out!} +\tbd{update / finish this discussion} \subsection{Command Sequences} @@ -4035,4 +4072,7 @@ \subsection{IPP Pipelines Overview} +\tbd{add the use of Q/A measurements from the IPP CDR Response +document} + The IPP as a whole performs all of the image analysis functions required by the Pan-STARRS telescopes, including images from the full @@ -4380,10 +4420,10 @@ header or new exp table?), the exposure is added to the `raw exposure' table for images of that type. The allowed types are `detrend', (all -bias, dark, flat images), `object', `focus'(??), etc. (** The -different tables represent different analysis modes. This process -also adds an entry to the exp ID / image file match **). This process -also adds all science (OBJECT) exposures to the P1 exposure table (for -mosaic data) or the P2 chip table (for single detector data). These -tables are used to trigger the Phase 1 and Phase 2 analysis stages. +bias, dark, flat images), `object', `focus'(??), etc. The different +tables represent different analysis modes. This process also adds an +entry to the exp ID / image file match. This process also adds all +science (OBJECT) exposures to the P1 exposure table (for mosaic data) +or the P2 chip table (for single detector data). These tables are +used to trigger the Phase 1 and Phase 2 analysis stages. \subsection{Phase 1} @@ -4559,9 +4599,9 @@ rules. -\note{Phase 4 run can be defined by selecting an observation group, a +\tbd{Phase 4 run can be defined by selecting an observation group, a set of exposures, or a set of rules related to a spatial region (eg, region, time range, and filter}. -\note{Phase 4 discussion (and diagram) needs more work} +\tbd{Phase 4 discussion (and diagram) needs more work} \subsection{Analysis Version and Recipes}