Index: /trunk/doc/design/ippSRS.tex =================================================================== --- /trunk/doc/design/ippSRS.tex (revision 1066) +++ /trunk/doc/design/ippSRS.tex (revision 1067) @@ -1,3 +1,3 @@ -%%% $Id: ippSRS.tex,v 1.4 2004-06-05 00:49:48 eugene Exp $ +%%% $Id: ippSRS.tex,v 1.5 2004-06-22 08:35:17 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -73,5 +73,5 @@ \paragraph{``Should''} When used in this specification, the word -``should'' refers to a desired chracteristic of a system component or +``should'' refers to a desired characteristic of a system component or the complete system. @@ -111,58 +111,81 @@ \item Accept raw images from the summit at a sustained rate of 1 - exposure per 30 seconds. - -\item Accept metadata from the summit at a sustained rate of \tbd{XXX -MB / sec}. - -\item Produce high-quality master calibration images from the raw - calibration images. The master calibration images must not - introduce systematic uncertainties greater than \tbd{0.2\%}. - \tbd{Requirements on the speed of processing the calibration - images.} - -\item Pre-process the science images with the high-quality master - calibration images. + exposure (2~GB) per 30 seconds. + +\item Accept metadata from the summit at a sustained rate of \tbr{1 MB + per second}. + +\item Produce master calibration images from the raw calibration + images. The master calibration images must not introduce systematic + uncertainties in the photometry greater than \tbr{0.2\%}. + +\item Pre-process the science images with the master calibration + images. \item Merge multiple pre-processed science images -- from multiple - telescopes or from sequential, dithered exposures -- into single, - cleaned, stacked images with corresponding signal-to-noise maps. - -\item Subtract a static sky image from the cleaned, stacked images to - produce an image of only the transient objects. - -\item Excise the significant transients and outliers from the - pre-processed science images. \tbd{how to handle variable stars?} - -\item merge the cleaned images into the static sky image, and - update the corresponding exposure (S/N) maps. + telescopes or from sequential, dithered exposures -- into stacked + images with corresponding signal-to-noise maps. Pixels from the + input images which are outliers for the ensemble of corresponding + pixels must be excised. + +\item Subtract a static sky image from the stacked images to produce + an image of only the transient objects. + +\item Excise transients and outliers which exceed a user-configurable + threshold in the subtracted image from the pre-processed science + images. + +\item Merge the cleaned images into the static sky image, and update + the corresponding exposure (S/N) maps. \item Detect and measure parameters of objects on the four types of - images: pre-processed images, the stacked image, the difference - image, and the static sky image. + images: pre-processed images, the stacked image, the difference + image, and the static sky image. \item Determine astrometry of the detected objects relative to an - astrometric reference to an accuracy of \tbd{30 mas}, with a limit of - \tbd{xxx} on the outliers. - -\item Determine photometry of the detected objects relative to a - photometric reference to an accuracy of \tbd{5 millimag} relative - photometry and \tbd{10 millimag} absolute photometry in photometric - weather. \tbd{before vs after PS-1 AP Surver} \tbd{bright vs faint - errors} with a limit of \tbd{xxx} on the outliers. \tbd{limit - depends on filter} + astrometric reference. For the Commissioning phase of PS-1, the + astrometric calibration will be limited by the determination of the + optical model of the focal plane, and may be as poor as \tbr{750 + mas}. For the AP reference construction phase of PS-1, after the + optical model has been measured, the astrometry solution must be + limited by the reference catalog in use, and will be in the vicinity + of \tbr{75 mas (UCAC) - 250 mas (USNO B1.0)}. After the construction + of the AP astrometric reference catalog, the accuracy will be limited + by atmospheric variations, and must be no worse than \tbr{50 mas}, + with a goal of \tbr{10 mas}. + +\item Determine photometry of the detected objects, both within an + internal photometric system and in terms of appropriate external + photometric reference systems. For the Commissioning phase, the + accuracy of the photometric calibration will be limited by the + quantity and quality of the standard star observations, and the + consistency of the flat-field images across the camera; the scatter + must be less than \tbr{25 millimags}. During the AP reference + construction phase of PS-1, after the flat-field correction has been + measured, the photometric accuracy will be limited by the standard + star observations, the zero-point determinations, and in the case of + calibration to the external standard, the color corrections. The + photometric accuracy in this stage must be better than \tbr{10 + millimags}. After the construction of the AP Reference Catalog, the + photometric accuracy will be limited by knowledge of the flat-field, + variations in the atmosphere across the field, and the reference + catalogs. The photometric scatter in photometric weather must be + better than \tbd{5 millimag} for relative photometry (relative to the + internal filter system) and \tbd{10 millimag} for absolute photometry + (relative to other filter systems such as the SDSS filters). \item Produce a high-quality astrometric reference catalog from the - extracted objects on a time-scale of 6 months. The astrometric - reference must have an absolute accuracy of \tbd{30 mas} and a local - relative accuracy of \tbd{10 mas}. Proper motions of detected - objects with distances greater than 1000 AU must be determined with - an accuracy of \tbd{XXX mas / year}. + extracted objects within 6 months of the end of the AP Survey. The + astrometric reference must have an absolute accuracy of \tbr{30 mas} + and a local relative accuracy of \tbr{10 mas}. Proper motions of + detected non-solar-system objects must be determined with an + accuracy of \tbr{20 mas / year} for unsaturated, bright stars. \item Produce a high-quality photometric reference catalog from the - extracted point-source objects on a time-scale of 6 months. The - photometric reference must have an consistency across the sky of - \tbd{5 millimag} and an absolute calibration to the external system - defined by \tbd{SDSS} of \tbd{10 millimag}. + extracted point-source objects within 6 months of the end of the AP + Survey. The photometric reference must have an consistency across + the sky of \tbr{5 millimag} and an absolute calibration to the + external system (defined by \tbr{SDSS} and the CFHT Legacy Survey + Standards) with an accuracy of \tbr{10 millimag}. \item Publish the static sky images to the Pan-STARRS published static @@ -174,14 +197,15 @@ \item Provide access to external Pan-STARRS clients to the detected objects on time-scales of \tbr{10 minute} after the image is - obtained.\comment{this is a top-level science requirement.} - -\item Store the raw images for a particular period of, depending on - the survey source of the data. In PS-1, the AP and IVP Survey data - must be stored for the lifetime of the project. Other raw data must - be store for \tbr{1 month}. + obtained.\comment{this is derived from the top-level science + requirement.} + +\item Store the raw images for a period of time which depends on the + survey source of the data. In PS-1, the AP and IVP Survey data must + be stored for the lifetime of the project. Other raw data must be + stored for \tbr{1 month}. \item Store the detected objects for a period of time, depending on the type of detection. Transients from the P4$\Delta$ images may be - excised after \tbr{6 monts}. + excised after \tbr{6 months}. \end{enumerate} @@ -198,5 +222,6 @@ complete set of image processing tasks, including both calibration and science image processing. The IPP must respond to requests for data -from client science pipelines. +from client science pipelines. In the active state, the IPP must +respond to analysis priority requests issued by the IPP users. \subsubsection{Paused State} @@ -209,5 +234,5 @@ \label{req:interactive-state} -In interactive state, the IPP must accept imcoming data and metadata, +In interactive state, the IPP must accept incoming data and metadata, but must not automatically process the data. The IPP must respond to user commands to initiate portions of the data analysis. @@ -242,5 +267,5 @@ the delivered code must be in compliance with the language-independent UNIX operating system standard POSIX (Open Group Based Specifications -Issue 6, IEEE Std 1003.1, 2003). +Issue 6, IEEE Std 1003.1, 2004). \item Source code files must use the UNIX line-break convention (line-feed only). @@ -259,16 +284,16 @@ Functions visible at global scope that are part of the public API must -have names begining with \code{ps} and follow the naming conventions +have names beginning with \code{ps} and follow the naming conventions in the coding standard. Functions visible at global scope but which -are not part of the public interface must have names begining with +are not part of the public interface must have names beginning with \code{p_ps}. Functions that are local to a file must \textit{not} -start \code{ps} (or \code{p_ps}). +start with \code{ps} or \code{p_ps}. Variables visible at global scope which are part of the public API -must have names begining with \code{ps}, and follow the naming +must have names beginning with \code{ps}, and follow the naming conventions in the coding standard. Variables that are visible at global scope but which are not part of the public interface must have -names begining with \code{p_ps}. Variables that are local to a file -must \textit{not} start \code{ps} (or \code{p_ps}). +names beginning with \code{p_ps}. Variables that are local to a file +must \textit{not} start with \code{ps} (or \code{p_ps}). The names of all enumerated types and C-preprocessor symbols (but not @@ -281,5 +306,5 @@ When defining a function to convert from one type to another, the name -must be of the form \code{psOldToAlloc}, e.g.\hfil\break +must be of the form \code{psOldToNew}, e.g.\hfil\break \code{psEquatorialToEcliptic} (\emph{not} \code{psEquatorial2Ecliptic}). @@ -290,6 +315,5 @@ \textit{first}, following the pattern of \code{strcpy}; e.g. \begin{verbatim} -void psAddToVector(restrict psVec *outVec, const restrict psVec *inVec, - int val); +void psVectorCopy(restrict psVector *out, const restrict psVector *in); \end{verbatim} @@ -300,13 +324,8 @@ \item The constructor name should consist of the type name followed by \code{Alloc}; e.g. a type \code{psImage} would be created by a -function -\begin{verbatim} -psImage *psImageAlloc(int nrow, int ncol); -\end{verbatim} - -\item The type should be freed with a destructor named \code{typeFree}, e.g. -\begin{verbatim} -void psImageFree(psImage *img); -\end{verbatim} +function \code{psImage *psImageAlloc();}. + +\item The type should be freed with a destructor named +\code{typeFree}, e.g. \code{void psImageFree(psImage *image);}. \item The constructor must never return \code{NULL}, and no code calling the @@ -344,7 +363,7 @@ \subsubsection{CSCI Deliverable} -All final source code generated for the IPP is to be delivered via -CVS, including the test code. CVS revision history must be included -and made available via CVS. +All final source code generated for the IPP must be delivered via CVS, +including the test code. CVS revision history must be included and +made available via CVS. \subsubsection{Platform architectures and operating systems} @@ -367,7 +386,8 @@ x86/Linux combination. -All timing measurements are to execution time as measured on a -\tbd{Reference Pan-Starrs Computation Node} and assumed to be not -limited by network bandwidth. +\subsubsection{Timing measurements} + +Timing requirements specified in this document must be achieved on the +deployed Pan-STARRS analysis computers. \subsubsection{Software Configuration} @@ -381,8 +401,9 @@ software elements. The SCD provides a detailed description of the roles and responsibilities of these subsystems. In brief, the IPP -consists of a collection of science analysis stages, a set of -architectural components which provide the infrastructure needed to -run the analysis programs, and a collection of hardware on which all -of the software elements exist. +consists of: a collection of science analysis programs which perform +the stages of the data analysis; a set of architectural components +which provide the infrastructure needed to run the analysis programs; +and a collection of hardware on which all of the software elements +exist and operate. The architectural components consist of: @@ -395,30 +416,31 @@ any temporary image data products produced by the IPP. The Image Server is required to meet all of the image storage needs identified - in the top-level requirements above. The Image Server must accept - the incoming data and store it until it is no longer needed by other - portions of the IPP. + in the top-level requirements above. The Image Server may also store + large data files which do not contain imaging data. The Image Server + must accept the incoming data and store it until it is no longer + needed by other portions of the IPP. \item {\bf Astrometry \& Photometry Database (AP):} This component is - required to store and manipulate astronomical objects detected in - various images, as identified above, including individual - measurements of objects on the images, the summary information about - those objects, and reference object data. + required to store and manipulate astronomical objects detected in + images processed by the IPP, including individual measurements of + objects on the images, the summary information about those objects, + and reference object data. \item {\bf Metadata Database:} This component is required to store the - all other data which are neither image files nor astronomical object - data. The Metadata Database is the authoratative source for all - metadata data, including metadata which may be duplicated elsewhere, - such as in the headers of images in the image database. + all other data which are neither image files nor astronomical object + data. The Metadata Database is the authoritative source for all + metadata data, including metadata which may be duplicated elsewhere, + such as in the headers of images in the image database. \item {\bf Controller:} In order to perform the analysis stages - required by the IPP, it is necessary to use distributed computing - processes on a large number of computers. The Controller is - required to manage the collection of analysis stages performed on - these machines. - -\item {\bf Scheduler:} This component is a decision-making mechanism - required to guide the operation of the IPP: to evaluate the - currently available collection of data, to identify the necessary - analysis, and to assign the analysis tasks to the Controller. + required by the IPP, it is necessary to use distributed computing + processes on a large number of computers. The Controller is required + to manage the collection of analysis stages performed on these + machines. + +\item {\bf Scheduler:} This component is a decision-making mechanism + required to guide the operation of the IPP: to evaluate the currently + available collection of data, to identify the necessary analysis, and + to assign the analysis tasks to the Controller. \end{enumerate} @@ -440,5 +462,5 @@ The IPP Image Server must store images on a distributed collection of -computer disks. Individual instinces of a file are only required to +computer disks. Individual instances of a file are only required to be stored on a single machine (striping across computers is not a requirement). @@ -447,13 +469,14 @@ image on a specific machine. If such a request cannot be honored (ie, the machine is down), the IPP Image Server must select an appropriate -machine and notify the requesting agent of the new locations. - -The IPP Image Server store multiple copies of each image, the number -of copies specified independently for each by the user. +machine and notify the requesting agent of the new location. + +The IPP Image Server must store multiple copies of each image upon +request, the number of copies specified independently for each file by +the user. The IPP Image Server must maintain a record of all image copies currently available in the repository. This record must include the image name, location (which machine), the image size, and the state of -the image. +the image (available, locked, deleted). The IPP Image Server must lock images in the repository on request. @@ -465,5 +488,5 @@ which it resides) upon request. -The IPP Image Server must return a specified image upon request. +The IPP Image Server must provide a specified image upon request. The IPP Image Server must delete images in the repository on request. @@ -475,4 +498,9 @@ MB/sec. +\tbd{archive lifetime} + +\tbd{reliability} + +\tbd{backups} \subsubsection{AP Database} @@ -542,4 +570,6 @@ \item given detection, return source image data. +\item given detection, return object. + \item given $(RA,DEC)$, return all images overlapping coordinate. @@ -552,6 +582,6 @@ magnitudes based on calibration information. -\item given a collection of detections, determine the object avergae - magnitude. +\item given a collection of detections in a filter, determine the + object average magnitude in that filter. \item given a collection of objects and detections, determine the @@ -559,5 +589,5 @@ \item given a region, return all possible combinations of the object - or detection magnitudes $(M1 - M2)$. + or detection magnitudes $(M_1 - M_2)$. \item given a list of $(RA,DEC)$ entries, return all nearest objects. @@ -600,6 +630,5 @@ incoming rates. The expected rates are listed in Table~\ref{APrates}, along with the total data volume required for storage space over the -PS-1 lifetime. The AP Database must be able to keep up with these -rates. +PS-1 lifetime. \tbd{archive lifetime} @@ -611,10 +640,34 @@ \subsubsection{Metadata Database} -\tbd{this section needs to be reviewed and revised} +\begin{table} +\begin{center} +\caption{Metadata Classes\label{, and the while +the metadata}} +\begin{tabular}{l} +\hline +\hline +\hline +raw images \\ +pending images \\ +master detrend images \\ +processed images \\ +static sky images \\ +detrend residuals \\ +object detection statistics \\ +master detrend creation statistics \\ +astrometry residuals \\ +warping statistics \\ +processing timing \\ +software installation information \\ +software configuration information \\ +\hline +\end{tabular} +\end{center} +\end{table} The IPP requires a Metadata Database to store and provide access to metadata of various types and from various sources. Metadata in the -context of the IPP represents all data which is not included in the -two data stores discussed above (Images and Detection/Objects). +context of the IPP corresponds to all data which is not included in +the two data stores discussed above (Images and Detection/Objects). Metadata is generated at the telescope and during the various analysis stages @@ -624,30 +677,34 @@ master), for the extracted object lists. Metadata describing the environmental conditions at the telescope must also be stored and -provided as needed. - -If analysis results are exchanged via the metadata database, it must -provide access to the queried data on timescales of $<2$ seconds to -avoid slowing down the analysis systems. - -\tbd{need to extract specific requirements from this} - -\tbd{volume requirements} - -\tbd{queries} - -\tbd{description of images belong in the Metadata database, location - of images is in the Image server} - -\paragraph{Configuration Database} - -The IPP requires a Configuration Database to store and provide access -to information about the IPP itself. Examples of data in the -configuration database include the default parameters for the various -analysis programs, the description of the computing environment, the -process status information, etc. \tbd{part of metadata database?}. - -\tbd{some information must have access limited to specific responsible - people. ie, software / hardware configuration $\rightarrow$ sysadmin; - science parameters $\rightarrow$ science team.} +provided as needed. Table~\ref{metadata} lists the classes of +metadata which must be stored by the Metadata Database. + +If analysis results are exchanged between analysis stages via the +Metadata Database, it must provide access to the queried data on +timescales of $<2$ seconds to avoid slowing down the analysis systems. + +The Metadata Database must store the metadata for the lifetime of the +project. The Metadata Database must be capable of accepting a total +data volume after 2 years of operation of 128 GB. + +The Metadata Database must respond to simple queries which return the +data in the categories listed in Table~\ref{metadata} based on the +primary data key and with basic constraints of time ranges and other +simple conditional constraints. + +The Metadata must store descriptive information about the raw images +received from the summit and the current state of the data processing. +The Metadata must also store descriptive information for each of the +static sky images currently available. + +The IPP requires configuration information defining the organization +and configuration of the IPP itself. The Metadata database must store +the configuration information with restricted access so that only +specific people may change the information. Examples of configuration +data include the default parameters for the various analysis programs, +the description of the computing environment, and the process status +information, etc. The Metadata Database must restrict access to the +scientific parameters to a different group from the software and +hardware configuration parameters. \subsubsection{Controller} @@ -661,5 +718,5 @@ The IPP Controller must detect computers which crash or stop -responding. +responding and set their state to {\tt dead}. The IPP Controller must attempt to re-establish communication with @@ -674,11 +731,12 @@ unavailable, the IPP Controller must attempt to run the task on another node. If the node is available, the IPP Controller must -attempt to run the next task when the current task is completed. +attempt to run a given task only if no higher-priority tasks are +available and no task is currently being executed. The IPP Controller must monitor the output from the task and write it -to an associated log file. - -The IPP Controller must monitor the execution status of the task and -perform the following actions: +to an associated log destination. + +The IPP Controller must monitor the execution status of each task +currently executing on a node and perform the following actions: \begin{enumerate} \item identify the task as successful if it has a valid exit status. @@ -704,5 +762,5 @@ \subsubsection{Scheduler} -The IPP Scheduler intiates analysis tasks which it must send to the +The IPP Scheduler initiates analysis tasks which it must send to the IPP Controller. @@ -712,5 +770,5 @@ The IPP Scheduler must refer to several input data sources to decide -what tasks to intiate. These data sources include the IPP Metadata +what tasks to initiate. These data sources include the IPP Metadata Database, the Summit Metadata Database, and User requests. @@ -726,11 +784,11 @@ When the IPP Scheduler is placed in the {\em paused state}, it must -only intiate User-requested tasks. +only initiate User-requested tasks. When the IPP Scheduler is placed in the {\em interactive state}, it -must intiate User-requested tasks as well as data transfer tasks. +must initiate User-requested tasks as well as data transfer tasks. When the IPP Scheduler is placed in the {\em automatic state}, it must -intiate the most appropriate task based on the inputs. +initiate the most appropriate task based on the inputs. The IPP Scheduler must receive the exit status of tasks from the IPP @@ -755,6 +813,6 @@ group. -The science image analysis stages must perform their analyses quickly -enoough to keep up with the incoming data stream. The required +The science image analysis stages must perform their analysis quickly +enough to keep up with the incoming data stream. The required processing time is derived from the rate at which science images are obtained by PS-1. At a minimum, the Science Image Analysis must keep @@ -764,5 +822,5 @@ night within 12 hours. -The maximum latency between the aquisition of an image and the +The maximum latency between the acquisition of an image and the completion of the science image analysis is set by the science requirements of the fast transient recovery programs. The science @@ -786,5 +844,5 @@ extract bright stars from the image. This extraction must be done in less than \tbr{1 second}. The total number of stars and size of the -bright-star aquisition box must be a user-configurable parameter. +bright-star acquisition box must be a user-configurable parameter. In order for blind astrometry of an image to succeed, it is necessary @@ -796,5 +854,5 @@ between the science image to be processed and the static sky images. -The overlaps must overestimated by a small amount so that errors in +The overlaps must be overestimated by a small amount so that errors in astrometry at Phase 1 will not cause any valid static sky / science image pairs to be missed. The amount of overlap must be a @@ -802,12 +860,12 @@ Sky cells which do not have sufficient science image overlap \tbd{$< -5\%$} must be excluded. - -It is not unusual that an image be obtained with invalid coordinates +5\%$} must be excluded from the overlap table. + +It is not unusual for an image to be obtained with invalid coordinates or without any valid stars. For example, the telescope control system may make an error and report the wrong time or coordinates. Or, the image may be obtained in exceptionally poor conditions with no detected stars. Phase 1 must return a descriptive error message in -these conditions. +these conditions. \subsubsection{Phase 2 : image reduction} @@ -816,7 +874,15 @@ the detector are processed to remove instrumental signatures. +The Phase 2 analysis stage must consult the processing recipe to +define the necessary analysis steps performed by the Phase 2 stage. + Phase 2 must perform the analysis steps only if required by the -processing recipe. The processing recipe must respect exposure time -and background flux limits to select certain stages. +processing recipe. The processing recipe must define the stages to be +executed with optional exposure time and background flux limits to +require or exclude select certain stages. + +In the discussion below, various steps specify that the values are +user-configurable parameters. These parameters must be stored in and +extracted from the Metadata Database. \paragraph{Detrend Image Convolutions} @@ -835,5 +901,5 @@ The Phase 2 analysis must use the OT kernel to grow the traps in the -raw bad pixel mag. +raw bad pixel map. The Phase 2 analysis must mask saturated pixels and a user-specified @@ -845,8 +911,8 @@ \paragraph{Bias correction via overscan subtraction} -Phase 2 must be perform bias subtraction on the image. - -Phase 2 must choose the bias subtraction method and applied statistics -based on a user-configured parameter. +Phase 2 must perform bias subtraction on the image. + +Phase 2 must choose the bias subtraction method and analysis statistic +based on the user-configured parameters. The bias correction must be measured from the image overscan region. @@ -861,5 +927,5 @@ \item subtract a 1-D bias which varies along the overscan. The function to be used must include -a spline or a chebychev polynomial derived from the data values along +a spline or a Chebychev polynomial derived from the data values along the overscan, as specified by the user parameters. @@ -870,7 +936,7 @@ The statistic used to calculate the overscan constant or the inputs to the spline and polynomial fits must be derived from groups of pixels -on the basis of one of several statistics, as specified by the user -parameters. The choice of statistics must include the sample and -robust mean, median, and modes. +on the basis of one of several possible statistics, as specified by +the user parameters. The choice of statistics must include the sample +and robust mean, median, and modes. In the case of a single constant, all of the overscan pixel values are @@ -878,5 +944,5 @@ functional representation, the input values to the fit must represent the coordinate along the overscan, with the statistic derived from the -pixels in the perpedicular direction at each location. +pixels in the perpendicular direction at each location. If specified in the user parameters, sigma-clipping must be performed @@ -903,8 +969,10 @@ \paragraph{Flat-field correction} -The Phase 2 analysis must divide by the provided flat-field image. +The Phase 2 analysis must divide the science image by the provided +flat-field image. The division must handle zero-valued pixels in the flat-field image -without raising floating point exceptions. +without raising floating point exceptions, setting the corresponding +bit value in the mask. The flat-field images must be appropriately normalized (see section @@ -941,6 +1009,6 @@ bit value in the mask. -The Phase 2 analysis must extend the masked region be a -user-configurable growth factor. +The Phase 2 analysis must extend the masked region by a +user-configurable growth factor. The Phase 2 analysis must perform the cosmic ray detection only if it @@ -953,7 +1021,208 @@ The object detection must detect all objects above a user-configured -threshold. \tbd{valid range for the threshold?} The detection -threshold must be a function of the average background flux or the -image noise map. +threshold. The threshold must be a positive value; negative values +must invoke an error. The detection threshold must optionally be a +function of the average background flux or the local noise level. + +The object detection must measure the following object parameters: +\begin{enumerate} +\item object centroid and position errors +\item an extended object position ($x_g, y_g$) +\item instrumental PSF magnitude and error +\item local background level and error +\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object + and their covariance matrix +\end{enumerate} + +Minimal object classification must be performed to distinguish objects +which are consistent with a single PSF, objects which are +inconsistently large, objects which are inconsistently small, and +objects which are saturated. + +The resulting collection of detected objects must be saved along with +the relevant image metadata (\ie filter, exposure time, etc). + +\paragraph{Astrometry} + +The Phase 2 analysis must match the detected objects with known +astrometric reference objects. + +The astrometric reference object coordinates must be adjusted for +proper motion. + +The reference and detected object coordinates must be fit to determine +astrometric parameters for the individual OTAs. + +The OTA astrometric parameters must include Chebychev polynomials of the +coordinates up to 3rd order. + +The fitted number of polynomial orders must be a user-configured +parameter. + +The Cell astrometric parameters must not be allowed to vary in the +fit. + +The fit must be robust, rejecting outlier matches (either stars with +poorly determined proper motion or spurious matches). + +The resulting astrometric solution must be consistent across the OTA +field to within \tbr{300 milli-arcsec}. + +\paragraph{Postage Stamps} + +The Phase 2 analysis must extract subrasters (`postage stamps') +surrounding a user-specified list of coordinates from the flattened +images. + +The postage stamp images must be saved in the IPP Image Server. + +\subsubsection{Phase 3 : exposure analysis} + +The Phase 3 analysis must use the objects detected in Phase 2, matched +with a user-specified reference photometry catalog, to determine the +image photometric zero point and zero-point variations across the +field. + +If zero-point variations are significant \tbd{level TBD}, the +zero-point variations must be modeled with a Chebychev polynomial +correction of order 3 or less. + +The photometric nature of the FPA image must be categorized +\tbd{numerical scale?} on the basis of the zero-point consistency, the +transparency compared with recent long-term measurements in the +filter, and the external indicators of photometricity. + +The Phase 3 analysis must use the objects detected in Phase 2, matched +with an appropriate astrometric reference catalog, to improve the +distortion model used for the image. + +The resulting astrometric accuracy must be limited by the astrometric +reference catalog, ie, 250 mas for USNO-B1.0. + +\subsubsection{Phase 4 : image combination} + +Phase 4 is the image combination stage, in which multiple images of +the same portion of the sky are merged and confronted with the static +sky image. Requirements for the different steps of the process are +given below. + +\paragraph{Extract image pixels} + +The Phase 4 analysis must determine the corresponding set of image +pixels for a given sky cell. + +The corresponding image pixels must be extracted from the input +images, using the astrometric information for each OTA and Cell to +determine the exact overlaps. + +The Phase 4 analysis must not miss any pixels in this match, and it +must read no more than 20\% more pixels than necessary from the input +images. + +The Phase 4 analysis must skip any sky cells with fewer than 5\% of +their pixels overlapping the input images. + +\paragraph{Transform pixel coordinates} + +Pixels which have been extracted from the input images must be mapped +to the corresponding pixels in the sky image. + +The transformation must be based on the measured astrometric solution +for the input images relative to the reference catalog used to +generate the static sky image. + +This warping must use a locally-linear astrometric solution. + +The output image must maintain photometric consistency with the input +image to within 0.2\%. + +\tbd{interpolation? does interpolation method choice risk losing flux?} + +\paragraph{Flux matching} + +The Phase 4 analysis must determine appropriate photometry scaling +factors needed to combine the images photometrically. + +\tbd{is flux matched automatically by calibration?} + +\paragraph{Image outlier pixel rejection} + +When multiple images are combined, the group of input pixels which +contribute to an output pixel must be examined and pixels from the +group of images which are inconsistent with the ensemble \tbd{how +much?} must be identified and flagged. + +This outlier rejection must be performed optionally. + +\tbd{for moving objects and images which are not simultaneous, do we + identify the moving objects?} + +\tbd{use the spatial information? fit a 2-D Nth order polynomial to + the collection of pixels and then look for outliers} + +\paragraph{Initial cleaned image} + +The resulting collection of pixels must be used to construct a single +output image, cleaned of the outliers. + +\paragraph{PSF matching} + +The cleaned, combined image must be PSF matched with the static sky image. + +\paragraph{Image Subtraction} + +The static sky image must be subtracted from the stacked, cleaned +image. + +\tbd{what about different stellar colors?} + +\paragraph{Find objects in the image} + +The Phase 4 analysis must perform object detection on the difference +images. + +All objects in the difference image must be detected and the pixels +belonging to variable sources flagged in the input image. + +The object detection must detect all objects above a user-configured +threshold. Both positive and negative objects must be detected; the +specified threshold must define the absolute value of the detection +thresholds. The detection threshold must optionally be a function of +the average background flux or the local noise level. + +The object detection must measure the following object parameters: +\begin{enumerate} +\item object centroid and position errors +\item instrumental PSF magnitude and error +\item local background level and error +\item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ +\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix +\end{enumerate} + +Minimal object classification must be performed to distinguish objects +which are consistent with a single PSF, objects which are +inconsistent, and objects which are saturated. + +The resulting collection of detected objects must be saved along with +the relevant image metadata (\ie filter, exposure time, etc). + +\paragraph{Cleaned Input Image} + +The pixels flagged as being from the difference image sources must be +masked in the input images. + +A new, cleaned image must be constructed from the masked input images. + +\tbd{how to handle variable stars?} + +\paragraph{Find objects in the image} + +The Phase 4 analysis must perform object detection on the cleaned, +summed image. + +The object detection must detect all objects above a user-configured +threshold. The threshold must be a positive value; negative values +must invoke an error. The detection threshold optionally must be a +function of the average background flux or the local noise level. The object detection must measure the following object parameters: @@ -964,5 +1233,7 @@ \item local background level and error \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their - covarience matrix + covariance matrix +\item the Petrosian radius, magnitude, axis ratio, and angle +\item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. \end{enumerate} @@ -974,205 +1245,9 @@ the relevant image metadata (\ie filter, exposure time, etc). -\paragraph{Astrometry} - -The Phase 2 analysis must match the detected objects with known -astrometric reference objects. - -The astrometric reference object coordinates must be adjusted for -proper motion. - -The reference and detected object coordinates must be fit to determine -astrometric parameters for the individual OTAs. - -The OTA astrometric parameters must include Chebychev polynomials of the -coordinates up to 3rd order. - -The fitted number of polynomial orders must be a user-configured -parameter. - -The Cell astrometric parameters must not be allowed to vary in the -fit. - -The fit must be robust, rejecting outlier matches (either stars with -poorly determined proper motion or spurious matches). - -The resulting astrometric solution must be consistent across the OTA -field to within \tbd{0.2 arcsec}. - -\paragraph{Postage Stamps} - -The Phase 2 analysis must extract subrasters (`postage stamps') -surrounding a user-specified list of coordinates from the flattened -images. - -The postage stamp images must be saved in the IPP Image Server. - -\subsubsection{Phase 3 : exposure analysis} - -The Phase 3 analysis must use the objects detected in Phase 2, matched -with a user-specified reference photometry catalog, to determine the -image photometric zero point and zero-point variations across the -field. - -If zero-point variations are significant \tbd{level TBD}, the -zero-point variations must be modeled with a chebychev polynomial -correction of order 3 or less. - -The photometric nature of the FPA image must be categorized -\tbd{numerical scale?} on the basis of the zero-point consistency, the -transparency compared with recent long-term measurements in the -filter, and the external indicators of photometricity. - -The Phase 3 analysis must use the objects detected in Phase 2, matched -with an appropriate reference catalog, to improve the distortion model -used for this image. - -The resulting astrometric accuracy must be limited by the astrometric -reference catalog \tbd{30 mas for USNO?} - -\subsubsection{Phase 4 : image combination} - -Phase 4 is the image combination stage, in which multiple images of -the same portion of the sky are merged and confronted with the static -sky image. Requirements for the different steps of the process are -given below. - -\paragraph{Extract image pixels} - -The Phase 4 analysis must determine the corresponding set of image -pixels for a given sky cell. - -The corresponding image pixels must be extracted from the input -images, using the astrometric information for each OTA and Cell to -determine the exact overlaps. - -The Phase 4 analysis must not miss any pixels in this match, and it -must read no more than 20\% more pixels than necessary from the input -images. - -The Phase 4 analysis must skip any sky cells with fewer than 5\% of -their pixels overlapping the input images. - -\paragraph{Transform pixel coordinates} - -Pixels which have been extracted from the input images must be mapped -to the corresponding pixels in the sky image. - -The tranformation must be based on the measured astrometric solution -for the input images relative to the reference catalog used to -generate the static sky image. - -This warping must use a locally-linear astrometric solution. - -The output image must maintain photometric consistency with the input -image to within 0.2\%. \tbd{does interpolation method choice risk -losing flux?} - -\paragraph{Flux matching} - -The Phase 4 analysis must determine appropriate photometry scaling -factors needed to combine the images photometrically. - -\tbd{is flux matched automatically by calibration?} - -\paragraph{Image outlier pixel rejection} - -When multiple images are combined, the group of input pixels which -contribute to an output pixel must be examined and pixels from the -group of images which are inconsistent with the ensemble \tbd{how -much?} must be identified and flagged. - -This outlier rejection must be performed optionally. - -\tbd{for moving objects and images which are not simultaneous, do we - identify the moving objects?} - -\tbd{use the spatial information? fit a 2-D Nth order polynomial to - the collection of pixels and then look for outliers} - -\paragraph{Initial cleaned image} - -The resulting collection of pixels must be used to construct a single -output image, cleaned of the outliers. - -\paragraph{PSF matching} - -The cleaned, combined image must be PSF matched with the static sky image. - -\paragraph{Image Subtraction} - -The static sky image must be subtracted from the stacked, cleaned -image. - -\tbd{what about different stellar colors?} - -\paragraph{Find objects in the image} - -The Phase 4 analysis must perform object detection on the difference -images. - -All objects in the difference image must be detected and the pixels -belonging to variable sources flagged in the input image. - -The object detection must detect all objects above a user-configured -threshold. \tbd{valid range for the threshold?} The detection -threshold must be a function of the average background flux or the -image noise map. - -The object detection must measure the following object parameters: -\begin{enumerate} -\item object centroid and position errors -\item instrumental PSF magnitude and error -\item local background level and error -\item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ -\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covarience matrix -\end{enumerate} - -Minimal object classification must be performed to distinguish objects -which are consistent with a single PSF, objects which are -inconsistent, and objects which are saturated. - -The resulting collection of detected objects must be saved along with -the relevant image metadata (\ie filter, exposure time, etc). - -\paragraph{Cleaned Input Image} - -The pixels flagged as being from the difference image sources must be -masked in the input images. - -A new, cleaned image must be constructed from the masked input images. - -\paragraph{Find objects in the image} - -The Phase 4 analysis must perform object detection on the cleaned, -summed image. - -The object detection must detect all objects above a user-configured -threshold. \tbd{valid range for the threshold?} The detection -threshold must be a function of the average background flux or the -image noise map. - -The object detection must measure the following object parameters: -\begin{enumerate} -\item object centroid and position errors -\item an extended object position ($x_g, y_g$) -\item instrumental PSF magnitude and error -\item local background level and error -\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their - covarience matrix -\item the Petrosian radius, magnitude, axis ratio, and angle -\item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. -\end{enumerate} - -Minimal object classification must be performed to distinguish objects -which are consistent with a single PSF, objects which are -inconsistent, and objects which are saturated. - -The resulting collection of detected objects must be saved along with -the relevant image metadata (\ie filter, exposure time, etc). - \paragraph{Image Processing Q/A} Before the image is added to the static sky, it must pass Q/A tests. + +\tbd{how do we specify auotmatic Q/A tests? astrometry, photometry} \paragraph{Update static sky} @@ -1186,30 +1261,13 @@ It is required that the {\em total} processing for each exposure by -the Pan-STARRS system not take longer than $n \times T_{\rm min}$, -where $T_{\rm min}$ is the minimum time between exposures (30 sec), -and $n$ is a small positive number. Increasing $n$ results in a -proportionally higher expenditure on CPUs, hence it is strongly -desirable that $n \le 2$. - -Since we envision 4 OTAs (each 4k pixels, square) being processed by a -single CPU, we need Phase 4 to process 64 (input) Mpix in -approximately 30 sec (since Phase 4 is the most intensive, it should -receive the lion's share of the time budget), or 2 (input) Mpix per -second. - -\paragraph{Accuracies} - -Transformations/mappings from detector to sky must preserve both -photometric and astrometric accuracies: -\begin{itemize} -\item Relative photometric accuracy better than \tbd{0.005 mag} -\item Absolute photometric accuracy better than \tbd{0.02 mag} -\item Relative astrometric accuracy better than \tbd{0.01 arcsec} -\item Absolute astrometric accuracy better than \tbd{0.2 arcsec} -\end{itemize} +the Pan-STARRS system not take longer than the time between a complete +set of exposures. For PS-1, the primary mode of operation will use +four exposures to form a complete set (major frame), with 30 second +exposures times and 2 second readout times. Thus, the complete Phase +4 analysis must be performed on average within 120 seconds, assuming a +separate collection of computers are dedicated to the Phase 2 +analysis. \paragraph{Robustness} - -\tbd{what are the corresponding requirements?} It is essential that the static sky image (which may have been @@ -1218,6 +1276,10 @@ to an error upstream in the processing). +\tbd{what are the corresponding requirements?} + \subsubsection{Calibration Stages} \label{mkcal} + +tbd{Requirements on the speed of processing the calibration images.} The Calibration analysis stages must construct the various types of @@ -1241,4 +1303,8 @@ which the master bias is applied to the input images. +Outlier residual images, those for which the residual bias and +variance in the bias image are excessive ($> 1DN$), must be excluded +from the input image stack the the bias image reconstructed. + \paragraph{dark images} @@ -1260,4 +1326,8 @@ which the master dark is applied to the input images. +Outlier residual images, those for which the residual level and +variance are excessive ($> 1DN$), must be excluded from the input +image stack the the dark image reconstructed. + \paragraph{flat-field images} @@ -1284,4 +1354,9 @@ images, in which the master flat-field is applied to the input images. +Outlier residual images, those for which the residual level and +variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit of +the flat-field image), must be excluded from the input image stack the +the flat-field image reconstructed. + \paragraph{mask images} @@ -1332,13 +1407,15 @@ The \code{fringe} calibration stage must construct residual images, in which the master fringe image is applied to the input images, along -with all necessary preceeding calibration images. +with all necessary preceding calibration images. The \code{fringe} calibration stage must measure the residual fringe amplitude on the residual images. -\paragraph{low-k sky models} +\paragraph{low-spatial-frequency sky models} The \code{sky model} calibration stage must construct a sky model image from a stack of raw night-time sky images. + +\tbd{details of the image construction to be specified} \paragraph{Flat-field correction frame} @@ -1383,5 +1460,5 @@ future Pan-STARRS calibration. The generation of these catalogs is inherently a research project, and will require human control and -intervention. The IPP will be required to provide the data access, +intervention. The IPP is required to provide the data access, manipulation and visualization tools needed to construct these reference catalogs and to assess their quality. In this section, we @@ -1393,16 +1470,16 @@ \begin{center} \caption{Astrometric Reference Catalogs\label{AstroRefs}} -\begin{tabular}{lrrr} -\hline -\hline -Name & scatter & depth & filters \\ - & arcsec & mag & \\ -\hline -Hipparcos & & & \\ -Tycho2 & & & \\ -UCAC & & & \\ -YBx & & & \\ -USNO-Bx & & & \\ -2MASS & & & \\ +\begin{tabular}{lrrrrl} +\hline +\hline +Name & scatter limit & proper & depth & Nstars & filters \\ + & (milli-arcsec) & motion? &(mag) & (millions) & \\ +\hline +Hipparcos & 1 & 2 & 7.3 & 0.1 & V \\ +Tycho2 & 10 & 1 & 11.5 & 2.5 & B,V \\ +UCAC-2 & 20 & 1 & 16.0 & 48.0 & R \\ +USNO-A2.0 & 250 & N/A & 19.0? & 526.2 & B,R \\ +USNO-B1.0 & 200 & 20? & 21.0 & 1042.6 & B,R \\ +2MASS & 70 & N/A & 15.0? & 470.0 & J,H,K \\ \hline \end{tabular} @@ -1413,5 +1490,5 @@ reference on the basis of the observations obtained by the AP survey. The IPP must provide the analysis tools needed to generate the master -astometric reference catalog. Much of the required functionality is +astrometric reference catalog. Much of the required functionality is covered by the AP Database. @@ -1534,10 +1611,10 @@ The required set of Pan-STARRS modules and their functionality is -specfied in the document `Pan-STARRS Image Processing Pipeline Modules +specified in the document `Pan-STARRS Image Processing Pipeline Modules Supplementary Design Requirements' (PSDC-430-xxx), and details of -specific apgorithms are specfied in the document `Pan-STARRS Image +specific algorithms are specified in the document `Pan-STARRS Image Processing Pipeline Algorithm Design Document' (PSDC-430-006). -\subsection{PanSTARRS IPP Library} +\subsection{Pan-STARRS IPP Library} In order to facilitate testing and development, and to encourage @@ -1629,13 +1706,6 @@ \subsubsection{Overview} -\tbd{this section should be parred down a bit by referring more to the - hardware report}. - -\tbd{switch to passive voice (we will address foo $\rightarrow$ foo is -addressed)} - -This section discusses the Pan-STARRS Image Processing Pipeline (IPP) -PS-1 hardware requirements. The hardware requirements addressed in -this section consist of: +This section discusses the IPP PS-1 hardware requirements. The +hardware requirements addressed in this section consist of: \begin{itemize} @@ -1647,13 +1717,11 @@ \end{itemize} -We will address the various hardware requirements by referring to the -assumed data processing and data organization scenarios discussed in -the document \tbd{Pan-STARRS IPP Hardware Report, PSDC-4xx-xx}. The -organization of the data and certain aspects of the data processing -scheme have very large implications for the hardware requirements. We -use the values from that report representing the minimum data volume -and the optimum data organization. We address the data requirements -of the single-telescope Pan-STARRS-1 scenario based on the Design -Reference Mission \tbd{REF}. +The report, `The Pan-STARRS Image Processing Pipeline Computational +Challange' (PSDC-4xx-xx) discusses the assumptions and measurements +made to determine the IPP computing requirements, for both the PS-1 +configuration and the PS-4 configuration, under multiple assumptions +regarding the data volume. The requirements in this section are +derived from that report, and follow the minimal data volumne +assumptions for PS-1. \begin{table}[b] @@ -1664,10 +1732,10 @@ \hline Raw data & 200 TB \\ -static sky & 256 TB \\ -calibration frames & 5 TB \\ -metadata db & 0.3 TB \\ -object db & 4 TB \\ -\hline -total & 466 TB \\ +static sky & 235 TB \\ +calibration frames & 1.8 TB \\ +metadata db & 0.2 TB \\ +AP db & 24 TB \\ +\hline +total & 461 TB \\ \hline \end{tabular} @@ -1680,287 +1748,111 @@ principal areas: raw image data, static sky image data, master calibration images, the metadata database, and the object database. -We discuss each of these data items and their impact on the data -storage requirements for the IPP for PS-1. Table~\ref{storage} -summarizes the data storage requirements in the different scenarios. - -\paragraph{Raw Data Storage} - -There are two basic image types which will be acquired: night-time -science images and calibration images. The night-time science images -consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, -the PS-1 telescope can obtain images at a sustained rate of 1 image -per 30 seconds for the entire night of 10 hours (36000 seconds). A -total of 100 calibration images per night would be a substantial -overestimate of the typical expectation. Combining these numbers, we -can expect to receive a total of 1300 images, or 2.6 TB of data per -night. The total data storage requirements for the raw data are -governed by the number of nights' worth of data we are required to -keep online. \tbd{for the first year, we are required to keep all -images from the AP and IPV surveys. This amounts to a total of 200 -TB of data}. - -\paragraph{Static Sky Data Storage} - -The static sky is represented by images with 0.2 arcsec per pixel. -There will be one summed image and one weight image for each of the -\tbd{6} filters, each stored with 16 bits of resolution, for a total -of 24 bytes per sky pixel. At this resolution, there are 324 Mpix per -square degree, and we will observe a potential total area of 30,000 -square degrees. Allowing for 10\% overage for overlapping tiling, we -require a total of 10.7 Tpix to cover the sky once, or a total of -$\sim 256$ TB to maintain a single image of the static sky in all 6 -filters. - -\paragraph{Calibration Frame Storage} - -The possible required calibration frames consist of the bias, dark, -and mask images, along with one flat, one flat-correction, and -multiple sky/fringe library frames per filter. In fact, not all types -are needed at all stages. It is very likely that we will not require -bias or dark images, and mask images may be represented by a single -byte per pixel. Nonetheless, it is necessary for us to generate and -store all master calibration frames at least until we prove that they -are not needed. We assume a total of 21 calibration images are -necessary (one flat, fringe, and sky per filter, along with a bias, -dark, and mask). If we intend to keep all master calibration frames -for the project lifetime, and generate a new master on a weekly basis -(a reasonable time-scale), then we can expect to require a total of 5 -TB of calibration image by the end of the 2 years of PS-1. We note -that this is likely to be a drastic overestimate as we are unlikely to -need to regenerate all master calibration frames on a weekly -time-scale. - -\paragraph{Metadata Database Storage} - -The metadata data storage requirements are driven by the need to store -the data for the project lifetime. There are two types of metadata -generated at the summit: data associated with images and environmental -data. The environmental data consists of measurements on a regular -cadence, roughly 1 per minute, of a variety of parameters. We suggest -an expected of 1kB per entry, for a total of 1 GB over the two-year -term of PS-1. The additional systems, such as the DIMM, SkyProbe, NIR -Sky Camera, and the LRProbe will have higher data requirements, but -should be considered as separate, self-contained systems. Their data -products are distilled to a limited number of parameters per minute -which are included in the 1kB given above. Furthermore, items such as -guide-star history, if saved, will be saved with the image data and -represents only a small fraction of the total image data volume. Some -subset of the telescope diagnosic information may be a high volume -data product as well, but only retained by the telescope control -system for the purpose of diagnostic studies. Such data will be -excluded from this analysis. - -The image metadata consists of values associated with the FPA (1), the -OTAs (64), and the Cells (4096). Aside from the guide star history, -the total data requirements for each of these entries will be scaled -by the number of bytes required for the metadata from each data level. -Clearly, if the Cell entry is allowed to be large, it will dominate -the total Metadata data volume. We suggest an expected number of 64 -bytes per Cell, 256 B per OTA, and 1k per FPA, yielding a total -metadata volume per exposure of roughly 0.3 MB, completely dominated -by the Cell metadata. With the exposure rates above, we find a total -of metadata volume of 0.3 TB over the two-year term of PS-1. - -\paragraph{Object Database Storage} - -The hardware requirements for the IPP object database are rather -flexible: the total volume depends critically on the depth to which -the object detection analyses are performed (and thus the total number -of object detections) and the number of object parameters which are -measured. We can make very rough estimates that the total number of -detections over the 2 year lifetime of the project may be in the -vicinity of $10^{11}$. We can conservatively estimate the number of -bytes needed to represent each detection as 128 B, resulting in a -total data storage for the object detections of 12 TB. However, this -number depends strongly on the timescale for which the IPP is required -to maintain all object detections, and may potentially be -significantly reduced. +Table~\ref{storage} summarizes the data storage requirements for these +types of data. + +The IPP must store all raw images from the first year from the AP and +IVP surveys. This corresponds to 175,000 images, or 175 TB, assuming +1 GB per image and compression. The IPP will require space for 200 TB +of raw imagery to store the data from these two survey components +along with raw calibration, test, and other raw images not in the AP +and IVP surveys. + +The IPP must store a single copy of the complete static sky in all +four filters. With the assumed image sampling of 0.2 arcsec per +pixel, this corresponds to 9.7 Tpix per filter, or a total of 235 TB +for the 6 filters, with 2 bytes for the noise map and 2 bytes for the +image map. + +The IPP must also store other, smaller collections of data. The other +components contribute only a small fraction of the data storage +requirement. The metadata is a fraction of a terabyte, while the +calibration frames (all master detrend frames) represent at most a few +terabytes. The AP object and detection data make up a total of 24 +terabytes (see Table~\ref{APrates}). + +The IPP must have storage capacity for a total of 461 TB of data. \subsubsection{CPU Requirements} -Phase 2 and Phase 4 dominate the processing requirement, primarily -because they must keep up with the image delivery rate of 1 per 30 -seconds. We have performed benchmarks of a demonstration version for -both the Phase 2 and Phase 4 analyses. - -For the Phase 2, a substantial fraction of the processing time is -consumed by the need to perform FFTs on the images in order to -convolve them with the guide-star kernel, and in the smoothing used -for the object detection process. Additional processing time is -needed by the object detection, deblending, and analysis. Experiments -with the FFTW package show that FFTs may be performed on Intel -processors at rates of approximately 0.25 GHz-sec / Mpix for data sets -of order 1 Megapixel. The FFTs required for the Phase 2 analysis are -performed on the 512$^2$ pixel cells, so these numbers may roughly be -scaled linearly to determine the total time required for OTA -processing. A single FFT on a full OTA, with 64 Cells, therefore -requires roughly 4 GHz-sec. For the full Phase 2 analysis, there are -roughly 4 single direction FFTs required excluding those associated -with object detection; thus the total processing time for these FFTs -is approximately 16 GHz-sec. The addtional analysis steps, excluding -object detection and characterization, account for a small fraction of -this compute time, which we estimate at 10\%. The object detection -stage depends somewhat on the depth to which the analysis is -performed, and the number of measurements made per object. Typical -analysis performed by the Sextractor routine, which performs a -substantial number of per-object analyses, requires 27 GHz-sec for a -full OTA, including the FFTs used for smoothing. We can therefore -assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This -converts to a total of 12800 GHz-sec for a complete major frame. - -For Phase 4, the main computational tasks are combining the multiple -images, with cosmic-ray rejection, and performing the object detection -tasks. Nick Kaiser has done tests of the Phase 4 image combine and -rejection stages, and finds a total processing time of roughly 96 -GHz-sec for a full stack of 4 OTA images. If we add in an additional -34 GHz-sec for detailed object detection and image differencing, we -find a conservative estimage of 130 GHz-sec for a 4-image OTA stack, -equivalent to 7800 GHz-sec for a major frame. - -For PS-1, the typical time for a major frame is $4 \times 30$ seconds. -Some reduction in the load may be gained by reducing the complexity -and depth of analysis for PS-1. Depending on the details and depth of -the analysis, we may reduce the computational load by a factor of 2. - -\begin{table} -\begin{center} -\caption{Data I/O (MB per OTA or Sky-cell) \label{scenarios}} -\begin{tabular}{lr} -\hline -\hline -{\em Phase 2 input} \\ -from summit & $2 \times 32$ MB \\ -input image & {\bf 32 MB} \\ -calibration & {\bf 4 $\times$ 32 MB} \\ -mask image & {\bf 8 MB} \\ -\hline -network I/O: & 64 MB \\ -disk I/O: & 176 MB \\ - & \\ -{\em Phase 2 output} \\ -output image & {\bf 32 MB} \\ -output mask & {\bf 8 MB} \\ -image to P4 & $1.5 \times 32$ MB \\ -mask to P4 & $1.5 \times 8$ MB \\ -\hline -network I/O: & 60 MB \\ -disk I/O: & 40 MB \\ - & \\ -{\em Phase 4} & \\ -input images & $1.5 \times 4 \times 32$ MB \\ -input masks & $1.5 \times 4 \times 8$ MB \\ -static sky & 32x9/4 MB \\ -static weight & 32x9/4 MB \\ -\hline -input: & 304 MB \\ -output: & 96 MB \\ -\hline -\multicolumn{2}{l}{\em Bold-faced entries are access to local-disk} \\ -\multicolumn{2}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ -\end{tabular} -\end{center} -\end{table} +The IPP must provide sufficient computing resources to keep up with +the data analysis tasks. The minimal processing requirement is that +the analysis of a typical night's worth of data be completed within 12 +hours of the start of the night. With a typical night length of 8 +hours, and a maximum read rate of 1 image every 30 seconds, this +implies an average of 45 seconds per image. + +The science image analysis dominates the processing requirements. +Within the science image analysis, Phase 2 and Phase 4 dominate the +processing requirements. These two phases are performed in sequence +with separate computers performing the analyses. They may therefore +be addressed independently. + +The IPP must perform the Phase 2 analysis within an average time of 45 +seconds per single Gigapixel camera image. The Phase 2 analysis has +been measured to require 3200 GHz-sec on a x86/32 bit machine, +implying a requirement of NN GHz for the Phase 2 analysis, if NN sec +are devoted to I/O. + +The IPP must perform the Phase 4 analysis on a set of 4 input frames +within an average time of 180 seconds. The Phase 4 analysis has been +measured to require a total of 7800 GHz-sec on an x86/32 bit machine +for a major frame of 4 input Gigapixel camera images. + +\subsubsection{Network I/O Requirements} + +The switch I/O requirements are defined by the total number of bytes +per second serviced by the network switch. In the assumption that all +Phase 2 processing is performed locally on the nodes which store the +raw images and the corresponding detrend images, and that all Phase 4 +processing requires complete network distribution of both the initial +and updated static sky images, the total I/O for a 180 second +major-frame period is: +\begin{itemize} +\item 8 GB from summit to Phase 2 (4 images @ 2 GB each) +\item 18 GB from Phase 2 to Phase 4 (3 bytes per pixel for image + + mask, 50\% image overhead) +\item 9 GB from Static Sky to Phase 4 (2.25 static-sky pixels per + input image pixel, 4 bytes per pixel). +\item 9 GB from Phase 4 to Static Sky +\end{itemize} +for a grand total of 44 GB over 180 seconds, or 244 MB/second, of +which 26 GB are processed by the Phase 2 nodes and 36 are processed by +the Phase 4 nodes. The IPP must be capable of sustaining this network +load. + +\paragraph{Disk I/O Requirements} + +The disk I/O requirements are determined by the total number of bytes +read from and written to disk. For each major frame processed, the +total I/O to and from disk for Phase 2 is: +\begin{itemize} +\item 8 GB raw image from summit to Phase 2 nodes (4 images @ 2 GB each) +\item 8 GB raw image from Phase 2 disk to memory +\item 40 GB detrend image from Phase 2 disk to memory +\item 12 GB processed image from memory to Phase 2 disk (2 bytes image + + 1 byte mask). +\item 18 GB processed image from Phase 2 disk to Phase 4 +\end{itemize} +for a grand total of 86 GB I/O for Phase 2. Equivalently, for each +major frame processed, the total I/O to and from disk for Phase 4 is: +\begin{itemize} +\item 18 GB processed image from Phase 2 disk to Phase 4 +\item 9 GB static image from Phase 4 disk to memory +\item 9 GB static image from memory to Phase 4 disk +\end{itemize} +for a total of 36 GB I/O for Phase 4. \subsubsection{Per-Node I/O Requirements} Data I/O per node is defined as the number of bytes per second passed -through the node's network adapter. The data throughput for each node -depends strongly on the how the data is organized and processed. In -this section, we identify the data which is passed between nodes for -the two stages of the science analysis process. Table~\ref{scenarios} -lists the per-node data I/O for the analysis stages. - -For PS-1, there are 120 seconds of compute time allowed for each of -the Phase 2 and Phase 4 analyses for the collection of four images -which makes up a cannonical major frame. We use the data I/O volumes -and some assumptions about expected network and disk bandwidth to -estimate the I/O and processing timeline for the four scenarios. From -this analysis, we can judge the total CPU requirements in terms of -GHz, not just GHz-sec. We have assumed that GigE network adapters are -capable of delivering data at 50MB/sec sustained and that a disk RAID -can deliver sustained 100 MB/sec reads and writes. These numbers are -conservative estimates based on recent tests discussed below. Using -these assumptions, Table~\ref{throughput} lists the time allocations -for the processing stages. - -\paragraph{Phase 2 Node I/O Requirements} - -In the assumed data distribution scenario, there is a single CPU -allocated to each OTA in the OTA farm and a single CPU for each Sky -cell process. In addition, all data for the specified OTA are stored -on local disks attached to the same computer as the CPU, with the -result that all Phase 2 I/O is made to a local disk. For each science -OTA image which is observed, each OTA node will read from the network -a total of 2 raw images (one for the original image, one for a backup -copy) and write an average of roughly 1.5 processed images and masks -to the Phase 4 machines for a total of 124 MB of network I/O. During -the processing stage, the OTA node will read from disk a total of 176 -MB (4 calibration frames at 32 MB each, one 16 MB mask, and one raw -science image at 32 MB) and write a total of 40 MB (one processed -image at 32 MB and one mask at 8 MB). Given the assumptions for the -network and disk bandwidths (50 MB/s and 100 MB/s respectively), the -data volumes imply a total I/O period of 4.6 seconds. In this -instance, the network I/O is presumed to be sequential with the disk -I/O. - -\paragraph{Phase 4 Node I/O Requirements} - -Although it is easy to arrange the OTA data in such a way that the -majority of I/O is performed locally, it is not as easy to arrange -this for the Static Sky data used by the Phase 4 analysis. We -therefore make the assumption that the Phase 4 analysis will require -all input OTA data to be loaded across the network, as well as all -Static Sky data. This is somewhat of an overestimate as some of the -Static Sky data will be processed by machines with the data stored -locally, and clever Static-Sky data organization schemes can enhance -this chance. - -In the Phase 4 analysis, the images from the 4 separate telescopes are -combined into a single image, confronted with the appropriate segment -of the static sky, with output difference image and updated static sky -image. If we restrict input access to the individual OTA cells, the -maximum read overhead is 50\% (need to read a 10x10 set of cells for -an 8x8 input image). If the processing is performed on Static Sky -segments equivalent in size to the OTAs, the total volume of input -data per node is 304 MB (192 MB of processed science image, 48 MB of -input mask, 32 MB of static sky image and 32 MB of static sky weight -map) while the output data is 96 MB (32 MB static sky, 32 MB weight -map, and 32 MB difference image). Thus, we require a total of 400 MB -network I/O, which implies an I/O period of 8 seconds. - -\begin{table} -\begin{center} -\caption{Data Throughput \label{throughput}} -\begin{tabular}{lr} -\hline -\hline -Phase 2 per-node network I/O & 2.2 s \\ -Phase 2 per-node disk I/O (read) & 1.3 s \\ -Phase 2 per-node disk I/O (write) & 1.2 s \\ -Phase 2 CPU total & 25 s : 128 GHz \\ -Phase 4 per-node I/O & 8 s \\ -Phase 4 CPU total & 112 s : 70 GHz \\ -Phase 2 switch load & 264 MB/s \\ -Phase 4 switch load & 215 MB/s \\ -Phase 2 to Phase 4 switch load & 160 MB/s \\ -Summit to Phase 2 switch load & 70 MB/s \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Switch I/O Requirements} - -The switch I/O requirements are defined by the total number of bytes -per second serviced by the two switches in the system. - -The Phase 2 network I/O is 124 MB per OTA. With 64 OTAs per image, -and 30 seconds average between images, this implies a total of 264 -MB/s switch bandwidth. The Phase 4 network I/O is 400 MB per sky -cell. With 64 cells and 120 seconds between major frames, this is an -average switch bandwidth of 215 MB/s switch bandwidth. The total -switch-to-switch load is 304 MB per OTA, with an average timescale of -120 seconds. With 64 OTAs, this corresponds to 160 MB/s. The -summit-to-Phase 2 switch load is 70 MB/s. +through the node's network adapter. The data I/O per node is tied to +the total processing power and the total number of nodes. A useful +way to examine the per-node I/O requirements is to compare the I/O and +CPU requirements to determine the required number of processing nodes. +The assumption is made that each CPU is associated with a single disk +RAID which may deliver data at a rate of 100 MB/sec and a GigE +ethernet controller which may deliver data at a sustained rate of 50 +MB/sec, and that each CPU is equivalent to 4 GHz. The IPP must +therefore have a total of 26 Phase 2 nodes and 16 Phase 4 nodes. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -2005,5 +1897,5 @@ See Appendix A \& B of the IPP Library SDR (PSDC-430-007) for the test -verification matricies for the Pan-STARRS IPP Library +verification matrices for the Pan-STARRS IPP Library %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%