Index: trunk/doc/design/ippSDRS.tex =================================================================== --- trunk/doc/design/ippSDRS.tex (revision 2114) +++ trunk/doc/design/ippSDRS.tex (revision 2168) @@ -1,3 +1,3 @@ -%%% $Id: ippSDRS.tex,v 1.5 2004-10-14 05:06:31 eugene Exp $ +%%% $Id: ippSDRS.tex,v 1.6 2004-10-18 22:05:43 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -7,6 +7,6 @@ \shorttitle{IPP SDRS} \author{Eugene Magnier, Paul Price, Josh Hoblitt} -\group{\PS{} Algorithm Group} -\project{\PS{} Image Processing Pipeline} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} \organization{Institute for Astronomy} \version{DR} @@ -26,8 +26,21 @@ DR.03 & 2004.03.25 & Section reorganization \\ \hline DR.04 & 2004.04.13 & Most sections fleshed out \\ \hline -DR.05 & 2004.04.29 & Reorganisation for consistency --- PAP. \\ \hline +DR.05 & 2004.04.29 & Reorganisation for consistency \\ \hline \RevisionsEnd \listoffigures + +\begin{verbatim} +TODOs +- add hardware org diagram to section 3 +- clean 3.4 system ifs: describe types of interactions, which are push which are pull? +- 3.5: move to 3.1? summary of requirements? +- add Image Server figure +- discuss AP DB operations: addstar, delstar, relphot, etc +- discuss AP DB throughput issues +- unify controller discussion +- scheduler: distinguish states +\end{verbatim} + \pagebreak @@ -40,24 +53,58 @@ \subsection{Identification} -This document establishes additional design requirements, beyond those -specified in the Software Requirement Specification (PSDC-430-005), for -the Pan-STARRS Image Processing Pipeline (IPP) as applied to -Pan-STARRS 1 (PS-1), the initial demonstration telescope to be -constructed on Haleakala by Jan 2006. +This document establishes Software Design Requirements for the +Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) +Image Processing Pipeline (IPP) for the prototype telescope PS-1, and +is a System-level controlled specification/design description document +in the official Pan-STARRS engineering specification tree. \subsection{System Overview} -\PS{} is a survey telescope system being developed by the University -of Hawaii Institute for Astronomy (IfA), the Maui High Performance -Computing Center (MHPCC), Science Applications International -Corporation (SAIC), and Massachusetts Institute of Technology (MIT) -Lincoln Laboratory. The baseline system will consist of four 1.8m -telescopes, each with a 1 gigapixel camera capable of sustained image -rates of 2 per minute. A single initial test telescope (PS-1) will -be constructed on Haleakala and will see first light at the beginning -of 2006. The full four-telescope system (PS-4) will follow PS-1 by -roughly 2 years. +The Institute for Astronomy at the University of Hawaii is developing +a large optical synoptic survey telescope system, the Panoramic Survey +Telescope and Rapid Response System (Pan-STARRS). The science goals, +priorities, top-level concept of operations with associated +operational requirements, and system performance drivers with +associated system performance requirements are described in the +Pan-STARRS Science Goals Statement (SGS). As described in this +document, The system conceptual design for Pan-STARRS utilizes an +array of four 1.8m telescopes each with a 7 degree$^2$ field of view, +giving the system an \'etendue larger than all existing survey +instruments combined (defined as the product of the collecting area +$A$ multiplied by the field-of-view solid angle $\Omega$). Each +telescope will be equipped with a 1 billion pixel CCD camera with low +noise and rapid read-out, and the data will be reduced in near real +time to produce both cumulative static sky and difference images from +which transient, moving, and variable objects can be +detected. Pan-STARRS will be able to survey up to $\approx 6,000$ +degree$^{2}$ per night to a detection limit of approximately 24$^{th}$ +magnitude. This unique combination of sensitivity and sky coverage +will open up many new possibilities in time domain astronomy including +a major goal of surveying the Potentially Hazardous Object (PHO) +population down to a diameter of $\approx 300$ meters. In addition, +the Pan-STARRS data will be used to investigate a broad range of +astronomical problems of extreme current interest concerning the Solar +System, the Galaxy, and the Cosmos at large. A prototype single +telescope system, PS-1, is being developed as a preliminary step +before construction of the complete four telescope system. + +\begin{tabular}{ll} +Project sponsor:& AFRL, United States Air Force \\ +Acquirer: & University of Hawaii Institute for Astronomy \\ +User: & Astronomical community \\ +Developer: & University of Hawaii Institute for Astronomy, participating \\ + & institutions, and associated subcontractors +\end{tabular} \subsection{Document Overview} + +The Pan-STARRS IPP Software Requirements Specification contains the +complete system requirements of the Pan-STARRS PS-1 IPP in order to +achieve the top-level performance and operational requirements +specified by the SCD. The requirements flow begun in the SGS and +continued in the SCD is further developed in this SRS to provide +additional derived system and subsystem requirements. + +\subsection{Requirements Definitions} The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV @@ -66,6 +113,22 @@ that series is implied. -Open Issues and TBDs in this document are marked \tbd{in bold, red -type with surrounding square brackets}. +Open issues (TBDs) in this document are marked \tbd{in bold red}. + +Quantities which should be reviewed (TBRs) are marked \tbr{in bold +blue}. + +\subsubsection{``Shall''} When used in this specification, the word +``shall'' refers to an explicit requirement of a system component or +the complete system. + +\subsubsection{``Should''} When used in this specification, the word +``should'' refers to a desired characteristic of a system component or +the complete system. + +\subsubsection{``Will''} When used in this specification, the word +``will'' provides information about a characteristic of a related +system component or a complete related system. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \DocumentsInternalSection @@ -215,5 +278,5 @@ \begin{figure} \begin{center} -\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}} +%\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}} \caption{ \label{hardware} IPP Hardware Organization} \end{center} @@ -251,40 +314,8 @@ Database) are also shown. -%%% needs some work / move around elsewhere -\subsection{System Interfaces} - -\paragraph{MOPS and other Client Science Pipelines} - -The Client Science Programs (CSPs) and the Moving Object Processing -System (MOPS) are not a part of the IPP, but are external systems. We -include them here to show the required interfaces. - -The CSPs and MOPS may query the Pixel DB, the Metadata DB and the -Object DB. In addition, they may write certain fields to the object -DB (e.g., the external identifiers and class of object) as they -process objects, and they may retrieve pixel data from the Nodes. - -Since ``CSPs'' is a vague term, we now give some examples which may -help to illustrate the functionality. - -One example of a CSP is a web front-end to retrieve (published) images -and objects from the Pixel DB and Object DB. - -Another example would be a program interested in searching for -transiting extrasolar planets. Such a program may periodically poll -the Metadata DB for new processed observations in its region of -interest (such as the Galactic Plane), retrieve the object photometry -of all high signal-to-noise stars in the processed observations, and -attempt to identify a planetary transit in progress. - -Yet another example would be a Stationary Transient Object Pipeline, -which would periodically poll the Metadata DB for new processed -observations, and query the Object DB for variable sources which were -identified twice (so that they are not moving objects). - \subsection{System Design Decisions} -Since \PS{} is a survey project, all data from the telescopes will be -uniformly analysed by the \PS{} Image Processing Pipeline (IPP) and +Since Pan-STARRS is a survey project, all data from the telescopes will be +uniformly analysed by the Pan-STARRS Image Processing Pipeline (IPP) and the appropriate resulting data products made available to internal and external science analysis systems as they become available. The @@ -301,6 +332,6 @@ object photometry, and reference astrometry and photometry. -The IPP interacts closely with other \PS{} systems responsible for -other aspects of the \PS{} operation, including the summit systems +The IPP interacts closely with other Pan-STARRS systems responsible for +other aspects of the Pan-STARRS operation, including the summit systems (OATS), the science object database, the Moving Object Processing System (MOPS), and potentially other client science pipelines. @@ -311,5 +342,5 @@ \begin{figure} -\psfig{file=pics/ImageServer,width=15cm,angle=0} +% \psfig{file=pics/ImageServer,width=15cm,angle=0} \caption{The components of the IPP Image Server.} \label{fig:ImageServer} @@ -331,5 +362,5 @@ computer and storage system. In order to achieve the data throughput requirements, the IPP Image Server may distribute the images across -the processor nodes in an organized fashion, i.e.\ associating +the processor nodes in an organized fashion, i.e., associating specific machines with specific detectors. It is not the responsibility of the IPP Image Server to determine which computer @@ -356,6 +387,5 @@ Image Server requires a file system which provides files in the local file system. This may be done over many machines with a network file -system such as NFS or GFS. \tbd{select file system for IPP / test NFS -vs GFS vs Mogile, etc}. +system such as NFS or GFS. The IPP Image Server provides the storage and access mechanisms, but @@ -373,5 +403,5 @@ \end{itemize} -\paragraph{IPP Image Server Client APIs} +\subsubsection{IPP Image Server Client APIs} Clients interact with the IPP Image Server with a small number of C @@ -427,5 +457,5 @@ The IPP Image Server daemon uses a database to store the information about the data storage objects, their instances, and the available -hardware resources. A \tt{mysql} database engine is used to manage +hardware resources. A {\tt mysql} database engine is used to manage the database. The database tables defined for the Image Server are listed in Table~\ref{ImageServerTables}, and their current contents @@ -458,5 +488,5 @@ data volume. -\paragraph{IPP Image Server Maintenance Tools} +\subsubsection{IPP Image Server Maintenance Tools} The IPP Image Server provides a collection of administration tools @@ -473,5 +503,5 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsubsection{Metadata Database} +\subsection{Metadata Database} The IPP Metadata Database acts as a repository for all non-pixel data @@ -489,473 +519,461 @@ for the Metadata Database may be collected and inserted by a separate, dedicated process or analysis pipeline collection of processes. +Metadata which is large in volume or poorly structure may also be +stored in an appropriate container file (FITS Table, FITS Header, XML +File) in the Image Server with the Metadata DB providing pointers to +these files. + +The IPP Metadata Database is a simple database system, consisting of a +number of simple tables without extensive inter-table links. The +\code{mysql} database engine will be used to drive the database. + +\subsubsection{Metadata Tables} +\label{Metadata} + +The complete contents of the Metadata Database will not be completely +specified until the complete collection of data analysis scripts are +available. Even so, we can make a good first pass at the likely +collection of long-term tables, and some of the temporary processing +tables. Table~\ref{MetadtaDBTables} lists the Metadata tables +identified to date for the Metadata Database. The contents of these +tables are outlined in Appendix~\ref{MetadataContents}, with examples +for the data entries and thier data types in many cases. + +\subsubsection{Metadata Queries} + +The IPP provides simple queries to the Metadata Database tables using +autocoded APIs. These queries allow for a single row or a simple +collection of rows based on the primary key. The format of the API is +identical for all Metadata tables. New tables and APIs can be added +to the IPP system by adding to the autocoding table description +files. See Appendix~\ref{Autocode} for futher information. + +\begin{table} +\begin{center} +\caption{Metadata Database Tables\label{MetadataDBTables}} +\begin{tabular}{ll} +\hline +\hline +{\bf Table Name} & {\bf Description} \\ +\hline +Weather & Details on the weather, including internal temperatures. \\ +SkyProbe Transparency & Analysis of SkyProbe B \& V data. \\ +SkyProbe Absorption & Analysis of SkyProbe A data. \\ +SkyProbe Emission & Analysis of SkyProbe E data. \\ +DIMM & Summary of DIMM data analysis. \\ +NIR & Summary statistics from NIR camera. \\ +Dome Status & The time history of the dome status. \\ +Telescope Status & The time history of the telescope status. \\ +Raw FPAs & Information about the raw FPA exposures. \\ +Pending Science Chips & Science images to be processed and status. \\ +Processed Science Chips & Science images which have been migrated to the processed state. \\ +Observation Group & Details about a group of associated observations. \\ +Observation Frame & Major frame information. \\ +Science Processing stats & Details on processed cells. \\ +Chip / Sky overlaps & List of overlaps between sky cells and detectors. \\ +Processed Sky-Cell stats & Details of the sky cell processing. \\ +\hline +\end{tabular} +\end{center} +\end{table} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\paragraph{Metadata Tables} - -Table \tbd{NN} lists the Metadata tables identified for the Metadata -Database. - +\subsection{AP Database} + +The AP (Astrometry \& Photometry) Database is a mechanism to store +data related to astronomical objects derived from various sources with +a variety of associations. The AP Database deals with two related +concepts: {\em objects} and {\em detections}. The objects are +descriptions of astronomical objects while the detections are the +specific measurements of those objects, typically measured from +astronomical images. A collection of {\em detections} may be used to +derive average quantities which describe a particular {\em object}. A +third class of object information which must also be considered are +those supplied by external references. These may be treated as {\em +detections}, with the caveat that access to the raw measurements and +metadata are usually unavailable; the reported measurements and errors +must be accepted as they are reported. + +The AP Database stores the collections of detections which were +derived from specific images from any of the analysis stages. It +provides a mechanism to determine and (in conjunction with the Image +Server) locate the image from which a specific detection was derived. +The AP Database also makes it possible to extract all detections +derived from a specific image and to determine quantities such as the +coordinates of the detection in pixel coordinates on the image. + +The AP Database also has the capability to associate multiple +detections of a specific object. Several major classes of objects +will be present, each of which must be handled correctly. + +First, the most distant stars, compact galaxies, and QSOs will have +nearly fixed locations relative to other nearby stars, with only small +deviations for individual measurements. The association between +multiple detections of such objects is made on the basis of their +coincident positions. The AP Database determines the average position +of the object and the deviations of the individual detections from +that average on the basis of the ensemble of individual detection. + +Second, solar system objects do not have a fixed location. Detections +of such objects are linked by their orbits, and depend on both the +position and the time of the image. The AP Database does not attempt +to make this link, which is the role of the MOPS system. However, it +has the ability to accept identifications made externally with +specified detections and to return the identifier of the moving object +associated with the specific detections. These associations also +include descriptive information such as the offset of the detection +from the predicted location of the detection based on the orbit. This +functionality is required to allow the AP Database to ignore known +moving object detections from other types of queries. + +Third, stars in the general vicinity of the solar system fall in +between these first two classes of objects. Their proper motion and +parallax response is significant enough ($>1$ arcsec in 1 year) that +they are not well-described by an average location and a collection of +offsets. These objects are described by a distance and a proper +motion vector. The AP Database provides the association between the +specific detections and an average object which includes finite +parallax and proper motion. + +Fourth, many detections, especially in their initial states, will not +be associated with a specific astronomical object of any of the above +classes and are treated as orphans. Most of these will be spurious +(not represent real objects), some will be from solar system objects +for which orbits are not yet determined, some will be from faint stars +near the detection limits, some will be from short-term transients +which have only been detected once. The AP Database maintains these +detections until they have been associated with one of the objects +above. The AP Database provides mechanisms by which individual +detections may be migrated back and forth between the orphan state and +association with an astronomical object. + +For every object, and all orphaned detections, the AP Database also +provides the capability to determine the images which observed the +location of the object but for which no detection was made. The +minimum set of information which must be carried for these +non-detections is the image and the associated object or orphan. + +The AP Database also stores the relationships between various +photometric systems and, in some cases, the evolution of that +relationship. It provides mechanisms to convert between the measured +instrumental magnitude of a detection with a specific filter, +detector, and telescope, and at a particular time and the implied +magnitude in the average Pan-STARRS photometry system, given a +determined set of calibrations. It also provides the capability to +convert magnitudes in one system to the magnitudes in another system; +an example of such a conversion is between the average Pan-STARRS +filter systems and the various reference systems appropriate for those +filters. + +The AP Database provides interfaces to extract lists of objects and +detections based on various query parameters. It provides the +capability to extract all detections associated with a specific +object, all non-detections of that object, all non-detections of an +orphan, and summary statistics from these collections. It will also +return all objects or detections within specified spatial regions +including regions bounded by great circles (RA,DEC; GLAT,GLON; +ELAT,ELON) and regions described by a location and a search radius. +It will also return the image parameters associated with a specific +detection including image coordinates of the detection, exposure time, +time and date of the detection, etc. + +The IPP AP Database consists of the following components: + +\begin{itemize} +\item AP Database servers +\item AP Database client APIs +\item AP Database storage hardware +\item AP Database database engine +\item AP Database database tables +\end{itemize} + +\subsubsection{AP Database Tables} + +The AP Database divides the sky into a regions, which are in turn +sub-divided into regions, in a hierarchical series. The regions are +used to subdivide the tables of images, objects, and detections. +These three tables are the three largest in terms of both data volume +and number of rows. Since nearly all interactions with the AP +Database performed by the IPP are limited in spatial coverage, +subdividing the tables allows a specific interaction to search only a +small subset of the data. The table of images is the smallest of the +three; the table of detections is likely to be the largest. As a +result, the images tables will be subdivided at a shallow hierarchical +level, while the objects and detections are subdivided on deeper (more +finely sampled) levels. The region table defines the sky regions and +specifies if the region corresponds to an image table, and object +table, and/or a detection table. It also specified which regions in +the next level of the hierarchy are contained by the region, and which +parent region it belongs to. In addition to improving the spatial +access to the image, object, and detection data, the region table +allows for the multiple computers to serve the database tables. The +region file specifies the machine which stores the specific table. + +The table of Images lists all of the images which provided the data in +the AP Database. In general, these images correspond to the Chips. +\tbd{how does the AP Database know about the relationship between a +collection of chips?}. This table includes sufficient astrometric +parameters to represent the coordinates of the detections to a +sufficient accuracy: \tbr{3rd order polynomial across the chip?}. +\tbr{does the AP Database know about FPA, Chip, Distortion Model, etc? +I think it probably needs to if it is going to solve for distortion +models. however, this operation may be a combination of AP DB +interaction and MD DB interaction.} + +The Images in the image table group are stored in the Image table +which contains the (center? 0,0 pixel?) of the chip. A specific +coordinate can be specified to a single Image region table. However, +it is frequently useful to determine all regions which a specific +image overlaps. The Image Overlaps tables contains a list of the +image regions which are overlapped by each image. + +The Objects table group (divided by region) stores the average +parameters for each astronomical object. Certain details of this +table have not yet been specified. In particular, objects with +significant parallax and/or proper motion may potentially be stored in +a distinct table. Solar system objects, to the extent average +properties are maintained, are certainly stored in a separate table. +A related table, also divided in the same regions, is the Average +Magnitudes table. In this table, there are multiple rows per average +object, one for each of the primary filters of interest for which +photometric averaging is performed. + +The Detections table stores all of the measurements of astronomical +objects on specific images. \tbd{is this table divided into P2, P4S, +P4D tables? 3$\sigma$ objects vs 5$\sigma$ objects? We don't want to +store all detections in a single table, I think}. + +The Non-detections table stores information about detection failures +for each object. If an image is added to the database which overlaps +an object but the object is not detected, an entry is made in this +table. In fact, this table may store only the most recent +non-detection and the total number, or a similar reduced set of +non-detection statistics. + +The Filters table identifies all of the physical filters (specific, +named pieces of glass) known to the system. A related table, +photcodes, defines relationships between specific photometry systems. +A system may consist of a detector, telescope, and specific filter, or +it may be a derived photometry system. \tbd{distinguish between +reference, average, and detection photcodes}. + +\subsubsection{AP Database servers} + +The AP Database functions on a group of computers, with portions of +the tables stored on separate machines, as described above. The +association between a machine and the corresponding table or part of +the sky is defined by the Region table. Each machine has a +corresponding AP Database server which runs on that machine to +interact with the tables available on that machine. A client chooses +one of the machines and sends its query or data to be inserted to that +machine. The server then uses the region table to determine which +machines contain the relevant portion of the sky. The data to be +inserted is divided into corresponding region chunks and sent to the +appropriate servers. In the case of queries, the queries are +redirected to the appropriate server(s). The original server may +collect the results and return them to the original client. + +\subsubsection{AP DB Operations} + +\begin{itemize} +\item addstar +\item delstar +\item relphot +\item uniphot +\item mosastro +\item distortion +\end{itemize} + +\begin{table} +\begin{center} +\caption{AP Detection Classes \& Object Parameters\label{APdetections}} +\begin{tabular}{lrrrr} +\hline +\hline +Object Parameter & P2 & P4S & P4D & SS \\ +\hline +PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ +PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ +star/gal sep & + & + & + & + \\ +$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ +local sky data & + & + & + & + \\ +Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ +S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ +W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ +exp. spaced aps., Poisson noise, variance & - & - & - & + \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{AP Database Tables\label{APDBTables}} \begin{tabular}{ll} \hline -\multicolumn{2}{l}{\bf Metadata Tables} \\ -Weather & Details on the weather, including internal temperatures. \\ -SkyProbe & Analysis of SkyProbe data. \\ -LRProbe & Analysis of LRProbe data. \\ -DIMM & Analysis of DIMM data. \\ -NIR & Analysis of NIR data. \\ -Dome Status & The status of the dome. \\ -Telescope Status & The status of the telescope. \\ -Raw FPAs & Details on raw FPA exposures. \\ -Raw Chips & Details on raw chips. \\ -Raw Cells & Details on raw cells. \\ -Observation Group & Details on a group of observations to be processed. \\ -Chip Guide Stars & Details on guide stars \\ -Science Chip stats & Details on processed chips. \\ -Science Cell stats & Details on processed cells. \\ -Science FPA stats & Details on processed FPAs. \\ -Sky-Detector overlaps & List of overlaps between sky cells and detectors. \\ -Processed Sky-Cell stats & Details on sky cells. \\ -Calibration 1 input stats & Details on input images for Cal 1. \\ -Calibration 1 output stats & Details on output detrend images from Cal 1. \\ -Calibration 2 input stats & Details on input images for Cal 2. \\ -Calibration 2 output stats & Details on output detrend images from Cal 2. \\ -Calibration 3 input stats & Details on input images for Cal 3. \\ -Calibration 3 output stats & Details on output detrend images from Cal 3. \\ +\hline +{\bf Table Name} & {\bf Description} \\ +\hline +Region Table & spatial distribution of tables \\ +Images & The images that have objects in the DB. \\ +Image Overlaps & Image regions which are touched by specific images. \\ +Objects & The objects --- average properties of multiple detections of the same object. \\ +Average Magnitudes & Average photometry in multiple filters \\ +Detections & Detections of sources in an image. \\ +Non-Detections & Non-detections of objects in an image. \\ +Filters & Filters understood by the system. \\ +Photcodes & Transformations between different photometric systems \\ +Database Machines & computers used to store the tables \\ \hline \end{tabular} +\end{center} +\end{table} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\paragraph{Metadata Table Contents} - -Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the Metadata tables -listed above. - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Weather} \\ -Time & The time the weather information was measured. \\ -Temperature & The temperature at \tbd{some place. Will likely want temperatures for a range of locations: -external, dome, secondary, primary for starters.} \\ -Humidity & The relative humidity. \\ -Pressure & The (external) atmospheric pressure. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf SkyProbe} \\ -Time & The time the SkyProbe image was taken. \\ -Filter & Filter used for SkyProbe image. \\ -Transparency & The derived transparency. \\ -Error in transparency & The error in the derived transparency. \\ -Number of stars & The number of stars used to measure the transparency. \\ -Astrometry & The astrometry used on the SkyProbe image. \\ -Exposure time & The exposure time of the SkyProbe image. \\ -Sky brightness & The measured sky (surface) brightness, in physical units. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf LRProbe} \\ -Time & The time the LRProbe observation was taken. \\ -A band absorption & The absorption EW of the atmospheric A band. \\ -B band absorption & The absorption EW of the atmospheric B band. \\ -Absorption component 3 & The absorption EW by some other atmospheric component. \\ -Emission 1 & The emission EW of some sky line. \\ -emission 2 & The emission EW of another sky line. \\ -emission 3 & The emission EW of some other sky line. \\ -Number of stars & Number of stars used to measure the absorptions. \\ -Astrometry & The astrometry used on the LRProbe image. \\ -Exposure time & The exposure time of the LRProbe image. \\ -Sky brightness & The measured sky (surface) brightness, in physical units. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf DIMM} \\ -Time & The time the DIMM observation was taken. \\ -$\sigma_x$ & \tbd{The dispersion in $x$}. \\ -$\sigma_y$ & \tbd{The dispersion in $y$}. \\ -FWHM & The seeing full width at half maximum. \\ -Star coordinates & The coordinates of the measured star. \\ -Exposure time & The exposure time of the DIMM observation. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf NIR} \\ -Time & The time the NIR observation was taken. \\ -Sky brightness & The sky (surface) brightness in the NIR observation. \\ -Sky variance & The variance in the sky (surface) brightness. \\ -Astrometry & The astrometry used on the NIR image. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Dome Status} \\ -Time & The time for which the dome status is valid. \\ -Azimuth & The azimuth of the dome. \\ -Open status & Whether the dome is open or not. \\ -Lights status & Whether lights are on in the dome or not. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Telescope Status} \\ -Time & The time for which the telescope status is valid. \\ -Guide status & The status of the guiding. \\ -Altitude & The telescope altitude. \\ -Azimuth & The telescope azimuth. \\ -RA & The telescope Right Ascension (ICRS $\approx$ J2000). \\ -Dec & The telescope Declination (ICRS $\approx$ J2000).\\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Raw FPAs} \\ -Coords & Coordinates of the boresight (i.e. telescope pointing). \\ -Filter & Filter used for the exposure. \\ -Exposure status & Status of the exposure. \\ -Exposure time & Exposure time for the image. \\ -Airmass & Airmass at which the image was taken. \\ -ObsGroup ID & \tbd{The ObsGroup identification number.} \\ -Observer & The name of the observer, or the version of the telescope scheduler software. \\ -Program & The observing program being executed. \\ -Number of chips & The number of chips that comprise the FPA. \\ -NX, NY & \tbd{Assuming the chips are laid out rectilinearly,} the number of chips in each dimension. \\ -Astrometry & The astrometry used for the FPA. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Raw Chips} \\ -i, j & \tbd{Assuming a rectilinear FPA,} the chip number in each dimension. \\ -ID & Chip identification number. \\ -temps & The chip temperature. \\ -Astrometry & The astrometry used for the chip. \\ -Number of cells & The number of component cells. \\ -NX, NY & \tbd{Assuming the cells are rectilinear,} the number of cells in each dimension. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Raw Cells} \\ -Astrometry & The astrometry used for the cell. \\ -Validity & Is the cell working? \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Observation Group} \\ -ID & Identification number for the observation group. \\ -Number of images & Number of images in the observation group. \\ -Type & Type of observation. \\ -Status & Status of the observation group. \\ -\tbd{etc} & \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Chip guide stars} \\ -Chip ID & The identification number for the chip. \\ -Guide Star ID & The identification number for the guide star. \\ -X, Y & The centroided pixel coordinates of the guide star. \\ -RA, DEC & The sky coordinates of the guide star. \\ -$\sigma_{x}$, $\sigma_{y}$ & The dispersion in the centroids over the particular exposure.\\ -$\Delta X_{\rm max}$, $\Delta Y_{\rm max}$ & The maximum deviation in the centroid over the -particular exposure. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Science Chip stats} \\ -Chip ID & The chip identification number. \\ -State & \tbd{The state of the processing.} \\ -Major frame & \tbd{The major frame the chip belongs to.} \\ -ObsGroup & The observation group the science exposure belongs to. \\ -P1 astrom & The Phase 1 astrometry. \\ -P2 astrom & The Phase 2 astrometry. \\ -P3 astrom & The Phase 3 astrometry. \\ -Number of guide stars & Number of guide stars used for the exposure. \\ -Bias correction method & Method used to correct the bias. \\ -Bias stats & Summary statistics for bias (mean, number of parameters, deviation of residuals, -bias section used). \\ -Flat-field image & The flat-field image that was applied. \\ -Kernel convolution parameters & A description of the OT kernel. \\ -Flat-field stats & Summary statistics for flat-field (sigma of sky). \\ -Mask image & The mask image that was applied. \\ -Masking algorithm & \tbd{The algorithm used to mask the bad pixels.} \\ -Fringe images & The fringe model images that were used. \\ -Fringe stats & Summary statistics for fringes (fringe amplitude, sky sigma) \\ -Object detection stats & Summary statistics for object detection (number of objects, depth, other -input parameters). \\ -Updated astrometry & \tbd{Updated astrometry parameters.} \\ -Astrometry stats & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\ -Reference catalog & The reference catalog that was used for the astrometry. \\ -Updated photometry parameters & The parameters used to update the photometry: magnitude zero point -and other corrections. \\ -Photometry stats & Summary statistics for the photometry (number of stars, $sigma_m$) \\ -Reference catalog & The reference catalog that was used for the photometry. \\ -PSF stats & Summary statistics of the PSF. \\ -Chip state & \tbd{The state of the chip?} \\ -Software versions & Versions of each of the modules used in the processing. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Science Cell stats} \\ -Bias stats & Summary statistics for the bias (mean, parameters, dispersion of residuals, biassec) \\ -P1 astrom & The Phase 1 astrometry. \\ -P2 astrom & The Phase 2 astrometry. \\ -P3 astrom & The Phase 3 astrometry. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Science FPA stats} \\ -FPA ID & The FPA identification number. \\ -State & \tbd{The state of the FPA.} \\ -P1 astrom & The Phase 1 astrometry. \\ -P1 astrom stats & Summary statistics for the Phase 1 astrometry (number of stars, $\sigma_x$, $sigma_y$). \\ -P1 reference catalog & The reference catalog that was used for the astrometry. \\ -P1 software versions & The versions of each of the modules used in the Phase 1 processing. \\ -P1 bright stars & Pointers to the bright stars in the field. \\ -P1 ghosts & Pointers to the ghosts in the field. \\ -P1 large objects & Pointers to the large astronomical objects in the field. \\ -P1 PSF model & Description of the PSF model used in Phase 1. \\ -P3 astrom & The Phase 3 astrometry. \\ -P3 astrom stats & Summary statistics for the Phase 3 astrometry (number of stars, $sigma_x$, $sigma_y$). \\ -P3 reference catalog & The reference catalog that was used for the astrometry. \\ -P3 photom & The Phase 3 photometry. \\ -P3 photom stats & Summary statistics for the Phase 3 photometry (number of stars, $sigma_m$). \\ -P3 reference catalog & The reference catalog that was used for the photometry. \\ -P3 PSF model & Description of the PSF model used in Phase 3. \\ -P3 software versions & The versions of each of the modules used in the Phase 3 processing. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Sky-Detector overlaps} \\ -Chip ID & The identification number of the chip. \\ -Sky Cell ID & The identification number of the sky cell. \\ -State & \tbd{The state of the processing?} \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Processed Sky-Cell stats} \\ -Input Chips & Identification numbers of the chips used to produce the sky cell. \\ -PSF adjustments & \tbd{Adjustments to the PSF.} \\ -CR rejection stats & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ -Image combination parameters & Parameters used for the image combination. \\ -Difference image parameters & Parameters used for the image differencing. \\ -Average reference image depth / weight & \tbd{The weight of the reference image?} \\ -Difference image object detection stats & Summary statistics of the object detection (number of objects, -depth, other input parameters). \\ -Summed image object detection stats & Summary statistics of the object detection (number of objects, -depth, other input parameters). \\ -Software versions & Software versions of modules used in the sky cell processing. \\ -Processing stats & Summary statistics of the processing (CPU time, etc). \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 1 input stats} \\ -Input ID & The input chip identification number. \\ -Output ID & The identification number of the output detrend image. \\ -State & \tbd{State of the processing?} \\ -Accepted? & Is the detrend image of acceptable quality? \\ -Image stats & Assorted image statistics (mean flux, exposure time, airmass) \\ -Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 1 output stats} \\ -Output ID & The identification number of the output detrend image. \\ -Data type & The type of the detrend image (bias | dark | flat) \\ -Number accepted & Number of accepted input images that contributed. \\ -Number rejected & Number of rejected input images (no contribution). \\ -Summary stats & Summary statistics of the combination (deviation, normalisations). \\ -Applicability period & The time period the detrend image is applicable for. \\ -Software versions & The software versions of the modules used in processing. \\ -Processing stats & Summary statistics of the processing (CPU time, etc). \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 2 input stats} \\ -Input ID & The input chip identification number. \\ -Output ID & The identification number of the output detrend image. \\ -State & \tbd{State of the processing?} \\ -Accepted? & Is the detrend image of acceptable quality? \\ -Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ -Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ -Applied reduction & \tbd{Reduction method used?} \\ -Applied params & Parameters of reduction. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 2 output stats } \\ -Output ID & The identification number of the output detrend image. \\ -Data type & The type of the detrend image (bias | dark | flat) \\ -Number accepted & Number of accepted input images that contributed. \\ -Number rejected & Number of rejected input images (no contribution). \\ -Summary stats & Summary statistics of the combination (deviation, normalisations). \\ -Applicability period & The time period the detrend image is applicable for. \\ -Software versions & The software versions of the modules used in processing. \\ -Processing stats & Summary statistics of the processing (CPU time, etc). \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 3 input stats} \\ -Input ID & The input chip identification number. \\ -Output ID & The identification number of the output detrend image. \\ -State & \tbd{State of the processing?} \\ -Accepted? & Is the detrend image of acceptable quality? \\ -Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ -Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ -Applied reduction & \tbd{Reduction method used?} \\ -Applied params & Parameters of reduction. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 3 output metadata } \\ -Input images & Identification numbers of the input chips. \\ -Input image stats & Summary statistics of the input chips. \\ -Input object summary stats & Summary statistics of the objects on the input chips (number, density, etc) \\ -Object rejection criteria & Parameters of the rejection step. \\ -Phot stats & Summary statistics of the relative photometry (Mcal, dMcal, Klam, etc, bin size) \\ -Residual stats & Summary statistics of the residuals. \\ -Output image params & Parameters of the output image (size, etc) \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Astrometric Reference Generation output metadata } \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{1}{l}{\bf Photometric Reference Generation output metadata } \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Reference Data} \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Configuration Data} \\ -\hline -\end{tabular} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Metadata Queries} - -\tbd{How is the Metadata DB queried?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Object Database} - -The IPP Object Database (IOD) acts as a repository for data on all -astronomical objects. This database is required to provide organized -access to objects on the sky, including the access to the photometry -associated with specific input images, moving objects associated with -specific chips. Detailed requirements for the IOD are described in -\tbd{the IOD subsystem specification document xxx-xxx-xxxx}. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object DB Tables} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Object DB Tables} \\ -Images & The images that have objects in the DB. \\ -Objects & The objects --- average properties of multiple detections of the same object. \\ -Detections & Detections of sources in an image. \\ -Non-Detections & Non-detections of objects in an image. \\ -Filters & Filters understood by the system. \\ -Photcodes & \tbd{Transformations between different photometric systems?} \\ -Bright Objects & \tbd{Links to postage stamp images of bright objects.} \\ -Region Tables & \tbd{???} \\ -Average Magnitudes & \tbd{How is this different from an `object'?} \\ -USNO Objects & Objects from the USNO database. \\ -Reference Objects & The reference catalogs for astrometry and photometry. \\ -\hline -\end{tabular} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object DB Table Contents} - -\tbd{Dunno yet} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object DB Queries} - -\tbd{Dunno yet} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Controller} - -\tbd{can a process send a message back to the controller before - process is complete? messages via controller?} - -\tbd{does the controller or the image server decide if a machine is - offline or both?} - -\tbd{I/O tasks vs CPU tasks?} - -The IPP Controller is responsible for managing the processing stages. -The Controller manages the parallel processing of these stages in the -IPP computer hardware environment and reports the completion to the -Scheduler. The Controller must be able to manage more than a single -processing thread to make maximum use of available processor -resources. - -The Controller must honour demands that a processing stage run on a -particular Node. Requests that a processing stage run on a particular -node should be honoured if possible. Where no restriction is placed -on the choice of Node choice by the Scheduler, the processing stage -may be run on any available Node. +\subsection{Controller} + +The IPP uses a group of computers to store and process images and to +manipulate collections of detections. These computers perform any of +a large number of analysis stages or other processing tasks without +significant interprocess communication. It is necessary to have a +mechanism which initiates computing tasks on the different computers, +which monitors the tasks as they are executed, which handles the +output and the errors from these tasks, and which reacts to the +failure of any of the computing nodes. The system responsible for the +tasks in the IPP is the IPP Controller. + +The IPP Controller interacts with the collection of computers under +its management and with other subsystems in the IPP. The IPP +Controller receives a variety of inputs from other subsystems, +described below, and initiates actions such as adding a new process to +its queue. The IPP Controller also provides information to other +subsystems on demand about its processing history and current state. +Each physical computer may have multiple processors; since the IPP +Controller is managing processing tasks, it treats each processor +independently. It is up to the system configuration if each computer +needs to reserve one of its CPUs to manage background tasks or if the +IPP Controller should attempt to send one task per CPU and let the +kernel handle the I/O load. + +Computers managed by the IPP Controller are allowed to be in one of +several states, and the IPP Controller must interact with it in an +appropriate way for each of those states. A computer may be {\tt +alive}, {\tt dead} or {\tt off}. If the computer is {\tt alive}, it +responds to commands from the IPP Controller and may be used for tasks +subject to other constraints. If it is {\tt dead}, the computer is +not responsive and must not be used for executing tasks. The IPP +Controller must identify computers which have died and occasionally +test them to see if they are {\tt alive} again. Computers which are +{\tt off} are not available for tests and must not be tested. +Computers may be set to the {\tt off} or {\tt dead} states by external +subsystems; it is the responsibility of the IPP Controller to return a +computer to the {\tt alive} state if possible. An example scenario: a +computer crashes. At this point the IPP Controller should detect that +the computer is no longer responsive and mark it {\tt dead}. It +should occasionally try to re-establish communication with the +computer, potentially with longer and longer delays between attempts. +A human could be notified if the computer seems to remain {\tt dead} +for a very long time. In another circumstance, a person needs to work +on a computer. They should have the ability to notify the IPP +Controller that the machine is off, perhaps with a prior notification +that the machine should be prepared to go off. Only when the person +is done working and testing the machine, and tells the IPP Controller +that the machine is now {\tt dead} can the IPP Controller attempt to +re-start communications and processing on that computer. + +CPUs on computers which are in the {\tt alive} state may be in one of +two modes: {\tt busy} and {\tt free}. A CPU which is {\tt busy} +currently has a task assigned to it. The IPP Controller may only +assign one task to one CPU at a time. A CPU which is in the {\tt +free} state may have tasks assigned to it. The IPP Controller must +also respect a list of task restrictions which may require specific +tasks to run on specific CPUs or exclude specific tasks from specific +CPUs. + +The IPP Controller accepts tasks from other IPP subsystems. The task +requests include the specific command to be executed and are in the +form of a UNIX command which could be performed on any of the +computing nodes. Any input or output data structures in the commands +must be a valid resource regardless of the node on which the task is +executed. Input and output data resources must be unique where +necessary to avoid conflicts. The IPP Controller gives each task a +unique identifier, which is returned to the requesting entity. The +requestor may then use that ID to obtain status information on that +task or to send control signals to the specific task. + +Task requests may specify a desired node for the task execution. The +IPP Controller attempts to honor the request if the node is {\tt +alive}, but will execute it on another node if the requested one is +{\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP +Controller will choose another node if the specified task is not +allowed on the requested node. In all other cases, the IPP Controller +waits until the currently executing processes, and processes with +higher priority, are completed before executing the specified task on +the requested node. + +Task requests may specify an urgency level. The IPP Controller +determines the priority of the task on the basis of both the priority +and the age of the request. An executing task must be completed on a +CPU before any new task is started on that CPU, regardless of +priority. Tasks may be assigned a priority of 0 in which case they +are maintained in the queue and never executed. + +The IPP Controller monitors the output streams from the executing +tasks and the exit status of the tasks. Each task is associated with +a log file, to which all output is written. The status, including the +exit status, of each task is maintained by the IPP Controller so that +other subsystems may determine if specific tasks have started or +completed. + +The IPP Controller must accept commands from other IPP subsystems. +These commands include those which govern the processing of specified +tasks, those which govern the behavior of specific computing nodes, +and those which request information from the IPP Controller. The IPP +Controller must be able to halt the execution of a specified task, +delete an unexecuted task from the task list, change the priority of +tasks, and change the requested nodes for tasks. The IPP Controller +must also be able to stop the current execution of a task and push it +to the end of the queue and also change its priority. + +The IPP Controller must honor requests (normally from the users) to +change the mode of any computing node on demand between {\tt off} and +{\tt dead}. This would normally be done after a computer has been +rebooted and is release to the IPP Controller for its use. It must +also be able to change the list of allowed tasks as requested by +external commands. + +The IPP Controller must respond to informational requests regarding the +collection of machines and their states as well as the collection of +tasks and their states. The IPP Controller must monitor the execution +times of the different tasks and provide summary statistics. Finally, +the IPP Controller must respond to three top-level commands: {\tt finish}, +{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more +new tasks are accepted on the stack of task, and when all tasks in the +stack have completed, the IPP Controller must exit. When {\tt stop} is +requested, the currently executing tasks must be completed at which +point the IPP Controller must exit, but tasks remaining in the stack which +have not been started are flushed. When {\tt abort} is issued, the +IPP Controller immediately kills all executing tasks and exits. + +The IPP Controller and the IPP Image Server have related needs for +information from the combined storage-and-processing nodes regarding +which nodes are available. It is not yet clear if this information is +best stored in a single location (either IPP Controller or IPP Image +Server), which provides the information to other systems on demand, or +if both systems should maintain the information. Also, it may be +necessary to distinguish nodes which are available for processing from +those that are available to serve data as part of the IPP Image +Server. + +It may be useful for the Controller to distinguish between tasks +dominated by I/O and tasks dominated by data processing. It is +possible that one of each of these types of tasks may be sent to the +same node without significantly impacting the system performance. +Alternatively, it may be necessary to limit a single machine with 2 +CPUs to only one of each of these types of tasks (i.e., one processor +will be working on I/O while the other is working on processing). +Such details will be studied by the IfA IPP Team. The Controller maintains a table of processing nodes available to it @@ -973,20 +991,16 @@ clients and sends them new pending stages when they become free. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \subsubsection{Node Agents} -A Node Agent runs on each of the individual nodes to perform the -processing stages as directed by the Controller. The Node Agents -communicate with the Controller via a socket connection. - -A processing stage is executed in the UNIX user space, and is run as a fork by the -Node Agent. The Node Agent must monitor the standard error and -standard output of the processing stage and save them in separate buffers. If the -process dies, the Node Agent must detect the crash. The Node Agent -must respond to various commands from the Controller. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +A Node Agent runs on each of the individual nodes to perform the tasks +as directed by the Controller. The Node Agents communicate with the +Controller via a socket connection. + +A processing stage is executed in the UNIX user space, and is run as a +fork by the Node Agent. The Node Agent must monitor the standard +error and standard output of the processing stage and save them in +separate buffers. If the process dies, the Node Agent must detect the +crash. The Node Agent must respond to various commands from the +Controller, as follows: \paragraph{Report status} @@ -1012,6 +1026,4 @@ indication that there is no current processing stage (`none'). -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \paragraph{Report stdout} @@ -1023,11 +1035,7 @@ accept all of the buffer output. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \paragraph{Report stderr} Identical to `report stdout', but for stderr. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \paragraph{Kill processing stage} @@ -1038,6 +1046,4 @@ `done'. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \paragraph{Clear processing stage} @@ -1045,6 +1051,4 @@ and the Node state to `idle'. If a processing stage is currently running, it should be killed before the processing stage is cleared. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \paragraph{Start processing stage} @@ -1055,279 +1059,297 @@ of security, for example, by employing SSL authentication. +\subsubsection{Controller User Interface} + +The IPP Controller provides a mechanism for users (either other +programs or humans) to interact with it. The user interface provides +commands to check the current processing job queues, the tables of +successful and failed jobs, to stop or delete jobs, etc. + +\subsubsection{Notes} + +can a process send a message back to the controller before process is +complete? messages via controller? + +does the controller or the image server decide if a machine is offline +or both? + +I/O tasks vs CPU tasks? + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\paragraph{Matrix} - -\tbd{The Node Agent does not wear a suit, nor does it know kung fu.} +\subsection{Scheduler} + +The IPP is responsible for a variety of analysis tasks: processing of +the science images through several stages; routine assessment of the +detrend (instrumental calibration) images used in processing the +science images; construction of replacement detrend images when +needed; generation of astrometric and photometric reference catalogs +based on the collected dataset; and the performance of test analysis +programs. At any point, decisions need to be made about which of +these tasks should be performed, based on an analysis of the contents +of the metadata database, the requirements of the people monitoring +the IPP, and the near-term observing plans. The IPP Scheduler is the +mechanism that assesses these various inputs to guide the decisions +and initiate the actions. + +The IPP Scheduler acts as an intermediary between several components +of the IPP and also between the IPP and external agents such as OTIS +and the users who must monitor the behavior of the IPP. + +The IPP Scheduler sends commands to the IPP Controller for execution. +While the IPP Scheduler chooses the tasks to be performed, it is the +IPP Controller's responsibility to manage the specific tasks executing +on a given processing node. Examples of these tasks include the +process of copying or moving data from the Summit data systems to the +IPP Image Server; image processing analysis stages performed on the +science images by the appropriate processing nodes; and the analysis +of the data in the AP Database. This division of responsibilites +allows us to isolate and encapsulate the functionality of the IPP +Scheduler and the IPP Controller. With this separation, the IPP +Controller does not need to have any information about the details of +the tasks which it executes, while the IPP Scheduler does not need to +have detailed information about the available computer hardware. + +Communication between the IPP Scheduler and the IPP Controller is +bi-directional; the IPP Scheduler sends tasks to the IPP Controller, +while the IPP Controller informs the IPP Scheduler of the outcome of +those tasks. It is not specified whether the IPP Scheduler and IPP +Controller are components of a single software system or interacting +but distinct software components. + +The IPP Scheduler takes as input the current list of pending images, +both science and calibration, and a description of the current +observing plan or strategy on some time-scale. The IPP Scheduler also +takes input from humans who manage the IPP. + +The IPP Scheduler must choose between several types of analysis tasks +based on the contents of those lists and on the requirements of the +users. The list of tasks which the IPP Scheduler must decide between +includes: +\begin{itemize} +\item moving data from the Summit pixel server ($\sim 30$ second timescales) +\item running the science analysis stages ($\sim 30$ second timescales) +\item testing the validity of the current detrend images ($\sim$ + nightly) +\item constructing new detrend images ($\sim$ weekly) +\item updating and improving the photometric and astrometric reference + catalogs ($\sim$ yearly). +\end{itemize} + +The IPP Scheduler chooses between tasks which are relevant on several +different time-scales. The time-scales range from 2 times per minute +to once or twice a year, as noted in the list above. The IPP +Scheduler must also make use of user input in managing such choices. +Users have the option to specify that a particular task or set of +tasks is of higher or lower priority than the norm. + +The scheduler may be viewed as a complex state machine. Our goal is +to design the rules independently from the engine which parses the +rules to detemine which specific jobs to send to the controller. + +\subsubsection{Scheduler User Interface} + +The IPP Scheduler provides a user interface which allows a human +operator, or other processes, to monitor the current state of the +Scheduler. + +The IPP Scheduler defines the operating state of the IPP. When the +IPP is in the {\em automatic state}, the IPP Scheduler performs the +most appropriate of all possible tasks at a particular time. When the +IPP is in the {\em interactive state}, the IPP Scheduler performs only +the requested action regardless of the outcome of the decision trees. +In addition, in the interactive state, the IPP Scheduler must only +perform the requested actions and not attempt to perform the other +normally-required actions. The only exception to this exclusion is +that, in the interactive state, data is still copied from the summit +system. An additional IPP state is the {\em paused state}, intended +for tests or maintenance, in which case the IPP Scheduler does not +perform even the data copy tasks. Every task is performed on demand +by the user. The user command sets the IPP Scheduler in one of these +three states, {\em automatic}, {\em interactive}, and {\em paused}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Design : Science Analysis Tasks and Stages} + +In this section, we discuss the design of the science analysis stages +which perform the fundamental image analysis steps of the IPP. The +IPP science image processing stages perform analyses on the night-sky +science images to extract the science data from these images. These +consist of: Phase 1, the image processing preparation stage; Phase 2, +the image reduction stage; Phase 3, the exposure analysis stage; and +Phase 4, the image combination stage. These analysis tasks must +process the images in a timely manner so that the incoming data stream +will not overload the IPP Image Server. The decision to execute a +specific pipeline for a specific dataset is made by the Scheduler, +which sends the infomation to the Controller. The Controller executes +the pipeline for the data on an appropriate machine and monitors the +success or failure of the processing stage. + +The analysis stages are written as UNIX commands, which may be +executed by the IPP Controller, or may be executed individually by +hand. This aspect makes testing of the complete analysis system much +easier because the individual analysis stages may be tested +independently of each other and the IPP infrastructure. + +In keeping with this design model, the analysis stages have several +methods for accepting and returning the input and output data. All of +the analysis stages load an analysis recipe file, which defines the +details of the analysis. This includes the location of the data +sources (from the metadata, from the image headers, from other +external files, or supplied directly), and which steps to employ. For +example, in the discussion of the Phase 2 analysis below, the recipe +file may specify {\em if} a bias subtraction should be applied, {\em +where} to find the overscan region and {\em which} bias image, if any, +to apply. + +\tbd{further discussion of the recipe / configuration files?} + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsubsection{Scheduler} - -The IPP Scheduler is responsible for initiating the various processing -stages (which are executed by the IPP Controller), based on the state -of the survey as reflected by the IPP Metadata Database (IMD). - -The Scheduler shall maintain a list of processing stages, as well as -the required input and dependencies for each of the processing stagesFor example, the -dependencies for copying pixel data from OATS may be: -\begin{itemize} -\item OATS has new pixel data available; -\item The new pixel data has not been copied. -\end{itemize} -Similarly, the dependencies for executing Phase 2 processing on a chip -may be: -\begin{itemize} -\item The chip pixel data has been copied. -\item Phase 1 has run successfully on the metadata for the FPA to which - the chip belongs. -\item A reduced image (i.e., output from Phase 2) does not already - exist. -\end{itemize} - -When the dependencies are satisfied, the Scheduler shall prepare for -execution the particular processing stage on the appropriate data. -The Scheduler must query the Metdata DB for the most appropriate -calibration data, if required. The processing stage should be -filtered through the IPSDLO in order to assign the processing stage to -a particular Node (if desired) and to determine the URIs for the -required inputs. The processing stage is then passed to the -Controller. - -The Scheduler must also be able to send requests for new calibration -data to OATS, including required flat-fields, flat-field correction -observations, or other specialized observations needed to improve the -calibrations. The Scheduler must balance the need for improved -calibrations with the need to process the science images in a timely -manner given the capabilities of the science pipelines. - -\paragraph{Pollster} - -The Pollster is a program that polls OATS at regular intervals for the -existence of observations not contained in the Metadata DB. New -weather and image metadata are written to the Metadata DB. - -There is no reason why this architectural component cannot be -contained within another (such as the Scheduler), but it is shown here -as separate for simplicity. - -A polling model is adopted so that OATS' interface may be kept as -simple as possible --- OATS should not be concerned with whether the -IPP has received notifications. Under this polling model, it is -specifically the responsibility of the IPP to retrieve from OATS the -metadata that is not not already in the Metadata DB. - -\subsubsection{Pollster} - -The Pollster simply polls OATS on a regular basis for metadata -(including telescope exposures) which is not known by the IPP (i.e., -already written in the Metadata DB). On the discovery of such metadata, -it is written to the Metadata DB. +\subsection{Phase 1: image processing preparation} + +The Phase 1 analysis stage is performed on each science exposure (each +complete FPA image) to calculate basic astrometric data needed by the +later stages. Phase 1 uses the static (pre-determined) telescope +distortion model and a table of nominal OTA positions and rotations, +combined with the guide star pixel and celestial coordinates, to +determine the correct telescope bore-sight, field rotation and +magnification. The guide star coordinates are loaded from the +Metadata database. These calculations are performed by comparing the +observed guide star detector coodinates with the known astrometic +positions of these same stars as reported by an external astrometric +reference. The accuracy of the resulting astrometric solution is +expected to be $\sim 1$ arcsec across the field, sufficient in later +stages to match the vast majority of astrometric reference stars with +their detections with minimal effort. + +In some circumstances, science images may have no guide stars. This +may occur in the Pan-STARRS system if the detectors are not run in OTA +mode, for example for short snapshot images. This may also be the +case if the IPP is being run on non-Pan-STARRS data. In such a +circumstance, the Phase 1 stage uses the provided boresight +coordinates, exposure time, and camera zero-point to predict the pixel +coordinates of known bright stars expected to be found on the +detectors. It then extracts a large box ($\sim$ 30 $\times$ +30\arcsec) around these locations and performs extremely basic object +detection to determine the detector coordinates of those bright stars +which are not saturated but which are significantly above the +background level. By targetting known locations in the image files, +only a small amount of data will have to be read. + +If the image has invalid coordinates or no detectable bright stars, +Phase 1 fails and reports a descriptive error. + +Given the above astrometric solution, the Phase 1 analysis stage +constructs a table of the overlaps between the science image to be +processed and the static sky images that must be constructed. This +table will be used to guide the processing of the static sky in Phase +4. The overlaps should be generously calculated so that small errors +in astrometry at Phase 1 will not cause any valid static sky / science +image pairs to be missed because of the astrometric error at this +phase. It is acceptable for a small number of invalid overlaps to be +identified as these will be excluded in Phase 4. Static Sky cells +which do not have sufficient science image overlap \tbr{$< 5\%$} need +not be processed because the few new measured pixels do not add +significantly to the Static Sky. + +\subsubsection{Notes} + +\begin{verbatim} +possible command forms: + +P1 filename.fits [FPA is single fits file] +P1 filename.list [FPA is collection of files] +P1 FPA IA [FPA info from metadata db] + +sources for the input data: + +distortion model: + metadata table + XML file + FITS table + metadata -> image server + user provided on command line + recipe provided + +camera layout: + metadata table + XML file + FITS table + metadata -> image server + user provided on command line + recipe provided + +boresite coordinates guess: + image header (keywords from recipe) + metadata table + +guide stars + collection of video streams + collection of centroid time histories + list of centroids, coordinates +\end{verbatim} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{System UI} - -A user interface allows a human operator to monitor the Controller and -Scheduler through some user interface (UI). The System UI may -interact with the Controller and Scheduler via a socket connection -using a defined set of commands. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Execute processing stage} - -A new processing stages is sent to the Scheduler. The Scheduler may -filter the processing stages through the IPSDLO, or it may be -prevented from doing so by the user. The Scheduler then passes the -processing stages to the Controller for execution. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Kill processing stage} - -The user may kill an existing processing stage. The Controller is -commanded to kill the particular processing stage. - -\tbd{Should we allow a System UI to kill processing stages sent by -other System UIs?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Get status} - -The System UI may request the current status of the Controller, -including the list of pending, active, and completed processing stages -and the status of the individual processing stages. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Available Nodes} - -The System UI may view and configure the list of Nodes available to -the Controller (e.g., to remove a Node temporarily for maintenance). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Analysis Tasks and Stages} - -In this section, we review the processing stages which are executed on -the Nodes. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Phase 2 : image reduction} \subsubsection{Overview} -The processing stages are the software that process data. These -processing stages are divided into five categories which are -summarised in \S\ref{sec:processingStages}. Each of the processing -stages are described below. - -The processing stages are initiated by the Scheduler, parallized and -managed by the Controller, and executed through the Node Agents on the -nodes. Processing stages are purely serial, and so they may be run on -a single node at once without the need for interprocess communication. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Retrieval} - -The retrieval stages simply retrieve pixel data from an external -source (ordinarily OATS at the Summit, but it could conceivably be -some other external source) and store it on the nodes. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Science Image Processing} - -The IPP science image processing stages perform analyses on the -night-sky science images to extract the science data from these -images. These consist of: Phase 1, the image processing preparation -stage; Phase 2, the image reduction stage; Phase 3, the exposure -analysis stage; and Phase 4, the image combination stage. These -pipelines must process the images in a timely manner so that the -incoming data stream will not overload the IPS. The decision to -execute a specific pipeline for a specific dataset is made by the -Scheduler, which sends the infomation to the Controller. The -Controller executes the pipeline for the data on an appropriate -machine and monitors the success or failure of the processing stage. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 1: image processing preparation} - -The Phase 1 system operates on data from each FPA to calculate basic -astrometric information needed by other stages of the analysis. The -analysis includes: - -\begin{itemize} -\item preliminary astrometry based on the guide-star centroids -\item sky-cell / detector-cell overlaps -\end{itemize} - -The input to this analysis is the list of guide-star pixel centroids -and their celestial coordinates as saved in the metadata database, as -well as the FPA and chip organization and geometry, and the basic -optical distortion for the camera. For the sky-cell / detector-cell -overlaps, the sky tiling scheme is required. - -The output consists of calculated astrometric parameters (linear -transformation + static distortion) for each of the FPA chips. On the -basis of this astrometry, the overlap between the detectors and the -sky-cells is calculated. The output of this calculation is a list of -sky-cell / chip links in a database table. This list of links can be -used by the later stages to initiate the analyses. - -The phase 1 analysis is performed on an FPA basis to ensure that -enough reference stars are available for the astrometry calculation. -Phase 1 cannot be usefully calculated on the basis of a major frame -since the telescope positions are independent; no additional -information is available by combining stars from different FPAs. This -analysis does not restrict the definition of a major frame in any way. - -\tbd{Phase 1 command: P1 (exposure)} - -\tbd{Megacam: P1 654321o} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 2 : image reduction : new version} - -\tbd{how long are processed images kept?} - -\tbd{what subsystem deletes processed images?} - -\tbd{does 'remove' mean 'mask' or 'replace'} - -\tbd{what is the absolute astrometry accuracy at phase 2? 0.1 arcsec -== 0.33 pix?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Concept} - -Phase~2 processing within the \PS{} image processing pipeline is -the de-trend stage, where the images from the detector are processed -to remove instrumental signatures. - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/phase2}} -\caption{ \label{phase2} Phase 2 dataflow} -\end{center} -\end{figure} - -Prior to Phase~2, the Phase~1 process operates on an entire telescope -Focal Plane Array to set the boresight astrometric solution using -the guide stars and initial masking of ghost reflections. - -Phase~2 consists of the following modules: -\begin{enumerate} -\item Form OT kernel; -\item Convolve de-trend images with the OT kernel; +Phase 2 processing within the Pan-STARRS image processing pipeline is +the detrend stage, where the images from the detector are processed to +remove instrumental signatures. This analysis is performed on +individual chips, which can be identified as the data entity which has +a single, continuous astrometric solution. + +Phase 2 consists of the following operations, some of which as noted +may be skipped by the recipe: +\begin{itemize} +\item Load science image +\item Identify appropriate detrend images +\item Load detrend images +\item Form OT kernel +\item Convolve detrend images with the OT kernel \item Mask bad pixels -\item Mask diffraction spikes and optical ghosts; -\item Bias/dark/overscan subtraction; -\item Trim overscan; -\item Non-linearity correction; -\item Flat-field; -\item Subtract sky; -\item Identify CRs by morphology; -\item Determine PSF model; -\item Find and photometer objects in the image; -\item Improved astrometry; and -\item Bright object postage stamps. -\end{enumerate} -These modules are each explained below. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Form OT Kernel} - -The first module for Phase~2 is to form the OT kernel from the image -metadata of pixel shifts made during the exposure. This involves -decoding the metadata and converting it to a data type that can be -used to convolve by. The output is the OT convolution kernel. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Convolve de-trend images} - -\tbd{Must this be a formal convolution with the analytical OT kernel, -or can it be a convolution with a decomposed kernel?} - -\tbd{what is the source of the OT kernel? pixel server?} - -This module convolves the de-trend images with the OT convolution kernel -so that they can be used to de-trend the object image. The inputs +\item Mask diffraction spikes and optical ghosts +\item Bias/dark/overscan subtraction +\item Trim overscan +\item Non-linearity correction +\item Flat-field +\item Subtract sky +\item Identify CRs by morphology +\item Determine PSF model +\item Find and photometer objects in the image +\item Improved astrometry +\item Extract Bright object postage stamps +\end{itemize} + +Several of the steps are explained in detail below. + +\subsubsection{Form OT Kernel} + +Certain detrend images are convolved by the OT kernel, so that they +accurately represent the detrend images appropriate for the object +images, which have been shifted using OT. The detrend images which +must be convolved include: the flat-field and the +high-spatial-frequency fringe images. The appropriate kernel for each +cell of an OTA must be determined from the guide star history, +extracted from the IPP Metadata Database\footnote{or image header}. +If the OT kernel is not available, but the image metadata notes that +it should be, the convolution is skipped, with a warning. + +The first module for Phase 2 forms the OT kernel from the list of +pixel shifts made during the exposure. This involves decoding the +metadata and converting it to a data type that can be used to convolve +by. The output is the OT convolution kernel. + +\subsubsection{Convolve detrend images} + +This module convolves the detrend images with the OT convolution kernel +so that they can be used to detrend the object image. The inputs are: -\begin{enumerate} +\begin{itemize} \item The OT convolution kernel --- from the previous module; \item The appropriate dark frame --- from the IPP Pixel Server; @@ -1335,5 +1357,5 @@ \item The appropriate fringe frame(s) --- from the IPP Pixel Server; and \item The appropriate static bad pixel mask --- from the IPP Pixel Server. -\end{enumerate} +\end{itemize} The module convolves each of the dark frame, flat-field, and the fringe @@ -1341,18 +1363,20 @@ bad pixel mask are grown by the outline of the OT convolution kernel (see Section \ref{ap:masks}). The output results are: -\begin{enumerate} +\begin{itemize} \item The convolved flat-field; \item The convolved fringe frame(s); and \item The updated pixel mask. -\end{enumerate} -Each of these will be used for a later module. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Overscan Subtraction} +\end{itemize} +Each of these will be used for a later module. The convolution method +depends on the size and structure of the OT kernel. If the kernel is +small ($< 5x5$ pixels), direct convolution may be employed. If the +kernel is large, but may be decomposed using Gaussians, then it may be +convolved using a decomposition method. + +\subsubsection{Bias Correction / Overscan Subtraction} This module corrects the object exposures for the electronic pedestal introduced by the readout electronics. The inputs are: -\begin{enumerate} +\begin{itemize} \item The object image --- from the IPP Pixel Server; \item The pixel mask --- from the previous module; @@ -1361,5 +1385,5 @@ \item Detector characteristics (gain, read noise) --- from the Metadata. -\end{enumerate} +\end{itemize} The overscan is averaged (either in bulk, or individually by rows) or @@ -1370,18 +1394,16 @@ regions grown by an additional pixel to counter CCD ``blooming''. The output is: -\begin{enumerate} +\begin{itemize} \item The overscan-subtracted object image; and \item The updated pixel mask. -\end{enumerate} +\end{itemize} These will be used for a subsequent module. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Trim} +\subsubsection{Trim} This module trims the object image and each of the calibration frames to remove the outer edge which was affected by the OT during the exposure. The inputs, each from previous modules, are: -\begin{enumerate} +\begin{itemize} \item The overscan-subtracted object image; \item The corresponding pixel mask; @@ -1389,5 +1411,5 @@ \item The convolved fringe frame(s); and \item The dimension of the OT convolution kernel in each direction. -\end{enumerate} +\end{itemize} Each of the input frames (object image, flat-field, fringe frame(s) @@ -1397,15 +1419,13 @@ modules. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Non-Linearity Correction} +\subsubsection{Non-Linearity Correction} This module corrects images for non-linearity in the detector. The inputs are: -\begin{enumerate} +\begin{itemize} \item The trimmed object image --- from a previous module; and \item The detector non-linearity correction coefficient(s) --- from the Metadata. -\end{enumerate} +\end{itemize} The module corrects the flux in each pixel for non-linearity by applying @@ -1413,36 +1433,32 @@ is the corrected object image, which is used for a later module. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Flat field} +\subsubsection{Flat field} This module corrects the object image for variations in sensitivity over the image. The inputs are: -\begin{enumerate} +\begin{itemize} \item The object image corrected for non-linearity; \item The corresponding pixel mask; and \item The convolved, trimmed flat-field. -\end{enumerate} +\end{itemize} Each of these comes from a previous module. The module divides the object image by the flat-field, masking pixels that are non-positive in the flat-field. The outputs are: -\begin{enumerate} +\begin{itemize} \item The flattened object image; and \item The updated pixel mask. -\end{enumerate} +\end{itemize} Both of these will be used in later modules. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Subtract sky} +\subsubsection{Subtract sky} This module subtracts the sky background from the object image. The inputs are: -\begin{enumerate} +\begin{itemize} \item The object image --- from the previous module; \item The list of objects on the image --- from the object database; and \item The convolved, trimmed fringe frame(s) --- from a previous module. -\end{enumerate} +\end{itemize} The module masks (though {\em not} in the ``official'' pixel mask) all @@ -1456,15 +1472,13 @@ which is used for the next module. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Identify CRs by morphology} +\subsubsection{Identify CRs by morphology} This module identifies cosmic rays (or other hot pixels missed in the static bad pixel mask) on the basis of their morphology. The inputs are: -\begin{enumerate} +\begin{itemize} \item The object image; and \item The corresponding pixel mask. -\end{enumerate} +\end{itemize} Both of these come from a previous module. @@ -1473,16 +1487,14 @@ in each direction. Masked pixels are interpolated over. The outputs are the updated pixel mask, which is sent to the IPP pixel server for -use in Phase~3, and is also used for the next module; and the object image, +use in Phase 3, and is also used for the next module; and the object image, which is sent to the IPP Pixel Server. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Find objects} +\subsubsection{Detect and Measure objects} This module finds objects on the object image. The inputs are: -\begin{enumerate} +\begin{itemize} \item The sky-subtracted object image; and \item The corresponding pixel mask. -\end{enumerate} +\end{itemize} Both of these come from a previous module. @@ -1493,16 +1505,28 @@ object image. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Bright object postage stamps} +Object catalogs from Phase 2 shall consist of at least the +following elements for each object: +\begin{itemize} +\item Object centre, with corresponding errors; +\item Object magnitude, with corresponding error; +\item Object isophotal magnitude, with corresponding error; +\item Object FWHM; +\item Object elliptical axis lengths; and +\item Object position angle for ellipse. +\end{itemize} + +Though further details may be required for catalogs in Phase 4, +the above details are minimum requirements for Phase 2 catalogs. + +\subsubsection{Bright object postage stamps} This module saves postage stamps of bright objects, so that extra care with regard to astrometry and photometry can be taken with them at a later stage. The inputs, each from a previous module, are: -\begin{enumerate} +\begin{itemize} \item The sky-subtracted object image; \item The corresponding pixel mask; and \item The catalog of objects. -\end{enumerate} +\end{itemize} The module makes postage stamps of all objects brighter than a given @@ -1511,10 +1535,29 @@ the IPP Pixel Server. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Metadata Required} +\subsubsection{Pixel Masks} +\label{ap:masks} + +This section describes the requirements on Bad Pixel Masks (BPMs). +These will consist of bit masks for each pixel. For Phase 2, flags +are required for at least each of the following pixel attributes: +\begin{itemize} +\item The pixel is a charge trap; +\item The pixel is a bad column; +\item The pixel is saturated in the A/D converter; +\item The pixel is non-positive in the flat-field; +\item The pixel is part of a row that has excess noise; and +\item The pixel is determined to be a cosmic ray, based on its +morphology. +\end{itemize} + +Of these, only masks for the charge traps need to be grown by the +extent of the OT convolution kernel. For other pixel types, +orthogonal transfer of the flux in this pixel will not (necessarily) +affect the flux in neighbouring pixels + +\subsubsection{Phase 2 Metadata} The following metadata associated with the images are required for -Phase~2 operation: +Phase 2 operation: \begin{itemize} \item The orthogonal transfer (OT) image shifts made during the @@ -1526,66 +1569,33 @@ detrend images; \item Exposure time --- for the photometric calibration; -\item Detector gain --- for calculating photometric errors; and +\item Detector gain --- for calculating photometric errors and +determining the quality of the overscan; \item Detector read noise --- for calculating photometric errors and determining the quality of the overscan; \end{itemize} -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Pixel Masks} -\label{ap:masks} - -This section describes the requirements on Bad Pixel Masks (BPMs). -These will consist of bit masks for each pixel. For Phase 2, flags -are required for at least each of the following pixel attributes: -\begin{enumerate} -\item The pixel is a charge trap; -\item The pixel is a bad column; -\item The pixel is saturated in the A/D converter; -\item The pixel is non-positive in the flat-field; -\item The pixel is part of a row that has excess noise; and -\item The pixel is determined to be a cosmic ray, based on its -morphology. -\end{enumerate} - -Of these, only masks for the charge traps need to be grown by the -extent of the OT convolution kernel. For other pixel types, -orthogonal transfer of the flux in this pixel will not (necessarily) -affect the flux in neighbouring pixels - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Object Catalogs} -\label{ap:catalogs} - -Object catalogs from Phase 2 shall consist of at least the -following elements for each object: -\begin{enumerate} -\item Object centre, with corresponding errors; -\item Object magnitude, with corresponding error; -\item Object isophotal magnitude, with corresponding error; -\item Object FWHM; -\item Object elliptical axis lengths; and -\item Object position angle for ellipse. -\end{enumerate} - -Though further details may be required for catalogs in Phase~4, -the above details are minimum requirements for Phase~2 catalogs. - -\tbd{Phase 2 command: P2 (exposure.ota.fits)} -\tbd{Megacam: P2 654321o.fits[ccd00] - what are output names?} -\tbd{PS FPA is saved as a collection of MEF files. Megacam FPA is - saved as a single MEF file. how to handle this difference?} +\subsubsection{Notes} + +\tbd{how long are processed images kept?} + +\tbd{what subsystem deletes processed images?} + +\begin{figure} +\begin{center} +\resizebox{8cm}{!}{\includegraphics{pics/phase2}} +\caption{ \label{phase2} Phase 2 dataflow} +\end{center} +\end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\paragraph{Phase 3 : exposure analysis} +\subsection{Phase 3 : exposure analysis} The Phase 3 system operates on the combined Phase 2 results from an FPA to determine improved solutions for the image calibrations and to provide the parameters needed by Phase 4. The Phase 3 output is saved -by the IMD, and consists largely of improved values of the -calibrations already determined by Phase 2. The analysis performed by -this pipeline consists of: +by the Metadata Database, and consists largely of improved values of +the calibrations already determined by Phase 2. The analysis +performed by this pipeline consists of: \begin{itemize} @@ -1598,4 +1608,30 @@ \end{itemize} +In the Phase 2 analysis, the astrometric solutions were determined +independently for each chip. These solutions are limited by the +assumption of a static distortion and by the accuracy of the +astrometric reference. In the phase 3 analysis, the astrometric +solutions of the $N$ FPA images are improved by... + +For image combination in phase 4, should we use relative astrometry to +map N-1 images to 1, or are we sufficiently accurate to use absolute +astrometry to map N images to the sky-cells? + +In the Phase 2 analysis, the background is determined based only on +the available sky in a single chip. However, the background +structures are normally correlated on the scale of the FPA, so an +improved background solution can be determined by combining the +information from many chips. \tbd{is the background correlated +between FPAs?} + +Phase 3 photometric improvement + +In the Phase 4 analysis, the $N$ FPA images are optimally combined to +create a single image of the sky with bad-pixel and cosmic-ray +rejection. This combination requires the calculation of a set of PSF +kernels to convert each of the input images to a single, common PSF. +These PSF kernels are determined from the per-chip PSFs measured in +Phase 2. + \begin{figure} \begin{center} @@ -1605,57 +1641,18 @@ \end{figure} -In the Phase 2 analysis, the astrometric solutions were determined -independently for each chip. These solutions are limited by the -assumption of a static distortion and \tbd{by the accuracy of the -astrometric reference}. In the phase 3 analysis, the astrometric -solutions of the $N$ FPA images are improved by \tbd{???}. - -\tbd{what is the expected accuracy of the relative astrometric - solution compared to the absolute astrometric solution?} - -\tbd{for image combination in phase 4, should we use relative - astrometry to map N-1 images to 1, or are we sufficiently accurate - to use absolute astrometry to map N images to the sky-cells?} - -In the Phase 2 analysis, the background is determined based only on -the available sky in a single chip. However, the background -structures are normally correlated on the scale of the FPA, so an -improved background solution can be determined by combining the -information from many chips. \tbd{is the background correlated -between FPAs?} - -\tbd{Phase 3 photometric improvement??} \tbd{Phase 3 determined -accurate relative photometry between the N images which are to be -combined in the Phase 4 analysis. Is this more accurate than the -absolute photometry solution? (probably)} - -In the Phase 4 analysis, the $N$ FPA images are optimally combined to -create a single image of the sky with bad-pixel and cosmic-ray -rejection. This combination requires the calculation of a set of PSF -kernels to convert each of the input images to a single, common PSF. -These PSF kernels are determined from the per-chip PSFs measured in -Phase 2. - %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\paragraph{Phase 4 : image combination} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Phase 4 Concept} - -Phase 4 processing within the \PS{} image processing pipeline is -the final stage of processing for a science image. It operates on -each sky cell that has overlapping imaging data from the exposure(s) -being processed, and produces the main output image data products of -the pipeline --- the difference images and a deep static sky image --- -along with the associated catalogs of static and variable sources. - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/phase4}} -\caption{ \label{phase4} Phase 4 dataflow} -\end{center} -\end{figure} +\subsection{Phase 4 : image combination} + +\subsubsection{Overview} + +Phase 4 processing within the Pan-STARRS image processing pipeline is +the image combination stage of processing for a science image. It +operates on each sky cell that has overlapping imaging data from the +exposure(s) being processed, and produces a set of clean, combined +images of the sky. It also subtracts the current static sky image to +generate a difference image, which it uses to identify transient +objects. These are then excised from the summed image, which is in +turn then added to the static sky image. Prior to Phase 4, the Phase 3 process produces the following products: @@ -1665,16 +1662,15 @@ \item astrometric calibration with mapping to sky cells; and \end{itemize} + These will each be used by the Phase 4 modules: -\begin{enumerate} +\begin{itemize} \item Combine Images; \item Identify Sources; \item Transient Identification; and \item Add to Static Sky. -\end{enumerate} +\end{itemize} These modules are each explained below. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Combine Images} +\subsubsection{Combine Images} \tbd{for moving objects and images which are not simultaneous, do we @@ -1687,5 +1683,5 @@ telescope, rejecting artifacts such as cosmic rays and low altitude streaks. The inputs to this module are: -\begin{enumerate} +\begin{itemize} \item the sky-subtracted images that overlap the sky cell (or portions thereof) --- from the IPP Pixel Server (or directly from Phase 3); @@ -1697,5 +1693,5 @@ signal-to-noise (i.e.\ sky noise divided by the square of the seeing) --- from metadata associated with the images. -\end{enumerate} +\end{itemize} The module maps the detector images to the sky cell using the specified @@ -1714,5 +1710,5 @@ The outputs from this module are: -\begin{enumerate} +\begin{itemize} \item The combined sky cell image --- sent to the IPP Pixel Server and/or the next module; @@ -1722,9 +1718,7 @@ \item Catalog of sources on the combined sky cell image --- sent to the IPP Object Database. -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Identify Sources} +\end{itemize} + +\subsubsection{Identify Sources} This module identifies sources in the combined sky cell image. The @@ -1736,17 +1730,15 @@ is the catalog of sources on the combined sky cell image, which is to the IPP Object Database. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Transient Identification} + +\subsubsection{Transient Identification} \tbd{what about different stellar colors?} This module identifies variable/moving sources. The inputs are: -\begin{enumerate} +\begin{itemize} \item The combined sky cell image --- from the previous module or the IPP Pixel Server; and \item The current static sky image --- from the Sky Image Server. -\end{enumerate} +\end{itemize} The module subtracts the current static sky image from the combined sky @@ -1779,5 +1771,5 @@ The module outputs: -\begin{enumerate} +\begin{itemize} \item Combined sky cell image, with all variable sources masked --- used for the next module; @@ -1786,9 +1778,7 @@ \item Catalog of variable sources --- sent to the IPP Object Database. -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Add to Static Sky} +\end{itemize} + +\subsubsection{Add to Static Sky} \tbd{how to handle variable stars?} @@ -1798,5 +1788,5 @@ performed if the new data is of sufficient quality that it will not degrade the static sky image. The inputs are: -\begin{enumerate} +\begin{itemize} \item The combined sky cell image with variable sources masked --- from a previous module; @@ -1806,5 +1796,5 @@ each of the images --- estimate made from metadata associated with each image. -\end{enumerate} +\end{itemize} The sky cell image is added to the static sky. The sky cell image @@ -1816,14 +1806,12 @@ The output is: -\begin{enumerate} +\begin{itemize} \item The new static sky image --- sent to the Sky Image Server; \item The Catalog of sources on the new static sky image --- sent to the IPP Object Database; and \item The estimated limiting magnitude for the new static sky --- metadata associated with the the new static sky image. -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Notes} +\end{itemize} + +\subsubsection{Notes} \begin{itemize} @@ -1838,85 +1826,195 @@ \end{itemize} +\begin{figure} +\begin{center} +\resizebox{8cm}{!}{\includegraphics{pics/phase4}} +\caption{ \label{phase4} Phase 4 dataflow} +\end{center} +\end{figure} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Design : Calibration Image Processing} + +The Calibration Analysis Stages construct calibrations from the +relevant input data. Some of these combine multiple raw input images +together, after processing, to create a high-quality high-signal +master calibration image. Some of the calibrations are used to +correct other calibrations. Each of the calibration stages must also +provide the tools to test the quality of the input data against +existing master calibration data and to test the consistency of +multiple measurements of the calibration. + +The Calibration analysis stages may be performed on whatever +timescales are appropriate and necessary to maintain the quality and +relevance of the calibration images. Below, we list the specific +calibration data which must be constructed in the calibration analysis +stages. + +The IPP must generate basic calibration images using the raw bias, +dark, and flat-field (dome or twilight) images obtained by the +telescope as the input. The analysis of these images requires +relatively simple stacking of the input set of images. Outlier +rejection, both of complete input images as well as pixels within the +input stack, must be performed. In addition, each type of image +requires an appropriate normalization which may depend on the data +levels in other detectors in the input set. Each of these calibration +stages must be able to determine from the input stack if the relevant +calibration image needs to be updated and perform an initial test to +see which input images are consistent and valid. + +\subsection{Bias Images} + +Bias images may be needed to correct for structure in the bias. The +IPP must have the capability of constructing a master bias image from +a stack of raw bias frames. The input bias images, representing +offsets from the overscan level, are processed by subtracting the +overscan, including 1D structure if needed. + +The master bias frame construction uses outlier image and outlier +pixel rejection to construct a single high-quality bias frame. The +statistic used to determine pixel values from the input stack can be +set by the user to be one of the following: the sample mean, median, +and mode, robust mean, median, and mode, and the clipped mean and +median. Testing of the input images consists of constructing residual +images, in which the master bias is applied to the input images. +These images may be included or excluded from an additional iteration +of the stack on the basis of their pixel-to-pixel statistics. + +\subsection{Dark Images} + +Dark images may be needed to correct for structure in the dark +current. The IPP must have the capability of constructing a master +dark image from a stack of raw dark frames. The input dark images are +first corrected for the bias using whatever method is appropriate for +the science images. Master dark frames depend on their exposure time. +As such, the input dark frames must have a limited range of exposure +times, and the output dark frame includes the exposure time as part of +its associated metadata. + +The master dark frame construction uses outlier image and outlier +pixel rejection to construct a single high-quality dark frame. The +statistic used to determine pixel values from the input stack can be +set by the user to be one of the following: the sample mean, median, +and mode, robust mean, median, and mode, and the clipped mean and +median. Testing of the input images consists of constructing residual +images, in which the master dark image is applied to the input images. +These images may be included or excluded from an additional iteration +of the stack on the basis of their pixel-to-pixel statistics. A +collection of master dark frames with a range of exposure times are +used to determine the scaling of the dark frame as a function of +exposure time. + +\subsection{On-Off Dark Images for Light Leaks} + +A type of image which may be necessary for calibrations will be pairs +of images taken at night with the shutter closed with and without the +dome shutter closed. Such a pair of images can be used to determine +any light-leak in the camera which may contribute additional flux +across the mosaic. + +\subsection{Flat-Field Images} + +Master flat-field images must be constructed from a collection of +input flat-field images. The input flat-field images may be obtained +from any of the standard sources: the dome, the twilight sky, and the +night-time sky. The choice of flat-field input image must be +determined experimentally from observations during the commissioning +phase of the telescope. The IPP flat-field construction system must +be capable of handling any of these sources. + +An appropriate set of input images is selected on the basis of their +flux levels, time of observations, and the observing conditions. The +input flat-field images are processed (bias and dark corrected if +needed) and the resulting images are stacked. The master flat-field +construction uses image and pixel outlier rejection to construct a +single high-quality master flat-field frame. The statistic used to +determine pixel values from the input stack can be set by the user to +be one of the following: the sample mean, median, and mode, robust +mean, median, and mode, and the clipped mean and median. Testing of +the input images consists of constructing residual images, in which +the master flat-field image is applied to the input images. These +images may be included or excluded from an additional iteration of the +stack on the basis of their pixel-to-pixel statistics. + +\subsection{Mask Images} + +Preliminary bad-pixel mask images are generated on the basis of +comparison between raw flat-field images and a cleaned, stacked +master. The mask creation system accepts a collection of flat-field +images and identifies pixels which are consistently poorly flattened. +Pixels which are under-responsive are also identified as pixels to be +masked. + +\subsection{Sky \& Fringe Frames} + +Fringe-correction frames must be generated to remove the fringe +pattern caused by thin-film interference in the top layers of CCDs, +particularly in the redder passbands. Fringe correction frames may be +constructed on the basis of observations of the night-sky in the +appropriate filters or on the basis of dome fringe lamp observations. +The choice of the appropriate source will be determined experimentally +on the basis of data obtained during the commissioning phase. The IPP +must be capable of handing either source. The images are first +flattened to remove the pixel-to-pixel sensitivity variations of the +detector. The combination of multiple input fringe frames may not be +simply stacked since the amplitude of the fringe pattern varies +independently of other variations in the image. The amplitude of the +fringe pattern in the input frames is measured and the images scaled +to normalize the fringe amplitude to a consistent range (-1 to +1) for +all input images before they are combined with one of the standard +combination statistics (mean, median, mode, etc). The quality of the +input frames is tested by flattening the input image and applying the +master fringe-frame. The resulting residual image statistics are used +to select or exclude specific input images. + +\subsection{Shutter Correction Map} + +Shutter correction map images may be generated based on the timing +measurements of the shutter itself, or on the basis of dome-flat +images of decreasing exposure times down to the shortest available +exposures. + +\subsection{Low-k Sky Models} + +Large-scale background structure in images which is not caused by +thin-film interference must also be detected and corrected. Models of +this background structure may be a necessary input to the correction +proceedure. The IPP must have the capability of generating image +models of the large-scale structure patterns observed with the +telescope + +\subsection{Flat-Field Correction Frame} + +Flat-field images, whether constructed from the dome, twilight, or +night-sky images, do not perfectly correct the detector response in a +consistent fashion across the full field of the camera. The IPP must +have the capability of generating flat-field photometric correction +frames on the basis of the measured photometry of objects which are +moved to a variety of locations on the detector in a sequence of +images. The flat-field correction frames analysis stage makes use of +targetted observations following a specified dither pattern, and +extracts the photometered objects from the AP Database to determine +the necessary photometric corrections. The resulting image is applied +to the master flat-field image. Testing of the correction is +performed by applying the correction to the basic master flat-field +image, applying that flat-field image to the dithered photometry +observations, and performing the object detections. Comparion of the +photometry of individual stars at different locations on the mosaic +will demonstrate the consistency of the flat-field image. + +\subsection{Non-Linearity Correction} + +The IPP must have the capability of constructing a correction for +non-linearity in the detectors. These frames are constructed from +exposures of a uniform source with a range of exposure times. The +non-linearity correction frames provide polynomial correction +coefficients or a lookup table describing the correction. There is +likely to be a single non-linear correction for each OTA detector, or +potentially for each Cell. The IPP must handle these two cases. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\paragraph{Calibration Image Processing} - -The IPP Calibration Image Pipelines perform the tasks needed to -generate high-quality calibration images from the input image -dataset. These operations may be performed on whatever timescales are -appropriate and necessary to maintain the quality and relevance of the -calibration images. There are four distinct types of calibration -image pipelines: the basic detrend creation pipeline, the photometric -correction image creation pipeline, the fringe pattern generation -pipeline, and the sky foreground pattern generation pipeline. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Cal 1: Basic detrend image creation} - -The basic detrend image creation pipeline collects the appropriate -input detrend images (bias, dark, dome flat, etc) and generates a -master image by combining the input images in some optimal way -\tbd{median/sigma-clipping/etc}. The master image is used to -determine input image residuals so that poor input images can be -iteratively rejected. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Cal 2: Fringe pattern and sky foreground model creation} - -The fringe model creation and sky foreground model creation pipelines -use night-sky images with sufficient flux to measure the fringe or sky -models. The input images are processed and optimally combined to yield -a set of correction fringe patterns. The fringe pattern creation and -the sky foreground pattern creation have a similar processing -structure: both require processing of the input images, both determine -a set of principal components as a function of specific input -parameters. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Cal 3: Photometric flat correction image creation} - -The photometric flat-field correction uses images which have been -dithered with a large range of spatial scales, combined with the -uncorrected flat-field images, to generate a correction to the -flat-field image. This correction compenstates for non-uniform -illumination of the detector during the initial flat-field generation -stage. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Calibration Test Processing} - -The calibration test processing tests observations to determine if the -calibrations need updating. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{CalTest 1: Detrend frame testing} - -A newly-acquired master detrend frame, having been combined (using Cal -1 or Cal 2) are simply differenced from the old detrend frames. If -there exist significant residuals, the newly-acquired detrend frame -is adopted as the detrend frame of choice. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{CalTest 2: Photometric flat correction testing} - -Newly-acquired photometry of many objects (initially, this may be -standard star fields, but once the PS1 catalog is available, it should -be possible to use all photometry acquired over a given time period) -are compared with previously-acquired photometry. If there exist -significant residuals, a new photometric flat correction should be -produced from the newly-acquired photometry. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Reference Catalog Processing} +\section{System Design : Reference Catalog Processing} The IPP reference catalog pipelines use the data in the IPP Metadata @@ -1924,7 +2022,5 @@ astrometric and photometric calibration references. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{AstroRef: Astrometric Reference Catalog creation} +\subsection{AstroRef: Astrometric Reference Catalog creation} This processing stage shall use many observations over a given time @@ -1933,7 +2029,5 @@ published. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{PhotoRef: Photometric Reference Catalog creation} +\subsection{PhotoRef: Photometric Reference Catalog creation} This processing stage shall use many observations over a given time @@ -1943,43 +2037,24 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Design : Miscellaneous Tasks} + +In this section, we discuss the design of the science analysis stages +which perform the fundamental image analysis steps of the IPP. + +\subsection{Retrieval} + +The retrieval stages simply retrieve pixel data from an external +source (ordinarily OATS at the Summit, but it could conceivably be +some other external source) and store it on the nodes. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Reference Catalogs} - -The IPP will employ reference catalogs in order to calibrate the -photometry and astrometry. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Astrometric Reference Catalog} - -For PS1, this shall be UCAC. - -For PS4, this shall be the PS1 catalog. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Photometric Reference Catalog} - -For PS1, absolute photometry will not be available until the master -fit which will be performed when all data is taken. For purposes of -relative photometric extinction, the guide star brightnesses should be -sufficient. - -For PS4, the PS1 catalog shall be used. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Software Hierarchy} + +\section{Software Hierarchy} In order to facilitate testing and development, and to encourage flexibility, the IPP will be built in a layered fashion. The lowest level functions will be written in C and collected together into a -\PS{} library. These library functions will be used to write more +Pan-STARRS library. These library functions will be used to write more complex modules. The modules will be written in C but will make use of the SWIG tool to make their functionality available within other @@ -1997,27 +2072,27 @@ stringent. -\subsubsection{External Libraries} - -\PS{} will employ several external libraries to save duplicating +\subsection{External Libraries} + +Pan-STARRS will employ several external libraries to save duplicating functionality that is already available. These external libraries -will be wrapped by the \PS{} Library, insulating the project from the +will be wrapped by the Pan-STARRS Library, insulating the project from the implementation details of the external libraries. Examples of the external libraries are FFTW and SLALib. -\subsubsection{\PS{} Library} - -The \PS{} Library will consist of C structures describing the basic +\subsection{Pan-STARRS Library} + +The Pan-STARRS Library will consist of C structures describing the basic data types needed by the IPP and C functions which perform the basic data manipulation operations. Note that a subset of the library functions will be provided with SWIG interfaces as well to allow for their use in the creation of the processing stages. Examples of the -\PS{} Library are fourier transforms and transforming between pixel +Pan-STARRS Library are fourier transforms and transforming between pixel and celestial coordinates. -\subsubsection{Modules} +\subsection{Modules} The IPP analysis stages are broken down into modules which represent specific functional operations. The modules will be written in C -using the \PS{} Library functions and will be grouped into a \PS{} +using the Pan-STARRS Library functions and will be grouped into a Pan-STARRS Module Library. The modules will be provided with SWIG interfaces to all public APIs for their use in processing stages. Examples of @@ -2025,5 +2100,5 @@ (e.g.\ find objects on an image) will be used by multiple stages. -\subsubsection{Stages} +\subsection{Stages} The major IPP processing tasks are organized into stages, which @@ -2036,19 +2111,7 @@ images from multiple telescopes and search for transients). -\subsection{Modules} - -\tbd{What goes here? There will be modules?} - %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{\PS{} Library} - -See PSDC-430-007 for the design of the \PS{} Library, PSLib. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Interfaces} \subsection{Internal Interfaces} @@ -2076,8 +2139,4 @@ C:DB interactions -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \subsection{External Interfaces} @@ -2085,5 +2144,5 @@ This subsection describes the interfaces between the IPP and other -\PS{} systems and the external clients. The interfaces are +Pan-STARRS systems and the external clients. The interfaces are illustrated in Figure~\ref{fig:functionalities}. Incoming data is received by either the IPS (pixels), the IMD (metadata), or the IOD @@ -2092,8 +2151,7 @@ generated by the IPP Scheduler or the science processing pipelines. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{OATS} +\subsubsection{Camera} + +\subsubsection{OTIS} The Summit Pixel Server (SPS) sends raw image data, image metadata, @@ -2103,13 +2161,10 @@ to the IPS while the metadata is sent to the IMD. -The \PS{} Telescope Scheduler (PTS) sends information about the +The Pan-STARRS Telescope Scheduler (PTS) sends information about the telescope schedule to the IPP: observing plan for the night, or longer time scales. The IPP scheduler sends telescope schedule requests to the PTS (i.e.\ calibration needs). -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Published Static Sky Server} +\subsubsection{PSPS} The Static Image Server provides segments of the current static sky @@ -2118,18 +2173,5 @@ provides updated static sky images to the SIS when available. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Object Database} - -The Master Science Object Database receives new object photometry from -the IPP. The IPP IOD acts as a cache for object photometry data; -\tbd{an IPP subsystem will send photometry data in batches on some -timescale. Is this a function of the IOD?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Moving Object Processing System} +\subsubsection{MOPS} The Moving Object Processing System interfaces with the IPP to receive @@ -2137,8 +2179,5 @@ The MOPS may interface with the IMD as needed. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Other Client Science Pipelines} +\subsubsection{Other Preferred Client Science Pipelines} The client science pipelines may interface with the IPP via requests @@ -2147,153 +2186,52 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Computer Hardware} + +\subsection{PS-1 Cluster requirements} + + \begin{itemize} + \item CPU requirements + \item per-node I/O requirements + \item switch throughput requirements + \item storage profile + \end{itemize} + +\subsection{PS-1 Cluster Hardware Plan} + + \begin{itemize} + \item COTS equipment + \item number of processors needed + \item number of I/O ports needed + \item number of disk slots needed + \item switch choice + \item design choice for computer nodes + \item total rack space + \end{itemize} + +\subsection{PS-1 Cluster Expected Reliability} + +\subsection{PS-1 Cluster Support} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Computer Hardware} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Overview} - -This document discusses the likely range of the \PS{} Image -Processing Pipeline (IPP) hardware requirements. The hardware -requirements addressed in this document consist of: - -\begin{itemize} -\item Total Disk Volume -\item Total Processing Power -\item Sustained Switch Bandwidth -\item Sustained Node Network I/O -\item Sustained Disk I/O -\end{itemize} - -Even without the complete IPP design, it is possible to identify the -major drivers on the hardware requirements. The total disk volume -requirements are dominated by the need to store raw images for a -certain period, the need to store calibration images for a longer -period, and the need to store the static sky images. Of the various -analysis pipelines, and depending on the data organization as -discussed below, Phase 2 and Phase 4 present the most significant -demands in terms of data I/O throughput on the network. Phase 2 and -Phase 4 also present the most significant CPU demands. In this -discusion, Phase 2 refers to the per-chip pre-processing in which the -instrumental signature is removed and a first pass object detection is -performed. Phase 4 refers to the multiple chip combination in which -the pre-processed images are merged and combined, in both addition and -subtraction, with the static sky image, and up to three object -detection passes are performed. - -This document does not address the hardware requirements implied by -the Phase 0, 1, or 3 stages, nor the load required by the calibration -image creation stages. In the first instance, the operations are only -performed on the metadata and are extremely minimal both in terms of -data I/O and computation requirements. In the second case, the -processing is less time critical than the per-image processing and is -performed only infrequently (once per night to once per week or -month). This document also does not address any hardware requirements -introduced by the metadata manipulation. The software implementation -for metadata storage (RDBMS, FITS tables, etc) will have a very large -impact and will be evaluated along with the needed hardware at a later -date. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Scenarios} - -We will address the various hardware requirements by referring to a -set of data processing and data organization scenarios. The actual -hardware requirements will depend on design decisions which are not -yet available. It is possible to define the data organization in ways -which will minimize the hardware requirements, but which will increase -the software development effort. We will discuss both the worst-case -data organization scenario, which does not require significant -intelligence in the software systems, and the optimal data -organization scenario, which will require the software to track the -location of data products more carefully. In addition, this document -will address the data requirements of the complete \PS{} pipeline -with 4 telescopes as well as the single-telescope \PS{}-1 scenario -based on the Design Reference Mission [REF]. - -The IPP hardware system must provide both data storage and -computational resources. The IPP requires relativley large amounts of -data storage space, primarily for the image data. Image data is -organized in two categories. First, there is the per-chip data -- -data associated with specific chips, including the raw images, the -calibration images, and temporary processed images at various stages. -Second, there is the data associated with the static sky imagery, -which is in turn organized into smaller sky-cell units. The first -assumption we make is that the hardware is organized into nodes which -provide both data storage and computational resources. The second -assumption we make is that the data storage nodes are divided into two -classes: those which deal with the per-chip data and those that -provide the static sky storage. In addition, we assume that the -computational tasks related to Phase 2 take place on the per-chip -storage nodes and the Phase 4 computation takes place on the static -sky storage nodes. - -Figure~\ref{hardware} shows our basic concept for the hardware -organization for the IPP. This diagram shows the two types of compute -nodes: chip-level processing and storage nodes (dominated by Phase 2) -and static sky processing and storage nodes (mostly Phase 4). Also -shown are two switches used in this configuration; although it is -currently possible to buy a single switch which would have a -sufficient number of GigE ports for both sections of the PS-1 system, -such a two-switch organization may be needed for the full \PS{} -system. In such a case, the interswitch communication must also meet -the required throughput needs. We discuss the hardware requirements -in the assumption that such an organization will be necessary. - -The way in which the images are distributed among the storage and -compute nodes will largely determine the I/O bandwidth requirements. -For data bandwidth requirements calculations, it is necessary to make -some assumptions about the data organization. For the purposes of -this document, we explore two extreme-case options: -\begin{itemize} -\item Random Data Distribution --- Detector \& Sky data is randomly - distributed within the compute node of a given type (ie, chip data - is randomly distributed among the detector compute nodes). -\item Optimal Data Distribution --- Detector \& Sky data is optimally - distributed to compute Detector/Sky nodes (chip processing is always - on a machine with local chip data). -\end{itemize} -A second factor which will have a significant impact on the I/O -requirements is the image storage format for the processed and -calibration images. We have two basic choices: 32 bit floating point -format or 16 bit integer format with appropriate scaling. In the -former case, additional dynamic range is retained, while in the latter -case, we reduce the data volume by a factor of 2. While some may -argue that the higher dynamic range is necessary, arguments can be -made that the 16 bit range is sufficient. (In particular, the 16 bit -data provides a dynamic range far above the expected 1/1000 fractional -accuracy of the flat-field images). A related question is the number -of calibration images needed by the processing system. Since the -complete analysis is not yet defined, this number is difficult to -ascertain. However, we can make a range of assumptions which are -reasonable. We therefore adopt two data volume scenarios to explore -these possibilites: -\begin{itemize} -\item Standard Data Volume - 32 bit data for processed and calibration - images, average of 7 calibration frames per image. -\item Minimal Data Volume - 16 bit data for processed and calibration - images, average of 4 calibration frames per image. -\end{itemize} -In the discussion that follows, we explore the hardware requirements -implied by the collection of four combinations of these two sets of -scenario options. + +\clearpage + +\section{Appendices} + +\subsection{Image Server Database Table Contents} \begin{table} \begin{center} -\caption{Hardware Throughput Tests \label{existing-hardware}} -\begin{tabular}{lrrrr} -\hline -\hline -Test & where \& when & model & result \\ -\hline -node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ -node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ -RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ -Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ +\caption{Storage Object Table Contents\label{ImageServerTables:SO}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{so_id} & integer & internal storage object identifier \\ +\code{ext_id} & string & external storage object identifier (file ID) \\ +\code{comment} & string & user description of object \\ +\code{epoch} & date/time & last date of access \\ \hline \end{tabular} @@ -2301,36 +2239,19 @@ \end{table} -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Existing Hardware Throughput} - -We have collected a few representative tests of various pieces of -modern hardware to give a reference for the throughput capabilities. -A number of hardware configurations have been tested at CFHT for the -Elixir project, and we include here their recent reported hardware -RAID-5 I/O speeds and GigE card speeds. We also have included data -from VeriTest studies of Cisco switch throughput, commissioned by -Cisco for a 32 port GigE switch. These tests are summarized in -Table~\ref{existing-hardware}. - -\begin{table}[b] +\begin{table} \begin{center} -\caption{Data Storage Requirements \label{storage}} -\begin{tabular}{lrrrr} -\hline -\hline - & Standard / PS-4 - & Standard / PS-1 - & Minimal / PS-4 - & Minimal / PS-1 \\ -\hline -Raw data & 300 TB & 75 TB & 300 TB & 75 TB \\ -static sky & 512 TB & 64 TB & 256 TB & 32 TB \\ -calibration frames & 175 TB & 18 TB & 17 TB & 5 TB \\ -metadata db & 2 TB & 2 TB & 0.2 TB & 0.2 TB \\ -object db & 60 TB & 4 TB & 60 TB & 4 TB \\ -\hline -totals & 1050 TB & 163 TB & 633 TB & 116 TB \\ +\caption{Instance Table Contents\label{ImageServerTables:INT}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{ins_id} & integer & internal instance identifier \\ +\code{so_id} & integer & key to storage object table \\ +\code{uri} & string & location in hardware collection \\ +\code{sha1sum} & string & checksum information \\ +\code{assigned_location} & boolean & is location user-specified? \\ +\code{epoch} & date/time & last date of access \\ +\code{atime} & date/time & last date of access \\ \hline \end{tabular} @@ -2338,392 +2259,39 @@ \end{table} -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Data Storage Requirements} - -The \PS{} IPP data storage requirements may be divided into five -principal areas: raw image data, static sky image data, master -calibration images, the metadata database, and the object database. -We discuss each of these data items and their impact on the data -storage requirements for the IPP, and identify the impact of the -minimal vs standard data storage requirements as well as the -requirements specifically for PS-1. Table~\ref{storage} summarizes -the data storage requirements in the different scenarios. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Raw Data Storage} - -There are two basic image types which will be acquired: night-time -science images and calibration images. The night-time science images -consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, -the 4 telescopes can obtain images at a sustained rate of 1 image per -30 seconds per telescope for the entire night of 10 hours (36000 -minutes). A total of 100 calibration images per night would be a -substantial overestimate of the typical expectation. Combining these -numbers, we can expect to receive a total of 1300 image per telescope -per night, 5200 image total, or 10.4 TB of data per night. The total -data storage requirements for the raw data are governed by the number -of nights' worth of data we are required to keep online. A reasonable -number is one month to allow a full moon's cycle. Thus, for raw image -storage, we require a total of 300 TB data storage. For PS-1, this -number is simply scaled down by a factor of 4. The choice of the -minimal data volume does not affect these numbers because the raw data -is already stored with 16 bit pixels. - -\tbd{The PS-1 design reference may now require storage of the entire -first year of data, calculated to be 200 TB.} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Static Sky Data Storage} - -The static sky is represented by images with 0.2 arcsec per pixel. -There will be one summed image and one weight image for each of the 6 -filters, each stored in floating point format. At this resolution, -there are 324 Mpix per square degree, and we will observe a potential -total area of 30,000 square degrees. Allowing for 10\% overage for -overlapping tiling, we require a total of 10.7 Gpix to cover the sky -once, or a total of $\sim 512$ TB for the static sky images. In the -minimal data volume scenario, this value is reduced by a factor of 2, -while in PS-1, the reduction is a factor of roughly 8 because we only -intend to store the static sky for the ecliptic plane survey and the -small IPP verification program. - -\tbd{This last point is no longer valid - the PS-1 static sky may -require the entire 3pi.} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Calibration Frame Storage} - -The possible required calibration frames consist of the bias, dark, -and mask images, along with one flat, one flat-correction, and -multiple sky/fringe library frames per filter. In fact, not all types -are needed at all stages. For the standard data volume, we assume an -average of 7 calibration frames per image and filter. This results in -a total of 42 master calibration image per telescope. If we intend to -keep all master calibration frames for the project lifetime, and -generate a new master on a weekly basis (a reasonable time-scale), -then we can expect to require a total of 175 TB of calibration image -by the end of the 5 year lifetime of the project. For the case of -PS-1, the time period is only 2 years, and there is only 1 telescope, -resulting in a factor of 10 reduction in the volume. For the minimal -data case, we reduce the volume by another factor of 3.5. We also note -that this is likely to be a drastic overestimate as we are unlikely to -need to regenerate all master calibration frames on a weekly -time-scale. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Metadata Database Storage} - -The metadata data storage requirements are driven by the need to store -the data for the project lifetime. There are two types of metadata -generated at the summit: data associated with images and environmental -data. The environmental data consists of measurements on a regular -cadence, roughly 1 per minute, of a variety of parameters. We suggest -an expected of 1kB per entry, for a total of 2.6 GB over the lifetime -of the project. PS-1 will represent a smaller amount of data per -minute, and also a factor of 2.5 fewer minutes. We suggest PS-1 may -have a total environmental metadata set smaller by a factor of 5. The -additional systems, such as the DIMM, SkyProbe, NIR Sky Camera, and -the LRProbe will have higher data requirements, but should be -considered as separate, self-contained systems. Their data products -are distilled to a limited number of parameters per minute which are -included in the 1kB given above. Furthermore, items such as -guide-star history, if saved, will be saved with the image data and -represents only a small fraction of the total image data volume. Some -subset of the telescope diagnosic information may be a high volume -data product as well, but only retained by the telescope control -system for the purpose of diagnostic studies. Such data will be -excluded from this analysis. - -The image metadata consists of values associated with the FPA (4), the -chips (240), and the Cells (15360). Aside from the guide star -history, the total data requirements for each of these entries will be -scaled by the number of bytes required for the metadata from each data -level. Clearly, if the Cell entry is allowed to be large, it will -dominate the total Metadata data volume. If we suggest an expected -number of 64~bytes per Cell, 256~B per chips, and 1~kB per FPA, we find a -total metadata volume per exposure of roughly 1~MB, completely -dominated by the Cell metadata. With the exposure rates above, we -find a total of metadata volume of 1.8~TB over the lifetime of the -project. For PS-1, the total volume is reduced by a factor of 2.5 -(for the shorter lifetime) and another factor of 4 (for the lone -telescope). Neither data quantity is affected by the minimal vs -standard data volume choice. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object Database Storage} - -The hardware requirements for the IPP object database are rather -flexible: the total volume depends critically on the depth to which -the object detection analyses are performed (and thus the total number -of object detections) and the number of object parameters which are -measured. We can make very rough estimates that the total number of -detections over the 5 year lifetime of the project may be in the -vicinity of $5\times10^{11}$. We can conservatively estimate the -number of bytes needed to represent each detection as 128 B, resulting -in a total data storage for the object detections of 60 TB. However, -this number depends strongly on the timescale for which the IPP is -required to maintain all object detections, and may potentially be -significantly reduced. For the case of PS-1, the total number of -detections is likely to be reduced by a factor of 4 for the number of -telescopes, and potentially another significant factor ($\sim 4?$) by -limiting the depth of object detections. Again, the minimal data -volume scenario is irrelevant to the object database volume. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{CPU Requirements} - -Phase 2 and Phase 4 dominate the processing requirement, primarily -because they must keep up with the image delivery rate of 1 per 30 -seconds. We have performed benchmarks of a demonstration version for -both the Phase 2 and Phase 4 analyses. - -For the Phase 2, a substantial fraction of the processing time is -consumed by the need to perform FFTs on the images in order to -convolve them with the guide-star kernel, and in the smoothing used -for the object detection process. Additional processing time is -needed by the object detection, deblending, and analysis. Experiments -with the FFTW package show that FFTs may be performed on Intel -processors at rates of approximately 0.25~GHz-sec / Mpix for data sets -of order 1 Megapixel. The FFTs required for the Phase 2 analysis are -performed on the 512$^2$ pixel cells, so these numbers may roughly be -scaled linearly to determine the total time required for chip -processing. A single FFT on a full chip, with 64 cells, therefore -requires roughly 4~GHz-sec. For the full Phase 2 analysis, there are -roughly 4 single direction FFTs required excluding those associated -with object detection; thus the total processing time for these FFTs -is approximately 16~GHz-sec. The addtional analysis steps, excluding -object detection and characterization, account for a small fraction of -this compute time, which we estimate at 10\%. The object detection -stage depends somewhat on the depth to which the analysis is -performed, and the number of measurements made per object. Typical -analysis performed by the Sextractor routine, which performs a -substantial number of per-object analyses, requires 27~GHz-sec for a -full chip, including the FFTs used for smoothing. We can therefore -assume a total of 50~GHz-sec per chip for the Phase 2 processing. -This converts to a total of 12,000~GHz-sec for a complete major frame. - -For Phase 4, the main computational tasks are combining the multiple -images, with cosmic-ray rejection, and performing the object detection -tasks. Nick Kaiser has done tests of the Phase 4 image combine and -rejection stages, and finds a total processing time of roughly -96~GHz-sec for a full stack of 4 chips. If we add in an additional -34~GHz-sec for detailed object detection and image differencing, we -find a conservative estimage of 130~GHz-sec for a 4-image chip stack, -equivalent to 7800~GHz-sec for a major frame. - -For PS-1, the data processing will clearly require a smaller amount of -computational resources because of the lower image rate. However, the -total number of GHz-sec required for the complete analysis of 4 input -images and the combination with the static sky will remain -more-or-less the same. Some reduction in the load may be gained by -reducing the complexity and depth of analysis for PS-1. Depending on -the details and depth of the analysis, we may reduce the computational -load by a factor of 2. - \begin{table} \begin{center} -\caption{Data Scenarios (MB per Chip or Sky-cell) \label{scenarios}} -\begin{tabular}{lrrrr} -\hline -\hline - & Random / Standard & Random / Minimal & Optimal / Standard & Optimal / Minimal \\ -\hline -{\em Phase 2 input} & & & & \\ -from summit & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB \\ -input image & 32 MB & 32 MB & {\bf 32 MB} & {\bf 32 MB} \\ -calibration & $7 \times 64$ MB & $4 \times 32$ MB & {\bf 7 $\times$ 64 MB} & {\bf 4 $\times$ 32 MB} \\ -mask image & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ -\hline -network I/O: & 560 MB & 232 MB & 64 MB & 64 MB \\ -disk I/O: & (560 MB) & (232 MB) & 496 MB & 168 MB \\ - & & & & \\ -{\em Phase 2 output} & & & & \\ -output image & 64 MB & 32 MB & {\bf 64 MB} & {\bf 32 MB} \\ -output mask & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ -image to P4 & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB \\ -mask to P4 & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB \\ -\hline -network I/O: & 200 MB & 100 MB & 120 MB & 60 MB \\ -disk I/O: & (80 MB) & (40 MB) & 80 MB & 40 MB \\ - & & & & \\ -{\em Phase 4} & & & & \\ -input images & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & & \\ -input masks & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & & \\ -static sky & 64 MB & 64 MB & & \\ -static weight & 64 MB & 32 MB & & \\ -\hline -input: & 608 MB & 336 MB & & \\ -output: & 192 MB & 128 MB & & \\ -\hline -\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ -\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ +\caption{Volume Table Contents\label{ImageServerTables:VOL}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{vol_id} & integer & internal volume identifier \\ +\code{uri} & string & node name? \\ +\hline \end{tabular} \end{center} \end{table} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Per-Node I/O Requirements} - -Data I/O per node is defined as the number of bytes per second passed -through the node's network adapter. The data throughput for each node -depends strongly on the scenarios identified above. In this section, -we identify the data which is passed between nodes for each of the -different scenarios. Table~\ref{scenarios} lists the per-node data -I/O for the four scenarios. - -For PS-4, there are only 30 seconds of compute time allowed for each -of the Phase 2 and Phase 4 analyses. We use the data I/O volumes and -some assumptions about expected network and disk bandwidth to estimate -the I/O and processing timeline for the four scenarios. From this -analysis, we can judge the total CPU requirements in terms of GHz, not -just GHz-sec. We have assumed that GigE network adapters are capable -of delivering data at 50MB/sec sustained and that a disk RAID can -deliver sustained 100 MB/sec reads and writes. These numbers are -conservative estimates based on recent tests discussed above. Using -these assumptions, Table~\ref{throughput} lists the time allocations -for the complete set of scenarios for the case of PS-4. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Random / Standard Data Scenario} - -In the Random Data Distribution scenario, there is a single CPU -allocated to each chip in the detector farm and a single CPU for each Sky -cell process. The chip data are stored across random machines in the -detector farm, with the result that every Phase 2 processing requires -network access to the data. For each science chip which is -observed, each detector node will read from the network a total of 560 MB -(the 2 raw images for data storage and the 7 calibration frames, along -with one mask and one raw input image) and write a total of 200 MB -(one processed image and the mask along with the 1.5 processed images -and masks for the Phase 4 analysis). Given the assumption of 50 MB/s -from the network adapter, the total data volume implies an I/O period -of 15.2 seconds. Note that the disk I/O is parallel with the network -I/O and substantially underfills the disk bandwidth. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Random / Minimal Data Scenario} - -In the Random-Minimal, there is a single CPU allocated to each chip in -the detector farm and a single CPU for each Sky cell process, and the -chip data are stored across random machines in the detector farm. -However, the calibration and the processed science images are stored -at 2 bytes per pixel, the mask is set at 4 bits per pixel, and only 4 -calibration images are assumed. For each science chip which is -observed, each detector node will read from the network a total of 232 MB -(the 2 raw images for data storage and the 4 calibration frames, along -with one mask and one raw input image) and write a total of 100 MB -(one processed image and the mask along with the 1.5 processed images -for the Phase 4 analysis). Given the assumption of 50 MB/s from the -network adapter, the total data volume implies an I/O period of 6.6 -seconds. Again, note that the disk I/O is parallel with the network -I/O and substantially underfills the disk bandwidth. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Optimal / Standard Data Scenario} - -In the Optimal Data Distribution scenario, there is a single CPU -allocated to each chip in the detector farm and a single CPU for each -Sky cell process. In addition, all data for the specified chip are -stored on local disks attached to the same computer as the CPU, with -the result that all Phase 2 I/O is made to a local disk. For each -science chip which is observed, each detector node will read from the -network a total of 2 raw images (one for the original image, one for -the backup copy) and write an average of roughly 1.5 processed images -and masks to the Phase 4 machines for a total of 184 MB of network -I/O. During the processing stage, the detector node will read from -disk a total of 496 MB (7 calibration frames at 64 MB each, one 16 MB -mask, and one raw science image at 32 MB) and write a total of 80 MB -(one processed image at 64 MB and one mask at 8 MB). Given the -assumptions for the network and disk bandwidths (50 MB/s and 100 MB/s -respectively), the data volumes imply a total I/O period of 9.5 -seconds. In this instance, the network I/O is presumed to be -sequential with the disk I/O. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Optimal / Minimal Data Scenario} - -In the Optimal / Minimal Scenario, the minimal data sizes are used -with the optimal data distribution scheme. In this case, we reduce -the disk I/O volume to 168 read and 40 MB write, and the network -traffic to 124 MB. Given the assumptions for the network and disk -bandwidths, the data volumes imply a total I/O period of 4.6 seconds. -Again, the network I/O is presumed to be sequential with the disk I/O. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 4 Node I/O Requirements / Standard Data Volume} - -Although it is easy to arrange the detector data in such a way that -the majority of I/O is performed locally, it is not as easy to arrange -this for the Static Sky data used by the Phase 4 analysis. We -therefore make the assumption that the Phase 4 analysis will require -all input detector data to be loaded across the network, as well as -all Static Sky data. This is somewhat of an overestimate as some of -the Static Sky data will be processed by machines with the data stored -locally, and clever Static-Sky data organization schemes can enhance -this chance. - -In the Phase 4 analysis, the images from the 4 separate telescopes are -combined into a single image, confronted with the appropriate segment -of the static sky, with output difference image and updated static sky -image. If we restrict input access to the individual chip cells, the -maximum read overhead is 50\% (need to read a 10x10 set of cells for -an 8x8 input image). If the processing is performed on Static Sky -segments equivalent in size to the chips, the input data is 608 MB (384 -MB of processed science image, 96 MB of mask images, 64 MB of static -sky image and 64 MB of static sky weight map) while the output data is -192 MB (static sky, weight map, and difference image, each 64 MB). -Thus, we require a total of 800 MB network I/O. Given the network -bandwidth, this implies an I/O period of 16 seconds for Phase 4. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 4 Node I/O Requirements / Minimal Data Volume} - -In the minimal data volume scenario, the Phase 4 analysis volume is -significantly reduced. The total volume of input data is 336 MB (192 -MB of processed science image, 48 MB of input mask, 64 MB of static -sky image and 32 MB of static sky weight map) while the output data is -128 MB (64 MB static sky, 32 MB weight map, and 32 MB difference -image). Thus, we require a total of 464 MB network I/O, which implies -an I/O period of 9.3 seconds. +\clearpage + +\subsection{Metadata Database Table Contents} + +Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the +Metadata Database tables listed in Section~\ref{Metadata}. \begin{table} \begin{center} -\caption{Data Throughput for 4 Scenarios \label{throughput}} -\begin{tabular}{lrrrr} -\hline -\hline -& -\multicolumn{1}{c}{Random / Standard} & -\multicolumn{1}{c}{Random / Minimal} & -\multicolumn{1}{c}{Optimal / Standard} & -\multicolumn{1}{c}{Optimal / Minimal} \\ -\hline -Phase 2 per-node network I/O & 15.2 s & 6.6 s & 3.7 s & 2.5 s \\ -Phase 2 per-node disk I/O (read) & (5.6 s) & (2.3 s) & 5.0 s & 1.7 s \\ -Phase 2 per-node disk I/O (write) & (0.8 s) & (0.4 s) & 0.8 s & 0.4 s \\ -Phase 2 CPU total & 14 s : 860 GHz & 23 s : 520 GHz & 20 s : 600 GHz & 25 s : 480 GHz \\ -Phase 4 per-node I/O & 16 s & 9.3 s & & \\ -Phase 4 CPU total & 14 s : 490 GHz & 20 s : 390 GHz & & \\ -Phase 2 switch load & 6.1 GB/s & 2.7 GB/s & 1.5 GB/s & 1.0 GB/s \\ -Phase 4 switch load & 0.8 GB/s & 0.5 GB/s & 0.8 GB/s & 0.5 GB/s \\ -Phase 2 to Phase 4 switch load & 1.1 GB/s & 0.6 GB/s & 1.1 GB/s & 0.6 GB/s \\ -Summit to Phase 2 switch load & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s \\ +\caption{Weather Table: some sample weather points\label{WeatherTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the weather information was measured. \\ +Temperature 01 & float & The external temperature \\ +Temperature 02 & float & The temperature at top of the dome \\ +Temperature 03 & float & The temperature on the primary mirror \\ +Humidity & float & The relative humidity. \\ +Pressure & float & The (external) atmospheric pressure. \\ \hline \end{tabular} @@ -2731,95 +2299,19 @@ \end{table} -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Switch I/O Requirements} - -The switch I/O requirements are defined by the total number of bytes -per second serviced by the two switches in the system. For the -analysis of the Switch I/O requirements, the choice of data -distribution again has a major impact. We again test the four -scenarios discussed above: Random Data Distribution, Random / Minimal, -Optimal Data Distribution, and Optimal / Minimal. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Random / Standard Data Scenario} - -In the Random Data Distribution scenario, each detector node needs to -read a total of 560 MB from the network and write a total of 200 MB -every 30 seconds. With 240 detector nodes, this corresponds to a -total bandwidth of 6080 MB/sec, or 49 Gb/sec. Note that this includes -the bandwidth needed to copy data from the summit and make two copies -on the detector machines, as well as the bandwidth to send the processed -image portions to the Phase 4 machines. The Phase 4 processing adds -an additional 320 MB of network I/O per Sky-Cell group, and there are -roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing -adds an additional 750 MB/sec network bandwidth. In the architecture -defined in Figure \tbd{NN}, the Sky nodes and the detector nodes are each -attached to separate switches. An additional bandwidth requirement is -derived by the need to exchange data between these switches in for -Phase 4. The total amount of data exchanged between these switches is -480 MB per Sky-cell, for a total bandwidth of 1120 MB/sec. In -addition, the connection to the summit is a single, separate line -which needs to support the bandwidth requirement of copying all intial -raw images. In our simple model, each raw image is copied twice, -accounting for a total of 15360 MB every 30 seconds, or a bandwidth -load of 512 MB/sec. (Note that this last is double the actual -bandwidth requirement to the summit: a dedicated local circular buffer -would reduce the need for the second copy to come directly from the -summit.) - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Random / Minimal Data Scenario} - -In the Random / Minimal Scenario, the data volumes are significantly -reduced. The total Phase 2 bandwidth contribution is 332 MB over 30 -seconds for 240 nodes (2656 MB/sec) and the residual Phase 4 bandwidth -load is 224 MB per Sky cell over 30 seconds (522 MB/sec). The -inter-switch communication is now 240 MB per sky cell over 30 seconds, -or 560 MB/sec. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Optimal / Standard Data Scenario} - -In the Optimal Data Distribution, the Phase 2 network bandwidth is -reduced significantly to 184 MB per detector node, for a total of -1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at -750 MB/sec. The inter-switch communication also remains the same at -1.12 GB/sec. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Optimal / Minimal Data Scenario} - -In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth -drops to 124 MB per detector node, for a total of 1.0GB/sec, while the -Phase 4 network bandwidth is 552 MB/sec. The inter-switch -communication remains the same as the Random/Minimal Scenario at 560 -MB/sec. - -\begin{table}[t] +\begin{table} \begin{center} -\caption{\label{NP2} Phase 2 load per major frame (12000 GHz-sec)} -\begin{tabular}{lrrrr} -\hline -\hline -& Random/Standard -& Random/Minimal -& Optimal/Standard -& Optimal/Minimal \\ -\hline -network I/O (GB) & 182 & 80 & 44 & 30 \\ -PS-1 & & & & \\ - I/O (cpu-sec) & 3640 & 1600 & 880 & 600 \\ - CPU (cpu-sec) & 4000 & 4000 & 4000 & 4000 \\ - \# cpus & 64 & 47 & 41 & 38 \\ -PS-4 & & & & \\ - I/O (cpu-sec) & 1820 & 800 & 440 & 300 \\ - CPU (cpu-sec) & 2000 & 2000 & 2000 & 2000 \\ - \# cpus & 127 & 93 & 81 & 77 \\ +\caption{SkyProbe Transparency Table (sample entries)\label{SkyprobeBVTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the SkyProbe image was taken. \\ +Filter & string & Filter used for SkyProbe image. \\ +Transparency & float & The derived transparency. \\ +Number of stars & int & The number of stars used to measure the transparency. \\ +Astrometry & coords & The astrometry used on the SkyProbe image. \\ +Exposure time & float & The exposure time of the SkyProbe image. \\ +Sky brightness & float & The measured sky (surface) brightness, counts / second \\ \hline \end{tabular} @@ -2827,22 +2319,22 @@ \end{table} -\begin{table}[b] +\begin{table} \begin{center} -\caption{\label{NP4} Phase 4 load per major frame (7800 GHz-sec)} -\begin{tabular}{lrr} -\hline -\hline -& Standard -& Minimal \\ -\hline -network I/O (GB) & 48 & 28 \\ -PS-1 & & \\ - I/O (cpu-sec) & 960 & 557 \\ - CPU (cpu-sec) & 2600 & 2600 \\ - \# cpus & 30 & 26 \\ -PS-4 & & \\ - I/O (cpu-sec) & 480 & 278 \\ - CPU (cpu-sec) & 1300 & 1300 \\ - \# cpus & 59 & 53 \\ +\caption{Skyprobe Line Absorption Table (sample entries)\label{SkyprobeATable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the LRProbe observation was taken. \\ +Disperser ID & string & ID of the dispersing element \\ +Atm Component 1 & float & The strength of the 1st atmospheric component. \\ +Atm Component 2 & float & The strength of the 2nd atmospheric component. \\ +Atm Component 3 & float & The strength of the 3rd atmospheric component. \\ +Disperser ID & string & ID of the dispersing element \\ +Number of stars & int & Number of stars used to measure the absorptions. \\ +Astrometry & coords & The astrometry used on the LRProbe image. \\ +Exposure time & float & The exposure time of the LRProbe image. \\ +Sky brightness & float & The measured sky (surface) brightness, in physical units. \\ \hline \end{tabular} @@ -2850,53 +2342,461 @@ \end{table} +\begin{table} +\begin{center} +\caption{Skyprobe Line Emission Table (sample entries)\label{SkyprobeETable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the LRProbe observation was taken. \\ +Disperser ID & string & ID of the dispersing element \\ +Atm Component 1 & float & The strength of the 1st atmospheric component. \\ +Atm Component 2 & float & The strength of the 2nd atmospheric component. \\ +Atm Component 3 & float & The strength of the 3rd atmospheric component. \\ +Continuum & float & The strength of the continuum emission. \\ +Disperser ID & string & ID of the dispersing element \\ +Exposure time & float & The exposure time of the LRProbe image. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{DIMM Measurements Table\label{DimmTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the DIMM observation was taken. \\ +$\sigma_x$ & float & Raw dispersion in $x$. \\ +$\sigma_y$ & float & Raw dispersion in $y$. \\ +FWHM & float & Dervied seeing full width at half maximum. \\ +RA & float & The coordinates of the measured star. \\ +DEC & float & The coordinates of the measured star. \\ +Exposure time & float & The exposure time of the DIMM observation. \\ +Telescope ID & string & source of the DIMM data \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Near IR Wide-field Camera Results Table\label{NIR-Table}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the NIR observation was taken. \\ +Sky brightness & float & The sky (surface) brightness in the NIR observation. \\ +Sky variance & float & The variance in the sky (surface) brightness. \\ +Astrometry & coords & The astrometry used on the NIR image. \\ +FOV X & float & field width \\ +FOV Y & float & field height \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Dome Status Table\label{DomeStatusTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time for which the dome status is valid. \\ +Azimuth & float & The azimuth of the dome. \\ +Open status & boolean & Whether the dome is open or not. \\ +Lights status & boolean & Whether lights are on in the dome or not. \\ +Track status & boolean & Whether dome is tracking telescope or not. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Telescope Status\label{TelescopeStatusTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time for which the telescope status is valid. \\ +Guide status & enum & The status of the guiding. \\ +Altitude & float & The telescope altitude. \\ +Azimuth & float & The telescope azimuth. \\ +RA & float & The telescope Right Ascension (ICRS $\approx$ J2000). \\ +Dec & float & The telescope Declination (ICRS $\approx$ J2000).\\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Raw FPA Images\label{RawFPAs}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +ID & string & FPA image ID \\ +RA & float & Coordinates of the boresight (i.e. telescope pointing). \\ +DEC & float & Coordinates of the boresight (i.e. telescope pointing). \\ +Filter & string & Filter used for the exposure. \\ +Image Type & enum & image exposure type \\ +Exposure time & float & Exposure time for the image. \\ +Airmass & float & Airmass at which the image was taken. \\ +ObsFrame ID & int & Observation frame identification number, ties FPAs into major frame \\ +ObsGroup ID & int & Observation group identification number, ties FPAs into observing group \\ +Observer & string & The name of the observer, or the version of the telescope scheduler software. \\ +Program & string & The observing program being executed. \\ +Nchips readout & int & Number of detector chips read out \\ +Camera & string & Identification of camera source \\ +Telescope & string & Telescope used for observation \\ +Astrometry & coords & The astrometry used for the FPA. \\ +Chip Metadata & string & metadata resource file \\ +Cell Metadata & string & metadata resource file \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Pending Science Chips\label{PendingChips}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +FPA ID & string & FPA image ID \\ +Chip ID & string & Chip identification number. \\ +Proc Status & enum & Current Processing Status. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Processed Science Chips\label{ProcessedChips}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +FPA ID & string & FPA Image ID \\ +Chip ID & string & Chip identification number. \\ +Status & enum & Current Processing Status. \\ +Residual Stats & float & quality statistics. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Observation Group Information\label{OBS}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +ObsGroup ID & string & Identification number for the observation group. \\ +Number of images & string & Number of images in the observation group. \\ +Type & string & Type of observation. \\ +Status & string & Status of the observation group. \\ +\tbd{etc} & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Observation Frame Information\label{OBS}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +ObsFrame ID & string & Identification number for the observation frame. \\ +Number of images & string & Number of images in the observation group. \\ +Type & string & Type of observation. \\ +Status & string & Status of the observation group. \\ +\tbd{etc} & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Science Processing Stats\label{PSStats}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Chip ID & string & The chip identification number. \\ +State & string & The state of the processing. \\ +ObsFrame ID & string & The major frame the chip belongs to. \\ +ObsGroup ID & string & The observation group the chip belongs to. \\ +P1 astrom & string & The Phase 1 astrometry results file. \\ +P2 astrom & string & The Phase 2 astrometry results file. \\ +P3 astrom & string & The Phase 3 astrometry results file. \\ +N guide stars & string & Number of guide stars used for the exposure. \\ +Astrometry stats & string & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\ +Astrom catalog & string & The reference catalog that was used for the astrometry. \\ +Bias method & string & Method used to correct the bias. \\ +Bias stats & string & Summary statistics for bias \\ +Flat-field image & string & The flat-field image that was applied. \\ +Kernel data & & A description of the OT kernel. \\ +Flat-field stats & & Summary statistics for flat-field (sigma of sky). \\ +Mask image & string & The mask image that was applied. \\ +Mask method & string & The algorithm used to mask the bad pixels. \\ +Fringe images & string & The fringe model images that were used. \\ +Fringe stats & & Summary statistics for fringes (fringe amplitude, sky sigma) \\ +Object stats & & Summary statistics for object detection (number of objects, depth, other input parameters). \\ +Photometry data & & photometry information: magnitude zero point and other corrections. \\ +Photometry stats & & Summary statistics for the photometry (number of stars, $sigma_m$) \\ +Photom catalog & string & The reference catalog that was used for the photometry. \\ +PSF stats & & Summary statistics of the PSF. \\ +Software ver & string & Versions of each of the modules used in the processing. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Chip / Sky overlaps\label{overlaps}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Chip ID & string & The identification number of the chip. \\ +Sky Cell ID & string & The identification number of the sky cell. \\ +State & string & Processing state of overlap \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Processed Sky-Cell stats\label{}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Input Chips & string & Identification numbers of the chips used to produce the sky cell. \\ +PSF adjustments & string & \tbd{Adjustments to the PSF.} \\ +CR rejection stats & string & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ +Image comb params & string & Parameters used for the image combination. \\ +Diff image params & string & Parameters used for the image differencing. \\ +Average weight & string & The weight of the reference image \\ +P4D object stats & string & Summary statistics of the object detection (number of objects, depth, other input parameters). \\ +P4S object stats & string & Summary statistics of the object detection (number of objects, depth, other input parameters). \\ +Software versions & string & Software versions of modules used in the sky cell processing. \\ +Processing stats & string & Summary statistics of the processing (CPU time, etc). \\ +\hline +\end{tabular} +\end{center} +\end{table} +\clearpage %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Conclusions} - -Table~\ref{throughput} presents one way of analysing the hardware -requirements, making a specific set of assumptions about the number of -nodes for the two phases and the expected network and disk -bandwidths. The important conclusion in this analysis is the implied -number of GHz per processor, given the assumptions laid out. -Phase 2 is specified to have 240 detector nodes, while Phase 4 is specified -to have roughly 60 static sky nodes. The range of Phase 2 CPU -requirements implies that each CPU needs to have speeds in the range -of 2.0 - 3.6 GHz, which sound very plausible for the year 2007, since -these apply to PS-4. - -Another way to represent this information is to use the total number -of MB I/O and the total number of GHz-sec required for the two stages, -confront these with an assumption for the bandwidth per network -adapter and an assumption for the CPU speed and use those numbers to -calculate the minimum number of nodes (CPUs) needed to sustain the -timing requirements. There are quite a few parameters and options to -choose from. We have assumed that for PS-1, the time between major -frames (4 images combined in Phase 4) is 120 seconds, and 30 seconds -for PS-4. We have also assumed that each CPU has one network adapter -associated with it, and use the numbers of 50 MB/sec for PS-1 era -network adapters and 100 MB/sec for the PS-4 network adapters (since -there has been some steady improvement in GigE hardware over the past -year). We have also assumed each PS-1 CPU is rated at 3 GHz and those -for PS-4 are rated at 6 GHz (somewhat conservative since 3 GHz -machines are already available). Tables~\ref{NP2} and \ref{NP4} show -the load and resulting number of nodes for both Phase 2 and Phase 4 -for both the PS-1 and PS-4 assumptions, using the I/O numbers for all -of the scenarios above. Note that in these discussions, we make the -idealized assumption that the computational and I/O portions of each -process are completely serial. As a result, the CPU is completely -used to perform the I/O during the I/O phase, avoiding any concern -about I/O load on the processor during analysis. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Software Runtime Configuration Issues} + +The IPP Software requires extensive runtime configuration information. +This includes default parameters for analysis to be performed, +descriptions of how a particular analysis is performed, locations of +data sources, and so forth. The IPP may store this information in the +Metadata Database or in configuration files available to the user. +Both methods are implemented in the current design. In either method, +the necessary parameters are identical. In this section, we discuss +the contents of specific portions of the runtime configuration. + +\subsubsection{Camera Definition Information} + +Every camera which may be analysed by the IPP has differences in how +the data is represented. The IPP is built with the flexibility to +handle data from many different cameras, not just the Pan-STARRS +Gigapix cameras. This is partly to allow testing of the analysis +system on data from other telescopes, such as MegaPrime on CFHT and +Suprime on Subaru, but also to allow us to adapt to changes in the +design of the Gigapix cameras themselves. It also means the IPP +software may be used by astronomers for other analysis projects beyond +the IPP. + +Most cameras provide extensive descriptive information in the FITS +image headers when the images are read out. Typically, the location +and orientations of the individual detectors are defined by keywords +such as DATASEC and DETSEC. Other variations on these words are used +for cameras which place the pixels from multiple amplifiers in the +same FITS data segment. Other parameters, such as astrometric +information or exposure times, are stored in headers as well. It is +possible to use these header keywords to guide the analysis software, +but there are two difficulties. + +First, it is very common for different keywords to be used by +different cameras, sometimes even the same camera may use different +keywords for the same information at different times (major readout +software upgrades, for example, can be accompanied by keyword +revisions). In addition, within Pan-STARRS and the IPP, we would like +the capability to refer to the Metadata database as the authoratative +sources of some of these entries rather than the image headers. Given +this circumstance, it is at least necessary to define the appropriate +source for a given data concept appropriate to data from a specific +camera. + +The second problem arises when actually performing an analysis. In +many circumstances, the software needs to know what data to expect +even when an appropriate camera image is not available. This is +particularly true for a camera which is composed of multiple chips and +multiple amplifiers. It is a frequent circumstance than some subset +of the chips or amplifiers will either be unavailable or are invalid +for one reason or another. It is important for the software to have a +guide for what data should be available from a perfect readout of the +given camera so decisions can be made how to handle data which is not +complete. This is also important to validate that a particular +dataset, which appears to be from a known camera, actually corresponds +to that camera and has all of the necessary information where +expected. + +In order to facilitate the operation of the IPP with a variety of +cameras, and to allow the software the flexibility to change the +camera defintion dynamically, we define a collection of software +runtime configuration information which defines a given camera. This +information is represented below in the form of the PSLib Metadata +Config file, but may be stored in the Metadata Database or in an +alternate format as appropriate. + +We start by noting that the a single camera is represented as a Focal +Plane Array (FPA), divided into Chips, divided into Cells. For a +single FPA, all imaging data is stored in a FITS file or a collection +of FITS files. Software needs to know where in a given file or set of +files to find a particular Cell, what Cells to expect, what chips to +expect, and the relationships between those entities, etc. + +A single camera configuration file (or dataset) represents the +description of a complete FPA. In the configuration file, any +parameters which are specific to the complete FPA are placed on their +own lines. These include the definition of the keywords or database +locations. An incomplete example is given below. + +\begin{verbatim} +NCELL S32 NN +NCHIP S32 NN +EXPTIME-SRC STR HD:EXPTIME # need to specify PHU vs EXTNAME? +EXPTIME-KEY STR EXPTIME +DATE-KEY STR DATE-OBS +DATE-FMT STR YYYY/MM/DD + +TYPE CELL FILENAME EXTNAME CHIP DATASEC BIASSEC +CELL.nn CELL @ROOT@CELL AMP00 CHIP.00 CF:[0,0:0,0] HD:BIASSEC +CELL.01 CELL @ID/@ID@CELL.fits AMP01 CHIP.00 DB:??? +\end{verbatim} + +\subsubsection{Analysis Recipe Information} + +In order to maintain flexibility in the analysis details, the IPP uses +recipes to define how a particular analysis is implemented. Each +major analysis script (eg, Phase 2) has its own recipe configuration +information, which may be stored in the Metadata Database or in the +form of the PSLib Metadata Config file. This configuration +information includes all of the user configurable parameters. Many of +these may specify a specific value, or they may specify lookup methods +(database locations, or header locations). The specifies of each +depends on the context. Below, we provide an example recipe file for +the bias subtraction portion of Phase 2, giving several alternative +options for certain entries. Note that, for example, the overscan +subtraction may be specified as using a particular region given in the +recipe file, or on the basis of a particular header keyword. + +\begin{verbatim} +# BIAS: +BIAS.IMAGE STR NONE +BIAS.IMAGE STR FILE:bias.fits +BIAS.IMAGE STR DB:BEST +BIAS.IMAGE STR DB:CLOSE + +BIAS.OVERSCAN STR HD:BIASSEC +BIAS.OVERSCAN STR CF:[0,16:0,2048] +BIAS.OVERSCAN STR NONE + +BIAS.OVERSCAN.STATS STR MEDIAN +BIAS.OVERSCAN.STATS STR MEAN + +BIAS.OVERSCAN.FIT STR SPLINE +BIAS.OVERSCAN.FIT.NPTS S32 5 + +BIAS.OVERSCAN.FIT STR POLYNOMIAL +BIAS.OVERSCAN.FIT.ORDER S32 3 +BIAS.OVERSCAN.FIT.NBIN S32 5 +\end{verbatim} + +\subsection{I/O Code Autogeneration} + +Within IPP, we have a number of data collections which have multiple +representations. We define a tool to automatically generate code to +provide I/O APIs to read and write these data and data structures to +carry them within program. Within the IPP, we will use database +tables (ie, in the Metadata Database), FITS Tables (to exchange bulk +data), and XML (to exchange more complete datasets). + +I/O API Autocode template (example.def): +\begin{verbatim} +Name Example +Table EXAMPLE +EXTNAME EXAMPLE + +KEY XVALUE + +# name format unit comment +XVALUE F32 pixels "x coordinate" +BINNING S32 fraction "binning factor" +NAME STR[32] string "description of entry" +\end{verbatim} + +Running autocode on such a file would generate an output header and C +files \code{example.h, example.c} with the following structure and APIs: + +\begin{verbatim} +typedef struct { + psF32 XVALUE; // x coordinate + psS32 BINNING; // binning factor + char NAME[32]; // description of entry +} Example; + +psMetadata *psFITSTableInitExample (); +psExample *psFITSTableLoadExample (char *filename, int *Nrows); +bool psFITSTableSaveExample (char *filename); + +psMetadata *psDatabaseTableInitExample (); +psExample *psDatabaseTableLoadExample (char *filename, int *Nrows); +bool psDatabaseTableSaveExample (char *filename); +psExample *psDatabaseTableLoadExampleRow (char *filename, psF32 XVALUE); +\end{verbatim} + +\bibliographystyle{plain} +\bibliography{panstarrs} + +\end{document} + + \section{Notes} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Cell vs Chip vs FPA vs Major Frame} @@ -2913,5 +2813,5 @@ possibilities: -\begin{enumerate} +\begin{itemize} \item exposures in a major frame are always synchronized; the telescopes are required to take exposures in a coordinated fashion and @@ -2936,12 +2836,8 @@ coincident) than a major frame in which the offsets are larger in either dimension. -\end{enumerate} +\end{itemize} A decisions between these possibilities will drive some requirements either on the IPP side or on the PTS/TCS side. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Identifying ghosts, spikes, etc} @@ -2957,8 +2853,4 @@ addition of data. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \subsection{Pending Sky-cell / Detector table} @@ -2967,188 +2859,2 @@ initiate phase 4. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Appendices} - -\subsection{Image Server Database Tables} - -\begin{table} -\begin{center} -\caption{Storage Object Table Contents\label{ImageServerTables:SO}} -\begin{tabular}{ll} -\hline -\hline -{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ -\hline -\code{so_id} & integer & internal storage object identifier \\ -\code{ext_id} & string & external storage object identifier (file ID) \\ -\code{comment} & string & user description of object \\ -\code{epoch} & time/date & last date of access \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table} -\begin{center} -\caption{Instance Table Contents\label{ImageServerTables:INT}} -\begin{tabular}{ll} -\hline -\hline -{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ -\hline -\code{ins_id} & integer & internal instance identifier \\ -\code{so_id} & integer & key to storage object table \\ -\code{uri} & string & location in hardware collection \\ -\code{sha1sum} & string & checksum information \\ -\code{assigned_location} & boolean & is location user-specified? \\ -\code{epoch} & time/date & last date of access \\ -\code{atime} & time/date & last date of access \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table} -\begin{center} -\caption{Volume Table Contents\label{ImageServerTables:VOL}} -\begin{tabular}{ll} -\hline -\hline -{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ -\hline -\code{vol_id} & integer & internal volume identifier \\ -\code{uri} & string & node name? \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\bibliographystyle{plain} -\bibliography{panstarrs} -\end{document} - -%%%%%% Phase 0 has been dropped: identifying the moving objects is not needed - -\paragraph{Phase 0 : night preparation} - -Phase 0 is the night preparation phase of the IPP analysis system. -There may be potentially many pieces of information which apply to the -processing for an entire night and which take substantial time to -calculate. these are pre-calculated by the phase 0 stage and stored -in a database table for reference by other stages of the processing -system. Currently, the only quantity calculated by Phase 0 is the -collection of known moving object ephemerids. - -At various stages in the IPP analysis, it is necessary to know the -location of known moving objects (main belt asteroids, comets, -Kuiper-belt objects, any other classes of asteroids) in relation to -specific images obtained. If moving object orbits were trivial to -calculate, or if the number was limited, this would be a simple -problem of three dimensional intersections. However, complete orbits -are not trivial and there may be tens of thousands to millions of -possible objects of interest. To simplify the task, it is possible to -reduce the parameter space of the search by pre-calculating the orbit -segments of all objects for a given night and saving fiducial points -of the orbit in a database table. Later systems which require the -position of objects in a specific image can use linear interpolation -between these fiducial points to identify the likely objects, and -potentially additional non-linear orbital calculations to refine the -positions. - -The database table of object fiducial positions must include the -following information: - -\begin{itemize} -\item object ID -\item epoch -\item RA at epoch -\item DEC at epoch -\item dRA at epoch -\item dDEC at epoch -\item R magnitude? -\item date of calculation? -\item lifetime? -\end{itemize} - -The input for this calculation is the table of known moving objects -and their orbital elements, and the time range for the calculation. -If the calculation is slow, Phase 0 could be paralellized by object. -If Phase 0 is fast enough (\tbd{minutes?}), the process need not be -parallel. The {\tt lifetime} and {\tt date of calculation} allow old -Phase 0 entries to be removed when they are not needed. \tbd{This -cleaning phase could be a function of Phase 0.} Phase 0 need not be -run only for the current night. Any time a specific set of data is to -be analysed by the later stages, phase 0 should be run for the -appropriate time period. \tbd{Does there need to be a database table -with phase 0 runs and time periods defined? this could be the -reference used by later phases to decide if phase 0 has been run. they -could also trigger the phase 0 run if they notice it has not been run -(a job of the scheduler).} - -\tbd{what is the orbit calculation speed? does it scale with Npts? -what is the number of known objects now? in 5 years?} - - - -%%% phase 2 metadata -\milsection{Metadata} - -The following metadata associated with the images are required for -Phase~2 operation: -\begin{itemize} -\item The orthogonal transfer (OT) image shifts made during the -exposure --- in order to create a convolution kernel; -\item Time of observation --- for selecting the appropriate detrend -images; -\item Filter --- for selecting the appropriate detrend images; -\item Telescope identification --- for selecting the appropriate -detrend images; -\item Exposure time --- for the photometric calibration; -\item Detector gain --- for calculating photometric errors and -determining the quality of the overscan; -\item Detector read noise --- for calculating photometric errors and -determining the quality of the overscan; -\end{itemize} - -\milsection{Pixel Masks} -\label{ap:masks} - -This section describes the requirements on Bad Pixel Masks (BPMs). -These will consist in of bit masks for each pixel. For Phase 2, flags -are required for at least each of the following pixel attributes: -\begin{enumerate} -\item The pixel is a charge trap; -\item The pixel is a bad column; -\item The pixel is saturated in the A/D converter; -\item The pixel is non-positive in the flat-field; -\item The pixel is part of a row that has excess noise; and -\item The pixel is determined to be a cosmic ray, based on its -morphology. -\end{enumerate} - -Of these, only masks for the charge traps need to be grown by the -extent of the OT convolution kernel. For other pixel types, -orthogonal transfer of the flux in this pixel will not (necessarily) -affect the flux in neighbouring pixels - -\milsection{Object Catalogs} -\label{ap:catalogs} - -Object catalogs from Phase 2 shall consist of at least the -following elements for each object: -\begin{enumerate} -\item Object centre, with corresponding errors; -\item Object magnitude, with corresponding error; -\item Object isophotal magnitude, with corresponding error; -\item Object FWHM; -\item Object elliptical axis lengths; and -\item Object position angle for ellipse. -\end{enumerate} - -Though further details may be required for catalogs in Phase~4, -the above details are minimum requirements for Phase~2 catalogs. -