Index: trunk/doc/design/ippSSDD.tex =================================================================== --- trunk/doc/design/ippSSDD.tex (revision 6049) +++ trunk/doc/design/ippSSDD.tex (revision 6054) @@ -1,3 +1,3 @@ -%%% $Id: ippSSDD.tex,v 1.4 2006-01-19 03:51:41 eugene Exp $ +%%% $Id: ippSSDD.tex,v 1.5 2006-01-19 06:49:50 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -137,8 +137,9 @@ PSDC-430-012 & Pan-STARRS IPP Modules Supplementary Design Requirements Specification \\ \hline PSDC-430-014 & Pan-STARRS IPP PS-1 Cluster Support \\ \hline -\tbd{add the other subsystem SDDs} \DocumentsExternalSection Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ \DocumentsEnd + +\tbd{add the other subsystem SDDs} \section{Subsystem Overview} @@ -605,7 +606,7 @@ names. -\subsubsubsection{House keeping} - -\paragraph {Lock sweeping} In the event that a Storage Object operation fails to complete successfully +\paragraph{House keeping} + +\subparagraph{Lock sweeping} In the event that a Storage Object operation fails to complete successfully stale locks will have to be identified and removed from the IPP Pixel Data Server Database. This should be done periodically by comparing @@ -622,5 +623,5 @@ table. -\paragraph{Consistency sweeping} Periodically the IPP Pixel Data Server meta-data and Storage Object will need +\subparagraph{Consistency sweeping} Periodically the IPP Pixel Data Server meta-data and Storage Object will need to be checked for sanity. This would be similar to running fsck on a modern filesystem. Consistency sweeping should include Lock sweeping @@ -2031,4 +2032,7 @@ Phase 2. +\subsection{Summary of the Phase 1-3 Data Products} + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -2042,14 +2046,29 @@ sky, the sky cell, along with the associated pixels from a collection of images which have been processed through phases 1--3. The size and -exact representation of a static sky cell are yet to be determined. -The working concept is that the static sky cells contain roughly the -same number of pixels as an OTA (4k x 4k) and represent a portion of a -local tangent plane projection. In order to meet the image -degradation requirements, the pixel scale of the static sky is planned -to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw image -pixel scale. - -For each sky cell, the corresponding pixels are extracted from the -exposures being processed and mapped to the projection of the sky +exact representation of a static sky cell are configurable parameters +which may vary between surveys. In some cases, it may be feasible to +use a single large rectangular tangent plane image for the static sky +and accept the distortion near the edges. In other cases, the static +sky cells may be more appropriate defined to represent a relatively +small patch over which the distortion is known to be small. The +details of the static sky cell definition should not be a driver for +the Phase 4 analysis. In order to meet the image degradation +requirements, the pixel scale of the static sky is planned to be +0.2\arcsec, somewhat smaller than the 0.25\arcsec\ raw image pixel +scale; the choice of this scale must be determined with some caution. +If the pixels are too large, the degredation will be excessive; too +small, and the total volume of the static sky image data will be +excessively costly. + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/phase4}} +\caption{ \label{fig:phase2} Phase 2 dataflow - this diagram is old: update} +\end{center} +\end{figure} + +In the basic concept of the Phase 4 analysis, each sky cell is +examined independently. The corresponding pixels are extracted from +the exposures being processed and mapped to the projection of the sky cell. The pixels from the multiple input processed images are combined into a single, cleaned image. Outlier pixels may be optionally @@ -2061,5 +2080,6 @@ The remaining pixels are added to the existing static sky image. Object detection must be performed on the difference ($P4\Delta$) and -cleaned ($P4\Sigma$) images. +cleaned ($P4\Sigma$) images. This process is illustrated graphically +in Figure~\ref{BasicP4} \subsubsection{Image Warping and Matching} @@ -2154,10 +2174,111 @@ a time when the computing infrastructure is not under significant load. -%\begin{figure} -%\begin{center} -%\resizebox{6in}{!}{\includegraphics{pics/phase4}} -%\caption{ \label{fig:phase4} Phase 4 dataflow} -%\end{center} -%\end{figure} +\subsubsection{Magic and Phase 4 Modifications} + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/phase4a}} +\caption{ \label{fig:phase4a} Phase 4a} +\end{center} +\end{figure} + +The Pan-STARRS relationship with the U.S. Air Force has some +implications for the data processing which place some interesting +constraints on the IPP implementation. The U.S. Air Force is funding +the construction of PS-1, and we are thus subject to restrictions +under which the Air Force must operate telescopes. The Air Force has +diplomatic and security concerns about publically releasing images in +which artificial satelites are detected, particularly if those +satelites are identified. Historically, Air Force projects have been +restricted from releasing images with identified satelites, +considering the data to be of a sensitive nature. + +The language which governs such Air Force projects is in the process +of being modified so that images with identifable satelites can be +treated under the classification of 'For Official Use Only'. Even +under such an arrangement, however, the Air Force requires that the +satelites which appear in the PS-1 images be excised in such a way +that their orbits cannot be reliably identified. + +Satelite detected in the Pan-STARRS images will appear as long ($> +100$ pixels) streaks. The natural, and most effective, way to +identify such streaks is to search for them in the images after they +have been processed using the difference image processing in Phase 4. +A team from Boeing has been contracted to develop a software module to +idenfity streaks in these images. They delivered the initial release +of their software, which is currently called {\em Magic}, in the +beginning of January 2006. Further effort on that software will be +required to confront it with the real PS-1 image parameters. + +In order to mesh the operation of {\em Magic} with the IPP Phase 4 +analysis for PS-1, some modifications must be made to the operation +sequence. In the case of PS-1, unlike PS-4, the basic set of images +which confronted with the Static Sky image are obtained in sequence, +not essentially simultaneously. This is important for Magic because +the satelite streak in one image will not appear in the other three. +Thus the Magic operation must be performed on intermediate difference +images for individual exposures. To facilitate this, the basic Phase +4 described above is divided into three stages: Phase 4a, in which the +individual difference frames are constructed; Magic; and Phase 4b, in +which the results of Magic are used to filter the images before the +final stack and difference images are generated. + +Figure~\ref{phase4a} illustrates the operation of Phase 4a and Magic. +The individual warped image (P4$_W$) is constructed and differenced +against the corresponding Static Sky image. In this analysis, poisub, +which performs the kernel matching and difference image construction, +is run with somewhat looser constraints on the image difference +kernel. The lower-accuracy difference kernel will make the difference +image noisier in the vicinity of galaxies and bright stars. Unlike +the science transients, however, most of the pixels involved in a +satelite streak are not in close proximity to another objects; for SNe +and GRBs, the objects are preferentially located near host galaxies +where it is most critical to have a high-quality subtraction. +Allowing poisub a more relaxed image difference kernel results in a +substantial improvement in the convolution speed. The resulting +difference image is called P4$\Delta$'. + +{\em Magic} searches for streakes in the difference images by using a +heirarchy of Huff transforms. A single run of {\em Magic} expects to +have access to all of the P4$\Delta$' images from a single FPA image. +It starts with independent Huff transforms of the individual cells, +which can be performed in parallel under the PanTasks paralleliztion +scheme. The results of these Huff transforms are then merged to yield +the equivalent of a larger Huff-transformed image. The heirarchy +searches for significant streaks from the individual cells, the +equivalent 2x2 cell sets, then 2x2 of those, etc, until the entire +array has been searched. Significant peaks detected from the Huff +transforms are then used to restrict the pixels in the image space. +These pixels are examined and streaks detected by requiring a certain +fraction of the pixels along the purported streak to be lit. +Filtering using the cross-streak PSF can also be applied to enhance +the detection and minimize the false positives. Any streaks which are +detected are then excisized. In order to obfuscate the idenification +of the satelite, this step masks the streak pixels, with a box +somewhat wider than the streak (10-20 pixel wide), displaced by an +unknown, random amount relative to the streak center-line. This strip +is extended to the ends of the FPA array. Analysis of the expected +density of streaks by the Boeing team, using known satelites as the +input to the simulation, shows that we should expect to lose less than +1\% of the pixels due to these long excised regions. + +The result of the Magic process is a set of masks for each raw PS-1 +image. Any images which are released beyond the IPP cluster must have +these masks applied, or with manual inspection for visible streaks. +\tbd{time-stamps? ok to release without the valid time? or only with } + +The final portion of the analysis, Phase 4b, is illustrated in +Figure~\ref{phase4b}. In this stage, the intermediate result images +from Phase 4a can be used. The P4$_W$ images are masked with the +corresponding masks from Magic. These images are then stacked as +normal, and the resulting summed image is differenced with poisub, +using a more stringent limit on the image difference kernel. + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/phase4b}} +\caption{ \label{fig:phase4b} Phase 4b} +\end{center} +\end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -2748,4 +2869,5 @@ \item {\bf output stage} as a stand-alone program, psphot will produce output tables in various formats. +\end{itemize} \subsection{psastro} @@ -2773,5 +2895,5 @@ and 3. -In the conceptualy most straightforeward mode, each readout contained +In the conceptually most straightforeward mode, each readout contained in the incoming FPA structure is treated independently. The metadata describing the approximate astrometry of the readout are used to guess @@ -2811,5 +2933,40 @@ basic description of the astrometry is the collection of header keywords which define the boresite center coordinates (RA, DEC), the -rotation and platescales +location of the chip data arrays within the full mosaic (the IRAF +DATASEC keywords). More detailed astrometric information may be +defined using the WCS keywords. These unfortunately do not have a +standard representation of higher-order terms. Furthermore, the two +competing systems which have been proposed define only a single +transformation frame. In the IPP, it is important to carry around +more information which can be used to improve our astrometric +solutions in the future. Specifically, we would like to maintain at +least the transformation for the telescope optics independent from the +individual chip warps or tilts. Even more, we would like to have a +flexible astrometry definition format which can be extended in a +flexible fashion. We have defined a FITS table convention to carry +all of the elements of the astrometric transformation of a full FPA. +Within the table, transformations are generally defined to convert one +layer (eg, focal plane) to another. The form and the parameters of +the conversion make up the columns of the table. With this structure, +it is possible to add arbitrary layers as needed. The IPP, and +portions of the project (particularly Otis), will share a common +default astrometry model for the telescope. This will be defined on +the basis of measurements over the first weeks of observations. The +IPP will perform the astrometric analysis of individual images with +psastro, and the results can be saved in this tabular format. Over +time, these result tables can be used to improve the astrometric model +for the telescope, and to improve the astrometric reference catalog. + +\subsection{poisub} + +Poisub is the image difference analysis program. \tbd{Paul: please + flesh this out!}. + +\subsection{stac} + +STAC is the program which warps and optimally combines images from the +same region of the sky. It consists of two major stages: the warping +stage and the image combination stage with robust outlier rejection. +\tbd{Paul: flesh this out!} \section{Interfaces} @@ -3817,7 +3974,7 @@ ------ -* poisub / stack -* re-org the Phase 4 stuff to discuss Magic -* astrometry calibration data formats +* output data products +* DVO +* PanTasks +* ipptools / ippMonitor? * analysis stages, versions and iterations -* output data products