Index: /trunk/doc/modules/configProposal.tex =================================================================== --- /trunk/doc/modules/configProposal.tex (revision 1144) +++ /trunk/doc/modules/configProposal.tex (revision 1144) @@ -0,0 +1,466 @@ +%%% $Id: configProposal.tex,v 1.1 2004-06-30 09:43:35 price Exp $ +\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{Pan-STARRS IPP Image Hierachy} +\subtitle{Configuration File Proposal} +\author{Paul Price} +\shorttitle{Image Hierachy Configurations} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{DR} +\docnumber{PSDC-430-???} +% note the use of the docnumber & version number: +% the complete PSDC document number is given by +% \thedocnumber-\theversion + +\setlength{\topsep}{-2pt} + +\begin{document} +\maketitle +\sloppy + +% -- Revision History -- +% provide explicit values for the old versions +% use '\theversion' for the current version (set above) +% use \hline between each table row +\RevisionsStart +% version Date Description +DR & 2004 Jun 29 & Draft \\ \hline +\RevisionsEnd + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\DocumentsInternal +PSCD-130-001 & PS-1 Design Reference Mission \\ \hline +PSCD-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline +PSCD-430-005 & Pan-STARRS IPP SRS \\ \hline +PSCD-430-006 & Pan-STARRS IPP ADD \\ \hline +PSCD-430-008 & Pan-STARRS IPP Architecture SDR \\ +\DocumentsExternal +Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ +\DocumentsEnd + +\tableofcontents +\pagebreak +\pagenumbering{arabic} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Introduction} + +Although the \PS{} Image Processing Pipeline (IPP) is required +primarily to support \PS{} operations, it is important to test it +thoroughly beforehand on data from other sources. The great +heterogeneity of the format of FITS files used in the astronomical +community means that we require a configuration file to specify how +images, both from other sources and from \PS{}, are converted into the +internal representation of the IPP. It is the object of this document +to propose a format for this configuration file. + +This document, it should be noted, is a preliminary proposal, and +hence the format has not yet been specified in the excruciating detail +required for a file format. If this proposal is perceived to be of +some use, it shall be expanded upon in the near future. + +\section{Requirements} + +It is desirable to associate a single name with an entire focal plane +image. Some observatories write images from the focal plane to +multiple FITS files. Others write to multiple extensions within the +same FITS file. + +The naming of particular information within FITS headers is extremely +heterogeneous. For example, the exposure time is variably stored as +\code{EXPTIME} and \code{EXPOSURE} throughout different observatories. +Furthermore, not all observatories store particular information, for +example, the hour angle of the observation, so in some cases, these +will have to be calculated on the basis of other information. Even +further, some values may not be able to be derived from information in +the FITS headers at all, but need external information, e.g., the +orientation of the focal plane elements on the sky. + +In some cases, the extension names/numbers may not be known \emph{a +priori}, as in the case of \PS{}, where a file may contain readouts +from multiple cells. Nevertheless, it should be possible to +reconstruct the focal plane on the basis of information recorded in +the headers. + +\section{Proposal} + +I propose the following format, which I believe satisfies the above +requirements. + +\subsection{File} + +The file shall be human-readable, plain text. + +\subsection{Comments} + +For the ease of parsing, we specify that only lines beginning with a +hash mark (\#) are to be ignored as comment lines. A comment may not +start in the course of a line. + +\subsection{Key/Value} + +Options are specified as key/value pairs. In some cases (e.g., the +specification of bias sections), there may be several ``values'' to +specify a single value, listed one after the other, delimited by +whitespace. + +\subsection{Hierarchy} + +The configuration file may be split into each of the hierarchical +levels: FPA, Chip, Cell and Readout. Options may be specified at any +of the levels. Options specified at one level flow down to all lower +levels, unless explicitly overridden. Some options are additive as +they flow down (for example, the offset of an element is cumulative as +one descends down the hierarchy, since the chip is offset so far from +the centre of the focal plane, while a cell is offset with respect to +the chip). + +The hierarchical levels may each be specified in turn, from the +highest level (FPA) to the lowest level (Readout), followed by a +commma-delimited list of allowed values for the name of an element of +the hierarchical level (or an asterisk if any value is permitted). In +options that follow, the name is denoted by the hierarchical level +name in angled brackets: \code{}, \code{}, \code{}, +\code{}. The name shall only be available at a hierarchical +level equal or lower than the corresponding level. + +For example: + +\begin{verbatim} +FPA * +Chip chip00,chip01,chip10,chip11 +Cell 1,2 +Readout * +\end{verbatim} + +In this example, the FPA may have any name, the chip name must be one +of \code{chip00}, \code{chip01}, \code{chip10} or \code{chip11}, the +cell name must be one of ``1'' or ``2'', and the readout may also have +any name. In this case, this likely represents a focal plane +consisting of four pieces of silicon, each with two amplifiers. + +Of course, the above example does not specify any options, which we +shall discuss next. + +\subsection{Options} + +By options, we mean the association of a particular value or FITS +header with a concept that is used in the IPP. It is the options that +provide the translation between the FITS headers and the concepts used +by the IPP, such as airmass, right ascension, exposure time, etc. + +An option is specified at the appropriate hierarchical level, indented +by a single tab (for clarity in reading by humans), and consists of a +keyword, followed by a value or FITS header from which the value may +be read. + +Below we note the options required for the IPP, along with the option +keyword: +\begin{itemize} +\item Right Ascension of the observation or of the hierarchical level; + \code{ra}. +\item Declination of the observation or of the hierarchical level; + \code{dec}. +\item Equinox for the coordinates; \code{equinox}. +\item Rotation of the observation or of the hierarchical level; + \code{rotation}. +\item Handedness of hierarchical level; \code{flip} --- either $+1$ or + $-1$. +\item Pixel scale of hierarchical level; \code{scale}. +\item Filename to retrieve the hierarchical level from; + \code{filename}. +\item FITS extension name from which to retrieve the hierarchical + level; \code{extname}. +\item FITS extension number from which to retrieve the hierarchical + level; \code{extnum}. +\item Bias regions; \code{bias} --- there may be more than one, + comma-delimited, and each specified in the format + \code{[x1:x2,y1:y2]}. +\item Axis to use for overscan subtraction; \code{axis} --- may be one + of \code{x}, \code{y}, \code{rows} or \code{columns}. +\item Detector regions to crop after overscan subtraction; + \code{detector}. +\item RA offset for the hierarchical level from the level(s) above; + \code{deltara}. +\item Dec offset for the hierarchical level from the level(s) above; + \code{deltadec}. +\item Rotation offset for the hierarchical level from the level(s) above; + \code{deltarot}. +\item Difference in handedness for the hierarchical level from the + level(s) above; \code{deltaflip}. +\item Multiplication of scale for the hierarchical level from the + level(s) above; \code{deltascale}. +\item Exposure time; \code{exptime}. +\item Airmass; \code{airmass}. +\item Hour angle; \code{ha}. +\item Sidereal time; \code{st}. +\item Latitude of observatory; \code{obslat}. +\item Longitude of observatory; \code{obslong}. +\item Altitude of observatory; \code{obsalt}. +\item Date of exposure; \code{date}. +\item Time of exposure; \code{time}. +\item Time reference; \code{timeref} --- may be \code{start}, + \code{stop} or \code{mid}, to specify when the observation time is + referenced to. +\item Format of date, to specify how to convert; \code{dateformat}. +\item Format of time, to specify how to convert; \code{timeformat}. +\item Extension containing OT shifts; \code{shiftsext}. +\item File containing OT shifts; \code{shiftsfile}. +\item Filter; \code{filter} --- multiple FITS headers may be + specified, comma-delimited, in order to account for instruments + which have multiple filter wheels. +\item Object/field name; \code{name}. +\item Observation type; \code{obstype}. +\item CCD dark time; \code{darktime}. +\item CCD gain; \code{gain}. +\item CCD read noise; \code{readnoise}. +\item Binning in x; \code{xbin}. +\item Binning in y; \code{ybin}. +\item Telescope altitude; \code{altitude}. +\item Telescope azimuth; \code{azimuth}. +\end{itemize} + +Note that the \code{deltara}, \code{deltadec}, \code{deltarot} and +\code{deltaflip} options should be propagated down the hierarchical +levels, while for the other options, the option defined last applies. + +Some unavailable options (e.g., not all observatories record the +sidereal time) may be calculated from other available options after +parsing. However, once this has been attempted, if any required +option is missing, and if the option cannot default to a reasonable +value, then the read should fail. + +A value specified for an option may include the name of an +hierarchical level in angled brackets (i.e., \code{}, +\code{}, \code{}, \code{}), which shall be +replaced iteratively by each of the allowed names, in turn, of the +hierarchical level. + +\subsubsection{Extensions not known a priori} + +In some cases, the extension name or number may not be known \emph{a +priori}. In particular, OTA data or NIR data from some instruments +may include multiple reads of a cell simply included in the same MEF +file along with other cells. + +In this case, there is no choice for the program but to create a table +of contents of the FITS file specified by scanning the extensions, and +associating different extensions with the appropriate hierarchical +level on the basis of a particular header. We specify the special +option, \code{id}, the corresponding value for which indicates which +header may be examined to indicate the name of the instance of a +hierarchical level. See the PS-GPC example for a demonstration. + +The \code{id} option may only be specified when the file name has +already been specified unambiguously at higher hierarchical level(s). + +Once it is known which extension corresponds to which +FPA/chip/cell/readout combination, then the focal plane may easily be +reconstructed. + +\subsection{Dependent options} + +In cases where the option is not available in the FITS header, the +value must be specified manually in the configuration file. However, +often the value is dependent upon the particular instance of an +hierarchical level. For such cases, we specify a syntax similar to +a \code{switch} statement in C. + +When an option is followed by a colon, then this means that it is +dependent upon the particular hierarchical level that is specified in +angled brackets following the colon. For example: +\begin{verbatim} + gain: +\end{verbatim} +specifies that the \code{gain} option is dependent upon the chip. + +A dependent option is followed on subsequent lines by the possible +values for the name of the hierarchical level, indented by two tabs +(for readability by humans) and the corresponding value for the +option. For example: +\begin{verbatim} +chip 00,01,10,11 + gain: + 00 1.0 + 01 2.1 + 10 3.2 + 11 4.3 + readnoise: + 00 5 + 01 6 + 10 7 + 11 8 + exptime EXPOSURE +\end{verbatim} + +This specifies that chip 00 has gain 1.0 and read noise 5, chip 01 has +gain 2.1 and read noise 6, chip 10 has gain 3.2 and read noise 7, and +chip 11 has gain 4.3 and read noise 8, and all chips have an exposure +time specified by the \code{EXPOSURE} FITS header. + +\section{Examples} + +I here offer some practical examples to help decipher these (no doubt, +cryptic) instructions. Not all options are specified, but hopefully +sufficient to give an idea of the format. + +\subsection{CFHT 12k MEF format} + +Some CFHT 12k images are written in MEF FITS format, with each of the +chips as separate extensions in a single FITS file for each exposure. + +\begin{verbatim} +FPA * + file .fits + ra RA + dec DEC + exptime OPENTIME +Chip chip1,chip2,chip3,chip4,chip5,chip6,chip7,chip8,chip9,chip10,chip11,chip12 + extnum + dra: + chip1 -110 + chip2 -90 + chip3 -70 + chip4 -50 + chip5 -30 + chip6 -10 + chip7 10 + chip8 30 + chip9 50 + chip10 70 + chip11 90 + chip12 110 + ddec: + chip1 -20 + chip2 -20 + chip3 -20 + chip4 -20 + chip5 -20 + chip6 -20 + chip7 20 + chip8 20 + chip9 20 + chip10 20 + chip11 20 + chip12 20 +Cell * +Readout * + bias [1:100,1:4096],[2150:2248,1:4096] + axis x + detector [101:2148,1:4096] +\end{verbatim} + +This specifies, firstly, that any name may be used to specify an FPA +image (I suspect that this is usually the case). Filenames are +specified on the FPA level (since all the chips are in the same FITS +file). If the FPA name given to the IPP is \code{img12345}, then the +corresponding filename that the IPP searches for is +\code{img12345.fits}. + +The Right Ascension, Declination and exposure time are specified for +the FPA, and read from the \code{RA}, \code{DEC} and \code{OPENTIME} +headers, respectively. These headers are to be found in the Primary +Header Unit (PHU), since an extension has not been specified. + +The allowed chip names are \code{chip1}, \code{chip2}, \code{chip3}, +\code{chip4}, \code{chip5}, \code{chip6}, \code{chip7}, \code{chip8}, +\code{chip9}, \code{chip10}, \code{chip11} and \code{chip12}. A chip +is to be found in the extension of the same name (as specified by the +\code{}. + +The RA and Dec offsets are specified, such that \code{chip1} is the +lower-left hand chip and \code{chip12} is the upper-right hand chip on +the focal plane. + +All Cell and Readout names are valid. Two bias regions are specified +for each chip, namely a prescan region, from column 1 to 100, and an +overscan region, from column 2150 to 2248. The bias region is to be +collapsed along the x axis, and following overscan subtraction, only +columns 101 to 2148 are of any interest. + + +\subsection{PS GPC} + +We expect that a \PS{} GigaPixel Camera exposure will be written as +several MEF files, one OTA to each file, with an extension within that +file for each cell and each readout (since there may be multiple +readouts per cell to incorporate fast guiding). To specify all 64 +OTAs and 64 cells in each would make for a long example, so we only +show 4 OTAs and 4 cells in each. + +This is a case in which the extensions are not known \emph{a priori}, +since there may exist multiple readouts for one cell in the same MEF +file as other cells. Note the use of the \code{id} option to specify +this. + +\begin{verbatim} +FPA * +Chip ota00,ota01,ota10,ota11 + file ..fits + ra RA + dec DEC + exptime EXPOSURE + dra: + ota00 -20 + ota01 20 + ota10 -20 + ota11 20 + ddec: + ota00 -20 + ota01 -20 + ota10 20 + ota11 20 +Cell cell00,cell01,cell10,cell11 + id CELLNUM + dra: + cell00 -10 + cell01 10 + cell10 -10 + cell11 10 + ddec: + cell00 -10 + cell01 -10 + cell10 10 + cell11 10 + shiftsext ot +Readout * + id READNUM + bias BIASSEC + axis x + detector DETSEC +\end{verbatim} + +Note first of all that all FPA names are valid (as is usually the +case). Valid chip names are \code{ota00}, \code{ota01}, \code{ota10} +and \code{ota11}. If the FPA name is \code{img12345}, then the +filename for \code{ota01} is \code{img12345.ota01.fits}. The RA, Dec +and exposure time are derived from the \code{RA}, \code{DEC} and +\code{EXPOSURE} headers, respectively. The chips are laid out in a +$2\times 2$ mosaic. + +Now we come to the cells. The allowed cell names are \code{cell00}, +\code{cell01}, \code{cell10} and \code{cell11}. However, the +extension these cells reside in is not known \emph{a priori}, and so +the \code{id} option specifies that the program should create a table +of contents of the extensions in the file, and determine the chip name +from the \code{CELLNUM} header in each extension. The cells are laid +out in a $2\times 2$ pattern on the chip. The OT shifts made for the +cell may be found in the extension of name \code{ot}. + +Finally, the readouts. Again, the particular readout is not known +\emph{a priori}, but the particular readout may be deduced from the +\code{READNUM} header. The bias region is specified by the +\code{BIASSEC} header, the bias region is to be collapsed along the x +axis, and the region to be cropped following the overscan subtraction +is specified by the \code{DETSEC} header. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\bibliographystyle{plain} +\bibliography{panstarrs} + +\end{document}