Index: unk/doc/modules/configProposal.tex =================================================================== --- /trunk/doc/modules/configProposal.tex (revision 9619) +++ (revision ) @@ -1,466 +1,0 @@ -%%% $Id: configProposal.tex,v 1.1 2004-06-30 09:43:35 price Exp $ -\documentclass[panstarrs]{panstarrs} - -% basic document variables -\title{Pan-STARRS IPP Image Hierachy} -\subtitle{Configuration File Proposal} -\author{Paul Price} -\shorttitle{Image Hierachy Configurations} -\group{Pan-STARRS Algorithm Group} -\project{Pan-STARRS Image Processing Pipeline} -\organization{Institute for Astronomy} -\version{DR} -\docnumber{PSDC-430-???} -% note the use of the docnumber & version number: -% the complete PSDC document number is given by -% \thedocnumber-\theversion - -\setlength{\topsep}{-2pt} - -\begin{document} -\maketitle -\sloppy - -% -- Revision History -- -% provide explicit values for the old versions -% use '\theversion' for the current version (set above) -% use \hline between each table row -\RevisionsStart -% version Date Description -DR & 2004 Jun 29 & Draft \\ \hline -\RevisionsEnd - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\DocumentsInternal -PSCD-130-001 & PS-1 Design Reference Mission \\ \hline -PSCD-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline -PSCD-430-005 & Pan-STARRS IPP SRS \\ \hline -PSCD-430-006 & Pan-STARRS IPP ADD \\ \hline -PSCD-430-008 & Pan-STARRS IPP Architecture SDR \\ -\DocumentsExternal -Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ -\DocumentsEnd - -\tableofcontents -\pagebreak -\pagenumbering{arabic} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Introduction} - -Although the \PS{} Image Processing Pipeline (IPP) is required -primarily to support \PS{} operations, it is important to test it -thoroughly beforehand on data from other sources. The great -heterogeneity of the format of FITS files used in the astronomical -community means that we require a configuration file to specify how -images, both from other sources and from \PS{}, are converted into the -internal representation of the IPP. It is the object of this document -to propose a format for this configuration file. - -This document, it should be noted, is a preliminary proposal, and -hence the format has not yet been specified in the excruciating detail -required for a file format. If this proposal is perceived to be of -some use, it shall be expanded upon in the near future. - -\section{Requirements} - -It is desirable to associate a single name with an entire focal plane -image. Some observatories write images from the focal plane to -multiple FITS files. Others write to multiple extensions within the -same FITS file. - -The naming of particular information within FITS headers is extremely -heterogeneous. For example, the exposure time is variably stored as -\code{EXPTIME} and \code{EXPOSURE} throughout different observatories. -Furthermore, not all observatories store particular information, for -example, the hour angle of the observation, so in some cases, these -will have to be calculated on the basis of other information. Even -further, some values may not be able to be derived from information in -the FITS headers at all, but need external information, e.g., the -orientation of the focal plane elements on the sky. - -In some cases, the extension names/numbers may not be known \emph{a -priori}, as in the case of \PS{}, where a file may contain readouts -from multiple cells. Nevertheless, it should be possible to -reconstruct the focal plane on the basis of information recorded in -the headers. - -\section{Proposal} - -I propose the following format, which I believe satisfies the above -requirements. - -\subsection{File} - -The file shall be human-readable, plain text. - -\subsection{Comments} - -For the ease of parsing, we specify that only lines beginning with a -hash mark (\#) are to be ignored as comment lines. A comment may not -start in the course of a line. - -\subsection{Key/Value} - -Options are specified as key/value pairs. In some cases (e.g., the -specification of bias sections), there may be several ``values'' to -specify a single value, listed one after the other, delimited by -whitespace. - -\subsection{Hierarchy} - -The configuration file may be split into each of the hierarchical -levels: FPA, Chip, Cell and Readout. Options may be specified at any -of the levels. Options specified at one level flow down to all lower -levels, unless explicitly overridden. Some options are additive as -they flow down (for example, the offset of an element is cumulative as -one descends down the hierarchy, since the chip is offset so far from -the centre of the focal plane, while a cell is offset with respect to -the chip). - -The hierarchical levels may each be specified in turn, from the -highest level (FPA) to the lowest level (Readout), followed by a -commma-delimited list of allowed values for the name of an element of -the hierarchical level (or an asterisk if any value is permitted). In -options that follow, the name is denoted by the hierarchical level -name in angled brackets: \code{}, \code{}, \code{}, -\code{}. The name shall only be available at a hierarchical -level equal or lower than the corresponding level. - -For example: - -\begin{verbatim} -FPA * -Chip chip00,chip01,chip10,chip11 -Cell 1,2 -Readout * -\end{verbatim} - -In this example, the FPA may have any name, the chip name must be one -of \code{chip00}, \code{chip01}, \code{chip10} or \code{chip11}, the -cell name must be one of ``1'' or ``2'', and the readout may also have -any name. In this case, this likely represents a focal plane -consisting of four pieces of silicon, each with two amplifiers. - -Of course, the above example does not specify any options, which we -shall discuss next. - -\subsection{Options} - -By options, we mean the association of a particular value or FITS -header with a concept that is used in the IPP. It is the options that -provide the translation between the FITS headers and the concepts used -by the IPP, such as airmass, right ascension, exposure time, etc. - -An option is specified at the appropriate hierarchical level, indented -by a single tab (for clarity in reading by humans), and consists of a -keyword, followed by a value or FITS header from which the value may -be read. - -Below we note the options required for the IPP, along with the option -keyword: -\begin{itemize} -\item Right Ascension of the observation or of the hierarchical level; - \code{ra}. -\item Declination of the observation or of the hierarchical level; - \code{dec}. -\item Equinox for the coordinates; \code{equinox}. -\item Rotation of the observation or of the hierarchical level; - \code{rotation}. -\item Handedness of hierarchical level; \code{flip} --- either $+1$ or - $-1$. -\item Pixel scale of hierarchical level; \code{scale}. -\item Filename to retrieve the hierarchical level from; - \code{filename}. -\item FITS extension name from which to retrieve the hierarchical - level; \code{extname}. -\item FITS extension number from which to retrieve the hierarchical - level; \code{extnum}. -\item Bias regions; \code{bias} --- there may be more than one, - comma-delimited, and each specified in the format - \code{[x1:x2,y1:y2]}. -\item Axis to use for overscan subtraction; \code{axis} --- may be one - of \code{x}, \code{y}, \code{rows} or \code{columns}. -\item Detector regions to crop after overscan subtraction; - \code{detector}. -\item RA offset for the hierarchical level from the level(s) above; - \code{deltara}. -\item Dec offset for the hierarchical level from the level(s) above; - \code{deltadec}. -\item Rotation offset for the hierarchical level from the level(s) above; - \code{deltarot}. -\item Difference in handedness for the hierarchical level from the - level(s) above; \code{deltaflip}. -\item Multiplication of scale for the hierarchical level from the - level(s) above; \code{deltascale}. -\item Exposure time; \code{exptime}. -\item Airmass; \code{airmass}. -\item Hour angle; \code{ha}. -\item Sidereal time; \code{st}. -\item Latitude of observatory; \code{obslat}. -\item Longitude of observatory; \code{obslong}. -\item Altitude of observatory; \code{obsalt}. -\item Date of exposure; \code{date}. -\item Time of exposure; \code{time}. -\item Time reference; \code{timeref} --- may be \code{start}, - \code{stop} or \code{mid}, to specify when the observation time is - referenced to. -\item Format of date, to specify how to convert; \code{dateformat}. -\item Format of time, to specify how to convert; \code{timeformat}. -\item Extension containing OT shifts; \code{shiftsext}. -\item File containing OT shifts; \code{shiftsfile}. -\item Filter; \code{filter} --- multiple FITS headers may be - specified, comma-delimited, in order to account for instruments - which have multiple filter wheels. -\item Object/field name; \code{name}. -\item Observation type; \code{obstype}. -\item CCD dark time; \code{darktime}. -\item CCD gain; \code{gain}. -\item CCD read noise; \code{readnoise}. -\item Binning in x; \code{xbin}. -\item Binning in y; \code{ybin}. -\item Telescope altitude; \code{altitude}. -\item Telescope azimuth; \code{azimuth}. -\end{itemize} - -Note that the \code{deltara}, \code{deltadec}, \code{deltarot} and -\code{deltaflip} options should be propagated down the hierarchical -levels, while for the other options, the option defined last applies. - -Some unavailable options (e.g., not all observatories record the -sidereal time) may be calculated from other available options after -parsing. However, once this has been attempted, if any required -option is missing, and if the option cannot default to a reasonable -value, then the read should fail. - -A value specified for an option may include the name of an -hierarchical level in angled brackets (i.e., \code{}, -\code{}, \code{}, \code{}), which shall be -replaced iteratively by each of the allowed names, in turn, of the -hierarchical level. - -\subsubsection{Extensions not known a priori} - -In some cases, the extension name or number may not be known \emph{a -priori}. In particular, OTA data or NIR data from some instruments -may include multiple reads of a cell simply included in the same MEF -file along with other cells. - -In this case, there is no choice for the program but to create a table -of contents of the FITS file specified by scanning the extensions, and -associating different extensions with the appropriate hierarchical -level on the basis of a particular header. We specify the special -option, \code{id}, the corresponding value for which indicates which -header may be examined to indicate the name of the instance of a -hierarchical level. See the PS-GPC example for a demonstration. - -The \code{id} option may only be specified when the file name has -already been specified unambiguously at higher hierarchical level(s). - -Once it is known which extension corresponds to which -FPA/chip/cell/readout combination, then the focal plane may easily be -reconstructed. - -\subsection{Dependent options} - -In cases where the option is not available in the FITS header, the -value must be specified manually in the configuration file. However, -often the value is dependent upon the particular instance of an -hierarchical level. For such cases, we specify a syntax similar to -a \code{switch} statement in C. - -When an option is followed by a colon, then this means that it is -dependent upon the particular hierarchical level that is specified in -angled brackets following the colon. For example: -\begin{verbatim} - gain: -\end{verbatim} -specifies that the \code{gain} option is dependent upon the chip. - -A dependent option is followed on subsequent lines by the possible -values for the name of the hierarchical level, indented by two tabs -(for readability by humans) and the corresponding value for the -option. For example: -\begin{verbatim} -chip 00,01,10,11 - gain: - 00 1.0 - 01 2.1 - 10 3.2 - 11 4.3 - readnoise: - 00 5 - 01 6 - 10 7 - 11 8 - exptime EXPOSURE -\end{verbatim} - -This specifies that chip 00 has gain 1.0 and read noise 5, chip 01 has -gain 2.1 and read noise 6, chip 10 has gain 3.2 and read noise 7, and -chip 11 has gain 4.3 and read noise 8, and all chips have an exposure -time specified by the \code{EXPOSURE} FITS header. - -\section{Examples} - -I here offer some practical examples to help decipher these (no doubt, -cryptic) instructions. Not all options are specified, but hopefully -sufficient to give an idea of the format. - -\subsection{CFHT 12k MEF format} - -Some CFHT 12k images are written in MEF FITS format, with each of the -chips as separate extensions in a single FITS file for each exposure. - -\begin{verbatim} -FPA * - file .fits - ra RA - dec DEC - exptime OPENTIME -Chip chip1,chip2,chip3,chip4,chip5,chip6,chip7,chip8,chip9,chip10,chip11,chip12 - extnum - dra: - chip1 -110 - chip2 -90 - chip3 -70 - chip4 -50 - chip5 -30 - chip6 -10 - chip7 10 - chip8 30 - chip9 50 - chip10 70 - chip11 90 - chip12 110 - ddec: - chip1 -20 - chip2 -20 - chip3 -20 - chip4 -20 - chip5 -20 - chip6 -20 - chip7 20 - chip8 20 - chip9 20 - chip10 20 - chip11 20 - chip12 20 -Cell * -Readout * - bias [1:100,1:4096],[2150:2248,1:4096] - axis x - detector [101:2148,1:4096] -\end{verbatim} - -This specifies, firstly, that any name may be used to specify an FPA -image (I suspect that this is usually the case). Filenames are -specified on the FPA level (since all the chips are in the same FITS -file). If the FPA name given to the IPP is \code{img12345}, then the -corresponding filename that the IPP searches for is -\code{img12345.fits}. - -The Right Ascension, Declination and exposure time are specified for -the FPA, and read from the \code{RA}, \code{DEC} and \code{OPENTIME} -headers, respectively. These headers are to be found in the Primary -Header Unit (PHU), since an extension has not been specified. - -The allowed chip names are \code{chip1}, \code{chip2}, \code{chip3}, -\code{chip4}, \code{chip5}, \code{chip6}, \code{chip7}, \code{chip8}, -\code{chip9}, \code{chip10}, \code{chip11} and \code{chip12}. A chip -is to be found in the extension of the same name (as specified by the -\code{}. - -The RA and Dec offsets are specified, such that \code{chip1} is the -lower-left hand chip and \code{chip12} is the upper-right hand chip on -the focal plane. - -All Cell and Readout names are valid. Two bias regions are specified -for each chip, namely a prescan region, from column 1 to 100, and an -overscan region, from column 2150 to 2248. The bias region is to be -collapsed along the x axis, and following overscan subtraction, only -columns 101 to 2148 are of any interest. - - -\subsection{PS GPC} - -We expect that a \PS{} GigaPixel Camera exposure will be written as -several MEF files, one OTA to each file, with an extension within that -file for each cell and each readout (since there may be multiple -readouts per cell to incorporate fast guiding). To specify all 64 -OTAs and 64 cells in each would make for a long example, so we only -show 4 OTAs and 4 cells in each. - -This is a case in which the extensions are not known \emph{a priori}, -since there may exist multiple readouts for one cell in the same MEF -file as other cells. Note the use of the \code{id} option to specify -this. - -\begin{verbatim} -FPA * -Chip ota00,ota01,ota10,ota11 - file ..fits - ra RA - dec DEC - exptime EXPOSURE - dra: - ota00 -20 - ota01 20 - ota10 -20 - ota11 20 - ddec: - ota00 -20 - ota01 -20 - ota10 20 - ota11 20 -Cell cell00,cell01,cell10,cell11 - id CELLNUM - dra: - cell00 -10 - cell01 10 - cell10 -10 - cell11 10 - ddec: - cell00 -10 - cell01 -10 - cell10 10 - cell11 10 - shiftsext ot -Readout * - id READNUM - bias BIASSEC - axis x - detector DETSEC -\end{verbatim} - -Note first of all that all FPA names are valid (as is usually the -case). Valid chip names are \code{ota00}, \code{ota01}, \code{ota10} -and \code{ota11}. If the FPA name is \code{img12345}, then the -filename for \code{ota01} is \code{img12345.ota01.fits}. The RA, Dec -and exposure time are derived from the \code{RA}, \code{DEC} and -\code{EXPOSURE} headers, respectively. The chips are laid out in a -$2\times 2$ mosaic. - -Now we come to the cells. The allowed cell names are \code{cell00}, -\code{cell01}, \code{cell10} and \code{cell11}. However, the -extension these cells reside in is not known \emph{a priori}, and so -the \code{id} option specifies that the program should create a table -of contents of the extensions in the file, and determine the chip name -from the \code{CELLNUM} header in each extension. The cells are laid -out in a $2\times 2$ pattern on the chip. The OT shifts made for the -cell may be found in the extension of name \code{ot}. - -Finally, the readouts. Again, the particular readout is not known -\emph{a priori}, but the particular readout may be deduced from the -\code{READNUM} header. The bias region is specified by the -\code{BIASSEC} header, the bias region is to be collapsed along the x -axis, and the region to be cropped following the overscan subtraction -is specified by the \code{DETSEC} header. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\bibliographystyle{plain} -\bibliography{panstarrs} - -\end{document} Index: unk/doc/modules/notes.config =================================================================== --- /trunk/doc/modules/notes.config (revision 9619) +++ (revision ) @@ -1,207 +1,0 @@ -CFH12kMEF config: - -FPA * - file .fits - ra RA - dec DEC - exptime OPENTIME -Chip chip1,chip2,chip3,chip4,chip5,chip6,chip7,chip8,chip9,chip10,chip11,chip12 - extnum - dra: - chip1 -110 - chip2 -90 - chip3 -70 - chip4 -50 - chip5 -30 - chip6 -10 - chip7 10 - chip8 30 - chip9 50 - chip10 70 - chip11 90 - chip12 110 - ddec: - chip1 -20 - chip2 -20 - chip3 -20 - chip4 -20 - chip5 -20 - chip6 -20 - chip7 20 - chip8 20 - chip9 20 - chip10 20 - chip11 20 - chip12 20 -Cell * -Readout * - bias [2050:2148,1:4096] - axis x - use [1:2048,1:4096] - - -CFH12kSplit config: - -FPA: * -Chip: 1,2,3,4,5,6,7,8,9,10,11,12 - file ..fits - orientation: - 1 dRA dDec Rot Sign - 2 dRA dDec Rot Sign - 3 dRA dDec Rot Sign - 4 dRA dDec Rot Sign - 5 dRA dDec Rot Sign - 6 dRA dDec Rot Sign - 7 dRA dDec Rot Sign - 8 dRA dDec Rot Sign - 9 dRA dDec Rot Sign - 10 dRA dDec Rot Sign - 11 dRA dDec Rot Sign - 12 dRA dDec Rot Sign -Cell: * -Readout: * - bias [x1:x2,y1:y2] [x3:x4,y3:y4] [x5:x6,*] BIASHEADER1 BIASHEADER2 BIASHEADER3 - axis x - use [x1:x2,y1:y2] - - -MEGACAMraw config: - -FPA: * - file .fits -Chip: * - id CHIPHEADER - orientation: - 1 dRA dDec Rot Sign - 2 dRA dDec Rot Sign - ..... - 39 dRA dDec Rot Sign - 40 dRA dDec Rot Sign -Cell: * - id CELLHEADER - offset: - 1 dRA dDec - 2 dRA dDec -Readout: * - bias [x1:x2,y1:y2] [x3:x4,y3:y4] [x5:x6,*] BIASHEADER1 BIASHEADER2 BIASHEADER3 - axis x - use [x1:x2,y1:y2] - - -MEGACAMspliced config: - -FPA: * - file .fits -Chip: * - id CHIPHEADER - orientation: - 1 dRA dDec Rot Sign - 2 dRA dDec Rot Sign - ..... - 39 dRA dDec Rot Sign - 40 dRA dDec Rot Sign -Cell: * - id CELLHEADER -Readout: 2 - region: - 1 [x1:x2,y1:y2] [x3:x4,y3:y4] - 2 [x5:x6,y5:y6] [x7:x8,y7:y8] - bias: - 1 [x3:x4,y3:y4] - 2 [x7:x8,y7:y8] - axis x - use: - 1 [x1:x2,y1:y2] - 2 [x5:x6,y5:y6] - - -PS-GPC-raw config: - -FPA * -Chip 00,01,02,03,04,05,06,07,10,11,12,13,14,15,16,17,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,70,71,72,73,74,75,76,77 - file ..fits - orientation: - 00 dRA dDec Rot Sign - 01 dRA dDec Rot Sign - ..... - 76 dRA dDec Rot Sign - 77 dRA dDec Rot Sign -Cell: 00,01,02,03,04,05,06,07,10,11,12,13,14,15,16,17,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,70,71,72,73,74,75,76,77 - id CELLHEADER - offset: - 00 dRA dDec - 01 dRA dDec - ..... - 76 dRA dDec - 77 dRA dDec - shifts SHIFTS_EXT_NAME -Readout: * - id READOUT - bias BIASHEADER - axis x - use USEHEADER - - -PS-GPC-spliced config (corrected by Camera group on the summit): - -FPA: * - file .fits -Chip: 00,01,02,03,04,05,06,07,10,11,12,13,14,15,16,17,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,70,71,72,73,74,75,76,77 - id CHIPHEADER - orientation: - 00 dRA dDec Rot Sign - 01 dRA dDec Rot Sign - ..... - 76 dRA dDec Rot Sign - 77 dRA dDec Rot Sign -Cell: 00,01,02,03,04,05,06,07,10,11,12,13,14,15,16,17,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,70,71,72,73,74,75,76,77 - id CELLHEADER - offset: - 00 dRA dDec - 01 dRA dDec - ..... - 76 dRA dDec - 77 dRA dDec - shifts SHIFTS_EXT_NAME -Readout: 0 - use ALL - - - -Example for "small" GPC: - -FPA * - ra RA - dec DEC - exptime EXPOSURE -Chip ota00,ota01,ota10,ota11 - file ..fits - dra: - ota00 -20 - ota01 20 - ota10 -20 - ota11 20 - ddec: - ota00 -20 - ota01 -20 - ota10 20 - ota11 20 -Cell cell00,cell01,cell10,cell11 - id CELLNUM - dra: - cell00 -5 - cell01 5 - cell10 -5 - cell11 5 - ddec: - cell00 -5 - cell01 -5 - cell10 5 - cell11 5 - shiftsext ot -Readout * - id READNUM - bias BIASSEC - axis x - use DETSEC -