Index: /trunk/doc/modules/ModulesSDRS.tex =================================================================== --- /trunk/doc/modules/ModulesSDRS.tex (revision 6756) +++ /trunk/doc/modules/ModulesSDRS.tex (revision 6757) @@ -1,3 +1,3 @@ -%%% $Id: ModulesSDRS.tex,v 1.75 2006-02-16 18:57:51 eugene Exp $ +%%% $Id: ModulesSDRS.tex,v 1.76 2006-04-01 03:07:09 price Exp $ \documentclass[panstarrs,spec]{panstarrs} @@ -103,10 +103,11 @@ All modules need to load some configuration information defining parameters which may be configured at run-time. We break these -parameters down into three levels: +parameters down into four levels: \begin{itemize} \item Options for the particular site installation of the pipeline: the {\it site}; -\item Options specifying the instrument setup, and in particular the - format of the FITS file: the {\it camera}; and +\item Options specifying the instrument setup: the {\it camera}; +\item Options specifying the format of the FITS file: the {\it + format}; and \item Options specifying the particular parameter choices that affect the details of an analysis: the {\it recipe}. @@ -205,12 +206,93 @@ \end{verbatim} -\subsubsection{Camera Configuration} - -The camera configuration is somewhat complicated and involved, since -it must not only specify how to translate the pixels from a FITS file -into a focal plane hierarchy (\S\ref{sec:focalplane}), but it must -also specify how to derive the various values the IPP needs -(\S\ref{sec:concepts}). Moreover, it must be able to do these for the -great variety of cameras in use in the astronomical community. +\subsubsection{Camera configuration} + +The camera configuration file is a fairly simple configuration file +containing information particular to a particular camera, regardless +of the file format used to represent that camera. The camera configuration +consists of the following elements: +\begin{itemize} +\item \code{FORMATS} of type \code{METADATA}: this contains a list of + known FITS file formats with the file names (of type \code{STR}) of + the configuration files; +\item \code{FPA} of type \code{METADATA}: this contains a list of + chips, each with a string list (type \code{STR} of the component + cells; and +\item \code{RECIPES} of type \code{METADATA}: this contains a list of + recipes used for the camera with the file names (of type \code{STR} + of the configuration files. +\end{itemize} + +An example camera configuration file: + +\begin{verbatim} +# Camera configuration file for MegaCam: describes the camera + +# File formats that we know about +FORMATS METADATA + RAW STR megacam_raw.config + SPLICE STR megacam_splice.config + SPLIT STR megacam_split.config +END + + +# Description of camera --- all the chips and the cells that comprise them +FPA METADATA + ccd00 STR left right + ccd01 STR left right + ccd02 STR left right + ccd03 STR left right + ccd04 STR left right + ccd05 STR left right + ccd06 STR left right + ccd07 STR left right + ccd08 STR left right + ccd09 STR left right + ccd10 STR left right + ccd11 STR left right + ccd12 STR left right + ccd13 STR left right + ccd14 STR left right + ccd15 STR left right + ccd16 STR left right + ccd17 STR left right + ccd18 STR left right + ccd19 STR left right + ccd20 STR left right + ccd21 STR left right + ccd22 STR left right + ccd23 STR left right + ccd24 STR left right + ccd25 STR left right + ccd26 STR left right + ccd27 STR left right + ccd28 STR left right + ccd29 STR left right + ccd30 STR left right + ccd31 STR left right + ccd32 STR left right + ccd33 STR left right + ccd34 STR left right + ccd35 STR left right +END + + +# Recipe options +RECIPES METADATA + PHASE2 STR phase2.config # Phase 2 recipe details + PSPHOT STR psphot.config # psphot details +END +\end{verbatim} + + +\subsubsection{FITS file format} + +The FITS file format configuration files are somewhat complicated and +involved, since they must not only specify how to translate the pixels +from a FITS file into a focal plane hierarchy +(\S\ref{sec:focalplane}), but must also specify how to derive the +various values the IPP needs (\S\ref{sec:concepts}). Moreover, they +must be able to do these for the great variety of cameras in use in +the astronomical community. Example camera configuration files are included in an appendix, but @@ -231,6 +313,6 @@ segments and binary data segments. The headers describe the format and layout of the data segments. The first of these groups is -traditionally called the 'primary header unit' (PHU) and the rest are -referred to as 'extensions'. The header segments may contain +traditionally called the ``primary header unit'' (PHU) and the rest are +referred to as ``extensions''. The header segments may contain extensive documentary information related to the interpretation of the data. Although the FITS format defines a standard representation of @@ -246,5 +328,5 @@ In the simplest case, the camera consists of a single chip consisting of a single cell always read with a single readout. In this case, the -image data could be written as part of the primary data unit. In a +image data could be written as part of the primary header unit. In a more complex case with multiple chips and multiple cells, the data may be organized in several ways. The data may be distributed into @@ -260,5 +342,5 @@ entire FITS file corresponds to (FPA, chip, or cell), and what level the extensions correspond to (chip, cell or no extensions at all). -Knowing these, and having a list of the components, we can construct +Knowing these, and having a list of the extensions, we can construct the focal plane hierarchy. @@ -274,38 +356,58 @@ To define the hierarchy, we specify the following keywords: \begin{itemize} - -\item \code{PHU} of type \code{STR}: May be one of \code{FPA}, - \code{CHIP}, or \code{CHIP}. This specifies the focal plane level - of the Primary Header Unit, and hence the entire FITS file (the - 'class' of the file) . - -\item \code{EXTENSIONS} of type \code{STR}: May be one of \code{CHIP}, - \code{CELL} or \code{NONE}, though not of a level higher than that - specified by the \code{PHU}. This specifies what each extension - represents. - -\item \code{CONTENTS} which may be of type \code{METADATA} or - \code{STR}, depending upon the \code{PHU} and \code{EXTENSIONS}, - specifies what the contents of the FITS file are: +\item \code{RULE} of type \code{METADATA}: contains headers with their + respective values that are required to be in the PHU of any FITS + file of this type. + +\item \code{FILE} of type \code{METADATA}: contains information on + the global format of the FITS file with the following entries: \begin{itemize} - \item \code{PHU=FPA, EXTENSIONS=CHIP}: Type \code{METADATA} with - the component keywords being the extension names and the values - the names of the cells, separated by commas or whitespace. - \item \code{PHU=FPA, EXTENSIONS=CELL}: Type \code{METADATA} with - the component keywords being the extension names and the values - the chip name and the cell type, separated by a colon. - \item \code{PHU=FPA, EXTENSIONS=NONE}: Type \code{METADATA} with - the component keywords being the chip names and the values the - names of the cells, separated by commas or whitespace. - \item \code{PHU=CHIP, EXTENSIONS=CELL}: Type \code{METADATA} with - the component keywords being the extension names and the values - the corresponding cell type. - \item \code{PHU=CHIP, EXTENSIONS=NONE}: Type \code{STR} with the - value being the cell types separated by commas or whitespace. + \item \code{PHU} of type \code{STR}: May be one of \code{FPA}, + \code{CHIP}, or \code{CHIP}. This specifies the focal plane level + of the Primary Header Unit, and hence the entire FITS file (the + 'class' of the file). + + \item \code{EXTENSIONS} of type \code{STR}: May be one of + \code{CHIP}, \code{CELL} or \code{NONE}, though not of a level + higher than that specified by the \code{PHU}. This specifies what + each extension represents. + + \item \code{FPA.NAME} of type \code{STR}: Specifies a header keyword + in the primary header for a unique identifier for the FPA name + (e.g., an observation number). + + \item \code{CHIP.NAME} of type \code{STR}: Need only be included if + \code{PHU} is \code{CHIP} or \code{CELL}. Specifies a header + keyword in the primary header for a unique identifier for the chip + name (e.g., the CCD identification number or name). + + \item \code{CELL.NAME} of type \code{STR}: Need only be included if + \code{PHU} is \code{CELL}. Specifies a header keyword in the + primary header for a unique identifier for the cell name (e.g., + the amplifier identification). \end{itemize} -\item \code{CELLS} of type \code{METADATA} with the component keywords - being the cell names or types, each of type \code{METADATA}. Within - each cell should be specified various PS concept values appropriate - for each cell. + +\item \code{CONTENTS} of type \code{METADATA}: Specifies what the + contents of the FITS file are. Each entry is an extension name with + the corresponding value being a string listing the source and the + cell type, separated by a colon (e.g., \code{ccd01:left + ccd01:right}). If \code{EXTENSIONS=NONE} then the \code{CONTENTS} + is ignored (since there are no extensions to list). + +\item \code{CELLS} of type \code{METADATA}: specifies the cell types. + Entries are the cell types, each of type \code{METADATA}, with the + values being PS concept values appropriate for each cell type (more + detail later) \tbd{link to more detail}. In the event that + \code{EXTENSIONS=NONE}, the \code{CELLS} is used as a list of all + cells present in the file. + +\item \code{TRANSLATION} of type \code{METADATA} + +\item \code{DEFAULTS} of type \code{METADATA} + +\item \code{DATABASE} of type \code{METADATA} + +\item \code{FORMATS} of type \code{METADATA} + \end{itemize} @@ -313,31 +415,176 @@ \begin{verbatim} +# The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file. + +# How to identify this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 72 +END + # How to read this data -PHU STR FPA # The FITS file represents an entire FPA -EXTENSIONS STR CELL # The extensions represent cells - +FORMAT METADATA + PHU STR FPA # The FITS file represents an entire FPA + EXTENSIONS STR CELL # The extensions represent cells + FPA.NAME STR EXPNUM # A PHU keyword for unique identifier within the hierarchy level +END + # What's in the FITS file? -CONTENTS METADATA - # Extension name, chip name:type - amp00 STR ccd00:left - amp01 STR ccd00:right - amp02 STR ccd01:left - amp03 STR ccd01:right - amp04 STR ccd02:left +CONTENTS METADATA + # Extension name, chip name:type + amp00 STR ccd00:left + amp01 STR ccd00:right + amp02 STR ccd01:left + amp03 STR ccd01:right + amp04 STR ccd02:left + amp05 STR ccd02:right + amp06 STR ccd03:left + amp07 STR ccd03:right + amp08 STR ccd04:left + amp09 STR ccd04:right + amp10 STR ccd05:left + amp11 STR ccd05:right + amp12 STR ccd06:left + amp13 STR ccd06:right + amp14 STR ccd07:left + amp15 STR ccd07:right + amp16 STR ccd08:left + amp17 STR ccd08:right + amp18 STR ccd09:left + amp19 STR ccd09:right + amp20 STR ccd10:left + amp21 STR ccd10:right + amp22 STR ccd11:left + amp23 STR ccd11:right + amp24 STR ccd12:left + amp25 STR ccd12:right + amp26 STR ccd13:left + amp27 STR ccd13:right + amp28 STR ccd14:left + amp29 STR ccd14:right + amp30 STR ccd15:left + amp31 STR ccd15:right + amp32 STR ccd16:left + amp33 STR ccd16:right + amp34 STR ccd17:left + amp35 STR ccd17:right + amp36 STR ccd18:left + amp37 STR ccd18:right + amp38 STR ccd19:left + amp39 STR ccd19:right + amp40 STR ccd20:left + amp41 STR ccd20:right + amp42 STR ccd21:left + amp43 STR ccd21:right + amp44 STR ccd22:left + amp45 STR ccd22:right + amp46 STR ccd23:left + amp47 STR ccd23:right + amp48 STR ccd24:left + amp49 STR ccd24:right + amp50 STR ccd25:left + amp51 STR ccd25:right + amp52 STR ccd26:left + amp53 STR ccd26:right + amp54 STR ccd27:left + amp55 STR ccd27:right + amp56 STR ccd28:left + amp57 STR ccd28:right + amp58 STR ccd29:left + amp59 STR ccd29:right + amp60 STR ccd30:left + amp61 STR ccd30:right + amp62 STR ccd31:left + amp63 STR ccd31:right + amp64 STR ccd32:left + amp65 STR ccd32:right + amp66 STR ccd33:left + amp67 STR ccd33:right + amp68 STR ccd34:left + amp69 STR ccd34:right + amp70 STR ccd35:left + amp71 STR ccd35:right END # Specify the cell data -CELLS METADATA - left METADATA # Left amplifier - CELL.BIASSEC STR BIASSEC - CELL.TRIMSEC STR DATASEC - CELL.PARITY S32 1 - END - right METADATA # Right amplifier - # This cell is read out in the opposite direction - CELL.BIASSEC STR BIASSEC - CELL.TRIMSEC STR [1025:2048,1:2048] - CELL.PARITY S32 -1 - END +CELLS METADATA + left METADATA # Left amplifier + CELL.NAME STR LeftSide + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR DATASEC + CELL.XPARITY S32 1 # We could have specified this as a DEFAULT, but this works + CELL.X0 S32 1 + CELL.Y0 S32 1 + END + right METADATA # Right amplifier + CELL.NAME STR RightSide + CELL.BIASSEC.SOURCE STR HEADER + CELL.TRIMSEC.SOURCE STR HEADER + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR DATASEC + CELL.XPARITY S32 -1 # This cell is read out in the opposite direction + CELL.X0 S32 2048 + CELL.Y0 S32 1 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE + CELL.TIME STR MJD-OBS + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.READDIR S32 1 # Cell is read in x direction + CELL.BAD S32 0 + CELL.TIMESYS STR UTC + CELL.YPARITY S32 1 +END + +# How to translation PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# FPA.BIAS METADATA +# TABLE STR Camera +# COLUMN STR gain +# chipId STR {CHIP.NAME} +# cellId STR {CELL.NAME} +# time STR {CELL.TIME} +# END +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). +END + + +# Where there might be some ambiguity, specify the format +FORMATS METADATA + FPA.RA STR HOURS + FPA.DEC STR DEGREES + CELL.TIME STR MJD +# CELL.BINNING STR SEPARATE + CELL.X0 STR FORTRAN + CELL.Y0 STR FORTRAN END \end{verbatim}