Index: trunk/doc/modules/ModulesSDRS.tex =================================================================== --- trunk/doc/modules/ModulesSDRS.tex (revision 6762) +++ trunk/doc/modules/ModulesSDRS.tex (revision 9689) @@ -1,3 +1,3 @@ -%%% $Id: ModulesSDRS.tex,v 1.77 2006-04-04 18:52:47 eugene Exp $ +%%% $Id: ModulesSDRS.tex,v 1.78 2006-10-21 00:58:33 price Exp $ \documentclass[panstarrs,spec]{panstarrs} @@ -47,4 +47,5 @@ PSDC-430-006 & Pan-STARRS PS-1 IPP Algorithm Design Document \\ \hline PSDC-430-011 & Pan-STARRS PS-1 IPP System/Subsystem Design Description \\ \hline +PSDC-430-??? & Pan-STARRS PS-1 IPP Configurations \\ \hline \DocumentsExternal Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ @@ -69,6 +70,5 @@ programs which perform complete data analysis tasks (an ``analysis stage''). The modules may be tied together within a C framework or -using a high-level scripting language. Bindings of the Modules are -made available to the scripting language using the program SWIG. +using a high-level scripting language. In order to preserve name space, globally-visible structures and @@ -76,896 +76,216 @@ Modules''. -\section{Runtime Configuration Data} - -PSLib defines a \code{psMetadata} structure which can carry labeled -data of arbitrary types. The associated functions implemented by -PSLib consist of tools to manipulate and extract data from -\code{psMetadata} collections. A particular application of the -\code{psMetadata} structure within PSLib is to carry the data from a -FITS header. Other general-purpose information is also carried with -the structure. Functions are available to fill a \code{psMetadata} -collection from a text-based configuration file using a human-readable -syntax, and to fill a \code{psMetadata} collection from a properly -formatted XML document. - -In the IPP Modules, we use \code{psMetadata} collections to carry -run-time configuration data used by the data analysis modules. Below, -in the discussion of the various modules, this configuration -information is defined by specifying the name of the data item of -interest, the conceptual meaning of that data item, and the allowed -values for the data item. In this section, we discuss top-level -concepts related to the configuration information, including the -sources of the run-time configuration data and special -operations used to extract information from the configuration system. - -\subsection{Configuration Data Sources} - -All modules need to load some configuration information defining -parameters which may be configured at run-time. We break these -parameters down into four levels: -\begin{itemize} -\item Options for the particular site installation of the - pipeline: the {\it site}; -\item Options specifying the instrument setup: the {\it camera}; -\item Options specifying the format of the FITS file: the {\it - format}; and -\item Options specifying the particular parameter choices that affect - the details of an analysis: the {\it recipe}. -\end{itemize} -Note that these are arranged in an hierarchical order, with the site -configuration being the most general, and the recipe configuration the -most specific. For example, not all sites will have to deal with all -cameras, and different cameras may require different recipes at -different times according to their particular quirks, analysis -experimentations, or their evolution. - -Each of the levels will have a metadata configuration file. In the -case of the site configuration, the filename shall be that specified -by the \code{-site} option on the command line if provided, the -environment variable \code{PS_SITE}, if defined, or \code{~/.ipprc} -otherwise. The camera configuration shall be specified by the -\code{-camera} option on the command line if provided, or shall be -inferred from a FITS header (more detail below). The recipe -configuration shall be specified by the \code{-recipe} option on the -command line if provided, or from the the camera configuration (more -detail below). - -\subsection{Configuration Files} - -\subsubsection{Site Configuration} - -The site configuration file must contain the following: -\begin{itemize} -\item The database configuration: - \begin{itemize} - \item \code{DBSERVER} of type \code{STR}: The database host name for - \code{psDBInit}. - \item \code{DBUSER} of type \code{STR}: The database user name for - \code{psDBInit}. - \item \code{DBPASSWORD} of type \code{STR}: The corresponding - database password for \code{psDBInit}. - \item \code{DBNAME} of type \code{STR}: The database name for - \code{psDBInit}. - \end{itemize} -\item \code{CAMERAS} of type \code{METADATA}: A list of instruments - that the system understands. Cameras are specified as separate - metadata entries, with the name of the camera as the key, and the - filename of the camera configuration file (of type \code{STR}) as - the data. -\end{itemize} -and may also contain the following psLib configuration options: -\begin{itemize} -\item \code{TIME} of type \code{STR}: The time configuration file (for - \code{psTimeInitialize}). -\item \code{LOGLEVEL} of type \code{S32}: The log level for - \code{psLogSetLevel}. -\item \code{LOGFORMAT} of type \code{STR}: The log format for - \code{psLogSetFormat}. -\item \code{LOGDEST} of type \code{STR}: The log destination for - \code{psLogSetDestination}. -\item \code{TRACE} of type \code{METADATA}: A list of components with - the desired trace level (of type \code{S32}) for each. -\end{itemize} - -\tbd{No doubt there is a need for better security than storing the -database password directly in the file, but we push this problem onto -the stack for now.} - -\tbd{We will add other data sources in the future, e.g., file paths, -configuration for Nebulous and DVO, etc.}. - - -An example site configuration file: - -\begin{verbatim} -### Example .ipprc file - -### Database configuration -DBSERVER STR ippdb.ifa.hawaii.edu # Database host name (for psDBInit) -DBUSER STR ipp # Database user name (for psDBInit) -DBPASSWORD STR password # Database password (for psDBInit) - -### Setups for each camera system -CAMERAS METADATA - MEGACAM_RAW STR megacam_raw.config - MEGACAM_SPLICE STR megacam_splice.config - GPC1_RAW STR gpc1_raw.config - LRIS_BLUE STR lris_blue.config - LRIS_RED STR lris_red.config -END - -### psLib setup -TIME STR time.config # Time configuration file -LOGLEVEL S32 3 # Logging level; 3=INFO -LOGFORMAT STR THLNM # Log format -LOGDEST STR STDOUT # Log destination -TRACE METADATA # Trace levels - psLib.math.psPolynomial S32 6 - psLib.image.psImageConvolve S32 2 -END -\end{verbatim} - -\subsubsection{Camera configuration} - -The camera configuration file is a fairly simple configuration file -containing information particular to a particular camera, regardless -of the file format used to represent that camera. The camera configuration -consists of the following elements: -\begin{itemize} -\item \code{FORMATS} of type \code{METADATA}: this contains a list of - known FITS file formats with the file names (of type \code{STR}) of - the configuration files; -\item \code{FPA} of type \code{METADATA}: this contains a list of - chips, each with a string list (type \code{STR} of the component - cells; and -\item \code{RECIPES} of type \code{METADATA}: this contains a list of - recipes used for the camera with the file names (of type \code{STR} - of the configuration files. -\end{itemize} - -An example camera configuration file: - -\begin{verbatim} -# Camera configuration file for MegaCam: describes the camera - -# File formats that we know about -FORMATS METADATA - RAW STR megacam_raw.config - SPLICE STR megacam_splice.config - SPLIT STR megacam_split.config -END - - -# Description of camera --- all the chips and the cells that comprise them -FPA METADATA - ccd00 STR left right - ccd01 STR left right - ccd02 STR left right - ccd03 STR left right - ccd04 STR left right - ccd05 STR left right - ccd06 STR left right - ccd07 STR left right - ccd08 STR left right - ccd09 STR left right - ccd10 STR left right - ccd11 STR left right - ccd12 STR left right - ccd13 STR left right - ccd14 STR left right - ccd15 STR left right - ccd16 STR left right - ccd17 STR left right - ccd18 STR left right - ccd19 STR left right - ccd20 STR left right - ccd21 STR left right - ccd22 STR left right - ccd23 STR left right - ccd24 STR left right - ccd25 STR left right - ccd26 STR left right - ccd27 STR left right - ccd28 STR left right - ccd29 STR left right - ccd30 STR left right - ccd31 STR left right - ccd32 STR left right - ccd33 STR left right - ccd34 STR left right - ccd35 STR left right -END - - -# Recipe options -RECIPES METADATA - PHASE2 STR phase2.config # Phase 2 recipe details - PSPHOT STR psphot.config # psphot details -END -\end{verbatim} - - -\subsubsection{FITS file format} - -The FITS file format configuration files are somewhat complicated and -involved, since they must not only specify how to translate the pixels -from a FITS file into a focal plane hierarchy -(\S\ref{sec:focalplane}), but must also specify how to derive the -various values the IPP needs (\S\ref{sec:concepts}). Moreover, they -must be able to do these for the great variety of cameras in use in -the astronomical community. - -Example camera configuration files are included in an appendix, but -below we explain the components. - -\paragraph{FITS File to Focal Plane Hierarchy} - -The Focal Plane hierarchy (\code{pmFPA, pmChip, pmCell, pmReadout}) is -explained in more detail in \S\ref{sec:focalplane}. The top level, an -FPA contains one or more chips, which correspond to a contiguous piece -of silicon. A chip contains one or more cells, which correspond to a -single amplifier. A cell contains one or more readouts, which -correspond to individual reads of the detector. - -The FITS data storage formation is a standard in the astronomical -community for storing astronomical images. A FITS file consists of an -arbitrary number of coupled human readable \code{ASCII} header -segments and binary data segments. The headers describe the format -and layout of the data segments. The first of these groups is -traditionally called the ``primary header unit'' (PHU) and the rest are -referred to as ``extensions''. The header segments may contain -extensive documentary information related to the interpretation of the -data. Although the FITS format defines a standard representation of -the data, the header metadata is not so consistently defined within -the astronomical community. Also, the flexibility of the data format -means that different representations are possible for the same -fundamental collection of data. The tools presented in this section -provide a method to define and constrain the wide range of possible -FITS representations of astronomical images. - -Within the FITS data representation, there are various choices which -can and have been made for the placement of the pixels in the file. -In the simplest case, the camera consists of a single chip consisting -of a single cell always read with a single readout. In this case, the -image data could be written as part of the primary header unit. In a -more complex case with multiple chips and multiple cells, the data may -be organized in several ways. The data may be distributed into -multiple files or in multiple FITS data extensions. A single camera -image may be written as a collection of files for individual chips -with separate extensions for each cell (CFH12K.split, GPC). Another -camera may write a single file with multiple extensions for each cell -(Megacam.raw), or multiple extensions per chip, with each cell -representing portions of the chip image (Megacam.splice, CFHT-IR). - -In all of these representations, there are only two basic distinctions -in how the pixel data is stored: what level in the hierarchy the -entire FITS file corresponds to (FPA, chip, or cell), and what level -the extensions correspond to (chip, cell or no extensions at all). -Knowing these, and having a list of the extensions, we can construct -the focal plane hierarchy. - -Note that a single data extension, consisting of a uniform grid of -pixels, can only naturally represent a cell or a chip. In order to -represent the entire focal plane array as a single grid, some -artificial choices would be made to fill-in or ignore the gaps between -chips and their relative rotations. Within our framework, a complete -focal plane mosaic of multiple chips could be represented as a single -extension by treating the collection of pixels as if they were from a -single chip. - -To define the hierarchy, we specify the following keywords: -\begin{itemize} -\item \code{RULE} of type \code{METADATA}: contains headers with their - respective values that are required to be in the PHU of any FITS - file of this type. - -\item \code{FILE} of type \code{METADATA}: contains information on - the global format of the FITS file with the following entries: - \begin{itemize} - \item \code{PHU} of type \code{STR}: May be one of \code{FPA}, - \code{CHIP}, or \code{CHIP}. This specifies the focal plane level - of the Primary Header Unit, and hence the entire FITS file (the - 'class' of the file). - - \item \code{EXTENSIONS} of type \code{STR}: May be one of - \code{CHIP}, \code{CELL} or \code{NONE}, though not of a level - higher than that specified by the \code{PHU}. This specifies what - each extension represents. - - \item \code{FPA.NAME} of type \code{STR}: Specifies a header keyword - in the primary header for a unique identifier for the FPA name - (e.g., an observation number). - - \item \code{CHIP.NAME} of type \code{STR}: Need only be included if - \code{PHU} is \code{CHIP} or \code{CELL}. Specifies a header - keyword in the primary header for a unique identifier for the chip - name (e.g., the CCD identification number or name). - - \item \code{CELL.NAME} of type \code{STR}: Need only be included if - \code{PHU} is \code{CELL}. Specifies a header keyword in the - primary header for a unique identifier for the cell name (e.g., - the amplifier identification). - \end{itemize} - -\item \code{CONTENTS} of type \code{METADATA}: Specifies what the - contents of the FITS file are. Each entry is an extension name with - the corresponding value being a string listing the source and the - cell type, separated by a colon (e.g., \code{ccd01:left - ccd01:right}). If \code{EXTENSIONS=NONE} then the \code{CONTENTS} - is ignored (since there are no extensions to list). - -\item \code{CELLS} of type \code{METADATA}: specifies the cell types. - Entries are the cell types, each of type \code{METADATA}, with the - values being PS concept values appropriate for each cell type (more - detail later) \tbd{link to more detail}. In the event that - \code{EXTENSIONS=NONE}, the \code{CELLS} is used as a list of all - cells present in the file. - -\item \code{TRANSLATION} of type \code{METADATA} - -\item \code{DEFAULTS} of type \code{METADATA} - -\item \code{DATABASE} of type \code{METADATA} - -\item \code{FORMATS} of type \code{METADATA} - -\end{itemize} - -An example: - -\begin{verbatim} -# The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file. - -# How to identify this type -RULE METADATA - TELESCOP STR CFHT 3.6m - DETECTOR STR MegaCam - EXTEND BOOL T - NEXTEND S32 72 -END - -# How to read this data -FORMAT METADATA - PHU STR FPA # The FITS file represents an entire FPA - EXTENSIONS STR CELL # The extensions represent cells - FPA.NAME STR EXPNUM # A PHU keyword for unique identifier within the hierarchy level -END - -# What's in the FITS file? -CONTENTS METADATA - # Extension name, chip name:type - amp00 STR ccd00:left - amp01 STR ccd00:right - amp02 STR ccd01:left - amp03 STR ccd01:right - amp04 STR ccd02:left - amp05 STR ccd02:right - amp06 STR ccd03:left - amp07 STR ccd03:right - amp08 STR ccd04:left - amp09 STR ccd04:right - amp10 STR ccd05:left - amp11 STR ccd05:right - amp12 STR ccd06:left - amp13 STR ccd06:right - amp14 STR ccd07:left - amp15 STR ccd07:right - amp16 STR ccd08:left - amp17 STR ccd08:right - amp18 STR ccd09:left - amp19 STR ccd09:right - amp20 STR ccd10:left - amp21 STR ccd10:right - amp22 STR ccd11:left - amp23 STR ccd11:right - amp24 STR ccd12:left - amp25 STR ccd12:right - amp26 STR ccd13:left - amp27 STR ccd13:right - amp28 STR ccd14:left - amp29 STR ccd14:right - amp30 STR ccd15:left - amp31 STR ccd15:right - amp32 STR ccd16:left - amp33 STR ccd16:right - amp34 STR ccd17:left - amp35 STR ccd17:right - amp36 STR ccd18:left - amp37 STR ccd18:right - amp38 STR ccd19:left - amp39 STR ccd19:right - amp40 STR ccd20:left - amp41 STR ccd20:right - amp42 STR ccd21:left - amp43 STR ccd21:right - amp44 STR ccd22:left - amp45 STR ccd22:right - amp46 STR ccd23:left - amp47 STR ccd23:right - amp48 STR ccd24:left - amp49 STR ccd24:right - amp50 STR ccd25:left - amp51 STR ccd25:right - amp52 STR ccd26:left - amp53 STR ccd26:right - amp54 STR ccd27:left - amp55 STR ccd27:right - amp56 STR ccd28:left - amp57 STR ccd28:right - amp58 STR ccd29:left - amp59 STR ccd29:right - amp60 STR ccd30:left - amp61 STR ccd30:right - amp62 STR ccd31:left - amp63 STR ccd31:right - amp64 STR ccd32:left - amp65 STR ccd32:right - amp66 STR ccd33:left - amp67 STR ccd33:right - amp68 STR ccd34:left - amp69 STR ccd34:right - amp70 STR ccd35:left - amp71 STR ccd35:right -END - -# Specify the cell data -CELLS METADATA - left METADATA # Left amplifier - CELL.NAME STR LeftSide - CELL.BIASSEC.SOURCE STR HEADER - CELL.TRIMSEC.SOURCE STR HEADER - CELL.BIASSEC STR BIASSEC - CELL.TRIMSEC STR DATASEC - CELL.XPARITY S32 1 # We could have specified this as a DEFAULT, but this works - CELL.X0 S32 1 - CELL.Y0 S32 1 - END - right METADATA # Right amplifier - CELL.NAME STR RightSide - CELL.BIASSEC.SOURCE STR HEADER - CELL.TRIMSEC.SOURCE STR HEADER - CELL.BIASSEC STR BIASSEC - CELL.TRIMSEC STR DATASEC - CELL.XPARITY S32 -1 # This cell is read out in the opposite direction - CELL.X0 S32 2048 - CELL.Y0 S32 1 - END -END - -# How to translate PS concepts into FITS headers -TRANSLATION METADATA - FPA.NAME STR EXPNUM - FPA.AIRMASS STR AIRMASS - FPA.FILTER STR FILTER - FPA.POSANGLE STR ROTANGLE - FPA.RA STR RA - FPA.DEC STR DEC - FPA.RADECSYS STR RADECSYS - CELL.EXPOSURE STR EXPTIME - CELL.DARKTIME STR DARKTIME - CELL.GAIN STR GAIN - CELL.READNOISE STR RDNOISE - CELL.SATURATION STR SATURATE - CELL.TIME STR MJD-OBS - CELL.XBIN STR CCDBIN1 - CELL.YBIN STR CCDBIN2 -END - -# Default PS concepts that may be specified by value -DEFAULTS METADATA - CELL.READDIR S32 1 # Cell is read in x direction - CELL.BAD S32 0 - CELL.TIMESYS STR UTC - CELL.YPARITY S32 1 -END - -# How to translation PS concepts into database lookups -DATABASE METADATA - TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS -# FPA.BIAS METADATA -# TABLE STR Camera -# COLUMN STR gain -# chipId STR {CHIP.NAME} -# cellId STR {CELL.NAME} -# time STR {CELL.TIME} -# END -# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME -# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME - -# A database entry refers to a particular column (COLUMN) in a -# particular table (TABLE), given certain PS concepts (GIVENPS) that -# match certain database columns (GIVENDBCOL). -END - - -# Where there might be some ambiguity, specify the format -FORMATS METADATA - FPA.RA STR HOURS - FPA.DEC STR DEGREES - CELL.TIME STR MJD -# CELL.BINNING STR SEPARATE - CELL.X0 STR FORTRAN - CELL.Y0 STR FORTRAN -END -\end{verbatim} - -Observe how the \code{CONTENTS} specifies the extension name, which we -know from the \code{EXTENSIONS} is a cell, and that each extension is -associated with a chip, and has a cell type. - -\paragraph{Deriving concept values} -\label{sec:derivingconcepts} - -The PS concepts are described in more detail in \S\ref{sec:concepts}. -Basically, astronomical cameras generally store the important details -(``concepts'') in different ways. This is generally manifested in the -choice of different FITS header keywords to describe the same concept, -but one can also imagine deriving values from a database or a known -default. - -We therefore specify the following keywords: -\begin{itemize} -\item \code{TRANSLATION} of type \code{METADATA} is a translation - table for understanding PS concepts in terms of FITS headers. The - PS concept (keyword) is derived from the FITS header given in the - value. -\item \code{DATABASE} of type \code{METADATA} is a formula for - obtaining a PS concept from the database. Each component is of a - user-specified type containing \code{TABLE}, \code{COLUMN}, - \code{GIVENDBCOL} and \code{GIVENPS}. The idea is that to obtain - the value of a PS concept, one refers to a particular \code{COLUMN} - in a particular \code{TABLE}, where the value of certain PS concepts - (\code{GIVENPS}; multiple values separated by a comma or semicolon) - match certain database columns (\code{GIVENDBCOL}; multiple values - separated by a comma or semicolon). -\item \code{DEFAULTS} of type \code{METADATA} is a set of default - values of PS concepts for the camera. The PS concept (keyword) is - assigned the value. There is also limited dependency allowed; see - \S\ref{sec:concepts}. -\end{itemize} - -An example: -\begin{verbatim} -# How to translate PS concepts into FITS headers -TRANSLATION METADATA - FPA.NAME STR EXPNUM - FPA.AIRMASS STR AIRMASS - FPA.FILTER STR FILTER - FPA.POSANGLE STR ROTANGLE - FPA.RA STR RA - FPA.DEC STR DEC - FPA.RADECSYS STR RADECSYS - FPA.MJD STR MJD-OBS - CELL.EXPOSURE STR EXPTIME - CELL.DARKTIME STR DARKTIME - CELL.XBIN STR CCDBIN1 - CELL.YBIN STR CCDBIN2 - CELL.SATURATION STR SATURATE -END - -# Default PS concepts that may be specified by value -DEFAULTS METADATA - CELL.BAD S32 0 - CELL.PARITY.DEPEND STR CHIP.NAME - CELL.PARITY METADATA - amp00 S32 1 - amp01 S32 -1 - amp02 S32 1 - amp03 S32 -1 - END -END - -# How to translate PS concepts into database lookups -DATABASE METADATA - TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS - CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME - CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME -END -\end{verbatim} - -The \code{.DEPEND} entry in the \code{DEFAULTS} will be explained in -\S\ref{sec:concepts}. - -\paragraph{Indentification by rule} -\label{sec:camerarule} - -The function \code{pmConfigCameraFromHeader} requires that the camera -configuration also contains a rule on how to recognise that a FITS -header comes from that camera. - -We therefore specify another keyword: \code{RULE} of type -\code{METADATA}: Contains a list of FITS headers keywords and values -(of the appropriate type) against which actual headers are compared to -determine if it matches the camera type. - -An example is: -\begin{verbatim} -# How to identify this type -RULE METADATA - TELESCOP STR CFHT 3.6m - DETECTOR STR MegaCam - EXTEND BOOL T - NEXTEND S32 72 -END -\end{verbatim} - -\paragraph{Recipes} - -The camera configuration file must also contain filenames for the -recipe configuration files. We include \code{RECIPES} of type -\code{METADATA} with component keywords being the various recipe names -and the values (of type \code{STR}) the corresponding recipe -configuration filename. - -An example: -\begin{verbatim} -# Recipes for LRIS -RECIPES METADATA - PHASE1 STR lris_phase1.config - PHASE2 STR lris_phase2.config - PHASE4 STR lris_phase4.config -END -\end{verbatim} - -\subsubsection{Recipe Configuration} - -\tbd{The contents of the recipe configuration file are dependent upon -the particular module, and hence are not specified here at this time.} - - -\subsection{PS Concepts} - -\subsubsection{Ingest} - -For different camera systems, these concepts are not always known by -the same name, nor are they generally obtained in the same manner, and -so their source or value must be specified in the camera configuration -file. At ingest, the value of a concept shall be found by searching in -the following order: -\begin{itemize} -\item The cell data from the \code{CELLS} metadata in the camera configuration. -\item The FITS header via the \code{TRANSLATION} table. -\item The \code{DATABASE} lookup. -\item The \code{DEFAULTS} value. -\end{itemize} - -\subsubsection{Dependencies for defaults} - -In the \code{DEFAULTS} table in the camera configuration, we allow the -specification of the concept with an additional suffix, \code{DEPEND}. -The value (of type \code{STR}) of the \code{CONCEPT.DEPEND} is the -name of a concept on which the first concept depends. For example, it -might depend on the chip name. Then the first concept becomes of type -\code{METADATA}, with the component keywords being the value of the -second concept (on which the first depends). To avoid infinite -recursion, no further dependency is permitted. We also allow an entry -\code{CONCEPT.DEFAULT} specifiying the default value of the concept if -a match is not made with the dependcency list. An example of the -dependency: - -\begin{verbatim} -# Default PS concepts that may be specified by value -DEFAULTS METADATA - CELL.GAIN.DEPEND STR CHIP.NAME - CELL.GAIN.DEFAULT STR 1.0 - CELL.GAIN METADATA - ccd00 F32 1.2 - ccd01 F32 3.4 - ccd02 F32 5.6 - END -END -\end{verbatim} - -\subsubsection{FORMATS} - -Because of the variety of methods for specifying these concepts -(especially in FITS headers), we must also specify additional -information in the camera configuration that specifies how to -interpret the data provided. These are provided in an entry -\code{FORMATS} (of type \code{METADATA}) in the camera configuration. -Within the \code{FORMATS} metadata, there is a string for each of the -concepts that requires a format to be specified. - -\paragraph{CELL.TIME} - -The time at which the shutter opens is represented in a variety of -ways in FITS files, so care must be taken to specify what the format -is in the file under consideration. Permitted values of -\code{CELL.TIME.FORMAT} are: - -\begin{itemize} -\item \code{JD}: The value pointed to by \code{CELL.TIME} is to be - interpreted as a Julian Date. -\item \code{MJD}: The value pointed to by \code{CELL.TIME} is to be - interpreted as a Modified Julian Date. -\item \code{ISO}: The value pointed to by \code{CELL.TIME} is to be - interpreted as an ISO date-time (yyyy-mm-ddThh:mm:ss.ss). -\item \code{SEPARATE}: The date and time are specified separately, and - the \code{CELL.TIME} contains the headers for the date and the time - separated by whitespace or a comma. Then it is necessary to add - additional qualifiers to specify the formats of these: - \begin{itemize} - \item \code{PRE2000}: The year is in the old style two-digit format - popular before the year 2000, and it should be assumed that the - date is in the twentieth century. - \item \code{BACKWARDS}: The date is in the format dd-mm-yyyy or - dd/mm/yyyy. - \item \code{SOD}: The time is specified as seconds-of-day. - \end{itemize} -\end{itemize} - -Note that the FITS standard is that the time in the header refers to -the {\it start} of the observation. - -\tbd{the PRE2000 and BACKWARDS qualifiers should be replace with -explicit format definitions in the form YYYY/MM/DD} - -\tbd{In the future, we might add additional qualifiers that calculate -the start time of the observation based on someone foolishly putting -the end- or mid-time in the header.} - -\tbd{Should we move CELL.TIMESYS into the format as well?} - -\paragraph{FPA.RA and FPA.DEC} - -The RA and Declination of the boresight might be specified in a few -ways. We need to specify both how the value is interpreted and the -units. \code{FPA.RA.FORMAT} and \code{FPA.DEC.FORMAT} should be one -of the following: - -\begin{itemize} -\item \code{HOURS}: The value pointed to by the concept should be - interpreted as being in hours. -\item \code{DEGREES}: The value pointed to by the concept should be - interpreted as being in degrees. -\item \code{RADIANS}: The value pointed to by the concept should be - interpreted as being in radians. -\end{itemize} - -How the value is interpreted can be determined from the type of the -header: if it is of type \code{STR}, then we can reasonably assume -that it is in sexagesimal format with colons or spaces as separators; -and if it is of type \code{F32} (or \code{F64}), then we can assume -that it is in decimal format. - -\subsubsection{Implicit format information} - -While details like the units of the right ascension in the header must -be specified explicitly, some other details can be determined from -implicit information. - -\begin{itemize} -\item \code{FPA.RA} and \code{FPA.DEC}: if the value on ingest is of -type \code{STRING}, then it may be interpreted as sexagesimal -notation, ``\code{dd:mm:ss.ss}'', or ``\code{dd:mm.mmm}''. A space -may be used instead of a colon to separate the values. Otherwise, if -the value is of a numerical type (\code{F32} or \code{F64}), then that -is the appropriate value. -\item \code{CELL.XBIN} and \code{CELL.YBIN}: if the value on ingest is -of type \code{STRING}, then it may be interpreted as ``\code{x,y}'', -where \code{x} is the binning in x, and \code{y} is the binning in y. -A space may be used instead of a comma, and there may even be a space -before or after the comma (or both). Otherwise, if the value is of a -numerical type (\code{S32}, etc), then that is the appropriate value. -\item \code{CELL.BIASSEC} and \code{CELL.TRIMSEC}: These values on -ingest should always be of type \code{STRING}. If they contain a -square bracket, then they may be interpreted as a list of standard -region specifications, ``\code{[x0:x1,y0:y1];[x2:x3,y2:y3];...}'', -where the semi-colon may be replaced by spaces. Otherwise, the string -may be interpreted as a FITS header (or headers, separated by spaces, -commas or semi-colons) that contains the appropriate values. -\end{itemize} - -\tbd{the use of implicit interpretation of formats should be - discouraged: format interpretation guides should be provided} - -\subsection{Configuration APIs} - -\begin{prototype} -bool pmConfigRead(psMetadata **site, psMetadata **camera, psMetadata **recipe, - int *argc, char **argv, const char *recipeName); -psMetadata *pmConfigCameraFromHeader(const psMetadata *site, const psMetadata *header); -psMetadata *pmConfigRecipeFromCamera(const psMetadata *camera, const char *recipeName); -\end{prototype} - -\code{pmConfigRead} shall load the \code{site} configuration -(according to the above rule for determining the source). The -\code{camera} configuration shall also be loaded if it is specified on -the command line (\code{argc, argv}); otherwise it shall be set to -\code{NULL}. The \code{recipe} shall also be loaded from the command -line (if specified) or, if the camera configuration has been loaded, -from the camera configuration and recipe specification therein (see -below). In dealing with the command line parameters, the functions -shall use the appropriate functions in psLib to retrieve and remove -the relevant options from the argument list; this simplifies -assignment of the mandatory arguments, since all the optional command -line arguments are removed leaving only the mandatory arguments. The -following psLib setups shall also be performed if they are specified -in the site configuration: -\begin{itemize} -\item the function shall call \code{psTimeInitialize} with the - configuration file specified by \code{TIME}. -\item the function shall call \code{psLogSetLevel} with the logging - level specified by \code{LOGLEVEL}. -\item the function shall call \code{psLogSetFormat} with the log - format specified by \code{LOGFORMAT}. -\item the function shall call \code{psTraceSetLevel} with the component names and - trace levels specified by the \code{TRACE}. -\end{itemize} -Note that additional log/trace command-line options may be specified -and interpretted using the \code{psArgumentVerbosity} function from -psLib. These options should (in the case of logging) override the -configuration-supplied information or (in the case of tracing) -supplement it. - -\code{pmConfigCameraFromHeader} shall load the \code{camera} -configuration based on the contents of the FITS \code{header}, using -the list of known cameras contained in the \code{site} configuration. -If more than one camera matches the FITS header, a warning shall be -generated and the first matching camera returned. - -\code{pmConfigRecipeFromCamera} shall load the \code{recipe} -configuration based on the \code{recipeName} and the list of known -recipes contained in the \code{camera} configuration (details below). - -\begin{prototype} -bool pmConfigValidateCamera(const psMetadata *camera, const psMetadata *header); -\end{prototype} - -This function, used by \code{pmConfigCameraFromHeader}, shall return -\code{true} if the FITS \code{header} matches the rule contained in -the \code{camera} configuration (see \S\ref{sec:camerarule}); -otherwise it shall return \code{false}. - -\begin{prototype} -psDB *pmConfigDB(psMetadata *site); -\end{prototype} - -\code{pmConfigDB} shall use the \code{site} configuration data to open -a database handle. \tbd{This is fairly straightforward at the moment, -but will change when we beef up security.} - -\subsubsection{Example usage} - -The following is provided as an example of how the above functions -are envisioned in use. - -\begin{verbatim} -int main(int argc, char *argv[]) -{ - // Parse other command-line arguments here - psMetadata *site = NULL; // Site configuration - psMetadata *camera = NULL; // Camera configuration - psMetadata *recipe = NULL; // Recipe configuration - if (! pmConfigRead(&site, &camera, &recipe, &argc, argv, "moduleName")) { - psLogMsg("moduleName", PS_LOG_ERROR, "Can't find site configuration!\n"); - exit(EXIT_FAILURE); - } - // Parse other command-line arguments here - - // The command-line argument list now contains only mandatory arguments - // Assume the first of these is an input image - char *imageName = argv[1]; // Name of FITS file - psFits *imageFH = psFitsOpen(imageName, "r"); // File handle for FITS file - if (! imageFH) { - psLogMsg("moduleName", PS_LOG_ERROR, "Can't open input image %s\n", imageName); - exit(EXIT_FAILURE); - } - psMetadata *header = psFitsReadHeader(NULL, imageFH); // FITS header - - if (!camera && !(camera = pmConfigCameraFromHeader(site, header))) { - psLogMsg("moduleName", PS_LOG_ERROR, "Can't find camera configuration!\n"); - exit(EXIT_FAILURE); - } - - if (! recipe && !(recipe = pmConfigRecipeFromCamera(camera, "moduleName"))) { - psLogMsg("moduleName", PS_LOG_ERROR, "Can't find recipe configuration!\n"); - exit(EXIT_FAILURE); - } - - // Now go on and do stuff - .... -} -\end{verbatim} +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Configuration} + +Configuration information in the IPP is provided on four levels: the +site, camera, format and recipe configurations. The use of these, and +their contents are described in an additional document, PSDC-430-???, +``Pan-STARRS PS1 IPP Configurations''. Here we define functions that +read these configuration files. + + +\subsubsection{Configuration information} + +This structure stores the configuration information: the site, camera +and recipe configuration, the command-line arguments, the pmFPAfiles +used, and the database handle. +\begin{datatype} +typedef struct { + psMetadata *site; ///< Site configuration + psMetadata *camera; ///< Camera specification + const char *cameraName; ///< Camera name + psMetadata *recipes; ///< Recipes for processing + psMetadata *arguments; ///< Processed command-line arguments + psMetadata *files; ///< pmFPAfiles used for analysis + psDB *database; ///< Database handle + int *argc; ///< Number of command-line arguments + char **argv; ///< Command-line arguments (raw version) + // Private members + p_pmRecipeSource recipesRead; ///< Which recipe sources have been read + psMetadata *recipesSource; ///< Where each recipe came from +} pmConfig; +\end{datatype} + +The site, camera and recipe configurations are represented here. The +camera format configuration is stored elsewhere (within the +\code{pmFPA}). + +Note that the \code{arguments} are different from the \code{argc} and +\code{argv}, since it is the parsed version. This metadata may also +be used to carry program options, if desired. + +The allocator for this structure is: +\begin{prototype} +pmConfig *pmConfigAlloc(int *argc, /// Number of command-line arguments + char **argv /// Command-line arguments + ); +\end{prototype} + + +\subsection{Reading configuration files} + +Read configuration information from the command line: +\begin{prototype} +pmConfig *pmConfigRead(int *argc, ///< Number of command-line arguments + char **argv ///< Array of command-line arguments + ); +\end{prototype} +pmConfigRead loads the site configuration (the file name is specified +by "-site SITE_FILE" on the command-line, the PS_SITE environment +variable, or it is \code{$HOME/.ipprc}). The configuration search %$ +path is set. The camera configuration is loaded if it is specified on +the command line ("-camera CAMERA_FILE"). Recipes specified on the +command line ("-recipe RECIPE_NAME RECIPE_SOURCE") are also loaded. +These command-line arguments are removed from from the command-line, +to simplify parsing. The psLib log, trace and time setups are also +performed if specified in the site configuration. + +Read a configuration file: +\begin{prototype} +bool pmConfigFileRead(psMetadata **config, ///< Config to output + const char *name, ///< Name of file + const char *description ///< Description of file + ); +\end{prototype} +Read a metadata configuration file into the supplied metadata. +Produce an error and return false if there's a problem. + +Set static configuration information: +\begin{prototype} +void pmConfigSet(const char *path ///< Search paths for configuration files; colon-delimited directories + ); +\end{prototype} +The search path for the configuration files is a local static +variable, set by this function. + +Free static memory used in the configuration system: +\begin{prototype} +void pmConfigDone(void); +\end{prototype} + + +\subsection{Camera configuration} + +Validate a header against the camera format: +\begin{prototype} +bool pmConfigValidateCameraFormat(const psMetadata *cameraFormat, ///< Camera format containing the RULE + const psMetadata *header // FITS header for the PHU + ); +\end{prototype} +Given a FITS header (the PHU header), check it against the RULE +metadata contained within the camera format; return true if it +matches. + +Determine the camera format (and camera if unknown) from examining the +header: +\begin{prototype} +psMetadata *pmConfigCameraFormatFromHeader(pmConfig *config, ///< The configuration + const psMetadata *header ///< The FITS header + ); +\end{prototype} +Given a FITS header, check it against all known cameras (unless we +already know which camera, from pmConfigRead) and all known formats +for those cameras in order to identify which is appropriate. The +first matching format is accepted; further matches produce warnings. +The accepted camera is saved in the configuration. The accepted +format is returned. + +Return the camera configuration specified by name: +\begin{prototype} +psMetadata *pmConfigCameraByName(pmConfig *config, ///< The configuration + const char *cameraName ///< The camera name header + ); +\end{prototype} +Given a camera name, returns the camera configuration metadata. + +Make the supplied header conform to the nominated camera format: +\begin{prototype} +bool pmConfigConformHeader(psMetadata *header, ///< Header to conform + const psMetadata *format ///< Camera format + ); +\end{prototype} +Given a FITS header, make it conform to the RULE in the specified +camera format. This is useful for switching between formats, or +generating fake data that must be recognised by +pmConfigCameraFormatFromHeader. + + +\subsection{Recipes} + +Recipes are read from a variety of sources: the command line, the site +configuration, and the camera configuration. In addition, recipes +specified on the command line may be a symbolic link to recipes +defined in the other locations. Since the various sources of recipes +may be read at different times (e.g., the camera configuration may be +read from the command line, or it may only be read once a FITS file +has been opened for inspection), we specify the following list of +recipe sources: +\begin{datatype} +typedef enum { + P_PM_RECIPE_SOURCE_NONE = 0x00, ///< None yet + P_PM_RECIPE_SOURCE_SITE = 0x01, ///< Site configuration + P_PM_RECIPE_SOURCE_CAMERA = 0x02, ///< Camera configuration + P_PM_RECIPE_SOURCE_CL = 0x04, ///< Command-line + P_PM_RECIPE_SOURCE_SYMBOLIC = 0x14, ///< Symbolic link, specified on command-line + P_PM_RECIPE_SOURCE_ALL = 0xff ///< All sources +} p_pmRecipeSource; +\end{datatype} +This structure is private to psModules --- there is no need for the +user to know about it. + +See PSDC-430-???, ``IPP Configurations'' for information on symbolic +links, and how these are identified and resolved, and also for the +precedence of sources. + +Read recipes: +\begin{prototype} +bool pmConfigReadRecipes(pmConfig *config ///< Configuration + ); +\end{prototype} +Attempt to read recipes from the sources that are available but have +not already been read.Having read a recipe, attempt to resolve +symbolic links that were specified on the command line. + + +\subsection{Database} + +Setup the database: +\begin{prototype} +psDB *pmConfigDB(pmConfig *config ///< Configuration + ); +\end{prototype} +Initialise the database connection using the DBSERVER, DBNAME, DBUSER, +DBPASSWORD values provided in the site configuration. Stores the +database handle in the configuration, and also returns it. + + +\subsection{File sets} + +Read the command-line for files (or a text file containing a list of +files): +\begin{prototype} +psArray *pmConfigFileSets(pmConfig *config, ///< Configuration, containing command-line arguments + const char *file, ///< CL argument specifying a filename + const char *list ///< CL argument specifying a text file with a list of filenames + ); +\end{prototype} +Given the 'file' and 'list' arguments (e.g., "-file" and "-list"), +find the arguments associated with these words and interpret them as +lists of files. Return an array of the resulting filenames. + +Stuff associated files from the command-line into a metadata: +\begin{prototype} +bool pmConfigFileSetsMD(psMetadata *metadata, ///< Metadata into which to stuff the array + pmConfig *config, ///< Configuration (which command-line arguments) + const char *name, ///< Name for array in the metadata + const char *file, ///< CL argument specifying a filename + const char *list ///< CL argument specifying a text file with a list of filenames + ); +\end{prototype} +Calls pmConfigFileSets to parse the command line for filenames (or a +list which provides filenames), and stuffs the array of filenames into +the metadata under "name". %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -984,13 +304,13 @@ plane hierarchy. -After ingest (performed in \code{pmFPARead}, the user may safely -assume that all of the above concepts exist at the appropriate level -(meaning the user needn't be hampered by excessive error checking), is -of the specified type (meaning the user doesn't need to worry about -whether the value of interest is stored in, e.g., floating point or -double precision or even a colon-delimited string) and in the -specified format (meaning the user doesn't need to know, e.g., whether -the right ascension is in radians or degrees) --- all the conversions -are handled by the ``concepts'' functions at ingest. +After ingest, the user may safely assume that all of the above +concepts exist at the appropriate level (meaning the user needn't be +hampered by excessive error checking), is of the specified type +(meaning the user doesn't need to worry about whether the value of +interest is stored in, e.g., floating point or double precision or +even a colon-delimited string) and in the specified format (meaning +the user doesn't need to know, e.g., whether the right ascension is in +radians or degrees) --- all the conversions are handled by the +``concepts'' functions at ingest. Most of the structures and functions in this section are intended to @@ -1007,49 +327,74 @@ \begin{datatype} -typedef psMetadataItem* (*p_pmConceptReadFunc)(pmFPA *fpa, pmChip *chip, pmCell *cell, psDB *db); -typedef bool (*p_pmConceptWriteFunc)(pmFPA *fpa, pmChip *chip, pmCell *cell, psDB *db); typedef struct { - psMetadataItem *blank; // Blank value of concept; also contains the name - p_pmConceptReadFunc read; // Function to call to read the concept - p_pmConceptWriteFunc write; // Function to call to write the concept -} p_pmConceptSpec; + psMetadataItem *blank; ///< Blank value of concept; also contains the name and comment + pmConceptParseFunc parse; ///< Function to call to read the concept + pmConceptFormatFunc format; ///< Function to call to write the concept +} pmConceptSpec; \end{datatype} -\code{blank} is a \code{psMetadataItem} that provides the name, type -and default/blank value for the concept. \code{read} and \code{write} -provide the functions to read and write. - -A concept specification may be allocated: -\begin{prototype} -p_pmConceptSpec *p_pmConceptSpecAlloc(psMetadataItem *blank, pmConceptReadFunc read, - pmConceptWriteFunc write); -\end{prototype} +The allocator is: +\begin{prototype} +pmConceptSpec *pmConceptSpecAlloc(psMetadataItem *blank, ///< Blank value; contains the name + pmConceptParseFunc parse, ///< Function to call to parse the concept + pmConceptFormatFunc format ///< Function to call to format the concept + ); +\end{prototype} + +\code{blank} provides the name, type and default/blank value for the +concept. \code{parse} and \code{format} provide the functions to +parse and format the concepts: + +\begin{datatype} +typedef psMetadataItem* (*pmConceptParseFunc)(const psMetadataItem *concept, ///< Concept to parse + const psMetadataItem *pattern, ///< Pattern for parsing + const psMetadata *cameraFormat, ///< Camera format definition + const pmFPA *fpa, ///< FPA for concept, or NULL + const pmChip *chip, ///< Chip for concept, or NULL + const pmCell *cell ///< Cell for concept, or NULL + ); +typedef psMetadataItem* (*pmConceptFormatFunc)(const psMetadataItem *concept, ///< Concept to format + const psMetadata *cameraFormat, ///< Camera format definition + const pmFPA *fpa, ///< FPA for concept, or NULL + const pmChip *chip, ///< Chip for concept, or NULL + const pmCell *cell ///< Cell for concept, or NULL + ); +\end{datatype} + +A concept that has just been read requires parsing, so that the final +product stored in the \code{concepts} metadata is of the correct type +and format. The parsing function requires a \code{pattern} (generally +the \code{blank}), which provides the name and comment for the parsed +concept. The camera format (\code{cameraFormat}) allows looking up of +the \code{FORMATS} or other information, and the \code{fpa}, +\code{chip} and \code{cell} allow lookup of other concepts. + +Before writing, concepts are formatted so that the source's format +(likely differing from that of the psModules concepts) may be +respected. Again, providing the camera format (\code{cameraFormat}), +\code{fpa}, \code{chip} and \code{cell} allow the formatting function +to access information necessary to format. + \subsection{Registering a concept} -The concept specifications that have been registered shall be stored in -three \code{psMetadata}s, one for each level (FPA, chip, cell). +The concept specifications that have been registered are stored in +three (static) \code{psMetadata}s, one for each level (FPA, chip, +cell). Registering a concept is achieved by: \begin{prototype} -bool pmConceptRegister(psMetadataItem *blank, pmConceptReadFunc read, - pmConceptWriteFunc write, pmConceptLevel level); -\end{prototype} - -\code{pmConceptRegister} shall generate a concept specification from -the provided \code{blank}, and \code{read} and \code{write} functions, +bool pmConceptRegister(psMetadataItem *blank, ///< Blank value; contains the name and default comment + pmConceptParseFunc parse, ///< Function to call to parse the concept, or NULL + pmConceptFormatFunc format, ///< Function to call to format the concept, or NULL + pmFPALevel level ///< Level at which to store concept in the FPA hierarchy + ); +\end{prototype} + +\code{pmConceptRegister} generates a concept specification from the +provided \code{blank}, and \code{parse} and \code{format} functions, and register it in the metadata specified by the \code{level}. -\code{pmConceptLevel} simply specifies which level in the focal plane -hiearchy the concept applies to: -\begin{datatype} -typedef enum { - PM_CONCEPT_LEVEL_FPA, // Store in the FPA - PM_CONCEPT_LEVEL_CHIP, // Store in the chip - PM_CONCEPT_LEVEL_CELL // Store in the cell -} pmConceptLevel; -\end{datatype} - -A \code{read} function of \code{NULL} indicates that there is no +A \code{parse} function of \code{NULL} indicates that there is no special interpretation of the concept required, and that it can be used as read. A \code{write} function of \code{NULL} indicates that @@ -1060,34 +405,44 @@ \subsection{Default concepts} -Below is a list of concepts that the IPP will use, with the expected -type and a short description. +Below is a list of concepts that are defined in the psModules, with +the expected type and a short description. \begin{itemize} -\item \code{FPA.NAME} (\code{psString}): An identifier (e.g., observation number) for the FPA instance -\item \code{FPA.AIRMASS} (F32): Airmass at which the observation is made (boresight) -\item \code{FPA.FILTER} (\code{psString}): Filter used in observation -\item \code{FPA.POSANGLE} (F32): Position angle for camera -\item \code{FPA.RADECSYS} (\code{psString}): System of RA,Dec (e.g., J2000 or ICRS) -\item \code{FPA.RA} (F64): Right Ascension of boresight in radians -\item \code{FPA.DEC} (F64): Declination of boresight in radians -\item \code{CHIP.NAME} (\code{psString}): The name of the chip (unique within the FPA) --- set at FITS read -\item \code{CELL.NAME} (\code{psString}): The name of the cell (unique within the parent chip) --- set at FITS read -\item \code{CELL.GAIN} (F32): CCD gain (e/ADU) +\item \code{FPA.CAMERA} (STR): Camera used +\item \code{FPA.FOCUS} (F32): Telescope focus +\item \code{FPA.AIRMASS} (F32): Airmass at boresight +\item \code{FPA.FILTER} (STR): Filter used +\item \code{FPA.POSANGLE} (F32): Position angle of instrument +\item \code{FPA.RADECSYS} (STR): Celestial coordinate system +\item \code{FPA.RA} (F64): Right Ascension of boresight +\item \code{FPA.DEC} (F64): Declination of boresight +\item \code{FPA.OBSTYPE} (STR): Type of observation +\item \code{FPA.OBJECT} (STR): Object of observation +\item \code{FPA.ALT} (F64): Altitude of telescope +\item \code{FPA.AZ} (F64): Azimuth of telescope +\item \code{FPA.TIMESYS} (STR): Time system +\item \code{FPA.TIME} (psTime): Time of exposure +\item \code{CHIP.XPARITY} (S32): Orientation in x compared to the rest of the FPA, -1 or +1 +\item \code{CHIP.YPARITY} (S32): Orientation in y compared to the rest of the FPA, -1 or +1 +\item \code{CHIP.X0} (S32): Position of (0,0) on the FPA +\item \code{CHIP.Y0} (S32): Position of (0,0) on the FPA +\item \code{CHIP.TEMP} (F32): Temperature of chip +\item \code{CELL.GAIN} (F32): CCD gain (e/count) \item \code{CELL.READNOISE} (F32): CCD read noise (e) -\item \code{CELL.SATURATION} (F32): CCD saturation point (ADU) -\item \code{CELL.BAD} (F32): CCD bad pixel point (ADU) -\item \code{CELL.XPARITY} (S32): Direction of CCD readout in x relative to the rest of the chip -\item \code{CELL.YPARITY} (S32): Direction of CCD readout in y relative to the rest of the chip -\item \code{CELL.READDIR} (S32): Read direction: line (1) or column (2) -\item \code{CELL.EXPOSURE} (F32): Exposure time of image (sec) -\item \code{CELL.DARKTIME} (F32): Dark time for image (sec) -\item \code{CELL.TRIMSEC} (\code{psRegion*}): Trim region -\item \code{CELL.BIASSEC} (\code{psList*} of \code{psRegion*}): Overscan region(s) -\item \code{CELL.XBIN} (S32): CCD binning in x -\item \code{CELL.YBIN} (S32): CCD binning in y -\item \code{CELL.TIMESYS} (\code{psTimeType}): Time system in use -\item \code{CELL.TIME} (\code{psTime*}): Time of observation start -\item \code{CELL.X0} (S32): x position of cell (0,0) on the chip -\item \code{CELL.Y0} (S32): y position of cell (0,0) on the chip +\item \code{CELL.SATURATION} (F32): Saturation level (counts) +\item \code{CELL.BAD} (F32): Bad level (counts) +\item \code{CELL.XPARITY} (S32): Orientation in x compared to the rest of the chip, -1 or +1 +\item \code{CELL.YPARITY} (S32): Orientation in y compared to the rest of the chip, -1 or +1 +\item \code{CELL.READDIR} (S32): Read direction, rows=1, cols=2 +\item \code{CELL.EXPOSURE} (F32): Exposure time (sec) +\item \code{CELL.DARKTIME} (F32): Time since flush (sec) +\item \code{CELL.TRIMSEC} (psRegion): Trim section +\item \code{CELL.BIASSEC} (psList of psRegions): Bias sections +\item \code{CELL.XBIN} (S32): Binning in x +\item \code{CELL.YBIN} (S32): Binning in y +\item \code{CELL.TIMESYS} (STR): Time system +\item \code{CELL.TIME} (psTime): Time of exposure +\item \code{CELL.X0} (S32): Position of (0,0) on the chip +\item \code{CELL.Y0} (S32): Position of (0,0) on the chip \end{itemize} @@ -1098,5 +453,5 @@ dark time.} -The concept specifications for the above shall be registered by +The concept specifications for the above may be registered by \code{pmConceptsInit}: \begin{prototype} @@ -1106,9 +461,9 @@ Since defined concept specifications are required before any concept ingest can take place, all functions that work with the concepts must -call \code{pmConceptsInit} first. +call \code{pmConceptsInit} first. This is usually performed internal +to the pmConcepts, so the user need not call this function. The concept specification metadata containers and the concept -specifications that have been registered shall all be freed by -\code{pmConceptsDone}: +specifications that have been registered are freed by: \begin{prototype} void pmConceptsDone(void); @@ -1117,31 +472,129 @@ leak, since the metadata containers are defined \code{static}. -\subsection{Reading, Writing and Blanking} - -Reading concepts is the act of determining their values and setting -them in the \code{concepts} metadata in the focal plane hierarchy. -Writing concepts is the act of taking the \code{concepts} metadata -which is in the focal plane hierarchy and preparing them for output. + +\subsection{Concept sources} + +Concepts may be read from, and written to a variety of sources: the +\code{CELLS} information within the camera format, the FITS header +(either the primary header, or extension), default values from the +camera format, or the database. + +Since some sources become available at different times from others, we +need to provide some specificity to reading and writing concepts (or +we're forced to wait until everything's available, which we don't want +to do). Concepts may be read from or written to multiple sources at +once by OR-ing them. + +\begin{datatype} +typedef enum { + PM_CONCEPT_SOURCE_NONE = 0x00, ///< No concepts + PM_CONCEPT_SOURCE_CELLS = 0x01, ///< Concept comes from the camera information + PM_CONCEPT_SOURCE_DEFAULTS = 0x02, ///< Concept comes from defaults + PM_CONCEPT_SOURCE_PHU = 0x04, ///< Concept comes from PHU + PM_CONCEPT_SOURCE_HEADER = 0x08, ///< Concept comes from FITS header + PM_CONCEPT_SOURCE_DATABASE = 0x10, ///< Concept comes from database + PM_CONCEPT_SOURCE_ALL = 0xff ///< All concepts +} pmConceptSource; +\end{datatype} + + +\subsection{Blanking} + By ``blanking'', we mean setting the concepts to a default or blank value (e.g., \code{NaN} for floating point); this takes place before reading, and can be used to set up a focal plane hierarchy without -reading from any particular source. - -The following functions shall read, write or blank (as appropriate) -the concepts at the appropriate level in the focal plane hierarchy: -\begin{prototype} -bool p_pmConceptsReadFPA(pmFPA *fpa); -bool p_pmConceptsReadChip(pmChip *chip); -bool p_pmConceptsReadCell(pmCell *cell); -bool p_pmConceptsWriteFPA(pmFPA *fpa); -bool p_pmConceptsWriteChip(pmChip *chip); -bool p_pmConceptsWriteCell(pmCell *cell); -bool p_pmConceptsBlankFPA(pmFPA *fpa); -bool p_pmConceptsBlankChip(pmChip *chip); -bool p_pmConceptsBlankCell(pmCell *cell); -\end{prototype} - -Under ordinary circumstances, these functions will be called by -\code{pmFPARead}, \code{pmFPAWrite} and \code{pmFPAConstruct}. +reading from any particular source. This is not normally done by the +user, but it is performed on the focal plane hierarchy at construction +(see \code{pmFPAConstruct}). + +The following functions blank the concepts at the appropriate level: + +\begin{prototype} +bool pmConceptsBlankFPA(pmFPA *fpa); +bool pmConceptsBlankChip(pmChip *chip); +bool pmConceptsBlankCell(pmCell *cell); +\end{prototype} + + +\subsection{Reading} + +Reading a concept is the act of looking up the value, parsing it, and +setting the parsed concept in the \code{concepts} metadata in the +focal plane hierarchy at the appropriate level. It is not normally +necessary for the user to call these functions, since they are called +when constructing or reading the FPA (see \code{pmFPAConstruct}, +\code{pmFPARead}). + +The following functions read the concepts at the appropriate level: + +\begin{prototype} +bool pmConceptsReadFPA(pmFPA *fpa, ///< FPA for which to read concepts + pmConceptSource source, ///< Source for concepts + bool propagateDown, ///< Propagate to lower levels? + psDB *db ///< Database handle + ); +bool pmConceptsReadChip(pmChip *chip, ///< Chip for which to read concepts + pmConceptSource source, ///< Source for concepts + bool propagateUp, ///< Propagate to higher levels? + bool propagateDown, ///< Propagate to lower levels? + psDB *db ///< Database handle + ); +bool pmConceptsReadCell(pmCell *cell, ///< Cell for which to read concepts + pmConceptSource source, ///< Source for concepts + bool propagateUp, ///< Propagate to higher levels? + psDB *db ///< Database handle + ); + +\end{prototype} +In each case, we require the hierarchy component for which to read the +concepts, an indication of the source from which to read concepts +(which source has recently become available), the database handle (for +reading concepts from the database), and flags indicating whether we +should propagate the read to higher or lower levels in the hierarchy. + +An additional function reads concepts for as many hierarchy components +as are supplied: +\begin{prototype} +bool pmConceptsRead(pmFPA *fpa, ///< FPA for which to read concepts + pmChip *chip, ///< Chip for which to read concepts, or NULL + pmCell *cell, ///< Cell for which to read concepts, or NULL + pmConceptSource source, ///< The source of the concepts to read + psDB *db ///< Database handle + ); +\end{prototype} + + +\subsection{Writing} + +Writing a concept is the act of formatting the concept for the native +format, and writing it to the appropriate place (here, ``writing'' +often means preparing for writing, by updating the header metadata, or +checking the concept value against the default). The user will not +normally use these functions directly, since they are called when +writing the FPA (see \code{pmFPAWrite}). + +\begin{prototype} +bool pmConceptsWriteFPA(const pmFPA *fpa, ///< FPA for which to write concepts + pmConceptSource source, ///< Source for concepts + bool propagateDown, ///< Propagate to lower levels? + psDB *db ///< Database handle + ); +bool pmConceptsWriteChip(const pmChip *chip, ///< Chip for which to write concepts + pmConceptSource source, ///< Source for concepts + bool propagateUp,///< Propagate to higher levels? + bool propagateDown, ///< Propagate to lower levels? + psDB *db ///< Database handle + ); +bool pmConceptsWriteCell(const pmCell *cell, ///< FPA for which to write concepts + pmConceptSource source, ///< Source for concepts + bool propagateUp, ///< Propagate to higher levels? + psDB *db ///< Database handle + ); +\end{prototype} +In each case, we require the hierarchy component for which to read the +concepts, an indication of the source to which to write concepts +(which source requires updating), the database handle (for writing +concepts to the database), and flags indicating whether we should +propagate the write to higher or lower levels in the hierarchy. @@ -1150,10 +603,10 @@ The values of concepts may be copied from one source to another: \begin{prototype} -bool pmFPACopyConcepts(pmFPA *target, pmFPA *source); -\end{prototype} - -\code{pmFPACopyConcepts} shall iterate through the focal plane -hierarchy, copying the values of the concepts from the \code{source} -to the \code{target}. +bool pmFPACopyConcepts(pmFPA *target, const pmFPA *source); +\end{prototype} + +\code{pmFPACopyConcepts} iterates through the focal plane hierarchy, +copying the values of the concepts from the \code{source} to the +\code{target}.