Index: trunk/doc/pslib/psLibSDRS.tex =================================================================== --- trunk/doc/pslib/psLibSDRS.tex (revision 314) +++ trunk/doc/pslib/psLibSDRS.tex (revision 320) @@ -1,8 +1,9 @@ -%%% $Id: psLibSDRS.tex,v 1.14 2004-03-30 05:40:01 price Exp $ +%%% $Id: psLibSDRS.tex,v 1.15 2004-03-31 21:27:48 eugene Exp $ \documentclass[panstarrs]{panstarrs} %\documentclass[panstarrs]{panstarrs} % basic document variables -\title{Pan-STARRS IPP Library Design} +\title{Pan-STARRS IPP Library\\ +Supplementary Design Requirements} \author{Paul Price, Eugene Magnier, Robert Lupton} \shorttitle{PSLib Design} @@ -24,40 +25,89 @@ % use \hline between each table row \RevisionsStart -% version Date Description -\theversion & 2003 Mar 11-- & Draft \\ +% version Date Description +DR-1 & 2003 Mar 11 & Draft \\ +\hline +DR-2 & 2003 Mar 29 & Reorganized, spell-checked \\ \RevisionsEnd -\pagebreak +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\DocumentsInternal +PSCD-430-xxx & PS-1 Design Reference Mission \\ +\hline +PSCD-430-004 & Pan-STARRS IPP C Code Conventions \\ +\hline +PSCD-430-005 & Pan-STARRS IPP SRS \\ +\hline +PSCD-430-006 & Pan-STARRS IPP ADD \\ +\hline +PSCD-430-007 & Pan-STARRS IPP PSLib SDR \\ +\hline +PSCD-430-008 & Pan-STARRS IPP Architecture SDR \\ + +\DocumentsExternal +Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ +\DocumentsEnd + \tableofcontents - \pagebreak \pagenumbering{arabic} -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{\PS{} Library (PSLib)} - -PSLib is a library of basic functions required for IPP (and MOPS) -operations. We expect that it will be, to a large extent, the major -workhorse of the IPP. - -PSLib will consist of a variety of data structures and associated APIs -to perform common functions. It will provide low-level capabilities -in each of the following areas: +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Introduction} + +This document describes the Pan-STARRS Image Processing Pipeline (IPP) +Toolkit Library, PSLib. Any large software project such as the IPP +benefits from the existence of a library of basic software functions +which can be used throughout the project to simplify programming +tasks. Among the benefits are the ability to reuse code, +simplification of the testing process, streamlining of the code, and +the isolation and encapsulation of concepts which may be subject to +change. The component functions of such a library should represent +well-defined, concise operations which can be coded with only a modest +number of lines. PSLib is a library of basic functions required by +the IPP, but should include many programming concepts which may be useful +for other software projects, especially those which deal with +astronomical data handling tasks. + +PSLib consists of a collection of library function calls -- the +Application Programming Interfaces (APIs) -- and the associated data +structures. The capabilities provided by PSLib are grouped into the +following areas: +% \begin{itemize} -\item System utilities; -\item Data containers; -\item Data manipulation; and -\item Astronomy-specific tasks. +\item System Utilities +\item Basic Data Collections +\item Data Manipulation +\item Astronomy-Specific Functions. \end{itemize} - -Below we sketch out approximately 200~APIs which specify the required -capabilities. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +This list is sorted in a hierarchical order: the later entries depend +on the functions and data types of the earlier entries. + +The installed base of code for PSLib consists of header files, the +binary library code, \code{libpslib.a} and the shared-library +equivalent, \code{libpslib.so}. Assuming these components have been +installed into the library and search path, PSLib may be used within a +program by including the line \code{#include } into the C +code and linking with \code{-lpslib}. + +This document describes the data structures and details the functions +calls. The specified data structures and functions follow the naming +conventions detailed in the IPP Code Conventions (PSDC-430-004). In +particular, these coding conventions restrict the namespace used by +the library functions by requiring that all globally visible symbols +start with the two letters \code{ps}. Further namespace organization +is achieved by encouraging functions to be named in the form +psNounVerbPhrase, where Noun is the data type of principle relevance +and VerbPhase describes the operation applied to that data type. For +example, the function which copies an image (of type \code{psImage}) +is called \code{psImageCopy()}. + +\tbd{discuss the use of external libraries (list them)} + +\tbd{discussion of thread safe?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{System Utilities} @@ -70,12 +120,10 @@ The \PS{} software system will need a level of memory management placed between the operating system (\code{malloc}/\code{free}) and -the high level routines (e.g.\ \code{psMetaDataAlloc}). - -This layer is in addition to the possibility that specific heavily -used data types may need their own special-purpose memory managers; -but as we have specified that all user-level objects be allocated via -\code{typeAlloc/typeFree} functions, we will easily be able to -implement such functionality without impacting the facilities -described here. +the high level routines (e.g.\ \code{psMetaDataAlloc}). This layer is +in addition to the possibility that specific heavily used data types +may need their own special-purpose memory managers. However, since we +require that all user-level objects be allocated via associated +\code{Alloc/Free} functions, we will easily be able to implement such +functionality without impacting the facilities described here. \subsubsection{Rationale} @@ -107,5 +155,5 @@ \item While debugging complex scientific code, it is very useful to be - able to trace a given data structure as it passed through the + able to trace a given data structure as it passes through the processing pipeline. @@ -115,266 +163,280 @@ \end{itemize} -\subsubsection{A Minimal Specification} - -The previous section laid out a number of desiderata for a memory -management system. Rather than implement our own heap-manager at this -stage of the project, we propose to put in place a sufficient set of -data structures and APIs, but initially implement only a simple subset -(e.g.\ that we not implement a multi-bucket memory manager that takes -in 100Mb chunks from the system and performs sophisticated -defragmentation and garbage collection). - -Subject to agreement with the IfA, some of the functions proposed -here may be omitted from the initial implementation; in particular -the routines \code{psMemCheckCorruption} and \code{psMemCheckLeaks} -may be hard to build efficiently in a simple implementation based -upon adding extra fields to allocation requests. - -\paragraph{Extra information to be saved by Memory Allocation Functions} - -It is required that a table of all allocated memory blocks be -maintained by the \PS{} memory system. - -Each allocated block should preserve at least the information present -in the following struct definition: +\subsubsection{Memory Management} + +In the following sections, we specify the API set, and define the +appropriate data structures, needed by the PSLib memory management +system in order to meet the requirements specified by the desiderata +listed above. + +Within the PSLib memory management system, every allocated memory +block which is provided to the user is bounded by two additions memory +segments. The segment preceeding the user-memory contains data +describing the allocated block, using the \code{psMemBlock} structure. +The final element of this structure is a \code{void} pointer called +\code{magic} and is assigned a special value, \code{PS_MEM_MAGIC}. +The segment following the user-memory block consists of a single +\code{void} pointer, and is also assigned the special value of +\code{PS_MEM_MAGIC}. + +In practice, these bounding memory blocks mean that when a user is +requests $N$ bytes of memory, the memory management system in fact +allocates \code{N + sizeof(psMemBlock) + sizeof(void)} bytes, starting +at a paricular address, \code{ADDR}. It then fills in the first +\code{sizeof(psMemBlock)} bytes with the data of the \code{psMemBlock} +structure, and the last \code{sizeof(void)} bytes with the +\code{PS_MEM_MAGIC}. It returns to the user the pointer corresponding +to the address \code{ADDR + sizeof(psMemBlock)}. If the memory +management system reallocates a block of memory, it must also allocate +the additional space and fill in the boundary values. If the memory +management system is give a specific pointer for some operation, it is +able to find the corresponding \code{psMemBlock} by simply subtracting +\code{sizeof(psMemBlock)} from the pointer address. + +The purpose of the two boundary markers is to catch corruption and to +act as an aid in low-level debugging. In the first case, memory over- +and under-run errors are likely to overwrite the special values in +either the leading or trailing boundaries. The typical situation is +one where the coder mis-counts the range and either fills the data +just before the start of the valid memory or just after the end of the +valid memory. These actions will alter the boundary-post values and +can be detected by the memory management system. In the second case, +hexidecimal dumps of large blocks of memory are easier to examine if +the value of \code{PS_MEM_MAGIC} is chosen to catch the eye. A +traditional value for \code{PS_MEM_MAGIC} is \code{0xdeadbeef} which +is also easily recognized in a dump of the memory table. + +The PSLib memory management system must maintain a private table of +the allocated memory blocks. The table includes a list of pointers to +structures of type \code{psMemBlock}, defined as follows: +% \begin{verbatim} typedef struct { - const void *magic0; //!< initialised to p_psMEMMAGIC - const unsigned long id; //!< a unique ID for this allocation - const char *file; //!< set from __FILE__ in e.g. p_psAlloc - const int lineno; //!< set from __LINE__ in e.g. p_psAlloc - int refCounter; //!< how many times pointer is referenced - const void *magic; //!< initialised to p_psMEMMAGIC + const unsigned long id; ///< a unique ID for this allocation + const char *file; ///< set from __FILE__ in e.g. p_psAlloc + const int lineno; ///< set from __LINE__ in e.g. p_psAlloc + int refCounter; ///< how many times pointer is referenced + int state; ///< current state of memory block + const void *magic; ///< initialised to PS_MEM_MAGIC } psMemBlock; \end{verbatim} - -The value of \code{P_PS_MEMMAGIC} shall be \code{(void *)0xdeadbeef}% -\footnote{Why this choice? Tradition, and because it's easy to notice - in a hex dump.} - -\paragraph{APIs for using Memory Allocation Functions} - -The types and function prototypes for the part of the memory API -concerned with allocating and freeing memory are shown below. - -\begin{verbatim} -/// Memory allocation. Underlying private function called by macro psAlloc. -void *p_psAlloc(size_t size, //!< Size required - const char *file, //!< File of call - int lineno //!< Line number of call - ); - -/// Memory re-allocation. Underlying private function called by macro psRealloc. -void *p_psRealloc(void *ptr, //!< Pointer to re-allocate - size_t size, //!< Size required - const char *file, //!< File of call - int lineno //!< Line number of call - ); - -/// Free memory. Underlying private function called by macro psFree. -void p_psFree(void *ptr, //!< Pointer to free - const char *file, //!< File of call - int lineno //!< Line number of call - ); - -/// Memory allocation. psAlloc sends file and line number to p_psAlloc. -#define psAlloc(size) p_psAlloc(size, __FILE__, __LINE__) - -/// Memory re-allocation. psRealloc sends file and line number to p_psRealloc. -#define psRealloc(ptr, size) p_psRealloc(ptr, size, __FILE__, __LINE__) - -/// Free memory. psFree sends file and line number to p_psFree. -#define psFree(size) p_psFree(size, __FILE__, __LINE__) -\end{verbatim} - -N.b. +% +The first element in the structure is a sequential memory block ID. +The memory management system must maintain an internal memory block ID +counter from which a new ID may be supplied to each newly allocated +block of memory and saved in the element \code{id}. This ID should +also be the key to the memory block in the memory block table. + +The two entries \code{file} and \code{lineno} are set to the line +number and file at which the memory was originally allocated. This is +most easily implemented by the use of the C preprocessor macros +\code{__LINE__} and \code{__FILE__}. For this reason, we specify +below that the basic memory managment functions be implemented as +preprocessor macros which wrap the intrinsic C level functions. + +The \code{psMemBlock} structure element \code{refCounter} is provided +so APIs may cleanly manage multiple references to the same block of +memory. As discussed below, the basic free function, \code{psFree}, +is specified to free the memory block only if the reference counter is +set to 1. See the discussion of the \code{psDlist} and \code{psHash} +data containers for an example of the usage. Usage of this feature is +strongly encouraged, but not enforced by the memory management system. + +The element \code{state} defines the state of the memory block. In +order to trace double frees and other memory errors, the memory block +reference is not automatically deleted when the assocated memory is +deleted. Valid states are \code{PS_MEM_FREE} and \code{PS_MEM_ALLOC}. + +\subsubsection{APIs for Allocation and Freeing} + +PSLib provides the following APIs to create and destroy memory blocks: +% +\begin{verbatim} +void *psAlloc(size_t size); +void *psRealloc(void *ptr, size_t size); +void *psFree(void *ptr); +\end{verbatim} +% +From the user's perspective, the functions \code{psAlloc}, +\code{psRealloc}, and \code{psFree} have identical semantics to the +standard C library functions \code{malloc}, \code{realloc}, and +\code{free}. In fact, these functions shall be implemented as C +preprocessor macros which call the following private functions: +% +\begin{verbatim} +void *p_psAlloc(size_t size, char *file, int line); +void *p_psRealloc(void *ptr, size_t size, char *file, int line); +void p_psFree(void *ptr, char *file, int line); +\end{verbatim} +% +In these function calls, \code{size} is the number of bytes required +to be allocated, \code{ptr} is the pointer to the allocated memory +block, \code{file} is the file containing the calling function (set by +\code{__FILE__}), and \code{line} is the calling function line number +in the source-code file (set by \code{__LINE__}). Because the user +should only see the preprocessor versions (ie, \code{psAlloc}), we do +not distinguish between these two in the following discussion. + +In order to enforce the use of the PSLib versions, the header file +shall take steps to ensure that code calling the functions +\code{malloc}, \code{calloc}, \code{realloc}, or \code{free} will not +compile. This may be achieved by defining preprocessor macros which +mask these functions with invalid statements (\eg{} \code{#define +malloc(S) for}). In exceptional cases, such as the memory management +system itself, programmers may choose to override this prohibition by +defining the symbol \code{PS_ALLOW_MALLOC}. Application code will +call \code{p_psAlloc,p_psRealloc,p_psFree} via the macros defined +above. + +The functions \code{psAlloc} and \code{psRealloc} shall never return a +\code{NULL} pointer. If they are unable to provide the requested +memory they should attempt to obtain the desired memory by calling the +routine registered by \code{psMemExhaustedSetCB} (see subsubsection +\ref{secMemAdvanced}), and if still unsuccessful, call \code{psAbort()}. + +Note that we have not specified an equivalent of the \code{calloc} +function. The \code{calloc} function provides two aspects which +\code{malloc} originally did not: aligned memory and inialization. +Neither of these are required: under POSIX, \code{malloc} is required +to be aligned. Also, for all structures it is necessary to explicity +define the initialization independently since a byte value of 0 is +usually insufficient. + +\subsubsection{Callback Routines} + +The PSLib memory management system uses callback functions to handle +certain errors and trace conditions. The callbacks are registered by +the programmer and called by the basic memory management functions if +needed. The four callbacks currently defined are called in the +following situations: +% \begin{itemize} -\item - The functions \code{psAlloc}, \code{psRealloc}, and \code{psFree} - are defined, and are required to be equivalent to \code{p_psAlloc}, - \code{p_psRealloc}, and \code{p_psFree} with the final two arguments - \code{__FILE__} and \code{__LINE__}. - - In the descriptions that follow, I shall not distinguish between the - functions with and without the \code{p_} prefix. - -\item - Except as specified below, the functions \code{psAlloc}, - \code{psRealloc}, and \code{psFree} have identical semantics - to the standard C library functins \code{malloc}, \code{realloc}, - and \code{free}. - -\item - The header file (\file{psMemory.h}) shall take steps to ensure that - code calling the functions \code{malloc}, \code{calloc}, - \code{realloc}, or \code{free} shall not compile (\eg{} - \code{#define malloc(S) for}) unless the symbol - \code{PS_ALLOC_MALLOC} is defined. - -\item - In all cases, application code will call - \CODE|p_{psAlloc,psRealloc,psFree}| via the macros defined above. - -\item - The functions \code{psAlloc} and \code{psRealloc} shall never - return a \code{NULL} pointer. If they are unable to provide - the requested memory they should attempt to obtain the desired - memory by calling the routine registered by \code{psMemExhaustedSetCB} (see - subsubsection \ref{secMemAdvanced}), and if still unsuccessful, - call \code{abort()}. - -\item - The memory management routines shall maintain the field - \code{psMemBlock.refcntr}. It should be set to \code{1} - when a block is returned to the user. It is an error to - attempt to free a block with \code{psMemBlock.refcntr != 1} - (see subsubsection \ref{secMemRefcounter}). - -\item - Where practical and efficient, the memory manager shall call the - routine registered using the \code{psMemProblemSetCB} (see section - \ref{secMemAdvanced}) whenever a corrupted block of memory is - discovered. For example, doubly-freed blocks can be detected by - checking \code{psMemBlock.refcntr}. - -\item - There is no \code{psCalloc} function. Initialisation of data is - almost always more complex than setting all bytes to 0. - Correspondingly, the system call \code{calloc} should be disabled - using the pre-processor. +\item when insufficient memory is available (\code{psMemExhaustedCB}) +\item when a memory block is found to be corrupted (\code{psMemProblemCB}) +\item when a specified memory ID is allocated (\code{psMemAllocateCB}) +\item when a specified memory ID is freed (\code{psMemFreeCB}) \end{itemize} - -\paragraph{APIs for Tracing and Debugging Memory} +% +The callback functions are defined in terms of specific callback +types, specified below. The callbacks are set using functions with +names of the form \code{psCallbackSet}. In all cases, the +`\code{Set}' routine takes a pointer to the desired callback +function and returns a pointer to the one that was previously +installed. If the function pointer is \code{NULL}, the default +callback function is reinstalled. We discuss the use of each of the +four callbacks below. + +\subsubsubsection{\tt psMemExhaustedCB} + +If not enough memory is available to satisfy a request by +\code{psAlloc} or \code{psRealloc}, these functions attempt to find an +alternative solution by calling the \code{psMemExhaustedCB}, a +function which may be set by the programmer in appropriate +circumstances, rather than immediately fail. The typical use of such +a feature may be when a program needs a large chunk of memory to do an +operation, but the exact size is not critical. This feature gives the +programmer the opportunity to make a smaller request and try again, +limiting the size of the operating buffer. This callback has the +following form: +% +\begin{verbatim} +typedef void *(*psMemExhaustedCB)(size_t size); +psMemExhaustedCB psMemExhaustedCBSet(psMemExhaustedCB func); +\end{verbatim} +% +The callback function is called with the attempted size and is +expected to return a pointer to the allocated memory or \code{NULL}. +Until the callback function is set with \code{psMemExhaustedCBSet}, +the default callback function immediately returns \code{NULL}, in +which case \code{psAlloc} will call \code{psAbort}. + +\subsubsubsection{\tt psMemProblemCB} + +At various occasions, the memory manager can check the state of the +memory stack. If any of these checks discover that the memory stack +is corrupted, the \code{psMemProblemCB} is called. The callback +has the following form: +% +\begin{verbatim} +typedef void (*psMemProblemCB)(psMemBlock *ptr, char *file, int lineno); +psMemProblemCB psMemProblemCBSet(psMemProblemCB func); +\end{verbatim} +% +This callback may be used to report the error and other status +information. No return value is accepted, and no specific operations +are expected. The callback is for informational purposes only. Where +practical and efficient, the memory manager shall call the routine +registered using the \code{psMemProblemCBSet} whenever a corrupted +block of memory is discovered. For example, doubly-freed blocks can +be detected by checking \code{psMemBlock.refcntr}. + +\subsubsubsection{\tt psMemAllocateCB \& psMemFreeCB} + +Two private variables, \code{p_psMemAllocateID} and +\code{p_psMemFreeID}, can be used to trace the allocation and freeing +of specific memory blocks. If the first (\code{p_psMemAllocateID}) is +set and a memory block with that ID is allocated, the corresponding +callback is called just before memory is returned to the calling +function. If the second (\code{p_psMemFreeID}) is set and the memory +block with the ID is about to be freed, the corresponding callback is +called just before the memory block is freed. These variables are +internal and private to the memory manager and should be set using the +following two functions: +% +\begin{verbatim} +long psMemAllocateCBSetID(long id); +long psMemFreeCBSetID(long id); +\end{verbatim} +% +The corresponding callback have the following form: +% +\begin{verbatim} +typedef long (*psMemAllocateCB)(const psMemBlock *ptr); +psMemAllocateCB psMemAllocateCBSet(psMemAllocateCB func); +typedef long (*psMemFreeCB)(const psMemBlock *ptr); +psMemFreeCB psMemFreeCBSet(psMemFreeCB func); +\end{verbatim} +% +The callback functions are called with a pointer to the corresponding +memory block. The routines \code{psMemFreeIDSet} and +\code{psMemAllocateIDSet} accept the desired ID value and return the +old value to the user. The return values of the handlers installed by +\code{psMemAllocateCBSet} and \code{psMemFreeCBSet} are used to +increment the values of \code{p_psMemAllocateID} and +\code{p_psMemFreeID} respectively. For example, the return value +\code{0} implies that the value is unchanged; if the value is \code{2} +the callback will be called again when the memory ID counter has +increased by two. This functionality may be useful to check, for +example, every 100th block allocated. The function, \code{long +psMemGetId(void);} returns the next identification number to be +assigned to a memory block. This function can be used to guide the +choice of ID set with the functions above. + +\subsubsection{Memory Tracing and Corruption Checks} \hlabel{secMemAdvanced} -The types and function prototypes for this part of the memory API -are shown below. - -\begin{verbatim} -/// prototype of a basic callback used by memory functions -typedef int (*psMemCallback)(const psMemBlock *ptr); - -/// prototype of a callback used in error conditions -typedef void (*psMemProblemCallback)(const psMemBlock *ptr, const char *file, int lineno); - -/// prototype of a callback used when memory runs out -typedef void *(*psMemExhaustedCallback)(size_t size); - -/// Set callback for problems -psMemProblemCallback psMemProblemSetCB(psMemProblemCallback func //!< Function to run - ); - -/// Set callback for out-of-memory -psMemExhaustedCallback psMemExhaustedSetCB(psMemExhaustedCallback func //!< Function to run - ); - -/// Set call back for when a particular memory block is allocated -psMemCallback psMemAllocateSetCB(psMemCallback func //!< Function to run - ); - -/// Set call back for when a particular memory block is freed -psMemCallback psMemFreeSetCB(psMemCallback func - ); - -/// get next memory ID -int psMemGetId(void); - -/// set p_psMemAllocateID to id -long psMemSetAllocateID(long id //!< ID to set - ); - -/// set p_psMemFreeID to id -long psMemSetFreeID(long id //!< ID to set - ); - -/// Check for memory leaks -int psMemCheckLeaks(int id0, //!< don't list blocks with id < id0 - psMemBlock ***arr, //!< pointer to array of pointers to leaked blocks, or NULL - FILE *fd //!< print list of leaks to fd (or NULL) - ); -/// Check for memory corruption -int psMemCheckCorruption(int abort_on_error //!< Abort on detecting corruption? - ); -\end{verbatim} - -\paragraph{Callback Routines} - -The four `\code{SetCB}' routines are: - -\begin{tabular}{ll|l} -\textbf{type} & \textbf{Name} & \textbf{Function of callback} \\ -\code{psMemProblemCallback} & \code{psMemProblemSetCB} & -Called when a problem is detected with data being managed on the heap \\ -\code{psMemExhaustedCallback} & \code{psMemExhaustedSetCB} & -Called when \code{psAlloc} is unable to satisfy a memory request. \\ -\code{psMemCallback} & \code{psMemAllocateSetCB} & -Callback is called when the \code{psMemBlock} with a specified ID is -allocated. \\ -\code{psMemCallback} & \code{psMemFreeSetCB} & -Callback is called when the \code{psMemBlock} with a specified ID is -freed. \\ -\end{tabular} - -N.b. -\begin{itemize} -\item - In all cases, the `\code{SetCB}' routine takes a pointer to the - desired callback function, and returns a pointer to the one that was - previously installed. If the function pointer is \code{NULL}, the - default callback function is reinstalled. - -\item - The routine installed by \code{psMemExhaustedSetCB} should return - a pointer to the desired memory or \code{NULL}; in the latter case - \code{psAlloc} will call \code{abort()} - -\item - \code{psMemGetId} returns the next identification number to be - assigned to a \code{psMemBlock.id}. - -\item - The handler specified by \code{psMemAllocateSetCB} is called just - before the pointer with \code{psMemBlock.id} equal to - \code{p_psMemAllocateID} is returned to the user. The variable - \code{p_psMemAllocateID} should not be set directly in any - delivered code, \code{psMemSetAllocateID} should be used instead. - -\item - The handler specified by \code{psMemFreeSetCB} is called just - before the pointer with \code{psMemBlock.id} equal to - \code{p_psMemFreeID} is freed. The variable - \code{p_psMemFreeID} should not be set directly in any delivered - code, \code{psMemSetFreeID} should be used instead. - -\item - The routines \code{psMemSetFreeID} and \code{psMemSetAllocateID} - accept the desired ID value (see previous two items) and return - the old value to the user. - -\item - The return values of the handlers installed by \code{psMemAllocateSetCB} - and \code{psMemFreeSetCB} are used to increment the values of - \code{p_psMemAllocateID} and \code{p_psMemFreeID} respectively. - - For example, the return value \code{0} implies that the value is unchanged; - if the value is \code{2} the callback will be called again when the - memory ID counter has increased by two. -\end{itemize} - -\paragraph{Memory Tracing and Corruption Routines} - +The PSLib memory management system includes features to facilitate +tracing the memory allocation and freeing process and to debug memory +errors in the calling code. The types and function prototypes for +this part of the memory API are shown below. +% +\begin{verbatim} +int psMemCheckLeaks(long id0, psMemBlock ***array, FILE *fd); +int psMemCheckCorruption(int abort_on_error); +\end{verbatim} +% The routine \code{psMemCheckLeaks} may be used to check for memory leaks. The return value is the number of blocks that have been -allocated but not freed. - -Only blocks with \code{psMemBlock.id} greater than \code{id0} -are checked; this allows the user to ignore blocks allocated -by initialisation routines. - -If the argument \code{arr} is non-\code{NULL}, then upon entering -the call \code{**arr} should be \code{NULL}. Upon return it is set -to an array of \code{psMemBlock *} pointers, one for each block +allocated but not freed. Only blocks with \code{psMemBlock.id} +greater than \code{id0} are checked; this allows the user to ignore +blocks allocated by initialization routines. + +If the argument \code{array} is non-\code{NULL}, then \code{**array} +is set to an array of \code{psMemBlock *} pointers when the function +returns. These pointers represent the blocks which have been allocated but not freed. It is the caller's responsibility to free -this array with \code{psFree}. +this array with \code{psFree}. Also note that \code{**array} should be +\code{NULL} (or not point to allocated memory) upon entering the call +or the corresponding memory reference will be lost. If the argument \code{fd} is non\code{NULL}, a one-line summary @@ -384,83 +446,47 @@ The routine \code{psMemCheckCorruption} checks the entire heap for corruption, calling the routine registered with -\code{psMemProblemSetCB} for each block detected as being corrupted. +\code{psMemProblemCBSet} for each block detected as being corrupted. The return value is the number of corrupted blocks detected. If the -argument \code{abort_on_error} is true, -\code{psMemCheckCorruption} shall call \code{abort()} as soon as -memory corruption is detected. - -\paragraph{Reference Counting} +argument \code{abort_on_error} is true, \code{psMemCheckCorruption} +shall call \code{psAbort} as soon as memory corruption is detected. + +\subsubsection{Reference Counting} \hlabel{secMemRefcounter} -The memory management routines include a field -\code{psMemBlock.refcntr} which must equal \code{1} whenever -a pointer is presented to \code{psFree}. - -The API for this field is: -\begin{verbatim} -/// Return reference counter -int psMemGetRefCounter(void *vptr //!< Pointer to get refCounter for - ); - -/// Increment reference counter and return the pointer -void *psMemIncrRefCounter(void *vptr //!< Pointer to increment refCounter, and return - ); - -/// Decrement reference counter and return the pointer -void *psMemDecrRefCounter(void *vptr //!< Pointer to decrement refCounter, and return - ); -\end{verbatim} - -The functions \code{psMemIncrRefCounter} and \code{psMemDecrRefCounter} shall -return \code{NULL} if passed a \code{NULL} pointer. - -\subsubsubsection{Rationale} - -The \code{psMemBlock.refcounter} is clearly useful for detecting -attempts to free memory that is already free. A more complex -application is for allowing pointers to complex data-objects (e.g.\ -images) to appear in more than one data structure simultaneously: - -\begin{verbatim} -typedef struct { - char *name; - int value; -} psSimple; - -psSimple *psSimpleAlloc(const char *name, int val) -{ - psSimple *simp = psAlloc(sizeof(psSimple)); - simp->name = strcpy(psAlloc(strlen(name) + 1), name); - simp->value = val; - - return simp; -} - -void psSimpleFree(psSimple *simp) -{ - if (simp == NULL) { return; } - - if (psMemGetRefCounter(simp) > 1) { - (void)psMemDecrRefCounter(simp); - return; - } -} -\end{verbatim} - - -Because of the use of the \code{refcounter} field, we can safely put items of -this type onto many lists: -\goodbreak -\begin{verbatim} -simp = psSimpleAlloc("RHL", 0); -psDlistAppend(list1, psMemIncrRefCounter(simp)); -psDlistAppend(list2, psMemIncrRefCounter(simp)); -psSimpleFree(simp); -\end{verbatim} - -(Note: in fact there is no need to explicitly increment the counter -in this case, as the \code{psDlistAppend} (section \ref{psDlist}) -API specifies that it -does it for you.) +As mentioned above, the memory management system provides the +\code{refCounter} element in \code{psMemBlock} to allow for the +management of multiple references to the same block of memory. +External APIs which make use of this structure must increment the +counter for every additional reference to an allocated memory block, +and decrement it when those references are removed. The memory +management routines respect the use of the \code{refCounter} field: +\code{psFree} will not free a block for which \code{refCounter != 1}, +and \code{psAlloc} will initialize the field to 1. \tbd{should psFree +give an error if refCounter $>$ 1 or actually do the decrement?} +However, they do not (and cannot practically) enforce the use of the +counters; this is a requirement of external APIs which intend to use +the feature. + +Several APIs are provided to manage the reference counters. These +APIs are: +% +\begin{verbatim} +int psMemGetRefCounter(void *vptr); +void *psMemIncrRefCounter(void *vptr); +void *psMemDecrRefCounter(void *vptr); +\end{verbatim} +% +The functions all take a pointer to the start of a user block of +memory. The first simply returns the value of the reference counter. +The next two functions increment or decrement the reference counter, +returning the pointer which was passed in. These functions must +validate the memory pointer by determining the corresponding +\code{psMemBlock.id} and checking for consistency in the internal +memory block table (the table pointer for \code{psMemBlock.id} should +be in the valid range and should correspond to the address of the +\code{psMemBlock}). For an example implementation of the +\code{refCounter} facilities, see the discussion of \code{psDlist} + +\tbd{REF}. \subsection{Tracing and Logging} @@ -468,283 +494,286 @@ This section defines the \PS{} Tracing and Logging APIs; the former refers to debug information that we wish to be able to turn on and off -without recompiling (although it will \emph{not} be available in -production code); the latter means information about the processing -that must be collected and saved, even in the production system. - -We envision that we will make extensive use of \code{psTrace} throughout -the \PS{} code. +without recompiling (it will \emph{not} be available in production +code); the latter means information about the processing that must be +collected and saved, even in the production system. We envision that +we will make extensive use of \code{psTrace} throughout the \PS{} +code. \subsubsection{Tracing APIs} \hlabel{psTrace} -\begin{verbatim} -#if defined(PS_NTRACE) -# define psTrace(facil, level, ...) /* do nothing */ -#else -# define psTrace(facil, level, ...) \ - p_psTrace(facil, level, __VA_ARGS__) -#endif - -/// Send a trace message -void p_psTrace(const char *facil, ///< facilty of interest - int level, ///< desired trace level - ... ///< trace message arguments - ); - -/// Set trace level -int psSetTraceLevel(const char *facil, ///< facilty of interest - int level ///< desired trace level - ); - -/// Get the trace level -int psGetTraceLevel(const char *name ///< facilty of interest - ); - -/// turn off all tracing, and free trace's allocated memory -void psTraceReset(void); - -/// print trace levels -void psPrintTraceLevels(void); -\end{verbatim} - -\begin{itemize} - \item - Logging is provided by the function \code{psTrace}, - which is actually a macro. When the macro \code{PS_NTRACE} - is defined, all occurrences of \code{psTrace} shall - be replaced by whitespace. - - \item - The first argument to \code{psTrace} is a name of the form - \code{aaa.bbb.ccc}. The second is an integer, - \code{level}. The remaining arguments are a printf-style format - and a (possibly empty) set of values for that formatting - string. When this trace is active, \code{psTrace} shall generate - the requested output on \code{stdout}, preceded by \code{level} - spaces. - - \item - The call \code{psSetTraceLevel(name, level)} shall set the trace - level for \code{name} to \code{level}. - - \item - The call \code{psGetTraceLevel(name)} shall return the level - associated with \code{name}. If \code{name} has not been - associated with a level, the routine shall remove the \textit{last} - element of \code{name} and attempt to look up its level; this - procedure shall be applied recursively until the name is reduced - to \code{""}; if this name has no level associated with - it, then the value \code{0} shall be returned (see examples - below). - - \item - The call \code{psTrace(name, level, ...)} - shall print the message if and only if - \code{psGetTraceLevel(name)} returns a value greater than - or equal to \code{level}. - - \item - \code{psPrintTraceLevels} shall print a listing of all - declared levels, displayed as a hierarchy with sub-components - sorted within each component (see examples below). - - Note in particular that the root of the tree, the name \code{""} - should print as \code{(root)}, and nodes which have not - been assigned a value should list their level as \code{.}. - - \item - All of the tracing facilities shall be SWIGed, with the exception - of \code{p_psTrace}. - - \item - There is no requirement that \code{psTrace} be usable from - within the functions that implement the tracing or memory - management systems. - - \item - \code{psTraceReset} shall reset all tracing to the state that it - had at program initiation, including freeing any memory that the - tracing subsystem may have allocated. -\end{itemize} - -\paragraph{Examples of the use of Tracing Facilities} - -For example, after the commands: -\begin{verbatim} - psSetTraceLevel("utils.dlist.add.head", 9); - psSetTraceLevel("utils.dlist.add", 3); - psSetTraceLevel("utils.dlist.remove", 4); - psSetTraceLevel("coadd", 2); - psSetTraceLevel("coadd.CR.remove.morphology", 5); - psSetTraceLevel("utils.hash", 2); - psSetTraceLevel("utils.dlist.add", 9); - psSetTraceLevel("utils", 1); -\end{verbatim} -the command \code{psPrintTraceLevels()} should print: -\begin{verbatim} -(root) 0 - utils 1 - hash 2 - dlist . - remove 4 - add 9 - head 9 - coadd 2 - CR . - remove . - morphology 5 -\end{verbatim} -where \code{.} means that the trace level should be inherited from its parent. - -After this set of \code{psSetTraceLevel} commands, and if -\code{PS_NTRACE} is not defined, the following commands -\begin{verbatim} - psTrace("utils.dlist.remove", 2, "Removing psDList key \"%s\"\n", "my_key"); - psTrace("utils", 2, "Initialising utilities library\n"); - psTrace("", 2, "This is turned on by trace component \"\""); - psTrace("utils.dlist", 2, "Initialising psDList\n"); - psTrace("utils.dlist.remove", 4, "Removing psDList key \"%s\" (value: \"%d\")\n", "my_key", 12345); - psTrace("utils.hash.remove", 4, "Removing hash key \"%s\" (value: \"%d\")\n", "my_key", 12345); - psTrace("utils.dlist.add", 1, "Adding psDList key \"%s\"\n", "your_key"); - psTrace("utils.dlist.find", 2, "Looking up psDList key \"%s\"\n", "some_key"); - psTrace("coadd.CR.remove", 4, "Removing CRs\n"); - psTrace("coadd.CR.remove.morphology", 4, "CRs are not fuzzy\n"); -\end{verbatim} -should produce this output: -\begin{verbatim} - Removing psDList key "my_key" - Removing psDList key "my_key" (value: "12345") - Adding psDList key "your_key" - CRs are not fuzzy -\end{verbatim} - -Note that -\begin{description} -\item - \code{utils.dlist} messages are at level 1, inherited from \code{utils}, so the - \code{Initialising utilities library}, \code{Initialising psDList}, and - \code{Looking up psDList key} messages are \emph{not} printed (the traces are at level 2). - -\item - \code{utils.dlist.remove} messages are at level 4, and are printed. - -\item - \code{utils.hash} messages are at level 2, and are not printed (the traces are at level 4) - -\item - \code{coadd.CR.remove} is at level 2 (inherited from \code{coadd}) so \code{Removing CRs} - isn't printed. \code{coadd.CR.remove.morphology} is at level 4, so \code{CRs are not fuzzy} is printed. -\end{description} +A code-tracing facility should allow the programmer to place messages +in the code which, when called, will print some useful information +about the containing block of code. Ideally, different messages may +be specified to have different levels (of severity or interest). For +a given run of the program, the level of interest should be set to +provide more or less feedback, depending on the needs of the +programmer. In a typical situation, low-level messages would be +placed generously throughout the code, indicating the flow of the +program. Higher-level messages would be placed in a limited number of +special locations, such as the start of major code segments or where a +particularly unusual condition is met. Top-level messages would be +placed in code triggered under serious error conditions. A normal run +of the program would have the trace messages printed only for the +top-level. If the user needs to dig deeper into the code, the trace +level should be set lower, and the more detailed messages could be +examined. In a case of a real, poorly-understood problem with the +code, the trace threshold would be placed to the bottom and the +lowest-level step-by-step messages would be printed. + +The PSLib tracing facility provides the above functionality along with +the ability to assign different trace levels to different types of +messages. Each trace message when placed in the code is assigned to +be part of a specific tracing 'facility', defined in more detail +below. The trace level for that specific message is also set when the +message is placed. Each facility may have its trace level set +independently. Thus, it is possible to request detailed trace output +for one facility while minimizing the verbosity of the trace output +from the rest of the program. + +The trace facilities consist of a hierarchy of names. A trace +facility is defined by a string consisting of words separated by dots, +with a hierarchy equivalent to that of UNIX directory names. The +top-level facility is simply \code{'.'} (one dot). The next level +would be \code{'.A'}, followed by \code{.A.B}, and so on. The +relationship is seen in two ways. First, a facility inherits the +trace level of its parent unless specified. Second, the hierarchy is +used to format the listing of the trace facilities. The first of +these rules provides a mechanism to define the default trace levels +for any facility even if it has not been registered explicitly since +all named facilities are implicitly children of the top level facility +(\code{.}). The second rule is simply an organizational technique to +make the listing of facility information clear. + +The API to place a trace message in the code, and simultaneously set +its trace level and facility, is: +% +\begin{verbatim} +void psTrace(char *facil, int myLevel,...); +\end{verbatim} +% +where the last argument is a printf-style formatting code and possible +arguments to that formatting statement, to be implemented using the +\code{vprintf} functions. This command specifies the name of the +facility to which the message belongs (\code{facil}), the trace level +for this message in that facility (\code{myLevel}) and the message +itself. + +The trace level for any facility may be set at any time with the +following function: +% +\begin{verbatim} +int psSetTraceLevel(char *facil, int level); +\end{verbatim} +% +where \code{level} specifies the current trace level for the facility +named by \code{facil}. The currently defined trace level for a given +facility is determined by the function: +% +\begin{verbatim} +int psGetTraceLevel(char *facil); +\end{verbatim} +% +which returns the trace level of the named facility following the +rules specified above. A specified trace message (identified by +\code{psTrace}) shall be printed if and only if +\code{psGetTraceLevel(facil)} returns a value greater than or equal to +the value of \code{myLevel} for that message. + +PSLib includes a utility function for examining the current tracing +levels of all facilities: \code{void psPrintTraceLevels(void);}. This +function will print the hierarchy of trace facilities along with the +current trace level for each facility. For example, a particular +program may have a few facilities defined, along with their trace +levels. A call to \code{psPrintTraceLevels} may produce a listing +which appears as: +\begin{verbatim} +. 0 +.debias 1 +.flatten 1 +.flatten.divide 2 +.object.findpeak 1 +.object.getsky 1 +\end{verbatim} + +Considering the same program, the programmer might place a variety of +trace messages throughout the \code{flatten} portion of the code with +different types of messages, such as: +% +\begin{verbatim} +psTrace("flatten", 2, "starting flatten function\n"); +psTrace("flatten", 0, "ERROR: flat-field image \%s is invalid\n", filename); +psTrace("flatten.divide", 2, "doing the divide\n"); +psTrace("flatten.divide", 3, "trying the loop, i = \%d\n", i); +psTrace("flatten.divide", 1, "got an invalid pixel value (NaN) at \%d,\%d\n"); +psTrace("flatten.divide", 2, "divide is done\n"); +\end{verbatim} +% +Under the trace levels set above, if the code actually reached each of +these trace messages, the following messages would be printed: +% +\begin{verbatim} +ERROR: flat-field image foo.fits is invalid (printed) +doing the divide (printed) +got an invalid pixel value (NaN) at 500,20 (printed) +divide is done (printed) +\end{verbatim} +% +while these two would not be printed because their facility level was +too low: +% +\begin{verbatim} +starting flatten function +trying the loop, i = 0 +trying the loop, i = 1 +trying the loop, i = 2 +... +\end{verbatim} +% + +The tracing facility should be made available, or not, on compilation. +If the C pre-processor macro \code{PS_NO_TRACE} is defined, all trace +code shall be replaced by empty space so that none of the code is +compiled. This can be implemented via macro front-ends to private +versions of the user APIs. In addition, a function \code{void +psTraceReset(void)} will set all trace levels to 0. + +\tbd{ability to send trace messages to output locations other than stdout?} \subsubsection{Message Logging} \hlabel{psLogMsg} -\begin{verbatim} -enum { PS_LOG_ABORT = 0, PS_LOG_ERROR, PS_LOG_WARN, PS_LOG_INFO }; //!< Status codes for log messages - -enum { PS_LOG_TO_STDERR, PS_LOG_TO_STDOUT }; //!< Destinations for log messages - -/// Logs a message -void psLogMsg(const char *name, int level, const char *fmt, ...); - -/// Logs a message from varargs -void p_psVLogMsg(const char *name, int level, const char *fmt, va_list ap); - -/// Sets the log destination +Message logging is similar in some respects to tracing. Like trace +messages, log messages are placed in the code at various locations to +provide output describing the current state of the program. Like +the PSLib trace facility, a good log facility should have the +capability of associating each message with an importance or severity +level, and at any point, the level for which messages are actually +printed should be set in a flexible manner. Unlike trace messages, +however, log messages are always part of the code and are available in +the production version as well as in test versions. + +The PSLib logging facility does not include the extensive facility +levels which are provided by the trace facility. Less control over +the granularity is needed for the log messages than for the trace +messages. + +A log message is placed in the code with the command: +% +\begin{verbatim} +void psLogMsg(char *name, int myLevel, char *fmt, ...); +void psVLogMsg(char *name, int myLevel, char *fmt, va_list ap); +\end{verbatim} +where \code{name} is a word to describe the source of the message, +\code{myLevel} is the severity level of this message, and \code{fmt} +is a printf-style formatting statement defining the actual message, +and is followed by the arguments to the formatting code. The second +form, \code{psVLogMsg} is an equivalent command which takes a +\code{va_list} argument. + +A log message may have any level specified in the range 0-9, though +the first 4 levels are associated with symbolic names: +% +\begin{verbatim} +enum { PS_LOG_ABORT = 0, PS_LOG_ERROR, PS_LOG_WARN, PS_LOG_INFO }; +\end{verbatim} +% +The default level is set to \code{PS_LOG_INFO}. + +At any time, the program may set the current log level, the level +above which log messages are ignored, using the function: +% +\begin{verbatim} +int psSetLogLevel(int level); +\end{verbatim} +% +A specific message invoked with \code{psLogMsg} is only printed if its +value of \code{myLevel} is less than the current value set by +\code{psSetLogLevel}. + +Log messages are sent to the destination most recently set using: +% +\begin{verbatim} int psSetLogDestination(int dest); - -/// Sets the log level -int psSetLogLevel(int level); - -/// sets the log format +\end{verbatim} +% +The only two values that are initially defined are +\code{PS_LOG_TO_STDERR} and \code{PS_LOG_TO_STDOUT} to write to +\code{stderr} and \code{stdout} respectively. \tbd{define other +destinations? why not use file descriptors?, etc?}. + +The output format is controlled with the function: +% +\begin{verbatim} void psSetLogFormat(const char *fmt); \end{verbatim} - -The function \code{psSetLogLevel} may be used to set the current -level of logging; the previous value is returned. -The default value is \code{PS_LOG_INFO}. Valid values are 0---9 -inclusive (note that only the first four are required to have -symbolic names). - -A call to \code{psLogMsg(name, level, msg, ...)} shall generate -a log message if \code{level} is less than or equal to the -value most recently set using \code{psSetLogLevel}. The function -\code{p_psLogMsg} is identical, except that it expects a -final \code{va_list} argument. - -The format of the log message shall be of the form: -\begin{verbatim} -YYYY:MM:DD hh:mm:ssZ|hostname|l|name |msg -\end{verbatim} -\code{YYYY}, \code{MM}, \code{DD}, \code{hh}, \code{mm}, and \code{ss} -are the year, month (Jan == 1), day of the month, hours (0--23), -minutes, and seconds when the log message was received. Note that the -timestamp is in ISO order, and that the timezone is GMT (hence the -\code{Z}). - -The \code{hostname} is returned by \code{gethostname}, \code{l} is a -letter associated with the level (\code{A}, \code{E}, \code{W}, and -\code{I} for \code{PS_LOG_ABORT}, \code{PS_LOG_ERROR}, \code{PS_LOG_WARN}, -and \code{PS_LOG_INFO} respectively. Other levels are represented -numerically (\code{5} etc.). The other two field, \code{name} and -\code{msg} are arguments to \code{psLogMsg}; note that \code{name} has -a fixed width of 15 characters. If \code{msg} doesn't end in a newline, -a single newline is emitted to terminate the message. - -An example message is: -\begin{verbatim} -2004:02:24 20:14:18Z|alibaba.IfA.Hawaii.Edu|I|utils |Hello World -\end{verbatim} - -Log messages are sent to the destination most recently set using -\code{psSetLogDestination}. The only two values that are initially -defined are \code{PS_LOG_TO_STDERR} and \code{PS_LOG_TO_STDOUT} to -write to \code{stderr} and \code{stdout} respectively. - -The fields included in the log message may be controlled using \code{psSetLogFormat} which -expects a string consisting of the letters \code{H} (host), \code{L} (level), \code{M} (message), -\code{N} (name), and \code{T} (time). The default is \code{THLNM}, as shown above. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +% +which expects a string consisting of the letters \code{H} (host), +\code{L} (level), \code{M} (message), \code{N} (name), and \code{T} +(time). The default is \code{THLNM}, which produces log messages in +the form: +\begin{verbatim} +YYYY:MM:DD hh:mm:ssZ | hostname | L | name | msg +\end{verbatim} +where \code{YYYY}, \code{MM}, \code{DD}, \code{hh}, \code{mm}, and +\code{ss} are the year, month (Jan == 1), day of the month, hours +(0--23), minutes, and seconds when the log message was received. Note +that the timestamp is in ISO order, and that the timezone is GMT +(hence the \code{Z}). The \code{hostname} is returned by +\code{gethostname}, \code{L} is a letter associated with the level +(\code{A}, \code{E}, \code{W}, and \code{I} for \code{PS_LOG_ABORT}, +\code{PS_LOG_ERROR}, \code{PS_LOG_WARN}, and \code{PS_LOG_INFO} +respectively. Other levels are represented numerically (\code{5} +etc.). The other two field, \code{name} and \code{msg} are the +arguments to \code{psLogMsg}; note that \code{name} has a fixed width +of 15 characters. If \code{msg} doesn't end in a newline, a single +newline is emitted to terminate the message. An example message is: +% +\begin{verbatim} +2004:02:24 20:14:18Z | alibaba.IfA.Hawaii.Edu |I | utils | Hello World +\end{verbatim} +% +The possible order of the format entries is fixed and not determined +by the order of the letters. Selecting an output format with fewer +than the complete set of 5 entries simply removes those entries from +the output messages. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Miscellaneous Utilities} -\begin{verbatim} -#define PS_STRING(S) #S // converts argument S to string - -/// Prints an error message and aborts -void psAbort(const char *name, ///< Category of code that caused the abort - const char *fmt, ///< Format - ... ///< Extra arguments to use format - ); - -/// Prints an error message and doesn't abort -void psError(const char *name, ///< Category of code that caused the abort - const char *fmt, ///< Format - ... ///< Extra arguments to use format - ); - -/// Allocates and returns a copy of a string -char *psStringCopy(const char *str ///< string to copy - ); - -/// Allocates nChar and returns a copy of the string or segment -char *psStringNCopy(const char *str, ///< string to copy - int nChar //!< Number of characters (including \0 ) - ); -\end{verbatim} - -\code{psAbort} shall call \code{psMsgLog} with a level of \code{PS_LOG_ABORT}, -and then call \code{abort}. -\code{psError} shall call \code{psMsgLog} with a level of \code{PS_LOG_ERROR}, -and then return. -In cases of doubt, a good choice for -\code{name} is \code{__func__}. - -\code{psStringCopy} shall allocate and return a copy of the input string. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +We require a several very low-functions. Two functions provide +conveniences tied to the logging facilities: +% +\begin{verbatim} +void psAbort(char *name, char *fmt,...); +void psError(char *name, char *fmt,...); +\end{verbatim} +% +The first function, \code{psAbort}, shall call \code{psMsgLog} with a +level of \code{PS_LOG_ABORT}, and then call \code{abort}. The second +function, \code{psError}, shall call \code{psMsgLog} with a level of +\code{PS_LOG_ERROR}, and then return. In cases of doubt, a good +choice for \code{name} is \code{__func__}. + +A few useful string functions are also necessary: +% +\begin{verbatim} +PS_STRING(S); +char *psStringCopy(char *string); +char *psStringNCopy(char *string, int nChar); +\end{verbatim} +% +The first function simply converts the argument to a string +\tbd{explanation of usage and rational?}. The second function, +\code{psStringCopy}, shall allocate a sufficient memory block and +return a copy of the input string. Similarly, \code{psStringNCopy} +shall allocate exactly \code{nChar} bytes and copy as much of the +input string as possible into the resulting block without overrunning +the available space. The last byte of the output string is required +to be \code{NULL}, as well as any additional bytes if \code{string} is +not long enough to fill the output string. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Basic Data Collections} @@ -933,24 +962,24 @@ /** Doubly-linked list element */ typedef struct psDlistElem { - struct psDlistElem *prev; //!< previous link in list - struct psDlistElem *next; //!< next link in list - void *data; //!< real data item + struct psDlistElem *prev; ///< previous link in list + struct psDlistElem *next; ///< next link in list + void *data; ///< real data item } psDlistElem; /** Doubly-linked list */ typedef struct { - int n; //!< number of elements on list - psDlistElem *head; //!< first element on list (may be NULL) - psDlistElem *tail; //!< last element on list (may be NULL) - psDlistElem *iter; //!< iteration cursor + int n; ///< number of elements on list + psDlistElem *head; ///< first element on list (may be NULL) + psDlistElem *tail; ///< last element on list (may be NULL) + psDlistElem *iter; ///< iteration cursor } psDlist; /** Special values of index into list */ enum { - PS_DLIST_HEAD = 0, //!< at head - PS_DLIST_TAIL = -1, //!< at tail - PS_DLIST_UNKNOWN = -2, //!< unknown position - PS_DLIST_PREV = -3, //!< previous element - PS_DLIST_NEXT = -4 //!< next element + PS_DLIST_HEAD = 0, ///< at head + PS_DLIST_TAIL = -1, ///< at tail + PS_DLIST_UNKNOWN = -2, ///< unknown position + PS_DLIST_PREV = -3, ///< previous element + PS_DLIST_NEXT = -4 ///< next element }; \end{verbatim} @@ -960,10 +989,10 @@ \begin{verbatim} /** Constructor */ -psDlist *psDlistAlloc(void *data //!< initial data item; may be NULL +psDlist *psDlistAlloc(void *data ///< initial data item; may be NULL ); /** Destructor */ -void psDlistFree(psDlist *list, //!< list to destroy - void (*elemFree)(void *) //!< destructor for data on list +void psDlistFree(psDlist *list, ///< list to destroy + void (*elemFree)(void *) ///< destructor for data on list ); @@ -971,31 +1000,31 @@ /** Add to list */ -psDlist *psDlistAdd(psDlist *list, //!< list to add to (may be NULL) - void *data, //!< data item to add - int where //!< index, PS_DLIST_HEAD, or PS_DLIST_TAIL +psDlist *psDlistAdd(psDlist *list, ///< list to add to (may be NULL) + void *data, ///< data item to add + int where ///< index, PS_DLIST_HEAD, or PS_DLIST_TAIL ); /** Append to a list */ -psDlist *psDlistAppend(psDlist *list, //!< list to append to (may be NULL) - void *data //!< data item to add +psDlist *psDlistAppend(psDlist *list, ///< list to append to (may be NULL) + void *data ///< data item to add ); /** Remove from a list */ -void *psDlistRemove(psDlist *list, //!< list to remove element from - void *data, //!< data item to remove - int which //!< index of item, or PS_DLIST_UNKNOWN, or PS_DLIST_NEXT, or - //!< PS_DLIST_PREV +void *psDlistRemove(psDlist *list, ///< list to remove element from + void *data, ///< data item to remove + int which ///< index of item, or PS_DLIST_UNKNOWN, or PS_DLIST_NEXT, or + ///< PS_DLIST_PREV ); /** Retrieve from a list */ -void *psDlistGet(const psDlist *list, //!< list to retrieve element from - int which //!< index of item, or PS_DLIST_NEXT, or PS_DLIST_PREV +void *psDlistGet(const psDlist *list, ///< list to retrieve element from + int which ///< index of item, or PS_DLIST_NEXT, or PS_DLIST_PREV ); /** Convert doubly-linked list to an array */ -psVoidPtrArray *psDlistToArray(psDlist *dlist //!< List to convert +psVoidPtrArray *psDlistToArray(psDlist *dlist ///< List to convert ); /** Convert array to a doubly-linked list */ -psDlist *psArrayToDlist(psVoidPtrArray *arr //!< Array to convert +psDlist *psArrayToDlist(psVoidPtrArray *arr ///< Array to convert ); \end{verbatim} @@ -1015,20 +1044,20 @@ \begin{verbatim} /** Set the iterator */ -void psDlistSetIterator(psDlist *list, //!< list to retrieve element from - int where, //!< index, PS_DLIST_HEAD, or PS_DLIST_TAIL - int which //!< index of item, or PS_DLIST_UNKNOWN, or PS_DLIST_NEXT, or - //!< PS_DLIST_PREV +void psDlistSetIterator(psDlist *list, ///< list to retrieve element from + int where, ///< index, PS_DLIST_HEAD, or PS_DLIST_TAIL + int which ///< index of item, or PS_DLIST_UNKNOWN, or PS_DLIST_NEXT, or + ///< PS_DLIST_PREV ); /** Get next element */ -void *psDlistGetNext(psDlist *list, //!< list to retrieve element from - int which //!< index of item, or PS_DLIST_UNKNOWN, or PS_DLIST_NEXT, or - //!< PS_DLIST_PREV +void *psDlistGetNext(psDlist *list, ///< list to retrieve element from + int which ///< index of item, or PS_DLIST_UNKNOWN, or PS_DLIST_NEXT, or + ///< PS_DLIST_PREV ); /** Get previous element */ -void *psDlistGetPrev(psDlist *list, //!< list to retrieve element from - int which //!< index of item, or PS_DLIST_UNKNOWN, or PS_DLIST_NEXT, or - //!< PS_DLIST_PREV +void *psDlistGetPrev(psDlist *list, ///< list to retrieve element from + int which ///< index of item, or PS_DLIST_UNKNOWN, or PS_DLIST_NEXT, or + ///< PS_DLIST_PREV ); \end{verbatim} @@ -1048,4 +1077,54 @@ \code{psDlist}). +\subsubsubsection{Rationale} + +\tbd{defer this example to the psDlist section?} + +The \code{psMemBlock.refcounter} is clearly useful for detecting +attempts to free memory that is already free. A more complex +application is for allowing pointers to complex data-objects (e.g.\ +images) to appear in more than one data structure simultaneously: + +\begin{verbatim} +typedef struct { + char *name; + int value; +} psSimple; + +psSimple *psSimpleAlloc(const char *name, int val) +{ + psSimple *simp = psAlloc(sizeof(psSimple)); + simp->name = strcpy(psAlloc(strlen(name) + 1), name); + simp->value = val; + + return simp; +} + +void psSimpleFree(psSimple *simp) +{ + if (simp == NULL) { return; } + + if (psMemGetRefCounter(simp) > 1) { + (void)psMemDecrRefCounter(simp); + return; + } +} +\end{verbatim} + +Because of the use of the \code{refcounter} field, we can safely put items of +this type onto many lists: +\goodbreak +\begin{verbatim} +simp = psSimpleAlloc("RHL", 0); +psDlistAppend(list1, psMemIncrRefCounter(simp)); +psDlistAppend(list2, psMemIncrRefCounter(simp)); +psSimpleFree(simp); +\end{verbatim} + +(Note: in fact there is no need to explicitly increment the counter +in this case, as the \code{psDlistAppend} (section \ref{psDlist}) +API specifies that it +does it for you.) + \subsection{Hash Tables} \hlabel{psHash} @@ -1087,5 +1166,4 @@ \end{verbatim} - A hash table is an abstract type \code{psHash}. The argument \code{nbucket} to \code{psHashAlloc} is a non-binding suggestion @@ -1108,4 +1186,7 @@ \code{psHashRemove} removes the entry associated with the key from the table, and returns the \code{data}; if the key's invalid it returns NULL. + +\code{psHashKeylist} returns the complete list of defined keys +associated with the psHash table. \tbd{API is not yet specified.} \section{Data manipulation} @@ -1123,5 +1204,5 @@ \item (Fast) Fourier Transforms; \item General functions; and -\item Minimisation and fitting routines. +\item Minimization and fitting routines. \end{itemize} @@ -1137,6 +1218,6 @@ /** A bitmask of arbitrary length. */ typedef struct { - int n; //!< Number of chars that form the mask - char *bits; //!< The bits + int n; ///< Number of chars that form the mask + char *bits; ///< The bits } psBitMask; \end{verbatim} @@ -1146,9 +1227,9 @@ \begin{verbatim} /** Constructor */ -psBitMask *psBitMaskAlloc(int n //!< Number of bits required +psBitMask *psBitMaskAlloc(int n ///< Number of bits required ); /** Destructor */ -void psBitMaskFree(psBitMask *restrict myMask //!< Bit mask to destroy +void psBitMaskFree(psBitMask *restrict myMask ///< Bit mask to destroy ); \end{verbatim} @@ -1165,7 +1246,7 @@ /** Set a bit mask */ psBitMask * -psBitMaskSet(psBitMask *outMask, //!< Output bit mask or NULL - const psBitMask *myMask, //!< Input bit mask - int bit //!< Bit to set +psBitMaskSet(psBitMask *outMask, ///< Output bit mask or NULL + const psBitMask *myMask, ///< Input bit mask + int bit ///< Bit to set ); \end{verbatim} @@ -1174,6 +1255,6 @@ /** Check a bit mask. Returns true or false */ int -psBitMaskTest(const psBitMask *checkMask, //!< Bit mask to check - int bit //!< Bit to check +psBitMaskTest(const psBitMask *checkMask, ///< Bit mask to check + int bit ///< Bit to check ); \end{verbatim} @@ -1182,8 +1263,8 @@ /** apply the given operator to two bit masks */ psBitMask * -psBitMaskOp(psBitMask *outMask, //!< Output bit mask or NULL - const psBitMask *restrict inMask1, //!< Input bit mask 1 - char *operator, //!< bit mask operator (AND, OR, XOR) - const psBitMask *restrict inMask2 //!< Input bit mask 2 +psBitMaskOp(psBitMask *outMask, ///< Output bit mask or NULL + const psBitMask *restrict inMask1, ///< Input bit mask 1 + char *operator, ///< bit mask operator (AND, OR, XOR) + const psBitMask *restrict inMask2 ///< Input bit mask 2 ); \end{verbatim} @@ -1198,6 +1279,6 @@ /** Sort an array. Inputs not restrict-ed to allow sort in place */ psFloatArray * -psSort(psFloatArray *out, //!< Sorted array to return. May be NULL - const psFloatArray *myArray //!< Array to sort +psSort(psFloatArray *out, ///< Sorted array to return. May be NULL + const psFloatArray *myArray ///< Array to sort ); \end{verbatim} @@ -1205,5 +1286,5 @@ We also require the ability to sort one array based on another. In order to facilitate this, we will have a sort function return an array -containing the indices for the unsorted list in the order appropriate +containing the induces for the unsorted list in the order appropriate for the sorted array: @@ -1211,6 +1292,6 @@ /** Sort an array, along with some other stuff. Returns an index array */ psIntArray * -psSortIndex(psIntArray *restrict out; //!< Output index array (may be NULL) - const psFloatArray *restrict myArray //!< Array to sort +psSortIndex(psIntArray *restrict out; ///< Output index array (may be NULL) + const psFloatArray *restrict myArray ///< Array to sort ); \end{verbatim} @@ -1251,9 +1332,9 @@ /** Do Statistics on an array. Returns a status value. */ psStats * -psArrayStats(const psFloatArray *restrict myArray, //!< Array to be analysed - const psIntArray *restrict maskArray, //!< Ignore elements where (maskArray & maskVal) != 0 - //!< May be NULL - unsigned int maskVal, //!< Only mask elements with one of these bits set in maskArray - psStats *stats //!< stats structure defines stats to be calculated and how +psArrayStats(const psFloatArray *restrict myArray, ///< Array to be analysed + const psIntArray *restrict maskArray, ///< Ignore elements where (maskArray & maskVal) != 0 + ///< May be NULL + unsigned int maskVal, ///< Only mask elements with one of these bits set in maskArray + psStats *stats ///< stats structure defines stats to be calculated and how ); \end{verbatim} @@ -1265,32 +1346,32 @@ /** generic statistics structure */ typedef struct { - double sampleMean; //coeff[0][0] + x*T.x->coeff[1][0] + y*T.x->coeff[0][1]; +q = T.y->coeff[0][0] + x*T.y->coeff[1][0] + y*T.y->coeff[0][1]; +\end{verbatim} +% +where we have excluded the basic cross-term (\code{x*y}) by using the +mask: \code{T.x->mask[1][1] = 0; T.y->mask[1][1] = 0;} + +\begin{verbatim} +/** The optical distortion terms. The lowest two terms are the x and y axis of the target system. The higher + * two terms represent magnitude and color terms. + */ +typedef struct { + psDPolynomial4D *x; + psDPolynomial4D *y; +} psDistortion; +\end{verbatim} + +Like \code{psCoordXform}, \code{psDistortion} contains two +\code{psDPolynomial4D} structures representing polynomials of +arbitrary order as a function of four, rather than two dimensions. +There is one of these structures for each of the two output +dimensions. In this structure, the highest two dimensions could +represent a magnitude and a color. As an example, consider the simple +transformation from one linear coordinate frame \code{x,y} (say a +single CCD) of an object with magnitude and color \code{m,c} to a +second frame \code{p,q} (say, a second CCD image). If we have only +first order terms in the transformation \code{psCoordXform T}, the new +coordinates would be represented by the terms: +% +\begin{verbatim} +p = T.x->coeff[0][0][0][0] + x*T.x->coeff[1][0][0][0] + y*T.x->coeff[0][1][0][0] + m*T.x->coeff[0][0][1][0] + c*T.x->coeff[0][0][0][1] +q = T.y->coeff[0][0][0][0] + x*T.y->coeff[1][0][0][0] + y*T.y->coeff[0][1][0][0] + m*T.y->coeff[0][0][1][0] + c*T.y->coeff[0][0][0][1] +\end{verbatim} +% +where we have again excluded the cross-terms (\code{x*y}) by using the +mask. + +We require corresponding functions to apply the transformations: +% +\begin{verbatim} +/** apply the coordinate transformation to the given coordinate */ +psCoord *psCoordXformApply (psCoord *out, ///< Output coordinates, or NULL + const psCoordXform *frame, ///< coordinate transformation + const psCoord *coords ///< input coordiate + ); +\end{verbatim} +% +\begin{verbatim} +/** apply the optical distortion to the given coordinate, magnitude, color */ +psCoord *psDistortionApply (psCoord *out, ///< Output coordinates, or NULL + const psdistortion *pattern, ///< optical distortion pattern + const psCoord *coords, ///< input coordinate + float mag, ///< magnitude of object + float color ///< color of object + ); +\end{verbatim} +% + +\subsubsection{Celestial Coordinate Conversions} + +We need to be able to convert between ICRS, Galactic and Ecliptic +coordinates, and potentially between arbitrary spherical coordinate +systems. All of these basic spherical transformations represent +rotations of the spherical coordinate reference. We specify a general +transformation function which takes a structure \code{psCoordSphere} +defining the relationship between two spherical coordinate systems. +We also define a function to generate \code{psCoordSphere}, which +contains the appropriate sines and cosines of the angles, based on the +three angles describing the location of the pole and the relative +equatorial rotations of the two systems. We also specify special +coordinate transformations between standard coordinate systems which +define the value of psCoordSphere internally and automatically. + +\begin{verbatim} +/** Convert to new spherical coordinate system */ +psCoord * +psCoordTransform (psCoord *coord, + psCoordSphere *sphere) ///< ICRS coordinates to convert +; +\end{verbatim} + +\begin{verbatim} +/** Construct to new spherical coordinate system */ +psCoordSphere * +psCoordSphereDefine (double pole1, double pole2, double rotation); +\end{verbatim} + +\begin{verbatim} +/** Convert ICRS to Ecliptic */ +psCoord * +psCoordinatesItoE(const psCoord *restrict coordinates ///< ICRS coordinates to convert + ); +\end{verbatim} + +\begin{verbatim} +/** Convert Ecliptic to ICRS */ +psCoord * +psCoordinatesEtoI(const psCoord *restrict coordinates ///< Ecliptic coordinates to convert + ); +\end{verbatim} + +\begin{verbatim} +/** Convert ICRS to Galactic */ +psCoord * +psCoordinatesItoG(const psCoord *restrict coordinates ///< ICRS coordinates to convert + ); +\end{verbatim} + +\begin{verbatim} +/** Convert Galactic to ICRS */ +psCoord * +psCoordinatesGtoI(const psCoord *restrict coordinates ///< Galactic coordinates to convert + ); +\end{verbatim} + + +\subsubsection{Projections} + +We require functions to convert between spherical and linear +coordinate systems based on a variety of projections. The required +projections include: +\begin{itemize} +\item TAN +\item SIN +\item AIT +\item PAR +\item GLS +\end{itemize} + +\begin{verbatim} +psCoord *psCoordProject (psCoord *coord, char *projection); +\end{verbatim} + +\subsubsection{Offsets} + +We require a function to calculate the offset between two positions on +the sky, as well as a function to apply an offset to a position. + +\begin{verbatim} +/** Get offset (RA,Dec) on the sky between two positions position1 and position2 may not be identical */ +psCoord * +psGetOffset(const psCoord *restrict position1, ///< Position 1 + const psCoord *restrict position2, ///< Position 2 + const char *type ///< Type of offset: Linear, Spherical/Arcsec, Spherical/Degreees etc + ); +\end{verbatim} + +\begin{verbatim} +/** Apply an offset to a position */ +psCoord * +psApplyOffset(const psCoord *restrict position, ///< Position + const psCoord *restrict offset, ///< Offset + const char *type ///< Type of offset: Linear, Spherical/Arcsec, Spherical/Degreees etc + ); +\end{verbatim} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Astronomy Images} -\subsubsection{Terminology} - -Some brief review of terminology would be useful so that previous -definitions do not influence the understanding of this document. - -A ``readout'' is a read of the detector. - -A ``cell'' is defined as the smallest element of the detector readout; -usually associated with an amplifier. Correspondingly, each cell has -its own overscan region. There may be multiple readouts in a cell if -the cell was used to provide fast guiding. - -A ``chip'' is defined as a contiguous piece of silicon, and consists -of a group of cells. - -A ``focal plane'' is defined as a mosaic of chips, and consists of a -group of chips. +\subsubsection{Overview} + +Above, we have defined a basic container for a single 2D collection of +pixels (\code{psImage}), along with basic operations to manipulate the +image pixels. For astronomical applications, this data structure is +insufficient for two reasons. First, it does provide sufficient +additional metadata to describe the data in detail. Second, astronomy +applications frequent involve multiple, related images. For +PanSTARRS, and for general astronomical applications, we require a +richer collection of data structures which describe a very general +image concept. We have defined several layers in the hierarchy which +are necessary to describe the image data which will be produced by the +PanSTARRS Gigapixel cameras as well as other standard astronomical +images. + +A simple 2D image is a basic data unit for much of astronomical +imaging. If we consider various optical and IR array cameras, a +single readout of the detector produces a collection of pixels +measurements. We define our lowest-level astronomical image +structure, \code{psReadout}, to contain the pixels produced by a +single readout of the detector, along with metadata needed to define +that readout: the origin and binning of the image relative to the +original detector pixels explicitly in the structure, a pointer to the +appropriate overscan region, and pointers to the general metadata and +derived objects, if any. + +A single detector may produce more than one read which is associated. +For example, infrared detectors frequently produce an image +immediately after the detector is reset followed by an image after the +basic exposure is complete. Both readouts correspond to the same +pixels, though the binning or rastering may be different between the +two readouts. Another example is the video sequence produced by the +PanSTARRS Gigapix camera guide cells, each of which represents a +series of many images from a subraster of pixels in the detector +readout portion. The second level of our image container hierarchy, +\code{psCell}, consists of a collection of readouts from a single +detector. + +In the PanSTARRS Gigapix camera, the basic readout region is a +fraction of the full imaging area of a single CCD chip. The chip is +divided into 64 cells, any fraction of which may have been readout +for a given exposure. In other cameras, such as Megacam at CFHT, the +individual CCDs have multiple amplifiers addressing contiguous +portions of the detector. In such cameras, each amplifier produces a +separate collection of pixels. In the third level of our image +container hierarchy, the data structure \code{psChip} represents a +collection of different cells. + +The top level of our image container hierarchy is a complete focal +plane array (\code{psFPA}). This structure represents the collection +of chips in the camera, all of which are read out in a given +exposure. For example, take a mosaic camera consisting of eight $2k\times 4k$ @@ -2641,4 +2930,20 @@ a few cells may be dead). Some cells (those containing guide stars for the orthogonal transfer) will contain multiple readouts. + +These data structures represent containers with which to carry around +the collection of related image data. There is no requirement on the +functions or the structures that each instance of one of these data +structures represent the physical hardware. For example, it is not +necessary that an instance of \code{psFPA} always carry the data for +all 60 (or 64) Gigapixel camera OTAs. The usage of these structures +is such that all astronomical operations which apply to a CCD image +should be performed on an instance of \code{psFPA}. If a particular +circumstance only requires a single 2D image, then that is represented +by an instance of \code{psFPA} with one \code{psChip}, which in turn +has one \code{psCell}, which in turn has one \code{psReadout}. + +These container levels also include in their definition the information +needed to transform the coordinates in one of the levels to the +coordinate system relevant at the higher levels. \subsubsection{A Readout} @@ -2654,12 +2959,11 @@ /** a Readout: a collection of pixels */ typedef struct { - int x0, y0; //!< Offset from the lower-left corner + int x0, y0; ///< Offset from the lower-left corner psImage *image; ///< imaging area of cell psDlist *objects; ///< objects derived from cell psImage *overscan; ///< bias region (subimage) of cell - psMetaDataSet *md; //!< Readout-level metadata + psMetaDataSet *md; ///< Readout-level metadata } psReadout; \end{verbatim} - \subsubsection{A Cell} @@ -2686,5 +2990,5 @@ int nReadouts; ///< number of readouts in this cell realization; each may have its own ///< image, objects and overscan. - struct psReadout *readouts; //!< Readouts from the cell + struct psReadout *readouts; ///< Readouts from the cell psMetaDataSet *md; ///< Cell-level metadata @@ -2719,5 +3023,5 @@ psMetaDataSet *md; ///< Chip-level metadata psCoordXform *chipToFPA; ///< Transformations from chip coordinates to FPA coordinates - psCoordXform *FPAtoChip; //!< Transformations from FPA coordinates to chip + psCoordXform *FPAtoChip; ///< Transformations from FPA coordinates to chip struct psFPA *parentFPA; ///< FPA which contains this chip @@ -2734,11 +3038,11 @@ transformation will consist of two 4D polynomials (i.e.\ a function of two coordinates in position, the magnitude of the object, and the -colour of the object) in order to correct for optical distortions and +color of the object) in order to correct for optical distortions and the effects of the atmosphere; hence we expect that it is prudent to include a reverse transformation which will be derived from -numerically inverting the forward transformation. Since colours are -involved in the transformation, it is necessary to specify the colour +numerically inverting the forward transformation. Since colors are +involved in the transformation, it is necessary to specify the color the transformation is defined for. We also include some values to -characterise the quality of the transformation: the root mean square +characterize the quality of the transformation: the root mean square deviation for the x and y transformation fits, and the $\chi^2$ for the transformation fit. @@ -2756,12 +3060,11 @@ psDistortion *TPtoFP; ///< Transformation term from psDistortion *FPtoTP; ///< Transformation term from - psFixedPattern *pattern; //!< Fixed pattern residual offsets + psFixedPattern *pattern; ///< Fixed pattern residual offsets psExposure *exp; ///< information about this exposure psPhotSystem colorPlus, colorMinus; ///< Colour reference - float rmsX, rmsY; //!< Dispersion in astrometric solution - float chi2; //!< chi^2 of astrometric solution + float rmsX, rmsY; ///< Dispersion in astrometric solution + float chi2; ///< chi^2 of astrometric solution } psFPA; \end{verbatim} - \subsubsection{Exposure information} @@ -2775,23 +3078,23 @@ typedef struct { // Telescope longitude, latitude and height are stored separately, since they don't change with pointing - double ra, dec; //!< Telescope boresight - double ha; //!< Hour angle - double zd; //!< Zenith distance - double az; //!< Azimuth - double lst; //!< Local Sidereal Time - float mjd; //!< MJD of observation - float rotAngle; //!< Rotator position angle - float temp; //!< Air temperature, for estimating refraction - float pressure; //!< Air pressure, for calculating refraction - float humidity; //!< Relative humidity, for calculating refraction - float exptime; //!< Exposure time + double ra, dec; ///< Telescope boresight + double ha; ///< Hour angle + double zd; ///< Zenith distance + double az; ///< Azimuth + double lst; ///< Local Sidereal Time + float mjd; ///< MJD of observation + float rotAngle; ///< Rotator position angle + float temp; ///< Air temperature, for estimating refraction + float pressure; ///< Air pressure, for calculating refraction + float humidity; ///< Relative humidity, for calculating refraction + float exptime; ///< Exposure time /* Derived quantities */ - float posAngle; //!< Position angle - float parallactic; //!< Parallactic angle - float airmass; //!< Airmass, calculated from zenith distance - float pf; //!< Parallactic factor + float posAngle; ///< Position angle + float parallactic; ///< Parallactic angle + float airmass; ///< Airmass, calculated from zenith distance + float pf; ///< Parallactic factor char *cameraName; ///< name of camera which provided exposure char *telescopeName; ///< name of telescope which provided exposure - psGrommit *grommit; //!< Data needed to convert from the sky to the tangent plane + psGrommit *grommit; ///< Data needed to convert from the sky to the tangent plane } psExposure; \end{verbatim} @@ -2809,15 +3112,15 @@ /** Constructor */ psExposure * -psExposureAlloc(double ra, double dec, //!< Telescope boresight - double ha, //!< Hour angle - double zd, //!< Zenith distance - double az, //!< Azimuth - double lst, //!< Local Sidereal Time - float mjd, //!< MJD - float rotAngle, //!< Rotator position angle - float temp, //!< Temperature - float pressure, //!< Pressure - float humidity, //!< Relative humidity - float exptime //!< Exposure time +psExposureAlloc(double ra, double dec, ///< Telescope boresight + double ha, ///< Hour angle + double zd, ///< Zenith distance + double az, ///< Azimuth + double lst, ///< Local Sidereal Time + float mjd, ///< MJD + float rotAngle, ///< Rotator position angle + float temp, ///< Temperature + float pressure, ///< Pressure + float humidity, ///< Relative humidity + float exptime ///< Exposure time ); \end{verbatim} @@ -2827,5 +3130,5 @@ Astrometry is a basic functionality required for the IPP that will be -used repeatedly, both for low-precision (roughly where is my favourite +used repeatedly, both for low-precision (roughly where is my favorite object?) and high-precision (what is the proper motion of this star?). As such, it must be flexible, yet robust. Accordingly, we will wrap @@ -2847,10 +3150,10 @@ The following steps are required to convert from the cell coordinates to -the sky: +the sky \tbd{convert Cell->Chip and Chip->FP to psDistortion}: \begin{itemize} \item Cell $\longleftrightarrow$ Chip: two 2D polynomials, $(X,Y) = f(x,y)$; \item Chip $\longleftrightarrow$ FP: two 2D polynomials, $(p,q) = g(X,Y)$; \item FP $\longleftrightarrow$ TP: two 4D polynomials, $(l,m) = -h(p,q,m,c)$, where $m$ and $c$ are the magnitude and colour of the +h(p,q,m,c)$, where $m$ and $c$ are the magnitude and color of the object, respectively; and \item TP $\longleftrightarrow$ Sky: SLALib transformation using a @@ -2861,10 +3164,7 @@ Plane is a four-dimensional polynomial, in order to account for any possible dependencies in the astrometry on the stellar magnitude and -colour; the former serves as a check for charge transfer +color; the former serves as a check for charge transfer inefficiencies, while the latter will correct chromatic refraction, both through the atmosphere and the corrector lenses. - -\textbf{[If the magnitude terms serve to check CTI, then shouldn't we -put them in the cell $\leftrightarrow$ chip section?]} We require structures to contain each of the above transformations as @@ -2897,5 +3197,4 @@ \end{verbatim} - \subsubsection{Fixed Pattern} @@ -2911,13 +3210,13 @@ /** The fixed pattern residual offsets. These are specified via a coarse grid of x and y offsets. */ typedef struct { - int nX, nY; //!< Number of elements in x and y - double x0, y0; //!< Position of the lower-left corner of the grid on the focal plane - double xScale, yScale; //!< Scale of the grid - double **x, **y; //!< The grid of offsets in x and y + int nX, nY; ///< Number of elements in x and y + double x0, y0; ///< Position of the lower-left corner of the grid on the focal plane + double xScale, yScale; ///< Scale of the grid + double **x, **y; ///< The grid of offsets in x and y } psFixedPattern; \end{verbatim} -\subsubsection{Finding} +\subsubsection{Position Finding} We require functions to return the structure containing given @@ -2932,5 +3231,5 @@ /** returns Chip in FPA which contains the given FPA coordinate */ psChip * -psChipInFPA (psChip *out, //!< Chip to return, or NULL +psChipInFPA (psChip *out, ///< Chip to return, or NULL const psFPA *fpa, ///< FPA description const psCoord *coord ///< coordinate in FPA @@ -2941,5 +3240,5 @@ /** returns Cell in Chip which contains the given chip coordinate */ psCell * -psCellInChip(psCell *out, //!< Cell to return, or NULL +psCellInChip(psCell *out, ///< Cell to return, or NULL const psChip *chip, ///< chip description const psCoord *coord ///< coordinate in chip @@ -2954,7 +3253,7 @@ /** Return the cell in FPA which contains the given FPA coordinates */ psCell * -psCellInFPA(psCell *out, //!< Cell to return, or NULL - const psFPA *fpa, //!< FPA description - const psCoord *coord //!< Coordinate in FPA +psCellInFPA(psCell *out, ///< Cell to return, or NULL + const psFPA *fpa, ///< FPA description + const psCoord *coord ///< Coordinate in FPA ); \end{verbatim} @@ -2965,10 +3264,10 @@ We require functions to convert between the various coordinate frames -(Section~\ref{sec:coordinateFrames}). The heirarchy of the coordinate +(Section~\ref{sec:coordinateFrames}). The hierarchy of the coordinate frames and the transformations between each are shown in Figure~\ref{fig:coco}. The functions that employ the transformations are shown in Figure~\ref{fig:cocoFunc}. In addition to transformations between each adjoining coordinate frame in the -heirarchy, we also require higher-level functions to convert between +hierarchy, we also require higher-level functions to convert between the Cell and Sky coordinate frames; these will simply perform the intermediate steps. @@ -2993,7 +3292,7 @@ /** Convert (RA,Dec) to cell and cell coordinates */ psCoord * -psCoordSkyToCell(psCoord *out, //!< Coordinates to return, or NULL - psCell *cell, //!< Cell to return - const psFPA *fpa //!< FPA description +psCoordSkyToCell(psCoord *out, ///< Coordinates to return, or NULL + psCell *cell, ///< Cell to return + const psFPA *fpa ///< FPA description ); \end{verbatim} @@ -3002,7 +3301,7 @@ /** Convert cell and cell coordinate to (RA,Dec) */ psCoord * -psCoordCellToSky(psCoord *out, //!< Coordinates to return, or NULL - const psCell *cell, //!< Cell to get coordinates for - const psCoord *coord //!< cell coordinates to transform +psCoordCellToSky(psCoord *out, ///< Coordinates to return, or NULL + const psCell *cell, ///< Cell to get coordinates for + const psCoord *coord ///< cell coordinates to transform ); \end{verbatim} @@ -3011,7 +3310,7 @@ /** Quick and dirty cell to (RA,Dec) --- employs cellToSky transformation */ psCoord * -psCoordCellToSkyQuick(psCoord *out, //!< Coordinates to return, or NULL - const psCell *cell, //!< Cell description - const psCoord *coord //!< cell coordinates to transform +psCoordCellToSkyQuick(psCoord *out, ///< Coordinates to return, or NULL + const psCell *cell, ///< Cell description + const psCoord *coord ///< cell coordinates to transform ); \end{verbatim} @@ -3020,7 +3319,7 @@ /** Convert (RA,Dec) to tangent plane coords */ psCoord * -psCoordSkyToTP(psCoord *out, //!< Coordinates to return, or NULL - const psExposure *exp, //!< Exposure description - const psCoord *coord //!< input Sky coordinate +psCoordSkyToTP(psCoord *out, ///< Coordinates to return, or NULL + const psExposure *exp, ///< Exposure description + const psCoord *coord ///< input Sky coordinate ); \end{verbatim} @@ -3029,7 +3328,7 @@ /** Convert tangent plane coords to focal plane coordinates */ psCoord * -psCoordTPtoFPA(psCoord *out, //!< Coordinates to return, or NULL - const psFPA *fpa, //!< FPA description - const psCoord *coord //!< input TP coordinate +psCoordTPtoFPA(psCoord *out, ///< Coordinates to return, or NULL + const psFPA *fpa, ///< FPA description + const psCoord *coord ///< input TP coordinate ); \end{verbatim} @@ -3038,5 +3337,5 @@ /** converts the specified FPA coord to the coord on the given Chip */ psCoord * -psCoordFPAtoChip (psCoord *out, //!< Coordinates to return, or NULL +psCoordFPAtoChip (psCoord *out, ///< Coordinates to return, or NULL const psChip *chip, ///< Chip of interest const psCoord *coord ///< input FPA coordinate @@ -3047,5 +3346,5 @@ /** converts the specified Chip coord to the coord on the given Cell */ psCoord * -psCoordChiptoCell (psCoord *out, //!< Coordinates to return, or NULL +psCoordChiptoCell (psCoord *out, ///< Coordinates to return, or NULL const psCell *cell, ///< Cell of interest const psCoord *coord ///< input Chip coordinate @@ -3056,5 +3355,5 @@ /** converts the specified Cell coord to the coord on the parent Chip */ psCoord * -psCoordCelltoChip (psCoord *out, //!< Coordinates to return, or NULL +psCoordCelltoChip (psCoord *out, ///< Coordinates to return, or NULL const psCell *cell, ///< Cell description const psCoord *coord ///< input Cell coordinate @@ -3065,5 +3364,5 @@ /** converts the specified Chip coord to the coord on the parent FPA */ psCoord * -psCoordChiptoFPA (psCoord *out, //!< Coordinates to return, or NULL +psCoordChiptoFPA (psCoord *out, ///< Coordinates to return, or NULL const psChip *chip, ///< Chip description const psCoord *coord ///< input Chip coordinate @@ -3074,7 +3373,7 @@ /** Convert focal plane coords to tangent plane coordinates */ psCoord * -psCoordFPAToTP(psCoord *out, //!< Coordinates to return, or NULL - const psFPA *fpa, //!< FPA description - const psCoord *coord //!< input FPA coordinate +psCoordFPAToTP(psCoord *out, ///< Coordinates to return, or NULL + const psFPA *fpa, ///< FPA description + const psCoord *coord ///< input FPA coordinate ); \end{verbatim} @@ -3083,7 +3382,7 @@ /** Convert tangent plane coords to (RA,Dec) */ psCoord * -psCoordTPtoSky(psCoord *out, //!< Coordinates to return, or NULL - const psExposure *exp, //!< Exposure description - const psCoord *coord //!< input TP coordinate +psCoordTPtoSky(psCoord *out, ///< Coordinates to return, or NULL + const psExposure *exp, ///< Exposure description + const psCoord *coord ///< input TP coordinate ); \end{verbatim} @@ -3092,7 +3391,7 @@ /** Convert Cell coords to FPA coordinates */ psCoord * -psCoordCellToFPA(psCoord *out, //!< Coordinates to return, or NULL - const psCell *cell, //!< Cell description - const psCoord *coord //!< Input cell coordinates +psCoordCellToFPA(psCoord *out, ///< Coordinates to return, or NULL + const psCell *cell, ///< Cell description + const psCoord *coord ///< Input cell coordinates ); \end{verbatim} @@ -3108,7 +3407,7 @@ /** Get the airmass for a given position and sidereal time */ float -psGetAirmass(const psCoord *coord, //!< Position on the sky - double siderealTime, //!< Sidereal time - float height //!< Height above sea level +psGetAirmass(const psCoord *coord, ///< Position on the sky + double siderealTime, ///< Sidereal time + float height ///< Height above sea level ); \end{verbatim} @@ -3117,6 +3416,6 @@ /** Get the parallactic angle for a given position and sidereal time */ float -psGetParallactic(const psCoord *coord, //!< Position on the sky - double siderealTime //!< Sidereal time +psGetParallactic(const psCoord *coord, ///< Position on the sky + double siderealTime ///< Sidereal time ); \end{verbatim} @@ -3125,8 +3424,8 @@ /** Estimate atmospheric refraction, along the parallactic */ float -psGetRefraction(float colour, //!< Colour of object +psGetRefraction(float colour, ///< Colour of object psPhotSystem colorPlus, ///< Colour reference psPhotSystem colorMinus, ///< Colour reference - const psExposure *exp //!< Telescope pointing information, for airmass, temp and pressure + const psExposure *exp ///< Telescope pointing information, for airmass, temp and pressure ); \end{verbatim} @@ -3135,5 +3434,5 @@ /** Calculate the parallax factor */ double -psGetParallaxFactor(const psExposure *exp //!< Exposure details +psGetParallaxFactor(const psExposure *exp ///< Exposure details ); \end{verbatim} @@ -3161,9 +3460,9 @@ \begin{verbatim} typedef struct { - int ID; //!< ID number for this photometric system - char *name; //!< Name of photometric system - char *camera; //!< Camera for photometric system - char *filter; //!< Filter used for photometric system - char *detector; //!< Detector used for photometric system + int ID; ///< ID number for this photometric system + char *name; ///< Name of photometric system + char *camera; ///< Camera for photometric system + char *filter; ///< Filter used for photometric system + char *detector; ///< Detector used for photometric system } psPhotSystem; \end{verbatim} @@ -3173,10 +3472,10 @@ \begin{verbatim} typedef struct { - psPhotSystem src; //!< Source photometric system - psPhotSystem dst; //!< Destination photometric system + psPhotSystem src; ///< Source photometric system + psPhotSystem dst; ///< Destination photometric system psPhotSystem pP, pM; ///< Primary colour reference psPhotSystem sP, sM; ///< Secondary colour reference float pA, sA; ///< Colour offset for primary and secondary references - psPolynomial3D transform; //!< Transformation from source to destination + psPolynomial3D transform; ///< Transformation from source to destination } psPhotTransform; \end{verbatim} @@ -3192,5 +3491,5 @@ are highly reddened, for example. Derived photometric quantities may have been corrected for airmass variations, in which case only color -terms may be measureable. The structure defines the transformation +terms may be measurable. The structure defines the transformation between a source photometric system (\code{src}) and a target photometric system (\code{dst}). The photometric system of a primary @@ -3213,5 +3512,50 @@ \subsection{Astronomical objects} -\textbf{[Deferred.]} +\subsubsection{Positions of Major SS Objects} + +We require the ability to calculate the position of major Solar System +objects, as well as Lunar phase. + +\begin{verbatim} +/** Get Sun Position */ +psCoord * +psGetSunPos(float mjd ///< MJD to get position for + ); +\end{verbatim} + +\begin{verbatim} +/** Get Moon position */ +psCoord * +psGetMoonPos(float mjd, ///< MJD to get position for + double latitude, ///< Latitude for apparent position + double longitude ///< Longitude for apparent position + ); +\end{verbatim} + +\begin{verbatim} +/** Get Moon phase */ +float +psGetMoonPhase(float mjd ///< MJD to get phase for + ); +\end{verbatim} + +\begin{verbatim} +/** Get Planet positions */ +psCoord * +psGetSolarSystemPos(const char *solarSystemObject, ///< Named S.S. object + float mjd ///< MJD to get position for + ); +\end{verbatim} + +\tbd{The rest of this section is tentative} + +We may need a variety of other SLALib-type functions: + +\begin{itemize} +\item Lunation calculations to/from psTime +\item rise/set for specified object +\item 12 \& 18 deg twilight for closest sunrise / sunset +\item length of current night +\end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%