Index: /trunk/doc/ipptools/.cvsignore =================================================================== --- /trunk/doc/ipptools/.cvsignore (revision 4976) +++ /trunk/doc/ipptools/.cvsignore (revision 4976) @@ -0,0 +1,2 @@ +*.log *.dvi *.aux *.toc *.log *.out *.lof *.tbr *.tbd +pstask.pdf Index: /trunk/doc/ipptools/pstask.tex =================================================================== --- /trunk/doc/ipptools/pstask.tex (revision 4975) +++ /trunk/doc/ipptools/pstask.tex (revision 4976) @@ -18,4 +18,8 @@ \maketitle +\tableofcontents +\pagebreak +\pagenumbering{arabic} + \section{Overview} @@ -26,6 +30,11 @@ (Phase 1-4, detrend creation, etc) are illustrated in this document and the relevant MDDB tables are listed. The collection of diagrams -shows the IPP tasks and the Metadata Database (MDDB) tables needed to -manage the flow of data through the system. +shows the IPP tasks and the Metadata Database tables needed to manage +the flow of data through the system. + +\note{The flat-field correction analysis stage is not yet defined in + this document} + +\note{AP DB interfaces are mostly ignored} \section{PStask queues and metadata tables : persistent vs ephemeral state} @@ -33,23 +42,55 @@ \begin{figure} \begin{center} -\resizebox{4in}{!}{\includegraphics{pics/pstasks.01.ps}} +\includegraphics[scale=0.85]{pics/pstasks.01.ps} \caption{\label{queues} PStask queues and MDDB tables} \end{center} \end{figure} -The left-hand portion of the diagram illustrates the recommended -interaction between the metadata database tables and the scheduler -internal queues. Some table in the metadata database defines a list -of data items which are to be processed by some analysis job. The -scheduler uses a two-step approach to define the analysis jobs based -on this list. First, one scheduler task queries the MDDB for a list -of pending items, adds the returned items to an internal scheduler -queue. The process of adding the elements to the queue is defined so -that only unique items are added: already existing items are skipped. -The entries in the queue consist of the data items of interest and an -internal temporary state. At first, this would be 'pending'. A -second scheduler tasks pops 'pending' entries one-by-one from this -internal queue, submits a job based on the entry, and sets the -temporary state in the internal queue to 'running'. The internal +There are some subtleties in the interaction between IPP task +scheduler, PStasks, the metadata tables which store the system state, +and the jobs which are immediately being performed. There is a choice +to be made between rigorously maintaining the system state in the +Metadata DB at all times or in keeping an intermediate set of state +tables. Keeping the exact system state in the Metadata DB tables +would require more queries to/from the database and may introduce +additional latencies which are undesirable. This is because any +attempt by the scheduler to initiate a new job would require the +scheduler to mark the corresponding data item in the Metadata DB (the +item which acts as the trigger) with a `pending' state, and then mark +it again as `done' when the job actually completes. This also has the +drawback that, if the scheduler crashes, some initial process would be +required on start up to find all Metadata DB items which are in the +`pending' state (examining all possible items which can be in such a +state) and reset them to the `new' state. + +We implement an alternative in which the scheduler maintains an +internal, ephemeral stack of the pending jobs, and only updates the +system state entries in the Metadata DB when jobs are actually +completed. In this scenario, as far as the Metadata DB tables are +concerned, data items transition only between a `new' and a `done' +state. Any jobs which are pending when the system crashes or the +power is lost are simply dropped, and will be automatically +re-constructed when the system restarts. In this paradigm, no +intermediate operation state is saved, and no partially completed job +can be recovered. Since the IPP is defined in terms of a fine +granularity, with jobs lasting no more than 30 - 120 seconds, this +model will not have a large impact. + +Figure~\ref{queues} illustrates this ephemeral vs persistent state +information and the interation between the metadata tables and +PStasks. The left-hand portion of the diagram illustrates the +recommended interaction between the metadata database tables and the +scheduler internal queues. Some table in the metadata database +defines a list of data items which are to be processed by some +analysis job. The scheduler uses a two-step approach to define the +analysis jobs based on this list. First, one scheduler task queries +the MDDB for a list of pending items, adds the returned items to an +internal scheduler queue. The process of adding the elements to the +queue is defined so that only unique items are added: already existing +items are skipped. The entries in the queue consist of the data items +of interest and an internal temporary state. At first, this would be +`pending'. A second scheduler tasks pops `pending' entries one-by-one +from this internal queue, submits a job based on the entry, and sets +the temporary state in the internal queue to `running'. The internal state is needed to prevent the scheduler from re-submitting a job for the same data item before the first job is done or assessed. Since @@ -59,5 +100,5 @@ updated noting the completion. This may be done either by the job itself or by the scheduler. In addition, the state of the entry in -the queue can be set to either 'done' or the entry can be simply +the queue can be set to either `done' or the entry can be simply removed from the queue. @@ -83,7 +124,7 @@ copy of the image file from the location specified by the summit data system to the appropriate location within the IPP Image Server -(Nebulous). (As an alternative to the above, the 'pending images' -table may be part of the summit database system, and the 'get images' -command may query the summit directly. In this scenario, the 'copy +(Nebulous). (As an alternative to the above, the `pending images' +table may be part of the summit database system, and the `get images' +command may query the summit directly. In this scenario, the `copy image' command reports to the summit data system that an individual image file has been copied.) @@ -97,18 +138,18 @@ \begin{figure} \begin{center} -\resizebox{4in}{!}{\includegraphics{pics/pstasks.02.ps}} +\includegraphics[scale=0.85]{pics/pstasks.02.ps} \caption{\label{pcopy} Summit Copy Tasks} \end{center} \end{figure} -This diagram illustrates the MDDB tables used to copy data (images and -metadata tables) from the summit. The left-hand portion of the -diagram shows the tables involved in copying images from the summit -system. The table of pending image files lists the URLs of the +Figure~\ref{pcopy} illustrates the MDDB tables used to copy data +(images and metadata tables) from the summit. The left-hand portion +of the diagram shows the tables involved in copying images from the +summit system. The table of pending image files lists the URLs of the individual image files available for transfer, along with their associated exposure ID and the camera which generated the image. Two other entries assist in interpreting the file: the class and the class ID. The final entry in this table is the current copy state of the -file, can have the value of 'ready' or 'copied'. +file, can have the value of `ready' or `copied'. The class defines the data grouping represented by this image file and @@ -124,27 +165,8 @@ corresponding to this file. This value is necessary to make decisions on how to copy the data based on the chip / cell before the data is -available to IPP components. Here are likely values for the class and -class ID for some common cameras: - -\begin{table} -\begin{center} -\caption{Camera and Data Classes\label{classes}} -\begin{tabular}{lll} -\hline -\hline -camera & class & classID \\ -\hline -GPC & chip & chip02 \\ -skyprobe & fpa & sp01 \\ -Megacam & fpa & MegacamSpliced \\ -Suprime & chip & chip0 \\ -\hline -\end{tabular} -\end{center} -\end{table} - -and so forth. The system described is sufficiently flexible to allow -us to transfer the GPC images by cell if we eventually decide that is -more efficient. +available to IPP components. Table~\ref{classes} lists likely values +for the class and class ID for some common cameras. The system +described is sufficiently flexible to allow us to transfer the GPC +images by cell if we eventually decide that is more efficient. The copy process copies the file from the given URL to the appropriate @@ -158,6 +180,23 @@ server? eg: ref:DIR0001/file0001.fits might be in a directory which is defined in a table of directories.) After an image file is -successfully copied, the corresponding state in the 'pending chip' -table is updated from 'ready' to 'copied'. +successfully copied, the corresponding state in the `pending chip' +table is updated from `ready' to `copied'. + +\begin{table} +\begin{center} +\caption{Camera and Data Classes\label{classes}} +\begin{tabular}{llll} +\hline +\hline +camera & class & classID \\ +\hline +GPC & chip & chip02 \\ +skyprobe & fpa & sp01 \\ +Megacam & fpa & MegacamSpliced \\ +Suprime & chip & chip0 \\ +\hline +\end{tabular} +\end{center} +\end{table} The right hand portion of this diagram illustrates the process of @@ -177,14 +216,14 @@ \begin{figure} \begin{center} -\resizebox{4in}{!}{\includegraphics{pics/pstasks.03.ps}} +\includegraphics[scale=0.85]{pics/pstasks.03.ps} \caption{\label{phase0} Phase 0 Tasks} \end{center} \end{figure} -This diagram illustrates phase 0, in which the image files are +Figure~\ref{phase0} illustrates phase 0, in which the image files are categorised, examined for summary information and basic statistics, and moved to the tables used to trigger further analysis. The process -first examines the 'new image files' table. It selects images from -this table which have not yet been examined (state is 'new'). The +first examines the `new image files' table. It selects images from +this table which have not yet been examined (state is `new'). The file header is examined and relevant metadata is extracted (eg, RA, DEC, times, and so forth to be defined later). The process may also @@ -192,20 +231,20 @@ background level. These statistics, whether derived from the header or the pixel values, are placed along with image summary information -in the 'raw image files' table, and the state field of the 'new image -files' table is set to 'ready'. +in the `raw image files' table, and the state field of the `new image +files' table is set to `ready'. The process is also responsible for moving the exposures to the tables used for triggering the analysis process. If the image class is FPA, the image can be advanced without waiting for any other image files. -If the class is Chip or Cell, the process must also examine the 'new +If the class is Chip or Cell, the process must also examine the `new exposure' table for this exposure ID. The number of class files available for this exposure is listed in this table. The process must the select all image files matching the exposure ID with state of -'ready' and compare the number avalable to the number expected. If +`ready' and compare the number avalable to the number expected. If the two match, then a new exposure is ready. Based on the image type (from the most recently examined image file header or new exp table?), -the exposure is added to the 'raw exposure' table for images of that -type. The allowed types are 'detrend', (all bias, dark, flat images), -'object', 'focus'(??), etc. (** The different tables represent +the exposure is added to the `raw exposure' table for images of that +type. The allowed types are `detrend', (all bias, dark, flat images), +`object', `focus'(??), etc. (** The different tables represent different analysis modes. This process also adds an entry to the exp ID / image file match **). This process also adds all science @@ -218,15 +257,15 @@ \begin{figure} \begin{center} -\resizebox{4in}{!}{\includegraphics{pics/pstasks.04.ps}} +\includegraphics[scale=0.85]{pics/pstasks.04.ps} \caption{\label{phase1} Phase 1 Tasks} \end{center} \end{figure} -This diagram shows the tables involved in running the P1 analysis -stage. There are paths for exposures to enter the analysis +Figure~\ref{phase1} shows the tables involved in running the P1 +analysis stage. There are paths for exposures to enter the analysis automatically from the P0 analysis (arrow on left) or to be added manually based on a selection from the raw exposure table. Exposures to be analysed by Phase 1 are added to the P1 exposure table with the -state 'new'. Exposures may be added multiple times for processing and +state `new'. Exposures may be added multiple times for processing and reprocessing. The P1 exp table keeps a record of the old attempts for debugging and analysis. Each time an exposure is added to the P1 exp @@ -257,30 +296,30 @@ \begin{figure} \begin{center} -\resizebox{4in}{!}{\includegraphics{pics/pstasks.05.ps}} +\includegraphics[scale=0.85]{pics/pstasks.05.ps} \caption{\label{phase2} Phase 2 Tasks} \end{center} \end{figure} -This diagram shows the tables involved in running the P2 analysis -stage. There are paths for images to enter the analysis automatically -from the P1 analysis (arrow on left) or to be added manually based on -a selection from the raw exposure and raw image file tables. Images -to be analysed by Phase 2 are added to the P2 image table with the -state 'new'. When images are added to this table, a single entry is -also added to the P2 exposure table listing the P1 and P2 versions for -this exposure. These version numbers must be integers starting with -1. If this image did not have a P1 analysis, the P1 version is set to -0. Exposures may be added multiple times for processing and -reprocessing. The P2 image table keeps a record of the old attempts -for debugging and analysis. As with P1, each time a collection of -associated images from an exposure is added to the P2 image table, it -is given a new, unique version number, allowing the system as a whole -to track different analysis attempts. The top portion of the diagram -represents the user-space tool which may be used to re-submit the -images for an exposure or a group of exposures, potentially selected -on the basis of a query from the raw SCIENCE exposure and raw image -file tables. - -The P2 image table is examined to select the 'new' images. These +Figure~\ref{phase2} shows the tables involved in running the P2 +analysis stage. There are paths for images to enter the analysis +automatically from the P1 analysis (arrow on left) or to be added +manually based on a selection from the raw exposure and raw image file +tables. Images to be analysed by Phase 2 are added to the P2 image +table with the state `new'. When images are added to this table, a +single entry is also added to the P2 exposure table listing the P1 and +P2 versions for this exposure. These version numbers must be integers +starting with 1. If this image did not have a P1 analysis, the P1 +version is set to 0. Exposures may be added multiple times for +processing and reprocessing. The P2 image table keeps a record of the +old attempts for debugging and analysis. As with P1, each time a +collection of associated images from an exposure is added to the P2 +image table, it is given a new, unique version number, allowing the +system as a whole to track different analysis attempts. The top +portion of the diagram represents the user-space tool which may be +used to re-submit the images for an exposure or a group of exposures, +potentially selected on the basis of a query from the raw SCIENCE +exposure and raw image file tables. + +The P2 image table is examined to select the `new' images. These images are used to generate P2 analysis jobs. The P2 analysis uses the input url to find and load the image file. The url may be a file @@ -293,5 +332,5 @@ objects, including the astrometry calibration, are written to the P2 image table, along with summary statistics from the P2 analysis. The -state is also updated (to 'done'). +state is also updated (to `done'). Whenever the exposure is completed, the value of Ndone in the P2 @@ -305,25 +344,25 @@ \begin{figure} \begin{center} -\resizebox{4in}{!}{\includegraphics{pics/pstasks.06.ps}} +\includegraphics[scale=0.85]{pics/pstasks.06.ps} \caption{\label{phase3} Phase 3 Tasks} \end{center} \end{figure} -This diagram illustrates the tables involved in the Phase 3 analysis. -The P3 exposure table lists the exposure ID, the P3 analysis version, -the P2 analysis version to be used as input to this P3 analysis, and -the recipe to be used. The P2 exposure and image tables are used, in -conjunction with the P2 version information, to select the P2 output -measured objects and the astrometric calibrations from P2 and P1. -These measured objects are matched with the reference catalog objects, -and calibrated astrometry and photometry is produced for the full -exposure. The location of the resulting astometry calibration table -is stored back in the P3 exposure table. If the recipe file -specifies, the 2-D photometric and background / fringe corrections may -also be performed at this stage. Since these analyses require -reference data, the recipe may also be used to skip these analysis if -such reference data is unavailable or unreliable. At the end of Phase -3, the objects from the exposure are inserted into the photometry -database (this is not shown). +Figure~\ref{phase3} illustrates the tables involved in the Phase 3 +analysis. The P3 exposure table lists the exposure ID, the P3 +analysis version, the P2 analysis version to be used as input to this +P3 analysis, and the recipe to be used. The P2 exposure and image +tables are used, in conjunction with the P2 version information, to +select the P2 output measured objects and the astrometric calibrations +from P2 and P1. These measured objects are matched with the reference +catalog objects, and calibrated astrometry and photometry is produced +for the full exposure. The location of the resulting astometry +calibration table is stored back in the P3 exposure table. If the +recipe file specifies, the 2-D photometric and background / fringe +corrections may also be performed at this stage. Since these analyses +require reference data, the recipe may also be used to skip these +analysis if such reference data is unavailable or unreliable. At the +end of Phase 3, the objects from the exposure are inserted into the +photometry database (this is not shown). The astrometric calibration portion of Phase 3 is principally needed @@ -340,5 +379,5 @@ \begin{figure} \begin{center} -%\resizebox{4in}{!}{\includegraphics{pics/pstasks.03.ps}} +\includegraphics[scale=0.85]{pics/pstasks.07.ps} \caption{\label{phase4} Phase 4 Tasks} \end{center} @@ -363,9 +402,15 @@ image combination and image differencing. It may even be the case that only the combination portion of Phase 4 is performed on the AP -Survey data. +Survey data. More generally, the image groups selected for Phase 4 analysis may be chosen on the basis of a query of the AP Database (DVO) with some -rules. This may be +rules. + +\note{Phase 4 run can be defined by selecting an observation group, a + set of exposures, or a set of rules related to a spatial region (eg, + region, time range, and filter}. + +\note{Phase 4 discussion (and diagram) needs more work} \section{Analysis Version and Recipes} @@ -376,6 +421,5 @@ it possible to run and re-run the analysis at any stage without deleting the earlier results. As different analysis attempts are -performed for a given image, the versions branch out, like the diagram -in Figure NNN. +performed for a given image, the versions branch out. Also note that at every stage, the entries include a recipe @@ -411,20 +455,21 @@ \begin{figure} \begin{center} -\resizebox{4in}{!}{\includegraphics{pics/pstasks.08.ps}} +\includegraphics[scale=0.85]{pics/pstasks.08.ps} \caption{\label{detrend} Detrend Creation Tasks} \end{center} \end{figure} -This diagram illustrates the tables needed for the generic detrend -construction process, using the flat-field construction as an example. -This diagram is somewhat more complex than the preceeding versions. -In this diagram, both single jobs and multiple jobs are represented by -the process elements (the blue ellipses). In some cases, more that -one task will be needed to perform the function illustrated by a -single process task. The complexity of this diagram is enhanced by -the need for multiple iterations and both single chip and full mosaic -processing. At the moment, the distinction between mosaic and single -chip cameras is not specifically discussed. Finally, the triggers -which initiate a specific detrend analysis are glossed over. +Figure~\ref{detrend} illustrates the tables needed for the generic +detrend construction process, using the flat-field construction as an +example. This diagram is somewhat more complex than the preceeding +versions. In this diagram, both single jobs and multiple jobs are +represented by the process elements (the blue ellipses). In some +cases, more that one task will be needed to perform the function +illustrated by a single process task. The complexity of this diagram +is enhanced by the need for multiple iterations and both single chip +and full mosaic processing. At the moment, the distinction between +mosaic and single chip cameras is not specifically discussed. +Finally, the triggers which initiate a specific detrend analysis are +glossed over. The detrend analysis is initiated by choosing a type of detrend image @@ -506,11 +551,241 @@ flat-field images, the same structure should be capable of constructing the biases, dark, fringes, etc. In some cases, as noted -above, the 'process' stage is a null operation. +above, the `process' stage is a null operation. + +\pagebreak + +\appendix +\section{IPP top-level commands} + +In this section, we define all the necessary top-level commands needed +by the IPP to implement the data flow discussed in the previous +sections. These commands are user commands, and are visible in the +UNIX command space. The commands are discussed in the context of both +the automatic processing and processing of individual entries. It is +an important goal that the user commands should provide a +simple-enough interface that they can be used without needing to have +the entire infrastructure of the IPP in place to function. In +practice, this means that data items which can be acquired from the +Metadata DB tables can also be provided on the command line and/or in +the recipe file. In some cases, user commands are provided to allow a +manual intervention beyond the automatic processing loops. This is +particularly true of the \code{submit.Px} type of commands. + +\note{the command names are for illustration purposes only. any + suggestions for better / fancier names are welcome...} + +\begin{verbatim} + +copy.image + input: url, exp ID, camera, class, class ID + + This program copies the image file from the given URL, updates the + new image file table with the descriptive metadata (exp ID, camera, + class, class ID), and notifies the external subsystem that the copy + has succeeded. The destination host for the image file is + determined by selecting the class ID from the host-for-class table. + +copy.table + input: url, table ID, table type + + This program copies the metadata table file from the given URL, + determines which IPP metadata table it corresponds to, adds the + table rows to that metadata table, and notifies the external + subsystem that the table copy has succeeded. + +classify.image + input: (url) or (exp ID + class ID) + + This program examines the header of the specified image file, + determines the image type (detrend, science, etc), extracts a + specific set of information from the header, and adds a new entry to + the raw image files table. It also checks the new exposure table + for the corresponding exp ID, counts the files with the same exp ID + that are in the raw image files table, and if they match, adds an + entry to the raw exposure table for the appropriate exposure type. + Depending on the camera format (mosaic / single chip), the process + also adds an entry to either the P1 exposure table or the P2 image table. + +Phase.1 + input: exp ID + + This program determines the chips which correspond the exposure, + loads the guide stars (if they exist) or reads the pixels around + bright stars (if the guide stars do not exit. It determines the + centroids for these stars, reads in the astrometric reference stars + in the field, matches observed stars to reference stars, determines + an astrometric solution, using the default telescope / camera model + as the starting point, and writes out the result to the P1 output + file, along with the FITS table of observed star measurements (X, Y, + inst mag). It writes the summary statistics back to the P1 exposure + table, an new entry in the P2 exposure table, and entries for each + of the image files in the P2 images table. + +Submit.P2 + input: exposure selection criteria + + This program uses the provided selection criteria (eg, exp ID, time + range + filter, etc) and selects the corresponding exposures and + image files. If the output is specified to the database, it creates + new entries in the P2 images and P2 exposure tables, incrementing + the version fields if these exposure already exist. If the output + is specified to stdout / file, it writes the corresponding + information in a human (and/or machine) readable format. Note: it + is not necessary that a P2 exposure defined by this tool include all + possible or available image files. Note also: this program must + validate that the selected exposures have completed Phase 1, or are + not required to complete Phase 1. + +Phase.2 + input: (url) or (exp ID + class I) + + This program reads in the selected image file, determines the + matching detrend images, detrends the image data, performs object + detection and analysis, loads the corresponding + astrometric/photometric reference data, determines the astrometric + and photometric calibrations. It writes out the detrended image, + the corresponding mask, and a FITS table of the measured objects, + with the astrometric parameters in the FITS table header. It + updates the P2 images table with the statistics of the analysis, and + updates the P2 exposure table status. If this analysis is the last + of the Nclass P2 image analyses to be done on this P2 exp ID and + version, then the program makes a new entry in the P3 exposure + table. + +Submit.P3 + input: exposure selection criteria + + This program uses the provided selection criteria (eg, exp ID, time + range + filter, etc) and selects the corresponding exposures which + have succeeded in Phase 2 (a specific P2 version may be specified). + If the output is specified to the database, it creates new entries + in the P3 exposure table, incrementing the version field if this + exposure already exists. If the output is specified to stdout / + file, it writes the corresponding information in a human (and/or + machine) readable format. + +Phase.3 + input: (exp ID) or (list of P2 object files) + + This program finds the collection of P2 object files for the + specified exposure, reads in the object data and astrometric + calibration terms, loads the corresponding astrometric/photometric + reference data, determines the improved astrometric and photometric + calibrations. It writes out a single FITS table of the measured + objects, with the astrometric parameters in the FITS table header, + and astrometric calibration data in a new astrometry file for this + exposure. It updates the P3 exposure table with the statistics of + the analysis + +Submit.P4 + input: selection criteria + + This program uses the provided selection criteria (eg, exp ID list, + observation group ID, sky region + time range + filter, etc) and + selects the corresponding exposures which have succeeded in Phase 3 + (or Phase 2 if Phase 3 is not needed for this camera / recipe). It + determines the matching sky cells which are overlapped by the + selected exposures. If the output is specified to the database, it + creates new entries in the P4 run table and the P4 run input table. + If the output is specified to stdout / file, it writes the + corresponding information in a human (and/or machine) readable + format. + +Phase.4 + input: (run ID) or (sky cell + list of P2 / P3 image files) + + This program uses the skycell and the list of input exposures to + find the collection of image files which overlap the given skycell. + It then reads the image pixels, warps them to match the skycell + geometry, and stacks the image pixels. It reads the pixels from the + skycell and performs the image difference analysis, it photometers + the difference image, cleans the input summed image on the basis of + the detections, and photometers the input summed image. It writes + out the statistics of the analysis to the P4 run table, the P4 sum + table, and the P4 delta table. If requested, it improves the + signal-to-noise in the skycell image, and updates the skycell table. + +Detrend.init + input: (detrend creation criteria) + + This program adds a new entry to the master detrend run table based + on the given criteria. The criteria may define a time range for the + input images, a detrend type, flux ranges, a filter, an observation + group ID. The new master detrend run is created with the iteration + set to 00 and the state set to `new'. + +Detrend.get.input.list + input: det ID, version + + This program uses the selection criteria defined by the master + detrend run (eg, exp ID list, observation group ID, time range, type + + filter, etc) to select the corresponding detrend exposures. It + creates a new entry in the master detrend frames table, with state + `new' and iteration of 00. It also writes the exposures to the + input detrend exposure table. + +Detrend.get.images + input: exp ID + + This program identifies the detrend images which correspond to the + selected detrend exposure ID and adds an entries for the image files + in the input detrend images table, with state set to `raw'. + +Detrend.process + input: url / detrend type + + This program performs the pre-processing needed by the selected + detrend image in order to prepare it for combination with other + detrend images. When completed, the entry for this file in the + input detrend images table is update to `proc'. The processing may + be as simple as a null operation (eg, for a bias) or as complex as + bias, dark, flat-field, renormalize (eg, for a fringe). + +Detrend.stack + input: det ID, version, iteration, class ID? + + This program performs the image stacking for a single image class + ID. The stacking process may depend on the detrend type (different + for bias from flat from fringe). The resulting statitics are + written to the master detrend image table. If this is the last of + Nclass entries for the given master detrend frame, then the master + detrend frames state is updated. + +Detrend.merge + input: det ID, version, iteration + + This program examines the results of the Nclass master detrend + images and performs any necessary re-normalizations. It also + examines the statistics of the individual stacks and summarizes them + in the master detrend frame table.. + +Detrend.residuals + input: det ID, version, iteration, class ID, url + + This program performs the detrending on the given processed input + image using the corresponding detrend frame. The residual image is + saved, and an entry is written to the residual images table giving + the image location and the residual statistics for this image. + +Detrend.assess + input: det ID, version, iteration + + This program examines the collection of residual image statistics + and creates ensemble statistics for each residual exposure (NOTE: do + we need a residual exposure table??). It examines the ensemble of + exposure statistics and determines if 1) the complete stack meets + the success criteria and 2) which residual exposure do / do not meet + the success criteria. It updates the master detrend run table, the + master detrend frame table, and the residual image / exposure table + to note images which should be included / excluded in a future + iteration. +\end{verbatim} + \section{Metadata Database Tables used for IPP Job Flow} \label{sec:MetadataTableContents} -The tables presented here define in greater detail the contents of the -Metadata tables show in the figures above. In some cases, the +The tables presented below define in greater detail the contents of +the Metadata tables shown in the figures above. In some cases, the quantities (eg, the analysis result statistics) are illustrative, not definitive. In certain tables, data is provided which is redundant @@ -523,5 +798,5 @@ \begin{center} \caption{Pending Image Files\label{tab:PendingImageFiles}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -542,5 +817,5 @@ \begin{center} \caption{New Image Files\label{tab:NewImageFiles}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -559,6 +834,22 @@ \begin{table}[bh] \begin{center} +\caption{Host for Class\label{tab:HostForClass}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +camera & string & camera name & MegaPrime / GPC \\ +class ID & string & identify for class & chip00 / cell0102 \\ +hostname & string & preferred host computer & po01 / alala.ifa.hawaii.edu \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} \caption{Raw Image Files\label{tab:RawImageFiles}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -578,5 +869,5 @@ FHWM & float & average image quality & 2.5 (arcsec) \\ \hline -\multicolumn{3}{l}{note: stats below the line are measured, perhaps they go elsewhere?} \\ +\multicolumn{4}{l}{note: stats below the line are measured, perhaps they go elsewhere?} \\ \end{tabular} \end{center} @@ -586,5 +877,5 @@ \begin{center} \caption{Pending Metadata Tables\label{tab:PendingTables}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -603,5 +894,5 @@ \begin{center} \caption{Copied Metadata Tables\label{tab:CopiedTables}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -621,5 +912,5 @@ \begin{center} \caption{New Exposures\label{tab:NewExp}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -639,5 +930,5 @@ \begin{center} \caption{Raw SCIENCE Exposure\label{tab:RawScienceExp}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -679,5 +970,5 @@ \begin{center} \caption{Raw Detrend Exposures\label{tab:RawDetrendExp}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -718,8 +1009,10 @@ \end{table} +\clearpage + \begin{table}[bh] \begin{center} \caption{P1 Exposures\label{tab:P1-Exp}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -740,9 +1033,32 @@ Moff & float & measure ZP offset & 0.5 (mag) \\ dMoff & float & scatter in Moff & 0.1 (mag) \\ +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{P2 Exposures\label{tab:P2-Exp}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +exp ID & string & exposure ID & 654321o \\ +class & string & file data class & Cell / Chip / FPA \\ +Nfiles & int & number of P2 image files & 3 \\ +Ndone & int & number completed & 2 \\ +P1 version & int & P1 version number & 01 \\ +P2 version & int & P2 version number & 01 \\ +state & string & P2 analysis state & new / done / fail? \\ +\hline +\end{tabular} +\end{center} +\end{table} \begin{table}[bh] \begin{center} \caption{P2 Images\label{tab:P2-Images}} -\begin{tabular}{lll} +\begin{tabular}{llll} \hline \hline @@ -752,7 +1068,6 @@ class & string & file data class & Cell / Chip / FPA \\ class ID & string & identify for class & chip00 / cell0102 \\ -version & int & P1 version number & 01 \\ recipe & string & analysis recipe name & basic / flattest \\ -state & string & P1 analysis state & new / done \\ +state & string & P2 analysis state & new / done / fail? \\ input.url & string & file location & rootname.fits \\ output.image.url & string & file location & rootname.P2.01.fits \\ @@ -762,4 +1077,8 @@ bias & float & measured bias value & 5.0 \\ dbias & float & bias residual scatter & 5.0 \\ +fringe & float & fringe amplitude & 10.0 \\ +dfringe & float & fringe scatter & 10.0 \\ +sky & float & reduced background & 5.0 \\ +dsky & float & background scatter & 5.0 \\ Nstars & int & number of astrom stars & 50 \\ sigma.X & float & astrom scatter in X & 1.0 (arcsec) \\ @@ -768,15 +1087,236 @@ Moff & float & measure ZP offset & 0.5 (mag) \\ dMoff & float & scatter in Moff & 0.1 (mag) \\ -\hline -bias-image -dark-image -flat-image -\hline -\end{tabular} -\end{center} -\end{table} - - - +runtime & float & processing time & 20.2 (sec) \\ +\hline +\multicolumn{4}{l}{note: the stats below the line are examples to be extended} \\ +\end{tabular} +\end{center} +\end{table} + +The P2 analysis produces a reduced (P2) output image, a table of (P2) +measured objects, and astrometric calibration terms. The output +objects are stored as a FITS table. The astrometric solution for the +image is stored in the header of the output object file. Interesting +statistics from the analysis are written to the MDDB table. +Additional information describing the analysis is written to the log +file in a machine-readable format. Additional entries can be added to +the MDDB table if we find they are useful for understanding the +processing results. + +\begin{table}[bh] +\begin{center} +\caption{P3 Exposures\label{tab:P3-Exp}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +exp ID & string & exposure ID & 654321o \\ +class & string & file data class & Cell / Chip / FPA \\ +Nfiles & int & number of P2 image files & 3 \\ +Ndone & int & number completed & 2 \\ +P1 version & int & P1 version number & 01 \\ +P2 version & int & P2 version number & 01 \\ +P3 version & int & P3 version number & 01 \\ +state & string & P3 analysis state & new / done / fail? \\ +recipe & string & analysis recipe name & basic / flattest \\ +output & string & location of output file & rootname.P3.01.smf \\ +P3-log & string & location of P3 logfile & rootname.P3.01.log \\ +\hline +Nstars & int & number of astrom stars & 50 \\ +sigma.X & float & astrom scatter in X & 1.0 (arcsec) \\ +sigma.Y & float & astrom scatter in Y & 1.2 (arcsec) \\ +Mcal & float & nominal zeropoint & 25.2 (mag) \\ +Moff & float & measure ZP offset & 0.5 (mag) \\ +dMoff & float & scatter in Moff & 0.1 (mag) \\ +runtime & float & processing time & 20.2 (sec) \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\clearpage + +\begin{table}[bh] +\begin{center} +\caption{Master Dark Frames\label{tab:DarkFrames}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +det ID & string & detrend frame ID & 654321o \\ +version & int & version of det frame & 02 \\ +label & string & descriptive name & basic \\ +recipe & string & creation recipe & basic \\ +type & string & detrend type & bias / dark / flat \\ +class & string & file data class & Cell / Chip / FPA \\ +Nfiles & int & number of files & 2 \\ +\hline +exptime & float & exposure time & 5.0 \\ +Ninput & int & number of input frames & 5 \\ +sigma & float & stack residual stats & 2.0 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Master Flat Frames\label{tab:FlatFrames}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +det ID & string & detrend frame ID & 654321o \\ +version & int & version of det frame & 02 \\ +label & string & descriptive name & basic \\ +type & string & detrend type & bias / dark / flat \\ +class & string & file data class & Cell / Chip / FPA \\ +Nfiles & int & number of files & 2 \\ +\hline +filter.ID & string & filter ID (glass) & g.PS1.01 / none \\ +filter.Band & string & filter bandpass name & g / none \\ +Ninput & int & number of input frames & 5 \\ +sigma & float & stack residual stats & 2.0 \\ +Ntotal & float & ave.total input counts & 1e6 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Master Fringe Frames\label{tab:FringeFrames}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +det ID & string & detrend frame ID & 654321o \\ +version & int & version of det frame & 02 \\ +label & string & descriptive name & basic \\ +type & string & detrend type & bias / dark / flat \\ +class & string & file data class & Cell / Chip / FPA \\ +Nfiles & int & number of files & 2 \\ +\hline +filter.ID & string & filter ID (glass) & g.PS1.01 / none \\ +filter.Band & string & filter bandpass name & g / none \\ +Ninput & int & number of input frames & 5 \\ +sigma & float & stack residual stats & 2.0 \\ +Ntotal & float & ave.total input counts & 1e6 \\ +fringe & float & fringe amplitude & 52.0 \\ +dfringe & float & fringe stdev & 5.0 \\ +airmass.min & float & minimum airmass & 1.00 \\ +airmass.max & float & maximum airmass & 1.20 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Master Detrend Image Files\label{tab:DetImageFiles}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +URL & string & file location & neb://file001.fits \\ +det ID & string & detrend frame ID & 654321o \\ +{\it camera} & string & camera name & MegaPrime / GPC \\ +class & string & file data class & Cell / Chip / FPA \\ +class ID & string & identify for class & chip00 / cell0102 \\ +{\it type} & string & exposure type & bias / flat / science \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\clearpage + +\begin{table}[bh] +\begin{center} +\caption{Master Detrend Run\label{tab:DetRun}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +det ID & string & detrend frame ID & 654321o \\ +camera & string & camera name & MegaPrime / GPC \\ +version & int & version of det run & 02 \\ +type & string & detrend type & bias / dark / flat \\ +criteria & string & image selection criteria & filter flags \\ +state & string & analysis state & new / done \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Input Detrend Exp\label{tab:InputDetExp}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +det ID & string & detrend frame ID & 654321o \\ +{\it camera} & string & camera name & MegaPrime / GPC \\ +{\it version} & int & version of det run & 02 \\ +exp ID & string & exposure ID & 654321o \\ +state & string & analysis state & new / raw / proc \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Input Detrend Image Files\label{tab:InputDetImageFiles}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +raw.URL & string & location of raw file & neb://file001.fits \\ +proc.URL & string & location of proc file & neb://file001.fits \\ +det ID & string & detrend frame ID & 654321o \\ +{\it version} & int & version of det run & 02 \\ +{\it camera} & string & camera name & MegaPrime / GPC \\ +{\it type} & string & exposure type & bias / flat / science \\ +class & string & file data class & Cell / Chip / FPA \\ +class ID & string & identify for class & chip00 / cell0102 \\ +state & string & analysis state & new / raw / proc \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Input Detrend Resid Image Files\label{tab:InputDetResidImageFiles}} +\begin{tabular}{llll} +\hline +\hline +{\bf Field Name} & {\bf Datatype } & {\bf Description} & {\bf Examples} \\ +\hline +image.URL & string & location of resid image & neb://file001.fits \\ +thumbnail.URL & string & location of resid jpeg & neb://file001.fits \\ +det ID & string & detrend frame ID & 654321o \\ +version & int & version of det run & 02 \\ +iteration & int & det creation iteration & 03 \\ +{\it camera} & string & camera name & MegaPrime / GPC \\ +{\it type} & string & exposure type & bias / flat / science \\ +class & string & file data class & Cell / Chip / FPA \\ +class ID & string & identify for class & chip00 / cell0102 \\ +state & string & analysis state & new / raw / proc \\ +scatter & float & residual scatter & 1.0 \\ +\hline +\end{tabular} +\end{center} +\end{table} \end{document}