IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 4976


Ignore:
Timestamp:
Sep 7, 2005, 6:31:17 PM (21 years ago)
Author:
eugene
Message:

added pics and description

Location:
trunk/doc/ipptools
Files:
2 added
10 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/ipptools

    • Property svn:ignore set to
      *.log
      *.dvi
      *.aux
      *.toc
      *.log
      *.out
      *.lof
      *.tbr
      *.tbd
      pstask.pdf

  • trunk/doc/ipptools/pstask.tex

    r4902 r4976  
    1818\maketitle
    1919
     20\tableofcontents
     21\pagebreak
     22\pagenumbering{arabic}
     23
    2024\section{Overview}
    2125
    … …  
    2630(Phase 1-4, detrend creation, etc) are illustrated in this document
    2731and the relevant MDDB tables are listed.  The collection of diagrams
    28 shows the IPP tasks and the Metadata Database (MDDB) tables needed to
    29 manage the flow of data through the system.
     32shows the IPP tasks and the Metadata Database tables needed to manage
     33the flow of data through the system.
     34
     35\note{The flat-field correction analysis stage is not yet defined in
     36  this document}
     37
     38\note{AP DB interfaces are mostly ignored}
    3039
    3140\section{PStask queues and metadata tables : persistent vs ephemeral state}
    … …  
    3342\begin{figure}
    3443\begin{center}
    35 \resizebox{4in}{!}{\includegraphics{pics/pstasks.01.ps}}
     44\includegraphics[scale=0.85]{pics/pstasks.01.ps}
    3645\caption{\label{queues} PStask queues and MDDB tables}
    3746\end{center}
    3847\end{figure}
    3948
    40 The left-hand portion of the diagram illustrates the recommended
    41 interaction between the metadata database tables and the scheduler
    42 internal queues.  Some table in the metadata database defines a list
    43 of data items which are to be processed by some analysis job.  The
    44 scheduler uses a two-step approach to define the analysis jobs based
    45 on this list.  First, one scheduler task queries the MDDB for a list
    46 of pending items, adds the returned items to an internal scheduler
    47 queue.  The process of adding the elements to the queue is defined so
    48 that only unique items are added: already existing items are skipped.
    49 The entries in the queue consist of the data items of interest and an
    50 internal temporary state.  At first, this would be 'pending'.  A
    51 second scheduler tasks pops 'pending' entries one-by-one from this
    52 internal queue, submits a job based on the entry, and sets the
    53 temporary state in the internal queue to 'running'.  The internal
     49There are some subtleties in the interaction between IPP task
     50scheduler, PStasks, the metadata tables which store the system state,
     51and the jobs which are immediately being performed.  There is a choice
     52to be made between rigorously maintaining the system state in the
     53Metadata DB at all times or in keeping an intermediate set of state
     54tables.  Keeping the exact system state in the Metadata DB tables
     55would require more queries to/from the database and may introduce
     56additional latencies which are undesirable.  This is because any
     57attempt by the scheduler to initiate a new job would require the
     58scheduler to mark the corresponding data item in the Metadata DB (the
     59item which acts as the trigger) with a `pending' state, and then mark
     60it again as `done' when the job actually completes.  This also has the
     61drawback that, if the scheduler crashes, some initial process would be
     62required on start up to find all Metadata DB items which are in the
     63`pending' state (examining all possible items which can be in such a
     64state) and reset them to the `new' state.
     65
     66We implement an alternative in which the scheduler maintains an
     67internal, ephemeral stack of the pending jobs, and only updates the
     68system state entries in the Metadata DB when jobs are actually
     69completed.  In this scenario, as far as the Metadata DB tables are
     70concerned, data items transition only between a `new' and a `done'
     71state.  Any jobs which are pending when the system crashes or the
     72power is lost are simply dropped, and will be automatically
     73re-constructed when the system restarts.  In this paradigm, no
     74intermediate operation state is saved, and no partially completed job
     75can be recovered.  Since the IPP is defined in terms of a fine
     76granularity, with jobs lasting no more than 30 - 120 seconds, this
     77model will not have a large impact.
     78
     79Figure~\ref{queues} illustrates this ephemeral vs persistent state
     80information and the interation between the metadata tables and
     81PStasks.  The left-hand portion of the diagram illustrates the
     82recommended interaction between the metadata database tables and the
     83scheduler internal queues.  Some table in the metadata database
     84defines a list of data items which are to be processed by some
     85analysis job.  The scheduler uses a two-step approach to define the
     86analysis jobs based on this list.  First, one scheduler task queries
     87the MDDB for a list of pending items, adds the returned items to an
     88internal scheduler queue.  The process of adding the elements to the
     89queue is defined so that only unique items are added: already existing
     90items are skipped.  The entries in the queue consist of the data items
     91of interest and an internal temporary state.  At first, this would be
     92`pending'.  A second scheduler tasks pops `pending' entries one-by-one
     93from this internal queue, submits a job based on the entry, and sets
     94the temporary state in the internal queue to `running'.  The internal
    5495state is needed to prevent the scheduler from re-submitting a job for
    5596the same data item before the first job is done or assessed.  Since
    … …  
    59100updated noting the completion.  This may be done either by the job
    60101itself or by the scheduler.  In addition, the state of the entry in
    61 the queue can be set to either 'done' or the entry can be simply
     102the queue can be set to either `done' or the entry can be simply
    62103removed from the queue.
    63104
    … …  
    83124copy of the image file from the location specified by the summit data
    84125system to the appropriate location within the IPP Image Server
    85 (Nebulous).  (As an alternative to the above, the 'pending images'
    86 table may be part of the summit database system, and the 'get images'
    87 command may query the summit directly.  In this scenario, the 'copy
     126(Nebulous).  (As an alternative to the above, the `pending images'
     127table may be part of the summit database system, and the `get images'
     128command may query the summit directly.  In this scenario, the `copy
    88129image' command reports to the summit data system that an individual
    89130image file has been copied.)
    … …  
    97138\begin{figure}
    98139\begin{center}
    99 \resizebox{4in}{!}{\includegraphics{pics/pstasks.02.ps}}
     140\includegraphics[scale=0.85]{pics/pstasks.02.ps}
    100141\caption{\label{pcopy} Summit Copy Tasks}
    101142\end{center}
    102143\end{figure}
    103144
    104 This diagram illustrates the MDDB tables used to copy data (images and
    105 metadata tables) from the summit.  The left-hand portion of the
    106 diagram shows the tables involved in copying images from the summit
    107 system.  The table of pending image files lists the URLs of the
     145Figure~\ref{pcopy} illustrates the MDDB tables used to copy data
     146(images and metadata tables) from the summit.  The left-hand portion
     147of the diagram shows the tables involved in copying images from the
     148summit system.  The table of pending image files lists the URLs of the
    108149individual image files available for transfer, along with their
    109150associated exposure ID and the camera which generated the image.  Two
    110151other entries assist in interpreting the file: the class and the class
    111152ID.  The final entry in this table is the current copy state of the
    112 file, can have the value of 'ready' or 'copied'.
     153file, can have the value of `ready' or `copied'.
    113154
    114155The class defines the data grouping represented by this image file and
    … …  
    124165corresponding to this file.  This value is necessary to make decisions
    125166on how to copy the data based on the chip / cell before the data is
    126 available to IPP components.  Here are likely values for the class and
    127 class ID for some common cameras:
    128 
    129 \begin{table}
    130 \begin{center}
    131 \caption{Camera and Data Classes\label{classes}}
    132 \begin{tabular}{lll}
    133 \hline
    134 \hline
    135 camera   & class  & classID \\
    136 \hline
    137 GPC      & chip   & chip02 \\
    138 skyprobe & fpa    & sp01 \\
    139 Megacam  & fpa    & MegacamSpliced \\
    140 Suprime  & chip   & chip0 \\
    141 \hline
    142 \end{tabular}
    143 \end{center}
    144 \end{table}
    145 
    146 and so forth.  The system described is sufficiently flexible to allow
    147 us to transfer the GPC images by cell if we eventually decide that is
    148 more efficient. 
     167available to IPP components.  Table~\ref{classes} lists likely values
     168for the class and class ID for some common cameras.  The system
     169described is sufficiently flexible to allow us to transfer the GPC
     170images by cell if we eventually decide that is more efficient.
    149171
    150172The copy process copies the file from the given URL to the appropriate
    … …  
    158180server?  eg: ref:DIR0001/file0001.fits might be in a directory which
    159181is defined in a table of directories.) After an image file is
    160 successfully copied, the corresponding state in the 'pending chip'
    161 table is updated from 'ready' to 'copied'.
     182successfully copied, the corresponding state in the `pending chip'
     183table is updated from `ready' to `copied'.
     184
     185\begin{table}
     186\begin{center}
     187\caption{Camera and Data Classes\label{classes}}
     188\begin{tabular}{llll}
     189\hline
     190\hline
     191camera   & class  & classID \\
     192\hline
     193GPC      & chip   & chip02 \\
     194skyprobe & fpa    & sp01 \\
     195Megacam  & fpa    & MegacamSpliced \\
     196Suprime  & chip   & chip0 \\
     197\hline
     198\end{tabular}
     199\end{center}
     200\end{table}
    162201
    163202The right hand portion of this diagram illustrates the process of
    … …  
    177216\begin{figure}
    178217\begin{center}
    179 \resizebox{4in}{!}{\includegraphics{pics/pstasks.03.ps}}
     218\includegraphics[scale=0.85]{pics/pstasks.03.ps}
    180219\caption{\label{phase0} Phase 0 Tasks}
    181220\end{center}
    182221\end{figure}
    183222
    184 This diagram illustrates phase 0, in which the image files are
     223Figure~\ref{phase0} illustrates phase 0, in which the image files are
    185224categorised, examined for summary information and basic statistics,
    186225and moved to the tables used to trigger further analysis.  The process
    187 first examines the 'new image files' table.  It selects images from
    188 this table which have not yet been examined (state is 'new').  The
     226first examines the `new image files' table.  It selects images from
     227this table which have not yet been examined (state is `new').  The
    189228file header is examined and relevant metadata is extracted (eg, RA,
    190229DEC, times, and so forth to be defined later).  The process may also
    … …  
    192231background level.  These statistics, whether derived from the header
    193232or the pixel values, are placed along with image summary information
    194 in the 'raw image files' table, and the state field of the 'new image
    195 files' table is set to 'ready'. 
     233in the `raw image files' table, and the state field of the `new image
     234files' table is set to `ready'.
    196235
    197236The process is also responsible for moving the exposures to the tables
    198237used for triggering the analysis process.  If the image class is FPA,
    199238the image can be advanced without waiting for any other image files.
    200 If the class is Chip or Cell, the process must also examine the 'new
     239If the class is Chip or Cell, the process must also examine the `new
    201240exposure' table for this exposure ID.  The number of class files
    202241available for this exposure is listed in this table.  The process must
    203242the select all image files matching the exposure ID with state of
    204 'ready' and compare the number avalable to the number expected.  If
     243`ready' and compare the number avalable to the number expected.  If
    205244the two match, then a new exposure is ready.  Based on the image type
    206245(from the most recently examined image file header or new exp table?),
    207 the exposure is added to the 'raw exposure' table for images of that
    208 type.  The allowed types are 'detrend', (all bias, dark, flat images),
    209 'object', 'focus'(??), etc.  (** The different tables represent
     246the exposure is added to the `raw exposure' table for images of that
     247type.  The allowed types are `detrend', (all bias, dark, flat images),
     248`object', `focus'(??), etc.  (** The different tables represent
    210249different analysis modes.  This process also adds an entry to the exp
    211250ID / image file match **).  This process also adds all science
    … …  
    218257\begin{figure}
    219258\begin{center}
    220 \resizebox{4in}{!}{\includegraphics{pics/pstasks.04.ps}}
     259\includegraphics[scale=0.85]{pics/pstasks.04.ps}
    221260\caption{\label{phase1} Phase 1 Tasks}
    222261\end{center}
    223262\end{figure}
    224263
    225 This diagram shows the tables involved in running the P1 analysis
    226 stage.  There are paths for exposures to enter the analysis
     264Figure~\ref{phase1} shows the tables involved in running the P1
     265analysis stage.  There are paths for exposures to enter the analysis
    227266automatically from the P0 analysis (arrow on left) or to be added
    228267manually based on a selection from the raw exposure table.  Exposures
    229268to be analysed by Phase 1 are added to the P1 exposure table with the
    230 state 'new'.  Exposures may be added multiple times for processing and
     269state `new'.  Exposures may be added multiple times for processing and
    231270reprocessing. The P1 exp table keeps a record of the old attempts for
    232271debugging and analysis.  Each time an exposure is added to the P1 exp
    … …  
    257296\begin{figure}
    258297\begin{center}
    259 \resizebox{4in}{!}{\includegraphics{pics/pstasks.05.ps}}
     298\includegraphics[scale=0.85]{pics/pstasks.05.ps}
    260299\caption{\label{phase2} Phase 2 Tasks}
    261300\end{center}
    262301\end{figure}
    263302
    264 This diagram shows the tables involved in running the P2 analysis
    265 stage.  There are paths for images to enter the analysis automatically
    266 from the P1 analysis (arrow on left) or to be added manually based on
    267 a selection from the raw exposure and raw image file tables.  Images
    268 to be analysed by Phase 2 are added to the P2 image table with the
    269 state 'new'.  When images are added to this table, a single entry is
    270 also added to the P2 exposure table listing the P1 and P2 versions for
    271 this exposure.  These version numbers must be integers starting with
    272 1.  If this image did not have a P1 analysis, the P1 version is set to
    273 0.  Exposures may be added multiple times for processing and
    274 reprocessing. The P2 image table keeps a record of the old attempts
    275 for debugging and analysis.  As with P1, each time a collection of
    276 associated images from an exposure is added to the P2 image table, it
    277 is given a new, unique version number, allowing the system as a whole
    278 to track different analysis attempts.  The top portion of the diagram
    279 represents the user-space tool which may be used to re-submit the
    280 images for an exposure or a group of exposures, potentially selected
    281 on the basis of a query from the raw SCIENCE exposure and raw image
    282 file tables.
    283 
    284 The P2 image table is examined to select the 'new' images.  These
     303Figure~\ref{phase2} shows the tables involved in running the P2
     304analysis stage.  There are paths for images to enter the analysis
     305automatically from the P1 analysis (arrow on left) or to be added
     306manually based on a selection from the raw exposure and raw image file
     307tables.  Images to be analysed by Phase 2 are added to the P2 image
     308table with the state `new'.  When images are added to this table, a
     309single entry is also added to the P2 exposure table listing the P1 and
     310P2 versions for this exposure.  These version numbers must be integers
     311starting with 1.  If this image did not have a P1 analysis, the P1
     312version is set to 0.  Exposures may be added multiple times for
     313processing and reprocessing. The P2 image table keeps a record of the
     314old attempts for debugging and analysis.  As with P1, each time a
     315collection of associated images from an exposure is added to the P2
     316image table, it is given a new, unique version number, allowing the
     317system as a whole to track different analysis attempts.  The top
     318portion of the diagram represents the user-space tool which may be
     319used to re-submit the images for an exposure or a group of exposures,
     320potentially selected on the basis of a query from the raw SCIENCE
     321exposure and raw image file tables.
     322
     323The P2 image table is examined to select the `new' images.  These
    285324images are used to generate P2 analysis jobs.  The P2 analysis uses
    286325the input url to find and load the image file.  The url may be a file
    … …  
    293332objects, including the astrometry calibration, are written to the P2
    294333image table, along with summary statistics from the P2 analysis.  The
    295 state is also updated (to 'done'). 
     334state is also updated (to `done'). 
    296335
    297336Whenever the exposure is completed, the value of Ndone in the P2
    … …  
    305344\begin{figure}
    306345\begin{center}
    307 \resizebox{4in}{!}{\includegraphics{pics/pstasks.06.ps}}
     346\includegraphics[scale=0.85]{pics/pstasks.06.ps}
    308347\caption{\label{phase3} Phase 3 Tasks}
    309348\end{center}
    310349\end{figure}
    311350
    312 This diagram illustrates the tables involved in the Phase 3 analysis.
    313 The P3 exposure table lists the exposure ID, the P3 analysis version,
    314 the P2 analysis version to be used as input to this P3 analysis, and
    315 the recipe to be used.  The P2 exposure and image tables are used, in
    316 conjunction with the P2 version information, to select the P2 output
    317 measured objects and the astrometric calibrations from P2 and P1.
    318 These measured objects are matched with the reference catalog objects,
    319 and calibrated astrometry and photometry is produced for the full
    320 exposure.  The location of the resulting astometry calibration table
    321 is stored back in the P3 exposure table.  If the recipe file
    322 specifies, the 2-D photometric and background / fringe corrections may
    323 also be performed at this stage.  Since these analyses require
    324 reference data, the recipe may also be used to skip these analysis if
    325 such reference data is unavailable or unreliable.  At the end of Phase
    326 3, the objects from the exposure are inserted into the photometry
    327 database (this is not shown).
     351Figure~\ref{phase3} illustrates the tables involved in the Phase 3
     352analysis.  The P3 exposure table lists the exposure ID, the P3
     353analysis version, the P2 analysis version to be used as input to this
     354P3 analysis, and the recipe to be used.  The P2 exposure and image
     355tables are used, in conjunction with the P2 version information, to
     356select the P2 output measured objects and the astrometric calibrations
     357from P2 and P1.  These measured objects are matched with the reference
     358catalog objects, and calibrated astrometry and photometry is produced
     359for the full exposure.  The location of the resulting astometry
     360calibration table is stored back in the P3 exposure table.  If the
     361recipe file specifies, the 2-D photometric and background / fringe
     362corrections may also be performed at this stage.  Since these analyses
     363require reference data, the recipe may also be used to skip these
     364analysis if such reference data is unavailable or unreliable.  At the
     365end of Phase 3, the objects from the exposure are inserted into the
     366photometry database (this is not shown).
    328367
    329368The astrometric calibration portion of Phase 3 is principally needed
    … …  
    340379\begin{figure}
    341380\begin{center}
    342 %\resizebox{4in}{!}{\includegraphics{pics/pstasks.03.ps}}
     381\includegraphics[scale=0.85]{pics/pstasks.07.ps}
    343382\caption{\label{phase4} Phase 4 Tasks}
    344383\end{center}
    … …  
    363402image combination and image differencing.  It may even be the case
    364403that only the combination portion of Phase 4 is performed on the AP
    365 Survey data. 
     404Survey data.
    366405
    367406More generally, the image groups selected for Phase 4 analysis may be
    368407chosen on the basis of a query of the AP Database (DVO) with some
    369 rules.  This may be
     408rules. 
     409
     410\note{Phase 4 run can be defined by selecting an observation group, a
     411  set of exposures, or a set of rules related to a spatial region (eg,
     412  region, time range, and filter}.
     413
     414\note{Phase 4 discussion (and diagram) needs more work}
    370415
    371416\section{Analysis Version and Recipes}
    … …  
    376421it possible to run and re-run the analysis at any stage without
    377422deleting the earlier results.  As different analysis attempts are
    378 performed for a given image, the versions branch out, like the diagram
    379 in Figure NNN. 
     423performed for a given image, the versions branch out.
    380424
    381425Also note that at every stage, the entries include a recipe
    … …  
    411455\begin{figure}
    412456\begin{center}
    413 \resizebox{4in}{!}{\includegraphics{pics/pstasks.08.ps}}
     457\includegraphics[scale=0.85]{pics/pstasks.08.ps}
    414458\caption{\label{detrend} Detrend Creation Tasks}
    415459\end{center}
    416460\end{figure}
    417461
    418 This diagram illustrates the tables needed for the generic detrend
    419 construction process, using the flat-field construction as an example.
    420 This diagram is somewhat more complex than the preceeding versions.
    421 In this diagram, both single jobs and multiple jobs are represented by
    422 the process elements (the blue ellipses).  In some cases, more that
    423 one task will be needed to perform the function illustrated by a
    424 single process task.  The complexity of this diagram is enhanced by
    425 the need for multiple iterations and both single chip and full mosaic
    426 processing.  At the moment, the distinction between mosaic and single
    427 chip cameras is not specifically discussed.  Finally, the triggers
    428 which initiate a specific detrend analysis are glossed over.
     462Figure~\ref{detrend} illustrates the tables needed for the generic
     463detrend construction process, using the flat-field construction as an
     464example.  This diagram is somewhat more complex than the preceeding
     465versions.  In this diagram, both single jobs and multiple jobs are
     466represented by the process elements (the blue ellipses).  In some
     467cases, more that one task will be needed to perform the function
     468illustrated by a single process task.  The complexity of this diagram
     469is enhanced by the need for multiple iterations and both single chip
     470and full mosaic processing.  At the moment, the distinction between
     471mosaic and single chip cameras is not specifically discussed.
     472Finally, the triggers which initiate a specific detrend analysis are
     473glossed over.
    429474
    430475The detrend analysis is initiated by choosing a type of detrend image
    … …  
    506551flat-field images, the same structure should be capable of
    507552constructing the biases, dark, fringes, etc.  In some cases, as noted
    508 above, the 'process' stage is a null operation.
     553above, the `process' stage is a null operation.
     554
     555\pagebreak
     556
     557\appendix
     558\section{IPP top-level commands}
     559
     560In this section, we define all the necessary top-level commands needed
     561by the IPP to implement the data flow discussed in the previous
     562sections.  These commands are user commands, and are visible in the
     563UNIX command space.  The commands are discussed in the context of both
     564the automatic processing and processing of individual entries.  It is
     565an important goal that the user commands should provide a
     566simple-enough interface that they can be used without needing to have
     567the entire infrastructure of the IPP in place to function.  In
     568practice, this means that data items which can be acquired from the
     569Metadata DB tables can also be provided on the command line and/or in
     570the recipe file.  In some cases, user commands are provided to allow a
     571manual intervention beyond the automatic processing loops.  This is
     572particularly true of the \code{submit.Px} type of commands. 
     573
     574\note{the command names are for illustration purposes only.  any
     575  suggestions for better / fancier names are welcome...}
     576
     577\begin{verbatim}
     578
     579copy.image
     580  input: url, exp ID, camera, class, class ID
     581
     582  This program copies the image file from the given URL, updates the
     583  new image file table with the descriptive metadata (exp ID, camera,
     584  class, class ID), and notifies the external subsystem that the copy
     585  has succeeded.  The destination host for the image file is
     586  determined by selecting the class ID from the host-for-class table.
     587
     588copy.table
     589  input: url, table ID, table type
     590
     591  This program copies the metadata table file from the given URL,
     592  determines which IPP metadata table it corresponds to, adds the
     593  table rows to that metadata table, and notifies the external
     594  subsystem that the table copy has succeeded.
     595
     596classify.image
     597  input: (url) or (exp ID + class ID)
     598
     599  This program examines the header of the specified image file,
     600  determines the image type (detrend, science, etc), extracts a
     601  specific set of information from the header, and adds a new entry to
     602  the raw image files table.  It also checks the new exposure table
     603  for the corresponding exp ID, counts the files with the same exp ID
     604  that are in the raw image files table, and if they match, adds an
     605  entry to the raw exposure table for the appropriate exposure type.
     606  Depending on the camera format (mosaic / single chip), the process
     607  also adds an entry to either the P1 exposure table or the P2 image table.
     608 
     609Phase.1
     610  input: exp ID
     611
     612  This program determines the chips which correspond the exposure,
     613  loads the guide stars (if they exist) or reads the pixels around
     614  bright stars (if the guide stars do not exit.  It determines the
     615  centroids for these stars, reads in the astrometric reference stars
     616  in the field, matches observed stars to reference stars, determines
     617  an astrometric solution, using the default telescope / camera model
     618  as the starting point, and writes out the result to the P1 output
     619  file, along with the FITS table of observed star measurements (X, Y,
     620  inst mag).  It writes the summary statistics back to the P1 exposure
     621  table, an new entry in the P2 exposure table, and entries for each
     622  of the image files in the P2 images table.
     623
     624Submit.P2
     625  input: exposure selection criteria
     626
     627  This program uses the provided selection criteria (eg, exp ID, time
     628  range + filter, etc) and selects the corresponding exposures and
     629  image files.  If the output is specified to the database, it creates
     630  new entries in the P2 images and P2 exposure tables, incrementing
     631  the version fields if these exposure already exist.  If the output
     632  is specified to stdout / file, it writes the corresponding
     633  information in a human (and/or machine) readable format.  Note: it
     634  is not necessary that a P2 exposure defined by this tool include all
     635  possible or available image files.  Note also: this program must
     636  validate that the selected exposures have completed Phase 1, or are
     637  not required to complete Phase 1.
     638
     639Phase.2
     640  input: (url) or (exp ID + class I)
     641
     642  This program reads in the selected image file, determines the
     643  matching detrend images, detrends the image data, performs object
     644  detection and analysis, loads the corresponding
     645  astrometric/photometric reference data, determines the astrometric
     646  and photometric calibrations.  It writes out the detrended image,
     647  the corresponding mask, and a FITS table of the measured objects,
     648  with the astrometric parameters in the FITS table header.  It
     649  updates the P2 images table with the statistics of the analysis, and
     650  updates the P2 exposure table status.  If this analysis is the last
     651  of the Nclass P2 image analyses to be done on this P2 exp ID and
     652  version, then the program makes a new entry in the P3 exposure
     653  table. 
     654
     655Submit.P3
     656  input: exposure selection criteria
     657
     658  This program uses the provided selection criteria (eg, exp ID, time
     659  range + filter, etc) and selects the corresponding exposures which
     660  have succeeded in Phase 2 (a specific P2 version may be specified).
     661  If the output is specified to the database, it creates new entries
     662  in the P3 exposure table, incrementing the version field if this
     663  exposure already exists.  If the output is specified to stdout /
     664  file, it writes the corresponding information in a human (and/or
     665  machine) readable format.
     666
     667Phase.3
     668  input: (exp ID) or (list of P2 object files)
     669
     670  This program finds the collection of P2 object files for the
     671  specified exposure, reads in the object data and astrometric
     672  calibration terms, loads the corresponding astrometric/photometric
     673  reference data, determines the improved astrometric and photometric
     674  calibrations.  It writes out a single FITS table of the measured
     675  objects, with the astrometric parameters in the FITS table header,
     676  and astrometric calibration data in a new astrometry file for this
     677  exposure.  It updates the P3 exposure table with the statistics of
     678  the analysis
     679
     680Submit.P4
     681  input: selection criteria
     682
     683  This program uses the provided selection criteria (eg, exp ID list,
     684  observation group ID, sky region + time range + filter, etc) and
     685  selects the corresponding exposures which have succeeded in Phase 3
     686  (or Phase 2 if Phase 3 is not needed for this camera / recipe).  It
     687  determines the matching sky cells which are overlapped by the
     688  selected exposures.  If the output is specified to the database, it
     689  creates new entries in the P4 run table and the P4 run input table.
     690  If the output is specified to stdout / file, it writes the
     691  corresponding information in a human (and/or machine) readable
     692  format.
     693
     694Phase.4
     695  input: (run ID) or (sky cell + list of P2 / P3 image files)
     696
     697  This program uses the skycell and the list of input exposures to
     698  find the collection of image files which overlap the given skycell.
     699  It then reads the image pixels, warps them to match the skycell
     700  geometry, and stacks the image pixels.  It reads the pixels from the
     701  skycell and performs the image difference analysis, it photometers
     702  the difference image, cleans the input summed image on the basis of
     703  the detections, and photometers the input summed image.  It writes
     704  out the statistics of the analysis to the P4 run table, the P4 sum
     705  table, and the P4 delta table.  If requested, it improves the
     706  signal-to-noise in the skycell image, and updates the skycell table.
     707
     708Detrend.init
     709  input: (detrend creation criteria)
     710
     711  This program adds a new entry to the master detrend run table based
     712  on the given criteria.  The criteria may define a time range for the
     713  input images, a detrend type, flux ranges, a filter, an observation
     714  group ID.  The new master detrend run is created with the iteration
     715  set to 00 and the state set to `new'. 
     716
     717Detrend.get.input.list
     718  input: det ID, version
     719
     720  This program uses the selection criteria defined by the master
     721  detrend run (eg, exp ID list, observation group ID, time range, type
     722  + filter, etc) to select the corresponding detrend exposures.  It
     723  creates a new entry in the master detrend frames table, with state
     724  `new' and iteration of 00.  It also writes the exposures to the
     725  input detrend exposure table.
     726
     727Detrend.get.images
     728  input: exp ID
     729
     730  This program identifies the detrend images which correspond to the
     731  selected detrend exposure ID and adds an entries for the image files
     732  in the input detrend images table, with state set to `raw'.
     733
     734Detrend.process
     735  input: url / detrend type
     736
     737  This program performs the pre-processing needed by the selected
     738  detrend image in order to prepare it for combination with other
     739  detrend images.  When completed, the entry for this file in the
     740  input detrend images table is update to `proc'.  The processing may
     741  be as simple as a null operation (eg, for a bias) or as complex as
     742  bias, dark, flat-field, renormalize (eg, for a fringe).
     743
     744Detrend.stack
     745  input: det ID, version, iteration, class ID?
     746
     747  This program performs the image stacking for a single image class
     748  ID.  The stacking process may depend on the detrend type (different
     749  for bias from flat from fringe).  The resulting statitics are
     750  written to the master detrend image table.  If this is the last of
     751  Nclass entries for the given master detrend frame, then the master
     752  detrend frames state is updated.
     753
     754Detrend.merge
     755  input: det ID, version, iteration
     756
     757  This program examines the results of the Nclass master detrend
     758  images and performs any necessary re-normalizations.  It also
     759  examines the statistics of the individual stacks and summarizes them
     760  in the master detrend frame table..
     761
     762Detrend.residuals
     763  input:  det ID, version, iteration, class ID, url
     764
     765  This program performs the detrending on the given processed input
     766  image using the corresponding detrend frame.  The residual image is
     767  saved, and an entry is written to the residual images table giving
     768  the image location and the residual statistics for this image.
     769
     770Detrend.assess
     771  input: det ID, version, iteration
     772
     773  This program examines the collection of residual image statistics
     774  and creates ensemble statistics for each residual exposure (NOTE: do
     775  we need a residual exposure table??).  It examines the ensemble of
     776  exposure statistics and determines if 1) the complete stack meets
     777  the success criteria and 2) which residual exposure do / do not meet
     778  the success criteria.  It updates the master detrend run table, the
     779  master detrend frame table, and the residual image / exposure table
     780  to note images which should be included / excluded in a future
     781  iteration. 
     782\end{verbatim}
     783
    509784
    510785\section{Metadata Database Tables used for IPP Job Flow}
    511786\label{sec:MetadataTableContents}
    512787
    513 The tables presented here define in greater detail the contents of the
    514 Metadata tables show in the figures above.  In some cases, the
     788The tables presented below define in greater detail the contents of
     789the Metadata tables shown in the figures above.  In some cases, the
    515790quantities (eg, the analysis result statistics) are illustrative, not
    516791definitive.  In certain tables, data is provided which is redundant
    … …  
    523798\begin{center}
    524799\caption{Pending Image Files\label{tab:PendingImageFiles}}
    525 \begin{tabular}{lll}
     800\begin{tabular}{llll}
    526801\hline
    527802\hline
    … …  
    542817\begin{center}
    543818\caption{New Image Files\label{tab:NewImageFiles}}
    544 \begin{tabular}{lll}
     819\begin{tabular}{llll}
    545820\hline
    546821\hline
    … …  
    559834\begin{table}[bh]
    560835\begin{center}
     836\caption{Host for Class\label{tab:HostForClass}}
     837\begin{tabular}{llll}
     838\hline
     839\hline
     840{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     841\hline
     842camera            & string          & camera name              & MegaPrime / GPC           \\
     843class ID          & string          & identify for class       & chip00 / cell0102         \\
     844hostname          & string          & preferred host computer  & po01 / alala.ifa.hawaii.edu \\
     845\hline
     846\end{tabular}
     847\end{center}
     848\end{table}
     849
     850\begin{table}[bh]
     851\begin{center}
    561852\caption{Raw Image Files\label{tab:RawImageFiles}}
    562 \begin{tabular}{lll}
     853\begin{tabular}{llll}
    563854\hline
    564855\hline
    … …  
    578869FHWM              & float           & average image quality    & 2.5 (arcsec)              \\
    579870\hline
    580 \multicolumn{3}{l}{note: stats below the line are measured, perhaps they go elsewhere?}    \\
     871\multicolumn{4}{l}{note: stats below the line are measured, perhaps they go elsewhere?}    \\
    581872\end{tabular}
    582873\end{center}
    … …  
    586877\begin{center}
    587878\caption{Pending Metadata Tables\label{tab:PendingTables}}
    588 \begin{tabular}{lll}
     879\begin{tabular}{llll}
    589880\hline
    590881\hline
    … …  
    603894\begin{center}
    604895\caption{Copied Metadata Tables\label{tab:CopiedTables}}
    605 \begin{tabular}{lll}
     896\begin{tabular}{llll}
    606897\hline
    607898\hline
    … …  
    621912\begin{center}
    622913\caption{New Exposures\label{tab:NewExp}}
    623 \begin{tabular}{lll}
     914\begin{tabular}{llll}
    624915\hline
    625916\hline
    … …  
    639930\begin{center}
    640931\caption{Raw SCIENCE Exposure\label{tab:RawScienceExp}}
    641 \begin{tabular}{lll}
     932\begin{tabular}{llll}
    642933\hline
    643934\hline
    … …  
    679970\begin{center}
    680971\caption{Raw Detrend Exposures\label{tab:RawDetrendExp}}
    681 \begin{tabular}{lll}
     972\begin{tabular}{llll}
    682973\hline
    683974\hline
    … …  
    7181009\end{table}
    7191010
     1011\clearpage
     1012
    7201013\begin{table}[bh]
    7211014\begin{center}
    7221015\caption{P1 Exposures\label{tab:P1-Exp}}
    723 \begin{tabular}{lll}
     1016\begin{tabular}{llll}
    7241017\hline
    7251018\hline
    … …  
    7401033Moff              & float           & measure ZP offset        & 0.5 (mag)                 \\
    7411034dMoff             & float           & scatter in Moff          & 0.1 (mag)                 \\
     1035\end{tabular}
     1036\end{center}
     1037\end{table}
     1038
     1039\begin{table}[bh]
     1040\begin{center}
     1041\caption{P2 Exposures\label{tab:P2-Exp}}
     1042\begin{tabular}{llll}
     1043\hline
     1044\hline
     1045{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1046\hline
     1047exp ID            & string          & exposure ID              & 654321o                   \\
     1048class             & string          & file data class          & Cell / Chip / FPA         \\
     1049Nfiles            & int             & number of P2 image files & 3                         \\
     1050Ndone             & int             & number completed         & 2                         \\
     1051P1 version        & int             & P1 version number        & 01                        \\
     1052P2 version        & int             & P2 version number        & 01                        \\
     1053state             & string          & P2 analysis state        & new / done / fail?        \\
     1054\hline
     1055\end{tabular}
     1056\end{center}
     1057\end{table}
    7421058
    7431059\begin{table}[bh]
    7441060\begin{center}
    7451061\caption{P2 Images\label{tab:P2-Images}}
    746 \begin{tabular}{lll}
     1062\begin{tabular}{llll}
    7471063\hline
    7481064\hline
    … …  
    7521068class             & string          & file data class          & Cell / Chip / FPA         \\
    7531069class ID          & string          & identify for class       & chip00 / cell0102         \\
    754 version           & int             & P1 version number        & 01                        \\
    7551070recipe            & string          & analysis recipe name     & basic / flattest          \\
    756 state             & string          & P1 analysis state        & new / done                \\
     1071state             & string          & P2 analysis state        & new / done / fail?        \\
    7571072input.url         & string          & file location            & rootname.fits             \\
    7581073output.image.url  & string          & file location            & rootname.P2.01.fits       \\
    … …  
    7621077bias              & float           & measured bias value      & 5.0                       \\
    7631078dbias             & float           & bias residual scatter    & 5.0                       \\
     1079fringe            & float           & fringe amplitude         & 10.0                      \\
     1080dfringe           & float           & fringe scatter           & 10.0                      \\
     1081sky               & float           & reduced background       & 5.0                       \\
     1082dsky              & float           & background scatter       & 5.0                       \\
    7641083Nstars            & int             & number of astrom stars   & 50                        \\
    7651084sigma.X           & float           & astrom scatter in X      & 1.0 (arcsec)              \\
    … …  
    7681087Moff              & float           & measure ZP offset        & 0.5 (mag)                 \\
    7691088dMoff             & float           & scatter in Moff          & 0.1 (mag)                 \\
    770 \hline
    771 bias-image
    772 dark-image
    773 flat-image
    774 \hline
    775 \end{tabular}
    776 \end{center}
    777 \end{table}
    778 
    779 
    780 
     1089runtime           & float           & processing time          & 20.2 (sec)                \\
     1090\hline
     1091\multicolumn{4}{l}{note: the stats below the line are examples to be extended}     \\
     1092\end{tabular}
     1093\end{center}
     1094\end{table}
     1095
     1096The P2 analysis produces a reduced (P2) output image, a table of (P2)
     1097measured objects, and astrometric calibration terms.  The output
     1098objects are stored as a FITS table.  The astrometric solution for the
     1099image is stored in the header of the output object file.  Interesting
     1100statistics from the analysis are written to the MDDB table.
     1101Additional information describing the analysis is written to the log
     1102file in a machine-readable format.  Additional entries can be added to
     1103the MDDB table if we find they are useful for understanding the
     1104processing results.
     1105
     1106\begin{table}[bh]
     1107\begin{center}
     1108\caption{P3 Exposures\label{tab:P3-Exp}}
     1109\begin{tabular}{llll}
     1110\hline
     1111\hline
     1112{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1113\hline
     1114exp ID            & string          & exposure ID              & 654321o                   \\
     1115class             & string          & file data class          & Cell / Chip / FPA         \\
     1116Nfiles            & int             & number of P2 image files & 3                         \\
     1117Ndone             & int             & number completed         & 2                         \\
     1118P1 version        & int             & P1 version number        & 01                        \\
     1119P2 version        & int             & P2 version number        & 01                        \\
     1120P3 version        & int             & P3 version number        & 01                        \\
     1121state             & string          & P3 analysis state        & new / done / fail?        \\
     1122recipe            & string          & analysis recipe name     & basic / flattest          \\
     1123output            & string          & location of output file  & rootname.P3.01.smf        \\
     1124P3-log            & string          & location of P3 logfile   & rootname.P3.01.log        \\
     1125\hline
     1126Nstars            & int             & number of astrom stars   & 50                        \\
     1127sigma.X           & float           & astrom scatter in X      & 1.0 (arcsec)              \\
     1128sigma.Y           & float           & astrom scatter in Y      & 1.2 (arcsec)              \\
     1129Mcal              & float           & nominal zeropoint        & 25.2 (mag)                \\
     1130Moff              & float           & measure ZP offset        & 0.5 (mag)                 \\
     1131dMoff             & float           & scatter in Moff          & 0.1 (mag)                 \\
     1132runtime           & float           & processing time          & 20.2 (sec)                \\
     1133\hline
     1134\end{tabular}
     1135\end{center}
     1136\end{table}
     1137
     1138\clearpage
     1139
     1140\begin{table}[bh]
     1141\begin{center}
     1142\caption{Master Dark Frames\label{tab:DarkFrames}}
     1143\begin{tabular}{llll}
     1144\hline
     1145\hline
     1146{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1147\hline
     1148det ID            & string          & detrend frame ID         & 654321o                   \\
     1149version           & int             & version of det frame     & 02                        \\
     1150label             & string          & descriptive name         & basic                     \\
     1151recipe            & string          & creation recipe          & basic                     \\
     1152type              & string          & detrend type             & bias / dark / flat        \\
     1153class             & string          & file data class          & Cell / Chip / FPA         \\
     1154Nfiles            & int             & number of files          & 2                         \\
     1155\hline
     1156exptime           & float           & exposure time            & 5.0                       \\
     1157Ninput            & int             & number of input frames   & 5                         \\
     1158sigma             & float           & stack residual stats     & 2.0                       \\
     1159\hline
     1160\end{tabular}
     1161\end{center}
     1162\end{table}
     1163
     1164\begin{table}[bh]
     1165\begin{center}
     1166\caption{Master Flat Frames\label{tab:FlatFrames}}
     1167\begin{tabular}{llll}
     1168\hline
     1169\hline
     1170{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1171\hline
     1172det ID            & string          & detrend frame ID         & 654321o                   \\
     1173version           & int             & version of det frame     & 02                        \\
     1174label             & string          & descriptive name         & basic                     \\
     1175type              & string          & detrend type             & bias / dark / flat        \\
     1176class             & string          & file data class          & Cell / Chip / FPA         \\
     1177Nfiles            & int             & number of files          & 2                         \\
     1178\hline
     1179filter.ID         & string          & filter ID (glass)        & g.PS1.01 / none           \\
     1180filter.Band       & string          & filter bandpass name     & g / none                  \\
     1181Ninput            & int             & number of input frames   & 5                         \\
     1182sigma             & float           & stack residual stats     & 2.0                       \\
     1183Ntotal            & float           & ave.total input counts   & 1e6                       \\
     1184\hline
     1185\end{tabular}
     1186\end{center}
     1187\end{table}
     1188
     1189\begin{table}[bh]
     1190\begin{center}
     1191\caption{Master Fringe Frames\label{tab:FringeFrames}}
     1192\begin{tabular}{llll}
     1193\hline
     1194\hline
     1195{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1196\hline
     1197det ID            & string          & detrend frame ID         & 654321o                   \\
     1198version           & int             & version of det frame     & 02                        \\
     1199label             & string          & descriptive name         & basic                     \\
     1200type              & string          & detrend type             & bias / dark / flat        \\
     1201class             & string          & file data class          & Cell / Chip / FPA         \\
     1202Nfiles            & int             & number of files          & 2                         \\
     1203\hline
     1204filter.ID         & string          & filter ID (glass)        & g.PS1.01 / none           \\
     1205filter.Band       & string          & filter bandpass name     & g / none                  \\
     1206Ninput            & int             & number of input frames   & 5                         \\
     1207sigma             & float           & stack residual stats     & 2.0                       \\
     1208Ntotal            & float           & ave.total input counts   & 1e6                       \\
     1209fringe            & float           & fringe amplitude         & 52.0                      \\
     1210dfringe           & float           & fringe stdev             & 5.0                       \\
     1211airmass.min       & float           & minimum airmass          & 1.00                      \\
     1212airmass.max       & float           & maximum airmass          & 1.20                      \\
     1213\hline
     1214\end{tabular}
     1215\end{center}
     1216\end{table}
     1217
     1218\begin{table}[bh]
     1219\begin{center}
     1220\caption{Master Detrend Image Files\label{tab:DetImageFiles}}
     1221\begin{tabular}{llll}
     1222\hline
     1223\hline
     1224{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1225\hline
     1226URL               & string          & file location            & neb://file001.fits        \\
     1227det ID            & string          & detrend frame ID         & 654321o                   \\
     1228{\it camera}      & string          & camera name              & MegaPrime / GPC           \\
     1229class             & string          & file data class          & Cell / Chip / FPA         \\
     1230class ID          & string          & identify for class       & chip00 / cell0102         \\
     1231{\it type}        & string          & exposure type            & bias / flat / science     \\
     1232\hline
     1233\end{tabular}
     1234\end{center}
     1235\end{table}
     1236
     1237\clearpage
     1238
     1239\begin{table}[bh]
     1240\begin{center}
     1241\caption{Master Detrend Run\label{tab:DetRun}}
     1242\begin{tabular}{llll}
     1243\hline
     1244\hline
     1245{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1246\hline
     1247det ID            & string          & detrend frame ID         & 654321o                   \\
     1248camera            & string          & camera name              & MegaPrime / GPC           \\
     1249version           & int             & version of det run       & 02                        \\
     1250type              & string          & detrend type             & bias / dark / flat        \\
     1251criteria          & string          & image selection criteria & filter flags              \\
     1252state             & string          & analysis state           & new / done                \\
     1253\hline
     1254\end{tabular}
     1255\end{center}
     1256\end{table}
     1257
     1258\begin{table}[bh]
     1259\begin{center}
     1260\caption{Input Detrend Exp\label{tab:InputDetExp}}
     1261\begin{tabular}{llll}
     1262\hline
     1263\hline
     1264{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1265\hline
     1266det ID            & string          & detrend frame ID         & 654321o                   \\
     1267{\it camera}      & string          & camera name              & MegaPrime / GPC           \\
     1268{\it version}     & int             & version of det run       & 02                        \\
     1269exp ID            & string          & exposure ID              & 654321o                   \\
     1270state             & string          & analysis state           & new / raw / proc          \\
     1271\hline
     1272\end{tabular}
     1273\end{center}
     1274\end{table}
     1275
     1276\begin{table}[bh]
     1277\begin{center}
     1278\caption{Input Detrend Image Files\label{tab:InputDetImageFiles}}
     1279\begin{tabular}{llll}
     1280\hline
     1281\hline
     1282{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1283\hline
     1284raw.URL           & string          & location of raw file     & neb://file001.fits        \\
     1285proc.URL          & string          & location of proc file    & neb://file001.fits        \\
     1286det ID            & string          & detrend frame ID         & 654321o                   \\
     1287{\it version}     & int             & version of det run       & 02                        \\
     1288{\it camera}      & string          & camera name              & MegaPrime / GPC           \\
     1289{\it type}        & string          & exposure type            & bias / flat / science     \\
     1290class             & string          & file data class          & Cell / Chip / FPA         \\
     1291class ID          & string          & identify for class       & chip00 / cell0102         \\
     1292state             & string          & analysis state           & new / raw / proc          \\
     1293\hline
     1294\end{tabular}
     1295\end{center}
     1296\end{table}
     1297
     1298\begin{table}[bh]
     1299\begin{center}
     1300\caption{Input Detrend Resid Image Files\label{tab:InputDetResidImageFiles}}
     1301\begin{tabular}{llll}
     1302\hline
     1303\hline
     1304{\bf Field Name} & {\bf Datatype }  & {\bf Description}        & {\bf Examples} \\
     1305\hline
     1306image.URL         & string          & location of resid image  & neb://file001.fits        \\
     1307thumbnail.URL     & string          & location of resid jpeg   & neb://file001.fits        \\
     1308det ID            & string          & detrend frame ID         & 654321o                   \\
     1309version           & int             & version of det run       & 02                        \\
     1310iteration         & int             & det creation iteration   & 03                        \\
     1311{\it camera}      & string          & camera name              & MegaPrime / GPC           \\
     1312{\it type}        & string          & exposure type            & bias / flat / science     \\
     1313class             & string          & file data class          & Cell / Chip / FPA         \\
     1314class ID          & string          & identify for class       & chip00 / cell0102         \\
     1315state             & string          & analysis state           & new / raw / proc          \\
     1316scatter           & float           & residual scatter         & 1.0                       \\
     1317\hline
     1318\end{tabular}
     1319\end{center}
     1320\end{table}
    7811321
    7821322\end{document}
Note: See TracChangeset for help on using the changeset viewer.