Index: unk/doc/design/psched-tasks.txt =================================================================== --- /trunk/doc/design/psched-tasks.txt (revision 6001) +++ (revision ) @@ -1,446 +1,0 @@ -\documentclass[panstarrs,spec]{panstarrs} - -\title{PStask \& Metadata DB interactions} % put in your title -\subtitle{Job Flow in IPP} -\author{Eugene Magnier} -\audience{IPP} -%\shorttitle{PSPhot} -\group{Pan-STARRS IPP} -\project{Pan-STARRS IPP} -\organization{Institute for Astronomy} -\version{DR} -\docnumber{PSDC-xxx-xxx} - -\newcommand\ugriz{$u^\prime g^\prime r^\prime i^\prime z^\prime$} -\newcommand\grizy{$g r i z y$} - -\begin{document} -\maketitle - -This collection of diagrams shows the IPP tasks and the MDDB tables -needed to manage the flow of data through the system. - ---- - -The first diagram illustrates the recommended interaction between the -metadata database tables and the scheduler internal queues. Some -table in the metadata database defines a list of data items which are -to be processed by some analysis job. The scheduler uses a two-step -approach to define the analysis jobs based on this list. First, one -scheduler task queries the MDDB for a list of pending items, adds the -returned items to an internal scheduler queue. The process of adding -the elements to the queue is defined so that only unique items are -added: already existing items are skipped. The entries in the queue -consist of the data items of interest and an internal temporary state. -At first, this would be 'pending'. A second scheduler tasks pops -'pending' entries one-by-one from this internal queue, submits a job -based on the entry, and sets the temporary state in the internal queue -to 'running'. The internal state is needed to prevent the scheduler -from re-submitting a job for the same data item before the first job -is done or assessed. Since the job make take an arbitrary amount of -time, the scheduler requires a mechanism to remember which data items -it has already submitted. When the job eventually completes, the -metadata database table is updated noting the completion. This may be -done either by the job itself or by the scheduler. In addition, the -state of the entry in the queue can be set to either 'done' or the -entry can be simply removed from the queue. - -The purpose of this interaction is to maintain the temporary state -information within non-persistent elements of the scheduler rather -than using the metadata database tables to store this information. -This concept has two advantages. First, the scheduler internal queues -are in memory and relatively small, thus interfacing with them is -quite fast for the scheduler -- this should reduce the system latency. -Second, by keeping this information non-persistent, the system -responds correctly to stopping and restarting the scheduler. Any jobs -which have not been completed will not be marked in the database, and -will be restarted naturally by the scheduler. The alternative, of -writing a temporary state marker in the database would require a -restarted scheduler to initially clean all database tables of these -temporary state markers. - -The second diagram illustrates this process using the process of -copying the images from the summit as an example. The metadata -database table of interest in this case is the list of pending images, -with entries supplied by a job which queries the summit data systems. -The job which is actually performed is a remote copy of the image file -from the location specified by the summit data system to the -appropriate location within the IPP Image Server (Nebulous). (As an -alternative to the above, the 'pending images' table may be part of -the summit database system, and the 'get images' command may query the -summit directly. In this scenario, the 'copy image' command reports -to the summit data system that an individual image file has been -copied.) - -In the rest of this document, the use of psched internal queues to -manage the temporary data states is glossed over and assumed part of -the tasks defined in the process. - ---- - -Summit Copy - -This diagram illustrates the MDDB tables used to copy data (images and -metadata tables) from the summit. The left-hand portion of the -diagram shows the tables involved in copying images from the summit -system. The table of pending image files lists the URLs of the -individual image files available for transfer, along with their -associated exposure ID and the camera which generated the image. Two -other entries assist in interpreting the file: the class and the class -ID. The final entry in this table is the current copy state of the -file, can have the value of 'ready' or 'copied'. - -The class defines the data grouping represented by this image file and -may have values of: FPA, Chip, Cell. This value indicates that the -provided image file represents the specified portion of the camera -FPA. If the value is FPA, the file represents data from a complete -FPA, though the file may contain pixel data in multiple extensions or -other groupings to be identified later. If the value is chip, the -file contains only data for a single chip, presumably of multiple -chips available, and equivalently for Cell. Further discussion of the -FPA image hierarchy is given in the IPP documents (eg, Modules SDRS). -The class ID gives the identifier used to name the class level -corresponding to this file. This value is necessary to make decisions -on how to copy the data based on the chip / cell before the data is -available to IPP components. Here are likely values for the class and -class ID for some common cameras: - -camera class classID -GPC chip chip02 -skyprobe fpa sp01 -Megacam fpa MegacamSpliced -Suprime chip chip0 - -and so forth. The system described is sufficiently flexible to allow -us to transfer the GPC images by cell if we eventually decide that is -more efficient. - -The copy process copies the file from the given URL to the appropriate -IPP node and adds an entry to the table of new image files, consisting -of the same information as the pending image file table, though with a -new value for the URL. This URL may be an explicit filename, a -reference to an entry in the image server, or a web address, or -located on the image server (marked with file:, neb:, and http:, -respectively). (TBD: other possible file storage types? perhaps the -path could be abstracted without going to the level of the image -server? eg: ref:DIR0001/file0001.fits might be in a directory which -is defined in a table of directories.) After an image file is -successfully copied, the corresponding state in the 'pending chip' -table is updated from 'ready' to 'copied'. - -The right hand portion of this diagram illustrates the process of -copying a metadata table. The table of pending tables lists the URLs -for the tables which are ready, a unique table ID for each table, and -the table type. The copy function copies the listed table and uploads -the data to the IPP version of the same metadata database. Two -examples of metadata tables needed by the IPP for the basic image -processing system are illustrated: the table of new exposures and the -table of pending matches. The first lists the exposures which are -avilable from the summit system, and all represent entries which are -available from the Image server. the second represents the matches -between exposure IDs and chips - ---- - -Phase 0 - -This diagram illustrates phase 0, in which the image files are -categorised, examined for summary information and basic statistics, -and moved to the tables used to trigger further analysis. The process -first examines the 'new image files' table. It selects images from -this table which have not yet been examined (state is 'new'). The -file header is examined and relevant metadata is extracted (eg, RA, -DEC, times, and so forth to be defined later). The process may also -select a portion of the image pixel data to determine a rough bias and -background level. These statistics, whether derived from the header -or the pixel values, are placed along with image summary information -in the 'raw image files' table, and the state field of the 'new image -files' table is set to 'ready'. - -The process is also responsible for moving the exposures to the tables -used for triggering the analysis process. If the image class is FPA, -the image can be advanced without waiting for any other image files. -If the class is Chip or Cell, the process must also examine the 'new -exposure' table for this exposure ID. The number of class files -available for this exposure is listed in this table. The process must -the select all image files matching the exposure ID with state of -'ready' and compare the number avalable to the number expected. If -the two match, then a new exposure is ready. Based on the image type -(from the most recently examined image file header or new exp table?), -the exposure is added to the 'raw exposure' table for images of that -type. The allowed types are 'detrend', (all bias, dark, flat images), -'object', 'focus'(??), etc. (** The different tables represent -different analysis modes. This process also adds an entry to the exp -ID / image file match **). This process also adds all science -(OBJECT) exposures to the P1 exposure table (for mosaic data) or the -P2 chip table (for single detector data). These tables are used to -trigger the Phase 1 and Phase 2 analysis stages. - ---- - -Phase 1 - -This diagram shows the tables involved in running the P1 analysis -stage. There are paths for exposures to enter the analysis -automatically from the P0 analysis (arrow on left) or to be added -manually based on a selection from the raw exposure table. Exposures -to be analysed by Phase 1 are added to the P1 exposure table with the -state 'new'. Exposures may be added multiple times for processing and -reprocessing. The P1 exp table keeps a record of the old attempts for -debugging and analysis. Each time an exposure is added to the P1 exp -table, it is given a new, unique version number, allowing the system -as a whole to track different analysis attempts. This method is used -in all of the image analysis stages (and extrapolated to iterations in -the detrend analysis steps below). The top portion of the diagram -represents the user-space tool which may be used to re-submit an -exposure or a group of exposures, potentially selected on the basis of -a query from the raw SCIENCE exposure table. - -The P1 exposure table is examined to select the new exposures, these -are then used to generate the P1 analysis jobs. Within the analysis -job, the chips (image files) associated with the exposure are select -from the raw image file table. The analysis examines the contents of -these files, either extract the guide star information from the image -files (GS table extension) or searches for and centroids the pixels on -appropriate bright stars. The analysis results in astrometric -calibration terms which are written to the astrometric calibration -file for this exposure. The location of the astrometric calibration -file and the statistics of the measurement are written back to the P1 -exposure table. The images associated with exposures which are -successfully processed by P1 are then added to the P2 image table, -which is used to trigger the Phase 2 analysis. - ---- - -Phase 2 - -This diagram shows the tables involved in running the P2 analysis -stage. There are paths for images to enter the analysis automatically -from the P1 analysis (arrow on left) or to be added manually based on -a selection from the raw exposure and raw image file tables. Images -to be analysed by Phase 2 are added to the P2 image table with the -state 'new'. When images are added to this table, a single entry is -also added to the P2 exposure table listing the P1 and P2 versions for -this exposure. These version numbers must be integers starting with -1. If this image did not have a P1 analysis, the P1 version is set to -0. Exposures may be added multiple times for processing and -reprocessing. The P2 image table keeps a record of the old attempts -for debugging and analysis. As with P1, each time a collection of -associated images from an exposure is added to the P2 image table, it -is given a new, unique version number, allowing the system as a whole -to track different analysis attempts. The top portion of the diagram -represents the user-space tool which may be used to re-submit the -images for an exposure or a group of exposures, potentially selected -on the basis of a query from the raw SCIENCE exposure and raw image -file tables. - -The P2 image table is examined to select the 'new' images. These -images are used to generate P2 analysis jobs. The P2 analysis uses -the input url to find and load the image file. The url may be a file -on disk, an entry in the image server, Nebulous, etc. The master -detrend images matching the specific science image and the conditions -are selected by examining the table of master detrend frames. The -specific detrend image files are selected by using the master detrend -ID to select the matching the entries in the table of master detrend -files. After the analysis, the output image, mask, and FITS table of -objects, including the astrometry calibration, are written to the P2 -image table, along with summary statistics from the P2 analysis. The -state is also updated (to 'done'). - -Whenever the exposure is completed, the value of Ndone in the P2 -exposure table is incremented. If all P2 images matching the P2 -exposure version have been completed, the value of Ndone will match -Nclass, and in this case, the process adds an entry to the P3 exposure -table. - ---- - -Phase 3 - -This diagram illustrates the tables involved in the Phase 3 analysis. -The P3 exposure table lists the exposure ID, the P3 analysis version, -the P2 analysis version to be used as input to this P3 analysis, and -the recipe to be used. The P2 exposure and image tables are used, in -conjunction with the P2 version information, to select the P2 output -measured objects and the astrometric calibrations from P2 and P1. -These measured objects are matched with the reference catalog objects, -and calibrated astrometry and photometry is produced for the full -exposure. The location of the resulting astometry calibration table -is stored back in the P3 exposure table. If the recipe file -specifies, the 2-D photometric and background / fringe corrections may -also be performed at this stage. Since these analyses require -reference data, the recipe may also be used to skip these analysis if -such reference data is unavailable or unreliable. At the end of Phase -3, the objects from the exposure are inserted into the photometry -database (this is not shown). - -The astrometric calibration portion of Phase 3 is principally needed -for a mosaic camera. For single-chip cameras, Phase 3 may be used to -perform the photometric calibration and simply pass the astrometric -results along to the output file to be listed in the P3 exposure -table. In this way, later stages of the analysis (ie, Phase 4) can -use the P3 exposure table as input for all cameras, even if all the -funcionality of Phase 3 is not necessary for that camera. This would -be the case for the skyprobe camera, for example. - ---- - -Phase 4 - -At the end of Phase 3, the images are ready for Phase 4. The Phase 3 -diagram shows the output line adding the exposures to be processed by -Phase 4 to a Phase 4 table. However, this line is just for -illustration purposes. The rules for initiating a Phase 4 run are -somewhat more complicated than those for running Phases 1-3. Groups -of exposures which have an appropriate overlap should be chosen for -the Phase 4 analysis. In the steady-state period of PS-4, it may be -straightforward to choose the exposure groups: they would simply be -the exposures obtained nearly simultaneously by the four separate -cameras. The circumstance for PS-1 will be much more complicated (and -even PS-4 will probably be more complex than it seems at first -glance). For example, in PS-1, we will not have a static sky for most -of the AP Survery. In this circumstance, we cannot run P4, at least -until after the complete AP Reference catalog is built, and -potentially all exposures re-run through Phase 3. It may be useful -for the AP Survey data to split the Phase 4 analysis into two stages: -image combination and image differencing. It may even be the case -that only the combination portion of Phase 4 is performed on the AP -Survey data. - -More generally, the image groups selected for Phase 4 analysis may be -chosen on the basis of a query of the AP Database (DVO) with some -rules. This may be - --- - -Note that each of the stages P1-P4 refer to the processing version -from the previous stage. This allows the processing stage to request -the correct version of the results from the previous stage, and makes -it possible to run and re-run the analysis at any stage without -deleting the earlier results. As different analysis attempts are -performed for a given image, the versions branch out, like the diagram -in Figure NNN. - -Also note that at every stage, the entries include a recipe -identifier. This is used to select the analysis recipe which should -be used for this version. By default, the recipe should be set to the -current best recipe (use a default name for this?). This feature -allows the user to run test analyses with variations on the recipe -without altering the analysis system. For example, it is possible to -use a different flat-field set by specifying alternate rules for the -flat-field selection in a recipe file. If it is necessary to run the -P1-P3 analysis with the raw master flats, for example, the user simply -defines that selection in the recipe file and submits the images of -interest to P1 (or P2, etc), with the corresponding entry for the -recipe. - -The recipe file may also be used to specify alternative analysis paths -and desitinations. For example, it is not necessary that all analysis -stops with P4: the recipe file may be used to halt the analysis at P2 -or P3. In addition, the recipe file may be used to specify an -alternative destination for the output results. For example, to -generate the photometric flat-field correction frame from a collection -of dithered images, the user may not want the photometry results in -the main DVO database. By using the recipe to set an alternative DVO -database target, and by specifying the use of the raw master flat -rather than the corrected one, the analysis of the dithered images is -kept isolated from the other photometry database results. The -resulting photometry may be used to construct the new, corrected -flat-field images, and the processing of the same images using the new -flat-field images may be sent to the master DVO database. - ---- - -mkflat - -This diagram illustrates the tables needed for the generic detrend -construction process, using the flat-field construction as an example. -This diagram is somewhat more complex than the preceeding versions. -In this diagram, both single jobs and multiple jobs are represented by -the process elements (the blue ellipses). In some cases, more that -one task will be needed to perform the function illustrated by a -single process task. The complexity of this diagram is enhanced by -the need for multiple iterations and both single chip and full mosaic -processing. At the moment, the distinction between mosaic and single -chip cameras is not specifically discussed. Finally, the triggers -which initiate a specific detrend analysis are glossed over. - -The detrend analysis is initiated by choosing a type of detrend image -to be constructed and by specifying the criteria which will be used to -select the input raw detrend frames for the construction. For -example, these criteria could specify that all twilight flat images -over a certain period of days, perhaps with restrictions on the flux -levels or the time-from-sunset of the images. The detrend analysis -run is given an ID (det ID) which will also be used to identify the -resulting master detrend frame. - -Given the definition of a master detrend run, the input exposures are -selected from the raw detrend exposure table, and written to the input -detrend exposure table. In the next step, the corresponding image -files are selected from the table of raw image files. Since there -will be a different set of input raw images for each attempt at -creating a master detrend image, and since any given attempt may use -some of the same input images as any other attempt, a separate table -of input raw images is constructed. - -Each of the input raw images may be pre-processed before it may be -used to construct the detrend frame. For example, the input -flat-field images should (probably) be dark- and bias-corrected before -they are stacked. The information about these input processed images -is written to the input images table. If no processing is needed, -this step simply copies the appropriate information to the table, and -points back to the raw image, rather than a processed version. - -The input processed images are combined (stacked) to create a master -detrend image for the particular data element defined by the image -class (chip/cell/fpa). At this stage, not all input images should -necessarily be included in the stack. If residual statistics have -been measured for the input images (say, using a prior stack), then -some of the input image may be excluded. The table of residual images -is used to guide this process. The information describing the -resulting master image is written to the master images table. - -The statistics of the master detrend images must examined so that any -necessary renormalizations may be performed. For example, after -stacking the individual flat images, the resulting stacks must be -renomalized to account for the different ranges of input image fluxes. -This analysis is least-squares solution in which an appropriate scale -is determined for each input exposure and a separate gain is -determined for each of the chips or cells in the camera. This -analysis can only performed after all image stacks (ie, for all chips) -have been constructed. The resulting information is written to the -table of master detrend frames. - -Once the master detrend is constructed, the master detrend images may -be used to construct residual images for each of the input images. -These residual statistics, as well as the locations of the residual -images and other related data products (jpeg thumbnails?) are written -to the residual image table. Note the red arrow which by-passes the -stack construction and merge steps and skips directly to the residual -analysis. In some cases, it may be useful to have the input images -confronted with an existing detrend image, and the resulting residual -values used to guide the rest of the process. For example, in the -flat-field analysis, applying an earlier flat can result in a very -good first-pass rejection of poor input images. The logic to make -this leap must be part of the scheduler, since each of the individual -blocks represent complete processing jobs. - -Finally, the residual statistics from the complete mosaic (all input -images, all chips) are used to assess the quality of the newly -constructed master detrend image, and to potentially modify the -selection of input images. This latter process is performed by -marking the state of the residual images from this iteration. The -stacking process always examines the state information for the -residual images from the previous iteration, if it exists, when -constructing the master stack. Once a master detrend frame has been -judged of high enough quality, the state of the entry for the frame in -the master detrend frames table is set to an appropriate value to tell -the other routines that this image should be used as a master detrend. -The exact choice of which master detrend frame is used for a given -science image depends on the recipe along with information such as the -time period used or the conditions used. - -Note that, although this discussion focuses on the construction of -flat-field images, the same structure should be capable of -constructing the biases, dark, fringes, etc. In some cases, as noted -above, the 'process' stage is a null operation.