IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 4 and Version 5 of LAP_Science


Ignore:
Timestamp:
Apr 26, 2011, 3:54:06 PM (15 years ago)
Author:
watersc1
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • LAP_Science

    v4 v5  
    4343== Run definition ==
    4444
    45 A new lapRun is defined by calling `laptool -definerun -seq_id S -projection_cell PC_NAME -ra RA_CENTER -decl DEC_CENTER -radius R -filter F -label L`.  This adds a row to lapRun, and then creates a row in lapExp for all exposures that fall within the area on the sky defined by the -ra/-decl/-radius options.  If these exposures have been previously processed (and therefore have a row in lapExp associated with a different lapRun), the chip_id for the new lapExp is copied from that previous entry, provided that the private flag is not marked. 
     45A new `lapRun` is defined by calling `laptool -definerun -seq_id S -projection_cell PC_NAME -ra RA_CENTER -decl DEC_CENTER -radius R -filter F -label L`.  This adds a row to `lapRun`, and then creates a row in `lapExp` for all exposures that fall within the area on the sky defined by the -ra/-decl/-radius options.  If these exposures have been previously processed (and therefore have a row in `lapExp` associated with a different `lapRun`), the `chip_id` for the new lapExp is copied from that previous entry, provided that the `private` flag is not true. 
    4646
    4747== Chip-Warp processing ==
    4848
    49 Defining the run does not queue chipRuns for those exposures that have not been processed previously.  This is done by the `lap_science.pl --chip_mode --lap_id LAP` command.  This script calls `laptool -pendingexp -lap_id LAP` which returns a list of all the exposures included in this lapRun.  Each exposure is checked, and a chipRun queued if the chip_id entry is blank.
     49Defining the run does not queue chipRuns for those exposures that have not been processed previously.  This is done by the `lap_science.pl --chip_mode --lap_id LAP` command.  This script calls `laptool -pendingexp -lap_id LAP` which returns a list of all the exposures included in this `lapRun`.  Each exposure is checked, and a `chipRun` queued if the `chip_id` entry is blank.
    5050
    51 This stage also defines the pairs for differencing, using the object and comment fields to organize the pairs as is done in nightly science.  If a pair is found, the associated pair_id fields point at the chip_id of the companion exposure, and the pairwise flag is set to true.  If a pair is not found, pair_id is left as NULL, pairwise is set to false, and private is set to true.  This ensures that the difference against the quickstack will not be used by other lapRuns.
     51This stage also defines the pairs for differencing, using the object and comment fields to organize the pairs as is done in nightly science.  If a pair is found, the associated `pair_id` fields point at the `chip_id` of the companion exposure, and the `pairwise` flag is set to true.  If a pair is not found, `pair_id` is left as NULL, `pairwise` is set to false, and `private` is set to true.  This ensures that the difference against the quickstack will not be used by other `lapRun`s.
    5252
    53 Once all the chips have been queued, the state of the lapRun will be polled using the `lap_science.pl --status --lap_id LAP`.  This calls `laptool -exposures -lap_id LAP` which returns the status of each exposure at the chip, camera, fake, warp, diff, and magic stages.  In addition to gauging completion, exposure pairs are checked against each other to confirm that no faults have occurred.  If one member of a pair has an unrecoverable fault, that entry is marked with data_state = 'drop', and is no longer used.  The companion of that unrecoverable exposure is set to be private so it will be diffed against the quickstack. 
     53Once all the chips have been queued, the state of the `lapRun` will be polled using the `lap_science.pl --status --lap_id LAP`.  This calls `laptool -exposures -lap_id LAP` which returns the status of each exposure at the chip, camera, fake, warp, diff, and magic stages.  In addition to gauging completion, exposure pairs are checked against each other to confirm that no faults have occurred.  If one member of a pair has an unrecoverable fault, that entry is marked with `data_state` = 'drop', and is no longer used.  The companion of that unrecoverable exposure is set to be private so it will be diffed against the quickstack. 
    5454
    55 Setting the lapExp to private "steals" this processing run away from any other lapRun that may want to use it. If lap_science.pl detects that another lapExp with this exp_id/chip_id set has private = true, then it will requeue a chipRun for that exp_id, and record the new chip_id for itself, and set private = true.
     55Setting the `lapExp` to private "steals" this processing run away from any other `lapRun` that may want to use it. If `lap_science.pl` detects that another `lapExp` with this `exp_id`/`chip_id` set has `private` = true, then it will requeue a `chipRun` for that `exp_id`, and record the new `chip_id` for itself, and set `private` = true.
    5656
    57 The script will also check the status of any stacks via the `laptool -stacks -lap_id LAP` which returns the status of all associated stacks for this lapRun.  This information is used to set a pair of flags for both the quick and final stacks: the defined flag notes that stacks have been queued, and the have flag notes that the stacks have completed and may be used.
     57The script will also check the status of any stacks via the `laptool -stacks -lap_id LAP` which returns the status of all associated stacks for this `lapRun`.  This information is used to set a pair of flags for both the quick and final stacks: the `defined_`flag notes that stacks have been queued, and the `have_` flag notes that the stacks have completed and may be used.
    5858
    59 Using the information about the exposures and stacks for this lapRun, `lap_science.pl` will decide what the next step of the processing will be.  After all exposures have a warpRun state of full (or have been dropped due to quality), lap_science.pl checks which step is most appropriate:
     59Using the information about the exposures and stacks for this `lapRun`, `lap_science.pl` will decide what the next step of the processing will be.  After all exposures have a `warpRun.state` of full (or have been dropped due to quality), `lap_science.pl` checks which step is most appropriate:
    6060
    6161=== Quick stacks ===
    6262
    63 If any lapExp has private = true, then a quickstack is needed.  If defined_quick is false, the quickstack is queued, and the appropriate stackAssociation.sass_id is recorded as lapRun.quick_sass_id. 
     63If any `lapExp` has `private` = true, then a quickstack is needed.  If `defined_quick` is false, the quickstack is queued, and the appropriate `stackAssociation.sass_id` is recorded as `lapRun.quick_sass_id`. 
    6464
    6565=== Diffs ===
    6666
    67 If no lapExp has private = true, and they all have valid pair_id values, then they will all be pairwise diffed.  If any lapExp has private = true, and have_quick is true (marking that the quickstack has finished), then the quickstack diff for that exposure will be queued as well.
     67If no `lapExp` has `private` = true, and they all have valid `pair_id` values, then they will all be pairwise diffed.  If any `lapExp` has `private` = true, and `have_quick` is true (marking that the quickstack has finished), then the quickstack diff for that exposure will be queued as well.
    6868
    6969=== Final stacks ===
    7070
    71 Once the diffs have completed, the regular stdscience/distribution tools will be used to process all data through the magic and destreak steps.  Once these have completed, the warpRun.magicked value is used to determine if a lapExp is ready for the final stacking stage.  If all lapExps have a non-zero magicked value, or have a data_state = 'drop', and defined_final is false, then the final stack is queued, and the appropriate stackAssociation.sass_id is recorded as lapRun.final_sass_id.
     71Once the diffs have completed, the regular stdscience/distribution tools will be used to process all data through the magic and destreak steps.  Once these have completed, the `warpRun.magicked` value is used to determine if a `lapExp` is ready for the final stacking stage.  If all `lapExp`s have a non-zero `magicked` value, or have a data_state = 'drop', and `defined_final` is false, then the final stack is queued, and the appropriate `stackAssociation.sass_id` is recorded as `lapRun.final_sass_id`.
    7272
    7373=== Cleanup ===
    7474
    75 Once the final stacks have completed, the lapRun is also complete.  If have_final is true, then lapRun.state is set to 'full', and all lapExp.active is set to false.  Periodically, `lap_science.pl --cleanup` will poll all lapExp, and if all lapExp with a given chip_id have active = false, then the products of all stages generated from that chip_id will be marked for cleanup (STAGE.label = 'goto_cleaned', STAGE.state = 'goto_cleaned').
     75Once the final stacks have completed, the `lapRun` is also complete.  If have_final is true, then `lapRun.state` is set to 'full', and all `lapExp.active` is set to false.  Periodically, `lap_science.pl --cleanup` will poll all `lapExp`, and if all `lapExp` with a given `chip_id` have `active` = false, then the products of all stages generated from that `chip_id` will be marked for cleanup (`STAGE.label = 'goto_cleaned'`, `STAGE.state = 'goto_cleaned'`).
    7676
    7777== Proposed Changes ==
    … …  
    7979=== Update ===
    8080
    81 Even after cleanup, lapExp still records whether an exposure has been fully processed previously.  Therefore, instead of queuing a new chipRun for this exposure, the data products may be updated instead.  This will be faster than starting from scratch.  Implementing this is not significantly difficult, as it requires only relaxing the state restriction in `laptool -definerun` and having `lap_science.pl` know how to update a lapExp with STAGE.state = 'cleaned'.
     81Even after cleanup, `lapExp` still records whether an exposure has been fully processed previously.  Therefore, instead of queuing a new `chipRun` for this exposure, the data products may be updated instead.  This will be faster than starting from scratch.  Implementing this is not significantly difficult, as it requires only relaxing the state restriction in `laptool -definerun` and having `lap_science.pl` know how to update a `lapExp` with `STAGE.state = 'cleaned'`.
    8282
    8383=== Complete Private diffs ===
    8484
    85 If an unpaired lapExp spans multiple projection_cells, then if we make the requirement that the quickstacks associated with all of the projection_cells are complete before diffing, then we do not need to make that lapExp private.  The difference can then be done across the entire image, instead of only over the overlapping area.  This will reduce the amount of processing required, but will likely increase the mask fraction for that lapExp
     85If an unpaired `lapExp` spans multiple `projection_cell`s, then if we make the requirement that the quickstacks associated with all of the `projection_cell`s are complete before diffing, then we do not need to make that `lapExp` private.  The difference can then be done across the entire image, instead of only over the overlapping area.  This will reduce the amount of processing required, but will likely increase the mask fraction for that `lapExp`.
    8686