Nightly Science: pantasks utilities

nightly.stacks.onTurn on the entire automation code. Turn off the entire automation code. YYYY-MM-DDAdd a date to be processed. the dates currently in the book, along with their associated states.
ns.del.dateRemove a date from processing (dates that make it to stack are automatically removed). YYYY-MM-DD STATESet the state for a given date.

State values:

State: Criterion
NEW Registration has not completed yet (or registration for this date has not been checked).
REGISTERED Registration complete.
NEEDSBURNING No burntool tables found for some portion of the data.
QUEUEBURNING Burntool jobs are being sent to hosts for processing.
BURNING The first pass of sending out jobs has completed (signals ns.chips.load to check for completion).
QUEUECHIPS Burntool has finished, and chips can be queued.
TOWARP Chips have been queued, and ns.stacks.load is waiting for all stackable warps to finish.
QUEUESTACKS Stackable warps have finished, and stacks will be queued.
STACKING stacktool commands have run to queue all stackable data.
CONFIRM_STACKING stacking has been attempted at least once and may have succeeded or failed.
DROP No exposures were found for the date, or the date string was malformed.

The following states are internal, and should not appear in standard use:

State: Criterion
CHECKDQSTATS Check to see if a dqstats bundle can be created (unused in pantasks module).
QUEUEDQSTATS Queue a dqstats bundle (unused in pantasks module).
CHECKDETRENDS Check to see if a detrend verify run can be created (unused in pantasks module).
QUEUEDETRENDS Queue detrend verify runs (unused in pantasks module).
FORCETOWARP Internal state to ensure stacks are not prematurely queued.
STACKING_POSSIBLE_ERROR Internal state to note when new stacks queues have been attempted when such stacks already exist.
ERROR Burntool definition failed.
DETREND_QUEUED Successfully queued detrend verify runs (unused in pantasks module).
DETREND_DROP No detrend exposures were found for the data (unused in pantasks module). script

Useful modes for manually checking and queueing data: --date YYYY-MM-DD --check_registration

Confirm that registration is complete, and all needed exposures are available. If exposures are missing (due to an incomplete download or registration error), they are listed along with a note stating whether they are important to the automation or not. --date YYYY-MM-DD --check_chips --date YYYY-MM-DD --queue_chips

The first checks the chips to see if they're ready to be queued (confirms that burntool has finished for all exposures), and the second issues the appropriate chiptool commands. --date YYYY-MM-DD --check_stacks --date YYYY-MM-DD --queue_stacks --date YYYY-MM-DD --queue_stacks --force_stack_count

Similar to the chip mode options, check the number of warps against the input exposures to determine if all the warps for a stack are finished, and queue them if so. The --force_stack_count allows this counting to be skipped if an exposure is known to have a uncorrectable error (such as the camera stage "quality = 4007" issue that happens when guiding is lost).

Both chip and stack mode options support the --this_target_only and --this_filter_only which allow a specific target or filter to be queued without checking the rest. This is safe because previously queued runs will not be duplicated.

Other modes: --date YYYY-MM-DD --check_detrends --date YYYY-MM-DD --queue_detrends

Check and queue detrend verify runs based on the types of detrends defined in the config file. --date YYYY-MM-DD --check_dqstats --date YYYY-MM-DD --queue_dqstats

Check and queue dqstats runs to bundle image statistics for the IPP-OTIS interface. --date YYYY-MM-DD --clean_old --debug --date YYYY-MM-DD --clean_old

Check and queue old data cleanup based on the rules defined in the configuration file. config file

The configuration file (source:/trunk/ippconfig/recipes/nightly_science.config) defines the parameters for four different components:


This is a simple MULTI metadata object containing the strings of all filters defined.


These define control how data from different stages is cleaned. The parameters are:

MODE STR Mode name being cleaned. Used to index the methods, and to allow overrides (see below).
COMMAND STR Command used to queue the data for cleanup.
RETENTION_TIME S16 Number of days to retain data before cleanup. Values less than one day disable cleaning for that target.
ALTERNATE_CMD STR Used to mark the format of the options for the cleanup command.
A -dbname $dbname -updaterun -set_state goto_cleaned -full -set_label goto_cleaned -time_stamp_end $cleaning_date
B -dbname $dbname -updaterun -set_state goto_cleaned -state full -set_label goto_cleaned -label $label
NULL -dbname $dbname -updaterun -set_state goto_cleaned -state full -set_label goto_cleaned -label $label -data_group $data_group


Each target in ippconfig/recipes/nightly_science.config has the following parameters:

NAME STR Target name. Used as the base of the data_group variable.
TESS STR Tessellation ID.
OBSMODE STR obs_mode value from rawExp. This is a LIKE search, so % wildcards are supported.
OBJECT STR object from rawExp. This was not populated, so although it is supported, none of the current target uses it.
COMMENT STR comment from rawExp. Also a LIKE search.
STACKABLE BOOL This flag determines whether or not nightly stacks can be made of this target.
CLEAN_MODE S16 Using the clean mode names defined in the CLEAN_MODE section, an alternate retention time can be specified for this target/mode combination.


Nightly detrend verification runs are controlled by the following parameters:

NAME STR Detrend name.
DETTYPE STR Value of the det_type column in the rawExp table to select on.
EXPTYPE STR Value of the exp_type column in the rawExp table to select on.
REF_ID S64 det_id of the reference detrend to verify against.
REF_ITER S32 iteration of the reference detrend to verify against.
FILTER STR Optional filter to select on.
MAX S32 Optional maximum number of detrend frames to use in this verify run.

Current retention times

Number of days each stage's image products are retained. All catalogs and configuration information is retained permanently.

stage MD M31 ThreePi STS SweetSpot
chip 30 30 14 30 30
camera - - - - -
warp 7 7 7 7 60
stack - - - - -
diff 30 30 14 30 30
destreak 1 1 1 1 1
distribution 30 30 10 10 10