== Nightly Science: pantasks utilities ==

||{{{nightly.stacks.on}}}||Turn on the entire automation code.||
||{{{nightly.stacks.off}}}|| Turn off the entire automation code.||
||{{{ns.add.date YYYY-MM-DD}}}||Add a date to be processed.||
||{{{ns.show.dates}}}||List the dates currently in the book, along with their associated states.||
||{{{ns.del.date}}}||Remove a date from processing (dates that make it to stack are automatically removed).||
||{{{ns.set.date YYYY-MM-DD STATE}}}||Set the state for a given date.||

== State values: ==

||State:       || Criterion ||
||NEW          || Registration has not completed yet (or registration for this date has not been checked). ||
||REGISTERED   || Registration complete. ||
||NEEDSBURNING || No burntool tables found for some portion of the data. || 
||QUEUEBURNING || Burntool jobs are being sent to hosts for processing. ||
||BURNING      || The first pass of sending out jobs has completed (signals ns.chips.load to check for completion). ||
||QUEUECHIPS   || Burntool has finished, and chips can be queued. ||
||TOWARP       || Chips have been queued, and ns.stacks.load is waiting for all stackable warps to finish. ||
||QUEUESTACKS  || Stackable warps have finished, and stacks will be queued. ||
||STACKING     || stacktool commands have run to queue all stackable data.||
||CONFIRM_STACKING     || stacking has been attempted at least once and may have succeeded or failed.||
||DROP         || No exposures were found for the date, or the date string was malformed.||

The following states are internal, and should not appear in standard use:
||State:       || Criterion ||
||CHECKDQSTATS || Check to see if a dqstats bundle can be created (unused in pantasks module). ||
||QUEUEDQSTATS || Queue a dqstats bundle (unused in pantasks module). ||
||CHECKDETRENDS|| Check to see if a detrend verify run can be created (unused in pantasks module). ||
||QUEUEDETRENDS|| Queue detrend verify runs (unused in pantasks module). ||
||FORCETOWARP  || Internal state to ensure stacks are not prematurely queued. ||
||STACKING_POSSIBLE_ERROR|| Internal state to note when new stacks queues have been attempted when such stacks already exist.||
||ERROR|| Burntool definition failed. ||
||DETREND_QUEUED|| Successfully queued detrend verify runs (unused in pantasks module). ||
||DETREND_DROP|| No detrend exposures were found for the data (unused in pantasks module). ||


== automate_stacks.pl script ==

Useful modes for manually checking and queueing data:

{{{automate_stacks.pl --date YYYY-MM-DD --check_registration}}}

Confirm that registration is complete, and all needed exposures are available.  If exposures are missing (due to an incomplete download or registration error), they are listed along with a note stating whether they are important to the automation or not.

{{{automate_stacks.pl --date YYYY-MM-DD --check_chips}}}

{{{automate_stacks.pl --date YYYY-MM-DD --queue_chips}}}

The first checks the chips to see if they're ready to be queued (confirms that burntool has finished for all exposures), and the second issues the appropriate chiptool commands.  

{{{automate_stacks.pl --date YYYY-MM-DD --check_stacks}}}

{{{automate_stacks.pl --date YYYY-MM-DD --queue_stacks}}}

{{{automate_stacks.pl --date YYYY-MM-DD --queue_stacks --force_stack_count}}}

Similar to the chip mode options, check the number of warps against the input exposures to determine if all the warps for a stack are finished, and queue them if so.  The --force_stack_count allows this counting to be skipped if an exposure is known to have a uncorrectable error (such as the camera stage "quality = 4007" issue that happens when guiding is lost).

Both chip and stack mode options support the --this_target_only and --this_filter_only which allow a specific target or filter to be queued without checking the rest.  This is safe because previously queued runs will not be duplicated.

Other modes:

{{{automate_stacks.pl --date YYYY-MM-DD --check_detrends}}}

{{{automate_stacks.pl --date YYYY-MM-DD --queue_detrends}}}

Check and queue detrend verify runs based on the types of detrends defined in the config file.

{{{automate_stacks.pl --date YYYY-MM-DD --check_dqstats}}}

{{{automate_stacks.pl --date YYYY-MM-DD --queue_dqstats}}}

Check and queue dqstats runs to bundle image statistics for the IPP-OTIS interface.

{{{automate_stacks.pl --date YYYY-MM-DD --clean_old --debug}}}

{{{automate_stacks.pl --date YYYY-MM-DD --clean_old}}}

Check and queue old data cleanup based on the rules defined in the configuration file.

== automate_stacks.pl config file == 

The configuration file (source:/trunk/ippconfig/recipes/nightly_science.config) defines the parameters for four different components:

=== FILTERS ===

This is a simple MULTI metadata object containing the strings of all filters defined.

=== CLEAN_MODES ===

These define control how data from different stages is cleaned.  The parameters are:

|| MODE || STR || Mode name being cleaned. Used to index the methods, and to allow overrides (see below). ||
|| COMMAND || STR || Command used to queue the data for cleanup. ||
|| RETENTION_TIME || S16 || Number of days to retain data before cleanup. Values less than one day disable cleaning for that target. ||
|| ALTERNATE_CMD || STR || Used to mark the format of the options for the cleanup command.||
||               || A || -dbname $dbname -updaterun -set_state goto_cleaned -full -set_label goto_cleaned -time_stamp_end $cleaning_date ||
||               || B || -dbname $dbname -updaterun -set_state goto_cleaned -state full -set_label goto_cleaned -label $label ||
||               || NULL|| -dbname $dbname -updaterun -set_state goto_cleaned -state full -set_label goto_cleaned -label $label -data_group $data_group ||

=== TARGETS ===

Each target in ippconfig/recipes/nightly_science.config has the following parameters:

|| NAME  || STR || Target name. Used as the base of the data_group variable. ||
|| TESS  || STR || Tessellation ID. ||
|| OBSMODE || STR || obs_mode value from rawExp. This is a LIKE search, so % wildcards are supported. ||
|| OBJECT || STR || object from rawExp.  This was not populated, so although it is supported, none of the current target uses it.||
|| COMMENT || STR || comment from rawExp. Also a LIKE search. ||
|| STACKABLE || BOOL || This flag determines whether or not nightly stacks can be made of this target. ||
|| CLEAN_MODE || S16 || Using the clean mode names defined in the CLEAN_MODE section, an alternate retention time can be specified for this target/mode combination.||

=== DETRENDS ===

Nightly detrend verification runs are controlled by the following parameters:

|| NAME || STR || Detrend name. ||
|| DETTYPE || STR || Value of the det_type column in the rawExp table to select on. ||
|| EXPTYPE || STR || Value of the exp_type column in the rawExp table to select on. ||
|| REF_ID  || S64 || det_id of the reference detrend to verify against. ||
|| REF_ITER|| S32 || iteration of the reference detrend to verify against. ||
|| FILTER  || STR || Optional filter to select on.||
|| MAX     || S32 || Optional maximum number of detrend frames to use in this verify run. ||

== Current retention times ==
|| stage        || MD  || M31 || ThreePi || STS || SweetSpot ||
|| chip         || 30  || 30  || 14      || 30  || 30        ||
|| camera       || -   || -   || -       || -   || -         ||
|| warp         || 7   || 7   || 7       || 7   || 60        ||
|| stack        || -   || -   || -       || -   || -         ||
|| diff         || 30  || 30  || 14      || 30  || 30        ||
|| destreak     || 1   || 1   || 1       || 1   || 1         ||
|| distribution || 30  || 30  || 10      || 10  || 10        ||