Nightly Science v2

Steps

1. At UTC midnight, the nightly science tasks in the stdscience pantasks initialize books for a new day, and clean up old data according to the rules in nightly_science.config.
2. An exposure is observed at the summit.
3. The summitcopy pantasks populates the summitExp/summitImfile tables with the list of exposures available at the summit. It then downloads the fits files to MHPCC, and populates the newExp/newImfile tables.
4. The registration pantasks examines the downloaded fits images, and (using register_imfile.pl) registers an entry in the rawImfile table for each imfile (setting data_state to pending_burntool and burntool_state to 0). It then examines the series of previous exposures, and determines if this newly registered imfile can be run through burntool. This requires all previous imfiles known at the summit with the same class_id as the one under examination to have been registered and burntooled. If burntool cannot be run at this time (most likely due to a previous imfile not being downloaded yet), register_imfile.pl sets the data_state to pending_burntool and exits. If burntool can be run, it runs ipp_apply_burntool_single.pl and continues as below.
5. The registration pantasks periodically runs regtool -pendingburntoolimfile -date YYYY-MM-DD -valid_burntool 14 to examine the imfiles with data_state = pending_burntool and decide if they can now be burntooled. If so, the appropriate ipp_apply_burntool_single.pl command is constructed and run.
6. ipp_apply_burntool_single.pl takes the uris for the imfile under consideration and the previous uri, runs burntool and if successful, sets data_state = full and burntool_state = -14.
7. regtool -pendingexp refuses to complete the registration of the full exposure and populate rawExp until all component imfiles have data_state = full. This blocks the exposure level registration until all imfiles have been burntooled. register_exp.pl constructs the regtool command to populate rawExp, and decides if this exposure is a science exposure, based on the rules defined in the nightly_science.config file. If it is not a science exposure, then the end_stage is set to reg, and nightly science is finished with it. Science exposures have the appropriate values defined for the label, data_group, dist_group, end_stage, tess_id, workdir, and reduction, and after populating rawExp, an entry is written in chipRun queuing science processing on that exposure.
8. While exposures are taken, the nightly science tasks periodically run to examine whether stacks and diffs can be created.
9. For each stackable survey target defined in nightly_science.config, the number of chip and warp stage exposures is checked. For a stack to be queued it must satisfy the following criteria:
   1. No stack already exists with that set of data_group,skycell_id,filter,tess_id.
   2. The number of chip stage exposures equals the number of warp stage exposures with state = 'full' or with a quality value listed as acceptable in the configuration.
   3. If we have not yet reached end of night, then there must be at least the minimum number of warp exposures listed in the configuration. This prevents premature queuing of stacks in the event of a delay in summitcopy.
10. For each diffable survey target defined in nightly_science.config, all completed warps are grouped by object. For each object-group, the comment strings are compared, and any duplicates are resolved by taking the exposure with the latest dateobs value. Exposures are then grouped into pairs by dateobs and a diff is queued for each pair provided that no diff already exists with that pair of exposures in either direction.
11. This proceeds until any criterion for end of night is reached, as described below. Once nightly science reaches the end of night, it starts checking whether stacks and diffs have finished. It does this by comparing the number of potential runs to be queued and the number of runs that have been queued. When these are equal, it sets the states to the appropriate FINISHED_ value.

Configuration

The configuration is stored in ippconfig/recipes/nightly_science.config and has a standard IPP metadata format.

CLEAN_MODES

These entries list how the cleanup of old data is handled. Each CLEAN_MODE rule has three parts, a MODE which describes the name of the rule, COMMAND which lists the command to run to clean up data under this rule, and RETENTION_TIME which defines how long data is kept before this rule is invoked. The COMMAND entries allow for string replacement. The current values that can be replaced are:

• @DBNAME@
• @LABEL@
• @WORKDIR@
• @OBSMODE@
• @OBJECT@
• @TESS_ID@
• @DIST_GROUP@
• @DATA_GROUP@
• @REDUCTION@
• @CURRENT_DATE@

The values of RETENTION_TIME can be overruled for a given survey target if the MODE name is listed in a TARGET entry described below.

END_OF_NIGHT

These entries describe a set of rules used to match exposures to determine if observing has ended. Even if these rules are not met, end of night will always be true if the local time (HST) is later than 6:30AM. The rules must have a NAME, and can contain any of OBJECT,OBSMODE,EXPTYPE,COMMENT to define the rule.

FILTERS

This is simple a list of the filters to examine.

UNRECOVERABLE_QUALITY

This lists quality values that are acceptable for a warp stage exposure to have with respect to stack queuing. If a warp has one of these values, then it is counted as full, even if not.

MACRO_DEFINITIONS

These define a macro to be executed by pantasks upon successful completion of a task. The PROC_MODE describes the stage of nightly_science.pl the macro defined in MACRO should be executed. This is not well tested and is not used at the moment.

TARGETS

These rules define the properties of exposures for each survey target. Each target must have a NAME. Exposures that comprise this target are defined by one or more of the the OBSMODE,OBJECT,COMMENT fields, which accept SQL-like % wildcards. The DISTRIBUTION,TESS,REDUCTION fields allow the dist_group,tess_id,reduction to be set for the science processing. The two booleans STACKABLE,DIFFABLE define whether this target should be examined for stacks and diffs. The MIN_STACK sets the minimum number of warps needed to make a stack while observing is still happening. Finally, any of the CLEAN_MODES listed above may be set here to change the retention time for this data.

DETRENDS

These rules define how the detrend verify runs are queued. Each has a NAME and use DETTYPE,EXPTYPE to determine the type of verify run to generate. The reference detrend is set using the REF_ID,REF_ITER values. If a FILTER value is set, then only exposures in that filter are used for the verification. The number of exposures to use in the verification process is set by MAX_EXP.

State Values

Nightly science v2 is less state dependent than v1, and the old state parameter nsState is no longer used to control processing. The main state to monitor is nsObservingState. This shows whether nightly science believes we are OBSERVING or at END_OF_NIGHT. While the state is OBSERVING, new exposures should be registering, burntooling, and automatically queuing to chip stage. Once this changes to END_OF_NIGHT, stacks and diffs start checking to see if they are finished.

Both stacks and diffs start in the states nsStackState,nsDiffState = TOWARP,TOWARP. These signify that they are waiting for warps to finish. The standard nightly science task only runs the --queue_X versions of the nightly_science.pl code, as these suitably block multi- and premature queuing. The states will change to STACKING,DIFFING once they have attempted something. After END_OF_NIGHT, these will switch to FINISHED_STACKING,FINISHED_DIFFING when all suitable stacks and diffs have been constructed.

The ns.show.dates prints out the following values: YYYY-MM-DD nsState nsStackState nsDiffState nsObservingState

Manual Operation Modes

These commands show how to investigate the current status of the system, and how to manually fix certain issues. This assumes we are working with gpc1 data. All dates and times are UTC unless otherwise noted.

Look at the current status of observations/registration

This should probably be a mode to regtool with date and class_id options:

SQL: 
SELECT DISTINCT exp_name, 
  summitExp.dateobs AS registered, summitExp.imfiles, summitExp.fault AS summit_fault,
  pzDownloadExp.state AS download_state, newExp.state AS newExp_state, rawExp.state AS rawExp_state,    
  rawImfile.data_state AS imfile_state,
  newExp.exp_id, summitExp.exp_type, 
  rawExp.dateobs, rawExp.obs_mode AS obs_mode, rawExp.object AS object,
  rawImfile.burntool_state AS burntool_state, rawImfile.class_id,
  (pzDownloadExp.state = 'stop') AS is_downloaded,(rawImfile.burntool_state IS NOT NULL) AS is_registered 
FROM summitExp LEFT JOIN pzDownloadExp USING(exp_name) LEFT JOIN newExp ON exp_name = tmp_exp_name         
  LEFT JOIN rawExp USING(exp_id, exp_name) LEFT JOIN rawImfile USING(exp_name)
WHERE rawImfile.dateobs > '2010-12-18T00:00:00' AND class_id = 'XY45';

Look at the current list of pending imfiles

regtool -pendingimfile -dbname gpc1

Look at the current list of pending burntool imfiles

regtool -pendingburntoolimfile -date YYYY-MM-DD -valid_burntool 14 -dbname gpc1

Look at the current list of pending exposures

regtool -pendingexp -dbname gpc1

Revert an imfile that has failed registration

This has not happened before, but is increasingly likely as we start processing while still downloading and registering data. There is a revert task in the registration pantasks, but it is off by default.

regtool -revertprocessedimfile -dbname gpc1

Manually queue a burntool run

This writes the burntool table and logfile into nebulous, and sets the database values as described above upon success.

ipp_apply_burntool_single.pl --exp_id EXP_ID --class_id CLASS_ID --this_uri URI_FOR_EXP_ID --previous_uri URI_FOR_PREVIOUS_EXPOSURE --dbname gpc1

Look at the status of burntool for each science survey

nightly_science.pl --burntool_status --date YYYY-MM-DD

Check to see if stacks can be made

nightly_science.pl --check_stacks --date YYYY-MM-DD nightly_science.pl --check_stacks --date YYYY-MM-DD --this_target_only MD99 --this_filter_only g.00000 nightly_science.pl --check_stacks --date YYYY-MM-DD --this_target_only MD99 --force_stack_count

Queue a set of stacks

nightly_science.pl --queue_stacks --date YYYY-MM-DD

Check to see if diffs can be made

nightly_science.pl --check_diffs --date YYYY-MM-DD nightly_science.pl --check_diffs --date YYYY-MM-DD --this_target_only MD99 --this_filter_only g.00000

Queue a set of diffs

nightly_science.pl --queue_diffs --date YYYY-MM-DD nightly_science.pl --queue_diffs --date YYYY-MM-DD --this_target_only MD99 --this_filter_only g.00000

Clean old data

nightly_science.pl --clean_old --date YYYY-MM-DD

Look at what data would be cleaned

nightly_science.pl --clean_old --debug --date YYYY-MM-DD