= Nightly Science v2 = == Steps == 1. At UTC midnight, the nightly science tasks in the {{{stdscience}}} pantasks initialize books for a new day, and clean up old data according to the rules in nightly_science.config. 1. An exposure it observed at the summit. 1. The {{{summitcopy}}} pantasks identifies these exposures and populates the summitExp/summitImfile tables with the list of exposures available at the summit. It then downloads the fits files to MHPCC, and populates the newExp/newImfile tables. 1. The {{{registration}}} pantasks examines the downloaded fits images, and (using {{{register_imfile.pl}}}) registers an entry in the rawImfile table for each imfile (setting {{{data_state}}} to {{{pending_burntool}}} and {{{burntool_state}}} to {{{0}}}). It then examines the series of previous exposures, and determines if this newly registered imfile can be run through {{{burntool}}}. This requires all previous imfiles known at the summit with the same {{{class_id}}} as the one under examination to have been registered and burntooled. If {{{burntool}}} cannot be run at this time (most likely due to a previous imfile not being downloaded yet), {{{register_imfile.pl}}} sets the {{{data_state}}} to {{{pending_burntool}}} and exits. If {{{burntool}}} can be run, it runs {{{ipp_apply_burntool_single.pl}}} and continues as below. 1. The {{{registration}}} pantasks periodically runs {{{regtool -pendingburntoolimfile -date YYYY-MM-DD -valid_burntool 14}}} to examine the imfiles with {{{data_state = pending_burntool}}} and decide if they can now be burntooled. If so, the appropriate {{{ipp_apply_burntool_single.pl}}} command is constructed and run. 1. {{{ipp_apply_burntool_single.pl}}} takes the uris for the imfile under consideration and the previous uri, runs {{{burntool}}} and if successful, sets {{{data_state = full}}} and {{{burntool_state = -14}}}. 1. {{{regtool -pendingexp}}} refuses to complete the registration of the full exposure and populate rawExp until all component imfiles have {{{data_state = full}}}. This blocks the exposure level registration until all imfiles have been burntooled. {{{register_exp.pl}}} constructs the {{{regtool}}} command to populate rawExp, and decides if this exposure is a science exposure, based on the rules defined in the nightly_science.config file. If it is not a science exposure, then the {{{end_stage}}} is set to {{{reg}}}, and nightly science is finished with it. Science exposures have the appropriate values defined for the {{{label, data_group, dist_group, end_stage, tess_id, workdir, and reduction}}}, and after populating rawExp, an entry is written in chipRun queuing science processing on that exposure. 1. While exposures are taken, the nightly science tasks periodically run to examine whether stacks and diffs can be created. 1. For each stackable survey target defined in nightly_science.config, the number of chip and warp stage exposures is checked. For a stack to be queued it must satisfy the following criteria: 1. No stack already exists with that set of {{{data_group,skycell_id,filter,tess_id}}}. 1. The number of chip stage exposures equals the number of warp stage exposures with {{{state = 'full'}}} or with a {{{quality}}} value listed as acceptable in the configuration. 1. If we have not yet reached {{{end of night}}}, then there must be at least the minimum number of warp exposures listed in the configuration. This prevents premature queuing of stacks in the event of a delay in {{{summitcopy}}}. 1. For each diffable survey target defined in nightly_science.config, all completed warps are grouped by {{{object}}}. For each {{{object}}}-group, the comment strings are compared, and any duplicates are resolved by taking the exposure with the latest {{{dateobs}}} value. Exposures are then grouped into pairs by {{{dateobs}}} and a diff is queued for each pair provided that no diff already exists with that pair of exposures in either direction. 1. This proceeds until any criterion for {{{end of night}}} is reached, as described below. Once nightly science reaches the {{{end of night}}}, it starts checking whether stacks and diffs have finished. It does this by comparing the number of potential runs to be queued and the number of runs that have been queued. When these are equal, it sets the states to the appropriate {{{FINISHED_}}} value. == Configuration == The configuration is stored in {{{ippconfig/recipes/nightly_science.config}}} and has a standard IPP metadata format. === CLEAN_MODES === These entries list how the cleanup of old data is handled. Each {{{CLEAN_MODE}}} rule has three parts, a {{{MODE}}} which describes the name of the rule, {{{COMMAND}}} which lists the command to run to clean up data under this rule, and {{{RETENTION_TIME}}} which defines how long data is kept before this rule is invoked. The {{{COMMAND}}} entries allow for string replacement. The current values that can be replaced are: * {{{@DBNAME@}}} * {{{@LABEL@}}} * {{{@WORKDIR@}}} * {{{@OBSMODE@}}} * {{{@OBJECT@}}} * {{{@TESS_ID@}}} * {{{@DIST_GROUP@}}} * {{{@DATA_GROUP@}}} * {{{@REDUCTION@}}} * {{{@CURRENT_DATE@}}} The values of {{{RETENTION_TIME}}} can be overruled for a given survey target if the {{{MODE}}} name is listed in a {{{TARGET}}} entry described below. === END_OF_NIGHT === These entries describe a set of rules used to match exposures to determine if observing has ended. Even if these rules are not met, {{{end of night}}} will always be true if the local time (HST) is later than 6:30AM. The rules must have a {{{NAME}}}, and can contain any of {{{OBJECT,OBSMODE,EXPTYPE,COMMENT}}} to define the rule. === FILTERS === This is simple a list of the filters to examine. === UNRECOVERABLE_QUALITY === This lists {{{quality}}} values that are acceptable for a warp stage exposure to have with respect to stack queuing. If a warp has one of these values, then it is counted as full, even if not. === MACRO_DEFINITIONS === These define a macro to be executed by pantasks upon successful completion of a task. The {{{PROC_MODE}}} describes the stage of {{{nightly_science.pl}}} the macro defined in {{{MACRO}}} should be executed. This is not well tested and is not used at the moment. === TARGETS === These rules define the properties of exposures for each survey target. Each target must have a {{{NAME}}}. Exposures that comprise this target are defined by one or more of the the {{{OBSMODE,OBJECT,COMMENT}}} fields, which accept SQL-like % wildcards. The {{{DISTRIBUTION,TESS,REDUCTION}}} fields allow the {{{dist_group,tess_id,reduction}}} to be set for the science processing. The two booleans {{{STACKABLE,DIFFABLE}}} define whether this target should be examined for stacks and diffs. The {{{MIN_STACK}}} sets the minimum number of warps needed to make a stack while observing is still happening. Finally, any of the {{{CLEAN_MODES}}} listed above may be set here to change the retention time for this data. === DETRENDS === These rules define how the detrend verify runs are queued. Each has a {{{NAME}}} and use {{{DETTYPE,EXPTYPE}}} to determine the type of verify run to generate. The reference detrend is set using the {{{REF_ID,REF_ITER}}} values. If a {{{FILTER}}} value is set, then only exposures in that filter are used for the verification. The number of exposures to use in the verification process is set by {{{MAX_EXP}}}. == State Values == Nightly science v2 is less state dependent than v1, and the old state parameter {{{nsState}}} is no longer used to control processing. The main state to monitor is {{{nsObservingState}}}. This shows whether nightly science believes we are {{{OBSERVING}}} or at {{{END_OF_NIGHT}}}. While the state is {{{OBSERVING}}}, new exposures should be registering, burntooling, and automatically queuing to chip stage. Once this changes to {{{END_OF_NIGHT}}}, stacks and diffs start checking to see if they are finished. Both stacks and diffs start in the states {{{nsStackState,nsDiffState = TOWARP,TOWARP}}}. These signify that they are waiting for warps to finish. The standard nightly science task only runs the {{{--queue_X}}} versions of the nightly_science.pl code, as these suitably block multi- and premature queuing. The states will change to {{{STACKING,DIFFING}}} once they have attempted something. After {{{END_OF_NIGHT}}}, these will switch to {{{FINISHED_STACKING,FINISHED_DIFFING}}} when all suitable stacks and diffs have been constructed. The ns.show.dates prints out the following values: {{{ YYYY-MM-DD nsState nsStackState nsDiffState nsObservingState }}} == Manual Operation Modes == These commands show how to investigate the current status of the system, and how to manually fix certain issues. This assumes we are working with gpc1 data. All dates and times are UTC unless otherwise noted. === Look at the current status of observations/registration === This should probably be a mode to regtool with date and class_id options: {{{ SQL: SELECT DISTINCT exp_name, summitExp.dateobs AS registered, summitExp.imfiles, summitExp.fault AS summit_fault, pzDownloadExp.state AS download_state, newExp.state AS newExp_state, rawExp.state AS rawExp_state, rawImfile.data_state AS imfile_state, newExp.exp_id, summitExp.exp_type, rawExp.dateobs, rawExp.obs_mode AS obs_mode, rawExp.object AS object, rawImfile.burntool_state AS burntool_state, rawImfile.class_id, (pzDownloadExp.state = 'stop') AS is_downloaded,(rawImfile.burntool_state IS NOT NULL) AS is_registered FROM summitExp LEFT JOIN pzDownloadExp USING(exp_name) LEFT JOIN newExp ON exp_name = tmp_exp_name LEFT JOIN rawExp USING(exp_id, exp_name) LEFT JOIN rawImfile USING(exp_name) WHERE rawImfile.dateobs > '2010-12-18T00:00:00' AND class_id = 'XY45'; }}} === Look at the current list of pending imfiles === {{{ regtool -pendingimfile -dbname gpc1 }}} === Look at the current list of pending burntool imfiles === {{{ regtool -pendingburntoolimfile -date YYYY-MM-DD -valid_burntool 14 -dbname gpc1 }}} === Look at the current list of pending exposures === {{{ regtool -pendingexp -dbname gpc1 }}} === Revert an imfile that has failed registration === This has not happened before, but is increasingly likely as we start processing while still downloading and registering data. There is a revert task in the {{{registration}}} pantasks, but it is off by default. {{{ regtool -revertprocessedimfile -dbname gpc1 }}} === Manually queue a burntool run === This writes the burntool table and logfile into nebulous, and sets the database values as described above upon success. {{{ ipp_apply_burntool_single.pl --exp_id EXP_ID --class_id CLASS_ID --this_uri URI_FOR_EXP_ID --previous_uri URI_FOR_PREVIOUS_EXPOSURE --dbname gpc1 }}} === Look at the status of burntool for each science survey === {{{ nightly_science.pl --burntool_status --date YYYY-MM-DD }}} === Check to see if stacks can be made === {{{ nightly_science.pl --check_stacks --date YYYY-MM-DD }}} {{{ nightly_science.pl --check_stacks --date YYYY-MM-DD --this_target_only MD99 --this_filter_only g.00000 }}} {{{ nightly_science.pl --check_stacks --date YYYY-MM-DD --this_target_only MD99 --force_stack_count }}} === Queue a set of stacks === {{{ nightly_science.pl --queue_stacks --date YYYY-MM-DD }}} === Check to see if diffs can be made === {{{ nightly_science.pl --check_diffs --date YYYY-MM-DD }}} {{{ nightly_science.pl --check_diffs --date YYYY-MM-DD --this_target_only MD99 --this_filter_only g.00000 }}} === Queue a set of diffs === {{{ nightly_science.pl --queue_diffs --date YYYY-MM-DD }}} {{{ nightly_science.pl --queue_diffs --date YYYY-MM-DD --this_target_only MD99 --this_filter_only g.00000 }}} === Clean old data === {{{ nightly_science.pl --clean_old --date YYYY-MM-DD }}} === Look at what data would be cleaned === {{{ nightly_science.pl --clean_old --debug --date YYYY-MM-DD }}}