Changes between Initial Version and Version 1 of NightlyScienceV2

Timestamp:

Dec 21, 2010, 2:41:42 PM (16 years ago)

Author:

watersc1

Comment:

--

NightlyScienceV2

@@ +1 @@
+= Nightly Science v2 =
+
+== Steps ==
+
+ 1. At UTC midnight, the nightly science tasks in the {{{stdscience}}} pantasks initialize books for a new day, and clean up old data according to the rules in nightly_science.config.
+ 1. An exposure it observed at the summit.
+ 1. The {{{summitcopy}}} pantasks identifies these exposures and populates the summitExp/summitImfile tables with the list of exposures available at the summit. It then downloads the fits files to MHPCC, and populates the newExp/newImfile tables.
+ 1. The {{{registration}}} pantasks examines the downloaded fits images, and (using {{{register_imfile.pl}}}) registers an entry in the rawImfile table for each imfile (setting {{{data_state}}} to {{{pending_burntool}}} and {{{burntool_state}}} to {{{0}}}).  It then examines the series of previous exposures, and determines if this newly registered imfile can be run through {{{burntool}}}.  This requires all previous imfiles known at the summit with the same {{{class_id}}} as the one under examination to have been registered and burntooled.  If {{{burntool}}} cannot be run at this time (most likely due to a previous imfile not being downloaded yet), {{{register_imfile.pl}}} sets the {{{data_state}}} to {{{pending_burntool}}} and exits.  If {{{burntool}}} can be run, it runs {{{ipp_apply_burntool_single.pl}}} and continues as below.
+ 1. The {{{registration}}} pantasks periodically runs {{{regtool -pendingburntoolimfile -date YYYY-MM-DD -valid_burntool 14}}} to examine the imfiles with {{{data_state = pending_burntool}}} and decide if they can now be burntooled.  If so, the appropriate {{{ipp_apply_burntool_single.pl}}} command is constructed and run.
+ 1. {{{ipp_apply_burntool_single.pl}}} takes the uris for the imfile under consideration and the previous uri, runs {{{burntool}}} and if successful, sets {{{data_state = full}}} and {{{burntool_state = -14}}}.
+ 1. {{{regtool -pendingexp}}} refuses to complete the registration of the full exposure and populate rawExp until all component imfiles have {{{data_state = full}}}.  This blocks the exposure level registration until all imfiles have been burntooled. {{{register_exp.pl}}} constructs the {{{regtool}}} command to populate rawExp, and decides if this exposure is a science exposure, based on the rules defined in the nightly_science.config file.  If it is not a science exposure, then the {{{end_stage}}} is set to {{{reg}}}, and nightly science is finished with it.  Science exposures have the appropriate values defined for the {{{label, data_group, dist_group, end_stage, tess_id, workdir, and reduction}}}, and after populating rawExp, an entry is written in chipRun queuing science processing on that exposure.
+ 1. While exposures are taken, the nightly science tasks periodically run to examine whether stacks and diffs can be created.
+ 1. For each stackable survey target defined in nightly_science.config, the number of chip and warp stage exposures is checked.  For a stack to be queued it must satisfy the following criteria:
+   1. No stack already exists with that set of {{{data_group,skycell_id,filter,tess_id}}}.
+   1. The number of chip stage exposures equals the number of warp stage exposures with {{{state = 'full'}}} or with a {{{quality}}} value listed as acceptable in the configuration.
+   1. If we have not yet reached {{{end of night}}}, then there must be at least the minimum number of warp exposures listed in the configuration.  This prevents premature queuing of stacks in the event of a delay in {{{summitcopy}}}.
+ 1. For each diffable survey target defined in nightly_science.config, all completed warps are grouped by {{{object}}}.  For each {{{object}}}-group, the comment strings are compared, and any duplicates are resolved by taking the exposure with the latest {{{dateobs}}} value.  Exposures are then grouped into pairs by {{{dateobs}}} and a diff is queued for each pair provided that no diff already exists with that pair of exposures in either direction.
+ 1. This proceeds until any criterion for {{{end of night}}} is reached, as described below.  Once nightly science reaches the {{{end of night}}}, it starts checking whether stacks and diffs have finished.  It does this by comparing the number of potential runs to be queued and the number of runs that have been queued.  When these are equal, it sets the states to the appropriate {{{FINISHED_}}} value.
+
+== Configuration ==
+ 
+ The configuration is stored in {{{ippconfig/recipes/nightly_science.config}}} and has a standard IPP metadata format.
+
+=== CLEAN_MODES ===
+
+These entries list how the cleanup of old data is handled.  Each {{{CLEAN_MODE}}} rule has three parts, a {{{MODE}}} which describes the name of the rule, {{{COMMAND}}} which lists the command to run to clean up data under this rule, and {{{RETENTION_TIME}}} which defines how long data is kept before this rule is invoked.  The {{{COMMAND}}} entries allow for string replacement.  The current values that can be replaced are:
+ * {{{@DBNAME@}}}
+ * {{{@LABEL@}}}
+ * {{{@WORKDIR@}}}
+ * {{{@OBSMODE@}}}
+ * {{{@OBJECT@}}}
+ * {{{@TESS_ID@}}}
+ * {{{@DIST_GROUP@}}}
+ * {{{@DATA_GROUP@}}}
+ * {{{@REDUCTION@}}}
+ * {{{@CURRENT_DATE@}}}
+
+The values of {{{RETENTION_TIME}}} can be overruled for a given survey target if the {{{MODE}}} name is listed in a {{{TARGET}}} entry described below.
+
+=== END_OF_NIGHT ===
+
+These entries describe a set of rules used to match exposures to determine if observing has ended.  Even if these rules are not met, {{{end of night}}} will always be true if the local time (HST) is later than 6:30AM.  The rules must have a {{{NAME}}}, and can contain any of {{{OBJECT,OBSMODE,EXPTYPE,COMMENT}}} to define the rule.
+
+=== FILTERS ===
+
+This is simple a list of the filters to examine.
+
+=== UNRECOVERABLE_QUALITY ===
+
+
+
+
+== Faults ==
+
+== Manual Operation Modes ==
+