2016 Nightly Science Update

How to run the script to generate the commands for missing diffs.

This requires a build of the branches/czw_branch/20160809/ code, and is based on r39816. At minimum, ippScripts and ippconfig are needed.

 nightly_science.pl --dbname gpc1 --camera GPC1 --verbose --debug --check_diffs --force_registration --date `date +%Y-%m-%d`

In addition to the regular options, the effects of the others are:

• --debug Ensures all commands have -pretends appended for safety.
• --check_diffs Does not execute the commands automatically, just prints them out.
• --force_registration Should not be needed in general, but in case morning darks were not taken, this tells the script to assume it is the end of the night.

This command is very verbose, but the missing diffs will be towards the end, in the section with lines prefaced by "desp_diff_queue". Examples of the "current" (not this update) and "proposed" (this update) outputs are attached to this page.

Details from my email on the change.

The goal of this change is to ensure that all diffs that are desired by MOPS are automatically generated, removing the need for additional manual work. This should speed MOPS processing.

I have attached the output of the current nightly_science.pl script in the "check_diffs" mode, along with the output from the proposed change to nightly_science.pl for yesterday (2016-11-14). The check_diffs mode runs the same logic as is used for queuing diffs, but does not issue queuing commands, and as such represents a way to demonstrate what the effect of this proposed change will be.

The output starts with a summary of the active configuration. A diff illustrates that the current IPP production configuration does not match the trunk exactly, with default retention times for warps being long, and with differences to the target definitions that will likely need to be resolved on the IPP side.

Following the configuration is a set of checks of the database to attempt to determine if observing has reached the end of night, by looking for the standard dark frame observations in the database. This is used to know if it is safe to generate stacks, and also to queue the new diffs automatically.

Next is a scan over the observing targets and filters. A note is generated for each possible diff that should be possible, in this case noting that the diffs are complete. The format for this includes the date, target, input exp_id, template exp_id, input warp_id, template warp_id, the observation object, and the input comment string. The proposed change additionally adds the template comment string, which was left out of the original version.

This scan appears to me to show that all diffs generated in the current code are generated by the proposed code as well. Yesterday was a good example to run, as object ps1_32_1505 had a bad camera stage quality for the third visit. This is handled in both versions of the code, but the proposed change is more explicit about the rejection of this exposure (exp_id = 1165338) and the exposure rejected to ensure that image/template pairs are fully populated (exp_id = 1165358).

Following the standard diff scan, based on the information that the end of night has been reached and no new exposures are coming, the original code continues to a multi-date diff operation that tries to find unpaired exposures that exist on disk and construct differences from those. My understanding is that these products are not desired, and as such this has been disabled in the proposed change.

Replacing this is an attempt to construct "desperate diffs" (named that way simply because I couldn't think of a better word). This runs the pairing operation again, but with the exposures sorted in a reverse order. Diffs that have already been constructed are identified as already queued, but this sorting order allows for the rejection rules to generate the next-best diffs for cases where an exposure has been lost. For the case of ps1_32_1505, visit three is again rejected for camera quality issues, but this time visit one is rejected, allowing for the visit 2 - visit 4 diff to be constructed (exp_ids 1165320, 1165358). A technical note for IPP, these diffs have the -rerun option applied, which will allow one exposure to be in two diffs with the same labels.

Finally, after all this, an IPP metadata block is written out that is used by the pantasks processing to set state information, such that all possible diffs have been constructed, and that no further checks need to be run.