Detrend system

• Queuing. Most detrends can be constructed simply by issuing a dettool command with the appropriate selections and det_type as in the following example. Note that if specific input exposures are to be used (instead of a random selection from matching database entries), the -definebytag mode must be used for dettool.

  dettool -definebyquery -pretend -select_dateobs_begin 2011-05-01T00:00:00 -select_dateobs_end 2011-08-01T00:00:00 -ccd_temp_min -86.755 -ccd_temp_max -72.1583 -random_subset -random_limit 150  -det_type DARKTEST -mode master -workdir neb://any/gpc1/detrend_easter2012/dark.20110501/ -inst GPC1 -select_exp_type DARK -select_exp_time_min 0.001 -select_exp_time_max 300 -comment '%exp%' -time_begin 2012-03-07T00:00:00 -time_end 2012-03-07T00:00:00
  

• Processing. The first stage of detrend creation involves the processing of the raw detrend exposures into the form necessary to be stacked (or combined in some way). ppImage.config contains the recipes used for this processing, with the exact recipe chosen from the reductionClasses.mdc file
• Stacking. The individual exposures are combined to yield an appropriate detrend model. The method of combination is defined in the ppMerge.config file.
• Normalization. Normalization of the stacked exposure. A number of these operations are simply no-ops, so the stacked and normalized versions are identical.
• Residuals. The normalized detrend model is applied to the input detrend frames, to identify any discrepancy between the model and the inputs. These residuals are shown on ippmonitor, although the exact scaling of the jpeg images is not necessarily optimal.
• Rejection. Using a set of rejection rules defined in rejections.config, some of the least consistent input frames may be rejected and a new iteration constructed without these inputs. This process is not well defined for some detrends (such as FLATs), where the rules for rejection are not well constrained.
• Registration. An alternate method of constructing a detrend is to simply register a valid detrend file as the appropriate type. This is the main way manually constructed detrends (MASKs, LINEARITY models) are input into the system. This can also be used on automatically generated detrends that will be used for different det_ids (see below for examples). A detrend can be registered by issuing the following example commands. Note that the det_id is not known until after the -register_detrend command is issued, but is required for all subsequent registration commands.

   dettool -dbname gpc1 -register_detrend -filelevel chip -inst GPC1 -telescope PS1 -det_type NOISEMAP -time_begin 2010-09-01T00:00:00 -time_end 2011-05-01T00:00:00
   dettool -dbname gpc1 -register_detrend_imfile -det_id 868 -class_id XY01 -uri neb://any/gpc1/detrend_easter2012/noisemap.20100901//GPC1.NOISEMAP.860/GPC1.NOISEMAP.norm.860.0.XY01.fits
   [...]
   dettool -updatedetrun -det_id 868 -state stop
  

Detrend filenames

All detrends generated using the dettool/pantasks system are created with the following filename structure:

 WORKDIR/CAMERA.DET_TYPE.DET_ID/CAMERA.DET_TYPE.STAGE.DET_ID.ITERATION.CLASS_ID.fits

These components are as follows:

• WORKDIR : user specified (at time detRun queued) workdir
• CAMERA : camera for the detrend to be created (specified when queued)
• DET_TYPE : type of detrend generated (see below)
• DET_ID : database id for this detRun
• STAGE : generally "norm", as the normalized detrend is used in most cases.
• ITERATION : which rejection iteration this detrend is from.
• CLASS_ID : class_id that this detrend matches.

As a detrend may be created with one det_type and det_id, but subsequently registered as another detRun, it is not generally correct to assume that the det_id and det_type listed in the filename directly match the det_id/det_type requested and used by the detrend processing. This information is better obtained via the detselect command.

Detrend types

• DARK. Detrend to remove the camera dark current.
• DARKTEST. This is an intermediate testing version of a DARK. It is constructed based on the same recipes as a regular DARK, but will not be accidentally selected for processing when finished (as processing only selects a valid DARK detrend). However, as detrends are often constructed with this det_type (and then converted to a DARK, or registered as a DARK), filenames often contain this string.
• VIDEODARK. This is a special (currently theoretical) DARK that is constructed from frames that have had video data taken on a cell. This changes the electronic response of the detector, resulting in a different dark current on the images.
• FLAT. Detrend to remove the pixel-to-pixel response of the detector.
• FLATTEST. Similar to a DARKTEST above, this detrend is a FLAT that was constructed with a parallel name to avoid being accidentally used in processing.
• MASK. Manually constructed mask to remove bad areas of the detector.
• VIDEOMASK. Similar to a VIDEODARK above, but instead of correcting regions with discrepant dark current, these regions are masked out.
• CTEMASK. Automatically constructed mask that contains information about regions with Charge Transfer Efficiency issues.
• FRINGE. Detrend to remove fringing patterns in the infrared.
• LINEARITY. Manually constructed detrend listing the measured linearity response at low flux levels.
• NOISEMAP. Detrend containing the read noise values as a function of detector position. The regular noisemaps can be generated through the dettool system. We currently use boosted noisemaps that require subsequent processing to add these boosts.

Using detselect

-search

-select

Creation process

Current detrends