Detrend system

  • Queuing. Most detrends can be constructed simply by issuing a dettool command with the appropriate selections and det_type as in the following example. Note that if specific input exposures are to be used (instead of a random selection from matching database entries), the -definebytag mode must be used for dettool.
    dettool -definebyquery -pretend -select_dateobs_begin 2011-05-01T00:00:00 -select_dateobs_end 2011-08-01T00:00:00 -ccd_temp_min -86.755 -ccd_temp_max -72.1583 -random_subset -random_limit 150  -det_type DARKTEST -mode master -workdir neb://any/gpc1/detrend_easter2012/dark.20110501/ -inst GPC1 -select_exp_type DARK -select_exp_time_min 0.001 -select_exp_time_max 300 -comment '%exp%' -time_begin 2012-03-07T00:00:00 -time_end 2012-03-07T00:00:00
    
  • Processing. The first stage of detrend creation involves the processing of the raw detrend exposures into the form necessary to be stacked (or combined in some way). ppImage.config contains the recipes used for this processing, with the exact recipe chosen from the reductionClasses.mdc file
  • Stacking. The individual exposures are combined to yield an appropriate detrend model. The method of combination is defined in the ppMerge.config file.
  • Normalization. Normalization of the stacked exposure. A number of these operations are simply no-ops, so the stacked and normalized versions are identical.
  • Residuals. The normalized detrend model is applied to the input detrend frames, to identify any discrepancy between the model and the inputs. These residuals are shown on ippmonitor, although the exact scaling of the jpeg images is not necessarily optimal.
  • Rejection. Using a set of rejection rules defined in rejections.config, some of the least consistent input frames may be rejected and a new iteration constructed without these inputs. This process is not well defined for some detrends (such as FLATs), where the rules for rejection are not well constrained.
  • Registration. An alternate method of constructing a detrend is to simply register a valid detrend file as the appropriate type. This is the main way manually constructed detrends (MASKs, LINEARITY models) are input into the system. This can also be used on automatically generated detrends that will be used for different det_ids (see below for examples). A detrend can be registered by issuing the following example commands. Note that the det_id is not known until after the -register_detrend command is issued, but is required for all subsequent registration commands.
     dettool -dbname gpc1 -register_detrend -filelevel chip -inst GPC1 -telescope PS1 -det_type NOISEMAP -time_begin 2010-09-01T00:00:00 -time_end 2011-05-01T00:00:00
     dettool -dbname gpc1 -register_detrend_imfile -det_id 868 -class_id XY01 -uri neb://any/gpc1/detrend_easter2012/noisemap.20100901//GPC1.NOISEMAP.860/GPC1.NOISEMAP.norm.860.0.XY01.fits
     [...]
     dettool -updatedetrun -det_id 868 -state stop
    

Detrend filenames

All detrends generated using the dettool/pantasks system are created with the following filename structure:

 WORKDIR/CAMERA.DET_TYPE.DET_ID/CAMERA.DET_TYPE.STAGE.DET_ID.ITERATION.CLASS_ID.fits

These components are as follows:

  • WORKDIR : user specified (at time detRun queued) workdir
  • CAMERA : camera for the detrend to be created (specified when queued)
  • DET_TYPE : type of detrend generated (see below)
  • DET_ID : database id for this detRun
  • STAGE : generally "norm", as the normalized detrend is used in most cases.
  • ITERATION : which rejection iteration this detrend is from.
  • CLASS_ID : class_id that this detrend matches.

As a detrend may be created with one det_type and det_id, but subsequently registered as another detRun, it is not generally correct to assume that the det_id and det_type listed in the filename directly match the det_id/det_type requested and used by the detrend processing. This information is better obtained via the detselect command.

Detrend types

  • DARK. Detrend to remove the camera dark current.
  • DARKTEST. This is an intermediate testing version of a DARK. It is constructed based on the same recipes as a regular DARK, but will not be accidentally selected for processing when finished (as processing only selects a valid DARK detrend). However, as detrends are often constructed with this det_type (and then converted to a DARK, or registered as a DARK), filenames often contain this string.
  • VIDEODARK. This is a special (currently theoretical) DARK that is constructed from frames that have had video data taken on a cell. This changes the electronic response of the detector, resulting in a different dark current on the images.
  • FLAT. Detrend to remove the pixel-to-pixel response of the detector.
  • FLATTEST. Similar to a DARKTEST above, this detrend is a FLAT that was constructed with a parallel name to avoid being accidentally used in processing.
  • MASK. Manually constructed mask to remove bad areas of the detector.
  • VIDEOMASK. Similar to a VIDEODARK above, but instead of correcting regions with discrepant dark current, these regions are masked out.
  • CTEMASK. Automatically constructed mask that contains information about regions with Charge Transfer Efficiency issues.
  • AUXMASK. Auxiliary mask used to strictly mask the detector.
  • FRINGE. Detrend to remove fringing patterns in the infrared.
  • LINEARITY. Manually constructed detrend listing the measured linearity response at low flux levels.
  • NOISEMAP. Detrend containing the read noise values as a function of detector position. The regular noisemaps can be generated through the dettool system. We currently use boosted noisemaps that require subsequent processing to add these boosts.

Using detselect

A science processing of an exposure uses ppImage, which determines the detrend types required for processing based on the recipe definition. Once the recipe options are parsed to determine which detrends need to be applied, detselect library calls are used to obtain the correct detrend models. GPC1 currently (2012-05-15) uses DARK, FLAT, MASK, FRINGE, LINEARITY, and NOISEMAP detrends for normal science processing. Manual calls to the detselect command can be used to determine which detrends would be used for a given exposure.

The first step of using detselect to determine the appropriate detrend is to use the -search option, which returns a det_id for the requested detrend type for a specific date:

> detselect -search -det_type DARK -time 2012-05-15T02:02:02

detExp MULTI

detExp  METADATA  
   det_id           S64       866            
   iteration        S32       0              
   filelevel        STR       chip            
END

This example shows that for the given time (usually the dateobs value for the exposure of interest), the best DARK detrend is det_id 866. Note that unless otherwise needed, det_type should not be a TEST type (DARKTEST, FLATTEST). These are not generally used for science processing.

-select

After obtaining the det_id of the detrend needed, the exact detrend information can be obtained with a detselect -select command:

> detselect -select -det_id 866 -class_id XY06

detNormalizedImfile MULTI

detNormalizedImfile  METADATA  
   state            STR       stop            
   det_id           S64       866            
   iteration        S32       0              
   class_id         STR       XY06            
   uri              STR       neb://any/gpc1/detrend_easter2012/dark.20111101//GPC1.DARKTEST.858/GPC1.DARKTEST.norm.858.0.XY06.fits 
   bg               F64       nan            
   bg_stdev         F64       nan            
   bg_mean_stdev    F64       nan            
   user_1           F64       nan            
   user_2           F64       nan            
   user_3           F64       nan            
   user_4           F64       nan            
   user_5           F64       nan            
   path_base        STR       NULL            
   data_state       STR       full            
   fault            S16       0              
END

Without the -class_id option, the valid detrends for the full exposure will be returned. Note here that the filename of the DARK model is a DARKTEST with a different det_id. This detrend was originally constructed as a TEST, and then registered into a new detRun once being validated.

-show

Similar to dettool -search, -show finds all valid and active detrends of a given type with the set of constraints that they are restricted to:

> dettool -show -det_type NOISEMAP -inst GPC1 -telescope PS1

detRun MULTI

detRun  METADATA  
   det_id           S64       965            
   iteration        S32       0              
   filelevel        STR       chip            
   det_type         STR       NOISEMAP        
   mode             STR       register        
   registered       TAI       2012-07-07T03:16:54.000000
   filter           STR       NULL            
   time_begin       TAI       2011-05-01T00:00:00.000000
   time_end         TAI       NULL           
END

detRun  METADATA  
   det_id           S64       964            
   iteration        S32       0              
   filelevel        STR       chip            
   det_type         STR       NOISEMAP        
   mode             STR       register        
   registered       TAI       2012-07-07T03:16:46.000000
   filter           STR       NULL            
   time_begin       TAI       2010-09-01T00:00:00.000000
   time_end         TAI       2011-05-01T00:00:00.000000
END

detRun  METADATA  
   det_id           S64       963            
   iteration        S32       0              
   filelevel        STR       chip            
   det_type         STR       NOISEMAP        
   mode             STR       register        
   registered       TAI       2012-07-07T03:16:37.000000
   filter           STR       NULL            
   time_begin       TAI       2008-01-01T00:00:00.000000
   time_end         TAI       2010-09-01T00:00:00.000000
END

Without the -class_id option, the valid detrends for the full exposure will be returned. Note here that the filename of the DARK model is a DARKTEST with a different det_id. This detrend was originally constructed as a TEST, and then registered into a new detRun once being validated.

New detrend types

Adding a new detrend involves editing the following files:

  1. Add new detrend type to psModules/src/detrend/pmDetrendDB.{c|h}
  2. If this is a different type of fits file, add file handling to psModules/src/camera/pmFPAfile{.c|.h|FitsIO.c|IO.c}
  3. Edit ippconfig/recipes/ppImage.config, reductionClasses.mdc, rejections.mdc to properly define the configuration and processing of the detrend images.
  4. Edit ippScripts/scripts/detrend_*.pl to know about the new detrend type.
  5. Edit ppImage/ppMerge to correctly process and apply this detrend type.

Current detrends

Valid as of 2014-02-20. More exact information can be found on ippmonitor or in the gpc1 database.

det_type det_id reference det_id registered time_begin time_end Note
ASTROM 1064 2014-06-03 23:52:38 2008-05-06 00:00:00
KH_CORRECT 1065 2014-06-07 06:34:22 2008-05-06 00:00:00
MASK 945 2009-01-01 00:00:00 NULL
946 2009-12-09 00:00:00 NULL
947 2010-01-01 00:00:00 NULL
948 2011-01-06 00:00:00 NULL
949 2011-03-09 00:00:00 2011-03-10 23:59:59
950 2011-08-02 00:00:00 NULL
1072 2016-02-11T01:02:48.000000 2015-12-17T00:00:00.000000 NULL Fixes OTA62 bad cells.
FLAT 300 2010-01-21 20:25:06 2009-12-09 00:00:00 NULL g
301 2010-01-21 21:58:52 2009-12-09 00:00:00 NULL r
302 2010-01-21 21:58:52 2009-12-09 00:00:00 NULL i
303 2010-01-21 21:58:53 2009-12-09 00:00:00 NULL z
304 2010-01-21 21:58:54 2009-12-09 00:00:00 NULL y
305 2010-01-21 21:58:55 2009-12-09 00:00:00 NULL w
DARK 223 2009-07-16 23:08:00 2009-01-01 00:00:00 2009-12-09 00:00:00
299 2010-02-03 09:31:31 2009-12-09 00:00:00 NULL
863 862 2012-04-18 01:59:46 2010-01-23 00:00:00 2011-05-01 00:00:00 A-mode
864 856 2012-04-18 02:00:06 2011-05-01 00:00:00 2011-08-01 00:00:00
865 857 2012-04-18 02:01:26 2011-08-01 00:00:00 2011-11-01 00:00:00
866 858 2012-04-18 02:01:45 2011-11-01 00:00:00 2019-04-01 00:00:00
869-935 852 2012-04-18 02:15:56 2010-01-25 00:00:00* 2011-04-25 23:59:59* B-mode
FRINGE 296 2010-01-31 21:32:01 2009-12-09 00:00:00 NULL
LINEARITY 421 2010-08-18 03:15:28 NULL NULL
NOISEMAP 963 2012-07-07 03:16:37 2008-01-01 00:00:00 2010-09-01 00:00:00 Boosted
964 2012-07-07 03:16:46 2010-09-01 00:00:00 2011-05-01 00:00:00 ""
965 2012-07-07 03:16:54 2011-05-01 00:00:00 NULL ""
VIDEOMASK 951 2012-05-26 00:00:39 2009-01-01 00:00:00 NULL 50px/40pxl radius corners
952 2012-05-26 00:00:47 2009-12-09 00:00:00 NULL NULL
953 2012-05-26 00:00:55 2010-01-01 00:00:00 NULL NULL
954 2012-05-26 00:01:04 2011-01-06 00:00:00 NULL NULL
955 2012-05-26 00:01:13 2011-03-09 00:00:00 2011-03-10 23:59:59 NULL
956 2012-05-26 00:01:23 2011-08-02 00:00:00 2021-05-01 00:00:00 NULL
VIDEODARK 976 223 2013-05-25 01:15:21 2009-01-01 00:00:00 2009-12-09 00:00:00
977 299 2013-05-25 01:15:22 2009-12-09 00:00:00 2010-01-23 00:00:00
978 863 2013-05-25 01:15:22 2010-01-23 00:00:00 2011-05-01 00:00:00 A-mode
979 864 2013-05-25 01:15:22 2011-05-01 00:00:00 2011-08-01 00:00:00
980 865 2013-05-25 01:15:22 2011-08-01 00:00:00 2011-11-01 00:00:00
981 866 2013-05-25 01:15:22 2011-11-01 00:00:00 2019-04-01 00:00:00
982-1048869-935 2013-05-25 01:15:33 2010-01-25 00:00:00* 2011-04-25 23:59:59* B-mode
1049 978 2013-07-18T02:09:36 2010-09-12 00:00:00 2011-05-01 00:00:00 A-mode with OTA47fix
AUXMASK_STS 1050-1063 Need to be verified

Past detrends

(may not be fully complete)

det_type det_id reference det_id registered time_begin time_end Note
MASK 193 2009-07-03 02:21:54 2009-01-01 00:00:00 NULL
285 2010-01-25 00:12:50 2009-12-09 00:00:00 NULL
615 2010-12-24 01:36:56 2010-01-01 00:00:00 NULL
745 2011-04-05 02:20:50 2011-01-06 00:00:00 NULL
746 2011-04-05 02:43:27 2011-03-09 00:00:00 2011-03-10 23:59:59
847 2011-08-24 00:31:54 2011-08-02 00:00:00 NULL
NOISEMAP 936 854 2012-04-21 03:23:37 2011-05-01 00:00:00 2099-09-09 23:59:59 Boosted
937 859 2012-04-28 01:28:57 2008-01-01 00:00:00 2010-09-01 00:00:00 ""
938 860 2012-04-28 01:29:22 2010-09-01 00:00:00 2011-05-01 00:00:00 ""
VIDEOMASK 939 847 2012-05-02 02:34:36 2011-08-02 00:00:00 2021-05-01 00:00:00 50pxl/40pxl radius corners
940 746 2012-05-03 02:43:23 2011-03-09 00:00:00 2011-03-10 23:59:59 ""
941 745 2012-05-03 02:43:32 2011-01-06 00:00:00 NULL ""
942 615 2012-05-03 02:43:41 2010-01-01 00:00:00 NULL ""
943 285 2012-05-03 02:43:49 2009-12-09 00:00:00 NULL ""
944 193 2012-05-03 02:43:56 2009-01-01 00:00:00 NULL ""