Changes between Initial Version and Version 1 of EssenceTest

Timestamp:

Feb 24, 2009, 4:23:55 PM (17 years ago)

Author:

trac

Comment:

--

EssenceTest

@@ +1 @@
+=== Summary ===
+The ESSENCE survey from the CTIO 4m telescope in Chile provides a good dataset for testing the IPP. This page details the steps required to run the ESSENCE data through the IPP. 
+
+----
+Previous version that was a bit more for IPP 2.0 is available at [wiki:EssenceTest_IPP_2.0 EssenceTest_IPP_2.0]
+----
+Latest updates are applicable to IPP 2.4 & 2.5 (S.Rodney, 2008-04-07)
+----
+
+=== Annotated ESSENCE test walkthrough ===
+
+==== Configuration: ====
+ * ~/.ipprc 
+  * in this file you should be pointing to the site.config and system.config files, installed in the master config directory.
+  * also be sure to correctly set your PATH, as well as your LOG and TRACE levels.
+
+ * site.config  
+  * in the following we assume you have set up the paths to your raw data and to your base working directory in the DATAPATH metadata block  with the entries for DATA,WORK. e.g. :
+
+ # place your data directories here and refer to as path://PATH/remainder
+ DATAPATH        METADATA
+ [...]
+        DATA   STR     /gpfs1/Panstarrs/smsn/data/v9.0/W/rawdata
+        WORK   STR     /gpfs1/Panstarrs/data/W/
+ END
+
+When making program calls on the command line, refer to the first directory as path://DATA/ and the second as path://WORK/.
+
+ *system.config  :  
+  * set up the entries for DBSERVER, DBNAME, DBUSER, DBPASSWORD, E.g.,
+
+ DBSERVER        STR     tscdb                   # Database host name (for psDBInit)
+ DBNAME          STR     XXX                 # Database name (for psDBInit)
+ DBUSER           STR      ps1ipp                  # Database user name (for psDBInit)
+ DBPASSWORD    STR     somepassword            # Database password (for psDBInit), no quotes!
+
+NOTE: In the following we will assume that the dbname is not set to the database of interest.  This should be the case if you have more than one database in use on your installation, e.g. if there are multiple users or if one user has several projects going at once.
+
+ * Check ~/.ptolemyrc (which should be copied from ippconfig/dvo.site if it doesn't already exist) for:
+  * CONFDIR, e.g.,
+
+ CONFDIR                 /gpfs1/Panstarrs/code/ipp-2.2.lin64/share/ippconfig
+
+ * NOTE:  As of IPP 2.5, the 'ippconfig' directory contains a 'mosaic2' subdirectory with some good guesses at the configuration options in the files within.  This was formally the 'ctio_mosaic2' directory, but the name has been changed to be consistent with the internal naming conventions.
+  * in this mosaic2 config directory, you should have a camera.config file
+  * In that camera.config file you can set up the parameters that ipp uses to recognize and interact with files originating on the mosaic2 camera. In camera.config you must point to a format.config file, which tells the IPP what your camera's fits file format looks like. These formats must match up with what ppImage.config expects.  This should already be set in the distributed 'ippconfig/mosaic2' files.  Your camera.config file will also point to a filerules-mef.mdc file which defines how the various output files are named and what format they are written in (i.e. how they are compressed, binned, etc)  
+
+==== Initiating a database. ====
+
+Create an empty database: the dbadmin tool is a shell script that creates an SQL database and populates it with the tables needed by ippMonitor for user administration. The dbadmin script is probably not in your path, so find it in the ippMonitor source code directory and link to it or copy it somewhere.  Call dbadmin with no options for syntax help.
+
+ > dbadmin project [host] [password] essence
+
+Then initiate the new database
+ > pxadmin -create -dbname essence
+
+See [wiki:Simtest#Initiating_a_database Simtest#Initiating_a_database] for alternate methods.
+
+==== Injecting data ====
+
+Now we will inject our image files into the database, producing one db entry per imfile.
+
+'''WORKDIRS''' :  It is worth considering how you want your directory structure to be organized at this point.  If you have a lot of data you may not want everything to be planted in the same workdir.  For example, with the ESSENCE data it is useful to create subdirectories for each year of the survey, and a further level of subdirectories for the SuperMacho and the w-project data. The inject command might then look like:
+
+ > ipp_serial_inject.pl --telescope BLANCO --camera MOSAIC2 --workdir path://WORK/2003/SM/ --dbname essence /data/ipp003.0/SM_W/SM_SN03/*/sm*.fits
+ > ipp_serial_inject.pl --telescope BLANCO --camera MOSAIC2 --workdir path://WORK/2003/W/ --dbname essence /data/ipp003.0/SM_W/SM_SN03/*/w*.fits
+ > ipp_serial_inject.pl --telescope BLANCO --camera MOSAIC2 --workdir path://WORK/2004/SM/ --dbname essence /data/ipp003.0/SM_W/SM_SN04/*/sm*.fits
+ ...
+
+'''LABELS''' : The '''--label''' argument is useful for separating your data into subsets within the database. For example, with the ESSENCE data we may want to apply the same label to each year of the survey,  or perhaps to all the images of a given field  (See the ESSENCE collaboration site for more details: [http://www.ctio.noao.edu/wproject/fields/ ESSENCE fields]).   We can then use these labels to prevent that subset from being processed by using the '''-block''' option in chiptool or camtool.   
+
+These labels can be changed later from within the mySQL database, but it is easier to manage if you think about how to divide your data now.   For the example below I will assume that the label waa1 has been applied to all the images of the waa1 field from the w-project survey, regardless of the date of observation.    (This does not impact the process until we get to the chip and camera stage).
+
+
+First test: inject a single file into the db:
+ > ipp_serial_inject.pl --telescope BLANCO --camera MOSAIC2 --workdir path://WORK/ --dbname essence /data/ipp003.0/SM_W/SM_SN03/sm030927/dflat021.fits
+
+You should see the newly injected file on the ippMonitor under "New Exp."  Then do the whole set:
+ > ipp_serial_inject.pl --telescope BLANCO --camera MOSAIC2 --workdir path://WORK/ --dbname essence /data/ipp003.0/SM_W/SM_SN03/sm030927/[dflat,w,zero]*.fits
+
+(You may want to erase the previously injected file from the database, or else you will get multiple copies)
+
+
+==== Start pantasks ====
+
+See [wiki:Simtest#Start_pantasks Simtest#Start_pantasks] for details.
+
+ pantasks: module pantasks.pro
+ pantasks: module.tasks
+ pantasks: add.database [mydatabase]
+ [ add controller hosts]
+ 
+==== Registration ====
+
+Follow the steps in [wiki:Simtest#Registration Simtest#Registration]:
+
+  pantasks: all.off
+  pantasks: register.on
+  pantasks: run
+
+==== Detrend Construction ====
+
+The procedure here is essentially the same as for the [wiki:Simtest Simtest] walkthrough. You must substitute the appropriate path and database names in your calls to dettool, and we want to pay more attention to the final products. 
+
+When you create a detrend run using the dettool commands given below, the ipp by default sets it up in the "master" mode. It is trying to make a master bias frame using the set of input bias frames you have specified (with, e.g., the "-select_exp_type zero" option).  It will go through the input bias frames and iteratively reject some of them based on criteria set out in the rejections.config file. The base version of rejections.config is kept in the recipes directory.  You may specify a camera-level rejections.config file in your ctio_mosaic2/ directory that will override the recipes/ version.
+
+When working with a large multi-epoch dataset such as ESSENCE, we want to construct multiple detrend images to be applied to different subsets of data.  In the examples shown here, we are building detrends appropriate for the second year of the survey, 2003.  We limit the time range from which this detrend run can draw its input exposures by setting the -select_dateobs_begin/end arguments. We dictate which science exposures will use this detrend by setting the -time_begin/end dettool arguments.
+
+See [wiki:EssenceDetrendLog EssenceDetrendLog] for a complete log of S.Rodney's detrend creation commands. 
+
+==== BIAS ====
+
+ dettool -definebyquery -det_type BIAS -filelevel FPA -workdir path://WORK -telescope BLANCO -inst MOSAIC2 -select_exp_type zero -select_dateobs_begin 2003-09-25 -select_dateobs_end 2003-10-01 -time_begin 2003-09-25 -time_end 2003-10-01 -dbname essence
+
+ pantasks: register.off
+ pantasks: detrend.on
+ pantasks: run
+
+==== Bad Pixel Mask ====
+To build a bad pixel mask, we will go through five steps.  First we need a pre-mask DARK, which is then used to build a DARKMASK.  Then we make a pre-mask FLAT, followed by a FLATMASK.  Finally, we can combine the DARKMASK and FLATMASK together.  For more details of the process, and an explanation of the mask creation logic, see [wiki:Bad_Pixel_Masks Bad Pixel Masks].
+
+===== DARK and DARKMASK =====
+To find pixels that can not be reliably dark corrected, we take a set of dark images, apply a dark correction, and do some statistics on each image and on the stack of images.  Based on statistical cuts (which are set up in the recipes/rejections.config file) we flag some pixels to be masked.  The first step is to create a temporary 'master' DARK, which will be used to dark correct all the images in our input stack.  This dark can be created from the same dark images that it is about to be applied to. 
+
+Now we can start a detRun to create our pre-mask DARK with no rejections: 
+
+ dettool -simple -definebyquery -det_type DARK -dbname essence  -filelevel FPA -workdir path://WORK/2003/DET -telescope BLANCO -inst MOSAIC2 -select_exp_type dark -select_dateobs_begin 2003-09-25T12:00:00 -select_dateobs_end 2004-01-02T12:00:00 -time_begin 2003-09-25T12:00:00 -time_end 2004-01-02T12:00:00
+
+When that is finished, we can apply our new pre-mask DARK to a set of darks to get our DARKMASK:
+
+ dettool -simple -definebyquery -det_type DARKMASK -dbname essence  -filelevel FPA -workdir path://WORK/2003/DET -telescope BLANCO -inst MOSAIC2 -select_exp_type dark -select_dateobs_begin 2003-09-25T12:00:00 -select_dateobs_end 2004-01-02T12:00:00 -time_begin 2003-09-25T12:00:00 -time_end 2004-01-02T12:00:00
+
+===== FLAT and FLATMASK =====
+Essentially the same process:
+
+ * make a pre-mask FLAT
+ # Make a pre-flatmask FLAT in the VR filter using a few early images 
+ dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://WORK/2003/DET  -telescope BLANCO -inst MOSAIC2 -select_exp_type 'sky flat' -filter VR -select_filter 'VR SuperMacho c6027' -select_dateobs_begin 2003-09-25T12:00:00 -select_dateobs_end 2003-10-15T12:00:00 -time_begin 2003-09-25T12:00:00 -time_end 2003-10-15T12:00:00 -simple
+
+ * make a FLATMASK
+ # 2003 : Make a FLATMASK from the first few nights of data
+ dettool -definebyquery -det_type FLATMASK -filelevel FPA -workdir path://WORK/2003/DET  -telescope BLANCO -inst MOSAIC2 -select_exp_type 'sky flat' -filter VR -select_filter 'VR SuperMacho c6027' -select_dateobs_begin 2003-09-25T12:00:00 -select_dateobs_end 2003-10-15T12:00:00 -time_begin 2003-09-25T12:00:00 -time_end 2003-10-15T12:00:00 -simple -pretend | wc -l
+
+===== Combined MASK =====
+
+Now we use a serial perl script to 'OR' the two masks together, making a MASK that can be applied to all future detrend and science images:
+
+ ipp_detrend_combine.pl --det_type MASK --filelevel FPA --inst MOSAIC2 --telescope BLANCO --det_id1 43 --iteration1 0 --det_id2 56 --iteration2 0 --workdir file://data/ipp003.0/rodney/sresswork/2003/ --operation '|' --mask
+ 
+
+==== FLATs ====
+
+ dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://WORK/ -inst CTIO_MOSAIC2 -telescope BLANCO -filter "R Harris c6004" -select_exp_type "dome flat" -select_filter "R Harris c6004"
+
+ dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://WORK/ -inst CTIO_MOSAIC2 -telescope BLANCO -filter "I c6028" -select_exp_type "dome flat" -select_filter "I c6028"
+
+'''NOTE ON FILTERS:''' The -select_filter argument specifies which files will be accepted into the input list.  It must match the filter name exactly, and it is case specific.  If you want to combine input images that have been registered with slightly different filter names (e.g. "VR SuperMacho c6027"  and "VR supermacho c6027") then you will have to use a different dettool mode.  Try dettool -definebytag to provide an explicit list of exposure ids.
+The -filter argument tells dettool what to list as the filter name for the output FLAT image.  This should generally be a more general bandpass name (like 'VR' or 'i') and it should be listed in your camera.config file under the FILTER.ID metadata section.  A 'VR' flat will only be applied to science images that have filter names which translate back to 'VR' in the FILTER.ID metadata table.
+We expect the filter concept to be expanded and clarified somewhat in future releases.
+
+
+==== Detrend Analysis ====
+
+After creating a first master bias, you will want to examine the residuals from the component bias images used as input. In ippMonitor, go to Detrend Steps - Detrend Frames and click on the 'resid' link next to the bias you've just created. This will show you the residuals from the component images. Check out the jpeg images and the reported stats: background, stdev of components, stdev of image, stdev of binned image, S/N, etc.   
+
+You might want to make some plots of the statistics shown in this - or some other - ippMonitor table. For this we use the '''dvo''' program. '''dvo''' is a mongo-like plotting program that can also access SQL databases to ingest data. Within '''dvo''', you connect to a database with 'dbconnect', and you can issue sql commands with 'dbselect {sql command}'.  (Useful SQL queries can be grabbed from the bottom of the ippMonitor pages.) The results of the sql command are returned as '''dvo''' vectors (but all values are floats, so you can't retrieve string-valued data cells). Use 'help {cmd}' to get some command help.  In particular, try 'help style' for plotting style parameters. Here is an example of how to plot the raw detrend image backgrounds against their exposure times:
+
+ > dvo
+ dvo: dbconnect (hostname) (username) (database)
+ dvo: dbselect exp_time,bg FROM rawExp WHERE rawExp.exp_type != 'OBJECT'
+ dvo: vectors
+    N       name      size
+    0   exp_time         60
+    1         bg         60
+ dvo: limits bg exp_time
+ dvo: box
+ dvo: plot bg exp_time -x 2 -pt 7 -sz 2
+
+If you pick out some bad images that are degrading your master bias, you will want to make a new detrun that excludes those bad eggs. There are a ton of dettool modes and options available to pick out your desired subset of images.  Call dettool with no options to get a list of major modes. You can only run dettool with one of these major modes at a time.  Use "dettool -(major_mode)" to get a list of the options for the mode you are interested in. Some are required, others optional.  Here is an example that makes a master bias by calling out a couple of input frames explicitly using their exposure tags:
+
+ dettool -definebytag -det_type BIAS -workdir path://WORK/ -exptag zero001.53 -exptag zero010.62 -exptag zero018.70 
+
+(I grabbed the exposure tags from the ippMonitor tables)
+
+==== Chip and camera processing ====
+
+The next step is to apply the detrending to the science images. The available detrend steps are:
+ * overscan subtraction
+ * bias subtraction
+ * dark subtraction
+ * flat field normalization
+ * mask bad pixels
+ * fringe subtraction
+ * source identification and photometry
+ * astrometry on each chip
+ * astrometry for a mosaic of multiple chips
+ * save detrended image(s) 
+
+All of these steps can be turned on or off by editing this camera's version of ppImage.config, which should contain switches for the various steps used in the creation of detrend images as well as in the creation of detrended science images. Remember to point to the modified version of ppImage.config in your .ipprc and camera.config files!
+From within pantasks, starting the chip and camera level processing is simple:
+ 
+ pantasks: chip.on
+ pantasks: camera.on
+ pantasks: detrend.off
+ pantasks: run
+
+==== WARP ====
+
+The next stage is to warp the detrended images onto a common tangent plane projection of the sky.  The first step is to define the SKYCELLS onto which our images will be warped.  This is done with the skycells program, which will create two fits tables that hold all the skycell info : Images.dat and Photcodes.dat. 
+
+<pre>
+ USAGE: 
+ skycells (level) [-scale arcsec/pix] [-nx (Nx cells)] [-ny (Ny cells)]
+
+  -v                              : verbose mode
+  -triangles                  : save base triangles instead of skycells
+  -fix-ns                     : orient skycells with y-axis aligned with Dec
+  -scale                      : set pixel scale (default 1.0 arcsec / pixel)
+  -padding                    : pad skycells by this fraction in each dimension
+  -nmax                           : only keep nmax skycells in memory
+  -nx                             : subdivide skycell projection in x
+  -ny                             : subdivide skycell projection in y
+  -help                           : this list
+  -h                              : this list
+</pre>
+
+The required 'level' argument defines (something important)...
+
+Use the "-D CATDIR <dir>"  option to specify the directory where Images.dat and Photcodes.dat should be written.  The default is to make a subdirectoy called "default" in your cwd.
+
+After you've created the Images.dat and Photcodes.dat catalogs, be sure to point to them in your camera.config file:
+
+ TESSELLATIONS   METADATA
+       # Tessellation name with corresponding DVO catdir
+       Skycells        STR     file://path/to/catidir
+ END
+
+
+=== Required commands ===
+
+ skycells -D CATDIR skycells 8 -scale 0.2
+ 
+ ipp_serial_inject.pl --telescope BLANCO --camera MOSAIC2 --workdir path://PAP/WORK/DETREND --dbname pap DETREND/*.fits
+ ipp_serial_inject.pl --telescope BLANCO --camera MOSAIC2 --workdir path://PAP/WORK/R --dbname pap R/*.fits --label R --tess_id SKYCELLS
+ ipp_serial_inject.pl --telescope BLANCO --camera MOSAIC2 --workdir path://PAP/WORK/I --dbname pap I/*.fits --label I --tess_id SKYCELLS
+ 
+ dettool -dbname pap -definebyquery -det_type DARK -filelevel FPA -telescope BLANCO -inst MOSAIC2 -workdir path://PAP/WORK/DETREND/ -select_exp_type 'dark'
+ dettool -dbname pap -definebyquery -det_type FLAT -filelevel FPA -telescope BLANCO -inst MOSAIC2 -workdir path://PAP/WORK/DETREND/ -select_exp_type 'dome flat' -select_filter 'R Harris c6004' -filter 'R'
+ dettool -dbname pap -definebyquery -det_type FLAT -filelevel FPA -telescope BLANCO -inst MOSAIC2 -workdir path://PAP/WORK/DETREND/ -select_exp_type 'dome flat' -select_filter 'I c6028' -filter 'I'
+ 
+ dettool -dbname pap -definebyquery -det_type FLATMASK -filelevel FPA -telescope BLANCO -inst MOSAIC2 -workdir path://PAP/WORK/DETREND/ -select_exp_type 'dome flat' -select_filter 'R Harris c6004' -filter R
+ dettool -dbname pap -definebyquery -det_type FLATMASK -filelevel FPA -telescope BLANCO -inst MOSAIC2 -workdir path://PAP/WORK/DETREND/ -select_exp_type 'dome flat' -select_filter 'I c6028' -filter I
+ dettool -register_detrend -parent 5 -det_type MASK -filelevel FPA -filter R -inst MOSAIC2
+ dettool -register_detrend_imfile -det_id 7 -uri path://PAP/WORK/DETREND//MOSAIC2.FLATMASK.5/MOSAIC2.FLATMASK.norm.5.0.fits -class_id fpa
+ 
+ dettool -register_detrend -parent 6 -det_type MASK -filelevel FPA -filter I -inst MOSAIC2
+ dettool -register_detrend_imfile -det_id 8 -uri path://PAP/WORK/DETREND//MOSAIC2.FLATMASK.6/MOSAIC2.FLATMASK.norm.6.0.fits -class_id fpa
+ 
+ stacktool -dbname pap -queue -select_telescope BLANCO -select_inst MOSAIC2 -select_filter 'R Harris c6004' -workdir path://PAP/WORK/ -select_good_frac_min 0.2
+ stacktool -dbname pap -queue -select_telescope BLANCO -select_inst MOSAIC2 -select_filter 'I c6028' -workdir path://PAP/WORK/ -select_good_frac_min 0.2