Changes between Initial Version and Version 1 of EssenceTest_IPP_2.0

Timestamp:

Feb 24, 2009, 4:23:55 PM (17 years ago)

Author:

trac

Comment:

--

EssenceTest_IPP_2.0

@@ +1 @@
+=== Summary ===
+The ESSENCE survey from the CTIO 4m telescope in Chile provides a good dataset for testing the IPP. This page details the steps required to run the ESSENCE data through the IPP. 
+
+----
+=== Annotated ESSENCE test walkthrough ===
+
+==== Configuration: ====
+ * Check ~/.ipprc (or $PS_SITE) for:
+  * DATAPATH for DATA,WORK
+   * use DATA to specify where your raw data is. When making program calls on the command line, refer to this directory as path://DATA/
+
+  * DBSERVER, DBNAME, DBUSER, DBPASSWORD
+
+ * Check ~/.ptolemyrc for:
+  * CONFDIR
+
+ * If you are making config tweaks for the ctio mosaic2 camera, make a configuration directory for them.  
+  * in this ctio_mosaic2 config directory, you should have a camera.config file
+  * Point to that dir in your .ipprc file. e.g.:
+  ### Setups for each camera system
+  CAMERAS               METADATA        
+      MOSAIC2                   STR     /home/panstarrs/rodney/ctio_mosaic2/camera.config
+  * in that camera.config file you can set up the parameters that ipp uses to recognize and interact with files originating on the ctio_mosaic2 camera. In camera.config you must point to a format.config file, which tells the IPP what your camera's fits file format looks like. These formats must match up with what ppImage.config is 
+
+
+
+ 
+==== Initiating a database. ====
+
+Create an empty database: the dbadmin tool is a shell script that creates an SQL database and populates it with the tables needed by ippMonitor for user administration. The dbadmin script is probably not in your path, so find it in the ippMonitor source code directory and link to it or copy it somewhere.  Call dbadmin with no options for syntax help.
+
+ > dbadmin project alala ipp sress1
+
+Then initiate the new database
+ > pxadmin -create -dbname sress1
+
+See [wiki:Simtest#Initiating_a_database Simtest#Initiating_a_database] for alternate methods.
+
+==== Injecting data ====
+
+First test: inject a single file into the db:
+ > ipp_serial_inject.pl --workdir path://WORK/ --dbname sress1 /data/ipp003.0/SM_W/SM_SN03/sm030927/dflat021.fits
+
+You should see the newly injected file on the ippMonitor under "New Exp."  Then do the whole set:
+ > ipp_serial_inject.pl --workdir path://WORK/ --dbname sress1 /data/ipp003.0/SM_W/SM_SN03/sm030927/[dflat,w,zero]*.fits
+
+(You may want to erase the previously injected file from the database, or else you will get multiple copies)
+
+==== Start pantasks ====
+
+See [wiki:Simtest#Start_pantasks Simtest#Start_pantasks] for details.
+
+==== Registration ====
+
+Follow the steps in [wiki:Simtest#Registration Simtest#Registration].
+
+==== Detrend Construction ====
+
+The procedure here is essentially the same as for the [wiki:Simtest Simtest] walkthrough. You must substitute the appropriate path and database names in your calls to dettool, and we want to pay more attention to the final products. 
+
+When you create a detrend run using the dettool commands given below, the ipp by default sets it up in the "master" mode. It is trying to make a master bias frame using the set of input bias frames you have specified (with, e.g., the "-select_exp_type zero" option).  It will go through the input bias frames and iteratively reject some of them based on criteria set out in the rejections.config file. The base version of rejections.config is kept in the recipes directory.  You may specify a camera-level rejections.config file in your ctio_mosaic2/ directory that will override the recipes/ version.
+
+===== Bias =====
+
+ dettool -definebyquery -det_type BIAS -filelevel FPA -workdir path://SRESSWORK -inst MOSAIC2 -telescope BLANCO -select_exp_type zero
+
+ pantasks: run
+
+===== Dark =====
+
+ dettool -definebyquery -det_type DARK -filelevel FPA -workdir path://SRESSWORK/ -inst MOSAIC2 -telescope BLANCO -filter NONE -select_exp_type dark
+
+'''Note:''' most nights of essence data do not have dark frames. See, e.g., sm031026 for a handful of dark frames taken on a cloudy night.
+
+===== Flats =====
+
+ dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://SRESSWORK/ -inst MOSAIC2 -telescope BLANCO -filter "R Harris c6004" -select_exp_type "dome flat" -select_filter "R Harris c6004"
+
+ dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://SRESSWORK/ -inst MOSAIC2 -telescope BLANCO -filter "I c6028" -select_exp_type "dome flat" -select_filter "I c6028"
+
+'''NOTE:''' Make sure the -filter you give it here matches up with the filter name in the database entries for your science exposures. (i.e. don't do "-filter I" if your science exposures have "I c6028" as their filter name).
+
+==== Detrend Analysis ====
+
+After creating a first master bias, you will want to examine the residuals from the component bias images used as input. In ippMonitor, go to Detrend Steps - Detrend Frames and click on the 'resid' link next to the bias you've just created. This will show you the residuals from the component images. Check out the jpeg images and the reported stats: background, stdev of components, stdev of image, stdev of binned image, S/N, etc.   
+
+You might want to make some plots of the statistics shown in this - or some other - ippMonitor table. For this we use the '''dvo''' program. '''dvo''' is a mongo-like plotting program that can also access SQL databases to ingest data. Within dvo, you connect to a database with 'dbconnect', and you can issue sql commands with 'dbselect {sql command}'.  (Useful SQL queries can be grabbed from the bottom of the ippMonitor pages.) The results of the sql command are returned as dvo vectors (but all values are floats, so you can't retrieve string-valued data cells). Use 'help {cmd}' to get some command help.  In particular, try 'help style' for plotting style parameters. Here is an example of how to plot the raw detrend image backgrounds against their exposure times:
+
+ > dvo
+ dvo: dbconnect (hostname) (username) (database)
+ dvo: dbselect exp_time,bg FROM rawExp WHERE rawExp.exp_type != 'OBJECT'
+ dvo: vectors
+    N       name      size
+    0   exp_time         60
+    1         bg         60
+ dvo: limits bg exp_time
+ dvo: box
+ dvo: plot bg exp_time -x 2 -pt 7 -sz 2
+
+If you pick out some bad images that are degrading your master bias, you will want to make a new detrun that excludes those bad eggs. There are a ton of dettool modes and options available to pick out your desired subset of images.  Call dettool with no options to get a list of major modes. You can only run dettool with one of these major modes at a time.  Use "dettool -(major_mode)" to get a list of the options for the mode you are interested in. Some are required, others optional.  Here is an example that makes a master bias by calling out a couple of input frames explicitly using their exposure tags:
+
+ dettool -definebytag -det_type BIAS -workdir path://SRESSWORK/ -exptag zero001.53 -exptag zero010.62 -exptag zero018.70 
+
+(I grabbed the exposure tags from the ippMonitor tables)
+
+
+==== Chip and camera processing ====
+
+The next step is to apply the detrending to the science images. The available detrend steps are:
+ * overscan subtraction
+ * bias subtraction
+ * dark subtraction
+ * flat field normalization
+ * mask bad pixels
+ * fringe subtraction
+ * source identification and photometry
+ * astrometry on each chip
+ * astrometry for a mosaic of multiple chips
+ * save detrended image(s) 
+
+All of these steps can be turned on or off by editing this camera's version of ppImage.config, which should contain switches for the various steps used in the creation of detrend images as well as in the creation of detrended science images. Remember to point to the modified version of ppImage.config in your .ipprc and camera.config files!
+From within pantasks, starting the chip and camera level processing is simple:
+ 
+ pantasks: chip.on
+ pantasks: camera.on
+ pantasks: detrend.off
+ pantasks: run