Changes between Initial Version and Version 1 of DVO_Challenge

Timestamp:

Feb 24, 2009, 4:23:54 PM (17 years ago)

Author:

trac

Comment:

--

DVO_Challenge

@@ +1 @@
+This page contains scripts made by Mike Liu, which aren't really a cookbook but show examples for many dvo commands. [http://ps1sc.ifa.hawaii.edu/PS1wiki/images/Liu_dvo_practice.pdf Mike Liu's three dvo challenge slides from the 2008 Heidelberg meeting] also contain useful examples.
+
+If you have questions, the most sensible forum is the [http://pan-starrs.ifa.hawaii.edu/mailman/listinfo/ps-ipp-users ps-ipp-users@ifa.hawaii.edu mailing list (subscription page)], and the [http://pan-starrs.ifa.hawaii.edu/mailman/private/ps-ipp-users/threads.html ps-ipp-users@ifa.hawaii.edu list archive (accessible to subscribers)] for looking for previous answers. You need a username/pw to be able to see the web page, the usual combination for the science consortium pages works; and then you need your email address and mail list password to be able to see the archive.
+
+__TOC__
+
+== FAQ / Suggestions for Changes ==
+
+Please post both new questions about dvo, as well as answers to old ones if you have them!
+
+=== Questions ===  
+ * what is the comment character for DVO?
+ * errors in 2MASS phot?
+ * line continuation character (for extra long lines)
+ * is it possible to extract the detection limit for a given ra,dec or image,x,y?
+
+=== Change requests/suggestions ===
+
+ * command to clear all variables from memory
+ * change command name from 'vstat' to 'vstats' (or create an alias)
+ * 'czplot' should be able to work without specifying a min or max value
+ * need columnar printing tool for printing vectors
+ * major nuisance: how to know which variables are available for current catalog? Need "help catdir" or something like that
+ * when writing to file with 'write', should generate column headings (or able to add comments)
+ * consistency needed between input files and typing on command line, e.g. "#" and "!" usage
+
+== Examples == 
+
+These examples assume that you have a local IPP installation.
+
+=== DVO tutorial ===
+
+ ============================================================
+ DVO tutorial with Gene Magnier
+ 03/10/08
+ ============================================================
+ 
+ 
+ http://kiawe.ifa.hawaii.edu/Elixir/Ohana/DVO/
+ 
+ **ra values are in DEGREES**
+ 
+ 
+ ssh host_with_dvo_installed
+ 
+ Make sure you source the psconfig script that's appropriate for your shell,
+ and set up your environment for using ipp, e.g.:
+   psconfig ipp-2.5
+ 
+ 
+ Mucking around with dvo
+ 
+   > script dvo.logfile      make recording of session
+   > dvo
+ 
+   # Set up variables with catdir names
+   $catdir_2mass =  /local/path/to/catdir.2mass                  2MASS catalog
+   $catdir_merge =  /local/path/to/catdir.2mass/catdir.merge     2MASS+USNOB catalog
+ 
+ 
+   $a = 0      set scalar
+   echo $a     
+   $a = 5*6    do math
+ 
+   echo {5*6}           things in {} mean math operation
+   
+   create x 0 100 0.1   make an array full of values
+   echo x[]
+   echo x[0]
+   echo x[999]
+ 
+   set y = x^2          do an array operation
+   set b = $a^2         need to handle scalars with "$", but not arrays  
+     
+   create x 0 100 0.1
+   set x = y
+   limits x y           set limits for plotting box
+   box
+   set y = x^2
+   plot x y
+   png -name plot.png
+   ps -name plot.ps
+ 
+   mcreate foo 100 100    make array of 100x100 ("m" = matrix)
+   set xr = xramp(foo)    'xramp' is a function, makes 2d image as a slope in x-axis
+   set yr = xramp(foo)    'xramp' is a function, makes 2d image as a slope in y-axis
+   set zr = xr^2 + yr^2   
+   stats xr               give image statistics
+ 
+   tv xr 0 100            display the results with a range (must give a range)
+   clear -image
+   clear -graph
+ 
+   buffers               list all arrays
+   vectors               list all images
+   ??                    list all scalars
+ 
+ 
+ Messing around with databases
+ 
+    catdir $catdir_2mass     2MASS catalog
+    catdir $catdir_merge     2MASS+USNOB catalog
+ 
+    region 0 0 0.5       set center of region to (RA,Dec)=(0,0) with 0.5 deg half-box size
+    region 0 0 0.5 sin   sine projection
+    region 0 0 0.5 tan   tan projection
+    region 0 0 90 ait ; cgrid     Aitoff projection of the whole sky & show celestial grid
+ 
+    region 0 0 0.5 tan   tan projection
+    pmeasure -all -m 10 15 -pt 7 -c blue    plot all mag=[10,15] detections (J, H, and K)
+                                            using blue point symbol
+ 
+    pmeasure -all -m 10 15 -pt 7 -c blue -photcode 2MASS_J   plot only J-band (bigger symbol = brighter)
+    pmeasure -all -m 10 15 -pt 7 -c red -photcode 2MASS_K
+ 
+    mextract -region ra, dec, photcode, mag     extract info from define plotting region
+    region                                      show current value of region
+    vectors                                     show results
+    cplot ra dec        plot results on celestial sky
+ 
+    pcat -c red         overlay box sizes for sub-regions
+ 
+    skyregion 0 1 0 1   set search region (alternative to using 'region' command)
+    mextract RA, DEC, MAG   now don't have to use "-region" flag
+                            use capitals to define different input arrays
+                            (but catalog fields are case-insensitive)
+ 
+    czplot ra dec mag 10 15    plot {ra,dec} results with mag value encoded in symbol size
+                               must be given the min & max value
+ 
+    mextract -region RA, DEC, MAG where (MAG < 12.0)      restricted extraction
+    mextract -region RA, DEC, MAG where (MAG < 12.0) && (MAG > 2.0) && (RA < 0.25)
+    cplot RA DEC -c red
+ 
+    delete ra dec photcode mag RA DEC MAG
+    avextract -region ra dec 2MASS_J 2MASS_K    
+    avextract -region ra dec 2MASS_J 2MASS_K 2MASS_J:err
+    vectors           (note that resulting array names can't start with a number, so "v_" appended)
+    clear
+    czplot ra dec v_2MASS_J 15 10 -c blue -pt 7
+    pmeasure -all -m 10 15 -pt 7 -c red -photcode 2MASS_J
+ 
+    avextract ra, dec, 2MASS_J, glon, glat where (abs(glat) > 10)     extract based on galactic latitude
+ 
+    limits -n 1 -0.5 3.5 17.0 8.0; clear; box
+    plot {v_2MASS_J - v_2MASS_K} v_2MASS_K
+         NOTE: now that window 1 is active, this will be the default range for "-region"
+    dev -n 0       change back to device 0 
+    avextract -region ra, dec, 2MASS_J, 2MASS_K where (2MASS_J-2MASS_K > 0) && (2MASS_J-2MASS_K < 0.5)
+    plot -n 1 -c red {v_2MASS_J - v_2MASS_K} v_2MASS_K
+ 
+    cursor     poke plotting window, click on mouse to return coordinates
+               "q" to quit
+ 
+    write -f "%10.6f %10.6f %7.3f" test.dat ra dec      dump to file
+    !more test.dat         "!" is shell escape
+    exec more test.dat     also shell escape
+ 
+    list data -x "head test.dat"                        input files
+ 
+    subset r = ra if (dec < 0.2)
+    subset d = dec if (dec < 0.2)
+    subset m = mag if (dec < 0.2)
+    
+    set bool = (dec < 0.2)       create boolean variable for selection
+    setset r = ra if bool
+ 
+    macro foo                    create a macro
+      echo $1
+      for i 0 10
+      echo $i
+      end
+      end
+    foo 5
+    foo sdf sdfsd sdfdsf
+ 
+    input filename               import file of commands
+      in the file, can use "#" as comment character (though this doesn't work on the command line directly)
+      cannot use "!" in the file as shell escape, must use "exec"
+      
+    vstat              get statistics for a vector (no "s" on the command name)
+                         results are stored in variables $MEAN, $MIN, etc.
+    stats              get statistics for a scalar
+ 
+    sort ra dec mag    will re-sort the variables using first variable as the sorting field
+ 
+ 
+ 
+ ----------------------------------------------------------------------
+ 
+ 
+ the 'date' command is what you want here:
+ 
+ dvo: date -h
+ USAGE: date [-var variable] [-seconds] [-reftime seconds]
+ 
+ * by itself, just echoes the date & time
+ * to set a variable with the date, add -var name, where 'name' is the
+ name of your variable.  by default, this is a human readable date, eg
+ Wed Mar 12 08:30:11 2008
+ * to calculate time differences, add the -seconds flag, and the value
+ will be in seconds since 1970 Jan 01 00:00:00.  NOTE: There is a bug
+ in that the variable is represented as a float, and the value is now >
+ 1.2e9 seconds.  The difference is thus not very accurate.  To get
+ around this, use the -reftime feature to report the seconds since a
+ more recent date.  A value of something like 1205340000 works at the
+ moment.
+ 
+ examples:
+ 
+ dvo: date
+ Wed Mar 12 08:30:11 2008
+ 
+ dvo: date -var foo -seconds
+ dvo: echo $foo
+ 1205346664
+ 
+ dvo: date -var foo -seconds -reftime 1205340000
+ dvo: echo $foo
+ 6700
+ dvo: date -var bar -seconds -reftime 1205340000
+ dvo: echo $bar
+ 6708
+ dvo: echo {$bar - $foo}
+ 8
+ 
+
+=== Sample dvo script ===
+
+You can save this as nstars.dvo and then execute all the commands from the dvo shell with
+
+ input nstars.dvo
+
+ 
+ #---------------------------------------------------------------------------#
+ # Re-create 2MASS part of extraction used by Reid et al (2007, Paper XI)
+ #   used to find mid-M dwarfs
+ # 02/11/08 M. Liu
+ #---------------------------------------------------------------------------#
+ 
+ 
+ # define sky region and output file
+ # (remember that RA is in *degrees*)
+ $OUTFILE = junk
+ #$OUTFILE = nstars.2mass
+ $RA_MIN = 0*15  
+ $RA_MAX = 1*15
+ $DEC_MIN = -0
+ $DEC_MAX = 20
+ 
+ 
+ #$OUTFILE = big.2mass
+ #$RA_MIN = 0*15  
+ #$RA_MAX = 4*15
+ #$DEC_MIN = -0
+ #$DEC_MAX = 20
+ 
+ 
+ # cut by galactic latitude
+ $GLAT_MIN = 10  
+ 
+ 
+ # define selection criteria
+ $SELECT =  (2MASS_J < 12) && (2MASS_H - 2MASS_K > 0.23) && (abs(glat) > $GLAT_MIN)
+ 
+ 
+ #----------------------------------------------------------------------
+ # choose 2MASS
+ catdir $catdir_2mass 
+ 
+ # define region
+ skyregion $RA_MIN $RA_MAX $DEC_MIN $DEC_MAX
+ clear
+ limits $RA_MIN $RA_MAX $DEC_MIN $DEC_MAX
+ box
+ grid
+ 
+ # do extraction and keep track of time
+ echo "running AVEXTRACT"
+ echo "extraction area (sq.degs) = " {($RA_MAX- $RA_MIN) * ($DEC_MAX - $DEC_MIN)}
+ date
+ date -var d1 -seconds
+ avextract ra dec 2MASS_J 2MASS_H 2MASS_K glon glat  where $SELECT
+ echo "  found" ra[] "objects"
+ date -var d2 -seconds
+ echo "  run time =" {$d2-$d1} "seconds"
+ 
+ # plot results
+ vstat v_2MASS_J
+ zplot ra dec v_2MASS_J $MAX $MIN 
+ zplot ra dec v_2MASS_J $MIN $MAX -c red 
+ zplot ra dec v_2MASS_J $MAX $MIN -c blue -pt 7
+ 
+ # print results to file and sort by RA
+ write -f "%10.6f%+010.6f %7.3f %7.3f %7.3f %12.6f %10.6f" tmp.dat ra dec v_2MASS_J v_2MASS_H v_2MASS_K glon glat
+ exec sort -nk 1,1 tmp.dat > $OUTFILE
+ 
+ echo " "
+ echo "** FINISHED **"
+ echo "wrote results to outfile = " $OUTFILE
+ echo " "
+
+
+ 
+=== DVO tutorial (old) ===
+
+Since this tutorial is from 2007, some information here may be outdated by now.
+
+ ==================================================
+ DVO tutorial with Gene Magnier
+ 02/13/07
+ ==================================================
+ 
+ 
+ ssh local_host_with_dvo_installed
+ 
+ copy over test materials (works only on IfA cluster, if at all)
+   > cp -r ~ipp/715624o .
+ 
+ DVO overview
+   sky is stored in tables, can adjust density of RA/DEC divisions
+   default divisions are based on those used for HST Guide Star Catalog
+ 
+   photometry systems stored in ~ipp/psconfig/share/ippconfig/dvo.photcodes  
+     1st five lines give *average* magnitudes systems (not for any specific night)
+     "ref" lines are externally provided photometry
+     "dep" lines are average measurements for a given instrument (e.g. one Megacam chip)
+       data from specific nights are strapped onto this average
+       these can then be set onto the master photometric system
+ 
+ info for DVO commands
+    http://kiawe.ifa.hawaii.edu/Elixir/Ohana/DVO/
+ 
+ goofing around DVO
+   module navigate                      tool for graphically exploring database
+ 
+   catdir $catdir_merge    point to catalog of interest
+         each subdirectoy is a band in DEC, containing FITS photometry tables
+         "Images.dat" = contains list of images in directory (none in this case)             
+        this test directory is Tycho, 2MASS, and USNO-B merged
+ 
+   region 0 30 90 sin                 set viewing region
+      0,0 = center of the field in degrees
+      90 = radius of view
+      sin = sin projection (ait = Aitoff projection, also "tan", "par", "gls")
+    cgrid                     show celetial sphere
+ 
+   region 0 0 0.5 tan; cgrid     zoom into a region
+    
+   pmeasure -all -m 10 15        plot a bunch of data
+ 
+   help style                   show all plotting options
+ 
+   style -pt 7                   set for circles
+   clear; pmeasure -all -m 10 15  redraw
+ 
+   region 0 0 0.05 tan; cgrid        choose smaller region
+   clear; pmeasure -all -m 10 15     redraw
+   $MAG = 10                         set plotting region info
+   $dMAG = 5
+   navigate                         poke around
+     go into plotting window
+         m      show data for a given object
+                 shows "final" grizy magnitude
+                 then shows individual measurements from diff catalogs
+        x      quit navigate
+        h      show list of commands
+        S      redraw the stars
+ 
+   pcat         show boundaries of catalogs that contribute to region
+ 
+   mextract     extract vector of measurements from database (e.g. RA, DEC)
+   avextract    extract vector of measurements averaged for the merged catalog 
+   imextract    extract statistics about images (e.g. rms)
+ 
+   mextract help      show list of all possible columns
+   mextract ra        get list of RAs that overlap the defined region 
+   mextract mag:inst  
+   mextract mag:cat
+   avextract RA     get lists of RAs for merged catalog ("RA" is new name)
+   vectors          show list of defined vectors  
+ 
+   echo ra[]        show number of elements
+   echo ra[5000]    show value for an individual element
+ 
+   for i 0 10
+   echo ra[$i]      print first 10 elements
+ 
+   ra[7] = 10       change value of vector
+ 
+   mextract dec
+   lim -n 1 ra dec ; clear ; box ; plot ra dec -x 2 -pt 7     plot data
+ 
+   write        show list of options
+   write <file> <list of vectors> -f "format (C-style)"    write to file
+ 
+ 
+ 
+ Look around the 715624o/ stuff
+   *.sx are Sextractor output
+   *.smp has header info, data in ASCII (reorganization of Sextractor output), astrometry
+   *.xmp has astrometry info
+ 
+ and let's put some data into DVO
+   > cd 715642o/
+   > addstar -D CAMERA megacam 
+   > addstar -D CAMERA megacam -help
+   > addstar -D CAMERA megacam  715624o00.smp
+        produces "default/" directory which contains the database
+   > addstar -D CAMERA megacam "715624o??.smp"    enter them all into the database
+ 
+ go into dvo
+    imlist      show list of images entered in default/
+               (works b/c have set "default/" as nominal setting in "catdir" command")
+               gives field center {RA,DEC}, date, photcode info, airmass, relative phot info
+ 
+    region 280.4 0.15 1       set plotting region
+    images                    show image regions
+    pmeasure -all -m 10 20     plot all data
+    cursor                    poke around images and mark locations
+      can press any key (e.g. "1", "g", whatever) and location is saved
+      "q" to quit this
+    echo $R1    show results from 'cursor' results
+    echo $Rg
+ 
+    region $R1 $D1 0.1
+    pmeasure -all -m 10 20
+    ctrl-R  search old command history
+ 
+ 
+ make some CMDs from 2MASS
+   catdir $catdir_merge    
+   region 0 0 2
+   cmd 2MASS_J - 2MASS_H : 2MASS_H     generate vectors for plotting CMD ("xv" and "yv")
+   vectors                                 
+   lim -n 1 xv yv           set box limits based on vectors (for window 1)
+   plot xv yv -x 2 -pt 2    plot CMD
+ 
+   lim -1 4 18 5            set plotting limits
+   clear; box; plot xv yv
+   cursor
+       eyeball two regions, type "1" and "2" 
+   subset hs = yv if (xv > $X1) && (xv < $X2) && (yv < $Y1) && (yv > $Y2)  select subset of data
+   subset jhs = xv if (xv > $X1) && (xv < $X2) && (yv < $Y1) && (yv > $Y2)  select subset of data
+   plot -c red jhs hs
+ 
+   do extraction along a line
+     set yl = 11 - xv      vector that defines a line
+     subset jhs = xv if (yv< yl)
+     subset hhs = xv if (yv< yl)
+     plot -c blue jhs hs -n 1
+ 
+   set a boolean expression to do selection
+     set flag = (xv > $1)
+     subset jhs = xv if flag
+     subset hs = xv if flag
+     plot -c green jhs hs -n 1 
+ 
+ 
+ can also do data extraction in original 'mextract' commands
+ 
+ 
+ can input file to dvo
+    input <filename>
+ 
+ 
+ to add data from an external source    
+   make a copy of the "dvo.photcodes" file for yourself 
+     add new data are "ref" data, assign a unique number for 1st column
+     choose a unique name, e.g. "othercamera"
+   make a file with columns of {RA DEC magnitude error}
+ 
+    > addstar -D RADIUS 2.0 -help
+    > addstar -D RADIUS 2.0 -ref <filename> -photcode <othercamera> -time 2007/02/13,00:300:00