Changes between Initial Version and Version 1 of LBCreduce

Timestamp:

Feb 24, 2009, 4:23:57 PM (17 years ago)

Author:

trac

Comment:

--

LBCreduce

@@ +1 @@
+[wiki:Sebastian.Jester Sebastian's] notes on reducing LBCred data with IPP. Following [wiki:SimtestEAM SimtestEAM].
+
+= Preliminaries: setup =
+
+First I need to create the admin databases:
+
+ dbadmin project aida77 ipp jester_LBCred
+ pxadmin -create -dbname jester_LBCred
+
+and define some paths in site.config:
+
+ DATAPATH       METADATA
+        LBCin   STR     path://LBC/sci/
+        LBCrun  STR     path://LBC/ippreduce/
+        LBCflat STR     path://LBC/calib/flats/
+        LBCbias STR     path://LBC/calib/bias/
+        LBCdark STR     path://LBC/calib/dark/
+ END
+ 
+ # List of tessellations, and their DVO CATDIR
+ TESSELLATIONS  METADATA
+        BOOTESCELLS     STR     path://LBC/skycells/
+ END
+ 
+ # dvo databases used for output
+ DVO.CATDIRS     METADATA
+        BOOTESCAT       STR     path://LBC/catdir
+ END
+ 
+Then I set up some skycells
+
+ skycells 6 -scale 0.2 -fix-ns -D CATDIR /disk1/jester/data/LBT/z7qso/sci/skycells
+
+-fix-ns aligns the skycells with declination - otherwise we have a discontinuity (accidentally we're in a special part of the tessellation).
+
+Somehow, I can't plot up the images in dvo - this happens:
+
+ dvo: imbox -c red ../lbcr.20080212.105410.fits
+ error getting all elements for coordinate mode NONE
+
+= Injection =
+
+== For detrending ==
+
+Inject the data, giving skycell and dvodb names (and maybe also a reduction class?)
+
+ ipp_serial_inject.pl --telescope LBT-DX --camera LBC-RED \
+  --workdir path://LBCrun --dbname jester_LBCred \
+  --reduction PPIMAGE_OBDFR --tess_id BOOTESCELLS --dvodb BOOTESCAT
+  lbcr.*.fits
+
+Actually, telescope and camera (or instrument) are worked out by the database from format.config, so just given here for the user's reference.
+
+Calibration images are injected in the same way, the injection again works out each file's type from format.config:
+
+ ipp_serial_inject.pl --telescope LBT --camera LBCRED \
+  --workdir path://LBCrun --dbname jester_LBCred \
+  ../calib/*/lbcr.*.fits
+
+Start up pantasks and register the data: 
+ module pantasks.pro
+ module.tasks
+ controller host add aida77 # up to 3 times, but check the load on the machine first!
+ add.database jester_LBCred # or fan or kschmidt_lbc_stack
+ run
+
+To see what's going on, there are several commands:
+ status # Show status of various queue tasks
+ status -taskstats # show how long the various tasks took on average
+ controller status # Show how many jobs are queued and how many CPUs are working or not
+
+If something doesn't work, repeat the register_imfile.pl call that's failing in an xterm but with extra diagnostic output, e.g.:
+
+ register_imfile.pl  --exp_id 89 --tmp_class_id fpa --tmp_exp_name lbcr.20080211.103621 --uri path://LBC/sci/lbcr.20080211.103621.fits --dbname jester_LBCred --verbose --no-update
+
+Then you can 'revert' the file that's stuck:
+
+ regtool -revertprocessedimfile -dbname fan
+
+== For mosaicing == 
+
+=== Setup ===
+
+Directories in set up in /IPP/site.config
+ # in DATAPATH METADATA
+ LBTPROC STR    /disk2/jester/data/LBT/z7qso/IPPmosaic/
+ # in TESSELLATIONS     METADATA
+ LBTCELLS       STR     path://LBTPROC/skycells
+ # in DVO.CATDIRS     METADATA
+ LBTPROCCATDIR  STR     path://LBTPROC/catdir
+
+Skycell creation
+ 
+ skycells 6 -scale 0.2 -fix-ns -D CATDIR `ipp_datapath.pl path://LBTCELLS/skycells`
+
+=== Running ===
+
+In an xterm:
+
+ ipp_serial_inject.pl --telescope LBT-DX --camera LBCREDPROC --reduction PROCESSED \
+  --workdir path://LBTPROC --dbname kschmidt_lbc_stack --tess_id LBTCELLS \
+  --dvodb LBTPROCCATDIR ~fan/data/bootes12.fits
+
+Then start pantasks as above, but you only want the stages starting with register, none of the detrend ones. I typically say
+ all.off
+ register.on
+ chip.on
+ camera.on
+and if that works,
+ fake.on
+ warp.on
+ stack.on
+ diff.on # if you're into difference images
+
+= Start a detrend run =
+
+ppImage.config has a section with constraints that are applied to figure out which detrend files to use for which science image. [wiki:SimtestEAM SimtestEAM] mentions a pantasks module mkdetrend.pro which is in ipp-2.5 but only in the source tree at ipp-2.5/ippTasks/mkdetrend.pro - it doesn't get copied into the share/pantasks/modules/ directory for some reason.
+
+mysql detRun table defines date ranges for individual detrend runs