Science Analysis of VYSOS5 data with IPP

(see VYSOS5 Setup for more details on the VYSOS5 IPP Installation and Setup) (return to Data Analysis Examples)

Data Registration

I've defined a database 'vysos5' in mysql on vysosdata1, and am using that to track the VYSOS5 science processing.

All of the raw science images are stored in /drobopro/vysos/VYSOSData/VYSOS-5-Data/Images, which is linked to /home/ipp/alldata.

I injected all of the science images using a script which basically looks like this:

foreach f (alldata/200?????)
 set subdir = `basename $f`
 ipp_serial_inject.pl --telescope vysos --camera vysos --workdir path://PROCESS/$subdir --end_stage reg --dbname vysos5 $f/*.fts
 ipp_serial_inject.pl --telescope vysos --camera vysos --workdir path://PROCESS/$subdir --end_stage reg --dbname vysos5 $f/flats/*.fts
 ipp_serial_inject.pl --telescope vysos --camera vysos --workdir path://PROCESS/$subdir --end_stage reg --dbname vysos5 $f/Calibration/*.fts
end

I ran the registration process on the images injected into the database. There are currently 20796 science images and 7819 detrend images.

I am generating a master BIAS frame. An unfortunate aspect of the VYSSO5 headers means that the DARK and BIAS images have invalid times (1970/01/01). This makes it difficult to control which input images are used to make a master. I have used a random sample for this initial pass, in the assumption that the camera is reasonably stable.

Command to generate the basic master BIAS:

dettool -dbname vysos5 -definebyquery -random_subset -random_limit 25 -workdir path://PROCESS/detrend/bias -inst VYSOS5 -det_type BIAS -select_exp_type BIAS

Command to generate the basic master PSr FLAT:

dettool -simple -pretend -dbname vysos5 -definebyquery -workdir path://PROCESS/detrend/flat -inst VYSOS5 -det_type FLAT -filter PSr -select_exp_type SKYFLAT -select_filter PSr -random_subset -random_limit 25

Ran into trouble with the master flat construction -- it seems ppMerge is leaking some memory and crashing. For now, I've registered the flat from the demo.20090806 database.

Registered the astrometry model.

Launched an analysis of ic348 data:

set options = "-definebyquery"
set options = "$options -ra_min 56.0 -ra_max 56.5"
set options = "$options -decl_min 32.0 -decl_max 32.5"
set options = "$options -filter PSr"

set options = "$options -set_workdir path://PROCESS/ic348.20091014"
set options = "$options -set_end_stage camera"
set options = "$options -set_label ic348.20091014"

# set options = "$options -pretend -simple"

chiptool -dbname vysos5 $options

I initially attempted to run the pantasks and the processing on vysosdata1, which also is the data host and the mysql host. This was not very successful. The load on vysosdata1 was too large and the database interactions very quickly timed out, causing various failures.

To address this, I've had to set up the processing on the IPP fruitbowl in Hilo. This took a big of work:

• merged the vysos branch into the svn trunk
• updated and build the full ipp suite on cherry
• exported /home/ipp from vsysodata1 to the fruitbowl
• (Gavin) updated the fruitbowl automounts to place the main data disks in /exports/(hostname) as an local mount and /data/(hostname) as a remote mount
• added vysosdata1:/home/ipp to /data/vysos on the fruitbowl
• update the firewall to allow nfs and mysql traffic from the fruitbowl to vysosdata1
• added a 'vysos' user on the fruitbowl with UID matching the ipp UID on vysosdata1
• updated the database to change the named locations of the raw data (​file://home/ipp/alldata/...) to abstract paths (path://RAWDATA/...)
• updated the fruitbowl site.config configuration file to point at the correct locations for the raw and processed data targets.

Initially, I was going to have the processing read from /home/ipp/alldata/... and write results to /home/ipp/data/process. However, it turns out that the directory 'alldata' is a simlink to /drobopro, which is in turn an HFS+ device. Without a lot of work, I cannot NFS mount an HFS+ device on the fruitbowl machines. I decided it would be easier to just copy the data that was going to be processed to /home/ipp/data/rawdata/... and update the database. I defined path://RAWCOPY to point at the new location, and wrote a script (vysosdata1:~ipp/workdir/relocate.ic348) to copy the data and modify the database. Although the script has the ic348 in the name, it will work for any data associated with a labeled chipRun.