Science Analysis of VYSOS5 data with IPP

(return to Data Analysis Examples)

(see VYSOS5 Setup for more details on the VYSOS5 IPP Installation and Setup)

Data Registration

I've defined a database 'vysos5' in mysql on vysosdata1, and am using that to track the VYSOS5 science processing.

All of the raw science images are stored in /drobopro/vysos/VYSOSData/VYSOS-5-Data/Images, which is linked to /home/ipp/alldata.

I injected all of the science images using a script which basically looks like this:

foreach f (alldata/200?????)
 set subdir = `basename $f`
 ipp_serial_inject.pl --telescope vysos --camera vysos --workdir path://PROCESS/$subdir --end_stage reg --dbname vysos5 $f/*.fts
 ipp_serial_inject.pl --telescope vysos --camera vysos --workdir path://PROCESS/$subdir --end_stage reg --dbname vysos5 $f/flats/*.fts
 ipp_serial_inject.pl --telescope vysos --camera vysos --workdir path://PROCESS/$subdir --end_stage reg --dbname vysos5 $f/Calibration/*.fts
end

I ran the registration process on the images injected into the database. There are currently 20796 science images and 7819 detrend images.

Detrend Creation

I am generating a master BIAS frame. An unfortunate aspect of the VYSSO5 headers means that the DARK and BIAS images have invalid times (1970/01/01). This makes it difficult to control which input images are used to make a master. I have used a random sample for this initial pass, in the assumption that the camera is reasonably stable.

Command to generate the basic master BIAS:

dettool -dbname vysos5 -definebyquery -random_subset -random_limit 25 -workdir path://PROCESS/detrend/bias -inst VYSOS5 -det_type BIAS -select_exp_type BIAS

Command to generate the basic master PSr FLAT:

dettool -simple -pretend -dbname vysos5 -definebyquery -workdir path://PROCESS/detrend/flat -inst VYSOS5 -det_type FLAT -filter PSr -select_exp_type SKYFLAT -select_filter PSr -random_subset -random_limit 25

Ran into trouble with the master flat construction -- it seems ppMerge is leaking some memory and crashing. For now, I've registered the flat from the demo.20090806 database (see ~ipp/workdir/register.flat.sh).

I also registered the astrometry model (see ~ipp/workdir/register.astrom.sh).

Sample Science Analysis : IC348

Launched an analysis of ic348 data:

set options = "-definebyquery"
set options = "$options -ra_min 56.0 -ra_max 56.5"
set options = "$options -decl_min 32.0 -decl_max 32.5"
set options = "$options -filter PSr"

set options = "$options -set_workdir path://PROCESS/ic348.20091014"
set options = "$options -set_end_stage camera"
set options = "$options -set_label ic348.20091014"

# set options = "$options -pretend -simple"

chiptool -dbname vysos5 $options

I initially attempted to run the pantasks and the processing on vysosdata1, which also is the data host and the mysql host. This was not very successful. The load on vysosdata1 was too large and the database interactions very quickly timed out, causing various failures.

To address this, I've had to set up the processing on the IPP fruitbowl in Hilo. This took a big of work:

• merged the vysos branch into the svn trunk
• updated and build the full ipp suite on cherry
• exported /home/ipp from vsysodata1 to the fruitbowl
• (Gavin) updated the fruitbowl automounts to place the main data disks in /exports/(hostname) as an local mount and /data/(hostname) as a remote mount
• added vysosdata1:/home/ipp to /data/vysos on the fruitbowl
• update the firewall to allow nfs and mysql traffic from the fruitbowl to vysosdata1
• added a 'vysos' user on the fruitbowl with UID matching the ipp UID on vysosdata1
• updated the database to change the named locations of the raw data (​file://home/ipp/alldata/...) to abstract paths (path://RAWDATA/...)
• updated the fruitbowl site.config configuration file to point at the correct locations for the raw and processed data targets.

Initially, I was going to have the processing read from /home/ipp/alldata/... and write results to /home/ipp/data/process. However, it turns out that the directory 'alldata' is a simlink to /drobopro, which is in turn an HFS+ device. Without a lot of work, I cannot NFS mount an HFS+ device on the fruitbowl machines. I decided it would be easier to just copy the data that was going to be processed to /home/ipp/data/rawdata/... and update the database. I defined path://RAWCOPY to point at the new location, and wrote a script (vysosdata1:~ipp/workdir/relocate.ic348) to copy the data and modify the database. Although the script has the ic348 in the name, it will work for any data associated with a labeled chipRun.

2009.10.22: I needed to update the vysos5 database to bring it up to date with recent code changes. But, there seems to have been a recent change to the fruitbowl libraries which broke my IPP build. The IC348 data is processed to chip, but I cannot start camera until this gets fixed. update Gavin fixed this, and I was able to restart the camera analysis. It proceeded fine, though it required enlarging the astrometry guess region to ~1 CCD field since the pointing is apparently all over the place.

I uploaded ~144 resulting smf files into a DVO database on pikake, and explored the photometry a bit. I ran psphot on the database to calibrate out the transparency variations. It seems the photometry is not great -- the aperture magnitudes have better stability than the PSF magnitudes. This may mean I should be using a different PSF functional form, or perhaps I do not allow enough PSF variation.

VYSOS5 todo list (mostly for EAM)

• check on the psf vs aperture mags for specific images (psphot -visual)
• check tuning options for psphot PSF model
• update dvo 'lightcurve' to return aperture mags and detection flags
• modify relphot so it can use aperture mags for relative photometry