| Version 7 (modified by , 17 years ago) ( diff ) |
|---|
Notes on running IPP on SDSS stripe 82 multi-epoch data
Sebastian Jester jester at mpia.de
Use these as a guide for running IPP for the first time on data from a new camera.
Questions
- At what stage is the fake object insertion/dection run?
- Is warptool, stacktool, difftool operation going to be automated?
To-dos
- Look up the distortion correction for SDSS images and whether its WCS is good enough or not (ask Hubert Lampeitl?)
Useful for everything
To get more error messages to a command-line invocation of an IPP executable, add
-trace err 10
Step 0: Get a working installation of IPP
Gene set me up at IPP; there's a working version on panapple at MPIA, and I'm working on getting one set up on aida77-80.
See installing and using multiple IPP versions in parallel
Step 1: set up config files
See Configfiles.
Step 2: Run psphot on single SDSS image
First I create a file firstimg.lis which has the name of the input file. Then I run psphot thus:
psphot -list firstimg.lis firsttry
I could also have given it the input image on the command line with
psphot -file input.fits firsttry
firsttry is the rootname for the output. Here's what happens when I run psphot on fpC-001056-i1-0192.fit
Camera SDSS, format SDSS matches header.
Chip 0: 1 1
Cell 0: 1 1
Cell 0: 1 1
PHOTCODE is SDSS.i.21
System random seed value used seed = 4479799321607236930 hex
image sky : mean 1308.983600 stdev 10.202114
background sky : min 890.523438 mean 1308.983154 max 1349.394043 stdev 32.456470
subtracted background model: 0.283947 sec
built smoothed signficance image: 0.824891 sec
104 peaks: 0.104587 sec
100 sources, 100 moments, 4 failed: 0.116634 sec
found 0 satstar blend peaks, leaving 100 sources: 0.000100 sec
identified 11 blended objects: 0.000904 sec
psf clump X, Y: 1.646315, 1.765633
psf clump DX, DY: 0.056862, 0.044356
SN range (moments): 26.758667 - 1668.058228
SN range (peaks) : 26.420599 - 1264.513428
rough classification: 0.000428 sec
selected candidate 64 PSF objects
fit ext: 1.151582 sec for 63 of 64 sources
fit psf: 0.476687 sec for 45 of 64 sources
chisq vs flux fit: 0.006806 +/- 0.227294
chisq vs flux fit term 0: 0.035698 +/- 0.200890
chisq vs flux fit term 1: 2.193609 +/- 0.292923
apresid: -0.018228 +/- 0.021941; keeping 29 of 64 psf stars
try model PS_MODEL_QGAUSS, ap-fit: -0.018228 +/- 0.021941 : sky bias: 0.000000
dSxo: mean: 0.003162, median: 0.001405, stdev: 0.013455, npts: 29
dSyo: mean: 0.000718, median: 0.002953, stdev: 0.012298, npts: 29
ndSx: mean: 0.045632, median: -0.045638, stdev: 1.276208, npts: 29
ndSy: mean: -0.089024, median: 0.223447, stdev: 1.087374, npts: 29
generate residuals for 29 objects: 0.270703 sec
select psf model: 1.917943 sec
psf model PS_MODEL_QGAUSS, ApResid: -0.018228 +/- 0.021941
no external sources supplied
built models for 100 objects: 0.232271 sec
measure ensemble of PSFs: 0.396416 sec
GrowthCurve : apLoss : 0.012242
measure aperture residuals for 64 objects (2 skipped, 0 failed, 34 invalid)
measure full-frame aperture residuals for 27 of 64 objects: 0.879843 sec
aperture residual: -0.012413 +/- 0.019795 : 0.000000 bias, 0.000000 skysat
measure magnitudes : 0.558418 sec for 100 objects (86 with apertures)
replace background flux : 0.056657 sec
creating moments plot
saving plot to firsttry.mnt.png
creating psf model plot
saving plot to firsttry.psf.png
creating ap mag - psf mag plot
saving plot to firsttry.dap.png
complete psphot run: 8.708983 sec
found 0 leaks at psphot
I also tried this on the Mac in Heidelberg and get different results...:
Camera SDSS, format SDSS matches header.
Chip 0: 1 1
Cell 0: 1 1
Cell 0: 1 1
PHOTCODE is SDSS.i.21
System random seed value used seed = 2726282019464764904 hex
image sky : mean 1311.806081 stdev 8.951074
background sky : min 1305.236572 mean 1311.805908 max 1372.219238 stdev 8.360644
subtracted background model: 0.268568 sec
built smoothed signficance image: 0.662050 sec
99 peaks: 0.081675 sec
96 sources, 96 moments, 3 failed: 0.094184 sec
found 0 satstar blend peaks, leaving 96 sources: 0.000078 sec
identified 8 blended objects: 0.007648 sec
psf clump X, Y: 1.645948, 1.765079
psf clump DX, DY: 0.056189, 0.045065
SN range (moments): 26.627832 - 1668.060669
SN range (peaks) : 26.431179 - 1264.521973
rough classification: 0.012332 sec
selected candidate 64 PSF objects
fit ext: 0.840499 sec for 63 of 64 sources
fit psf: 0.416352 sec for 42 of 64 sources
chisq vs flux fit: 0.005232 +/- 0.371809
chisq vs flux fit term 0: -0.088947 +/- 0.187928
chisq vs flux fit term 1: 2.666661 +/- 0.175476
apresid: -0.037835 +/- 0.032469; keeping 26 of 64 psf stars
try model PS_MODEL_QGAUSS, ap-fit: -0.037835 +/- 0.032469 : sky bias: 0.000000
dSxo: mean: 0.001541, median: -0.001044, stdev: 0.014191, npts: 26
dSyo: mean: 0.001470, median: 0.001485, stdev: 0.011560, npts: 26
ndSx: mean: 0.125409, median: -0.191685, stdev: 1.329126, npts: 26
ndSy: mean: -0.105611, median: 0.018051, stdev: 1.013325, npts: 26
generate residuals for 26 objects: 0.283203 sec
select psf model: 1.554244 sec
psf model PS_MODEL_QGAUSS, ApResid: -0.037835 +/- 0.032469
no external sources supplied
built models for 96 objects: 0.170146 sec
measure ensemble of PSFs: 0.447365 sec
GrowthCurve : apLoss : 0.008937
measure aperture residuals for 64 objects (3 skipped, 0 failed, 29 invalid)
measure full-frame aperture residuals for 28 of 64 objects: 0.845621 sec
aperture residual: -0.012332 +/- 0.020148 : 0.000000 bias, 0.000000 skysat
measure magnitudes : 0.417724 sec for 96 objects (85 with apertures)
replace background flux : 0.027076 sec
skipping moments plot
skipping psf model plot
skipping ap-mag resid plot
complete psphot run: 5.548932 sec
I thought I was using exactly the same settings (copied all the config files 1:1 from IfA to MPIA), but maybe not? E.g. why is it skipping the moments plot in HD, but not in HI? According to Gene, the different numbers are explicable by different random numbers (i.e. seeds). The plots missing would only make sense if kapa wasn't compiled properly, but we think it was.
Step 3: inserting results into dvo and inspecting them
psphot produces its output catalog in firsttry.cmf which we load into dvo:
addstar -D CAMERA sdss -accept-astrom firsttry.cmf
-accept-astrom says that dvo should trust the WCS and ignore the fact that there are 0 astrometric stars reported in the header. Ohana commands like addstar require the -D CAMERA flag on the command line.
Then start dvo by typing 'dvo' and plot things:
# Make a plot window
region 0 0 90 ait
cgrid
current style -x 2 -c black -pt 2 -lt 0 -lw 0.500000 -sz 1.000000
# The above line is the output from csgrid. It's not something to type in.
# (May need to repeat both commands after adjusting size of kapa plot window)
# tell it where the catalogs are
catdir /data/ipp001.0/jester/stripe82/coadd/catdir
# make a new display for displaying images
images -n 1
# read a fits image into a buffer buf0
rd buf0 /data/ipp001.0/jester/stripe82/coadd/input/fpC-001056-i1-0192.fit
# display it with cuts given
tv buf0 1300 3000
# in the TV window, click left to pan, middle to zoom out, right to zoom in
# Tell IPP which region of the sky I'm interested in
skyregion 350 355 -2 0
# Extract individual measurements that exist in catdir for that region
mextract ra,dec,mag
# Show how many entries each vector has
vectors
N name size
0 ra 100
1 dec 100
2 mag 100
# convert ra,dec positions into display's x,y positions '''in place'''
coords buf0 -c ra dec
# and overlay the object positions on the display window
vload red ra dec
# To just plot the ra,dec on a coordinate grid (i.e., not an image), say
cplot ra dec
Step 4: warping and stacking
Set up a grid of skycells to which the output will be warped
DVO has Gene's tesselation of the sky built in. To create a table with the skycells of this tesselation, there's the skycells executable:
skycells 8 -scale .3 -D CATDIR /data/ipp001.0/jester/stripe82/coadd/skycells/
The '8' tells it the tiling level, where 1 is coarse and 10 is fine. -scale gives the arcseconds per pixel. The skycells table is put into the directory specified by CATDIR.
To look at the resulting tesselation, bring up dvo:
# Set catdir to the directory where the skycells live catdir /data/ipp001.0/jester/stripe82/coadd/skycells/ # Plot them up, first set up the grid (args are ra dec radius projection) region 352 -1 2 ait images # Overplot the outline of a fits image imbox -c red /data/ipp001.0/jester/stripe82/coadd/input/fpC-001056-i1-0193.fit # Same for a list of images - first read list into a list variable list myImlist -x "ls /data/ipp001.0/jester/stripe82/coadd/input/fpC*.fit" # Plot them up in a for loop for i 0 $myImlist:n >> imbox -c red $myImlist:$i >> end # The preceding was for individual fits files; if dvo already knows about images (via addstar), it works like this: # set catdir to place where images are catdir /data/ipp001.0/jester/stripe82/coadd/catdir/ # plot them up images -c red # To set up the next step, we need to know the id of the skycell. This can be done semi-interactively: cursor # then move the cursor into the skycell whose number you want to know and press any letter key, # e.g. d, to store the ra,dec of the point where you press the key (q to quit, also stores things in q). # $Rq and $Dq then store the ra,dec of that point, and you can use them in 'gimages' to get the skycell id: gimages $Rq $Dq -region output-> reading images (mode DVO_MODE_MEF) output-> 0 skycell.0484667 353.0672 -1.1307 1970/01/01,00:00:00 0 0 0.00 0.000 0.000 0.000 1
skycell.0484667 is the ID we need.
Warp images to the skycell grid
Setup: create a pixelless fits image containg the WCS of the skycell
This uses a shell command:
dvoImageExtract skycell.0484667 -D CATDIR /data/ipp001.0/jester/stripe82/coadd/skycells/ -o /data/ipp001.0/jester/stripe82/coadd/run_ipp/skycell_0484667_warptemplate.fits
to create the "warp template" fits file. It's a fits image with NAXIS=0, i.e., no pixels - just the header.
Run psWarp
You need to have the config files for the "simple" camera to run psWarp as follows:
pswarp -file ../input/fpC-001056-i1-0192.fit psWarp-001056-i1-0192 skycell_0484667_warptemplate.fits pswarp -file ../input/fpC-001755-i1-0330.fit psWarp-001755-i1-0330 skycell_0484667_warptemplate.fits
The 'output' parameter gives a rootname, so no .fits needed - it gets added automatically. The skycell needs to be specified with its full pathname.
Stack the images with ppStack
To run ppStack, you need an input file that specifies input images, weights, masks, seeing; here's an 
ppStack ppStack.mdc ppStack-run1
where ppStack-run1 is the output rootname.
Step 5: run psphot on the stacked image
Initially, this barfed with some config file issue, but Paul fixed it. Now it runs with 
psphot -file ppStack-run1.fits photstack-run1
Step 6: inject stacked psphot results into dvo
Step 7 (really chapter 2): run everything in automated fashion in pantasks
Preliminaries: set up a database in ippMon
The db administrator uses ippMonitor/dbadmin to create a new "project" in the mySQL database (this isolates different reduction runs from each other; I get the impression that it's useful to really have one project per run, as the easiest [and perhaps only?] way to start over with something is to delete all the tables in the project and start from scratch).
Administrator things
If the database is being set up for the very first time at all for IPP use, the admin need sto say (e.g.)
dbadmin init (dbserver) (dbname) (dbpassword) dbadmin init aida77 ipp ********
This creates an administrative database ippadmin which keeps track of everything else and is just required once per IPP installation.
To set up a new project database, then say
dbadmin project (dbserver) (dbuser) (dbname) dbadmin project aida77 ipp jester_s82
confirming with the password set above. This creates a project database jester_s82 that keeps track of all files in that project.
There's also something about creating a www user:
dbadmin user (dbserver) (dbuser) (username) (password)
but I'm still fighting with the web server. NB: in our case, the php script install location is
/srv/www/htdocs/ippMonitor
User things
The server, ipp-dbname, password and project are specified in ~/.ipprc and/or site.config. I have in site.config:
### Database configuration DBSERVER STR aida77 # Database host name (for psDBInit) DBUSER STR ipp # Database user name (for psDBInit) DBPASSWORD STR ******* # Database password (for psDBInit)
and in ~/.ipprc:
DBNAME STR jester_s82 # Database name (for psDBInit)
I as user then use pxadmin to create a new database, or delete or recreate it (make sure that DBNAME is pointing to the database you actually want to work on!)
pxadmin -create
There should proabably be something like an "IPP administrator's guide" (or maybe there is already one) that spells out how mySQL should be set up, what the best way is to make sure that different users don't interfere with each other's tables and projects (e.g. by having multiple mySQL users, if possible), etc.
To start a whole reduction afresh, at the moment the fastest/only way is
pxadmin -recreate
This will delete EVERYTHING in your database DBNAME...
Injecting and registering frames in ippMon
Now we need to tell ippMon about our images. This has been covered in 
Injection means telling IPP about the files
Adding a static bad pixel mask to the db
I've written a 
After pxadmin recreate, the bad-pixel masks need to be injected before the image files, so they are there and can be used'''
Injection with the "single chip = FPA" config
This is in
~jester/IPP/ippconfig/sdss/camera.config
The injection is simple here:
ipp_serial_inject.pl --telescope sdss --camera sdss \ --workdir path://S82/run_ipp --dbname jester \ --reduction PROCESSED /data/ipp001.0/jester/stripe82/coadd/input/fpC*.fits
The "reduction class" PROCESSED selects a given set of reduction steps that is defined in the ppImage.config file. In the sdssmosaic/ppImage.config, I renamed this class into SDSSphotonly, which runs psphot but not psastro, to distinguish it from SDSSphotast, which runs both.
Injection with the "six chips = mosaiced FPA" config
This is in
~jester/IPP/ippconfig/sdssmosaic/
and the camera name is SDSSmosaic. Whether an SDSS image is recognized as SDSSmosaic or the single-chip SDSS format depends on which of the lines in my ~/.ipprc is commented out - the format.config for both files look identical, since the files are identical.
The injection is more involved, since the injection needs to know about the grouping. There are two possibilities: ipp_serial_inject_mosaic.pl and ipp_serial_inject_split.pl
The former needs to be modified to know about the structure of SDSS. It then needs to be called once per group of 6 files that are part of the same "exposure".
The latter needs to have the input files put into subdirectories, with one dir per "exposure". You call it with a glob for all the directories [I think].
Since I need to write a script either way, either to copy to directories or to call with a suitable glob per exposure, I'll take the latter route so we don't need to change ipp_serial_inject_mosaic.pl.
To inject all the files in a given directory $D/stripe82/coadd/input/i/1056-0192/, use this command:
ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \ --workdir path://S82/run_ipp --dbname jester_s82 --tess_id S82CELLS --dvodb S82CATDIR \ $D/data/SDSS/stripe82/coadd/input/i/1056-0192/
tess_id and dvodb are new in 2.5. The tesselation_name and dvodb_name can be explicit paths to catdirs above, but also abstract names defined in site.config
SDSSphotonly corresponds to the PROCESSED reduction class (now PPIMAGE_OP_SOFT) defined in sdss[mosaic]/camara.config using the recipes in sdss[mosaic]/ppImage.config. The other new reduction class is SDSSphotast, which includes astrometry.
For the initial testing, I want to isolate any possible astrometry problems from photometry problems, so I'll continue to work with SDSSphotonly. However, I wanted to make sure that phot & ast both run through, so that I can rectify any mechanical problems while still at the IfA, and then worry only about functional problems later.
Registering
"Registering" means getting them ingested into ippMon:
# Start pantasks pantasks # Inside pantasks, load predefined procuedures module pantasks.pro module.tasks # Add host names for parallel processing controller host add ipp000 # etc. for more hosts. # Now by default, all analysis steps are on by default, but we want # to go through them step by step to make sure everything's working. # Hence we turn 'all' off, and then turn things on selectively. # Also, at this stage, IPP doesn't know by itself that fpC*.fits images # from SDSS don't need detrending, so we need to tell IPP by hand after the # register step. all.off register.on run
You can also save these steps in a macro, by puttin the commands in a file, wrapping them in "macro myMacro" and "end", and then reading the file in pantasks with the "input" command. In pantasks, one can then call the macro by just typing myMacro. On the IfA cluster, my macro that does these things is called thus:
input /home/panstarrs/jester/IPP/macros/inject.pro injectmine
Note that it is possible to 'controller add' the same host multiple times, and it will get multiple shares of the load.
Processing things with the pipeline
All the reduction steps are just switches in the pantasks.pro module; they can be run simply by turning them on, e.g.
chip.on camera.on
runs psphot.
The next step is warping, which requires some extra command-line work (but maybe there's a way to have IPP do this, too? There has to be one, eventually...)
Warping
Defining warp runs
In IPP 2.5, there's a new feature where you specify the skycell tessellation id already in the injection step (see above). You then just need to turn on the warping step, and it figures out what to warp where by itself, and does it.
Run them
Then
warp.on run
will run the actual warping. To see what's going on, say
status
for a general job count, and
status -taskstats
to see statistics on execution times.
Stacking
Now that everything is warped, we could stack it. For this, we need to define a 'stack run' with stacktool.
stacktool -definerun -workdir path://S82/run_ipp -skycell_id skycell.0472823 -warp_id 1 -warp_id 2 -warp_id 5 -tess_id Skycells
This tells IPP to stack things in the skycell given and include the pictures from the warp runs listed. The tess_id is defined in the TESSELLATIONS METADATA in camera.config, and points to the same path that I used to set up the tessellation above
At the moment, you need to figure out yourself which skycells and warp_ids to use. I do this:
# First, connect to the database
mysql -h alala -u ipp -p jester
[enter password]
# then run the following query
select distinct skycell_id, warp_id from warpSkyCellMap
where skycell_id in (select skycell_id from warpSkyCellMap
group by skycell_id having count(distinct warp_id) > 1)
order by skycell_id, warp_id;
The inner query shows you how many warp runs overlap a given skycell:
select skycell_id, count(distinct warp_id) as N_warp, count(*) as N_img from warpSkyCellMap group by skycell_id having N_img > 1 order by skycell_id ;
Conversely, you can find out how many skycells each of your images was split into:
select cam_id, class_id, warp_id, count(distinct skycell_id) from warpSkyCellMap group by class_id, warp_id, cam_id order by cam_id, class_id, warp_id;
If something fails, the way to get IPP to stop trying is
stacktool -updaterun -stack_id 2 -state stop
Diffing
In principle, we could also define a diff run with difftool, but since I'm not interested in image subtraction myself, at least not initially (sorting out brown dwarfs via proper motion may change this in the future), I am skipping this step.
However, the SDSS supernova survey has been preparing high-quality subtractions for this same data set, so it would be a very good comparison set, too. Gajus Miknaitis at Fermilab knows about the SDSS set, and I'm happy to help establishing contacts if necessary.
Running pshot on the stack
Don't know how that works in the pipeline yet.
Loading stacked psphot outputs into dvo
Should work as above, in principle.
Log 2008-05-28
I don't know where I'm at on aida77 so I'm starting afresh.
Set up IPP
Based on the date 20080912 in the names of databases etc. below compared to the dates of IPP installations present in /IPP, I am guessing that this example used
psconfig ipp-2.6.1
There are more versions of ipp in Heidelberg. Those from a numbered tarball have names like ipp-2.6.1; those that were obtained from cvs or svn checkouts are identified by symbolic names with dates such as cvs-20081104 or svn-20090414. cvs-bleed was created on 2008-12-17, ipp-magic-v0 from a tarball with the first magic-approved version of IPP on 2009-01-12.
Creating a database
Start with fresh jester_s82 reduction database:
pxadmin -recreate -dbname jester_s82
or create a new one for a new run:
ippadmin project aida77 ipp jester_s82_20080912 pxadmin -create -dbname jester_s82_20080912
Injecting
Normally I'd want to inject the bad pixel masks as described above, but I'm skipping that for now. Instead, go straight to injection, for now just with all the fields from 2 runs to see that it works:
ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \
--workdir path://S82TEST/run_ipp --dbname jester_s82_20080912 --tess_id S82CELLS --dvodb S82TESTCATDIR \
/IPP/data/SDSS/stripe82/coadd/input/i/{1056,1755}-*/
or for everything:
ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic --reduction SDSSphotast \ --workdir path://S82PROD/run_ipp --dbname jester_s82 --tess_id S82CELLS --dvodb S82PRODCATDIR \ /IPP/data/SDSS/stripe82/coadd/input/?/*/
The symbolic names like S82TEST are defined in ~/.ipprc or /IPP/site.config (or another site config file pointed to by your ~/.ipprc).
With different recipes
At some point, IPP allowed using recipes also for psphot, which are in sdssmosaic/psphot.config and sdssmosaic/reductionClasses.mdc For these, use things like
pxadmin -recreate -dbname jester_s82_20081217_ap3
ipp_serial_inject_split.pl --telescope SDSS --camera SDSSmosaic \
--reduction SDSSphotast_ap \
--workdir path://S82AP3_20081217 \
--dbname jester_s82_20081217_ap3 \
--tess_id S82CELLS \
--dvodb S82AP3CAT_20081217 \
/IPP/data/SDSS/stripe82/coadd/input/i/{1056,1755}-*/
This recipe turns on both aperture corrections and spatially variable PSF fits.
Pantasksing
Then start jobs in pantasks:
pantasks # If it's not in your ~/.pantasksrc module pantasks.pro module.tasks # There's also a file with the sort of commands that I'm describing here # in /home/jester/s82.pro which you can load with input /home/jester/s82.pro # Set database if different from DBNAME in ~/.ipprc add.database jester_s82_20080912 # Turn all reduction steps off, except for register, chip and camera: all.off register.on chip.on camera.on # Add some CPUs for the task scheduler to use # - need to be able to ssh without passwords to the host given # (see /IPP/README) controller host add aida77 -threads 2 # This command can be issued multiple times for adding more # CPUs on the same machine, or more machines. # The -threads 2 is optional and allows multi-threading, # which should be the fastest option. # aida77 (and aida78 etc.) have 2 dual-core CPUs. Hence: # Without the -threads, use at most 3 copies of aida77 # (fewer if more than one person are running pantasks!) # With the threads, use at most 2 copies of aida77. # Now start the processing run # say 'stop' to stop launching new jobs, or 'halt' to # stop processing altogether (and being flooded by error messages) # To see what's going on: controller status # lists active and pending jobs and the status of the hosts status # lists pending jobs and statistics of success/failures status -taskstats # includes information on how long each job ran
NB: with
$VERBOSE = 2
in pantasks, you see the commands as they're issued, which is useful for diagnosing problems. You can also simply re-run the command that's announced as failing, or look at the log file indicated. Normally, to re-run without generating further error messages, you need to remove the --log path://datata argument to see the output (which otherwise goes into a log file), and add --no-update so that the database isn't affected by the manual repeat.
Once the camera-stage processing is finished, there's a dvo catalog with the outputs in S82PRODCATDIR. This needs to be sorted:
cd `ipp_datapath.pl path://S82PRODCATDIR` addstar -D CATDIR . -resort relphot -D CATDIR . -averages
The resorted version is available for download at http://www.mpia.de/homes/jester/IPPstripe82_catdir_20080815.tgz - a tarfile of about 250 MB that expands into a directory of about 500 MB. I have asked our computer people also to make available all the IPP outputs directly; however, those are 250 GB (no compression) so they need to figure out a way to host those. The input fpC*.fits images from SDSS are another ~50 GB. Here is the 
