IPP Progress Report for the week 2011.03.07 - 2011.03.11

(Up to IPP Progress Reports)

Eugene Magnier

This was a productive week with Micael Wood-Vasey visiting. He is investigating the differences between the IPP and photpipe photometry of sources in the photfest stack-stack diffs. He started by looking into the differences between psphot measurements and psphotForced measurements. He uncovered some differences triggered by differences in the background model. Based on this, as well as ongoing concerns that the background model is too sensitive to bright stars, I looked into tweaks that may help. The current processing uses 256 pixel superpixels without oversampling, using a subset of 10k measurements in the superpixel. I discovered that the fitted sky value in the superpixels had high outliers more frequently that expected, and tracked this down to a low-level bug in the way the sky histogram is fitted -- it uses a fit to the lower fraction of the (presumed) gaussian sky distribution to estimate the peak. This can cause problems if the histogram is too noisy, resulting in a poor constraint on the high end of that fit. I fixed this by requiring the full range fit to be greater than the lower half fit, or else the fit from the full range is used. In discussion with Michael, we concluded that the number of samples is also too small, introducing more scatter than we really want. Also, the fairly small size of the superpixels (64 arcsec) makes the measurement too sensitive to bright stars. We have decided to try 400pixel superpixels, with 2x oversampling (ie, the superpixels are spaced by 400 pixels, but sample a region 800 pixels wide), with at least 50k pixels used per measurement.

I also explored the issue mentioned last week about regions where the sky is under subtracted (the model is unable to follow the shape of the sky sufficiently well). I tried to filter sources from this region on the basis of the flux in a small (psf-sized) aperture to that in a larger aperture. This seemed to work well for the image of interest, but in wider application it tends to exclude too many obvious galaxies. I tried an alternative of using the smoothed significance image in the peak culling process, and this seems to be a good balance:

I also looked into a problem we've been having with some stacks in which the target PSF is surprisingly large (eg, 30 pix FWHM when inputs were limited to 8). This is caused by poor PSF models, especially in cases where the input warp has very few sources. I have added code to ppStack to examine the input set of warp PSFs and reject inputs based on 2 criteria: 1) if the PSF is larger than some user-defined max or 2) if the PSF is larger than some number of sigma of the input set. I checked on the impact of this by examining the statistics of warps used for the reference stacks generated to date, and this cut would help a number of those stacks. This issue also explains a concern raised by Nigel and Peter about the large range of PSFs in the MD04 refstacks.

Serge Chastel

Heather Flewelling

  • sick half of Monday and half of thursday
  • ippconfig iterations (many) - finally is working - need to investigate camera stage to look for mask tweaks.
  • addstar - dvoverify can check if sorted (-s), and can tell the first unsorted file it encounters (-s -v), started working on (designing) changes for addstar to be more intelligent about verifying databases.
  • created gpc1gcirc.pl - finds the nearest raw 'sciency' (o%g%o) exposures for a given ra and dec. located in tools
  • czar friday - found raw files with inconsistent copies:
    • gpc1/20090628/o5010g0081o/o5010g0081o.ota45.fits
    • gpc1/20090628/o5010g0082o/o5010g0082o.ota24.fits
    • gpc1/20090628/o5010g0083o/o5010g0083o.ota45.fits
    • gpc1/20090628/o5010g0083o/o5010g0083o.ota52.fits
    • gpc1/20090628/o5010g0085o/o5010g0085o.ota25.fits

Roy Henderson

  • PSPS
    • retrieved more MD04 files for Jim
    • loaded more batches, including an init batch, for Conrad to test the DXLayer
    • wrote a class to encapsulate terrible cfitsio library. Cleaner, safer.
    • worked on parsing model fit table from Gene's newest stack cmf file
    • improvements to unit test for ippToPsps
    • figured out how to make a nice Mac OS app version of PSVO
  • IPP
    • Czar on Monday. No data.
    • added to ippToPsps and czartool documentation on wiki
    • started looking into porting Heather's nice ra/dec image locater script into PHP for ippMonitor
  • Other
    • Sick on Tuesday
    • Doctor's appointment Wednesday morning

Bill Sweeney

  • Some minor postage stamp server wrangling. It has gotten much better at taking care of itself.
  • Modified diff_skycell.pl and stack_skycell.pl to replicate precious output files. Tested changes and integrated into the production branch.
  • Modified the distribution system to facilitate the distribution of raw detrend images. Distributed raw detrend images for MPG.
  • Continued working on analysis of the sts astrometry problem.

Chris Waters

  • Reorganized ipphosts.mhpcc.config to balance the disk load across the cluster. This should utilize hosts with larger disks more than hosts with smaller disks.
  • Finished diffraction spike updates. New wider masks should cover the majority of spikes correctly, except where proper motion has moved the star away from the expected position.
  • Wrote script to identify and truncate large (> 30MB) logfiles. These usually result from jobs that fail repeatedly, so the truncation should not eliminate useful information. Needs to be parallelized and run across the cluster.
  • Various tests to attempt to determine a good initial reprocessing test set.
  • Bugfixes to ppImage to handle format/recipe setting of PATTERN correction correctly.
  • Create updated DQstats detection efficiency CDFs. Incomplete, due to a lack of z and y band images.
  • Checked the timer values created at each stage and compared to database. Added dtime_convolve to ppStack, as this takes up the majority of the processing time. Need to test database change and code changes before committing to trunk.
  • Identified slow processing of chip XY06 as being due to electronic noise on some cells. Created mask to block these cells while camera team worked on resolving the issue. After fix, discovered that there are still single pixel rows that have the electronic noise still.
  • Added options to dettool -updatedetrun to allow the start and end times of a detrend to be set via the tool.
  • Re-read detectability server code and identified a number of bugs that are probably the source of the trouble.