IPP Progress Report for the week 2009.10.19 - 2009.10.23

(Up to IPP Progress Reports)

Eugene Magnier

This week, we have been running some more example processing (MD08 stacks and SweetSpot), while also tracking down various bugs and trying to finish off some additional development tasks. Paul has been making good progress on stacks and diffs. On diffs, he has made a first pass at a completely working dual convolution implementation. This week we will run some large-scale data sets (MD and/or ThreePi) with this as the main difference algorithm to compare with earlier analysis runs. Chris has been concentrating on y-band fringes and verifying the flat-fields. The y-band fringe analysis issues turned out to be harder to track down than expected, but in the end he has gotten the code to work quite well. There were a number of bugs related to things like video cells and the pattern noise removal which took some hunting. We have run a good-size y-band test over the weekend that we can use to check the impact on magic.

I adjusted the photometry zero-point analysis code so that it can (a) use the entire exposure for the zero point calculation (this is now the default) and (b) measure the 'blue edge' of the mag_obs - mag_ref distribution, in the theory that this is representative of the disk population, and a bit less sensitive to the changes in population sample. The test I ran showed that I got the code backwards, so I need to fix that and also implement the bootstrap-resampling analysis of the error as suggested by Shaun Cole. I have also been digging into some of the photometry issues in the difference image analysis, but have not yet gotten very far on that. The MOPS team has reported inconsistencies between the magnitude and error distributions in detections from the difference images. Joel has also noted a problem with the forced photometry analysis which is potentially related.

Heather Flewelling

  • finished md5sums, finished fixing pairs of files with different md5sums, finished injecting md5sums into gpc1. There are some rawIms that are not in gpc1, I can't do anything about that.
  • processed MD08.20091015 through to stacks
  • released & distributed MD08.20091015 stacks
  • processing MD08.20091015 diffs and magic stage now.
  • made more plots for Ken.
  • started processing SweetSpot.20091020 (at warp stage now).

Bill Giebink

  • Continued on "realtest"
  • Out Friday and most of Monday

Paul Price

  • Stacking (branches/pap)
    • Generating unconvolved mask and variance along with image
    • Investigating rejection behaviour: some stars rejected from all images, some features on a single image (e.g., burn) can result in pixels from all images being rejected, scattering of pixels rejected on a single image
    • Some stars not aligned: both warp and convolved warp centroid can differ by up to 1.5 pix
  • Dual convolution subtraction (also on branches/pap)
    • Can suppress common basis functions between dual kernels
    • Dual convolution looks better, but produces more sources from photometry
    • Variance map is lower for result of dual convolution
    • Investigated variance calculation, seems fine (and renormalisation should take care of it anyway), yet there are more pixels in the (S/N)2 image above threshold in the dual convolution diff compared to the single
    • Lots of sources at the edge of masked regions (where there's more noise in the smoothing) --> can be filtered with PSF_QF > ~0.7
    • Increased smoothing also brings out more nasty features (e.g., wiggles, burns)
    • Now satisfied that dual convolution is doing the right thing
  • Fixed temp directory for stacks --> /local/tmp --> /local/ipp/tmp
  • Translation program for Dave Monet (trunk)
    • Implemented translator (ppMonet) which now lives in a new directory (ppTranslate) with ppMops (trunk@r25923)
    • Added to publish system
    • Added 'active' flag to publishClient table
    • Sending lots of data (5811 exposures) from MD, 3pi, SVS, Sweetspot, STS, flatcorr

Bill Sweeney

  • on vacation

Chris Waters

  • Fringe frames: Completed changes to fringe frame scale fitting, and tested that it removes the fringe signal on all chips. This appears to not work as well for chips with significant pattern noise, but will check the results of the first pipeline processed images.
  • Burntool pixel cleanup: Script written to delete these files (burntool processed images) from nebulous. Since we now calculate and store just the tables and do the pixel modifications on the fly, these are just taking up space. This deletion will begin once the raw image MD5SUM tests are done.
  • Flat field stability: Generated g-band skyflat from a single night, and used it to verify a set of sky flat images taken since April. Residual patterns point to illumination and cloud differences, suggesting that the detector response is stable.
  • Nebulous: Began work on fixing bugs in Nebulous by cobbling together a small cluster of computers and forcing them to run the Nebulous server. The server runs, and responds as expected, giving a safe test base for my edits.