IPP Progress Report for the week 2010.01.25 - 2010.01.29

(Up to IPP Progress Reports)

Eugene Magnier

We've been working on getting ready for the demo month. I built new flat-field correction frames for the new data set, using the images from the previous flat-correction round. We started running the first set of the magic verification data on Monday, but quickly discovered that there were some mis-masked portions of the array. After a couple of iterations, we were able to run some of the verification data through to stack, diff, and magic. We were not happy with the stack and diff results in the first pass, so Paul and I took another look at possible sources of errors in the code. We discovered two important algorithm issues: 1) the normalization term, which was solved for as part of the kernel analysis, is coupled to the solutions. sometimes the analysis can get the right answer, but sometimes the normalization is driven to the wrong value and the resulting kernels are very poor. we solved this by carefully measuring and applying aperture magnitudes to determine the relative normalizations of the two images. 2) the penalty function in the dual kernel analysis (used only for diffs, not stacks) had a mismatched scale, so that the penalties (used to prevent the kernels from going to large values) were 108 smaller than the terms they were penalizing (thus they had no real impact). By adjusting the choice of scaling factor, we solved this problem with the result that dual diffs now do not drive to large kernels. Paul also found an additional error that resulted in an error in the variance for those difference images with partially illuminated skycells. Our second pass on the magic verification suite turned up another (one line) bug that caused the masking to grow enormously. We fixed this error and are running our third attempt. Other minor points: I generated a new y-band fringe correction frame so that the y-band data can be processed for the magic verification test, and I generated a our first flat-corrected flat-field image for w-band (there were some minor configuration issues), so the w-band data may also be processed.

Heather Flewelling

  • culled raw files, freed up 1% of space.
  • liason duties: emailed Matt Holman
  • fixed various cells on the magic mask.
  • created magic verification test set: found from various emails between Gene, Ken, and Will, and added extra md and 3pi. The 'best' set:
    • magicverify.3pi.%20100127 3 pi fields
    • magicverify.md.%20100127 md04 with lots of satellite streaks
    • magicverify.dec07.%20100127 dec=-7.0
    • magicverify.ss.%20100127 sweetspot
    • magicverify.lgl.%20100127 low galactic lat
    • magicverify.m31.%20100127 m31
    • magicverify.shortmd.%20100127 mdfields g,r,i,z 30s
    • magicverify.md20.%20100127 mdfields (sets of 20, g,r,i,z)
    • magicverify.dec35.%20100127 dec=-3.5
  • stacked and diffed and diffed and magicked and reverted and reverted as necessary.
  • created VerifyStreaks for the above labels, and investigated causes of streaks.

Bill Giebink

Roy Henderson

  • workaround for issue of blank rows in DVO tables/jumps in smf data
  • now batching-up data as requested by PSPS: includes job and batch dirs with XML manifest files (md5sum etc)
  • created 'product' on Maui datastore (PSPS_test)
  • throughput test perl script now publishes data to datastore for Conrad to pick-up
  • lots of back and forth with Conrad regarding specifics of FITS format
  • working on SQL queries to pull out locations of diffs/batch files

Paul Price

  • Added new camera format configuration (format_20100122.config) to handle change to temperature keywords on camera side.
  • Added function, pmConfigRecipeValueByView to get recipe values dependent upon the camera component of interest. Applied to pattern subtraction within ppImage.
  • Parameter tuning for stacks; checked works on diff
  • Magic recertification
    • Merged pattern subtraction enhancements into branch
    • Diagnosed memory growth problems as due to change in temperature keywords
    • Merged new gpc1 configuration file into branch to get temperatures compatible with existing dark frames
    • Fix to PSF-matching code to measure normalisation independently from other kernel parameters: normalisation is measured in an aperture of carefully-chosen size
    • Fix to PSF-matching code to only worry about lit pixels to prevent covariance matrix from being evaluated at non-constrained position
    • Fix to variance renormalisation code to only worry about lit pixels to prevent insane renormalisation values
  • Compared diffs with dual/single and convolved/unconvolved stacks: now obsolete following above fixes, to be re-done with new code

Bill Sweeney

* Visiting MPG.

Chris Waters

  • Cleanup: Removed old burntooled pixel data and chip/warp/diffs from last year's processing.
  • Masking: Repaired masks to help magic.
  • Nightly Stacks: Began extending this code to handle data other than MD fields.
  • Crosstalk: Identified a possible new crosstalk rule, but have not implemented masking. The crosstalk artifact is much fainter than the previous ones, making it difficult to identify which chips have this issue.