IPP Progress Report for the week 2010.10.11 - 2010.10.15

(Up to IPP Progress Reports)

Eugene Magnier

I have been chasing down issues in the stack photometry analysis, and in the process have gotten side-tracked by concerns in the PSF matching code. The PSF-matched stacks were looking very poor (lots of ringing around bright stars). I identified some small bugs, but the main issue seemed to be a failure of the PSF matching code to generate well-matched images, despite reasonably easy inputs. The PSF matching code essentially employs the Alard-Lupton algorithm using in difference image analysis, so to explore this further, I worked with Mark Huber to compare ppSub and 'hotpants' (the photpipe implementation of Alard-Lupton) on some simple images. The result: even in some cases, ppSub was failing to get as good a subtraction as hotpants. By running hotpants on the same images and looking at the code, I was able to identify an important difference. The convolution kernels need to be generated with zero net flux (due to the way the algorithm is defined). This require modification of the even-order polynomial kernels. In the case of ppSub, the zero flux was achieved by subtracting a delta-function from each of those kernel terms; for hotpants, the zero-order smallest kernel was being subtracted in full. These seem equivalent mathematically, but when I changed the ppSub code to use the same technique, the resulting subtractions were much better matches to the hotpants version. Although the issue was most obvious in the single-convolution difference, this fix also had the important side-benefit of substantially improving the dual convolution analysis. A comparison of the prior version of the code (from 9/18, just before photfest) and this most recent fix can be seen here: ppSubModifications_201009.

How does the above discussion jibe with reports from photfest that the two codes produce effectively equivalent subtractions? For a couple of reasons: (a) the differences are somewhat subtle and can be overwhelmed by the unmatchable differences in real image, (b) ppSub was able to give good results for part of the test images.

I have also finally implemented the Cosmic Ray rejection and star/galaxy separation measurements suggested by Eddie Schlafly, and tested the cosmic ray masking. A discussion of the star/galaxy separation can be seen here: StarGalaxySeparation.

Serge Chastel

Heather Flewelling

Roy Henderson

  • PSPS
    • Continued creating 3PI batches, ready for loading once merge is complete
    • investigated huge slow-down for certain batches (see below)
  • IPP
    • Czartool
      • finally removed confusing cleanup-jumps from plots (with an option to put them back, if useful)
      • finished 'rate-of-processing' plots (for Ken's daily report)
      • method to figure out start/finish time for a given stage (for use with Ken's daily report)
      • added 1st derivatives plots for all stages
      • added 'hasPlateaued()' method for use by roboczar
      • issues with ippMonitor Db user permissions for czardb (fixed, though I need to look more closely at this)
    • DVO
      • found 0/360 boundary bug that was affecting ippToPsps. Applied a Band-Aid
      • investigated other minor bug with imlist command
    • Other IPP
      • wrote a quick backup script for czarDB and ippToPsps Dbs and mailing list code. Serge set up replication for mailing list Db.
  • Other
    • one day off to look after ill family
    • finally got Ken's new mailing lists set up
    • Czar for two days. Plain sailing.

Bill Sweeney

  • modified ipp_cleanup to deal with files that are lost on dead volumes (such as ipp020). Ran the program
  • Added modes to the chip, warp, and diff tool programs to support reverting of runs that failed to clean up properly.
  • Analyzed the flow of daily ipp processing and found a couple of bottlenecks that increased latency. Moved magic streak detection to the distribution pantasks. We still have some optimization to do. Right now warp processing gets delayed due to ThreePi? chip processing increasing latency for OSS and MD data for which timeliness is important.
  • Looked into the reasons that about 5% 3pi data is not getting fully processed through diff, magic, and distribution. Part of the problem is that the difference stage queuing is too restrictive. Visits that do not occur within 1 hour don't get diffed against each other. We should probably consider using the 'object' field in the database to insure that warp warp difference pairs are paired correctly. Especially for the great reprocessing.
  • Fixed problem with difference stage postage stamp images reported by Larry.
  • Modified distribution and postage stamp extraction programs to stop censoring pixels which only have the "diffraction spike" and "star core mask" bits set.
  • Improved timing parameters of certain pantasks jobs that were timing out frequently due to mysql server slowness. (Root problem was that mysqld was tired and needed to be restarted, but these changes will make the system more reliable).
  • Minor fixes to the postage stamp request form processing.
  • Fixed some cases where the postage stamp request parser was setting incorrect error codes.

Chris Waters

  • Linearity: held up due to programming bug in correction code.
  • Nebulous: reworked replicate query to find all objects that have a number of instances different than the requested number of instances. These files are then replicated or culled until the numbers match. Wrote and tested disk balancing code to shift second copies of files onto the hosts with the largest disks.
  • Identified and removed leftover burntooled pixel data that was missed last time we cleaned that up.