Changes between Version 6 and Version 7 of IPP_Progress_Report_20110124

Timestamp:

Feb 1, 2011, 2:01:17 PM (15 years ago)

Author:

eugene

Comment:

--

IPP_Progress_Report_20110124

@@ -5 +5 @@
 
 === Eugene Magnier ===
+
+It looks like I've fixed the ppSub / magic masking problems.  The last
+two issues I found this week where (a) I needed to turn on SVD dynamic range
+clipping in the linear algebra solution to avoid introducing noise in
+cases where the solution was particularly poorly conditioned and (b) I
+discovered that the covariance averaging process was weighting by the
+image variance (not the inverse variance).  As far as I can tell from
+the math and the results, a straight average (unweighted) is actually the correct
+answer.  This latter issue meant that the covariance of the difference
+image was dominated by the covariance of the less convolved image,
+resulting in a somewhat too small flux for a given detection limit.  This
+explains the magic issue, but it also explains the excess sources in
+the difference images which correspond to empty sky.  This weighting
+choice has actually been in place since long before the 8/23 software
+change.  I believe the reason it did not impact us earlier was that, before
+the 8/23 release, the dual convolution process penalties tended to
+result in a similar amount of smoothing on both images (with excessive overall smoothing), so the
+different weighting did not have too large of an impact.  The code
+since 8/23 has been trying harder to get the minimal convolution, with
+the result that the the mean variance of the 2 images are more likely
+to be rather different from one another.
+
+I ran the 3pi test set that Heather has been running.  Here are the
+numbers of magic streaks from the various test runs:
+
+|| label                                 || max(magic_id) || avg(magicMask.streaks) ||
+|| magictest.3Pi.trunk.default.20101230  ||108422.5833 ||       122.5833 ||
+|| magictest.3Pi.trunk.modified.20100105 ||109485.9667 ||       170.3000 ||
+|| magictest.3Pi.ipp20100701.20110106    ||109801.5000 ||        42.9000 ||
+|| magictest.r28950                      ||111028.7000 ||        36.7250 ||
+|| magictest.ippeam.r30312               ||113452.5000 ||       147.2500 ||
+|| magictest.ippeam.r30312.v2            ||113930.5000 ||       134.9500 ||
+|| magictest.ippeam.r30312.v3            ||114296.5000 ||       156.5000 ||
+|| magictest.ippeam.r30312.v4            ||114692.5000 ||       222.9737 ||
+|| magictest.ippeam.r30312.v5            ||114730.5000 ||        98.3611 ||
+|| magictest.ippeam.r30312.v8            ||118059.5000 ||        45.4412 ||
+
+At this point, we can get back to the task of running a good-sized mask / magic test suite to check on the quality of the new static masking.  
+
+Meanwhile, I've started to work on the reference database while that is processing.
 
 === Serge Chastel ===
@@ -13 +53 @@
    * Changing O_DIRECT into O_DSYNC didn't change performance (but what is true on ippc01 is maybe not on ippdb00)
    * Looking where nebulous code could be improved
+
 === Heather Flewelling === 
 
@@ -21 +62 @@
  * turned addstar back on (merging is off - will be turned on soon)
  * sick 2 days 
+
 === Roy Henderson ===
-
 
  * PSPS
@@ -37 +78 @@
  * Other
   * some work with the ipp metrics: rsync'd so they can be seen from the wiki
+
 === Bill Sweeney ===
  * Implemented "cleanup" for magicRuns. This will free up several TB of disk space on ipp050 and ipp053.