IPP Progress Report for the week 2011.01.24 - 2011.01.28

(Up to IPP Progress Reports)

Eugene Magnier

It looks like I've fixed the ppSub / magic masking problems. The last two issues I found this week where (a) I needed to turn on SVD dynamic range clipping in the linear algebra solution to avoid introducing noise in cases where the solution was particularly poorly conditioned and (b) I discovered that the covariance averaging process was weighting by the image variance (not the inverse variance). As far as I can tell from the math and the results, a straight average (unweighted) is actually the correct answer. This latter issue meant that the covariance of the difference image was dominated by the covariance of the less convolved image, resulting in a somewhat too small flux for a given detection limit. This explains the magic issue, but it also explains the excess sources in the difference images which correspond to empty sky. This weighting choice has actually been in place since long before the 8/23 software change. I believe the reason it did not impact us earlier was that, before the 8/23 release, the dual convolution process penalties tended to result in a similar amount of smoothing on both images (with excessive overall smoothing), so the different weighting did not have too large of an impact. The code since 8/23 has been trying harder to get the minimal convolution, with the result that the the mean variance of the 2 images are more likely to be rather different from one another.

I ran the 3pi test set that Heather has been running. Here are the numbers of magic streaks from the various test runs:

label max(magic_id) avg(magicMask.streaks)
magictest.3Pi.trunk.default.20101230 108422.5833 122.5833
magictest.3Pi.trunk.modified.20100105 109485.9667 170.3000
magictest.3Pi.ipp20100701.20110106 109801.5000 42.9000
magictest.r28950 111028.7000 36.7250
magictest.ippeam.r30312 113452.5000 147.2500
magictest.ippeam.r30312.v2 113930.5000 134.9500
magictest.ippeam.r30312.v3 114296.5000 156.5000
magictest.ippeam.r30312.v4 114692.5000 222.9737
magictest.ippeam.r30312.v5 114730.5000 98.3611
magictest.ippeam.r30312.v8 118059.5000 45.4412

At this point, we can get back to the task of running a good-sized mask / magic test suite to check on the quality of the new static masking.

Meanwhile, I've started to work on the reference database while that is processing.

Serge Chastel

  • MOPS czar
  • Confirmed that some diffs were not produced/published
  • Nebulous performance issues:
    • Changing O_DIRECT into O_DSYNC didn't change performance (but what is true on ippc01 is maybe not on ippdb00)
    • Looking where nebulous code could be improved

Heather Flewelling

  • deleted more stuff off ipp004 (this is a slow process), freeing up room for addstar.
  • liason (Zagursky, Deacon, Henry) - answered questions and copied files over.
  • magic investigations - looked on warps/diffs at artifacts that are triggering magic. Mostly streaky features, sent examples to Gene to investigate.
  • fringe - there are not a lot of examples with legitimate fringe. The fringe solutions are sometimes very crazy (random? not fitting properly). Started looking at the exposures used to generate the fringes (nothing to report yet).
  • turned addstar back on (merging is off - will be turned on soon)
  • sick 2 days

Roy Henderson

  • PSPS
    • after last week's meeting with Jim and Gene, I was able to populate some more stack fields
    • did some auditing after the (successful) merge of 10,000 batches
    • meeting with Larry, Jim, Conrad and Daniel regarding using PSVO as an interface for accessing MOPS data
    • some work on PSVO to enable queries to the MOPS database
  • IPP
    • ported Heather's exposure summary script into czartool. There is a link from the top of czartool
    • fixed minor bug in the czartool Pantasks class: was not getting current labels properly
    • added ability for czartool to grab current dates from pantasks servers, store them in the Db, then report them to the webpage
  • Other
    • some work with the ipp metrics: rsync'd so they can be seen from the wiki

Bill Sweeney

  • Implemented "cleanup" for magicRuns. This will free up several TB of disk space on ipp050 and ipp053.
  • Implemented changes to the publish system to support publishing detections from uncensored data. This will be used by MOPS.
  • Two days as processing czar.
  • Implemented priority ordering for distribution.
  • Debugged and solved ppStack memory explosion problem. This allowed the MD05 reference stacks to complete.
  • Fixed bug in destreaking code caused by cfitsio giving me a 64 bit weight image. (It was all NANs).
  • Investitgated why ns.stacks.run was failing repeatedly. The problem turned out to be related to the switch of MD05 to the V2 tessellation

Chris Waters

  • Updates to registration/nightly science code:
    • Automatically drop exposures at the download stage if they have imfiles with fault=110 (missing files that will never be downloaded).
    • Add -checkstatus mode to regtool to aid in diagnosing problems in the registration process.
    • Diagnose and fix diff queuing code that was prematurely stopping diff processing. Confirmed that this works correctly now.
    • Worked on ipp_apply_burntool_single.pl to allow multiple sequential exposures to be processed at once. This mode should help the system catch up when large blocks are marked as pending_burntool.
  • Reburntooled data that had not been updated to the newest version of burntool. All science data back to 2009-04-01 should now be ready for reprocessing.
  • Added new recipes and reduction classes for stacking that select the type of fits compression to be used, based on the type of stack created.
  • Began communication with OTIS about data quality/IPP-OTIS interface. Will need to rework the current code to be more useful.