IPP Progress Report for the week 2010.09.20 - 2010.09.24

(Up to IPP Progress Reports)

Eugene Magnier

I spent the week working on issues related to stacking, difference imaging, and running the test suite for the upcoming photfest. Some of the important development work:

I added fuzzy logic for suspect pixels in the stack masking. I had earlier fixed the propagation of "bad" mask bits, but the suspect bits were too aggressive. I've updated the code to do the following: (a) for bad pixels, set the output mask to the OR of all bad input pixels; (b) for good pixels (ie, an output value is calculated), set the output mask to the fuzzy AND of the inputs pixels *actually* used to calculate the output pixel value. The fuzzy AND raises a bit if 65% of the input pixels see that bit. This change was necessary to allow the nightly stacks to include information about suspect pixels due to, eg, excessive burntool or excessive diffraction spikes. The suspect nature of the pixel folds into the PSF_QF_PERFECT score (psf-weighted fraction of pixels with neither BAD nor SUSPECT pixels).

I've updated the rules for bright star masking (a SUSPECT not a BAD bit). The size of the bright star circles was OK for very bright stars (-23 inst mags), but too small for stars at or just beyond the saturation limit. I adjusted the slope of the rule to make a better fit to real star sizes.

I've adjusted the parameters for the newish parameters DIFF_R_P,M and DIFF_SN_P,M (distance to and s/n of the nearest neighbor). The max match distance was previously set too low, and clear cases were being missed. I've bumped it up (probably excessively) to 50 pixels. I think we could probably do better on this test by weighting the distance by the S/N, but I have not explored this area.

Serge Chastel

  • IPP: Czar on Monday and Tuesday (ipp020 crash, wrote log in wiki)
  • PSPS: Tests. Improved test report. Fixed unit tests and improved coverage.
  • MOPS: Started to look at Larry's questions: 1) Query IPP to know if a chunk is fnished? 2) What was done during a night? 3) Check where the warp-stack diffs are and copy them to ds

Heather Flewelling

  • vacation monday afternoon - Friday
  • ppMerge fits (unclear if it is better with/without variance, there is a difference, I need to investigate)
  • sporadically checked email during vacation:
    • restarted summitcopy on saturday
    • sent link to CZW - MD refstacks on ifaps1 following ipp020 failure

Roy Henderson

  • PSPS
    • restarted loading after some fairly major refactoring to my IPP code
    • stopped and started during the week due to IPP machine difficulties (ipp006 going down)
    • spent time creating masked chip images for Jim: learning to use the IPP chip_imfile code.
  • IPP
    • czartool:
      • fixed burntool bug (crazy negative numbers)
    • ippToPsps:
      • finished-up refactoring of Perl classes
      • rewrote certain maintenance scripts to use the above
      • new makeStacks.pl script to creak stacks using ...
      • new StackBatch class, a subclass of Batch, which implements functionality common to all batch types
      • changes to C-code to generate stack tables of correct shape. unpopulated as yet

Bill Sweeney

  • Debugged and fixed problem with dvomerge that Bertrand Goldman discovered. The problem only occurred when merging catdirs with different photcode tables.
  • Spent lots of time dealing with fallout of the ipp020 crash, especially with postage stamp server and update processing. Unfortunately this event happened at the same time that users requested tens of thousands of postage stamp requests. I have some ideas on how to further automate the error handling.
  • made some changes to pstamptool to more easily handle data that is not available.
  • Investigated the cause of some unusual features that the MOPS team saw in some 'warp - stack' difference images. They turned out to be caused by cell boundaries in areas where few images contributed to the stack template.
  • helped Gene get some special data for photfest distributed. Updated wiki page that describes how to control distribution.
  • got the special OSS observations through the system. (With a couple of false starts).

Chris Waters

  • linearity: fixed faulty code that wasn't correctly applying the corrections. Discovered correction data isn't smooth across the borders, so will need to fix, regenerate detrend data, and test again.
  • Helped recover processing from ipp020 crash, re-added reference stacks that were housed there from the ifaps1 computer, and turned on replication to duplicate the raw images that now only have one copy. Reference stacks are now replicated as well, to prevent this issue should a host crash in the future.
  • Enabled processing for MD11 data.
  • Iterated on detectability server to correctly process SSdiff images. Still not functional.
  • Identified that the majority of objects flagged as "ghosts" during the friday transient meeting at QUB are either real-looking objects or known crosstalk ghosts. Sent list back to transients for help identifying if crosstalk masks are propagating correctly.