IPP Progress Report for the week 2010.07.26 - 2010.07.30

(Up to IPP Progress Reports)

Eugene Magnier

I've been splitting my time this week on three areas: extended sources, image subtraction, and dvo database issues. For extended sources, I have concluded that the current psf-convolved fitting is working fine, but it is too slow for practical work. There are a couple of things that I can do to speed it up substantially: convert the convolution to use FFTs and add in threading. Both are fairly straight-forward given the rest of the IPP code.

Meanwhile, I have been looking at cases where the warp-warp subtractions are resulting in high numbers of false positives in an attempt to either characterize the false positives and/or to reduce them up front. Towards the first goal, I have added the ability to associate the diff detections with the nearest positive source above some S/N limit in both of the input images. I've added 5 new fields to the output cmf files: the S/N and distance to the source in the positive image (relative to the diff detection) and the same for the negative image, and a new set of flags, including bits the identify that a diff source is matched to a positive image source.

In terms of reducing the false positives, a major contributor seems to come from times when the image quality difference is large (eg, a sub-arcsecond image and one with 1.6 arcsec FWHM). It seems that at least 2 bad things are happening in ppSub in these cases (and maybe to some extent in less extreme cases): 1) the FWHM-based kernel scaling code saturates at one of the extremes, and 2) the normalization is poorly measured. I suspect (2) is more of a problem than (1), but both are a concern. I am looking into improving the robustness of the normalization measurement.

On DVO related issues: I have tested the updated dvomerge code -- I reworked the processing order so that only the new database is completely loaded rather than the output database -- and find that it have a much improved performance. It now runs at a rate that is essentially linear in the size of the input database (ie, the incoming mini database). It is still a bit slower than I might want: it runs at a rate equivalent to about 5-10 MByte per second, so it is clearly not I/O limited. It is single threaded, but it would not be difficult to change to a multi-threaded option (one thread per merged database sky region).

Serge Chastel

  • IppMonitor:
    • Plots for diff images are available
    • Plots for warp processed exposures should be ok now
    • Both can be seen on ippmonitor-test
  • IPP:
    • Checking if ippTools/share/addtool_find_cam_id.sql SQL query can be improved/optimized
  • PSPS:
    • Meeting with Conrad for testing activities
  • Big trouble for Belfast meeting: I lost my passport last Thursday.

Heather Flewelling

  • liason duties (answering questions for ifaps1)
  • passport issues relating to upcoming travel to belfast
  • generated plots for Ken for isp/gpc1 sky brightness
  • detrend investigations using SimTest Camera (100x100 pixel version)
    • I generated detrends with (for example, a 1st order temp term, and used ppMerge to fit that first order term)
    • 1st order in temp ppMerge sometimes works
    • 2nd order only in temp ppMerge works correctly
    • 1st and 2nd order: 2nd order works correctly, 1st does not
  • created a script that takes difftool query type arguments (and a grep) to find the diff logs and grep something out of them. This is checked into trunk/tools/difflog_grep.pl

Roy Henderson

  • PSPS
    • generated SAS test data for Jim:
      • wrote new code to create a new type of test batch
      • ran the above against all of SAS (467 exposures)
      • wrote a script to convert FITS to csv then cat into one-csv-per-exposure files
    • added to the ippToPsps section of the PSPS test plan
  • IPP
    • czartool:
      • labels in table are now listed by order of priority as set in gpc1
      • integrated plots into czartool:
        • can view histogram of processed/pending/faults for all stages for a given label, or all labels
        • can view time series for processed/pending/faults for a given stage and label, including all labels
        • can view time series of all stages for all labels
        • fixed reverts so that switching them on or off is reflected immediately on page
  • czar on friday: learned about cleanup from Chris
  • IfA morning coffee all week....

Bill Sweeney

  • Made the finishing touches on the two new stages of the IPP pipeline: chip_bg and warp_bg This fork in the pipeline was created by Paul Price just before he left. It removes the effects of the background subtraction and pattern correction for the chip stage and then produces warps. Data without background correction is required for the M31 project. Paul did a great job, Bill just needed to make some minor refinements to get it fully integrated into IPP processing.
  • Started working on changes to the dvomerge program to support combination of dvo catalogs which have different secfilt definitions.
  • Queued and shepherded the backlogged MD08 data through the pipeline.
  • Updated the released 3PI DVO catalog to use the measured zero points. This involved finding the input smf files extracting the zero points and running the program setphot.
  • 2 days as processing czar. Bill finds this a very stressful activity. (Large numbers of errors freak him out).
  • Fixed various minor bugs.
  • Changed the postage stamp server to avoid trying to queue updates from old runs that are incompatible with the current ipp.

Chris Waters