IPP Progress Report for the week 2010.08.16 - 2010.08.20

(Up to IPP Progress Reports)

Eugene Magnier

  • IPP Code Updates

We are in the process of switching the operational IPP code to a new software release. The new tag is ipp-20100823.

This update includes a number of features and fixes to issues which have been in progress for some time. Here is a summary list of the main salient points:

  1. a new "staticsky" distribution component : this is the method by which we will be distributing the results from the full photometry analysis of the static sky images.
  2. Full extended source analysis! psphot is now able to produce acceptable measurements of all of the extended source parameters on our list.
    1. for all sources, we are now including Kron parameters (Kron flux and error in an aperture 2.5 times the first radial moment (R1), plus the flux within 1 times R and 4 times R1, as well as the value of R1.
    2. in the static sky analysis, for all sources we now report the radial aperture measurements (this was available in the earlier MD04 example set).
    3. Petrosian and elliptical surface brightness profiles for a subset of extended sources (S/N above some cut off to be defined, probably based on galactic latitude)
    4. PSF-convolved model fits for a subset of extended sources for Exponential, DeVaucouleur?, and Sersic models.
  3. fixed the forced photometry non-negative flux bug
  4. added some other new stats:
    1. for PSF fits, we now have "psfWeightNotPoor" and "psfWeightNotBad" : the first gives the fraction of psf-weighted unmasked pixels considering any mask bits (except the internal 'mark' bit), while the former considers only 'bad' mask bits -- these are written to QF_PSF and QF_PSF_PERFECT in the CMF / SMF files. This may help to exclude false sources in subtracted burn regions (SUSPECT mask bit) or sources too close to a mask edge (CONV.POOR mask bit).
    2. raw aperture magnitudes (in addition to the curve-of-growth corrected version) for PSF measurements.
    3. in diff images, we now have several additional statistics related to the proximity of a diff detection to an input image source: DIFF_R_P, SN_P : distance and S/N of nearest bright source in the plus image, DIFF_R_M,SN_M : same for the minus image.
    4. for PSF sources, we have added several higher order moments: MOMENTS_M3C,S (third moments cos & sin theta), and same for 4th moments.
  5. somewhat better CR / EXT tests, including use of Kron magnitude as test for source size (CR vs EXT) instead of PSF-based aperture. (though these are not yet reported as a probability).
  6. save the radial profile aperture sizes in the headers
  7. swap read order in dvomerge to loop over the available input tables and merge them with existing output tables
  8. allow dvomerge to operate on only a region of the sky
  9. update to dvomerge to allow different average photcodes in the inputs and the output database.
  10. some important issues in ppSub that were resulting is somewhat poor matches when the IQ difference between the two images was large:
    1. ensure masked pixels are NANed in output diff image
    2. define separate penalties for each image (based on their fwhm values)
    3. define separate apertures for each image for flux normalization
    4. choose aperture based on curve-of-growth (was based on fixed fraction of full aperture flux, and thus noisy)
    5. some fine tuning of the penalty factor (this still seems arbitrary, and results are somewhat sensitive to the right value)

Note that there are three new output formats that are needed to make all of these features part of the standard products:

  • for chip-image processing: PS1_V3
  • for staticsky processing: PS1_SV1
  • for diff processing: PS1_DV2

I would like to run with the old versions for a few days to check that there are no problems, and to give people a bit of a warning. we need to be particularly careful that the MOPS folks do not get torqued by the diff output changes.

  • Data Samples:

I am placing two example data set (simulated images) that illustrate the extended source analysis and the difference image analysis

rsync://pikake.ifa.hawaii.edu/psphot.extended.simtest : this directory contains a set of images with fake galaxies (a range of sersic parameters, convolved with a Gaussian seeing kernel) and psphot fits to those same images. (inputs are image.fake.NN.fits, outputs are image.fake.pcm.x2.*)

rsync://pikake.ifa.hawaii.edu/ppsub.samples.simtest : this directory contains an extensive set of images with different PSFs all differenced against each other with the newly-updated psphot. There are two classes of images: those with .fg in the name have Gaussian PSFs (range of sizes, axis ratios, and angles), while those with .fp use the PS1_V1 model (a wingier PSF meant to mimic PS1).

rsync://pikake.ifa.hawaii.edu/ppsub.samples.simtest.oldcode : the same set of input image pairs diff'ed using the previous version of ppSub.

We will attempt to run MD04 with the static sky analysis tomorrow for people to play with.

  • MD04 Photometry

I have done some investigation of the high systematic photometry errors in the MD04 stacks as reported by Nigel M. Some of the plots related to this study can be found here: https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/MD04_Stack_Photometry_20100819

I first checked that the PSFs of the stars had a consistent shape by overlaying the normalized PSFs, and confirmed that they are indeed consistent.

I next examined the impact of different PSF models, to check the impact of the choice of model shape (ie, is the photometry more consistent the more accurately the PSF is modelled). This did not have a huge impact on the photometry (the bright-end sag was still apparent), but the bright end photometric scatter is less in my analysis (PSF - AP) than in Nigel's (PS1 - SDSS).

I then ran the same tests using non-poisson weights. This addresses the bright end sag, and perhaps tightens the photometry a bit.

Serge Chastel

  • Branched ipp trunk for testing
  • Hudson installation for IPP framework. It will likely be used to automate testing activities on IPP and PSPS
  • Czar on Thursday. I implemented some new "views" for the faults.

Heather Flewelling

  • ippBlog
  • czar for 2 days
    • answered czar questions
  • mops
    • requested that fabrizio move his files off of panstarrs to /data/ipp003.0 (he was filling it)
  • ifaps1
    • transferred more files for zagursky (raw md files and darks)
    • answered too many questions (zagursky)
    • transferred ref stacks for zagursky
    • emailed chambers, watters about dvo
    • discovered our users guessed our mysql password -> asked gavin/bill to change it.
  • ppMerge
    • working on bug - ppMerge does not respect -Db VARIANCES TRUE - it is not passed into fitting routine
    • got lots of much appreciated help from Roy to learn how to debug this.
  • addstar

Roy Henderson

  • PSPS
    • generated test batches from new SAS DVO Db (see below) and converted to csv-per-frame files for Jim.
    • lots more loading of 3PI from new DVO Db. Requires a certain amount of hand-holding
    • testing of new SCR 'survey' with Sue on sandbox
    • testing then loading of SCR on Maui. Encountered big problem with duplicate batches (my fault). Sue to the rescue...
    • changes to DXLayer so that it doesn't quit when failing to contact datastore or has problems downloading
  • IPP
    • fixed bug in addstar_run.pl script (non-mini DVO path was wrong)
    • SAS DVO Db creation fun with Heather
    • added framework to addstar to filter detections based on user-supplied mask in config
    • ippToPsps
      • ODM polling script now deletes from datastore once a batch is successfully loaded to ODM and merge worthy
      • investigated failures from loading: fixed one bug when zero-row detection tables cause fault at load stage
    • czartool
      • now using stacked histograms for pending/processed/faults

Bill Sweeney

  • enhanced the distribution system to deal with the new "static sky" stage.
  • changed psphotStack to save the configuration in a file. This is needed for distribution processing
  • enhanced dvo to allow mextract to return the extrernal image id that corresponds to a given measurement. This will enable users to more easily create postage stamp requests based on data extracted from DVO.
  • changed distribution of background restored data to include the variance images from the original chip and warp runs. The M31 team needs these images.
  • wrote and executed scripts to bundle up the missing variance images for data distributed prior to the work discussed in the last item.
  • various other bug fixes.
  • looked into problem of streaky stacks reported by a user. The problem is a race condition in our processing. Destreak does not wait for nightly stacks to be completed.
  • 1 day vacation, 1 day state holiday

Chris Waters

  • Automation: added handlers to pass macros to pantasks at the end of each stage. This allows us to turn on and off destreak between chip queuing and stack finishing. This should fix the issue of destreaking warps before they have been stacked.
  • Linearity: worked on implementation of software linearity fix for ppImage. Added linearity correction detrend information to database.