IPP Progress Report for the week 2010.04.12 - 2010.04.16

(Up to IPP Progress Reports)

Eugene Magnier

I spent Monday and Tuesday as processing czar, which gave me the opportunity to track down a few outstanding bugs and work on a bit of optimisation. I discovered that the footprint / peak culling process in psphot can sometimes become extremely slow, particularly if a footprint contains a large number of peaks. The culling rate seemed to be much slower than O(N). Part of this was due to the very large number of memory allocation operations for each tested peak, so I reworked the memory management in that function to allocate up-front the necessary support structures. In working out this problem, I also discovered that recent API modifications to psphot caused us to drop the sky error information, which in turn meant that some low-surface-brightness areas could be broken into too many peaks (due to the background sky error not being included in the calculation). Both of these fixes have substantially improved the speed to peak culling -- in a test set, I had been finding 20-30% of psphot analysis runs spending 100+ seconds on just this step; after the fixes, only 1-2% continued to spend substantial compute time on this step, with the rest down in the 2-3 second range.

I have also been working on psphot for extended sources to be sure we can generate matched aperture photometry for extended sources soon (rather than waiting for all of the other extended source measurements to be tested).

Heather Flewelling

  • processing czar 04-12-2010
  • Taught Roy about processing tasks as they came up.
  • sorted out ISP summitcopy - datastore issues. Erik Small made ISP datastore behave more like gpc1, Bill figured out how to get around problems like this in the future.
  • ISP registration - ISP images now register to camera stage, with camera label (only the OBJECT ones, ones which could potentially be processable), ISP registration merged in with gpc1
  • created ISP wiki of everything I could think of about ISP processing:
  • stack test of different plate scale (0.0258): label is MD04.20100415.scaletest.0258 , consists of 8 MD04 z band images from
    1. These are stuck due to failed warps (I need to sort this out). TESS_ID is MD04.V1
  • answered UIP's questions about processing ISP data - this is still in progress
  • added tess directories to ifaps1 (to make it more useable)
  • answered ifaps1 user questions throughout the week
  • Helped prepare for IFA Open House
  • manned PanSTARRS booth at IFA Open House
  • Helped Mike Lum (VYSOS data reduction) - this is still in progress

Bill Giebink

  • Prepped PS2 pier for ground penetrating radar team
  • Swapped ipp037 CPUs for new ones
  • Worked with radar team to map pier

Roy Henderson

  • PSPS
    • Installed the DXLayer on Maui cluster (a few issues, nothing major)
    • Created and loaded 1000 batches of real data. Handful of errors: see below
  • IPP
    • Added to processing czar wiki page
    • Improvements to error handling in ippToPsps
    • Changes to arguments for ippToPsps run-script (can run for a single exposure rather than whole label)
    • PSPS survey 'type' now taken from the gpc1 Db (using 'dist_group')
    • Bugs uncovered from PSPS loading:
      • Empty ImageMeta tables created if chip extension missing from input FITS: fixed
      • NULL values in smf files going into FITS output and getting blocked by loader: fixed
      • 0.00 values for flux getting blocked by loader: investigating

Paul Price

  • MOPS detection woes:
    • Inserted fake asteroids from Larry into chip stage products, manually ran through warp and diff, and shipped results. Based on detection efficiency measurement in psphot, only about 35% of asteroids (about 100 asteroids) from the input set are bright enough for detection.
    • Found SDSS report of 0.4 mag bias in asteroid photometry catalogues
    • MOPS photometry corrections don't appear to correspond to asteroid colours measured by SDSS
    • Re-running fake asteroid data with larger (10x) set of inputs from Larry
    • Depth of diff photometry appears to be g ~ 21 mag --> V ~ 20.6 mag. Chip is going to g ~ 21.9 mag, so should be doing better.
    • Shallow depth appears to be resolved with more careful combination of variance+covariance at diff stage. Recovered around 0.8 mag. Sent detailed report to ps-ipp-users and ps-mops-dev. Requires further work before merging into production system, especially worried about impact on Magic
    • Determined correct photometric corrections for MOPS: 1 mag offset in g appears to be ~0.4 mag catalogue bias + 0.5 mag photometric correction error.
  • Discovered a bug in "stacktool -definebyquery": wasn't grouping by tess_id. Inputs to MD.preDM.20100409 stacks have heterogeneous tess_ids, but this hasn't affected the stacks produced because "stacktool -inputskyfile" ensures the correct tessellation is used. Set failures (from insufficient inputs with the correct tess_id) to 'drop'.

Bill Sweeney

  • Spent the week teaching the postage stamp server how to queue and monitor update runs. The system can now queue updates for the chip and warp stages. Still to do are stack and diff (which is always a bear due to the diff stage's complexity. There are still a few issues to work out before deployment.

Chris Waters

  • Out for two days due to illness.
  • Finished first draft of detectability server. Waiting on the PStamp interface to stabilize before committing the final version.
  • Started work on matching IPP parameters to SDSS star/galaxy classifications for SAS.