IPP Progress Report for the week 2010.04.19 - 2010.04.23

(Up to IPP Progress Reports)

Eugene Magnier

We have largely been working to iron out wrinkles and issues as they come up, with special effort on the overall detection efficiency. We've been interacting a lot in the past 2-3 weeks with the MOPS team, and finding a number of interesting issues. Paul has been worrying about the most important of these: an error in the covariance calculation which was preventing the diff detection from reaching as deep as it should have been. With that error identified and corrected (as well as a related error in warp due to the change of pixel scale), he has been trying to understand the impact of the fix on the magic masking. We will need to iterate with Paul Sydney for final validation, but we can test the magic impact with the magic verification data set used before the start of Demo month. It may be necessary to tweak the magic threshold to reach the right balance between excess streaks and catching all real ones.

I spent a couple of days this week on the IfA TAC, which takes a lot of time (and mental energy). After that I've been working on the extended source photometry. This has been dormant for some months, but I've brought it back to life. I have updated the output functions to save the radial surface brightness measurements, with options to make it match the SDSS analysis. I am running tests on SDSS image to compare the psphot measurements with SDSS measurements of the same pixels. After these tests are done, it should not be hard to turn on this analysis in the 5-way image mode.

Heather Flewelling

  • working on addstar/mergestar
  • helped Mike Lum with IPP/single chip
  • copied warp files for Jim
  • queued addstar runs of ThreePi.DM.20100401 data (with new dvodb name)

Bill Giebink

  • Dropped off Fusion at Ford Dealer to have noise checked; picked it up
  • Dyed PMA cover cloth
  • Prepared cover attachments by gluing magnets to bars and applying flocking
  • Arranged installation of UPS for TCC (electrician, parts, etc.)

Roy Henderson

  • PSPS
    • Worked a lot with Sue, specifically the reload of real data and working through bugs in flux calibration code
    • DXLayer not remembering previously loaded batches: fixed (temporarily)
    • Created new datastore product for JHU testing for Maria. Loaded with real data minus those with invalid flux values (see below).
  • IPP
    • Wrote 'czartool' script to help with duties as...
    • Processing czar on Friday
    • Created SAS data for loading to PSPS
    • Investigated flux issue: now discarding detections where flux==0 and flux==NULL

Paul Price

  • Investigating impact of fix to diff variance/covariance:
    • Magic on a single, skycell of synthetic data: very small increase in number of streaks, 98 --> 100 (large number presumably due to lots of unsubtracted fake transient sources in diff), though only a 0.18 mag increase in depth.
    • Verified magic calculation of variance by reproducing thresholded image.
    • Magic on a single skycell produces more clusters and streaks, but not unreasonable (triggering on artifacts: ghost, bad cell bottom).
    • Magic on a single exposure produces more streaks (5 --> 37), most due to pattern row noise, cell edges, ghosts. Some were for no apparent reason.
    • Fill factor is difficult to measure due to small number statistics, and in any case it's relative to MOPS' understanding of the area of an exposure which should be larger than the actual area of GPC1 on the sky.
    • Verified from error distribution of fake asteroid detections that the error in the noise model was causing detections to be treated as lower significance.
    • Comparison of psphot detection efficiency estimates of the limiting magnitude with empirical measurements of the noise in the image reveal problems in the noise model.
    • Corrected diff noise model using simulated data. Magnitude limit of diff is now 0.4 mag brighter than warp. Still worried about warp noise model.
    • Fixed warp noise model: covariance needs to be resampled when changing the plate scale.
    • Magic on the same exposure as before now produces 9 streaks, all are due to ghosts except 1 burn
    • Asteroid limiting magnitude is now ~ 21.7 mag, compared to ~ 22 mag before the warp fix (which was unrealistic).
  • Processing czar
    • magic_id = 25663, skycell_id = skycell.1353.072 has corrupted input, cannot recreate since warps unavailable; set to drop.
    • Manually recreated input for magic 40212, skycell.1547.025.
    • Changed scripts to add date of processing to log.
    • MD pre-DM stacks completed. 36/7687 (0.5%) failed because most of the inputs were magicked.
  • Queued lots of cleanups
  • Added function PS::IPP::Metadata::Config->parse_list to quickly parse a list of metadatas; applied to ipp_cleanup.pl
  • Fixed photometry in ppSub: analysis metadata was getting clobbered.

Bill Sweeney

  • continued to work on triggering automated image recreation (update) through the postage stamp server. The code is integrated into ipp trunk but is not yet deployed. Still working on various workflow issues in the update process.
  • two days lost while working as processing czar
  • one day out due to injury

Chris Waters

  • Fixed bug in nightly science automation that was overqueueing stacks.
  • More work on detectability server. Still needs to be integrated into PStamp interface.
  • Fixed bug which was setting the data_group of diffs and stacks to the label, and not the data_group of the input data.
  • Switched Sweetspot processing to not make pairwise diffs. Still need to write tool to periodically create new reference stacks.
  • Include SVN revisions and mask fraction in the database. Needs to be finished and tested before being integrated into the trunk.