IPP Progress Report for the week 2011.01.03 - 2011.01.07

(Up to IPP Progress Reports)

Eugene Magnier

Over the past few weeks, we have been working on issues outstanding before we start the Grand Reprocessing. In addition, I have been working on the psphotStack analysis needed to generate the stack photometry & morphology data products needed by PSPS. Progress towards GR has been much slower than hoped, partly due to illness among the team (and general holiday drag), but also due to challenges in the work we are trying to do.

The major impediment at the moment on the GR work is in finalizing the static masks based on the detrend and non-linearity modifications of the last couple months. The IPP team have created an initial set of static masks, and over the holiday period, Heather has been leading the effort to test those masks. At this point, the testing has run into some difficulty because of the enhanced magic rates from the current code base. It is clear that we need to solve that issue before we can finalize the masks.

Before the Holidays, I was concentrating on the PSPS output issues. I finished modifications to psphotStack so that it performs the full set of photometry and morphology analyses on the input images. It then generates a series of PSF-matched image sets with multiple target PSFs, and performs the radial-aperture photometry analysis on these PSF-matched images. I provided Roy with examples of these data products from simulated data.

Last week, I created a set of tests scripts to verify the image processing outputs from simulated raw data through the stack process. Using this test suite, I was able to track down the bug causing the detection efficiency error. It appears that the 7/15 modification to the variance and covariance scaling missed an important factor in the detection efficiency analysis (which was included in the actual detection analysis). The implication of the bug is that the reported detection limits values since 8/23 (when the bug took effect) have been in error by nearly 2 magnitudes (ie, it is the reported limit that is in error; the actual detection efficiency was not impacted). In exploring this bug, I also discovered a bug that let psphot drop moderately near neighbors to brighter stars. This bug results in reduced efficiency as crowding increases.

I have since been pursuing the magic error introduced in the 8/23 release which results in too many magic streaks. I suspect that this is similar to the covariance scaling error mentioned above, but I have not yet been able to prove it.

Serge Chastel

  • back on Monday instead of Thursday;
  • czar on Thursday and Friday for IPP;
  • czar all week for MOPS;
  • trying to install mysql-5.1 on ippc00;
  • attempting to write down a synthesis about IPP current implementation

Heather Flewelling

  • various amounts of kicking burntool/registration to make it continue.
  • liason - emailed Matt Holman about skyprobe

Roy Henderson

  • PSPS
    • Loading
      • 3PI loading hampered, first by samba share issues, then by datastore issue (files on wrong machine after Db change). rsync'd to fix it
      • DXLayer kept stopping. Lots of restarting from home over the weekend
    • PSVO
      • implemented 'summary' page, showing totals per survey and filter, with buttons to plot any subset. More can be added
      • added functionality to change PSPS user
      • implement two inner classes in PSPSCommunicator to handle quick and slow jobs
      • can run and cancel slow jobs: changes to TableModel to load VOTable format exported from DRL
      • ran into weird things when querying detections using fGetNearbyObjEq() function. Reported to Graypan
      • added convenience option to grab detections for a given region using above function
      • lots of cosmetic changes
  • Sick on Friday

Bill Sweeney

  • spent lots of time looking for disk space on the cluster. Found significant portion from October - November that was missed by automatic cleanup
  • Looked into problem reported by a postage stamp user who asserted that requests for stamps from a reported detection returned no overlap. It turned out to be user error. The coordinates were outside the exposure.
  • Two days as acting processing czar.
  • Investigated by stack-stack diffs weren't getting queued. Once we cleared out the backlog MD03 still wasn't getting queued. The problem was an incorrect template entry in the survey book.
  • Lost several hours over two days due to dental work.
  • Investigated and fixed some problems with registration.
  • Investigated errors in distribution clean up.
  • Fixed some problems on the data store caused by incomplete clean up.
  • recovered from vacation email back log.
  • started thinking about generating "unmagicked" postage stamps from destreaked images by saving the excised pixels and putting them back for authorized users.

Chris Waters