IPP Progress Report for the week 2009.10.05 - 2009.10.09

(Up to IPP Progress Reports)

Eugene Magnier

This week, my goal was to start the re-processing of data taken in Run 3 & Run 4 with the upgrades to stacking, photometry, including the flat-field correction & y-band fringe corrections. Although we have had enough tests of the upgraded processing, work has been going a bit slower than hoped on the two detrend issues (flat-field and y-band fringes). While these were being worked on, we have been running more processing of data that had been collected just before the end of Run 4. This included a complete sweep through the 'SVS' data (the MOPS ecliptic-plane r-band data), with detections shipped to MOPS Tuesday and Wednesday. We also have processed the MD and M31 data taken in the last week before the shutdown.

On my side, I've been working on the flat-field correction analysis. Last weekend, I had processed the images taken for the flat-field correction. However, over the summer we re-organized part of the pipeline to better handle the DVO ingest process, defining a new 'addstar' analysis stage. I needed to make some changes to the flat-field correction analysis system to interact with this new addstar system, and I also found some errors in the implementation of the addstar stages. I also updated the flat-field correction system to distinguish between an analysis to generate a flat-field correction and one which tests the flat-field correction (these have slightly different behavior on completion). We now have flat-field corrections for gizy, and corresponding corrected flats for Run 3 & Run 4, and I have run the tests using those flats on the input data sets (still running as of Sunday 4pm). The r-band data had a minor logic error in the analysis database queries, but is running and should be done on Monday. I have also performed the flattest analysis on the input data, in which we apply the corrected flat to the data and measure the residual correction. I have only performed this measurement on the data used to construct the flat-field correction, so this is not yet a complete test of the system. However, it shows the internal consistency of the photometry. The results are extremely promising: I fit a Gaussian to the set of flat-field correction values, and the sigmas of those Gaussian fits are: (g,i,z,y) = (2.2, 1.8, 1.7, 2.5) milli-mags.

Heather Flewelling

  • continued md5sum checking of raw images, repaired a total of 182 images to date. More scripts written to make this easier to manage.
  • started to fill md5sums into the gpc1 database. waiting on 3 hosts to finish checks to continue the rest (then I will fill the gaps which are missing because of NFS errors, nebulous errors, or unresolved because of inconsistent md5sums)
  • helped Natalia get started with IPP software and dvo
  • processed MD01.20091007 and MD02.20091007 through to magic
  • processed SVS.20091005
  • finished processing MD10.20090925
  • destreaked MD09.20090925 and M31.20090925

Bill Giebink

(on vacation / at JHU)

Paul Price

  • MD08 stack processing
    • Low rate of failures: 0/19680 chip, 1/329 camera, 3/34403 warp, 15/3912 stack
    • Tested a few following Gene's merge: no problem
  • Organising UK tour
  • Stacking (branches/pap)
    • Produce non-convolved stack using convolved stack's rejection mask
    • Works, ready for merge

Bill Sweeney

Bill spent most of the week working on several issues relating to cleanup/update system, which is becoming important because we have begun deleting processed data.

  • Modified the postage stamp server to detect and queue update runs for requests where the images have been cleaned up.
  • wrote tasks and scripts to implement cleanup of the backup files saved in the magic destreaking process
  • magicdstool changes to support more flexible run state changes
  • implemented changes to the ippTools to support lookups of images by celestial coordinates
  • proposed and starting working on significant changes to the ipp database schema to support identifying data for update and distribution more cleanly.


  • debugged a couple of problems found by postage stamp users
  • built distribution bundles for latest processed data

Chris Waters

  • Cleanup: Ran cleanup on chip, warp, stack, and diff stages for a large list of old labels. This freed up 33 TB, or about 4% of the total cluster disk space. Updated and corrected bugs in ippTools and ipp_cleanup.pl to properly handle the cleanup of old data. Checked why certain labels do not clean successfully, and discovered that the majority are due to Nebulous issues (to be resolved next week).
  • Fringe: Investigated the way the fringe scale is calculated by looking at the samples. Switched to one background sample per cell (down from 9) and saw some improvement. Checking the chips that still do not scale correctly indicates that the samples that should be influencing the fit are being clipped as outliers. Rewriting this clipping should correctly scale the fringes.
  • Flat: Constructed remaining flat-verify runs for domeflats. The residual patterns likely correlate with dome screen tension (and do not correlate with anything other than date). Processed skyflat exposures with the domeflat, revealing a static pattern (due to the screen tension pattern not being averaged out). Working on constructing a skyflat for comparison.