IPP Progress Report for the week 20090927 - 2009.10.02

(Up to IPP Progress Reports)

Eugene Magnier

This week, I've spent much of my time making sure my recent work on psphot could be merged into the main SVN trunk and not break other things. I ran a good-sized test of the photometry using one of the r-band flat-field correction dithers. I needed to address a number of small bugs caught in that processs. Eventually, the full analysis completed without errors, and the results look very encouraging. The flat-field correction determined from that analysis is quite similar to the pattern observed by Eddie Schlafly in the difference between PS1 and SDSS photometry from 3pi data (reported at Harvard). Since the merge was complete, I have been running the complete set of filters of the flat-field correction data taken in late June 2009. A little work is needed to update the flat-field correction system since we changed the was 'addstar' is run in the pipeline. I expect that to be done by Monday, so that we can start processing with the improvements by Tuesday.

Heather Flewelling

  • finished processing MD09.20090925 and M31.20090925
  • still processing MD10.20090925
  • burntooled jun, jul, oct, nov, dec. of 2008. We have burntooled Jun 01, 2008 through now.
  • burntooled the previous nights data each morning. Working on a script to automate that.
  • continued md5sum checks on rawimage files (to verify that all copies of a raw file have the same md5sum, save the md5sums, and to fix the ones that do not).
    • found 63 corrupted raw image files (not this week, but since I've started)
    • all but 4 were fixed. Of those 4, 1 is lost, and the other 3 are missing only 1 or 2 cells.
    • wrote a script to help automate the repair of these files. Only works for a subset of the corrupt files.
  • found 80,000 files that had both copies on the same host. This has been fixed.

Bill Giebink

  • Continued working on psLib unit tests.
  • Spent a day at MHPCC filling PSPS cabinets with machines.

Paul Price

  • Stacking (branches/pap)
    • Mottling removed when using new PSF limits to throw out bad PSFs from envelope calculation
    • Testing with MD08.4940i.20090930, MD08.4941r.20090930, MD08.4956z.20090930, MD08.4979g.20090930
    • Many stacks (15%) failing because "No fake sources are suitable for PSF fitting", i.e., all PSFs rejected
    • Remove diffraction spikes, star cores, ghosts from the "bad" list, put in the "advisory" list, so now diffraction spikes don't get NANed in warps.
    • Enforce strict model limits at warp stage, since these are used to input to stack
    • Fixed model radius for PS1_V1 for PAR_7=0
    • Merged into trunk (r25738)
  • MD08 stack processing
    • Low rate of failures: 0/19680 chip, 1/329 camera, 3/34403 warp
    • Stacks underway
  • Created transients@…
  • Fixed concepts copying in ppstamp

Bill Sweeney

  • The major piece of this was implementing the 'byskycell' lookup method for the Postage Stamp Server. This feature enables users to select the images of interest for postage stamps based on a skycell identifier. This is a very efficient method for the system and is pretty easy for users to use. Especially those interested in the medium deep fields.
  • To support this effort he modified the ippTools to make it easy to select processed images at each stage of the pipeline based on the observation date, filter, and label.
  • Updated the his program for creating requests and started including it directly in the ipp build. The next step is finding an efficient way to find images based only on the coordinates.
  • Unfortunately part of last weekend and Monday was wasted (Note from EAM: the work was needed, if frustrating) due to a problem where the DataStore? returned corrupted data to the distribution client. After some difficult debugging it was determined to be a problem in one of the open source perl modules that we use as part of the IPP. A new version seems to avoid the problem.

Chris Waters

  • Cleanup: More chip/warp cleanups. Tests of detrend cleanup on non-simtest data. Identified even more data to be cleaned.
  • Burntool: Identified that burntool is now not expiring trails as expected. This can lead to oversubtractions on cells that are too noisy, as burntool believes it has found a reasonable fit. Updated ppImage.config BURNTOOL.TRAILS value to 0x07, which adds all pixels that burntool has altered to the mask as SUSPECT.
  • Nebulous: Added table "deleted" to active Nebulous database to allow the deletion of storage objects that have instances on inactive hosts. This now functions properly for so_ids that only have a single missing instance, but not for more (unclear why this is). Working on neb-replicate bug which is not properly randomizing hosts. Modified "dsget" to work around this bug in the meantime, preventing every backup copy from being put on ipp049.
  • y-band Fringe: Created a new fringe. Identified that the fringe isn't being scaled correctly for most chips, likely due to a quantization bug (fixed with Paul Price's help) and a sky subtraction bug (solution in progress).
  • Flat verification: Ran flat verify runs, looking into possible causes of residual structure.