IPP Progress Report for the week 2009.09.13 - 2009.09.18

(Up to IPP Progress Reports)

Eugene Magnier

We are finally pushing data out to the consortium again. We distributed MD09 data, and should be able to distribute M31 images by Monday. The burntool processing has been running slowly, but Heather & I made some modifications that allows the burntool to be run without so much hand-holding -- I had failed to increment the burntool version field, so Heather needed to watch carefully for processing errors. Heather & Chris have also been working on issues related to cluster data management. After Heather found a raw image file which had been corrupted (either in copy or in situ on disk), she has been running a script to check the md5sum for the raw images. In the cases so far, the corrupted images had a second uncorrupted image elsewhere in the cluster (due to the Nebulous duplications). Most of the examples we've seen appear to have been caused by a period where two summit copy processes were running at the same time, so hopefully there are not any unrecoverably damaged files. Bill S updated the schema and summit copy process to allow us to track compressed md5sum and file size (previously we were only tracking this for the uncompressed values reported by the summit). Chris has been making good progress on running the cleanup analysis. This is getting critical since many of our nodes are at 95% full, and our total usage is roughly 70% of capacity. There are less than 100k raw exposures, which should account for about 270TB, but we are using 560TB of our storage. The remainder is taken by the many versions of data analysis tests and distributions, so cleaning those old analysis runs will get us back most of our storage space.

In terms of development, this week I've continued to work on psphot. I've fixed the code that chooses the pixels used in the extended source analysis, so that it now includes all of the pixels in the 'footprint'. I've also made some improvements to the way extended source fitting failures are caught and handled, and I've gotten the radial profiles and Petrosian analysis working well within the full psphot context. I have also been looking at the analysis of the source sizes, and have fixed up the identification of PSF vs Extended Sources vs Cosmic Rays to make better use of the psf vs aperture magnitudes and the moments of sources. I have also improved the criterion by which the psf model chooses the order of the 2D variation in the PSF model. All of these are needed to make the photometry reliable under varying seeing conditions and to perform reliable extended source analyses.

Heather Flewelling

  • Processed MD09.20090910 through magic, sent email to harvard with link to data
  • Processed M31.20090917 through magic.
  • Processed MD10.20090915 through chip stage, half fail at cam stage (investigating, some psastro, some ppstats related. ppstats related to burntool state)
  • Processed M31.2008.20090918 through chip stage, half fail at cam stage (investigating...)
  • burntooled all dates between 06-01-09 and 09-16-09
  • edited wiki to keep track of which data I'm processing:
  • checked md5sum on 568794 gpc1 images, found 6 corrupted images, removed them and replicated the good copy.

Bill Giebink

  • Continued to identify and correct errors in psLib unit tests. Basic procedure is to:
    • identify failing tests,
    • determine if the test is still a valid measure of the underlying code, and,
    • if the test is not measuring anything useful then it's eliminated or replaced, also,
    • failures due to changed function parameters, different error checking, etc. are fixed.
  • In addtion, commented upon test deficiencies are being remedied where feasible.

Paul Price

  • Detection efficiency: merged branches/pap into trunk, r25383.
  • PS1SC:
    • Set up nightlogs@… list
    • Organising UK tour
    • Retrieved and sent dravg archive
  • Stacking:
    • Everything was getting rejected. Turned off "additional variance" in pmStack, and get decent rejections now. This is not well understood yet, but may be connected with a change made a month ago removing the "variance factor" terms, or renormalising the image subtraction chi2.
    • Can identify bad images using the iq statistics calculated in psphotImageQuality.c: iq_m2, iq_m2_uq-iq_m2_lq, iq_m3 and iq_m4 are useful quantities.
    • Sent Nigel and Peter a dump of ThreePi data from June/July that should be useful for determining good cuts on these quantities.
    • High-frequency structure in kernels caused by minimum ISIS width = 1 (FWHM, not sigma!): fixed ppSub recipe
    • Increased rejection threshold (COMBINE.REJ) to 2.5 from 2.0 standard deviations.
    • Using a given analytic PSF instead of that produced by pmPSFEnvelope produces reasonable results for the stack. Attempting to correlate resultant convolution kernels with iq statistics.
  • Fixed fuzz of FITS images so that expectation values are preserved (trunk@25439), after help from http://fits.gsfc.nasa.gov/registry/tilecompression.html

Bill Sweeney

  • Split the database queries for magic de-streak processing. The old way was taking too long and thus causing jobs to run multiple times.
  • Modified summit copy and registration scripts/tools/tables to save the md5sum and size of the compressed raw images. This will allow us to detect potential data corruption problems.
  • spent some time analyzing the results of the MD09 streak detection.
  • added new states to destreak processing to automate restoring destreaked images and to censor destreaked files that are found to contain unmasked streaks
  • restructured the queries for distribution bundling. The old way was taking too long and thus causing multiple jobs to run for a given component
  • worked with Johannes at MPG to set up a data store for submitting postage stamp requests at that site.
  • Fixed some bugs in postage stamp extraction triggered by requests submitted by Queen's University.
  • Built distribution bundles for the reprocessed and new MD09 exposures. Posted them to the ipp datastore.

Chris Waters

  • Began pushing old data through the chip and warp cleanup process.
  • Fixed resulting bugs in the cleanup code that did not arise during simtest tests.
  • Added burntool_state column to rawImfile table, and rewrote ipp_apply_burntool.pl and chip_imfile.pl to handle the new burntool versioning system.
  • Still working on getting register_imfile.pl and ppStats to successfully identify if the summit has run burntool, and if so, set burntool_state correctly in the database.
  • Identified new bug in burntool when run in a two-step process.