IPP Progress Report for the week 2011.03.21 - 2011.03.25

(Up to IPP Progress Reports)

Eugene Magnier

I examined the STS astrometry problems and found a solution which is appropriate for this particular field, if not in general. I chose psastro recipe values that allow a selection of reference stars down to 17.5 mags (instrumental mags of about -11.0), where stars are no longer affected by the astrometric error. I also updated the code so that the number of stars used in the cross-match can be a limited subset of the full set of stars for both the reference data and image being calibrated. This allows the calibration to get a cross match in a reasonable amount of time based on the brighter stars, but to use the fainter stars for the full fit.

I ran as set of MD04 stacks for PSPS and as a demo of the processing to be used in the Grand Reprocessing. This set includes 8-ways stacks for each filter (grizy) and an i-band stack using the same input images as the refstack currently being examined by the Durham group. I also ran psphotStack on the 8-ways stacks for PSPS. I needed to work through a couple of bugs caught in this processing, the most interesting of which resulted in an improvement to the background sampling. Until recently, we had been random-sampling pixels in the background supercell regions to avoid doing the statistics on all pixels in the image (slow). MWV pointed out that the way this was implemented, in cases where the masking fraction was high, could result in significant double or triple sampling of the same pixels. We worked out an alternative that avoided this problem. However, first implementation added 2-3 minutes to the processing time of each chip, which is unacceptable. MWV suggested and I implemented an alternative strategy based on the Fisher-Yates algorithm to select a random sequence with no repeats. This new implementation runs as fast as the old random-sampling method, but is not subject to the multi-sampling problem.

I worked out the confusion that we had regarding tests of the magic streak rates using the current trunk code. There were a couple of configuration errors coupling to bugs that were causing problems: the DETREND_CONSTRAINTS element was missing from the test recipe set, and this resulted in the wrong flats being selected for the tests. In addition, the new dark had multiple iterations 'accepted'; this, coupled with a bug in 'detselect' resulted in an inconsistent selection of the dark master. I fixed the code to prevent both errors, and also fixed the configuration so that the correct flat and darks were selected. I also clarified the rules for selection of the 'PATTERN.ROW' and 'PATTERN.CELL' options. I re-ran the 10 y-band images we had been using as a demo test, and showed that these were processed with the correct flat and dark, and the correct options for PATTERN.ROW,CELL, and the correct choice for non-linearity correction. This test shows that there are a small number of cells unmasked in the new mask the which were previously masked and which raise magic triggers. We need to update the new mask to address these cells (or portions of cells) before we can start reprocessing in earnest.

Finally, I spent much of the week working on the system zero points for the 3pi reference photometry database. I had earlier built an MD dvo database using all of the data taken to date. My goal is to use the relative photometry analysis of that data to generate the system zero point history (including any long-term secular trends and identification of nights with poor overall transparency -- cloudy nights). I had previosly run the program 'relphot' on the the database, to determine the per-exposure zero point offsets. This past week, I ran 'uniphot' on the database, which determines region-specific offsets needed to bring all of the exposures into a common system in which a single zero point applies for short time scales (say 1 day or 1 hour). After some updates to the code, this succeeded, but I was concerned about the amplitude of the scatter in the nightly zero points.

I examined the data in detail and discovered 2 issues that were causing higher than expected zero point variations. First, the relative photometry code was allowing too many galaxies to be included i the analysis. We are using the PSF photometry to constrain the relative zero points; the PSF photometry fit for galaxies is sensitive to the image quality: as the seeing changes from image to image, the galaxies appear to get brighter or fainter. I adjusted relphot to exclude galaxies, and this improved the scatter somewhat, but not enough. The second problem was that I was too generous in selecting input images when I built the database: every version of reprocessing of the same exposure was included, and these duplicates were causing significant confusion. I rebuilt the MD reference databases using only the nightly science processing, and this has helped substantially.

Serge Chastel

Heather Flewelling

  • started addstar merging (merging of all the minidvodbs). I was running it as heather, but had to stop it to investigate magictest (lots of building/rebuilding of code). Addstar does work though.
  • magictest - lots of investigations of why magic numbers are bad between different builds - found that some of the changes checked in caused skycells to have bad quality, ippconfig (pattern.*) to not work, and it was picking up random darks for each exposure.
  • answered questions for Bence and John.
  • transferred files for Niall.
  • vacation thursday, state holiday friday

Roy Henderson

  • PSPS
    • helped investigate with Sue why MDF frames have appeared in 3PI catalog
    • ippToPsps
      • finally finished refactoring all code to use new Fits class. Much cleaner now.
      • made a new class to encapsulate the initialization data, i.e. a clean OO interface to config data such as filterIDs, surveyIDs etc
      • new FitsGenerator class to encapsulate code for generating and populating FITS files
      • new Logger class that can write to stdout or log file, complete with debug, info and error levels. Also handles code timing.
      • added unit tests for stack batches and init batches, so a complete set.
  • IPP
    • fixed issue with burntool plots showing 0 before midnight - just a UTC/HST issue
  • Other
    • Responded to more Hungarian data access requests

Bill Sweeney

  • By request of PanPlanets? (STS) team, created a new tessellation. This required enhancing the skycells tool to allow multiple local tessellations to be combined. The resulting tessellation matches the way the observations are made. Three distinct partially overlapping boresites.
  • Ran some tests with the new STS tessellation. Haven't analyzed them yet.
  • investigated why not all of the ThreePi? rerun data made it through difference processing. It turned out that there were some errors in one of my scripts.
  • Made y band MD06 template stack.
  • researched some questions about detection efficiency from Bertrand.
  • one day vacation, 1 day holiday

Chris Waters

  • Finished testing and enable --continue mode in burntool to help catch up on registration if we fall behind.
  • Identify a set of input warps for John Tonry to allow a comparison of IPP stacks vs. his independent stacking method. Have not finished comparisons of these stacks yet.
  • Added a summit_id key to the summitExp, summitImfile, pzDownloadExp, pzDownloadImfile, newExp tables in the GPC1 database to speed the joins between these tables that are now required for the on-the-fly burntool and science processing we're now doing. Fixed miscellaneous bugs in this implementation as well.
  • Examined STS exposures that had astrometry shifts for bright stars, and compared the stars to the burntool tables to see if damage due to burntool trail removal can explain the shifts. This does not appear to be the case, as astrometry from the same exposures without the burntool correction applied show the same sort of shifts.