IPP Progress Report for the week YYYY.MM.DD - YYYY.MM.DD

(Up to IPP Progress Reports)

Eugene Magnier

This week, we've been continuing to push Run 3 and new Run 4 data through the system, and have been working on some older data as well, including older M31 multicolor data from Fall 2008. Heather ran into some trouble with astrometry on these fields, and at the same time on some of the newer Medium Deep fields. I spent some time understanding what was happening. It turns out these are unrelated issues: the newer data was failing because out reference database was partially missing -- when it was moved to its current location, some of the directories were missed. It looks like the older data has a different problem, and I have not yet identified the cause (either the model is out of sync on a change in the pointing model, or the pointing was intrinsically so much worse in that period that the guess we are using is way off).

I've also been continuing the work on details in psphot. I reviewed the state of the aperture residual analysis and identified some low-level bugs that were causing small but significant scatter in the aperture residuals. I had intended to run the flat-field correction analysis with the updated psphot code over the weekend, but this work went slower than hoped. I should be able to finish the psphot tests early next week, which should still give me time to run the flat-field correction before the end of the week.

We are planning to start a large-scale re-analysis of much of the data collected in Run 3 & 4 when the telescope is down (starting ~ Oct 1). This re-run will include the flat-field correction (thus, more consistent photometry across the field), the recent improvements to the stacking analysis from Paul (see below), and the latest iteration on burntool, among other bug fixes.

Heather Flewelling

  • finished processing MD10.20090915 from last week
  • processed MD01.20090921
  • processed MD02.20090923
  • burntooled data from 04-01-09 through 05-31-09, reburntooled data between 6-01-09 and 9-13-09, but only dates where we weren't certain it was the latest burntool, burntooled PR dates for chris B, august 2008, and sept 2008 dates (except for a few otas on sept and oct dates).
  • burntooled the previous night's data, once it's been copied from the summit
  • investigating why M31.2008.20090918 won't process (fails in psastro, still investigating).
  • sent instructions to Bill G on how to process data for realtest
  • continued md5sum check on raw images, found several more corrupt raw images, including 2 with no good copies. 2827439 images checked, 16 found to be corrupted.
  • learned how to destreak images, destreaking MD02.090923

Bill Giebink

  • Continued to fix broken psLib tests.
  • Started "realtest".

Paul Price

  • Fixed skycell hosts to include all wave 1, 2 and 3 machines.
  • Turned off ignore of XY27 in chip_imfile.pl
  • Stacks:
    • Look fine when target PSF is set analytically: existing image rejection is working to exclude bad images
    • Updated matching chi2 to be relative to the fake image which is common to all images
    • Bad input images have PSFs with PAR_7 < 0 (divot in PSF model); these caused the target PSF to have a similar divot
    • Branched (branches/pap) for implementation of strict and loose limits on PSF parameters
    • pmModelClassSetLimits() sets limits on all models, pmModel.modelSetLimits() sets limits for single model
    • Inputs with PSFs that exceed the limits are ignored when generating the target PSF
    • Stacking seems more robust; more testing is needed before declaring victory, but this might finally be it!
  • Fixed writing of S64, U64 FITS tables, and possibly images (trunk@25478)
  • Implemented "difftool -advance" with accompanying pantasks task definition (trunk@25509)
  • Updated warp and stack stages so that a warp PSF is not required (e.g., for PR images, where we don't convolve to make the stack). Note: it's still the user's job to ensure PSWARP's PSF recipe parameter is compatible with PPSTACK's CONVOLVE recipe parameter.

Bill Sweeney

This week Bill finally was able to find the time to solve several little nagging problems and holes in the magic destreaking and distribution software.

  • Added a task destreak.advance and the supporting magictool and SQL code to split off finishing a magicDSRun from processing of the components. (The previous implementation was causing errors due to race conditions).
  • Simplified the database tables used to manage distribution.
  • Implemented disttool mode to make it easier to set up "destinations" and their "interests" without using direct mysql commands.
  • Implemented magictool mode to handle censoring (preventing the release of) exposures for which\ magic missed real satellite streaks.
  • worked with users to fix bugs in the postage stamp server
  • started implementing 'byskycell' image lookup for the postage stamp server.
  • several other things too small to mention here.

Since it's not yet completely automatic Bill also spent many hours managing destreaking and distribution bundling of recently processed M31, MD01, and MD10 data. The functionality discussed above greatly improved the automation for the future.

Chris Waters

  • Pushed old data through chip stage cleanup. Fixed bugs in cleanup related to old data not conforming to the current IPP status. Updated warp cleanup to properly ignore skycells that were not created during processing.
  • Reading through Nebulous code to determine how much needs to be done to implement deleting a file that has only a single instance on a computer that is no longer accessible.
  • Confirmed that the spurious SN transients identified by Armin Rest are in the locations for crosstalk predicted by John Tonry.
  • Various burntool related tweaks: detect summit processing and correctly register that fact, only apply burntool masking to CHIP-like recipes, option to allow processing with an old version of burntool, add BTOOLAPP header keyword translation to old GPC1 formats.