IPP Progress Report for the week 2010.05.24 - 2010.05.30

(Up to IPP Progress Reports)

Eugene Magnier

At the start of the week, I was focused on preparing the next IPP tagged-release for the operational system. We've been tightening up our operations model, and especially the version control on the installation used for science operations. I updated the build system to ensure the complete IPP (including the approved version of magic) can be built with a single command. This reduces the chance for new installations to have errors in the configuration.

I have also been working on the portion of the pipeline needed to automate the stack photometry process. We are calling this the 'staticsky' stage of the pipeline. The automation is now complete: the operations team can initiate a staticsky analysis for any set of input stacks with any number of filters.

I am next working on re-generating a deep MD04 reference stack using all (or at least most) of the data taken to date. The first step of this process is to create an MD04 astrometry and photometry reference database using the PS1 data taken to date. That should allow us to build a stack with better registration as well as high-quality photometric calibration. We have 100-300 exposures per filter (and no more coming this year), so we should now be able make a very high-quality reference stack on which we can run the staticsky stack photometry. Once the process has been demonstrated with MD04, we can run it for the other MD fields that are no longer accessible.

Heather Flewelling

  • finishing up addtool minidvodb stuff - moving over to production cluster this week for testing.
  • ifaps1 - answered more questions
  • created dvodb catalog for Roy
    • addstar uses .ptolemyrc or ptolemy.rc (which was pointing to a deleted build, oops, something to watch out for when building/rebuilding)
  • debugged ISP registration problems (isp db was out of sync with gpc1, footools were failing)
  • asked Cindy to add new rows to OTIS db (shutopen, shutclose), as requested by ken and john.

Bill Giebink

Roy Henderson

  • PSPS
    • Continued loading demo month data, now approx 6000 exposures in Db. Issues encountered:
      • some duplicate exposures 'sneaked in', so wrote a script to generate a list from audit logs then Sue purged them from Db before merging
      • found bug (logic error) whereby code was not getting newest reprocessed smf data. Fixed query.
      • fixed bug arising after certain exposures (legitimately) failed, but caused all subsequent exposures to fail as well
      • loading halted when I encountered corrupt DVO database. Heather built me a new one for 3PI demo month
    • added two new tables and functions to the schema for handling detection and image quality flags
  • IPP
    • bulk of work was creating a database back-end for ippToPsps:
      • wrote 'createDb' script to generate the Db, complete with a revision table
      • table logging batch_id -> exp_id and various processing states. Populated from old audit logs
      • changed run script to use Db in place of old flat-file auditing
    • Czar for two days
      • Changed czartool to use a menu to be more in-keeping with the rest of ippMonitor
      • can now link from faults to a pages where logs can be located

Paul Price

  • Flat top significance images
    • Building stacks with updated software
    • Stacks produce significance images with peaked top
    • Stacks produce diff significance image with peaked top
    • All clear!
  • Writing PSF-matching paper
    • Text mostly done, working on figures/examples
    • Found small bug in noise model: changing scale of covariance matrix was introducing power
    • Measure covariance from PSF-matching at a small number of positions over the image and average
  • Processing czar:
    • Distribution server wasn't running; restarted
    • nebdiskd wasn't running, restarting was difficult because of Gentoo init script funkiness
    • Fixed warptool -advancerun and chiptool -advancerun
    • Merged dbconfig, ippTools, pstamp into operational branch and rebuilt
    • Deleted nebulous instances on full ipp015 causing warps to fail
    • Converted a couple of ppSub error cases to 'bad data quality': no sources to match, variance renormalisation out of bounds
    • Regenerated diff 56818/skycell.2213.067 and 56957/skycell.1117.097 due to corrupt file
    • Fixed corrupted raw file
  • Recovering after demise of laptop (needed to boot old machine via floppy to install Ubuntu)
  • Reworked locking in psThread.c to attempt to fix aborts
  • Turned diff statistics on

Bill Sweeney

Chris Waters

  • Masks: Finished mask work, and made a wiki page about the results: GPC1_FPA_Mask_20100525. Created final median static mask, and sat down with the camera team to discuss what could be done to minimize the effect of detector issues. Merged mask stats/software revision code into the trunk.
  • Work on JPEG creation for warp and later stages. JPEG creation code and tasks are largely finished. Incorporating stack association idea into this branch to allow multiple stack_id entries to be grouped into a single object on the basis of the projection center.
  • Modified ppSkycell to output fits images as well as JPEG images. Still need to do some work to add usable header information to these files.