IPP Progress Report for the week 2011.11.03 - 2011.11.07

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

• Continued Condor related development for stack production
• Queued a bunch of sextractor related scripts for Tonry (within Condor)
• Investigated the "multiple camRun entries creation" mystery (fixed: "chiptool -advanceexp" -chip_id was supported but not implemented)
• Ran chip-stack within Condor for MD04 (from 2009-04). 1685 exposures were processed in about 60 hours (while pantasks was running stdscience, LAP, and other MDs).
• Started investigation of ippdb02 replication concerns (it can't keep up with ippdb00)

Heather Flewelling

Roy Henderson

• Loading to PSPS:
  • continued to load, monitor and cleanup LAP data
  • investigated dodgy batch on datastore that caused DXLayer to stop. Found issue and fixed it.
  • supplied batches for sandbox testing for Sue and Conrad. Needed to write code to republish batches already processed for LAP.
  • lots of batches failed by ODM after skycell change (see below) due to conflict with initialization table: cleaned-up and resubmitted ~3000 batches
  • looked at newly merged data. Found issue with duplicated filter: my fault. Fixed it
  • attempts to sort out mess after more dxlayer files were lost. Had to figure out what the ODM has 'seen' and what it hasn't. Still work to do here.

• ippToPsps development
  • worked on missing stack fields after pointers from Chris and Mark:
    • populating psfFwhm_mean and psfFwhm_max from FITS header
    • skycelID was wrong. Fixed and added projectioncellID
    • populating psfLikelhood
  • DXLayer batch deletion:
    • wrote a class to encapsulate the dxlayer. Only a 'delete' method so far, which calls webservice
    • built this into to my cleanup code
  • added comment column to batch table in ipptopsps database. populated for out-of-the-ordinary batches, eg duplicated exposures from LAP a few weeks ago. This will help with audit.

• Documentation:
  • updated ippToPsps documentation with recent 'multi-client' developments
  • updated PSPS news page

Mark Huber

• MD reprocessing - still limiting to deep reference stacks
  • adding to documentation summary
  • MD03.refstack and sample nightly stacks finished, checked and distributed
  • MD04 refstacks and sample nightly stacks finished, ready to be evaluated for distribution. testing 200 input warp deep stack.
  • MD05 reprocessing sample finished, ready to start refstacks
• ppSub auto convolution direction testing
  • expanded the ppSim test suite Gene had setup for earlier tests, adding sample stack for testing against common code changes
  • selecting representative dataset to swap in for ppSim
• Processing throughput -- with Gavin/Cindy's help tracked down 2-6am degraded processing due to rsync ippc18->ippc19. Had the rsync moved to 6am to keep an eye on, nice'd and bandwidth limited the rsync. Bill rotated the massive pantask logs that alleviated the need to transfer large files problem.

Bill Sweeney

• Implemented vptool a tool for managing the database tables related to the new video photometry stage.
• Implemented script to run psvideophot program on the chip images that contain video cells.
• Next step is to flesh out the program that does the video photometry analysis to build the output fits table.
• Managed reprocessing of M31 data from 2010.
• Two days as processing czar.
• Vacation Friday.

Chris Waters