IPP Progress Report for the week 2011.11.03 - 2011.11.07

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

  • Continued Condor related development for stack production
  • Queued a bunch of sextractor related scripts for Tonry (within Condor)
  • Investigated the "multiple camRun entries creation" mystery (fixed: "chiptool -advanceexp" -chip_id was supported but not implemented)
  • Ran chip-stack within Condor for MD04 (from 2009-04). 1685 exposures were processed in about 60 hours (while pantasks was running stdscience, LAP, and other MDs).
  • Started investigation of ippdb02 replication concerns (it can't keep up with ippdb00)

Heather Flewelling

Roy Henderson

  • Loading to PSPS:
    • continued to load, monitor and cleanup LAP data
    • investigated dodgy batch on datastore that caused DXLayer to stop. Found issue and fixed it.
    • supplied batches for sandbox testing for Sue and Conrad. Needed to write code to republish batches already processed for LAP.
    • lots of batches failed by ODM after skycell change (see below) due to conflict with initialization table: cleaned-up and resubmitted ~3000 batches
    • looked at newly merged data. Found issue with duplicated filter: my fault. Fixed it
    • attempts to sort out mess after more dxlayer files were lost. Had to figure out what the ODM has 'seen' and what it hasn't. Still work to do here.
  • ippToPsps development
    • worked on missing stack fields after pointers from Chris and Mark:
      • populating psfFwhm_mean and psfFwhm_max from FITS header
      • skycelID was wrong. Fixed and added projectioncellID
      • populating psfLikelhood
    • DXLayer batch deletion:
      • wrote a class to encapsulate the dxlayer. Only a 'delete' method so far, which calls webservice
      • built this into to my cleanup code
    • added comment column to batch table in ipptopsps database. populated for out-of-the-ordinary batches, eg duplicated exposures from LAP a few weeks ago. This will help with audit.
  • Documentation:
    • updated ippToPsps documentation with recent 'multi-client' developments
    • updated PSPS news page

Mark Huber

  • MD reprocessing - still limiting to deep reference stacks
    • adding to documentation summary
    • MD03.refstack and sample nightly stacks finished, checked and distributed
    • MD04 refstacks and sample nightly stacks finished, ready to be evaluated for distribution. testing 200 input warp deep stack.
    • MD05 reprocessing sample finished, ready to start refstacks
  • ppSub auto convolution direction testing
    • expanded the ppSim test suite Gene had setup for earlier tests, adding sample stack for testing against common code changes
    • selecting representative dataset to swap in for ppSim
  • Processing throughput -- with Gavin/Cindy's help tracked down 2-6am degraded processing due to rsync ippc18->ippc19. Had the rsync moved to 6am to keep an eye on, nice'd and bandwidth limited the rsync. Bill rotated the massive pantask logs that alleviated the need to transfer large files problem.

Bill Sweeney

  • Implemented vptool a tool for managing the database tables related to the new video photometry stage.
  • Implemented script to run psvideophot program on the chip images that contain video cells.
  • Next step is to flesh out the program that does the video photometry analysis to build the output fits table.
  • Managed reprocessing of M31 data from 2010.
  • Two days as processing czar.
  • Vacation Friday.

Chris Waters

  • SAS: distribute SAS2 z-band synthetic reference catalog. This was forgotten before.
  • Offnight diffs: correct data_group for offnight diffs, and implement minimum time difference check to force offnight diffs to truly come from different nights.
  • Diskspace:
    • Implement neb-repair to attempt to repair a nebulous object that has inconsistent md5sums. Added this to chip_imfile.pl to allow the script to attempt to fix problems internally without faulting. This seems to have been largely successful, and has decreased the number of times manual intervention is needed to keep LAP processing moving.
    • Confirmed that the targetting shuffle is working correctly, and is freeing space on the wave2/wave3 storage nodes. Reworked shuffle to enhance the amount of free space created on these nodes by preventing the targetting from moving new imfiles to these nodes and instead focus on moving to the upgraded wave1 nodes. Added calls to neb-repair to increase the success rate at the shuffle, as it corrects missing copies and forces the ATRC computers to respond correctly.
    • The combination of these two improvements has filled ippb02, so steps need to be taken to resolve this.
  • Processing:
    • Transferred processing power between distribution and stack to allow the number of stacks completed per hour to match the number of stacks LAP processing seems to be capable of generating. This has had minimal effect on the distribution stage processing, as that still shows a plateauing in the number of runs completed. We are probably therefore still overpowered in distribution, but this allows us to catch up after it falls behind.
  • LAP:
    • Defined new regions for LAP to work on when the current set is finished.
    • Increased the number of LAP runs processing at the same time. This increases the disk footprint somewhat, but allows runs to get stuck without stopping all LAP processing.