IPP Progress Report for the week 2010.03.15 - 2010.03.19

(Up to IPP Progress Reports)

Eugene Magnier

I spent most of the week working on re-structuring the DVO code to handle large files (>231 bytes = 2GB). This task required changing a small number of basic data types in the Ohana code and modifying a large number of APIs to accept those changes. Now the Ohana code is capable of handling FITS tables with sizes > 2GB as well as tables with more than 232 rows and images with axes larger than 232 pixels. I also added gcc format checking macros to make this code more reliable.

I also spent some time updating a previously-existing program to merge two DVO databases either into a new third database, or to merge the first into the second. We are going to use this program (creatively called dvomerge) to enable a new paradigm for the DVO ingest: nightly science will be ingested into a nightly dvo database, which will in turn be merged into a master database (or databases -- one per survey component). This paradigm allows us to run consistency checks on the nightly database before ingesting it, and it will allow us to more-easily distribute the database. It will be possible for remote sites to accept just the nightly DVO databases and to perform their own local merge, rather than attempting to rsync the full database. The nightly databases should be in the range of 10-20GB, while the full-scale database is already >200GB for 3pi, and is expected to reach 20TB. In order to test these upgrades, I fleshed out the dvo test suite, adding tap-based tests for addstar ingest, relphot, relastro, and dvomerge (these are in Ohana/src/addstar/test).

Heather Flewelling

  • worked on ppImage and psastro ISP
  • went to Eventful universe conference (conference was Wednesday - Saturday, there were travel days Tuesday + Sunday + Monday)

Bill Giebink

  • Re-installed Win XP and Visual C++ on new RoboDIMM laptop
  • Compiled and ran RoboDIMM code on new laptop (not tested on stars, however).
  • Contacted Takeo again to try to get PS2 dome/cooling documentation
  • Installed a temporary fan in IPPC19 to stop beeping

Roy Henderson

  • Maui loading:
    • Finished converting all of Jim's simulation data for loading to the cluster
    • Lots of time spent attempting to submit to ODM on Maui. Worked through the various issues with Sue:
      • Path/permissions issues
      • ODM bugs
      • Time consuming Db reset: involved logging in to 6 separate Windows machines accessed via JHU
      • Batches now successfully merging (apart from object batches, which are missing a table)
  • JHU sandbox loading
    • Generated some medium deep data, but not able to load: changes needed at JHU
  • IPP side:
    • Some work needed on diff code: hadn't looked at it in awhile
    • Speed concerns: large DVO Db = large lookup times. Did some tests to confirm: 16 secs with one-exposure DVO Db vs. 11 mins with current 3PI Db.
    • Values like filterID, surveyID etc now looked-up from 'init' XML config
    • Finally checked code into SVN
    • Finally got some documentation written on the Wiki
    • Improved pspsSchema2Xml script to find the relevant *.sql file for each table

Paul Price

  • Queued M31 and STS processing; babysitting
  • ppStack development:
    • Verified fixing half-pixel warp bug improves stacks without need for astrometry tuning
    • Fixed sign bug in ppStack transparency correction
    • Investigating stack clipping: PS1_V1 target PSF is better
    • Plugged memory leaks in ppStack that were blowing up the memory footprint
    • Reorganised flow of ppStack so that convolved and unconvolved stacks aren't both in memory at the same time
    • Made growth of rejected pixels optional, and only used for convolved stack; verified unconvolved rejected pixels are not grown
  • Set up Transients data store, published demonstration set.
  • Fixed camera script: was skipping chips when it couldn't find the CMF file
  • Fixed destreaking scripts: was assuming we always have a dynamic mask
  • Ran psphot on SAS.123 unconvolved stacks, built DVO catalogue, shipped
  • New program: ppSubConvolve. This applies the (previously calculated) convolution kernel to an image, allowing quick reproduction of an inConv or refConv image.

Bill Sweeney

  • Shepherded data for M31 and STS thorough the system.
  • Helped some IfA users get started using the postage stamp server.
  • Debugged performance problems with the postage stamp server that those users' requests exposed.
  • Debugged and repaired a serious error in the magic destreaking software that I introduced March 8. Data for some subset of the data processed since then has been over masked.
  • Updated some cleaned up warps that contain a new supernova. This uncovered a bug in ipptools that I still need to investigate.

Chris Waters

  • Processing monitoring: First set of old raw file compression is nearly finished. Monitored science processing progress.
  • OTIS interface: Updated IPP->OTIS interface code, and added more control to the configuration file. Tested and confirmed that a test table uploaded to the datastore correctly.
  • Detectibility server: confirmed that psphotForced calculates the values needed for this, and started work on the scripts and interface with the postage stamp server.
  • Started work on a script to bundle detrend images for distribution.