IPP Progress Report for the week 2010.11.22 - 2010.11.26

(Up to IPP Progress Reports)

Eugene Magnier

I worked on two side projects this past week related to data storage. First, I added compression I/O support to the IPP metadata I/O functions. This will allow us to apply gzip all of the MDC configuration files which are saved at each stage. Uncompressed, those files now take up some 30TB of space on our cluster; compressed they should be about 2TB, so this is an interesting savings.

The other project was motivated by the realisation that our data replication has some equivalence to RAID 1, but it could be a RAID 5. We currently duplicate all raw exposures across the cluster to avoid loss of data if we lose a single storage node. Instead, we could identify groups of N files that are distributed across N unique hosts, and save a parity image of the N images on another unique host. This parity image would be the same size as the largest of the N input images. Like RAID 5, we could reproduce any single one of the N input images with the parity image and the remaining N-1 images. Thus we would be protected against the loss of a single node, just as we are currently protected. This data protection strategy would only require (1 + 1/N) times the input data, instead of 2 times the input data, as in the current strategy. There are some downsides to this strategy: recovery is somewhat slower (or blocks the rest of processing more) since it requires all N-1 remaining files to be read; the management is a bit more complex, and automation of the management within Nebulous would take some coding work. However, for some of the less in-demand data, we could reduce our storage footprint using manual scripts without too much effort.

I created a simple program to generate these parity images and to repair an image. I intend for us to move the older detrend images, the in-focus focus images, and the stare data to this backup strategy soon, which would save us some 60 - 80TB.

I also started work on the photometric and astrometric reference catalog: I made a subset database of the ThreePi? dvo database with the high-quality sources, and started to run some relphot and relastro tests. I also started work on the STS-desired forced photometry code in the difference image analysis.

Serge Chastel

  • IPP czar on Monday, Thursday, Friday
  • MOPS czar
  • Worked on tessellation 3d visualization: nothing interesting yet
  • Thurday off
  • Attempted to make changes and integrate new ppMops output format

Heather Flewelling

  • vacation

Roy Henderson

  • PSPS
    • Helped Robert Senger with PSPS loading stuff (datastore, manifest files etc), attempts to install dsreg
  • IPP
    • czartool
      • roboczar is now emailing ps-ipp-ops. Added rest of development group to list
      • czarpoll is now creating daily metrics at 6:30 every morning
    • spot-checked some exposures in new DVO Db: looks ok
    • metrics
      • new class to produce metrics for a date range (eg a lunation).
      • now reporting total mask fraction across all exposures for a given date range
      • code to create index HTML file for current available metrics
      • metruics created since beginning of czartool (July 18th) and last few lunations
    • documentation
      • started new page of user and developer documentation on IPP wiki for czartool, complete with graphics. Can be found here (still lots to do).

Bill Sweeney

  • Built reference DVO catalog for reprocessed OSS data. Used this to reprocess those data from camera through magic. The cleaned up those data to save space. Stacks were distributed to the data store
  • Found case where publication was repeatedly faulting and creating enormous log files. Fixed symptom. Filed ticket on problem.
  • 2 days as processing czar.
  • fixed bug where the detection efficiency extensions were getting lost during destreaking.
  • wrote and executed script to examine smf files to extract the detection efficiency parameters. Put them in a temporary table in the database. Eventually will add them to the real tables.
  • Helped the postage stamp server through a request storm. (The way updates are queued needs some work to avoid deadlock when a large number of updates are outstanding with the same label.)
  • Selected exposures to make new MD02 stacks. Processed them.
  • Found problem causing some warp-warp diff runs to get the inverse bit set incorrectly.
  • tried and failed to debug cluster problem over the weekend. Gene found that it was some dependencies on servers in Manoa that were having problems.

Chris Waters

  • Linearity: Finished code and detrend creation, and tested that the correction produces the results that we want. Merged into trunk, so this can be used to build the new detrends for reprocessing.
  • Realtime Burntool: Added mode to regtool to determine what to do with a given exposure. Need to finish coding ipp_register_imfile.pl scripting, and then add a mode to regtool to identify imfiles that were skipped on the initial pass but can now be burntooled.
  • Thanksgiving.