Back to GPC1 Data Distribution

Also see the ps1net google group and Mirroring Requirements by institution

Questions/Notes:

  1. Server setup?
  2. Performance of rsync on real-life updated dvo database tables? <br />This could be done using the Stripe82 data after successively updating dvo with more and more detections. May need to do this with the full stripe82 dataset, i.e. a few hundred square degrees, rather than the current 2.5x2.5 test set.
  3. Transfer rate needed? Gene estimates:
    • ~1 Mbit/s continuous for detections
    • ~3 Mbit/s continuous for average tables (what about extended-source parameters?)
    • ~75 Mbit/s continous for raw images (but might also want weights & masks - i.e. double the raw rate)

Johannes Koppenhöfer has managed transfer rates of 400 Mbit/s from Hawaii to Garching, via two parallel rsync transfers at 200 Mbit/s each, but so far only over timescales of 10 minutes. The best sustained rate was ~80 Mbit/s over a few hours from ipp000 or alala to Heidelberg via rsync, though this was a much earlier experiment that should be repeated soon.

Data Transfer Requirements

  • Case 1 : Total data volume based on IPP storage requirements:
    • Raw Images:
      • 1 exposure per 90 seconds
      • 11 hours per night
      • 250 clear nights per year
      • 3.5 years
      • +10% for Detrend Data
      • Total Raw Exposures = 423500 Exposures
      • Detrended Images have 5 bytes per pixel uncompressed (signal, weight, mask)
      • 2.2 bytes per pixel compressed
      • 48002 x 60 = 1382400000 pixels = 3.0 GB / exposure
      • Total Detrended Exposure Dataset = 1.27 PB
    • Static Sky:
      • 30000 square degrees
      • 0.2 arcsec per pixel
      • 10% overlap
      • = 10 Terapixels
      • 5 filters
      • 10 bytes per pixel (uncompressed, 32bit floats)
      • 5 bytes per pixel (compressed)
      • Total Static Sky Dataset = 250 TB
      • 4 copies to be shipped (1 per year + 1 final)
      • Total Static Sky = 1.0 PB
    • Total Data Volume : 2.27 PB = 1.8x1016 bits
    • Time : 3.5 years = 1.1x108 sec
    • Data Rate = 164 Mb/sec sustained
  • Case 2 : Total data volume based on survey data size
    • Raw Images:
      • 5500 3pi fields
      • 60 exposures
      • +10% for Detrend Data
      • +25% for Other Survey Data
      • Total Raw Exposures = 453750 Exposures
      • Detrended Images have 5 bytes per pixel uncompressed (signal, weight, mask)
      • 2.2 bytes per pixel compressed
      • 48002 x 60 = 1382400000 pixels = 3.0 GB / exposure
      • Total Detrended Exposure Dataset = 1.36 PB
    • Static Sky (as above)
      • Total Static Sky = 1.0 PB
    • Total Data Volume : 2.36 PB = 1.9x1016 bits
    • Time : 3.5 years = 1.1x108 sec
    • Data Rate = 173 Mb/sec sustained
  • Case 3 : Only Raw (masked) survey data and static sky (ie, end users must re-detrend)
    • Raw Images:
      • 5500 3pi fields
      • 60 exposures
      • +10% for Detrend Data
      • +25% for Other Survey Data
      • Total Raw Exposures = 453750 Exposures
      • Detrended Images have 2 bytes per pixel uncompressed (signal)
      • 1.0 bytes per pixel compressed
      • 48002 x 60 = 1382400000 pixels = 1.38 GB / exposure
      • Total Detrended Exposure Dataset = 626 TB
    • Static Sky (as above)
      • Total Static Sky = 1.0 PB
    • Total Data Volume : 1.63 PB = 1.3x1016 bits
    • Time : 3.5 years = 1.1x108 sec
    • Data Rate = 118 Mb/sec sustained

recommended TCP/IP settings (to be set in /etc/sysctl.conf):

net.core.rmem_max=62500000
net.core.wmem_max=62500000
net.core.netdev_max_backlog = 2500

net.ipv4.tcp_rmem=4096 87380 62500000
net.ipv4.tcp_wmem=4096 16384 62500000
net.ipv4.tcp_moderate_rcvbuf=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_ecn=1
net.ipv4.tcp_adv_win_scale=7
net.ipv4.tcp_syncookies=0
net.ipv4.tcp_congestion_control=cubic

Speed test to Munich/Garching (single job, single file):

  Hawaii->cluster: 224.21Mbit/s
  Hawaii->MPE:      30.96Mbit/s 
  Hawaii->USM:      53.28Mbit/s

rsync command used for tests:

\rsync -avP rsync://ipp0002.ifa.hawaii.edu/catdir.synth.grizy/n0000.tar.gz .