Back to [wiki:GPC1_DataDistribution GPC1 Data Distribution]

 Also see the [http://groups.google.com/group/ps1net ps1net google group] and [wiki:Mirroring_Requirements Mirroring Requirements by institution]

= Questions/Notes: =

 1. Server setup?
 1. Performance of rsync on real-life updated dvo database tables? <br />This could be done using the Stripe82 data after successively updating dvo with more and more detections. May need to do this with the full stripe82 dataset, i.e. a few hundred square degrees, rather than the current 2.5x2.5 test set.
 1. Transfer rate needed? Gene estimates: 
  * ~1 Mbit/s continuous for ''detections''
  * ~3 Mbit/s continuous for ''average tables'' ('''what about extended-source parameters?''')
  * ~75 Mbit/s continous for ''raw images'' (but might also want weights & masks - i.e. double the raw rate)
Johannes Koppenhöfer has managed transfer rates of 400 Mbit/s from Hawaii to Garching, via two parallel rsync transfers at 200 Mbit/s each, but so far only over timescales of 10 minutes. The best sustained rate was ~80 Mbit/s over a few hours from ipp000 or alala to Heidelberg via rsync, though this was a much earlier experiment that should be repeated soon.

== Data Transfer Requirements ==

 * Case 1 : Total data volume based on IPP storage requirements:
  * Raw Images: 
   * 1 exposure per 90 seconds
   * 11 hours per night
   * 250 clear nights per year
   * 3.5 years
   * +10% for Detrend Data
   * Total Raw Exposures = 423500 Exposures
   * Detrended Images have 5 bytes per pixel uncompressed (signal, weight, mask)
   * 2.2 bytes per pixel compressed
   * 4800^2^ x 60 = 1382400000 pixels = 3.0 GB / exposure
   * Total Detrended Exposure Dataset = 1.27 PB
  * Static Sky:
   * 30000 square degrees
   * 0.2 arcsec per pixel
   * 10% overlap
   * = 10 Terapixels
   * 5 filters
   * 10 bytes per pixel (uncompressed, 32bit floats) 
   * 5 bytes per pixel (compressed)
   * Total Static Sky Dataset = 250 TB
   * 4 copies to be shipped (1 per year + 1 final)
   * Total Static Sky = 1.0 PB
  * Total Data Volume : 2.27 PB = 1.8x10^16^ bits
  * Time : 3.5 years = 1.1x10^8^ sec
  * Data Rate = 164 Mb/sec sustained

 * Case 2 : Total data volume based on survey data size
  * Raw Images: 
   * 5500 3pi fields
   * 60 exposures
   * +10% for Detrend Data
   * +25% for Other Survey Data
   * Total Raw Exposures = 453750 Exposures
   * Detrended Images have 5 bytes per pixel uncompressed (signal, weight, mask)
   * 2.2 bytes per pixel compressed
   * 4800^2^ x 60 = 1382400000 pixels = 3.0 GB / exposure
   * Total Detrended Exposure Dataset = 1.36 PB
  * Static Sky (as above)
   * Total Static Sky = 1.0 PB
  * Total Data Volume : 2.36 PB = 1.9x10^16^ bits
  * Time : 3.5 years = 1.1x10^8^ sec
  * Data Rate = 173 Mb/sec sustained

 * Case 3 : Only Raw (masked) survey data and static sky (ie, end users must re-detrend)
  * Raw Images: 
   * 5500 3pi fields
   * 60 exposures
   * +10% for Detrend Data
   * +25% for Other Survey Data
   * Total Raw Exposures = 453750 Exposures
   * Detrended Images have 2 bytes per pixel uncompressed (signal)
   * 1.0 bytes per pixel compressed
   * 4800^2^ x 60 = 1382400000 pixels = 1.38 GB / exposure
   * Total Detrended Exposure Dataset = 626 TB
  * Static Sky (as above)
   * Total Static Sky = 1.0 PB
  * Total Data Volume : 1.63 PB = 1.3x10^16^ bits
  * Time : 3.5 years = 1.1x10^8^ sec
  * Data Rate = 118 Mb/sec sustained

== recommended TCP/IP settings (to be set in /etc/sysctl.conf): ==

{{{
net.core.rmem_max=62500000
net.core.wmem_max=62500000
net.core.netdev_max_backlog = 2500

net.ipv4.tcp_rmem=4096 87380 62500000
net.ipv4.tcp_wmem=4096 16384 62500000
net.ipv4.tcp_moderate_rcvbuf=1
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_sack=1
net.ipv4.tcp_ecn=1
net.ipv4.tcp_adv_win_scale=7
net.ipv4.tcp_syncookies=0
net.ipv4.tcp_congestion_control=cubic
}}}

== Speed test to Munich/Garching (single job, single file): ==

{{{
  Hawaii->cluster: 224.21Mbit/s
  Hawaii->MPE:      30.96Mbit/s 
  Hawaii->USM:      53.28Mbit/s
}}}

== rsync command used for tests: ==

{{{
\rsync -avP rsync://ipp0002.ifa.hawaii.edu/catdir.synth.grizy/n0000.tar.gz .
}}}