Changes between Version 76 and Version 77 of Cluster_Storage_Notes

Timestamp:

Oct 30, 2013, 3:26:05 PM (13 years ago)

Author:

watersc1

Comment:

--

Cluster_Storage_Notes

@@ -3 +3 @@
 = Cluster Storage Notes =
 
+== 2013-10-30b ==
+
+The following tables list the number of replications for each stage/component under four cases.  "rep" is the default full state case.  "clean" is for cleaned data.  "alt1" is a single replication version of a full processing (all items are retained, although none have two copies).  "alt2" is a science version, in which images are retained, as well as interesting catalogs.
+
+||  full || raw || 'fits' || 'bttable' || 'log' || 'compfits' ||
+||  rep || raw || 2 || 2 || 1 || 0 ||
+||  clean || raw || 0 || 2 || 1 || 2 ||
+||  alt1 || raw || 1 || 1 || 1 || 0 ||
+||  alt2 || raw || 1 || 1 || 0 || 0 ||
+
+
+||  full || chip || 'b1fits' || 'b2fits' || 'catalog' || 'fits' || 'log' || 'mask' || 'mdc' || 'mdl' || 'psf' || 'ptn' || 'stats' || 'trace' || 'weight' ||
+||  rep || chip || 2 || 2 || 2 || 1 || 1 || 1 || 2 || 2 || 2 || 2 || 1 || 1 || 1 ||
+||  clean || chip || 0 || 2 || 0 || 0 || 1 || 0 || 2 || 2 || 2 || 0 || 1 || 1 || 0 ||
+||  alt1 || chip || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 1 ||
+||  alt2 || chip || 0 || 0 || 0 || 1 || 0 || 1 || 0 || 1 || 1 || 0 || 0 || 0 || 1 ||
+
+
+||  full || cam || 'b1jpg' || 'b2jpg' || 'catalog' || 'log' || 'mask' || 'mdc' || 'mdl' || 'stats' ||
+||  rep || cam || 2 || 2 || 2 || 1 || 120 || 2 || 120 || 1 ||
+||  clean || cam || 2 || 2 || 2 || 1 || 120 || 2 || 120 || 1 ||
+||  alt1 || cam || 1 || 1 || 1 || 1 || 60 || 1 || 60 || 1 ||
+||  alt2 || cam || 0 || 0 || 1 || 0 || 0 || 0 || 0 || 0 ||
+
+
+||  full || diff || 'b1fits' || 'b1jpg' || 'b2fits' || 'b2jpg' || 'catalog' || 'fits' || 'kernel' || 'log' || 'mask' || 'mdc' || 'mdl' || 'psf' || 'stats' || 'weight' ||
+||  rep || diff || 0 || 2 || 0 || 2 || 4 || 2 || 2 || 1 || 4 || 2 || 2 || 2 || 1 || 4 ||
+||  clean || diff || 0 || 2 || 0 || 2 || 4 || 0 || 2 || 1 || 0 || 2 || 2 || 2 || 1 || 0 ||
+||  alt1 || diff || 0 || 1 || 0 || 1 || 2 || 2 || 2 || 1 || 2 || 1 || 1 || 2 || 1 || 2 ||
+||  alt2 || diff || 0 || 0 || 0 || 0 || 2 || 2 || 2 || 0 || 2 || 0 || 0 || 2 || 0 || 2 ||
+
+
+||  full || warp || 'b1fits' || 'b1jpg' || 'b2fits' || 'b2jpg' || 'catalog' || 'fits' || 'log' || 'mask' || 'mdc' || 'mdl' || 'psf' || 'skycell' || 'stats' || 'weight' ||
+||  rep || warp || 0 || 0 || 0 || 0 || 2 || 1 || 1 || 1 || 2 || 1 || 2 || 1 || 1 || 1 ||
+||  clean || warp || 0 || 0 || 0 || 0 || 0 || 0 || 1 || 0 || 2 || 1 || 2 || 0 || 1 || 0 ||
+||  alt1 || warp || 0 || 0 || 0 || 0 || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 1 || 0 ||
+||  alt2 || warp || 0 || 0 || 0 || 0 || 0 || 1 || 0 || 1 || 0 || 0 || 1 || 0 || 0 || 1 ||
+
+
+||  full || stack || 'b1fits' || 'b1jpg' || 'b2fits' || 'b2jpg' || 'catalog' || 'fits' || 'kernel' || 'log' || 'mask' || 'mdc' || 'miscfits' || 'psf' || 'stats' || 'trace' || 'weight' ||
+||  rep || stack || 0 || 2 || 0 || 2 || 2 || 4 || 0 || 1 || 4 || 2 || 12 || 2 || 1 || 1 || 4 ||
+||  clean || stack || 0 || 2 || 0 || 2 || 2 || 0 || 0 || 1 || 0 || 2 || 0 || 2 || 1 || 1 || 0 ||
+||  alt1 || stack || 0 || 1 || 0 || 1 || 1 || 2 || 0 || 1 || 2 || 1 || 6 || 2 || 1 || 1 || 2 ||
+||  alt2 || stack || 0 || 0 || 0 || 0 || 1 || 2 || 0 || 0 || 2 || 0 || 6 || 2 || 0 || 0 || 2 ||
+
+
+The following table contains the current "full" and "clean" count to size analysis.  The number of database entries for each stage are listed.  For the survey studies, chip/exposure level have a factor of 450000, with stacks having 1e6 + 132405 (one 3pi + current estimate of MD nightly stacks).
+
+||  #stage || gb/item || gb/id || tb/survey || tb_expect || full_count |||| gb/item || gb/id || tb/survey || tb_expect || clean_count ||
+||  chip || 0.0397 || 2.3811 || 1046.3797 || 84.5568 || 36364 |||| 0.0006 || 0.0350 || 15.3614 || 28.6888 || 840417 ||
+||  stack || 0.5015 || 0.5015 || 554.5917 || 1052.0697 || 2148191 |||| 0.0028 || 0.0028 || 3.1006 || 1.8559 || 677790 ||
+||  cam || 0.2824 || 0.2824 || 124.1011 || 239.0212 || 866709 |||| 0.2824 || 0.2824 || 124.1011 || 0.8152 || 2956 ||
+||  warp || 0.0439 || 3.1354 || 1377.8455 || 533.0823 || 174103 |||| 0.0005 || 0.0339 || 14.8847 || 22.3392 || 675369 ||
+||  diff || 0.1422 || 8.0601 || 3542.0213 || 74.6265 || 9481 |||| 0.0017 || 0.0964 || 42.3521 || 40.6701 || 432128 ||
+||  raw || 0.0391 || 2.3458 || 1030.8909 || 1147.3083 || 500818 |||| 0.0051 || 0.3089 || 135.7668 || 48.2983 || 160085 ||
+
+
+||  #stage || gb/item || gb/id || tb/survey ||  || gb/item || gb/id || tb/survey ||  ||
+||  chip || 0.0373 || 2.2358 || 982.5113 ||  || 0.0349 || 2.0956 || 920.9190 ||  ||
+||  stack || 0.2508 || 0.2508 || 277.4004 ||  || 0.2501 || 0.2501 || 276.5264 ||  ||
+||  cam || 0.1414 || 0.1414 || 62.1557 ||  || 0.0375 || 0.0375 || 16.4886 ||  ||
+||  warp || 0.0225 || 1.6123 || 708.5245 ||  || 0.0426 || 3.0488 || 1339.8083 ||  ||
+||  diff || 0.0955 || 5.4143 || 2379.3507 ||  || 0.0952 || 5.4001 || 2373.1093 ||  ||
+||  raw || 0.0196 || 1.1741 || 515.9532 ||  || 0.0195 || 1.1718 || 514.9377 ||  ||
 == 2013-10-30 ==