Changes between Version 35 and Version 36 of Cluster_Storage_Notes

Timestamp:

Nov 1, 2011, 2:13:48 PM (15 years ago)

Author:

watersc1

Comment:

--

Cluster_Storage_Notes

@@ -36 +36 @@
 CHIP update 3304
 }}}
-At the end of the week, a summary file is created (`neb://ipp_diskspace/YYYY-MM-DD/summary.dat`) by `ipp/trunk/tools/merge_counts_and_sizes.pl` that gives the usage in TB of each stage/product combination:
+At the end of the week, a summary file is created (`neb://ipp_diskspace/YYYY-MM-DD/summary.dat`) by `ipp/trunk/tools/nebulous_wiki.pl -S` that gives the usage in TB of each stage/product combination:
 {{{
 CHIP_B1FITS 4876.40501260757
@@ -61 +61 @@
 }}}
 
-A largely untested final summary should be generated by `ipp/trunk/tools/diskspace/generate_mappings.pl` that takes the usage.dat files, the run_im_counts.dat file, and the mapping file (hardcoded as `neb://ipp_diskspace/mappings_im.dat`) to match the sizes in each stage/product and the counts in each stage/data_state, and construct an understanding of how much disk space is being used by permanent products (raw imfiles), transient products (chip stage images), and final output products (stacks).  This has been delayed due to other concerns, and an issue with the perl installation on ippbXX.  This final summary would then be able to be used to recalculate the table in `trunk/tools/diskspace/sizes_from_counts.pl`.  This script takes a list of counts (any of the run_im_counts.dat files), and uses the average sizes for each stage/data_state, and calculates a description of which stages are using the most disk space.  Running this script on the count file from 2011-10-09 yields (in part):
+This script can also generate the wiki page tables displayed below, showing how each stage/product is distributed across the cluster.
+
+A largely untested final summary should be generated by `ipp/trunk/tools/diskspace/merge_counts_and_sizes.pl` that takes the usage.dat files, the run_im_counts.dat file, and the mapping file (hardcoded as `neb://ipp_diskspace/mappings_im.dat`) to match the sizes in each stage/product and the counts in each stage/data_state, and construct an understanding of how much disk space is being used by permanent products (raw imfiles), transient products (chip stage images), and final output products (stacks).  This has been delayed due to other concerns, and an issue with the perl installation on ippbXX.  This final summary would then be able to be used to recalculate the table in `trunk/tools/diskspace/sizes_from_counts.pl`.  This script takes a list of counts (any of the run_im_counts.dat files), and uses the average sizes for each stage/data_state, and calculates a description of which stages are using the most disk space.  Running this script on the count file from 2011-10-09 yields (in part):
 {{{
 CHIP     cleaned            PRODUCT        31179.7770     0.001840  16945531