Changes between Version 59 and Version 60 of Cluster_Storage_Notes

Timestamp:

Nov 27, 2012, 10:56:00 AM (14 years ago)

Author:

Serge CHASTEL

Comment:

--

Cluster_Storage_Notes

@@ -168 +168 @@
 Total* = Total excluding nodes: ['ipp004.0', 'ipp005.0', 'ipp064.0']
 }}}
-
-
-== Automatic Storage Monitoring ==
-
-Cron jobs are set up on all nebulous storage nodes to scan the nebulous disks and construct statistics about their contents.  These jobs (calling `ipp/trunk/tools/diskspace/nebulous_data.pl`) are striped across the cluster throughout the week, in an attempt to prevent all hosts running the disk scan simultaneously (which would presumably have an effect on processing throughput).  The usage statistics are placed in nebulous in files named `neb://ipp_diskspace/YYYY-MM-DD/ippXXX.Z.neb_usage.dat`, with the date string matching the date of the Sunday of the week the scan was performed.  The script parses each filename in the nebulous directory, and attempts to classify the stage it comes from as well as the type of product from that stage.  The total count of these files, and the total usage (in bytes) of these files.  These totals are what is stored in the statistics files.  An excerpt of one of these files is listed below, showing the CHIP stage file statistics from one host.
-{{{
-CHIP B1FITS 774142 320650755840
-CHIP B1JPG 71 18789330
-CHIP B2FITS 775168 34038596160
-CHIP B2JPG 70 902953
-CHIP BUNDLE 6649 190039346914
-CHIP CATALOG 462015 443818370880
-CHIP FITS 98121 1030044401280
-CHIP KERNEL 14 6658560
-CHIP LOG 764083 25942316933
-CHIP MASK 44888 93167104320
-CHIP MDC 774856 82952703287
-CHIP MDL 832113 34001887680
-CHIP PNG 9838 160628007
-CHIP PSF 774958 77240390400
-CHIP PTN 50884 60478741440
-CHIP SKYCELL 30 86400
-CHIP STATS 637305 2358654287
-CHIP TRACE 422839 29600227
-CHIP WEIGHT 44740 608909785920
-}}}
-
-At the end of the week, the gpc1 database is polled by `ipp/trunk/tools/diskspace/run_counter.pl` to identify the count of states for each stage, and the results stored in `neb://ipp_diskspace/YYYY-MM-DD/run_im_counts.dat`.  This gives the stage, the data_state, and then the count of the number of components with that data_state:
-{{{
-CHIP cleaned 16945531
-CHIP error_cleaned 25155
-CHIP error_scrubbed 5851
-CHIP full 631356
-CHIP purged 286850
-CHIP scrubbed 5993
-CHIP update 3304
-}}}
-At the end of the week, a summary file is created (`neb://ipp_diskspace/YYYY-MM-DD/summary.dat`) by `ipp/trunk/tools/nebulous_wiki.pl -S` that gives the usage in TB of each stage/product combination, as well as the count of these products found on disk (faked in this example):
-{{{
-CHIP_B1FITS 4876.40501260757 999
-CHIP_B1JPG 5.90300299786031 999
-CHIP_B2FITS 506.811673343182 999
-CHIP_B2JPG 0.397686927579343 999
-CHIP_BTTABLE 0 0
-CHIP_BUNDLE 2889.25785760116 999
-CHIP_CATALOG 7536.26382619143 999
-CHIP_FITS 14993.0452584894 999
-CHIP_KERNEL 3.89136224985123 999
-CHIP_LOG 557.324770034291 999
-CHIP_MASK 1228.97341176867 999
-CHIP_MDC 1149.67314596102 999
-CHIP_MDL 501.610046625137 999
-CHIP_PNG 10.054713034071 999
-CHIP_PSF 1185.04951536655 999
-CHIP_PTN 1154.56684827805 999
-CHIP_SKYCELL 0.0493767857551575 999
-CHIP_STATS 68.9840127192438 999
-CHIP_TRACE 5.00015713181347 999
-CHIP_UNKNOWN 0.0171183617785573 999
-CHIP_WEIGHT 9193.91913002729 999
-}}}
-
-This script can also generate the wiki page tables displayed below, showing how each stage/product is distributed across the cluster.
-
-A largely untested final summary should be generated by `ipp/trunk/tools/diskspace/merge_counts_and_sizes.pl` that takes the usage.dat files, the run_im_counts.dat file, and the mapping file (hardcoded as `neb://ipp_diskspace/mappings_im.dat`) to match the sizes in each stage/product and the counts in each stage/data_state, and construct an understanding of how much disk space is being used by permanent products (raw imfiles), transient products (chip stage images), and final output products (stacks).  This has been delayed due to other concerns, and an issue with the perl installation on ippbXX.  This final summary would then be able to be used to recalculate the table in `trunk/tools/diskspace/sizes_from_counts.pl`.  This script takes a list of counts (any of the run_im_counts.dat files), and uses the average sizes for each stage/data_state, and calculates a description of which stages are using the most disk space.  In addition, it uses the disk count values to estimate the total usage across the entire cluster. Running this script on the count file from 2011-11-07 yields (in part):
-{{{
-#                                      ---------based on sizes-----  --based on disk sizes/counts--     
-#Stage   state              type           Total usage use/db entry        Total usage use/db entry      N db entries
-CHIP     cleaned            PRODUCT          4892.5874     0.000272      146376.502127     0.008123      18020276
-CHIP     error_cleaned      PRODUCT           486.8947     0.019349         923.416495     0.036696         25164
-CHIP     error_scrubbed     PRODUCT            55.1817     0.009431         145.206937     0.024817          5851
-CHIP     full               TRANSIENT       11400.8763     0.021286       36359.449063     0.067885        535603
-CHIP     purged             PRODUCT            77.8811     0.000272        2330.047533     0.008123        286850
-CHIP     scrubbed           PRODUCT             1.6271     0.000272          48.680407     0.008123          5993
-CHIP     update             TRANSIENT         158.6413     0.019349         300.869967     0.036696          8199
-}}}
-This shows that of the space used by chip (on the disks scanned), 30% is consumed by the permanent output products of previous calculations, etc.  There is also a summary printed:
-{{{
-#       Type            based on sizes     based on disk sizes/counts
-        PRODUCT         70191.9236020825        449497.935598959
-        EXTERNAL        27428.9286857136        27428.9286857136
-        TRANSIENT       24491.4957386452        79790.7955015027
-        PERMANENT       109276.660096133        1011708.07967771
-        TOTAL           231389.008122574        1568425.73946388
-        EXP_TOT         231942.681785583
-}}}
-which displays the calculated sizes used by each of the three classes of products, along with the sum.  This sum seems to only be accurate to about 10%, and would certainly be improved by fixing the ippbXX perl issue and fully recalculating the size/count table used.
-
-
-
-
-----
-