IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 36 and Version 37 of Cluster_Storage_Notes


Ignore:
Timestamp:
Nov 7, 2011, 12:34:29 PM (15 years ago)
Author:
watersc1
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Cluster_Storage_Notes

    v36 v37  
    3636CHIP update 3304
    3737}}}
    38 At the end of the week, a summary file is created (`neb://ipp_diskspace/YYYY-MM-DD/summary.dat`) by `ipp/trunk/tools/nebulous_wiki.pl -S` that gives the usage in TB of each stage/product combination:
     38At the end of the week, a summary file is created (`neb://ipp_diskspace/YYYY-MM-DD/summary.dat`) by `ipp/trunk/tools/nebulous_wiki.pl -S` that gives the usage in TB of each stage/product combination, as well as the count of these products found on disk (faked in this example):
    3939{{{
    40 CHIP_B1FITS 4876.40501260757
    41 CHIP_B1JPG 5.90300299786031
    42 CHIP_B2FITS 506.811673343182
    43 CHIP_B2JPG 0.397686927579343
    44 CHIP_BTTABLE 0
    45 CHIP_BUNDLE 2889.25785760116
    46 CHIP_CATALOG 7536.26382619143
    47 CHIP_FITS 14993.0452584894
    48 CHIP_KERNEL 3.89136224985123
    49 CHIP_LOG 557.324770034291
    50 CHIP_MASK 1228.97341176867
    51 CHIP_MDC 1149.67314596102
    52 CHIP_MDL 501.610046625137
    53 CHIP_PNG 10.054713034071
    54 CHIP_PSF 1185.04951536655
    55 CHIP_PTN 1154.56684827805
    56 CHIP_SKYCELL 0.0493767857551575
    57 CHIP_STATS 68.9840127192438
    58 CHIP_TRACE 5.00015713181347
    59 CHIP_UNKNOWN 0.0171183617785573
    60 CHIP_WEIGHT 9193.91913002729
     40CHIP_B1FITS 4876.40501260757 999
     41CHIP_B1JPG 5.90300299786031 999
     42CHIP_B2FITS 506.811673343182 999
     43CHIP_B2JPG 0.397686927579343 999
     44CHIP_BTTABLE 0 0
     45CHIP_BUNDLE 2889.25785760116 999
     46CHIP_CATALOG 7536.26382619143 999
     47CHIP_FITS 14993.0452584894 999
     48CHIP_KERNEL 3.89136224985123 999
     49CHIP_LOG 557.324770034291 999
     50CHIP_MASK 1228.97341176867 999
     51CHIP_MDC 1149.67314596102 999
     52CHIP_MDL 501.610046625137 999
     53CHIP_PNG 10.054713034071 999
     54CHIP_PSF 1185.04951536655 999
     55CHIP_PTN 1154.56684827805 999
     56CHIP_SKYCELL 0.0493767857551575 999
     57CHIP_STATS 68.9840127192438 999
     58CHIP_TRACE 5.00015713181347 999
     59CHIP_UNKNOWN 0.0171183617785573 999
     60CHIP_WEIGHT 9193.91913002729 999
    6161}}}
    6262
    6363This script can also generate the wiki page tables displayed below, showing how each stage/product is distributed across the cluster.
    6464
    65 A largely untested final summary should be generated by `ipp/trunk/tools/diskspace/merge_counts_and_sizes.pl` that takes the usage.dat files, the run_im_counts.dat file, and the mapping file (hardcoded as `neb://ipp_diskspace/mappings_im.dat`) to match the sizes in each stage/product and the counts in each stage/data_state, and construct an understanding of how much disk space is being used by permanent products (raw imfiles), transient products (chip stage images), and final output products (stacks).  This has been delayed due to other concerns, and an issue with the perl installation on ippbXX.  This final summary would then be able to be used to recalculate the table in `trunk/tools/diskspace/sizes_from_counts.pl`.  This script takes a list of counts (any of the run_im_counts.dat files), and uses the average sizes for each stage/data_state, and calculates a description of which stages are using the most disk space.  Running this script on the count file from 2011-10-09 yields (in part):
     65A largely untested final summary should be generated by `ipp/trunk/tools/diskspace/merge_counts_and_sizes.pl` that takes the usage.dat files, the run_im_counts.dat file, and the mapping file (hardcoded as `neb://ipp_diskspace/mappings_im.dat`) to match the sizes in each stage/product and the counts in each stage/data_state, and construct an understanding of how much disk space is being used by permanent products (raw imfiles), transient products (chip stage images), and final output products (stacks).  This has been delayed due to other concerns, and an issue with the perl installation on ippbXX.  This final summary would then be able to be used to recalculate the table in `trunk/tools/diskspace/sizes_from_counts.pl`.  This script takes a list of counts (any of the run_im_counts.dat files), and uses the average sizes for each stage/data_state, and calculates a description of which stages are using the most disk space.  In addition, it uses the disk count values to estimate the total usage across the entire cluster. Running this script on the count file from 2011-11-07 yields (in part):
    6666{{{
    67 CHIP     cleaned            PRODUCT        31179.7770     0.001840  16945531
    68 CHIP     error_cleaned      PRODUCT         2652.1420     0.105432     25155
    69 CHIP     error_scrubbed     PRODUCT          232.8581     0.039798      5851
    70 CHIP     full               TRANSIENT      75617.5081     0.119770    631356
    71 CHIP     purged             PRODUCT          527.8040     0.001840    286850
    72 CHIP     scrubbed           PRODUCT           11.0271     0.001840      5993
    73 CHIP     update             TRANSIENT        348.3473     0.105432      3304
     67#                                      ---------based on sizes-----  --based on disk sizes/counts--     
     68#Stage   state              type           Total usage use/db entry        Total usage use/db entry      N db entries
     69CHIP     cleaned            PRODUCT          4892.5874     0.000272      146376.502127     0.008123      18020276
     70CHIP     error_cleaned      PRODUCT           486.8947     0.019349         923.416495     0.036696         25164
     71CHIP     error_scrubbed     PRODUCT            55.1817     0.009431         145.206937     0.024817          5851
     72CHIP     full               TRANSIENT       11400.8763     0.021286       36359.449063     0.067885        535603
     73CHIP     purged             PRODUCT            77.8811     0.000272        2330.047533     0.008123        286850
     74CHIP     scrubbed           PRODUCT             1.6271     0.000272          48.680407     0.008123          5993
     75CHIP     update             TRANSIENT         158.6413     0.019349         300.869967     0.036696          8199
    7476}}}
    7577This shows that of the space used by chip (on the disks scanned), 30% is consumed by the permanent output products of previous calculations, etc.  There is also a summary printed:
    7678{{{
    77         PRODUCT         272237.058713
    78         TRANSIENT       160852.076341
    79         PERMANENT       805663.824597
    80         SUM             1238752.959651
     79#       Type            based on sizes     based on disk sizes/counts
     80        PRODUCT         70191.9236020825        449497.935598959
     81        EXTERNAL        27428.9286857136        27428.9286857136
     82        TRANSIENT       24491.4957386452        79790.7955015027
     83        PERMANENT       109276.660096133        1011708.07967771
     84        TOTAL           231389.008122574        1568425.73946388
     85        EXP_TOT         231942.681785583
    8186}}}
    8287which displays the calculated sizes used by each of the three classes of products, along with the sum.  This sum seems to only be accurate to about 10%, and would certainly be improved by fixing the ippbXX perl issue and fully recalculating the size/count table used.