[[PageOutline]]

This wikipage gives some details about the "Files Monitoring" software.

= Objectives =

The objectives of the "Files Monitoring" software are:
 * Make sure that nebulous doesn't lose any OTA
 * Provide statistics about the storage nodes usage

= Implementation =
 * The software is still installed and run as {{{schastel}}}. TODO: Move it to {{{ippm}}}
 * The source code is in {{{~schastel/dev/FilesMonitoring}}}. In the rest of this documentation, all paths are relative to it. TODO: SVN it
 * On each storage node, every hour, the {{{NodeRelated/refresh_table.py}}} script is started (from the crontab file defined in {{{NodeRelated/deployment/scripts/crontab.<node name>}}}). To prevent cluster/nfs overwhelming, the scripts are not run at the same time. The crontab files as well as the deployment script can be generated from {{{NodeRelated/deployment/generate_crontabs_and_deployment_script.py}}}. Run it and follow its (last) instruction to deploy. Note that error messages are currently sent to {{{schastel@ifa.hawaii.edu}}}.
 * Every 4 hours (see schastel crontab on ippc11), the {{{Summary/snapshot.py}}} script is run and takes snapshots, i.e. statistics, of each storage node database.

= {{{NodeRelated/refresh_table.py}}} Details =

== User Doc ==
 * This script runs on each storage node. Each storage node should run a number of instances on this script equal to the number of its storage partitions (i.e. 1 for the mhpcc nodes, 3 for the atrc and the stsci nodes).
 * The script uses the local mysql server and its {{{DisksMonitoring}}} database
 * '''If the script (resp. the mysql server) are overwhelming the host (nfs, I/O, swap...), FEEL FREE TO KILL (resp. STOP) THEM'''. If you stop the mysql server, think of restarting it. The script should be robust enough to deal with it in the worst case, it will send an e-mail to the MAILTO recipient defined in {{{NodeRelated/deployment/scripts/crontab.<node name>}}}.

== Script Stages ==
 * Constants are (should be) all defined in the Constants pseudo-class
 * First, some initializations are performed (which disk is the script supposed to analyze, logging, ...)
 * If the last analyzed directory has been analyzed less than {{{Constants.max_refresh_interval}}} (= 6 hours) seconds ago, stop the script (otherwise the script is likely still running or has been intentionally killed)
 * Run an infinite loop
   * Get a directory to analyze (one of the 256 /export/<hostname>.<disk>/nebulous directories). Each directory has its own table: {{{neb_<disk>_<directory name>}}}, e.g. {{{neb_1_a4}}} is associated to the {{{/export/<hostname>.1/nebulous/a4}}} directory
   * Run {{{updatedb}}} on the directory to analyze (a faster implementation of updatedb/mlocate is installed)
   * Get all ins_id (nebulous instance id) from the {{{FilesMonitoring}}} database and the {{{mlocate}}} database:
   * Mark the files which are in the {{{FilesMonitoring}}} database but not in the {{{mlocate}}} database as deleted;
   * Add the files which are in the {{{mlocate}}} database but not in the {{{FilesMonitoring}}} database (since they are "new" files). New entries are undefined;
   * Optimize the related mysql table
   * Refresh the status of a maximum of {{{Constants.limit_undefined}}} (=10000) undefined entries using the {{{NodeRelated/update_table_undefined.py}}} script
   * Refresh the status of a maximum of {{{Constants.limit}}} (=1000) oldest entries using the {{{NodeRelated/update_table.py}}} script
   * Update the refresh_status information: set the last analyzed directory of the current disk to the current directory (timestamp is automatically updated).
   * Sleep for a while ({{{Constants.sleep}}} = 10 seconds )
== Deployment ==
TODO: 

= {{{NodeRelated/update_table_undefined.py}}} Details =
 * The script depends on {{{ipp.lifetime_templates}}} (hence the PYTHONPATH line in the crontab script)
 * This script can be run independently (and in parallel of the other scripts: {{{-h}}} for help)
 * The objective of this script is to classify "undefined" files in one of the categories defined in the {{{ipp.lifetime_templates}}} (see related paragraph TODO: Add link)
 * Its parameters are: the maximum number of undefined files to be classified (default: 10), the disk partition (default: 0), and the directory.
 * Whatever the value of the maximum number of undefined files to be classified, the files are classified by batches of 2500: we don't want to overwhelm the local mysql database / host. Therefore if you specify a maximum of 11000 files, there will be 5 iterations.
 * In each batch, 2500 undefined entries, that is, entries having an {{{undefined}}} status from the neb_<disk>_<directory name> table are selected, classified according to the rules defined in {{{ipp.lifetime_templates}}}, and then updated.

Note: The classifier defined {{{ipp.lifetime_templates}}} is dynamically imported: this means that there is no need to stop-and-restart the {{{NodeRelated/refresh_table.py}}} script to modify its behavior. It is the only reason why the {{{NodeRelated/update_table_undefined.py}}} is called as a script by {{{NodeRelated/refresh_table.py}}} and not as a class/function.

= {{{NodeRelated/update_table.py}}} Details =
 * This script is exactly the same as the {{{NodeRelated/update_table_undefined.py}}} script except that the condition to select entries is bases on their analysis date: the older are selected.
 * Since the same dynamic importation mechanism, this means that after some time any file will have its classification updated (patience...)

TODO: We don't like duplicate code, do we?
= {{{~schastel/local/share/python/ipp/lifetime_templates.py}}} Details =
See [wiki:Cluster_Storage_Notes]

This is where the file classifier and the classification definition are both defined. Since python allows easy dynamic class loading there is not need to have distinct definitions.

Each file name is classified:
 * The stage is determined (chip, cam, ..., unknown): This is the stage where the file is supposed to have been built;
 * The role is determined (b1fits, stats, ..., unknown): This is the role of the file;
 * The lifetime of the file is obtained from both the stage and the role: raw (keep the file till the end of times and possibly after), permanent (a product to be kept), ephemeral (product that will be cleaned up one day), unknown (it's unknown as soon as one is unknown)

To add a new stage, a new role:
 * The {{{initialize_IppTemplates}}} function has to be updated;
 * Add a stage definition in the first lines of the  {{{initialize_IppTemplates}}} function. A stage definition is made of a unique name (a Python str) and a Python list of regular expressions (a regular expression is a string as well). You need to define the lifetime of the role you want to the next lines. For instance, when the detrend stage was added, I added:
{{{
 Stage.add_stage(Stage( 'detrend', [ '^.*\.det.*$', ] ))
}}}
and then:
{{{
    Lifetime.add_lifetime( Lifetime('detrend', 'b1fits', 'permanent') )
...
    Lifetime.add_lifetime( Lifetime('detrend', 'b1jpg', 'permanent') )
}}}
 * It is perfectly acceptable to have a stage with no associated lifetime (e.g. skycal and staticsky)
 * I try to keep it clean and easily maintainable by grouping the lifetime definitions under the role associated to it. Please do the same...
 * It is strongly advised to add a relevant set of unit tests to the {{{Template}}} class when a new definition is added (Did I hear someone say "If it's not tested it doesn't work"?)

There is a "subtle" behavior for fits files in the classifier. When a filename is initially classified as fits, its analysis is pushed further to check if it is not a particular fits case (lines around the {{{if role.name != 'fits':}}} statement).

= {{{Summary/snapshot.py}}} Details =
 * The purpose of this script is 
   * 1) to extract statistics from the different neb_<disk>_<directory> tables
   * 2) to delete the entries which have been marked as deleted in those tables
 * The nodes are arbitrarily grouped in sets (set0, ..., ippb, ...)
 * In the loop over the hosts (given as the script arguments), for each neb_<disk>_<directory> table :
   * Insert or update in the {{{CurrentDiskUsage}}} table, the size and count statistics for each set of entries grouped by role, stage, lifetime, and status;
   * Delete the entries which have been marked as deleted from the current table;
   * Delete old entries from the {{{CurrentDiskUsage}}} table
   * Update the daily statistics (using the {{{UpdateDailyByNode}}} stored procedure)
   * Update the weekly statistics (using the {{{UpdateWeeklyByNode}}} stored procedure)
= Tools =

== {{{deneb_locate.py}}}: Locate a file using its nebulous storage key ==
 * The "nebulous storage key" is something like gpc1/sc_test_01/stack/MD08.V2/skycell.128/MD08.V2.skycell.128.stk.418540.ppStack.mdc
 * Equivalent of the {{{neb-ls -p}}} command but uses the different {{{DisksMonitoring}}} databases;
 * Is installed in {{{~ipp/local/bin}}} and can be executed as user {{{ipp}}};
 * {{{-h}}} to get help.
 
== {{{}}} TODO Statistics ==
 *