This wikipage gives some details about the "Files Monitoring" software.

Objectives

The objectives of the "Files Monitoring" software are:

• Make sure that nebulous doesn't lose any OTA
• Provide statistics about the storage nodes usage

Implementation

• The software is still installed and run as schastel. TODO: Move it to ippm
• The source code is in ~schastel/dev/FilesMonitoring. In the rest of this documentation, all paths are relative to it. TODO: SVN it
• On each storage node, every hour, the NodeRelated/refresh_table.py script is started (from the crontab file defined in NodeRelated/deployment/scripts/crontab.<node name>). To prevent cluster/nfs overwhelming, the scripts are not run at the same time. The crontab files as well as the deployment script can be generated from NodeRelated/deployment/generate_crontabs_and_deployment_script.py. Run it and follow its (last) instruction to deploy. Note that error messages are currently sent to schastel@ifa.hawaii.edu.
• Every 4 hours (see schastel crontab on ippc11), the Summary/snapshot.py script is run and takes snapshots, i.e. statistics, of each storage node database.

NodeRelated/refresh_table.py Details

User Doc

• This script runs on each storage node. Each storage node should run a number of instances on this script equal to the number of its storage partitions (i.e. 1 for the mhpcc nodes, 3 for the atrc and the stsci nodes).
• The script uses the local mysql server and its DisksMonitoring database
• If the script (resp. the mysql server) are overwhelming the host (nfs, I/O, swap...), FEEL FREE TO KILL (resp. STOP) THEM. If you stop the mysql server, think of restarting it. The script should be robust enough to deal with it in the worst case, it will send an e-mail to the MAILTO recipient defined in NodeRelated/deployment/scripts/crontab.<node name>.

Script Stages

• Constants are (should be) all defined in the Constants pseudo-class
• First, some initializations are performed (which disk is the script supposed to analyze, logging, ...)
• If the last analyzed directory has been analyzed less than Constants.max_refresh_interval (= 6 hours) seconds ago, stop the script (otherwise the script is likely still running or has been intentionally killed)
• Run an infinite loop
  • Get a directory to analyze (one of the 256 /export/<hostname>.<disk>/nebulous directories). Each directory has its own table: neb_<disk>_<directory name>, e.g. neb_1_a4 is associated to the /export/<hostname>.1/nebulous/a4 directory
  • Run updatedb on the directory to analyze (a faster implementation of updatedb/mlocate is installed)
  • Get all ins_id (nebulous instance id) from the FilesMonitoring database and the mlocate database:
  • Mark the files which are in the FilesMonitoring database but not in the mlocate database as deleted;
  • Add the files which are in the mlocate database but not in the FilesMonitoring database (since they are "new" files). New entries are undefined;
  • Optimize the related mysql table
  • Refresh the status of a maximum of Constants.limit_undefined (=10000) undefined entries using the NodeRelated/update_table_undefined.py script
  • Refresh the status of a maximum of Constants.limit (=1000) oldest entries using the NodeRelated/update_table.py script
  • Update the refresh_status information: set the last analyzed directory of the current disk to the current directory (timestamp is automatically updated).
  • Sleep for a while (Constants.sleep = 10 seconds )

Deployment

TODO:

NodeRelated/update_table.py Details

• The script depends on ipp.lifetime_templates

~schastel/local/share/python/ipp/lifetime_templates.py Details

See Cluster_Storage_Notes

Summary/refresh_table.py Details

Tools

deneb_locate.py: Locate a file using its nebulous storage key

• The "nebulous storage key" is something like gpc1/sc_test_01/stack/MD08.V2/skycell.128/MD08.V2.skycell.128.stk.418540.ppStack.mdc
• Equivalent of the neb-ls -p command but uses the different DisksMonitoring databases;
• Is installed in ~ipp/local/bin and can be executed as user ipp;
• -h to get help.

TODO Statistics