Index: /trunk/Nebulous-Server/docs/design.txt =================================================================== --- /trunk/Nebulous-Server/docs/design.txt (revision 5475) +++ /trunk/Nebulous-Server/docs/design.txt (revision 5476) @@ -1,5 +1,5 @@ # Copyright (C) 2004-2005 Joshua Hoblitt # -# $Id: design.txt,v 1.4 2005-09-14 03:24:45 jhoblitt Exp $ +# $Id: design.txt,v 1.5 2005-11-05 01:08:32 jhoblitt Exp $ Abstract @@ -7,56 +7,16 @@ Nebulous is a user-space distributed object (file) storage system. It is much -more a database of where objects are stored then a filesystem in the -traditional sense. Nebulous is considered distributed because of it's data -storage model, where objects are stored across a number of storage volumes and -may have redundant copies. While IPC is done in the traditional client/server +more of a database of objects locations then a file system in the traditional +sense. Nebulous is considered distributed because of it's data storage model, +where objects are stored across a number of storage volumes and may have one +or more redundant copies. While IPC is done in the traditional client/server model with a single[1] centralized server containing all storage object -meta-data. This system was designed, unlike most distributed filesystems, +meta-data. This system was designed, unlike most distributed storage system, specifically so that clients have the option of hosting one or more storage -volumes. Although, if you so choose, clients and storage volumes can reside -on independent hardware. Above all else, Nebulous is designed to make -efficient use of both disk and network bandwidth. - - -Overview --- - -There are 4 principle components required for a working Nebulous system; -server, client, data transport, and data storage. The Nebulous software -distribution provides only the server and the client software. Data -transport[2] and Data storage are left to 3rd party software packages. As -those functionalities are typically included with most operation system. -Instead, Nebulous build upon and benfits from advances made in those -technologies. - -Server --- - -The server is responsible for keeping track of storage objects, all instances -of that object, and enforcing locking semantics. Extensive logging and -tracing support is provided for debug and to allow for statics generation and -possible X optimization. - -Nebulous uses a centralized server model. This model was choosen because it -allows efficient X of storage object names. The current -'best' technique for a distributed metadata store is with distributed hash -tables. Unfortunately, no widely available DHT implementation allows efficent -I of key names. - - -[1] Multiple Nebulous servers should be possible via database replication. -[2] A future version of Nebulous may include it's own data transport layer -based on the WEB DAV protocol. - -Client <-> Server IPC --- - -The Nebulous Server and clients communicate via the SOAP protocol. Since SOAP -is platform independent this allows native Nebulous clients to be written in a -variety of languages. With in the Nebulous Server itself the IPC mechanism is -abstracted to allow ether multi-protocol support or complete replacement of -SOAP by another protocol. - -XXX SOAP namespace +volumes and that data may be stored and accessed locally on a node. This +allows network I/O to be dramatically reduced if the end user does simple data +locality optimizations. Although, if you so choose, clients and storage +volumes can reside on independent hardware. Above all else, Nebulous is +designed to make efficient use of both disk and network bandwidth. Data model @@ -105,4 +65,46 @@ XXX + +Components +-- + +There are 4 principle components required for a working Nebulous system; +server, client, data transport, and data storage. The Nebulous software +distribution provides only the server and the client software. Data +transport[2] and Data storage are left to 3rd party software packages. As +those functionalities are typically included with most operating systems. +Nebulous build upon and benefits from advances made in those technologies. + +Server +-- + +The server is responsible for keeping track of storage objects, all instances +of that object, and enforcing locking semantics. Extensive logging and +tracing support is provided for debug and to allow for statics generation and +possible X optimization. + +Nebulous uses a centralized server model. This model was choosen because it +allows efficient X of storage object names. The current +'best' technique for a distributed metadata store is with distributed hash +tables. Unfortunately, no widely available DHT implementation allows efficient +I of key names. + + +[1] Multiple Nebulous servers should be possible via database replication. +[2] A future version of Nebulous may include it's own data transport layer +based on the WEB DAV protocol. + +Client <-> Server IPC +-- + +The Nebulous Server and clients communicate via the SOAP protocol. Since SOAP +is platform independent this allows native Nebulous clients to be written in a +variety of languages. With in the Nebulous Server itself the IPC mechanism is +abstracted to allow ether multi-protocol support or complete replacement of +SOAP by another protocol. + +XXX SOAP namespace + + Storage volume state tracking. Index: /trunk/Nebulous/docs/design.txt =================================================================== --- /trunk/Nebulous/docs/design.txt (revision 5475) +++ /trunk/Nebulous/docs/design.txt (revision 5476) @@ -1,5 +1,5 @@ # Copyright (C) 2004-2005 Joshua Hoblitt # -# $Id: design.txt,v 1.4 2005-09-14 03:24:45 jhoblitt Exp $ +# $Id: design.txt,v 1.5 2005-11-05 01:08:32 jhoblitt Exp $ Abstract @@ -7,56 +7,16 @@ Nebulous is a user-space distributed object (file) storage system. It is much -more a database of where objects are stored then a filesystem in the -traditional sense. Nebulous is considered distributed because of it's data -storage model, where objects are stored across a number of storage volumes and -may have redundant copies. While IPC is done in the traditional client/server +more of a database of objects locations then a file system in the traditional +sense. Nebulous is considered distributed because of it's data storage model, +where objects are stored across a number of storage volumes and may have one +or more redundant copies. While IPC is done in the traditional client/server model with a single[1] centralized server containing all storage object -meta-data. This system was designed, unlike most distributed filesystems, +meta-data. This system was designed, unlike most distributed storage system, specifically so that clients have the option of hosting one or more storage -volumes. Although, if you so choose, clients and storage volumes can reside -on independent hardware. Above all else, Nebulous is designed to make -efficient use of both disk and network bandwidth. - - -Overview --- - -There are 4 principle components required for a working Nebulous system; -server, client, data transport, and data storage. The Nebulous software -distribution provides only the server and the client software. Data -transport[2] and Data storage are left to 3rd party software packages. As -those functionalities are typically included with most operation system. -Instead, Nebulous build upon and benfits from advances made in those -technologies. - -Server --- - -The server is responsible for keeping track of storage objects, all instances -of that object, and enforcing locking semantics. Extensive logging and -tracing support is provided for debug and to allow for statics generation and -possible X optimization. - -Nebulous uses a centralized server model. This model was choosen because it -allows efficient X of storage object names. The current -'best' technique for a distributed metadata store is with distributed hash -tables. Unfortunately, no widely available DHT implementation allows efficent -I of key names. - - -[1] Multiple Nebulous servers should be possible via database replication. -[2] A future version of Nebulous may include it's own data transport layer -based on the WEB DAV protocol. - -Client <-> Server IPC --- - -The Nebulous Server and clients communicate via the SOAP protocol. Since SOAP -is platform independent this allows native Nebulous clients to be written in a -variety of languages. With in the Nebulous Server itself the IPC mechanism is -abstracted to allow ether multi-protocol support or complete replacement of -SOAP by another protocol. - -XXX SOAP namespace +volumes and that data may be stored and accessed locally on a node. This +allows network I/O to be dramatically reduced if the end user does simple data +locality optimizations. Although, if you so choose, clients and storage +volumes can reside on independent hardware. Above all else, Nebulous is +designed to make efficient use of both disk and network bandwidth. Data model @@ -105,4 +65,46 @@ XXX + +Components +-- + +There are 4 principle components required for a working Nebulous system; +server, client, data transport, and data storage. The Nebulous software +distribution provides only the server and the client software. Data +transport[2] and Data storage are left to 3rd party software packages. As +those functionalities are typically included with most operating systems. +Nebulous build upon and benefits from advances made in those technologies. + +Server +-- + +The server is responsible for keeping track of storage objects, all instances +of that object, and enforcing locking semantics. Extensive logging and +tracing support is provided for debug and to allow for statics generation and +possible X optimization. + +Nebulous uses a centralized server model. This model was choosen because it +allows efficient X of storage object names. The current +'best' technique for a distributed metadata store is with distributed hash +tables. Unfortunately, no widely available DHT implementation allows efficient +I of key names. + + +[1] Multiple Nebulous servers should be possible via database replication. +[2] A future version of Nebulous may include it's own data transport layer +based on the WEB DAV protocol. + +Client <-> Server IPC +-- + +The Nebulous Server and clients communicate via the SOAP protocol. Since SOAP +is platform independent this allows native Nebulous clients to be written in a +variety of languages. With in the Nebulous Server itself the IPC mechanism is +abstracted to allow ether multi-protocol support or complete replacement of +SOAP by another protocol. + +XXX SOAP namespace + + Storage volume state tracking.