KR20130046441A - Method and system for extending data storage system functions - Google Patents

Abstract

A method and system for extending the functionality of a data storage system, the data storage system comprising data organization means and data storage resources, wherein the method incorporates data storage functions of the data storage system from other functions of the data storage system. Separating and transmitting at least some of the data storage functions to an intercepting system.

Description

METHOD AND SYSTEM FOR EXTENDING DATA STORAGE SYSTEM FUNCTIONS}

This application claims the priority of US Provisional Application 61 / 376,905, filed August 25, 2010, which is hereby incorporated by reference in its entirety.

The present invention relates generally to data storage systems, and more particularly to methods and systems for extending the functionality of data storage systems.

The file system on the computer allows the operating system of the computer to act as a trusted third party to enforce security and naming protocols between the communication processes, even if the communication processes are not active at the same time. One of the functions of the filesystem is to catalog and organize the data provided to it so that it can be retrieved later. To perform this function, the filesystem must manage storage resources. Typical storage resources appear in the filesystem in an object storage system, for example, as contiguous byte sequences, contiguous block sequences, or essentially as a key value store.

The filesystem is generally part of the operating system on a computer, but may exist as an extension of the operating system within an application or as a pure application accessed through a network connection using a client filesystem protocol. Various examples of these are known in the art. In all cases, the filesystem is designed to use a particular kind of storage resource, for example a disk, and will have restrictions associated with that storage resource. These limitations may be related to, for example, performance, capacity, parallelism, scalability, physical location, and the like. When a file system is not provided with functionality in advance, for example, the manipulation of stored data allows the file to apply other transformations including encryption, compression, replication, or context sensitive processing (data polymorphism). There are additional restrictions within the system itself. In this technology, context sensing generally refers to program features that change depending on what you are doing in the program. For example, context sensing helps to provide documentation of the specific features that the user is using and context or context sensing processing of the data allows the data to be processed differently depending on how or where you use the data.

Conventional systems and methods to address some of these limitations have been to use virtual devices to create virtual storage resources for the filesystem to be used. Because of the lack of available contexts that exist within the filesystem and are not passed to storage resources, this approach cannot provide context sensitive processing of the data, and also provides a view of the data that the filesystem sees versus the actual storage resources, e.g., full data storage. Limitations due to 1: 1 mapping of data on capacity cannot be removed.

It is therefore an object of the present invention to provide a novel system and method that extends the functionality of a data storage system, for example by extending the functionality of a data storage system to allow context sensitive data processing.

Summary of the Invention

According to an embodiment of the present invention, there is provided a method for extending the functionality of a data storage system, the data storage system comprising data organization means and data storage resources, wherein the method comprises data storage functions of the data storage system. A method is provided that includes separating from other functions of a storage system and transmitting at least some of the data storage functions to an intercepting system.

According to one aspect of the invention, the intercepting system is a complementary storage resource. The data organization means may be selected from a group comprising a file system, a key value store and a database.

According to another aspect of the invention, said separating is performed by providing interceptor means for communicating with said data organization means, said data storage resource, and an intercepting system, said interceptor means intercepting a filesystem operation to facilitate the operation of said operation. Determines whether a function should be handled by the data storage resource or by the intercepting system.

According to another aspect of the invention, the interceptor means intercepts a filesystem operation while the operation is still in context and before the operation is otherwise broken down into independent operations suitable for the data storage resource.

According to another aspect of the invention, at least another part of the data storage functions is maintained in the data storage system. According to another aspect of the invention, all of the data storage functions are transmitted to the intercepting system.

According to another aspect of the invention, the data storage system is selected from the group comprising a file system, a key value store, an object store and a network protocol.

According to another aspect of the invention, the data storage system is shared between multiple external interfaces.

According to another aspect of the invention, the interceptor means comprises a user space application program that cooperates with facilities of the operating system or file system. According to another aspect of the invention, the interceptor means comprises a file system protocol proxy application running on a network. According to another aspect of the invention, said interceptor means comprises a minifilter driver adapted to intercept filesystem operations in an operating system kernel.

According to another aspect of the invention, the intercepting system performs its functions using one or more complimentary storage resources, and the complementary storage resources are independent of the storage resource.

According to another aspect of the invention, the intercepting system implements capacity expansion of the data storage system. According to another aspect of the invention, the intercepting system improves the performance of the data storage system by changing one or more characteristics of the data on the storage resource. The one or more properties are preferably selected from the group comprising storage format, storage location and storage order.

According to another aspect of the invention, the method further comprises performing de-duplication by the intercepting system. According to another aspect of the invention, the method further comprises performing data polymorphism by the intercepting system. According to another aspect of the invention, the method further comprises implementing independent access control mechanisms for data by the intercepting system. According to another aspect of the invention, the method further comprises versioning data by the intercepting system.

According to another aspect of the invention, the method further comprises implementing one of the single or multilevel caching of the data storage system, wherein the implementing is performed by the interceptor means. According to another aspect of the invention, the method further comprises implementing locality optimization by pushing less used data to remote data storage systems and pulling more used data to adjacent data storage systems; The implementing step is performed by the interceptor means. The remote data storage systems include data storage systems that are physically remote or require more time to access.

According to another aspect of the present invention, the method further comprises implementing name based virtualization to enable invalid file names in the current data storage system by referencing data on another data storage system. And the implementing step is performed by the interceptor means.

According to another aspect of the present invention, the method further comprises implementing data backup and data replication, wherein the implementing is performed by the interceptor means.

According to another aspect of the invention, there is further included a step of implementing data virtualization comprising causing data under the same name to be physically located in different data storage systems, wherein the step of implementing is performed by the interceptor means. do.

According to another aspect of the present invention, the method further comprises providing separate capabilities to selected data in the interception system than at the storage resource, wherein the separate capabilities include performance characteristics, deduplication, data polymorphism, It is selected from a group that includes independent access control mechanisms, versioning, caching, locality, replication, and data virtualization. The selected data is preferably identified using a selection mechanism that employs one or more metadata pattern matching selected from the group comprising name, time stamp, size, history information, physical information and context information.

According to another embodiment of the present invention, there is provided a system for extending the functionality of a data storage system, wherein the data storage system includes data organization means and data storage resources, and the system includes data storage functions of the data storage system. A system is provided that includes separating means for separating from other functions of the storage system and means for transmitting at least some of the data storage functions to the intercepting system.

According to another aspect of this embodiment, the intercepting system may be a complementary storage resource.

According to another aspect of this embodiment, said separating means comprises an interceptor in communication with said data organization means, said data storage resource and said intercepting system, said interceptor intercepting a file system operation so that said function of said operation is performed by said data. And determine whether to be processed by a storage resource or by the intercepting system.

According to another aspect of this embodiment, the interceptor intercepts a filesystem operation while the operation is still in context and before the operation is otherwise broken down into independent operations suitable for the data storage resource.

According to another aspect of this embodiment, at least another part of the data storage functions is maintained in the data storage system. Alternatively, all of the data storage functions are sent to the intercepting system.

According to another aspect of this embodiment, the data storage system is selected from the group comprising a file system, a key value store, an object store and a network protocol.

According to another aspect of this embodiment, the data storage system is shared between multiple external interfaces.

According to another aspect of this embodiment, the interceptor includes a user space application program that cooperates with facilities of the operating system or file system.

According to another aspect of this embodiment, the interceptor comprises a file system protocol proxy application running on a network.

According to another aspect of this embodiment, the interceptor comprises a minifilter driver adapted to intercept filesystem operations in the operating system kernel.

According to another aspect of this embodiment, the intercepting system performs its functions using one or more complimentary storage resources, and the complementary storage resources are independent of the storage resource.

According to another aspect of this embodiment, the intercepting system implements capacity expansion of the data storage system.

According to another aspect of this embodiment, the intercepting system improves the performance of the data storage system by changing one or more characteristics of the data on the storage resource.

According to another aspect of this embodiment, said one or more characteristics are selected from the group comprising a storage format, a storage location and a storage order.

According to another aspect of this embodiment, the intercepting system is adapted to perform data de-duplication.

According to another aspect of this embodiment, the intercepting system is adapted to perform data polymorphism.

According to another aspect of this embodiment, the intercepting system is adapted to implement independent access control mechanisms for data.

According to another aspect of this embodiment, the intercepting system is adapted to perform data versioning.

According to another aspect of this embodiment, the interceptor is adapted to perform one of single or multilevel caching of the data storage system.

According to another aspect of this embodiment, the interceptor is adapted to perform locality optimization by pushing less used data to remote data storage systems and pulling more used data to adjacent data storage systems.

According to another aspect of this embodiment, the remote data storage systems include data storage systems that are physically remote or require more time to access.

According to another aspect of this embodiment, the interceptor is adapted to perform name based virtualization which causes invalid file names in the current data storage system to be valid by referring to data on other data storage systems.

According to another aspect of this embodiment, the interceptor is adapted to perform one of data backup and data replication.

According to another aspect of this embodiment, the interceptor is adapted to perform data virtualization comprising allowing data under the same name to be physically located in different data storage systems.

According to another aspect of this embodiment, the interception system includes performance improvement specific, data deduplication, data polymorphism, independent access control mechanisms, versioning, caching, locality, replication, and data virtualization. Adapted to perform one or more selected from the group.

According to another aspect of this embodiment, the data organization means is selected from the group comprising a file system, a key value store and a database.

Embodiments are described by way of example with reference to the accompanying drawings.
1 illustrates a high level architecture of a system according to the present invention.
2 illustrates a computer system in which the present invention may be used and / or implemented.
3 shows an embodiment of a method according to the invention.

The present invention provides novel systems and methods that affect all data related limitations, including but not limited to those that can be impacted using virtual storage resources. The present invention is directed to data storage systems other than file systems that provide a high level abstraction over actual storage resources as primary interfaces, such as databases and specific object databases, key value stores, certain network protocols and certain shared data systems. This functionality can be provided.

In addition, the present invention allows all traditional limitations of the data storage system to be altered without cooperation or modification of the file system. Thus, the present invention does not require a complete maintenance or implementation of a new file system, and the advantages and extended functions described herein require the development of a new file system, or on an existing system with or without the need for a new file system. To be implemented at the interface between and between new systems under development. Broadly, the present invention provides an intercepting system to be attached to a standing filesystem to selectively intercept filesystem operations while still containing contextual information before the operations are broken down into operations suitable for storage resources. That is, the file system includes the context operations provided by the computer's operating system and the instructions to perform them. When the file system interacts with storage resources, these context operations and commands are lost because the file system-storage resource interactions are only involved in retrieving, storing, and cataloging data. The intercepting system according to the present invention provides the functionality required by the file system at an interception point where the operation of the intercepting system is transparent to the file system. Thus, the present invention extends the functionality of the data storage resources and system by separating the storage responsibilities of the data storage system from other functions (file naming, locking, sharing, security, etc.) and optionally allowing actual storage tasks to be performed by individual components. It provides a method and system.

Referring to FIG. 1, one embodiment of the present invention is illustrated with a data storage system 100, an interceptor 200, and an intercepting system 300. Details of the preferred embodiment are described below in which various distributions of conventional data storage system functions are separated from the data storage system itself and divided to be performed by a data storage system or an intercepting system. The novel separation and subsequent distribution of functionality between the data storage system 100 and the intercepting system 300 are described herein in the art, particularly in a non-cooperative environment, between the data storage system 100 and the intercepting system 300. It is believed to be novel about providing a distribution. However, while the preferred embodiment of the present invention is directed to additional mechanisms for existing non-cooperative data storage systems, it may also be a structural feature that provides scalability to cooperative data storage systems to provide extended and enhanced functionality, Examples are described in more detail below.

Referring to FIG. 3, a step 310 of separating the data storage function of the data storage system from other functions of the data storage system, intercepting the file system operation to determine whether the function of the operation should be processed by the data storage resource. A method is provided for extending the functionality of a data storage system, including determining 315 whether it should be processed by and transmitting 320 at least a portion of the data storage function to an intercepting system. Alternatively, the intercepted data storage function continues with data storage resources (325).

The present invention generally functions within the context of a computer system and provides an extension of the data storage capabilities associated with a general computer system, an example of a typical computer system being shown in FIG. As shown, computer system 20 includes central processing unit (CPU) 24, random access memory (RAM) 28, input / output (I / O) interface 32, network interface 36, nonvolatile It has a number of physical and logical components, including storage 40, and a local bus 44 that allows the CPU 24 to communicate with other components. CPU 24 executes an operating system and a number of software systems. RAM 28 provides volatile storage that is responsive to CPU 24. I / O interface 32 allows input to be received from one or more devices, such as a keyboard, a mouse, and outputs information to devices such as displays and / or speakers. Network interface 36 allows communication with other systems. Nonvolatile storage 40 stores operating systems and programs. In operation of computer system 20, operating systems, programs, and data may be retrieved from nonvolatile storage 40 and placed in RAM 28 to facilitate execution.

Data storage system 100 is generally known in the art, and for the purposes of the present invention, is any machine, device, or device, capable of storing data using a given identifier (such as a file name), which is later stored in that identifier. Thereby retrieving at least a portion of the stored data on demand. The data storage system 100 includes a data organization means 105 and is a machine or module (non-volatile storage device in FIG. 2) referred to as a storage resource 110, which is operatively independent but optionally physically embedded. Data is stored in a primitive form according to the characteristics of the storage resource, from which the data storage system can later retrieve the stored data. The data storage system 100 can be shared between multiple external interfaces.

Representative examples of data organization means 105 include filesystems, key value stores, databases, and other machines layered on top of them, such as web caches and page files, and combinations thereof. Representative examples of storage resources 110 are physical disks, memory such as RAM, RAID arrays, paper tapes, documents, and the like. In a representative example of the preferred embodiment, storage system 100 may be a Windows ™ NTFS file system and storage resources may be hard disk drives.

The interceptor means 200 is a system for intercepting and processing data operations such as file system operations. According to the present invention, the interceptor means 200 separates the operation into appropriate context-free and more specific operations for the storage resource although the operation still has the context available, as is virtually the case for all storage resources. Intercept the data operation before it is done. The context may be, for example, application information including the source of the data request, details of the data to be requested, details of what the data is used for, and other information that requires knowledge of the current data operation.

According to the representative example described above, the interceptor means 200 is a minifilter or legacy filter driver that is designed and otherwise arranged to intercept filesystem operation at an appropriate level within the Windows ™ operating system kernel. Such drivers can be implemented independently or can be obtained commercially. In another example, the interception mechanism uses another application environment using an application level filesystem provider, such as FUSE (filesystem in user space), a stream (STREAMS) driver for network interception, or a physical transparent proxy machine. As an example, a built-in facility may be provided by, but not limited to, a filesystem stack. The interceptor means can also be a custom design software module that includes the functionality described herein. Interceptor means 200, which may be mixed with interceptors, may also be a filesystem protocol proxy application.

Interceptor means 200 manages the coupling between data storage system 100 and intercepting system 300. Intercepting system 300 is interested in certain operations of the data storage system. In a representative example, this may be all data input / output for redirecting data to another storage device. The intercepting system 300 constitutes an intercepting mechanism that delivers all data input / output operations laterally to the intercepting system 300, instead of delivering to the lower layers of the data storage system 100 as is traditionally done.

Upon receiving an intercepted operation, the intercepting system 300 has a number of options: (a) the intercepting system 300 does not process this operation and instructs the interceptor means 200 to continue to move as if there were no intercepts. Thereby determining that the interceptor means deliver the operation to a lower level of the data storage system 100 or (b) the intercepting system 300 transfers the operation to a lower level of the data storage system 100. May determine that there is a need to change the operation before the operation can be performed, or (c) the intercepting system 300 may determine to process the operation itself and take over responsibility for it, in which case the interception mechanism 200 It must respond to the upper and lower levels of the data storage system as appropriate for its operation, including error handling.

Intercepting system 300 may perform a variety of tasks, including those typically performed by complimentary storage resources, multimedia extenders, multimedia servers, home servers, application specific controllers, or storage resources. It can be any number of hardware devices, such as any similar system that can.

According to the present invention, an interceptor means 200 is provided in which the intercepting system can be responsible for operations that are intercepted in a transparent, ie non-identifiable manner by the data storage system. This allows existing filesystems to be extended with new functionality when data storage resources are separated from the filesystems. In contrast, conventional intermediary programs that perform arbitrary data interception functions, such as virus checkers, encryption shims, and the like, provide their functionality using native data storage systems.

The present invention also provides an existing data storage system that separates the location of stored data or metadata from the data storage system while maintaining the original semantics of the data storage system. Thus, while the data storage system is still responsible for any existing data or metadata already stored, the intercepting system may not be responsible for any new or changed data according to its policy unless the external semantics of the existing data storage system are changed. Alternatively, you can take responsibility for the metadata.

In the representative example described, the original filesystem is still responsible for all metadata, and the intercepting system takes over responsibility for some or all of the data. In particular, the present invention allows NTFS filesystem metadata, including directories, master file tables (MFTs) and the contents of each MFT entry, to be maintained on the filesystem's storage resources as if normally occurring, but the data itself was manipulated separately and inherently. Can be stored regardless of the configuration of the data storage system.

The present invention provides a number of advantages including providing a method for legacy, unaltered NTFS that supports modern file system features such as replication, de-duplication, caching, and pooling. Implementations, as well as features typically provided by storage resources (at the RAID level) and new unique features. The present invention makes this possible in a manner compatible with existing applications, since unchanged semantics exist in the external application interface of the filesystem. In other words, the filesystem remains unchanged and is available to the operating system without modification.

A conventional example of attempting to solve a similar problem as solved by the present invention is the Microsoft ™ Windows Home Server Project (WHS), a simple scalable data storage system intended for home users. WHS initially included a feature called drive extenders that extend NTFS on Windows systems using a file-based paradigm, that is, entire files can be placed on different destination NTFS filesystems, and the baseline (tombstone) ) Remains on the original NTFS filesystem, and this approach was not completely transparent to the application (the criteria that behaved differently from the original file) and the functionality that the drive extender attempted to provide initially used the virtual block storage resource model. Failed, due to the fact that it has been reimplemented by appearing to the filesystem as a storage resource, and then functionality has been completely removed from the WHS Another example of the original drive extender approach is described in the art, e. On the other hand, the present invention is evident in file system management. Ticks and metadata and to remain with the original storage resources, if necessary, the data itself is independently operated in any number of configurations of the original data storage system storage.

Although examples of such paragraphs have been provided for disk drive storage systems, the approach of the present invention also extends non-cooperative file systems to other non-cooperative data storage systems such as key value stores, object stores, various types of caches, databases, and the like. Can be used for Given the ability to locate data separately, differently, or even with additional processing compared to the design of the original data storage system, existing data storage systems can be used to provide many new features and corresponding functionality to applications. have. Applicants believe that the prior art does not disclose methods and systems that separate data storage responsibilities from other functions of the data storage system to extend the functionality of the data storage system beyond the limitations of the file system and the data storage system itself.

One of the advantages of including the systems and methods according to the present invention is that it can increase performance. The redirected data may be stored in a faster storage resource than the original data storage system uses, or may be stored in a manner that leads to faster reading and / or writing of the data, or both mechanisms may be used simultaneously. For example, the storage system's metadata may be maintained on the original storage resource, but the data itself is stored on the faster storage resource. In this example, the interceptor means instructs the filesystem to retrieve the metadata from the original storage resource and retrieve the data from the faster storage resource. When information and data are passed to the filesystem for processing by the operating system, the data appears to be retrieved directly from the original storage resource but retrieved faster. Performance may also be improved, for example, by caching data on secondary storage resources that may be faster and have improved other characteristics.

In one embodiment of the invention, the redirected data is stored in a sparse file on a faster storage resource. A sparse file is a file that contains holes in which data is not allocated on storage resources but is represented virtually. One of the problems with file caching systems is that they typically represent files cached in a space efficient format, requiring the creation of a data packing module that stores the actual data, and that the internal file location and the data for that location Is to move from one stored location to another. In order to cache data at the subfile level as needed, data that has not yet been retrieved from the original file must be represented virtually, not physically. Remember that using sparse files allows any filesystem that supports sparse files to provide this functionality, and to develop and provide this functionality within the present invention. Applicants believe that caching data in real files using sparse files is novel and those skilled in the art will recognize the ability to provide this internal functionality of the subfile cache using existing capabilities.

The invention can also be used in a data compression context, and the data can be compressed in a variety of ways that are not limited by the nature of the original data storage system. For example, there is normal stream compression or deduplication compression anywhere in the system where data is not stored again if it already exists. This type of data compression is generally not possible directly at the level of a data storage system using conventional techniques. Those skilled in the art will appreciate the individual benefits of providing data compression within any file system and data storage resource system regardless of the file system and data storage resource system used.

Another possible application is data polymorphism, which may appear differently depending on the context in which the data is intended to be used. While various data polymorphic applications are known in the art, they are generally performed at the application level and limited by the data storage system. According to the method of the present invention, data can be manipulated in an interception system, ie at the data storage system level, and presented to the file system in a context-changing manner so that no operating system and application resources or changes are required.

In the context of a document management system, the present invention provides extended semantics that allow redirected data to be versioned in a manner that maintains data storage system semantics and allows retrieval of existing versions of the data. Current systems require existing versions of data to be updated to include extended semantics or retrieved in a way that differentiates existing data from new data.

The redirected data input / output may also provide input to the distributed data storage system to control where the source data is located based on time related availability to the data storage system. In this regard, these features can be implemented at the data storage system level and will not appear differently to applications accessing the file system.

In a name virtualization context, data may no longer reside in the storage resources of the original data storage system, and may be located entirely elsewhere without the need to provide adaptability to the file system. Data virtualization is also enabled and one naming data no longer resides on the same storage resource, but may be distributed across multiple data storage systems and / or storage resources. In the context of data replication, redirected data input and output may drive the replication process to ensure redundancy of the data.

Although the above-described data and operations on data have known implementations, primarily at the application level, for example by using virtual machine implementations, the disadvantages of these implementations are described in the background of the present invention, and the present invention provides these features and functionality. This is to be provided at the data storage system level and thus to be independent of the file system or the operating system.

Implementing one of the single or multilevel caching of the data storage system by means of an interceptor, without departing from the spirit of the following claims, wherein the implementation step is performed by interceptor means, and less used data to the remote data storage system Various adaptations and implementations of the invention may be made, including implementing intensive optimization by pushing and pulling more used data into adjacent data storage systems. Remote data storage systems include data storage systems that are physically remote or require more time to access. In addition, interceptors may implement name based virtualization that makes file names invalid in current data storage systems by referencing data on other data storage systems. Data backup and replication are also possible by having the interceptor communicate directly with the filesystem. Data virtualization, including physically placing data under the same name in different data storage systems, is also possible at the data storage system level, and this implementation step is performed by interceptor means.

According to another aspect of the invention, the method further comprises providing individual capabilities for the selected data in the interception system rather than at the storage resource, wherein the individual capabilities comprise performance characteristics, de-duplication, data polymorphism, It is selected from the group including independent access control mechanisms, versioning, caching, locality, replication, and data virtualization. The selected data is preferably identified using a selection mechanism that employs one or more metadata pattern matching selected from the group comprising name, time stamp, size, history information, physical information and context information.

The above-described embodiments are intended as examples of the invention and modifications and variations are possible by those skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims (59)

As a way to extend the functionality of the data storage system,
The data storage system comprises data organization means and data storage resources, the method further comprising separating data storage functions of the data storage system from other functions of the data storage system and intercepting at least some of the data storage functions. Transmitting to a receiving system. The method of claim 1, wherein the intercepting system comprises a complementary storage resource. 2. The method of claim 1, wherein said separating is performed by providing interceptor means for communicating with said data organization means, said data storage resource, and said intercepting system, said interceptor means intercepting a filesystem operation to function of said operation. Determining whether this should be processed by the data storage resource or by the intercepting system. 4. The method of claim 3 wherein the interceptor means intercepts a filesystem operation while the operation is still in context and before the operation is decomposed into independent operations suitable for the data storage resource. The method of claim 1, wherein at least another portion of the data storage functions is maintained in the data storage system. The method of claim 1, wherein all of the data storage functions are transmitted to the intercepting system. The method of claim 1, wherein the data storage system is selected from the group comprising a file system, a key value store, an object store, and a network protocol. The method of claim 1, wherein the data storage system is shared among multiple external interfaces. 4. The method of claim 3, wherein the interceptor means comprises a user space application program that cooperates with facilities of an operating system or file system. 4. The method of claim 3, wherein the interceptor means comprises a filesystem protocol proxy application running on a network. 4. The method of claim 3, wherein the interceptor means comprises a minifilter driver configured to intercept filesystem operations in an operating system kernel. 4. The method of claim 3, wherein the intercepting system performs its functions using one or more complimentary storage resources, wherein the complimentary storage resources are independent of the storage resource. The method of claim 1, wherein the intercepting system implements capacity expansion of the data storage system. The method of claim 1, wherein the intercepting system improves the performance of the data storage system by changing one or more characteristics of the data on the storage resource or using a complementary storage resource. The method of claim 14, wherein the one or more characteristics are selected from the group comprising a storage format, a storage location, and a storage order. The method of claim 1, further comprising performing de-duplication by the intercepting system. 2. The method of claim 1, further comprising performing data polymorphism by the intercepting system. 2. The method of claim 1, further comprising implementing independent access control mechanisms for data by the intercepting system. The method of claim 1, further comprising versioning data by the intercepting system. 13. The method of claim 12, further comprising implementing one of single or multilevel caching of the data storage system, wherein the implementing is performed by the interceptor means. 13. The method of claim 12, wherein the complementary storage resource is an operating system memory cache that is normally used to cache data. 13. The method of claim 12, further comprising implementing locality optimization by pushing less used data to remote data storage systems and pulling more used data to adjacent storage resources or data storage systems. , Wherein said implementing step is performed by said interceptor means. 22. The method of claim 21 wherein the remote data storage systems include data storage systems that are physically remote or require more time to access. 13. The method of claim 12, further comprising implementing name based virtualization that makes invalid file names appear valid in the current data storage system by referencing data on other data storage systems. Implementing is performed by the interceptor means. 13. The method of claim 12, further comprising implementing one of data backup and data replication, wherein the implementing is performed by the interceptor means. 13. The method of claim 12, further comprising implementing data virtualization comprising allowing data under the same name to be physically located in different data storage systems, wherein the implementing is performed by the interceptor means. Way. 2. The method of claim 1, further comprising providing separate capabilities to selected data in the interception system than at the storage resource, wherein the separate capabilities are performance characteristic, deduplication, data polymorphism, independent access control mechanism. , Versioning, caching, locality, replication, and data virtualization. 27. The method of claim 26, wherein the selected data is identified using a selection mechanism that employs one or more metadata pattern matching selected from the group comprising name, time stamp, size, history information, physical information, and context information. The method of claim 1 wherein said data organization means is selected from the group comprising a filesystem, a key value store and a database. 13. The method of claim 12, wherein the complementary storage resource is a sparse file. As a system to extend the function of the data storage system,
The data storage system comprises data organization means and data storage resources, the system separating at least some of the data storage functions and separating means for separating the data storage functions of the data storage system from other functions of the data storage system. Means for transmitting to the intercepting system. 32. The system of claim 31, wherein the intercepting system comprises a complementary storage resource. 32. The apparatus of claim 31, wherein said separating means comprises an interceptor in communication with said data organizing means, said data storage resource and said intercepting system, said interceptor intercepting a file system operation such that the function of said operation is such that said data storage resource. And to determine whether to be processed by or by the intercepting system. 34. The system of claim 33, wherein the interceptor intercepts filesystem operations while the operations are still in context and before otherwise decomposing into independent operations suitable for the data storage resource. 32. The system of claim 31, wherein at least another portion of the data storage functions is maintained in the data storage system. 32. The system of claim 31 wherein all of the data storage functions are transmitted to the intercepting system. 32. The system of claim 31 wherein the data storage system is selected from the group comprising a file system, a key value store, an object store and a network protocol. 32. The system of claim 31 wherein the data storage system is shared among a plurality of external interfaces. 34. The system of claim 33, wherein the interceptor comprises a user space application program that cooperates with an operating system or facility of a file system. 34. The system of claim 33, wherein the interceptor comprises a file system protocol proxy application running on a network. 34. The system of claim 33, wherein the interceptor comprises a minifilter or legacy filter driver configured to intercept filesystem operations in an operating system kernel. 34. The system of claim 33, wherein the intercepting system uses one or more complimentary storage resources to perform their functions, and the complementary storage resources are independent of the storage resources. 34. The system of claim 33 wherein the intercepting system implements capacity expansion of the data storage system. 34. The system of claim 33, wherein the intercepting system improves the performance of the data storage system by changing one or more characteristics of the data on the storage resource. 34. The system of claim 33, wherein the one or more characteristics are selected from the group comprising a storage format, a storage location, and a storage order. 32. The system of claim 31, wherein the intercepting system is configured to perform data de-duplication. 32. The system of claim 31, wherein the intercepting system is configured to perform data polymorphism. 32. The system of claim 31, wherein the intercepting system is configured to implement independent access control mechanisms for data. 32. The system of claim 31, wherein the intercepting system is configured to perform data versioning. 34. The system of claim 33, wherein the interceptor is configured to perform one of single or multilevel caching of the data storage system. 34. The system of claim 33 wherein the complementary storage resource is an operating system memory cache that is normally used to cache data. 34. The method of claim 33, wherein the interceptor is configured to perform locality optimization by pushing less used data to remote data storage systems and pulling more used data to adjacent storage resources or data storage systems. system. 53. The system of claim 51, wherein the remote data storage systems comprise data storage systems that are physically remote or require more time to access. 34. The system of claim 33, wherein the interceptor is configured to perform name based virtualization such that invalid file names in a current data storage system appear valid by referring to data on other data storage systems. 34. The system of claim 33, wherein the interceptor is configured to perform one of data backup and data replication. 34. The system of claim 33, wherein the interceptor is configured to perform data virtualization comprising allowing data under the same name to be physically located in different data storage systems. 32. The system of claim 31, wherein the interception system is selected from the group comprising performance improvement characteristics, deduplication, data polymorphism, independent access control mechanisms, versioning, caching, locality, replication, and data virtualization. A system that is configured to perform one or more of the following. 32. The system of claim 31, wherein said data organization means is selected from the group comprising a file system, a key value store and a database. 43. The system of claim 42, wherein the complementary storage resource is a sparse file.

Images