CN114466015A - Data storage system and method based on multi-cloud architecture - Google Patents
Data storage system and method based on multi-cloud architecture Download PDFInfo
- Publication number
- CN114466015A CN114466015A CN202210088806.5A CN202210088806A CN114466015A CN 114466015 A CN114466015 A CN 114466015A CN 202210088806 A CN202210088806 A CN 202210088806A CN 114466015 A CN114466015 A CN 114466015A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- cloud
- information
- file data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 34
- 238000000034 method Methods 0.000 title claims abstract description 30
- 238000007726 management method Methods 0.000 claims abstract description 49
- 230000005540 biological transmission Effects 0.000 claims abstract description 19
- 238000012545 processing Methods 0.000 claims description 33
- 230000011218 segmentation Effects 0.000 claims description 19
- 238000000605 extraction Methods 0.000 claims description 6
- 238000000638 solvent extraction Methods 0.000 claims description 6
- 238000012546 transfer Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 6
- 239000000284 extract Substances 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 230000009977 dual effect Effects 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 241001522296 Erithacus rubecula Species 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Storage Device Security (AREA)
Abstract
The invention discloses a data storage system and a method based on a multi-cloud architecture, wherein the system comprises: a user access layer, a storage management layer, and a data transfer layer. The user access layer is used for identity authentication to realize safe access; the storage management layer is used for integrating API interfaces provided by different cloud service providers to form a uniform virtual cloud disk operation interface; the data service layer is used for dynamically dividing the file data and storing the meta information of the file data in a local database after the division is finished; the data transmission layer is used for conducting block encryption on the file data and uploading the file database blocks to the corresponding cloud based on a load balancing algorithm. According to the file data encryption method and system, the file data can be uploaded to the cloud after being encrypted, the safety of the data is guaranteed, and the meta information of the file data is stored in the local database, so that the control right of the local database on the file data is guaranteed, and the safety of the data is further guaranteed.
Description
Technical Field
The invention relates to the technical field of data security storage, in particular to a data storage system and method based on a multi-cloud architecture.
Background
Data are stored in a third-party cloud service manufacturer in a public cloud environment, data security is guaranteed by the cloud service manufacturer, and the transparency of security service of the third-party cloud service manufacturer brings users to be untrusty of cloud storage security. The user has complex operation, low efficiency and poor usability in the uploading and downloading process by compressing the encrypted file. The private cloud mode is complex to build, high in cost and not applicable to small and medium-scale enterprise organizations. Therefore, the existing technical scheme cannot effectively ensure the reliability and integrity of the data when the data is stored.
Thus, there is a need for improvements and enhancements in the art.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a data storage system and method based on a multi-cloud architecture, aiming at solving the problem that the reliability and integrity of data cannot be effectively ensured when the data is stored in the prior art.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
in a first aspect, the present invention provides a data storage system based on a multi-cloud architecture, wherein the system includes:
the user access layer is used for carrying out identity authentication and realizing safe access;
the system comprises a storage management layer, a cloud disk management layer and a cloud disk management layer, wherein the storage management layer is used for integrating AP I interfaces provided by different cloud service providers to form a uniform virtual cloud disk operation interface;
the data service layer is used for dynamically segmenting file data and storing the meta information of the file data in a local database after segmentation is finished;
and the data transmission layer is used for carrying out block encryption on the file data and uploading the file database blocks to a corresponding cloud terminal based on a load balancing algorithm.
In one implementation, the data service layer includes:
the file information extraction module is used for extracting file information from the file data and determining the data volume corresponding to the file information according to the file information;
the segmentation judging module is used for judging whether to segment the file data or not based on the comparison between the data volume and a threshold value;
and the dynamic segmentation module is used for segmenting the file data to obtain a plurality of data blocks and segmentation paths and generate a hash value.
In one implementation, the data service layer further includes:
and the meta information management module is used for taking the parameters generated after the file data are divided as meta information and storing the meta information in a local database.
In one implementation, the data service layer further includes:
and the redundancy block processing module is used for performing redundancy coding processing on the data blocks obtained after the file data is divided.
In one implementation, the data transport layer includes:
the load balancing processing module is used for determining a cloud end of the data block subjected to the redundant coding processing based on a scheduling strategy of load balancing;
and the data encryption processing module is used for encrypting the data block and feeding the key information back to the meta-information management module so as to feed the key information back to the local database through the meta-information management module.
In a second aspect, an embodiment of the present invention further provides a data storage method based on the data storage system based on the multi-cloud architecture in any one of the foregoing schemes, where the method includes:
acquiring user identity information and performing identity authentication;
loading a virtual cloud disk operation interface when the user identity information is determined to be legal;
dynamically dividing file data based on the virtual cloud disk operation interface, and storing meta information of the file data in a local database after division;
and carrying out block encryption on the file data, and uploading the file database blocks to a corresponding cloud terminal based on a load balancing algorithm.
In one implementation, the dynamically partitioning file data based on the virtual cloud disk operation interface includes:
extracting file information from the file data, and determining the data volume corresponding to the file information according to the file information;
comparing the data quantity with a threshold value, and judging whether to divide the file data;
when the division is determined, the file data is divided to obtain a plurality of data blocks and division paths, and a hash value is generated.
In one implementation, the saving the meta information of the file data in a local database after the splitting includes:
and taking the parameters generated after the file data is divided as meta-information, and storing the meta-information in a local database.
In one implementation, the storing the meta information of the file data in a local database after the splitting further includes:
and carrying out redundancy coding processing on the data blocks obtained after the file data is divided.
In one implementation, the block-wise encrypting the file data and block-wise uploading the file database to a corresponding cloud based on a load balancing algorithm includes:
determining a cloud end for the data block subjected to the redundant coding processing based on a scheduling strategy of load balancing;
and encrypting the data block, and feeding the key information back to the meta-information management module so as to feed the key information back to the local database by implementing the meta-information management module.
Has the advantages that: compared with the prior art, the invention provides a data storage method based on a multi-cloud architecture, and the system comprises the following steps: a user access layer, a storage management layer, and a data transfer layer. The user access layer is used for identity authentication to realize safe access; the storage management layer is used for integrating AP I interfaces provided by different cloud service providers to form a uniform virtual cloud disk operation interface; the data service layer is used for dynamically dividing the file data and storing the meta information of the file data in a local database after the division is finished; the data transmission layer is used for conducting block encryption on the file data and uploading the file database blocks to the corresponding cloud based on a load balancing algorithm. According to the file data encryption method and system, the file data can be uploaded to the cloud after being encrypted, the safety of the data is guaranteed, and the meta information of the file data is stored in the local database, so that the control right of the local database on the file data is guaranteed, and the safety of the data is further guaranteed.
Drawings
Fig. 1 is an overall framework diagram of a distributed multi-cluster data storage system according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a data service layer in a distributed multi-cluster data storage system according to an embodiment of the present invention.
Fig. 3 is a functional diagram of a structure of a data transport layer in a distributed multi-cluster data storage system according to an embodiment of the present invention.
Fig. 4 is a flowchart of a distributed multi-cluster data storage method according to a preferred embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Data are stored in a third-party cloud service manufacturer in a public cloud environment, data security is guaranteed by the cloud service manufacturer, and the transparency of security service of the third-party cloud service manufacturer brings users to be untrusty of cloud storage security. The user has complex operation, low efficiency and poor usability in the uploading and downloading process by compressing the encrypted file. The private cloud mode is complex to build, high in cost and not applicable to small and medium-scale enterprise organizations. Therefore, the existing technical scheme cannot effectively ensure the reliability and integrity of the data when the data is stored.
For this purpose, the present embodiment provides a data storage system based on a multi-cloud architecture, and specifically as shown in fig. 1, the data storage system based on a multi-cloud architecture in the present embodiment includes a user access layer, a storage management layer, and a data transmission layer. In this embodiment, the user side performs data storage and upload with a multi-cloud-end vendor through the data storage system. The user side in this embodiment may be a mobile side (such as a mobile phone), a PC side, or a dual local database. And the user side logs in the user access layer through the multiple cloud accounts, and performs identity authentication based on the user access layer to realize safe access. As shown in fig. 2, the user access layer implements a secure access login function of the user, and the user autonomously manages login accounts of the cloud terminals. The login key is stored in the local equipment (such as a mobile terminal, a PC terminal and the like) of the user, and the local access is carried out by the user during login, so that the safety and the privacy of the user are ensured. When the user provides a legal related cloud storage login key, the virtual network disk module is loaded, and the function is delivered to the storage management layer and the data service layer. The storage management layer in this embodiment is used to integrate API interfaces provided by different cloud service providers to form a unified virtual cloud disk operation interface, so that a user can upload and download files, delete files, create directories, share files and other basic operations, and the intuitive storage space management interface is more convenient for the user to operate. The design essence is that a plurality of actual storage cloud ends are abstracted into a uniform storage resource pool based on a virtual file system concept. And storing the file metafile information into a local database through an ORM technology, and establishing a directory according to the metafile information pointing to the file. Including file name, creation time, file size, etc. Specifically, the storage management layer in this embodiment provides a Restful interface by building a front end interface on the Web end through, for example, a unified interface for liblog to access a cloud computing service, and supports method operations such as Upload (), Download (), log (), File _ operation (), Dir _ operation (), User _ operation (), and the like. And the storage management layer displays basic operation pages such as uploading and downloading, user login information management pages, cloud load utilization rate monitoring pages and the like on a front-end interface, so that the user operation management is facilitated. In this embodiment, the storage management layer integrates API interfaces provided by different cloud manufacturers to form a unified management operation platform interface.
The data service layer in this embodiment is configured to dynamically partition file data, and store meta information of the file data in a local database after the partition is completed. In specific implementation, the data service layer in this embodiment includes: the device comprises a file information extraction module, a segmentation judgment module, a dynamic segmentation module, a meta-information management module and a redundant block processing module. Specifically, the file information extraction module in this embodiment is configured to extract file information from the file data, and determine a data size corresponding to the file information according to the file information. The segmentation judging module is used for comparing the data volume with a threshold value and judging whether to segment the file data. The dynamic segmentation module is used for segmenting the file data to obtain a plurality of data blocks and segmentation paths, and generating a hash value. And the meta-information management module is used for taking the parameters generated after the file data are divided as meta-information and storing the meta-information in a local database. And the redundant block processing module is used for carrying out redundant coding processing on the data blocks obtained after the file data is segmented.
As shown in fig. 1 and 2, the data service layer first extracts file information based on the file information extraction module and extracts a byte stream from the file information, resulting in a file size (i.e., a data amount of the file information). And then, partitioning the file data by adopting a dynamic partitioning algorithm, wherein the size of the file is firstly set as a parameter in the algorithm, and when the size of the file exceeds a set threshold, the file is partitioned into pieces, wherein the size of each split _ n does not exceed the set threshold. Specifically, the data service layer in this embodiment compares the file size with a threshold N based on the segmentation judgment module, and if the file size is larger than the threshold N, the file data needs to be segmented. At this time, the file data is divided based on the dynamic division module, a plurality of data blocks and a division path are obtained, a hash value is generated, and at the time of division, a pointer pos _ cur at a division line is recorded. When merging, the files are merged according to the pointer and the file path, and the integrity of the files is verified through the hash value generated by the original files. And when the file size is lower than the threshold value N, directly encrypting without cutting in view of performance. In this embodiment, the parameters generated by the file segmentation are used as meta information, that is, different segmentation strategies are adopted for data with different sizes, and when a user adds or deletes file data, the corresponding algorithm is dynamically adjusted and selected, so that the performance of the system is improved, and the waste of small files on the transmission efficiency is avoided. And the RS erasure codes realize an n + m redundancy strategy, and n blocks of data are processed by an algorithm to obtain m blocks of redundancy data. When any n blocks have errors, the lost file can be recovered according to the erasure codes through an algorithm, so that the reliability of the data is ensured. The meta information of the file data is generated and stored in a user local database or related equipment by a meta information management module, and the control right of the data is ensured to be local. When the integrity of the file is lost and the timestamp is not updated, the loss of part of the file blocks is indicated, and the redundant processing module is immediately started. The module for redundancy processing performs redundancy coding processing on the data block obtained by dividing the file data.
Therefore, in the embodiment, the synchronous file information can be maintained by transmitting the file information through the data service layer. The file management method includes that locally cached meta-information is compared, and when file information is changed in an adding and deleting mode, files in a virtual cloud disk corresponding to timestamps generated by the meta-information and the file information are correspondingly changed. When the meta information changes and the timestamp is updated, it indicates that the file was changed by the user and the update is synchronized. And generating a chart corresponding to the cloud occupancy rate condition according to the file _ path and the file _ size, and feeding the chart back to a front-end page of the user access layer.
The data transmission layer in this embodiment is configured to perform block encryption on the file data, and upload the file database in blocks to a corresponding cloud based on a load balancing algorithm. In specific implementation, the data transmission layer includes: the device comprises a load balancing processing module and a data encryption processing module. Specifically, the load balancing processing module is configured to determine a cloud end for the data block subjected to the redundancy coding processing based on a scheduling policy of load balancing. And the data encryption processing module is used for encrypting the data block and feeding back the key information to the meta-information management module so as to feed back the key information to the local database through the meta-information management module.
As shown in fig. 3, the data transmission layer design of the present embodiment implements a block encryption transmission mechanism for files. The confidentiality of data in the process of transmitting the data from a user to the cloud is guaranteed, and the data control right of the user is guaranteed by encrypting the block file. And by combining a load balancing processing module of the data service layer, the load balancing algorithm design considers the transmission performance difference of different cloud services so as to ensure the efficiency of the file in the uploading process. And carrying out encryption operation on the data in the process of uploading the file block to the cloud end to form a ciphertext and ensure the confidentiality of the data. The load balancing processing module is transparent to a user and is realized by adopting a file-driving-based mode, when the user performs I/O request operation through a virtual cloud disk operation interface and a corresponding system interface, the system establishes connection with each cloud server after receiving a message instruction, calls each module of a data transmission layer to realize parallel transmission, the user uploads a file to be processed through encryption, segmentation and parallel transmission, and downloads the file to be processed through integrity verification, decryption, combination, parallel transmission and other operations and finally provided for the user file data in a plaintext form. The data encryption processing module is divided into symmetric encryption and asymmetric encryption. Common symmetric encryption algorithms: DES, AES, 3DES, etc. Common asymmetric encryption algorithms: RSA, ECC, etc. Asymmetric encryption, while highly confidential, is inefficient. Because the fragmentation mechanism and the metafile data are stored in the local of the user, the user has certain confidentiality, and the AES-256 symmetric encryption method is adopted from the aspect of efficiency and practical application. In this embodiment, the scheduling policy of load balancing based on the LVS (Linux Virtual Server, that is, Linux Virtual Server) includes Weighted round robin, minimum connection, local minimum link, shortest expected delay, and the like, and because there is a difference in the quality of service provided by each cloud, the system adopts a Weighted minimum connection scheduling (WLC) policy. The service quality parameters comprise space utilization rate, connection utilization rate, throughput rate, average response time, space price ratio and the like which are used as performance parameter indexes, and service performance is evaluated and distributed by setting a weight parameter Wn according to the indexes. Compared with the traditional WLC algorithm, the system is dynamic in acquisition of the applicable scene service information, excessive cost is caused by excessive parameters, and the weight value can change along with the service quality of each cloud. And taking the throughput rate, the space utilization rate and the space price ratio as main parameters to generate a weight parameter Wn. The rationality of cloud resource allocation is better guaranteed through an improved WLC algorithm.
Therefore, the system of the embodiment can realize the storage of the key data of the user in the public cloud environment. Such as medical records databases, system history information, financial data, etc., which are dynamically generated and have a certain data size and sharing requirements. The proposal effectively and feasibly protects the security of the user data in the public cloud environment. And the problem that the storage service is unavailable due to the downtime of a third-party service is avoided, and the data availability is ensured. In addition, the problem of difficult data migration caused by operator locking can be avoided. As more and more data is stored on the mesh disk, the greater the dependency of a business or organization on the mesh disk, the more difficult it is to switch to other mesh disk services. And when data is migrated, it is difficult to migrate a large number of files from one network disk to another network disk in a short time. According to the proposal, the application scene and the performance requirement of the user are considered by combining the multi-cloud architecture with the load balancing related technology, so that the data controllability is ensured. Most importantly, the system of the embodiment guarantees the data control right and confidentiality of the user. And the encryption and decryption mode and the storage mode of the public cloud third-party service form in the storage process are completely ensured by a storage provider. The control right of data storage and transmission is changed from a third-party storage provider to a user by the technologies of dynamic segmentation, encryption distribution transmission, redundancy recovery strategy and metafile information generation management.
Based on the foregoing embodiment, the present embodiment further provides a data storage method based on the data storage system based on the foregoing embodiment, and specifically as shown in fig. 4, the data storage method of the present embodiment includes the following steps:
and S100, acquiring user identity information and performing identity authentication.
Specifically, the user side stores and uploads data with a multi-cloud-end manufacturer through the data storage system. The user side in this embodiment may be a mobile side (such as a mobile phone), a PC side, or a dual local database. And the user side logs in the user access layer through the multi-cloud account, and performs identity authentication based on the user access layer to realize safe access.
And S200, loading a virtual cloud disk operation interface when the user identity information is determined to be legal.
When the user provides a legal related cloud storage login key, the virtual network disk module is loaded, and the function is delivered to the storage management layer and the data service layer. The storage management layer in this embodiment is used to integrate API interfaces provided by different cloud service providers to form a unified virtual cloud disk operation interface, so that a user can upload and download files, delete files, create directories, share files and other basic operations, and the intuitive storage space management interface is more convenient for the user to operate.
And step S300, dynamically dividing the file data based on the virtual cloud disk operation interface, and storing the meta information of the file data in a local database after division.
Specifically, the present embodiment extracts file information based on the file information extraction module, and extracts a byte stream from the file information, to obtain a file size (i.e., a data size of the file information). And then, partitioning the file data by adopting a dynamic partitioning algorithm, wherein the size of the file is firstly set as a parameter in the algorithm, and when the size of the file exceeds a set threshold, the file is partitioned into pieces, wherein the size of each split _ n does not exceed the set threshold. Specifically, the data service layer in this embodiment compares the file size with a threshold N based on the segmentation judgment module, and if the file size is larger than the threshold N, the file data needs to be segmented. At this time, the file data is divided based on the dynamic division module, a plurality of data blocks and a division path are obtained, a hash value is generated, and at the time of division, a pointer pos _ cur at a division line is recorded. When merging, the files are merged according to the pointer and the file path, and the integrity of the files is verified through the hash value generated by the original files. And when the file size is lower than the threshold value N, directly encrypting without cutting in view of performance. In this embodiment, the meta information of the file data is generated and stored in the user local database or the related device by the meta information management module, so as to ensure that the control right of the data is local. When the integrity of the file is lost and the timestamp is not updated, the loss of part of the file blocks is indicated, and the redundant processing module is immediately started. The module for redundancy processing performs redundancy coding processing on the data block obtained by dividing the file data.
And S400, block encryption is carried out on the file data, and the file database is uploaded to a corresponding cloud terminal in a block mode based on a load balancing algorithm.
Specifically, in this embodiment, the cloud is determined based on a scheduling policy of load balancing for the data block subjected to the redundant coding processing. Then, the data block is encrypted, and the key information is fed back to the meta-information management module, so that the key information is fed back to the local database by implementing the meta-information management module.
The implementation principle of each step in the data storage method based on the multi-cloud architecture in this embodiment is the same as the implementation principle of each module in the data storage system based on the multi-cloud architecture in the above embodiment, and therefore, the description is omitted here.
Therefore, compared with the existing cloud storage in the form of a multi-cloud architecture, the embodiment adopts a meta-information management mode to ensure the data control right of the user. The user saves the metadata information in a hash value form through a local double-copy database, and the problems that the user worries about the middleware and the hidden danger that the metadata is lost are solved. The embodiment considers the actual performance requirement and application scenario. The load balancing processing module framework guarantees higher throughput rate and lower delay by considering factors such as network bandwidth, network delay, price-to-space ratio of cloud services and the like. And different encryption strategies are adopted according to different sizes of the files, and the efficiency problem in encryption transmission is optimized according to the fragmentation mechanism of the files. The embodiment provides a reasonable and convenient mode for user operation and management. And the system is added with a data integrity detection function, when data is unavailable due to single-point failure at a certain cloud, lost block recovery is carried out in time according to a redundancy erasure code recovery algorithm, lost information can be fed back to a user in time, and service continuity and data reliability are guaranteed. And visual interfaces such as monitoring and configuration are added to the front-end interface, so that a user can conveniently view and adjust the storage strategy in real time.
In summary, the present invention discloses a data storage system and method based on a multi-cloud architecture, the system includes: a user access layer, a storage management layer, and a data transfer layer. The user access layer is used for identity authentication to realize safe access; the storage management layer is used for integrating API interfaces provided by different cloud service providers to form a uniform virtual cloud disk operation interface; the data service layer is used for dynamically dividing the file data and storing the meta information of the file data in a local database after the division is finished; the data transmission layer is used for conducting block encryption on the file data and uploading the file database blocks to the corresponding cloud based on a load balancing algorithm. According to the file data encryption method and system, the file data can be uploaded to the cloud after being encrypted, the safety of the data is guaranteed, and the meta information of the file data is stored in the local database, so that the control right of the local database on the file data is guaranteed, and the safety of the data is further guaranteed.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A data storage system based on a multi-cloud architecture, the system comprising:
the user access layer is used for carrying out identity authentication and realizing safe access;
the system comprises a storage management layer, a cloud disk management layer and a cloud disk management layer, wherein the storage management layer is used for integrating API interfaces provided by different cloud service providers to form a uniform virtual cloud disk operation interface;
the data service layer is used for dynamically segmenting file data and storing the meta information of the file data in a local database after segmentation is finished;
and the data transmission layer is used for carrying out block encryption on the file data and uploading the file database blocks to a corresponding cloud terminal based on a load balancing algorithm.
2. The multi-cloud architecture based data storage system of claim 1, wherein said data service layer comprises:
the file information extraction module is used for extracting file information from the file data and determining the data volume corresponding to the file information according to the file information;
the segmentation judging module is used for judging whether to segment the file data or not based on the comparison between the data volume and a threshold value;
and the dynamic segmentation module is used for segmenting the file data to obtain a plurality of data blocks and segmentation paths and generate a hash value.
3. The multi-cloud architecture based data storage system of claim 2, wherein said data service layer further comprises:
and the meta information management module is used for taking the parameters generated after the file data are divided as meta information and storing the meta information in a local database.
4. The multi-cloud architecture based data storage system of claim 3, wherein said data service layer further comprises:
and the redundancy block processing module is used for performing redundancy coding processing on the data blocks obtained after the file data is divided.
5. The multi-cloud architecture based data storage system of claim 4, wherein said data transport layer comprises:
the load balancing processing module is used for determining a cloud end of the data block subjected to the redundancy coding processing based on a scheduling strategy of load balancing;
and the data encryption processing module is used for encrypting the data block and feeding the key information back to the meta-information management module so as to feed the key information back to the local database through the meta-information management module.
6. A data storage method based on the multi-cloud architecture based data storage system of any one of claims 1 to 5, wherein the method comprises:
acquiring user identity information and performing identity authentication;
loading a virtual cloud disk operation interface when the user identity information is determined to be legal;
dynamically dividing file data based on the virtual cloud disk operation interface, and storing meta information of the file data in a local database after division;
and carrying out block encryption on the file data, and uploading the file database blocks to a corresponding cloud terminal based on a load balancing algorithm.
7. The data storage method based on the multi-cloud architecture of claim 6, wherein the dynamically partitioning file data based on the virtual cloud disk operation interface comprises:
extracting file information from the file data, and determining the data volume corresponding to the file information according to the file information;
comparing the data quantity with a threshold value, and judging whether to divide the file data;
when the division is determined, the file data is divided to obtain a plurality of data blocks and division paths, and a hash value is generated.
8. The data storage method based on the multi-cloud architecture according to claim 6, wherein the saving the meta information of the file data in a local database after the splitting comprises:
and taking the parameters generated after the file data is divided as meta-information, and storing the meta-information in a local database.
9. The method according to claim 8, wherein the storing the meta information of the file data in a local database after the splitting further comprises:
and carrying out redundancy coding processing on the data blocks obtained after the file data is divided.
10. The method for storing data based on the multi-cloud architecture according to claim 6, wherein the block-wise encrypting the file data and block-wise uploading the file database to a corresponding cloud terminal based on a load balancing algorithm comprises:
determining a cloud end for the data block subjected to the redundant coding processing based on a scheduling strategy of load balancing;
and encrypting the data block, and feeding the key information back to the meta-information management module so as to feed the key information back to the local database by implementing the meta-information management module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210088806.5A CN114466015B (en) | 2022-01-25 | 2022-01-25 | Data storage system and method based on multi-cloud architecture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210088806.5A CN114466015B (en) | 2022-01-25 | 2022-01-25 | Data storage system and method based on multi-cloud architecture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114466015A true CN114466015A (en) | 2022-05-10 |
| CN114466015B CN114466015B (en) | 2024-03-15 |
Family
ID=81412527
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210088806.5A Active CN114466015B (en) | 2022-01-25 | 2022-01-25 | Data storage system and method based on multi-cloud architecture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114466015B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115022044A (en) * | 2022-06-02 | 2022-09-06 | 联通(广东)产业互联网有限公司 | Storage method and system based on multi-cloud architecture |
| CN115801843A (en) * | 2023-01-30 | 2023-03-14 | 湖南一特医疗股份有限公司 | Medical service platform and method based on cloud technology |
| CN117155871A (en) * | 2023-10-31 | 2023-12-01 | 山东衡昊信息技术有限公司 | Port industrial Internet point position low-delay concurrent processing method |
| CN117688106A (en) * | 2024-02-04 | 2024-03-12 | 广东东华发思特软件有限公司 | Efficient distributed data storage and retrieval system, method and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103118089A (en) * | 2013-01-22 | 2013-05-22 | 华中科技大学 | Safe storage method based on a plurality of cloud storage systems and system thereof |
| CN105827512A (en) * | 2016-04-29 | 2016-08-03 | 电子科技大学 | E-mail communication method through SMCLOUD |
| CN106453384A (en) * | 2016-11-09 | 2017-02-22 | 鹤荣育 | Security cloud disk system and security encryption method thereof |
| CN109583221A (en) * | 2018-12-07 | 2019-04-05 | 中国科学院深圳先进技术研究院 | Dropbox system based on cloudy server architecture |
| CN110363017A (en) * | 2019-07-15 | 2019-10-22 | 华瑞新智科技（北京）有限公司 | Mix the data safety sharing method and system based on client encryption under cloud environment |
| CN110851080A (en) * | 2019-11-04 | 2020-02-28 | 紫光云技术有限公司 | Distributed storage management system of multi-cloud-disk platform |
-
2022
- 2022-01-25 CN CN202210088806.5A patent/CN114466015B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103118089A (en) * | 2013-01-22 | 2013-05-22 | 华中科技大学 | Safe storage method based on a plurality of cloud storage systems and system thereof |
| CN105827512A (en) * | 2016-04-29 | 2016-08-03 | 电子科技大学 | E-mail communication method through SMCLOUD |
| CN106453384A (en) * | 2016-11-09 | 2017-02-22 | 鹤荣育 | Security cloud disk system and security encryption method thereof |
| CN109583221A (en) * | 2018-12-07 | 2019-04-05 | 中国科学院深圳先进技术研究院 | Dropbox system based on cloudy server architecture |
| CN110363017A (en) * | 2019-07-15 | 2019-10-22 | 华瑞新智科技（北京）有限公司 | Mix the data safety sharing method and system based on client encryption under cloud environment |
| CN110851080A (en) * | 2019-11-04 | 2020-02-28 | 紫光云技术有限公司 | Distributed storage management system of multi-cloud-disk platform |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115022044A (en) * | 2022-06-02 | 2022-09-06 | 联通(广东)产业互联网有限公司 | Storage method and system based on multi-cloud architecture |
| CN115801843A (en) * | 2023-01-30 | 2023-03-14 | 湖南一特医疗股份有限公司 | Medical service platform and method based on cloud technology |
| CN117155871A (en) * | 2023-10-31 | 2023-12-01 | 山东衡昊信息技术有限公司 | Port industrial Internet point position low-delay concurrent processing method |
| CN117155871B (en) * | 2023-10-31 | 2024-01-12 | 山东衡昊信息技术有限公司 | Port industrial Internet point position low-delay concurrent processing method |
| CN117688106A (en) * | 2024-02-04 | 2024-03-12 | 广东东华发思特软件有限公司 | Efficient distributed data storage and retrieval system, method and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114466015B (en) | 2024-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12008131B2 (en) | Systems and methods for a cryptographic file system layer | |
| CN114466015A (en) | Data storage system and method based on multi-cloud architecture | |
| US8762743B2 (en) | Encrypting data objects to back-up | |
| CN106156359B (en) | A kind of data synchronization updating method under cloud computing platform | |
| Celesti et al. | Adding long-term availability, obfuscation, and encryption to multi-cloud storage systems | |
| CN106294585B (en) | A kind of storage method under cloud computing platform | |
| US8838968B2 (en) | System and method for virtual machine data protection in a public cloud | |
| US10375166B2 (en) | Caching device and method thereof for integration with a cloud storage system | |
| US8452731B2 (en) | Remote backup and restore | |
| US20140082376A1 (en) | System, Method and Apparatus for Securely Saving/Retrieving Data on a Data Storage | |
| CN110381061A (en) | Cloudy storage method, method for down loading, device and the storage medium of file | |
| CN106341371A (en) | Cloud storage data encryption method and cloud storage system | |
| US9390101B1 (en) | Social deduplication using trust networks | |
| CN106060176A (en) | Cloud computing application architecture and cloud computing service method based on mixed cloud | |
| CN110851080A (en) | Distributed storage management system of multi-cloud-disk platform | |
| US10812590B2 (en) | System for generating distributed cloud data storage on disparate devices | |
| CN110633578A (en) | Intelligent shared cloud storage method and system | |
| CN110046510B (en) | Cross-cloud data migration method, device and system | |
| KR20160038496A (en) | Storage integrating method and system using file system | |
| Akingbade | Cloud Storage problems, benefits and solutions provided by Data De-duplication | |
| GB2496258A (en) | Encrypting data objects to back-up | |
| Srikanth et al. | Decentralized Cloud Storage using Unutilized Storage in PC | |
| JP6435616B2 (en) | Storage device, storage system, storage system control method and control program | |
| US11546411B1 (en) | Backing up confidential data to user devices on the same local network | |
| TWI856887B (en) | A distribution backup system, method non-volatile computer-readable storage medium thereof based on object storage gateway |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |