CN111158606A - Storage method, storage device, computer equipment and storage medium - Google Patents
Storage method, storage device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111158606A CN111158606A CN201911418326.5A CN201911418326A CN111158606A CN 111158606 A CN111158606 A CN 111158606A CN 201911418326 A CN201911418326 A CN 201911418326A CN 111158606 A CN111158606 A CN 111158606A
- Authority
- CN
- China
- Prior art keywords
- data block
- data
- offset
- read
- uploading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a storage method, a storage device, a computer device and a storage medium. The method comprises the following steps: acquiring a plurality of data blocks according to an original file; acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file; and after the uploading of each data block is finished, uploading the metadata file to the object storage system and taking the metadata file as a metadata object. The method can get rid of the limitation of uploading file size in the traditional object storage system.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a storage method, an apparatus, a computer device, and a storage medium.
Background
With the development of computer technology, the application of big data is more and more extensive, but the application of big data is not separated from the support of a large amount of data, and the data needs to be stored for the resource sharing of data by a plurality of platforms, so that the data storage is more challenged in the face of huge data storage requirements.
The object storage has the advantages of mass storage, infinite capacity expansion, quick access, low cost and the like, and is a storage system very suitable for static data. It supports two interfaces for uploading files: normal upload and fragmented upload. The common uploading interface uploads the file content to an object storage server at one time through an HTTP request to generate an object; the fragment uploading interface is used for dividing the file into a plurality of fragments, one fragment at each time, and finally combining the fragments into an object. The common uploading interface has limitation to the size of the file, and the typical maximum size is 5 gigabytes; the fragment upload interface has two limitations: 1) fragment number 2) fragment size, and the combination of these two limitations determines the size of the largest object that can be created and stored using the fragment upload interface, which is typically 5T bytes, in the mainstream public cloud object storage service.
If the file data to be stored is large, for example, when the backup set file generated by the application such as the backup virtual machine and the distributed database is stored, the traditional storage technology cannot meet the corresponding storage requirement because the maximum limit of the storage object is exceeded, and the traditional storage technology cannot support the storage requirement of the sparse file.
Disclosure of Invention
In view of the above, it is necessary to provide a storage method, an apparatus, a computer device, and a storage medium with good timeliness and high accuracy.
In a first aspect, an embodiment of the present invention provides a storage method, where the method includes:
acquiring a plurality of data blocks according to an original file;
acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file;
and after the uploading of each data block is finished, uploading the metadata file to the object storage system and taking the metadata file as a metadata object.
In an embodiment of the present application, the uploading the data block to a data object corresponding to the data block in an object storage system includes:
uploading the data block to a current data object in the object storage system to serve as a fragment of the current data object, and recording the number of fragments of the current data object;
if the fragmentation number of the current data object reaches a preset fragmentation threshold value or the data block is the last data block needing to be uploaded, sending a merging instruction to the object storage system, and acquiring a new initial data object as the current data object; the merging instructions are used for merging the fragments in the current data object into the data object.
In an embodiment of the application, after the uploading of each data block is completed, and before the step of uploading the metadata file to the object storage system and serving as the metadata object, the method further includes:
generating index information corresponding to each data block according to a first offset of each data block in the original file;
and storing the index information corresponding to each data block into an index information part in the metadata file.
In one embodiment of the present application, the method further comprises:
acquiring a reading request for data to be read in the original file, wherein the reading request comprises a starting offset and an ending offset of the data to be read in the original file;
acquiring at least one data block to be read in the metadata file according to the starting offset and the ending offset;
acquiring sub-data to be read in each data block to be read according to the data block information, the starting offset and the ending offset of each data block to be read;
and determining the data to be read according to the subdata to be read in each data block to be read.
In an embodiment of the present application, the obtaining at least one data block to be read in the metadata file according to the starting offset and the ending offset includes:
if the metadata file comprises an index information part, acquiring initial index information corresponding to the initial offset and ending index information corresponding to the ending offset in the index information part, and determining the at least one data block to be read in the index information part according to the initial index information and the ending index information;
if the metadata file does not comprise an index information part, acquiring a first offset and a data block size of each data block in the data block information part, and determining a data block starting offset and a data block ending offset corresponding to each data block; and determining the at least one data block to be read according to the data block starting offset and the data block ending offset corresponding to each data block and the position relation between the starting offset and the ending offset.
In an embodiment of the present application, the obtaining sub-data to be read in each data block to be read according to the data block information of each data block to be read, the starting offset, and the ending offset includes:
acquiring the data type of each data block to be read;
if the data type of the data block to be read is valid data, acquiring a starting point and an ending point of the subdata to be read according to a first offset of the data block to be read, a size of the data block, a first offset of the data block in the original file, a data object identifier of a corresponding data object, a second offset in the data object, the starting offset and the ending offset; acquiring the subdata to be read in a corresponding data object according to the starting point and the ending point of the subdata to be read;
if the data type of the data block to be read is invalid data, obtaining the null value length of the subdata to be read according to the first offset, the data block size, the starting offset and the ending offset of the data block to be read, and generating invalid data corresponding to the null value length to serve as the subdata to be read.
In a second aspect, an embodiment of the present invention provides a storage apparatus, where the apparatus includes:
the data block generating module is used for acquiring a plurality of data blocks according to the original file;
the data block uploading module is used for acquiring the data type of each data block, uploading the data block to a data object corresponding to the data block in an object storage system if the data type of the data block is valid data, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file;
and the metadata file uploading module is used for uploading the metadata file to the object storage system after the uploading of each data block is finished and taking the metadata file as a metadata object.
In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the following steps when executing the computer program:
acquiring a plurality of data blocks according to an original file;
acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file;
and after the uploading of each data block is finished, uploading the metadata file to the object storage system and taking the metadata file as a metadata object.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the following steps:
acquiring a plurality of data blocks according to an original file;
acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file;
and after the uploading of each data block is finished, uploading the metadata file to the object storage system and taking the metadata file as a metadata object.
According to the storage method, the storage device, the computer equipment and the storage medium, a plurality of data blocks are obtained according to the original file; acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of the data block in an original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file; and after the uploading of each data block is finished, uploading the metadata file to an object storage system and taking the metadata file as a metadata object. According to the storage method provided by the application, the original file is divided into the plurality of data blocks and stored in different data objects, the limitation of the file size in a traditional object storage system is eliminated, the corresponding storage steps are respectively carried out on the effective data blocks and the ineffective data blocks in the original file, the scheme of storing the coefficient file in the object storage system is realized, and the data blocks stored in different data objects can be rapidly and effectively read by generating the metadata object and uploading the metadata object to the object storage system.
Drawings
Fig. 1 is an implementation environment diagram of a storage method provided in an embodiment of the present application;
fig. 2 is a flowchart of a storage method according to an embodiment of the present application;
FIG. 3 is a flow chart of another storage method provided by an embodiment of the present application;
FIG. 4 is a flow chart of another storage method provided by an embodiment of the present application;
FIG. 5 is a flow chart of another storage method provided by an embodiment of the present application;
FIG. 6 is a flow chart of another storage method provided by an embodiment of the present application;
FIG. 7 is a flow chart of another storage method provided by an embodiment of the present application;
FIG. 8 is a block diagram of a memory device according to an embodiment of the present disclosure;
FIG. 9 is a block diagram of another memory device provided by an embodiment of the present application;
fig. 10 is a block diagram of a computer device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The storage method provided by the application can be applied to the application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented by an independent server or a server cluster formed by a plurality of servers. The server is provided with an object storage system for storing data uploaded by the terminal 102.
Referring to fig. 2, it shows a storage method provided in this embodiment, which is described by taking the method applied to the terminal in fig. 1 as an example, and includes the following steps:
step 202, obtaining a plurality of data blocks according to the original file.
In an embodiment of the present application, after the terminal receives an upload instruction of an original file from a user, the terminal divides the original file into a plurality of data blocks, and in an embodiment, when the terminal performs a data block partitioning operation on the original file, any one of the partitioning manners may be used, and the size of the plurality of data blocks obtained by partitioning may be the same size or different sizes. While generating each data block, recording a first offset of each data block in the original file and a data block size of each data block, wherein the first offset is used for representing the position of a start byte of the data block in an original element.
In an embodiment of the application, the terminal may divide the data blocks of the original file according to a preset blocking method, where each data block may be uploaded after the terminal completes the blocking operation of all the data blocks of the original file, or each obtained data block may be uploaded to the object storage system when each data block is obtained. Specifically, the original file may include a common file type and a data stream type, when the original file is of the common file type, the terminal may predict file information of the original file, the file information may include a file size and a file composition, and the terminal may perform block processing on the original file according to the file information. When the original file is of a data stream type, file information of the original file cannot be acquired before data reading is completed, and the start and the end of the data stream are judged only through an end symbol of the data stream or other control modes.
In an embodiment of the present application, if there is a portion of blank data in the original file, that is, the original file is a sparse file, the continuous blank data in the original file may be divided into one data block, and the size of the data block is the length of the continuous blank data. For example, if the original file includes 1000 bytes in total, where the 100 th byte to the 300 th byte in the original file are blank data, the 100 th byte to the 300 th byte may be divided into the same data block, and the size of the data block is 201 bytes.
In an embodiment of the present application, when the terminal obtains each data block, the terminal performs a corresponding storage operation according to a data type of each data block. Specifically, the data type of the data block may include valid data and invalid data, where if a real (non-empty) byte exists in the data block, the data type of the data block is valid data; and if all bytes in the data block are empty bytes, the data type of the data block is invalid data.
In an embodiment of the present application, if the data type of the data block is valid data, the data block is uploaded to a data object corresponding to the data block in an object storage system using a fragment upload interface, and in an upload process, if the upload operation of the data block is completed, the data block and first data block information corresponding to the data block are recorded in a data block information portion in a preset metadata file. When the data block is recorded, generating a data block number of the data block according to the uploading sequence of the data block, and storing the data block number into first data block information corresponding to the data block in a data information part; when data block information corresponding to the data block is recorded, obtaining a data object identifier of the data object corresponding to the data block and a second offset in the data object according to the data object uploaded by the data block and the position of the data block in the data object, and storing the data object identifier and the second offset in the data object into first data block information corresponding to the data block in a data information part; meanwhile, the data block size of the data block and the first offset of the data block in the original file are recorded in the first data block information corresponding to the data block in the data block information part in the metadata file, and specifically, the second offset is used for representing the position of the starting byte of the data block in the data object.
In an embodiment of the present application, the terminal may upload a plurality of data blocks to corresponding data objects in the object storage system at the same time, for example, three data blocks a, b, and c exist, and a data object 1 corresponding to the data block a, a data object 2 corresponding to the data block b, and a data object 3 corresponding to the data block c exist in the object storage system. At the same time, the terminal can respectively upload three data blocks a, b and c to the corresponding data object 1, data object 2 and data object 3 through a plurality of threads, if the uploading completion sequence is b, a and c, and the number of the data block corresponding to b is m, the number of the data block corresponding to a is m +1, and the number of the data block corresponding to c is m + 2. For a data block b, when the data block b is completely uploaded to a corresponding data object 2, the data block b and corresponding first data block information thereof are recorded in a data block information part of a metadata file of the terminal, wherein the first data block information comprises a data block number m, the size of the data block b, a first offset of the data block b in an original file, a data object identifier of the data object 2, and a second offset of the data block b in the data object 2.
Sparse files (sparse files) are computer files that attempt to more efficiently use the space of a file system when the contents of the file are mostly empty. The principle is that, instead of storing empty data blocks on a disk by occupying real space, only real (non-empty) data blocks are written to the disk as they are, with short information (metadata) representing the empty data blocks. When the data is stored in the object storage system, complete file data or fragment data is needed, and storage of empty data blocks in the data object is not supported, in actual use, a loose file is stored in the object storage system, and NULL (0) is needed to fill the empty data blocks in the file, which wastes a large amount of actual storage space and transmission bandwidth. Therefore, in an embodiment of the present application, when the terminal determines that the data type of the data block is invalid data, that is, all bytes in the data block are invalid data, the data block is not uploaded to the object storage system, and meanwhile, the terminal records the data block and second data block information corresponding to the data block in the metadata file, where the second data block information does not store a first offset of the data block in an original file, a data object identifier of a corresponding data object, and a second offset in the data object, compared to first data block information corresponding to valid data.
In an embodiment of the present application, the second data block information corresponding to the data block of which the data type is invalid data is stored in the metadata file, and in the process of reading the data block, the data block can be accurately restored by reading the second data block information corresponding to the data block in the metadata file in the metadata object.
And step 206, after the uploading of each data block is finished, uploading the metadata file to the object storage system and taking the metadata file as a metadata object.
In an embodiment of the application, after the terminal completes the uploading operation of all the data blocks corresponding to the original file, the metadata file also includes all the data blocks and first data block information corresponding to each data block. The terminal uploads the metadata file including all the data blocks and the corresponding first data block information to the object storage system to generate a corresponding metadata object, and at the moment, the metadata object and at least one data object corresponding to the original file are already stored in the remote object storage system, and uploading and storage are completed.
In an embodiment of the present application, the metadata file further includes an attribute information part, where the attribute information part is configured to store a file attribute of the original file, and specifically, the file attribute may include attributes of a file size, a generation time, a modification time, an owner, and the like of the original file, and optionally, the attribute information part is further configured to store a uniform prefix of a plurality of data objects used by the original file, and a storage offset of the data block information part in the metadata file.
In the storage method provided by the embodiment of the application, a plurality of data blocks are obtained according to an original file; acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises a data block number, a data block size, a first offset of the data block in an original file, a data object identifier of a corresponding data object and a second offset in the data object; and after the uploading of each data block is finished, uploading the metadata file to an object storage system and taking the metadata file as a metadata object. According to the storage method provided by the application, the original file is divided into the plurality of data blocks and stored in different data objects, the limitation of the file size in the traditional object storage system is eliminated, and the data blocks stored in different data objects can be rapidly and effectively read by generating the metadata object and uploading the metadata object to the object storage system.
Referring to fig. 3, a flow chart of another storage method provided by the present embodiment is shown, which can be applied to the terminal 102 in the implementation environment described above. On the basis of the embodiment shown in fig. 2, the step 202 may specifically include the following steps:
In an embodiment of the present application, when a first data block needs to be uploaded to the object storage system, the terminal may start a new initial data object through the fragment upload interface, that is, the object storage system may assign a unique identifier of the initial data object to the terminal, where the unique identifier may represent a position of the initial data object in the object storage system, and the initial data object is empty and does not include any fragment. The terminal generates a corresponding data object name according to the initial data object set by the user, takes the initial data object as a current data object, uploads a data block to be uploaded to the current data object as a fragment of the data object when uploading the data block, and records the fragment number of the current data object.
In an embodiment of the present application, if after a data block is uploaded, the number of fragments of the current data object reaches a preset fragmentation threshold, at this time, it indicates that the current data object cannot store a newly uploaded data block. The object storage system reassigns a new initial data object to the terminal as the current data object to receive the new data block. In a further embodiment, after the number of slices of the current data object has reached the preset slice threshold, a merge instruction for the current data object is sent to an object storage system, and the object storage system merges the slices in the current data object into a final data object according to the merge instruction.
In an embodiment of the present application, if the uploaded data block is the last data block that needs to be uploaded, that is, after the uploading operation of the data block is completed, when no other data block needs to be uploaded, the terminal sends a merge instruction for the current data object to the object storage system, and the object storage system merges the fragments in the current data object into a final data object according to the merge instruction. In a specific embodiment, if there are three data blocks a, b, and c that have not yet been uploaded, the uploading step of the data block a to the corresponding data object 1 and the uploading step of the data block b to the corresponding data object 2 are being executed at one time point, and then when the data block a completes the uploading process, there are data blocks c that need to be uploaded, when the number of fragments in the data object 1 does not reach the preset fragmentation threshold value, the terminal will continue to perform the uploading operation of uploading the data block c to the data object 1, and then when the data block b completes the uploading process, no data block needs to be uploaded, therefore, the terminal sends a merge instruction to the data object 2, so that the object storage system merges the fragments in the data object 2, and similarly, after the data block c completes the uploading process, the terminal sends a merge instruction to the data object 1.
In the storage method provided by the embodiment of the application, the data block is uploaded to the current data object in the object storage system to serve as the fragment of the current data object, and the number of fragments of the current data object is recorded; if the fragmentation number of the current data object reaches a preset fragmentation threshold value or the data block is the last data block needing to be uploaded, sending a merging instruction to the object storage system, and acquiring a new initial data object as the current data object; the merging instructions are used for merging the fragments in the current data object into the data object. According to the storage method provided by the embodiment of the application, after the preset fragmentation threshold corresponding to the data object is reached, the data block is continuously uploaded to the new data object, so that the size limit of the original file to be stored is removed while the existing object storage rule is met.
Referring to fig. 4, a flowchart of another storage method provided in this embodiment is shown, which can be applied to the terminal 102 in the above implementation environment. On the basis of the above embodiment, the method may further include the steps of:
In an embodiment of the present application, before uploading the metadata file to the object storage system, a first offset of each data block in the original file may be further obtained, and index information corresponding to each data block is generated according to an order of each first offset in the original file. The index information includes a relative order of the data blocks in the original file and a first offset corresponding to each data block. In a specific embodiment, if there are three data blocks a, b, and c, where the first offset corresponding to the data block a is the 50 th byte, the first offset corresponding to the data block b is the 0 th byte, and the first offset corresponding to the data block c is the 150 th byte, the data blocks may be sorted according to the order of the first offsets, and the sorting result is b, a, and c, and then the index information corresponding to the data blocks a, b, and c may be generated as 2, 1, and 3.
In step 404, the index information corresponding to each data block is stored in the index information section of the metadata file.
In one embodiment of the present application, after the index information corresponding to each data block is obtained, the index information corresponding to each data block is recorded and stored in the index information section of the metadata file.
In one embodiment of the present application, after the index information part is generated in the metadata file, the storage offset of the index information part in the metadata file may be further recorded in the attribute information part of the metadata file.
In the storage method provided by the embodiment of the application, index information corresponding to each data block is generated according to a first offset of each data block in the original file; and storing the index information corresponding to each data block into an index information part in the metadata file. According to the storage method provided by the embodiment of the application, since the index information part in the metadata file stores the index information of each data block, namely the storage sequence of each data block in the original file is stored, when each data block is read, the corresponding original file can be quickly restored through the index information of each data block, and the reading efficiency of the file is improved.
Referring to fig. 5, a flow chart of another storage method provided by the present embodiment is shown, which can be applied to the terminal 102 in the implementation environment described above. On the basis of the embodiment shown in fig. 2, the method may further include the following steps:
In an embodiment of the present application, after the terminal uploads the original file to the object storage system, a metadata object and at least one data object corresponding to the original file are formed. If a reading request of a user for the original file is received, extracting a starting offset and an ending offset of data to be read in the original file in the reading request, wherein the starting offset is the position of a starting byte of the data to be read in the original file, and the ending offset is the position of an ending byte of the data to be read in the original file. The data to be read can be intercepted in the original file through the starting offset and the ending offset.
In an embodiment of the present application, a terminal may first search locally whether a metadata file corresponding to the original file exists, and if the metadata file exists locally, determine at least one data block to be read that needs to be acquired according to a data block information portion or an index information portion in the metadata file, that is, at least one data block to be read that corresponds to the data to be read.
In an embodiment of the present application, if the terminal does not find the metadata file corresponding to the original file locally, accessing the metadata object corresponding to the original file in the object storage system, and obtaining the data block information portion according to the storage offset of the data block information portion in the attribute information portion in the metadata object; or the index information part is acquired according to the storage offset of the index information part in the attribute information part in the metadata object.
Referring to fig. 6, a flow chart of another storage method provided by the present embodiment is shown, which can be applied to the terminal 102 in the implementation environment described above. On the basis of the embodiment shown in fig. 5, the step 504 may specifically include the following steps:
In an embodiment of the present application, when the metadata file includes an index information portion, a start data block corresponding to the start offset and an end data block corresponding to the end offset may be determined according to a first offset of each data block in the index information portion, start index information corresponding to the start data block and end index information corresponding to the end data block are obtained, and at least one data block to be read is determined in the index information portion according to the start index information and the end index information.
In a specific embodiment, if there are 8 data blocks a to h, the corresponding index information may be represented as 1 to 8, where a first offset corresponding to the b data block is a 30 th byte, a first offset corresponding to the c data block is a 40 th byte, a first offset corresponding to the f data block is a 210 th byte, a first offset corresponding to the g data block is a 240 th byte, at this time, a start offset is a 36 th byte, an end offset is a 230 th byte, the start data block is a b data block, corresponding start index information is 2, the end data block is an f data block, and corresponding end index information is 6, and according to the start index information 2 and the end index information 6, the index information of at least one data block to be read is 2 to 6, that is, the data blocks to be read are b to f.
In one embodiment of the present application, if the metadata file does not include the index information portion, the first offset and the data block size of each data block may be obtained in the data block information portion of the metadata file according to the first offset B of each data blockoffAnd a data block size L, which can determine the data block start offset B of each data block in the original fileoffAnd ending offset (B)off+ L) and determining the starting offset R of the file to be readsAnd ending offset ReDetermine [ B ] corresponding to each data blockoff,Boff+L]And [ R ]s,Re]And judging whether an intersection exists or not, and if so, judging the data block as a data block to be read.
Referring to fig. 7, a flowchart of another storage method provided in this embodiment is shown, which can be applied to the terminal 102 in the above implementation environment. On the basis of the embodiment shown in fig. 5, the step 506 may specifically include the following steps:
In an embodiment of the present application, when the data block to be read is valid data, a first offset B of the data block to be read is obtainedoffData block size L, second offset in data object SoffThe initial offset R of the data to be read in the original filesEnd offset is ReThe starting point S and the end point E for calculating the read data from the data object are formulated as:
and according to the obtained start and end points SE, obtaining the sub data to be read corresponding to each data block to be read from the data object corresponding to the data object identification.
And step 508, determining the data to be read according to the sub-data to be read in each data block to be read.
In an embodiment of the present application, an arrangement order of sub-data to be read in each data block to be read is determined according to the first offset of each data block to be read, and the sub-data to be read can be merged into the data to be read according to the arrangement order.
In the storage method provided by the embodiment of the application, a reading request for data to be read in the original file is acquired, wherein the reading request comprises a starting offset and an ending offset of the data to be read in the original file; acquiring at least one data block to be read in the metadata file according to the starting offset and the ending offset; acquiring sub-data to be read in each data block to be read according to the data block information of each data block to be read, the starting offset and the ending offset; and determining the data to be read according to the subdata to be read in each data block to be read. According to the storage method provided by the embodiment of the application, the corresponding data block to be read can be obtained in the object storage system according to the starting offset and the ending offset corresponding to the data to be read, and the data to be read can be obtained according to each data block to be read, so that the data reading speed and stability are improved.
It should be understood that, although the steps in the above-described flowcharts are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in the above-described flowcharts may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or the stages is not necessarily sequential, but may be performed alternately or alternatingly with other steps or at least a portion of the sub-steps or stages of other steps.
Referring to fig. 8, a block diagram of a memory device 800 according to an embodiment of the present application is shown. As shown in fig. 8, the storage device 800 may include: a data block generation module 801, a data block uploading module 802, and a metadata file uploading module 803, wherein:
the data block generating module 801 is configured to obtain a plurality of data blocks according to an original file;
the data block uploading module 802 is configured to acquire a data type of each data block, upload the data block to a data object corresponding to the data block in an object storage system if the data type of the data block is valid data, and record first data block information corresponding to the data block in a data block information portion in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file;
the metadata file uploading module 803 is configured to upload the metadata file to the object storage system after the uploading of each data block is completed, and use the metadata file as a metadata object.
In an embodiment of the present application, the data block uploading module 802 is specifically configured to:
uploading the data blocks to a current data object in an object storage system, taking the data blocks as fragments of the current data object, and recording the number of the fragments of the current data object;
if the number of fragments of the current data object reaches a preset fragmentation threshold value or the data block is the last data block to be uploaded, sending a merging instruction to an object storage system, and acquiring a new initial data object as the current data object; the merge instruction is to merge the slices in the current data object into a data object.
Referring to fig. 9, a block diagram of a memory device 900 according to an embodiment of the present application is shown. As shown in fig. 9, the storage apparatus 900 may include, in addition to the modules included in the storage apparatus 800, optionally: an index information generating module 804 and a reading module 805. Wherein:
the index information generating module 804 is configured to generate index information corresponding to each data block according to a first offset of each data block in an original file; and storing the index information corresponding to each data block into an index information part in the metadata file.
The reading module 805 is configured to obtain a reading request for data to be read in an original file, where the reading request includes a start offset and an end offset of the data to be read in the original file; acquiring at least one data block to be read in the metadata file according to the starting offset and the ending offset; acquiring sub data to be read in each data block to be read according to the data block information, the initial offset and the end offset of each data block to be read; and determining the data to be read according to the subdata to be read in each data block to be read.
In an embodiment of the present application, the reading module 805 is further specifically configured to:
if the metadata file comprises an index information part, acquiring initial index information corresponding to the initial offset and ending index information corresponding to the ending offset in the index information part, and determining at least one data block to be read in the index information part according to the initial index information and the ending index information;
if the metadata file does not comprise an index information part, acquiring a first offset and a data block size of each data block in a data block information part, and determining a data block starting offset and a data block ending offset corresponding to each data block; and determining at least one data block to be read according to the data block starting offset and the data block ending offset corresponding to each data block and the position relation between the starting offset and the ending offset.
In an embodiment of the present application, the reading module 805 is further specifically configured to:
acquiring the data type of each data block to be read;
if the data type of the data block to be read is valid data, acquiring a starting point and an ending point of subdata to be read according to a first offset of the data block to be read, the size of the data block, a first offset of the data block in an original file, a data object identifier of a corresponding data object, a second offset, a starting offset and an ending offset in the data object; acquiring subdata to be read in a corresponding data object according to the starting and ending points of the subdata to be read;
if the data type of the data block to be read is invalid data, obtaining the null value length of the subdata to be read according to the first offset, the size of the data block, the initial offset and the ending offset of the data block to be read, and generating invalid data corresponding to the null value length to serve as the subdata to be read.
For the specific definition of the storage device, reference may be made to the above definition of the storage method, which is not described herein again. The modules in the storage device can be implemented in whole or in part by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 10. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a storage method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 10 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
acquiring a plurality of data blocks according to an original file;
acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file;
and after the uploading of each data block is finished, uploading the metadata file to an object storage system and taking the metadata file as a metadata object.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring a plurality of data blocks according to an original file;
acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file;
and after the uploading of each data block is finished, uploading the metadata file to an object storage system and taking the metadata file as a metadata object.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method of storage, the method comprising:
acquiring a plurality of data blocks according to an original file;
acquiring the data type of each data block, if the data type of the data block is valid data, uploading the data block to a data object corresponding to the data block in an object storage system, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file; and after the uploading of each data block is finished, uploading the metadata file to the object storage system and taking the metadata file as a metadata object.
2. The method of claim 1, wherein uploading the data block into a data object corresponding to the data block in an object storage system comprises:
uploading the data block to a current data object in the object storage system to serve as a fragment of the current data object, and recording the number of fragments of the current data object;
if the fragmentation number of the current data object reaches a preset fragmentation threshold value or the data block is the last data block needing to be uploaded, sending a merging instruction to the object storage system, and acquiring a new initial data object as the current data object; the merging instructions are used for merging the fragments in the current data object into the data object.
3. The method of any of claims 1-2, wherein after the uploading of each of the data blocks is complete and before the uploading of the metadata file to the object storage system as a metadata object, the method further comprises:
generating index information corresponding to each data block according to a first offset of each data block in the original file;
and storing the index information corresponding to each data block into an index information part in the metadata file.
4. The method of claim 3, further comprising:
acquiring a reading request for data to be read in the original file, wherein the reading request comprises a starting offset and an ending offset of the data to be read in the original file;
acquiring at least one data block to be read in the metadata file according to the starting offset and the ending offset;
acquiring sub-data to be read in each data block to be read according to the data block information, the starting offset and the ending offset of each data block to be read;
and determining the data to be read according to the subdata to be read in each data block to be read.
5. The method according to claim 4, wherein the obtaining at least one data block to be read in the metadata file according to the start offset and the end offset comprises:
if the metadata file comprises an index information part, acquiring initial index information corresponding to the initial offset and ending index information corresponding to the ending offset in the index information part, and determining the at least one data block to be read in the index information part according to the initial index information and the ending index information.
6. The method according to claim 5, wherein the obtaining at least one data block to be read in the metadata file according to the start offset and the end offset comprises:
if the metadata file does not comprise an index information part, acquiring a first offset and a data block size of each data block in the data block information part, and determining a data block starting offset and a data block ending offset corresponding to each data block; and determining the at least one data block to be read according to the data block starting offset and the data block ending offset corresponding to each data block and the position relation between the starting offset and the ending offset.
7. The method of claim 5, wherein the obtaining the sub-data to be read in each data block to be read according to the data block information of each data block to be read, the start offset, and the end offset comprises:
acquiring the data type of each data block to be read;
if the data type of the data block to be read is valid data, acquiring a starting point and an ending point of the subdata to be read according to a first offset of the data block to be read, a size of the data block, a first offset of the data block in the original file, a data object identifier of a corresponding data object, a second offset in the data object, the starting offset and the ending offset; acquiring the subdata to be read in a corresponding data object according to the starting point and the ending point of the subdata to be read;
if the data type of the data block to be read is invalid data, obtaining the null value length of the subdata to be read according to the first offset, the data block size, the starting offset and the ending offset of the data block to be read, and generating invalid data corresponding to the null value length to serve as the subdata to be read.
8. A storage device, the device comprising:
the data block generating module is used for acquiring a plurality of data blocks according to the original file;
the data block uploading module is used for acquiring the data type of each data block, uploading the data block to a data object corresponding to the data block in an object storage system if the data type of the data block is valid data, and recording first data block information corresponding to the data block in a data block information part in a metadata file; the first data block information comprises an effective data block identifier, a data block number, a data block size, a first offset of a data block in the original file, a data object identifier of a corresponding data object and a second offset in the data object; if the data type of the data block is invalid data, stopping uploading the data block to the object storage system, and recording second data block information of the data block in a data block information part in the initial metadata file, wherein the second data block information comprises an invalid data block identifier, a data block number, a data block size and first offset of the data block in the original file;
and the metadata file uploading module is used for uploading the metadata file to the object storage system after the uploading of each data block is finished and taking the metadata file as a metadata object.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911418326.5A CN111158606B (en) | 2019-12-31 | 2019-12-31 | Storage method, storage device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911418326.5A CN111158606B (en) | 2019-12-31 | 2019-12-31 | Storage method, storage device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111158606A true CN111158606A (en) | 2020-05-15 |
| CN111158606B CN111158606B (en) | 2020-11-03 |
Family
ID=70560455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911418326.5A Active CN111158606B (en) | 2019-12-31 | 2019-12-31 | Storage method, storage device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111158606B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113381995A (en) * | 2021-06-08 | 2021-09-10 | 珠海格力电器股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114630192A (en) * | 2022-03-16 | 2022-06-14 | 深圳法政信息技术有限公司 | A video recording method |
| CN115328858A (en) * | 2022-08-19 | 2022-11-11 | 济南浪潮数据技术有限公司 | Fragment uploading method, device and medium of distributed storage system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1595523A (en) * | 2003-09-12 | 2005-03-16 | 联想(北京)有限公司 | A disk storage method |
| CN103118089A (en) * | 2013-01-22 | 2013-05-22 | 华中科技大学 | Safe storage method based on a plurality of cloud storage systems and system thereof |
| US20140089264A1 (en) * | 2012-09-24 | 2014-03-27 | Fusion-Io, Inc. | Snapshots for a non-volatile device |
| CN103744875A (en) * | 2013-12-19 | 2014-04-23 | 记忆科技(深圳)有限公司 | Data rapid transferring method and system based on file system |
| CN104462141A (en) * | 2013-09-24 | 2015-03-25 | 中国移动通信集团重庆有限公司 | Data storage and query method and system and storage engine device |
| CN105612491A (en) * | 2013-09-16 | 2016-05-25 | 网络装置公司 | Management of extent based metadata with dense tree structures within a distributed storage architecture |
| CN106657213A (en) * | 2016-09-14 | 2017-05-10 | 深圳峰创智诚科技有限公司 | File transmission method and device |
| CN107872489A (en) * | 2016-09-28 | 2018-04-03 | 杭州海康威视数字技术股份有限公司 | A kind of file section method for uploading, device and cloud storage system |
-
2019
- 2019-12-31 CN CN201911418326.5A patent/CN111158606B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1595523A (en) * | 2003-09-12 | 2005-03-16 | 联想(北京)有限公司 | A disk storage method |
| US20140089264A1 (en) * | 2012-09-24 | 2014-03-27 | Fusion-Io, Inc. | Snapshots for a non-volatile device |
| CN103118089A (en) * | 2013-01-22 | 2013-05-22 | 华中科技大学 | Safe storage method based on a plurality of cloud storage systems and system thereof |
| CN105612491A (en) * | 2013-09-16 | 2016-05-25 | 网络装置公司 | Management of extent based metadata with dense tree structures within a distributed storage architecture |
| CN104462141A (en) * | 2013-09-24 | 2015-03-25 | 中国移动通信集团重庆有限公司 | Data storage and query method and system and storage engine device |
| CN103744875A (en) * | 2013-12-19 | 2014-04-23 | 记忆科技(深圳)有限公司 | Data rapid transferring method and system based on file system |
| CN106657213A (en) * | 2016-09-14 | 2017-05-10 | 深圳峰创智诚科技有限公司 | File transmission method and device |
| CN107872489A (en) * | 2016-09-28 | 2018-04-03 | 杭州海康威视数字技术股份有限公司 | A kind of file section method for uploading, device and cloud storage system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113381995A (en) * | 2021-06-08 | 2021-09-10 | 珠海格力电器股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114630192A (en) * | 2022-03-16 | 2022-06-14 | 深圳法政信息技术有限公司 | A video recording method |
| CN115328858A (en) * | 2022-08-19 | 2022-11-11 | 济南浪潮数据技术有限公司 | Fragment uploading method, device and medium of distributed storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111158606B (en) | 2020-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6778795B2 (en) | Methods, devices and systems for storing data | |
| CN111177302B (en) | Service bill processing method, device, computer equipment and storage medium | |
| CN105190573B (en) | The reduction redundancy of storing data | |
| US9377959B2 (en) | Data storage method and apparatus | |
| KR102031588B1 (en) | Method and system for implementing index when saving file | |
| CN111158606B (en) | Storage method, storage device, computer equipment and storage medium | |
| EP3376393B1 (en) | Data storage method and apparatus | |
| CN103955530B (en) | Data reconstruction and optimization method of on-line repeating data deletion system | |
| EP2933733A1 (en) | Data processing method and device in distributed file storage system | |
| WO2015199577A1 (en) | Metadata structures for low latency and high throughput inline data compression | |
| CN105468642A (en) | Data storage method and apparatus | |
| CN106030498A (en) | Storage device, data processing method, and storage system | |
| US20200301944A1 (en) | Method and apparatus for storing off-chain data | |
| CN112416880A (en) | Method and device for optimizing storage performance of mass small files based on real-time merging | |
| CN112632070B (en) | Method and device for storing and copying massive diversified complex spacecraft simulation data | |
| CN103067519A (en) | Method and device of data distribution storage under heterogeneous platform | |
| EP3207461A1 (en) | Compression of cascading style sheet files | |
| CN112817962B (en) | Data storage method and device based on object storage and computer equipment | |
| CN115470156A (en) | RDMA-based memory use method, system, electronic device and storage medium | |
| CN102282545B (en) | Storage system | |
| CN113791735A (en) | Video data storage method and device, computer equipment and storage medium | |
| CN115168499B (en) | Database table fragmentation method and device, computer equipment and storage medium | |
| CN111143130B (en) | Data recovery method and device, computer readable storage medium and computer equipment | |
| CN115840731A (en) | File processing method, computing device and computer storage medium | |
| CN113934362B (en) | Data storage method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |