CN112115206B - A method and device for processing object storage metadata - Google Patents
A method and device for processing object storage metadata Download PDFInfo
- Publication number
- CN112115206B CN112115206B CN201910531925.1A CN201910531925A CN112115206B CN 112115206 B CN112115206 B CN 112115206B CN 201910531925 A CN201910531925 A CN 201910531925A CN 112115206 B CN112115206 B CN 112115206B
- Authority
- CN
- China
- Prior art keywords
- library
- metadata
- read
- object storage
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种处理对象存储元数据的方法和装置，涉及计算机技术领域。该方法的一具体实施方式包括：将当前对象存储元数据存储至写入库，当所述写入库写满或发生故障时，将所述写入库降级为只读库；按照设定的时间间隔，将所述只读库中的对象存储元数据合并至稳定库；清空所述只读库，并将清空后的所述只读库作为备用库，当所述写入库写满或发生故障时，将所述备用库作为新的写入库。该实施方式因为采用多个数据库集群互相配合的技术手段，所以克服了当对象数量较多时单机数据库性能不稳定的技术问题，进而达到在对象数量极大的情况下也能稳定处理元数据的技术效果，同时还能达到轻松扩展到支持千亿级别对象的目的。
The present invention discloses a method and device for processing object storage metadata, and relates to the field of computer technology. A specific implementation of the method includes: storing the current object storage metadata in a write library, and when the write library is full or fails, downgrading the write library to a read-only library; merging the object storage metadata in the read-only library into a stable library at a set time interval; clearing the read-only library, and using the cleared read-only library as a backup library, and when the write library is full or fails, using the backup library as a new write library. Because this implementation adopts technical means of multiple database clusters cooperating with each other, it overcomes the technical problem of unstable performance of a single-machine database when the number of objects is large, thereby achieving the technical effect of stable processing of metadata even when the number of objects is extremely large, and at the same time can achieve the purpose of easily expanding to support hundreds of billions of objects.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing object storage metadata.
Background
In recent years, object storage is used as the most important supporting service of public cloud, supports various services such as live broadcast, video on demand, pictures and the like, and particularly develops various video services in the years, the data of various videos and pictures are more and more, and the object storage needs a large-scale metadata management system to manage metadata. Generally, an object generally includes an object name, object data (data without format), and some attributes (Meta, such as modification time of the object) attached to the object. Object storage is a distributed storage service used to store objects. The name of the object, the properties of the object, and the location in which the object data is stored are the metadata of the object (i.e., the object storage metadata). There are mainly two ways to access object storage metadata, one is to obtain the data and Meta information of an object by the name of the object, and one is to query all objects behind a certain object. The current common mode is to use a relational database to manage object storage metadata, the technology of the database is mature, and each large company has a rich maintenance experience, so that the relational database is a very suitable metadata storage system.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
1. The object storage metadata is managed through the relational database, the storage capacity is seriously dependent on the service capacity of a single machine, and when the number of objects is large (reaching the billion level), the performance of the single machine relational database is unstable.
2. The problem of unstable performance of a single machine is solved through splitting the database into tables, so that range query becomes complicated and even infeasible, the database needs to be further split due to service growth, and the cost is extremely high.
Disclosure of Invention
In view of this, the embodiments of the present invention provide a method and apparatus for processing object storage metadata, which can solve the problem of unstable performance of a stand-alone database when the number of objects is large, and is easy to expand, and can easily expand to support objects of trillion levels.
In order to achieve the above object, according to one aspect of the embodiment of the present invention, there is provided a method for processing object storage metadata, including storing current object storage metadata into a write-in library, degrading the write-in library to a read-only library when the write-in library is full or fails, merging object storage metadata in the read-only library into a stable library at a set time interval, emptying the read-only library, taking the emptied read-only library as a standby library, and taking the standby library as a new write-in library when the write-in library is full or fails.
The method comprises the steps of selecting a set number of object storage metadata from a current stable base randomly according to a set time interval, calculating a splitting point according to the object storage metadata, generating a metadata table and a metadata range record table according to the splitting point, wherein the metadata range record table records the storage range of the metadata table, and merging the object storage metadata in the read-only base into the stable base based on the metadata table and the metadata range record table.
Optionally, merging the object storage metadata in the read-only library into a stable library based on the metadata table and the metadata range record table, wherein the method comprises the steps of sequentially traversing and reading the object storage metadata of the current stable library and the object storage metadata of the read-only library, and writing the object storage metadata into the metadata table; the metadata table and the metadata range record table form a new stable base, the object storage metadata in the current stable base and the read-only base and the object storage metadata in the new stable base are subjected to offline comparison and verification, after verification is passed, the configuration pointed to the current stable base in the cluster configuration is modified to be pointed to the new stable base, and the current stable base is deleted.
Optionally, the method further comprises the steps of inquiring object storage metadata corresponding to N object names based on the object names, wherein N is a positive integer, and returning an inquiring result set according to the object storage metadata corresponding to the object names.
The method comprises the steps of selecting N object storage metadata corresponding to N object names, randomly reading the N object storage metadata from a write-in library, a read-only library and a stable library to obtain three read result sets, merging and traversing the three read result sets sequentially according to the object names, and after merging and traversing are completed, if the data volume in the query result set does not reach a set threshold value, starting the process of merging and traversing the three read result sets again by the last object name until the data volume in the query result set reaches the set threshold value.
The method comprises the steps of selecting object names, merging and traversing three read result sets according to the object names, wherein the three read result sets comprise setting priorities, sequentially reducing the priorities of the read result sets of a write-in library, a read-only library and a stable library, merging and traversing the three read result sets according to the priorities sequentially according to the object names, judging whether object storage metadata corresponding to the object names carry a deletion mark or not if one object name appears in the three read result sets in total, jumping out if the object names carry the deletion mark, starting merging and traversing by the next object name, returning the object storage metadata corresponding to the object names to the query result set if the object names do not carry the deletion mark, and returning the object storage metadata corresponding to the object names to the query result set if one object name appears in the three read result sets in total for a plurality of times, and returning the object storage metadata corresponding to the object names read from the read result set with the highest priority to the query result set.
According to one aspect of the embodiment of the invention, a device for processing object storage metadata is provided, which comprises a degradation processing module, a merging processing module and a standby processing module, wherein the degradation processing module is used for storing the current object storage metadata into a write-in library, degrading the write-in library into a read-only library when the write-in library is full or fails, merging the object storage metadata in the read-only library into a stable library according to a set time interval, and the standby processing module is used for emptying the read-only library, taking the emptied read-only library as a standby library, and taking the standby library as a new write-in library when the write-in library is full or fails.
The merging processing module is further used for randomly selecting a set number of object storage metadata from a current stable base according to a set time interval, calculating a splitting point according to the object storage metadata, generating a metadata table and a metadata range record table according to the splitting point, recording the storage range of the metadata table in the metadata range record table, and merging the object storage metadata in the read-only base to the stable base based on the metadata table and the metadata range record table.
The merging processing module is further used for sequentially traversing and reading the object storage metadata of the current stable library and the object storage metadata of the read-only library, writing the object storage metadata into the metadata table, forming a new stable library by the metadata table and the metadata range record table, carrying out offline comparison and verification on the object storage metadata in the current stable library and the read-only library and the object storage metadata in the new stable library, modifying the configuration pointed to the current stable library in cluster configuration to be pointed to the new stable library after verification is passed, and deleting the current stable library.
Optionally, the device further comprises a list query module, wherein the list query module is used for querying object storage metadata corresponding to N object names based on the object names, N is a positive integer, and a query result set is returned according to the object storage metadata corresponding to the object names.
The list query module is further used for randomly reading N pieces of object storage metadata from a write-in library, a read-only library and a stable library to obtain three read result sets, merging and traversing the three read result sets sequentially according to the object names, and after the merging and traversing are completed, if the data quantity in the query result set does not reach a set threshold value, starting the process of merging and traversing the three read result sets again by the last object name until the data quantity in the query result set reaches the set threshold value.
The list query module is further used for randomly reading N pieces of object storage metadata from the write-in library, the read-only library and the stable library to obtain three read result sets, setting priority, sequentially reducing the priority of the read result sets of the write-in library, the read-only library and the stable library, merging and traversing the three read result sets according to the priority sequentially according to the object names, judging whether the object storage metadata corresponding to the object names carries a deletion mark if one object name appears in the three read result sets in total, jumping out if the object storage metadata corresponding to the object names carries the object storage metadata, merging and traversing the object storage metadata corresponding to the object names if the object storage metadata carries the object storage metadata, returning the object storage metadata corresponding to the object names to the query result sets if the object names do not appear in the three read result sets in total, and returning the object storage metadata corresponding to the object names read from the read result set with the highest priority to the query result set if the object names appear in total for a plurality of times.
According to an aspect of an embodiment of the present invention, there is provided an electronic device including one or more processors, and a storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement a method of storing metadata for a processing object as provided in the foregoing embodiment.
According to an aspect of an embodiment of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements a method of processing object storage metadata as provided by the foregoing embodiment.
The embodiment of the invention has the advantages that the technical means of mutually matching a plurality of database clusters is adopted, so that the technical problem that the performance of a single database is unstable when the number of objects is large is solved, the service capability of the single database is not limited, the technical effect of stably processing metadata under the condition that the number of objects is extremely large is achieved, and the aim of easily expanding the metadata to support billions of objects is also achieved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a basic flow of a method of processing object storage metadata according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of basic modules of an apparatus for processing object storage metadata according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a system architecture for processing object storage metadata according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a merge function implementation in a system that processes object storage metadata in accordance with an embodiment of the present invention;
FIG. 5 is a schematic diagram of a fabric evolution failure rollback in a system that processes object storage metadata in accordance with an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
Fig. 7 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a basic flow of a method of processing object storage metadata according to an embodiment of the present invention. As shown in fig. 1, an embodiment of the present invention provides a method for processing object storage metadata, including:
And S101, storing the current object storage metadata into a write-in warehouse, degrading the write-in warehouse to a read-only warehouse when the write-in warehouse is full or fails, and storing the current object storage metadata into the write-in warehouse, wherein the object storage metadata are incremental data (readable and writable). When the write-in warehouse is full or fails, the incremental data is converted into stock data, the stock data is set to be read-only, and the stock data is data which cannot be modified at the moment.
S102, merging object storage metadata in the read-only library into a stable library according to a set time interval;
and S103, emptying the read-only library, taking the emptied read-only library as a standby library, and taking the standby library as a new write library when the write library is full or fails.
The embodiment of the invention solves the technical problem that the performance of the single database is unstable when the number of the objects is large because the plurality of database clusters are mutually matched, thereby achieving the technical effect of stably processing the metadata under the condition of extremely large number of the objects.
In step S102 of the embodiment of the invention, merging the object storage metadata in the read-only library into the stable library according to a set time interval, wherein the method comprises the steps of randomly selecting a set number of object storage metadata from the current stable library according to the set time interval, calculating a split point according to the object storage metadata, generating a metadata table and a metadata range record table according to the split point, wherein the metadata range record table records the storage range of the metadata table, and merging the object storage metadata in the read-only library into the stable library based on the metadata table and the metadata range record table.
In the embodiment of the invention, the data in the read-only library is periodically combined into the stable library to reduce the reading consumption, and the combination process is performed on two read-only data sets (namely, the data can only be read from the read-only library and the stable library and cannot be written into the read-only library and the stable library), so that the operation difficulty in the practical application process is reduced.
Based on step S102 of the foregoing embodiment, in the embodiment of the present invention, based on the metadata table and the metadata range record table, merging the object storage metadata in the read-only library into the stable library includes sequentially traversing and reading the object storage metadata of the current stable library and the object storage metadata of the read-only library, writing into the metadata table, forming a new stable library by the metadata table and the metadata range record table, performing offline comparison verification on the object storage metadata in the current stable library and the read-only library and the object storage metadata in the new stable library, modifying the configuration pointed to the current stable library in the cluster configuration to be pointed to the new stable library after verification is passed, and deleting the current stable library.
In the embodiment of the invention, the data in the read-only library is periodically combined into the stable library to reduce the reading consumption, and the combination process is carried out on two read-only data sets, so that the operation difficulty in the actual application process is reduced.
The method further comprises the steps of inquiring object storage metadata corresponding to N object names based on the object names, wherein N is a positive integer, and returning an inquiring result set according to the object storage metadata corresponding to the object names. The embodiment of the invention has the function of list query, such as querying metadata corresponding to 1000 keys starting from a certain object name (Key).
The method comprises the steps of randomly reading N pieces of object storage metadata from a write-in library, a read-only library and a stable library to obtain three read result sets, merging and traversing the three read result sets sequentially according to the object names, and executing the process of merging and traversing the three read result sets again by the last object name until the data amount in the query result set reaches a set threshold value if the data amount in the query result set does not reach the set threshold value after merging and traversing.
For example, a query starts with 1000 keys, which 1000 keys may appear in a write library, a read-only library, and a stable library, 1000 metadata are fetched from each of the three databases, and then returned data is determined according to the business rule, which may be 500, so there is a loop to process until 1000 results are fetched. The embodiment of the invention can solve the problem that the query becomes complex due to the adoption of the database and the table, and has high efficiency and accurate return result in the query process.
Based on the embodiment, the method comprises the steps of sequentially traversing three read result sets according to the object names in a merging mode, wherein the priority of the read result sets of a write-in library, a read-only library and a stable library is sequentially reduced, sequentially traversing the three read result sets according to the priority in a merging mode according to the object names, judging whether object storage metadata corresponding to one object name carries a deletion mark or not if one object name appears in the three read result sets in total, jumping out if the object name does not appear in total, starting merging and traversing by the next object name, returning the object storage metadata corresponding to the object name to the query result set if the object name does not appear in the three read result sets in total, and returning the object storage metadata corresponding to the object name to the query result set from the read result set with the highest priority if the object name appears in the three read result sets in total. The embodiment of the invention adopts a mode of setting priority to inquire and screen in three libraries, can solve the problem that the inquiry becomes complex due to the adoption of the library division and the table division, has high efficiency in the inquiry process and accurate return result.
Fig. 2 is a schematic diagram of basic modules of an apparatus for processing an object storage metadata according to an embodiment of the present invention. As shown in fig. 2, an embodiment of the present invention provides a processing apparatus 200 for storing metadata by an object, including:
the degradation processing module 201 is configured to store the current object storage metadata into a write-in library, and degrade the write-in library into a read-only library when the write-in library is full or fails;
A merging processing module 202, configured to merge object storage metadata in the read-only library into a stable library according to a set time interval;
And the standby processing module 203 is configured to empty the read-only library, take the emptied read-only library as a standby library, and take the standby library as a new write library when the write library is full or fails.
The embodiment of the invention adopts the technical means of mutually matching a plurality of database clusters, so that the technical problem that the performance of a single database is unstable when the number of objects is large is solved, the technical effect of stably processing metadata under the condition of extremely large number of objects is achieved, and the aim of easily expanding to support billions of objects is also achieved.
In the embodiment of the present invention, the merging processing module 202 is further configured to randomly select a set number of object storage metadata from a current stable library according to a set time interval, calculate a split point according to the object storage metadata, generate a metadata table and a metadata range record table according to the split point, wherein a storage range of the metadata table is recorded in the metadata range record table, and merge the object storage metadata in the read-only library into the stable library based on the metadata table and the metadata range record table.
In the embodiment of the invention, the data in the read-only library is periodically combined into the stable library to reduce the reading consumption, and the combination process is performed on two read-only data sets (namely, the data can only be read from the read-only library and the stable library and cannot be written into the read-only library and the stable library), so that the operation difficulty in the practical application process is reduced.
The merging processing module is further used for sequentially traversing and reading the object storage metadata of the current stable base and the object storage metadata of the read-only base and writing the object storage metadata into the metadata table, forming a new stable base by the metadata table and the metadata range record table, carrying out offline comparison and verification on the object storage metadata of the current stable base and the read-only base and the object storage metadata of the new stable base, modifying configuration pointed to the current stable base in cluster configuration to be pointed to the new stable base after verification is passed, and deleting the current stable base.
In the embodiment of the invention, the data in the read-only library is periodically combined into the stable library to reduce the reading consumption, and the combination process is carried out on two read-only data sets, so that the operation difficulty in the actual application process is reduced.
In the embodiment of the invention, the device further comprises a list query module, wherein the list query module is used for querying object storage metadata corresponding to N object names based on the object names, N is a positive integer, and a query result set is returned according to the object storage metadata corresponding to the object names. The embodiment of the invention has the function of List query (List), for example, the metadata corresponding to 1000 keys starting from a certain object name (Key) is queried.
In the embodiment of the invention, the list query module is further used for randomly reading N pieces of object storage metadata from a write-in library, a read-only library and a stable library to obtain three read result sets, sequentially merging and traversing the three read result sets according to the object names, and after the merging and traversing are completed, if the data quantity in the query result set does not reach a set threshold value, starting the process of merging and traversing the three read result sets again by the last object name until the data quantity in the query result set reaches the set threshold value. The embodiment of the invention can solve the problem that the query becomes complex due to the adoption of the database and the table, and has high efficiency and accurate return result in the query process.
In the embodiment of the invention, the list query module is further used for randomly reading N pieces of object storage metadata from a write-in library, a read-only library and a stable library to obtain three read result sets, setting priority, sequentially reducing the priority of the read result sets of the write-in library, the read-only library and the stable library, sequentially merging and traversing the three read result sets according to the priority according to the object names, judging whether the object storage metadata corresponding to the object names carries a deletion mark if one object name appears in the three read result sets in total, jumping out if the object storage metadata corresponding to the object names carries the deletion mark, merging and traversing the object storage metadata corresponding to the object names if the object storage metadata does not carry the object storage metadata, returning the object storage metadata corresponding to the object names to the query result sets if the object storage metadata does not carry the object storage metadata, and returning the object storage metadata corresponding to the object names read from the read result set with the highest priority if the object names appear in the three read result sets in total. The embodiment of the invention adopts a mode of setting priority to inquire and screen in three libraries, can solve the problem that the inquiry becomes complex due to the adoption of the library division and the table division, has high efficiency in the inquiry process and accurate return result.
FIG. 3 is a system architecture diagram of managing object storage metadata according to an embodiment of the present invention. As shown in fig. 3, the system for managing the object storage metadata comprises a stable base, a read-only base, a write-in base, a standby base, a monitoring service, a merging service and a configuration center, wherein the stable base is composed of a plurality of database tables and is used for storing historical stock data, the stable base only supports reading data, the stable base also comprises metadata tables and metadata range record tables, the write-in base is composed of a plurality of database tables and is used for storing data which are currently being read and written, the data can be read from the write-in base and can be written into the write-in base, the write-in base also comprises the metadata tables and the metadata range record tables, the read-only base is formed by switching from the write-in base when the write-in base fails or is full, the standby base is only used for supporting reading data, the standby base is used for switching into a new write-in base when the write-in base fails or is full, the monitoring service is used for monitoring whether the write-in base fails or is full, the configuration center is used for providing consistent cluster information for the database, and the read-in base is used for merging the read-in base according to the set time interval after the read-in base is complete.
The basic operation of the system for managing objects to store metadata is as follows:
1. write (Put), can be written directly into write library (Writeable Meta).
2. Delete (Delete) and write the data with Delete markers in the write library.
3. Reading (Get), namely sequentially reading data corresponding to the object name from a write library (Writeable Meta), a read-only library (ReadOnly Meta) and a Stable library (Stable Meta), and judging as follows:
when the data without the deletion mark is read, jumping out and cycling back to the data;
reading the data with the deletion mark, and jumping out and cycling back to the absence of the object;
if the data is not read, the returned object does not exist.
4. List inquiry (List)
Taking out N pieces of data from a write library (Writeable Meta), a read-only library (ReadOnly Meta) and a Stable library (Stable Meta);
Setting priority, and sequentially reducing the priority of writing into a warehouse, reading out from a warehouse and stabilizing the warehouse;
traversing the three result sets according to the priority merging, and merging until one result set is taken out:
if one Key only appears once, if the object storage metadata (namely metadata) corresponding to the Key carries a deletion mark, the metadata directly jumps out, and if the metadata does not carry the deletion mark, the metadata is returned;
if a Key appears multiple times, the metadata with the highest priority is subject to.
After the merging is finished, if the number of the results is insufficient, the last key is used as a starting mark, and the merging traversing process is re-executed. Fig. 4 is a schematic diagram of a merge function implementation in a system for processing object storage metadata according to an embodiment of the present invention. As shown in fig. 4, the data of the read-only library is periodically merged into the stable library to reduce the read consumption, and in the present system, the merging process is performed on two read-only data sets, which are as follows:
judging whether the stable base needs to be split (when the number of objects is excessive or the service capability of the current system is insufficient), randomly selecting some data to calculate splitting points if the stable base needs to be split, generating a new metadata Table (MetaN Table) according to the splitting points, generating a metadata range record Table (Table Meta) and the range of the Table Meta record MetaN to integrally form a tree structure, 1 traversing the data of the read stable base and writing the data into a new MetaN, 2 traversing the data of the read ReadOnly Meta and writing the data into MetaN (deleting the data with a deletion mark), comparing the object storage metadata of the current stable base and the read-only base participating in merging with the object storage metadata of the new stable base offline to ensure the merging before and after logic, 3 modifying the cluster configuration, modifying the configuration of the current stable base pointed to the current stable base in the cluster configuration to point to the new stable base, and deleting the current stable base to finish the merging.
In the actual operation process, various new storage systems are continuously appeared, based on the requirement of architecture evolution, new systems are tried to be introduced in the production environment, and in general, double writing is needed in this case, but verification of double writing data and rollback when failure can be a troublesome matter. In the system for processing object storage metadata according to the embodiment of the present invention, as shown in fig. 5, in normal circumstances, a New write dual-write New storage device (New Store) and a database (Mysql) are newly written, the New storage device is used as a main storage device, the write-in is periodically changed into a read-only storage, and the consistency of data in the New Store and Mysql is compared, once any problem occurs, a standby storage is directly started, and the service state of the original database is fully backed.
The system realizes a high-availability large-scale global ordered KV cluster based on a plurality of Mysql clusters, creates a large global ordered KV on a plurality of databases based on the thought of LSM Tree, can randomly migrate multi-cluster write points, solves the problem of Mysql single-point faults, solves the problem of Mysql reading and writing due to the Tree structure of logic inside a logic cluster, and converts dynamic data comparison into static data comparison. In the prior art, the HBase, tikv and other systems realize global ordered KV, the scale can be greatly expanded, and the large-scale global ordered requirement is realized, but when the clusters are used, the problem that the whole metadata management service cannot be written in due to single cluster faults can be encountered, and the writing points of the system can be randomly migrated, so that the problem of single cluster faults is solved. Meanwhile, the system can keep the basic architecture unchanged, and only uses HBase, tikv and other systems to replace Mysql, so as to realize higher availability.
The cluster data scale of the system is not limited by the service capability of a single database, and can reach the object of trillion levels, in particular:
The processing capability can be greatly expanded, namely the stock data is read-only data, the reading capability can be improved by simply increasing the backup of the database, the capability of reconstructing the stock data is provided, only one database is needed to be accessed for each access, the capacity can be improved by infinitely expanding the number of the databases without causing the reduction of the processing performance, the incremental data can be quickly read-only, and only a small cluster of the incremental data is needed to be maintained, so that the access of the incremental data is greatly reduced.
The reliability is improved, the availability of the stock data can be improved by simply increasing the number of the slave libraries, and the incremental data clusters (write-in libraries) can be quickly switched to the backup clusters (read-only libraries or standby libraries) when the incremental data clusters are unavailable, so that the writing of the system is not influenced.
The support of architecture evolution can switch the current database into a new system (or a novel storage device), the method is very simple to import the stock data into the new system and can switch back to the original database (or the system) without risk when the new system has problems due to the read-only of the stock data, the incremental data clusters can be quickly switched, the new system can be used as a writable cluster (write-in library), only a small amount of data is stored in the read-only cluster (read-only library), and the writable cluster can be quickly switched to a backup cluster of an old system when the new system has faults.
Fig. 6 illustrates an exemplary system architecture 600 of a method of processing object storage metadata or an apparatus of processing object storage metadata to which embodiments of the present invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 is used as a medium to provide communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 605 via the network 604 using the terminal devices 601, 602, 603 to receive or send messages, etc. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc., may be installed on the terminal devices 601, 602, 603.
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 605 may be a server providing various services, such as a background management server providing support for shopping-type websites browsed by the user using the terminal devices 601, 602, 603. The background management server can analyze and other data of the received product information inquiry request and feed back the processing result to the terminal equipment.
It should be noted that, the method for storing metadata in a processing object according to the embodiment of the present invention is generally executed by the server 605, and accordingly, the device for storing metadata in a processing object is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
According to an embodiment of the present invention, there is also provided an electronic device and a readable storage medium.
The electronic equipment comprises one or more processors and a storage device, wherein the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are enabled to realize the method for storing metadata by the processing object.
The computer readable medium of the embodiment of the present invention stores thereon a computer program which, when executed by a processor, implements the method for storing metadata for a processing object provided by the present invention.
Referring now to FIG. 7, there is illustrated a schematic diagram of a computer system 700 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the system 700 are also stored. The CPU701, ROM702, and RAM703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Connected to the I/O interface 705 are an input section 706 including a keyboard, a mouse, and the like, an output section 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like, a storage section 708 including a hard disk, and the like, and a communication section 709 including a network interface card such as a LAN card, a modem, and the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 701.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, a processor may be described as including a degradation processing module, a merge processing module, and a standby processing module. The names of these modules do not in any way limit the module itself, and for example, the merge processing module may also be described as "a module for merging object storage metadata in the read-only library into a stable library".
As a further aspect, the invention also provides a computer readable medium which may be comprised in the device described in the above embodiments or may be present alone without being fitted into the device. The computer readable medium carries one or more programs, and when the one or more programs are executed by the equipment, the equipment comprises the steps of storing the current object storage metadata into a write-in warehouse, degrading the write-in warehouse into a read-only warehouse when the write-in warehouse is full or fails, merging the object storage metadata in the read-only warehouse into a stable warehouse according to a set time interval, emptying the read-only warehouse, taking the emptied read-only warehouse as a standby warehouse, and taking the standby warehouse as a new write-in warehouse when the write-in warehouse is full or fails.
According to the method for processing the object storage metadata, which is disclosed by the embodiment of the invention, the technical problem that the performance of a single database is unstable when the number of objects is large is solved because the technical means that a plurality of database clusters are matched with each other is adopted, so that the technical effect of stably processing the metadata under the condition that the number of objects is extremely large is achieved, and the aim of easily expanding the metadata to support billions of objects is fulfilled.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (8)
1. A method of processing object storage metadata, comprising:
storing the current object storage metadata into a write-in library, and degrading the write-in library into a read-only library when the write-in library is full or fails;
Randomly selecting a set number of object storage metadata from a current stable library according to a set time interval, and calculating a split point according to the object storage metadata;
generating a metadata table and a metadata range record table according to the split point, wherein the storage range of the metadata table is recorded in the metadata range record table;
merging object storage metadata in the read-only library into a stable library based on the metadata table and the metadata range record table so as to reduce the reading consumption;
and emptying the read-only library, taking the emptied read-only library as a standby library, and taking the standby library as a new write library when the write library is full or fails.
2. The method of claim 1, wherein merging object storage metadata in the read-only library into a stable library based on the metadata table and the metadata range record table comprises:
Sequentially traversing and reading the object storage metadata of the current stable library and the object storage metadata of the read-only library, and writing the metadata into the metadata table;
The metadata table and the metadata range record table form a new stable library;
Performing offline comparison and verification on the object storage metadata in the current stable library and the read-only library and the object storage metadata in the new stable library;
and after verification is passed, modifying the configuration pointing to the current stable library in the cluster configuration to point to the new stable library, and deleting the current stable library.
3. The method according to claim 1, wherein the method further comprises:
Inquiring object storage metadata corresponding to N object names based on the object names, wherein N is a positive integer;
And returning the query result set according to the object storage metadata corresponding to the object name.
4. A method according to claim 3, wherein querying object storage metadata corresponding to N object names comprises:
randomly reading N pieces of object storage metadata from each of a write-in library, a read-only library and a stable library to obtain three read result sets;
sequentially merging and traversing the three read result sets according to the object names;
and after the merging traversal is finished, if the data quantity in the query result set does not reach the set threshold, starting to execute the merging traversal process on the three read result sets again by using the name of the last object until the data quantity in the query result set reaches the set threshold.
5. The method of claim 4, wherein merging through three sets of read results based on the object names in turn comprises:
setting priority, and sequentially reducing the priority of a read result set of a write-in library, a read-only library and a stable library;
sequentially merging and traversing three read result sets according to the object names and the priorities:
If one object name appears once in the three reading result sets, judging whether the object storage metadata corresponding to the object name carries a deletion mark, if so, jumping out, starting merging and traversing by the next object name, and if not, returning the object storage metadata corresponding to the object name to the query result set;
if one object name appears multiple times in the three read result sets, the object storage metadata corresponding to the object name is returned to the query result set, wherein the object storage metadata is read from the read result set with the highest priority.
6. An apparatus for processing object storage metadata, comprising:
The degradation processing module is used for storing the current object storage metadata into a write-in library, and degrading the write-in library into a read-only library when the write-in library is full or fails;
The merging processing module is used for randomly selecting a set number of object storage metadata from a current stable library according to a set time interval, calculating a splitting point according to the object storage metadata, generating a metadata table and a metadata range record table according to the splitting point, wherein the storage range of the metadata table is recorded in the metadata range record table;
And the standby processing module is used for emptying the read-only library, taking the emptied read-only library as a standby library, and taking the standby library as a new write library when the write library is full or fails.
7. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-5.
8. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910531925.1A CN112115206B (en) | 2019-06-19 | 2019-06-19 | A method and device for processing object storage metadata |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910531925.1A CN112115206B (en) | 2019-06-19 | 2019-06-19 | A method and device for processing object storage metadata |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112115206A CN112115206A (en) | 2020-12-22 |
| CN112115206B true CN112115206B (en) | 2025-05-16 |
Family
ID=73796709
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910531925.1A Active CN112115206B (en) | 2019-06-19 | 2019-06-19 | A method and device for processing object storage metadata |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112115206B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113448946B (en) * | 2021-07-05 | 2024-01-12 | 北京星辰天合科技股份有限公司 | Data migration method and device and electronic equipment |
| CN113609104B (en) * | 2021-08-19 | 2023-11-03 | 京东科技信息技术有限公司 | Method and device for accessing distributed storage system by key value of partial fault |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108319602A (en) * | 2017-01-17 | 2018-07-24 | 广州市动景计算机科技有限公司 | Data base management method and Database Systems |
| CN109074362A (en) * | 2016-05-02 | 2018-12-21 | 谷歌有限责任公司 | Fractionation and moving range in distributed system |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6918013B2 (en) * | 2001-07-16 | 2005-07-12 | Bea Systems, Inc. | System and method for flushing bean cache |
| US7953709B2 (en) * | 2008-03-27 | 2011-05-31 | Emc Corporation | Systems and methods for a read only mode for a portion of a storage system |
| US8996466B2 (en) * | 2008-12-01 | 2015-03-31 | Sap Se | Extend crud to support lifecyle management and business continuity |
| US9411682B2 (en) * | 2010-01-14 | 2016-08-09 | Hewlett Packard Enterprise Development Lp | Scrubbing procedure for a data storage system |
| US8515915B2 (en) * | 2010-09-24 | 2013-08-20 | Hitachi Data Systems Corporation | System and method for enhancing availability of a distributed object storage system during a partial database outage |
| US9626115B2 (en) * | 2015-01-14 | 2017-04-18 | International Business Machines Corporation | Threshold based incremental flashcopy backup of a raid protected array |
| CN111007991B (en) * | 2015-08-13 | 2024-01-26 | 北京忆恒创源科技股份有限公司 | Method for separating read-write requests based on NVDIMM and computer thereof |
| CN108509153B (en) * | 2018-03-23 | 2021-07-06 | 新华三技术有限公司 | OSD selection method, data writing and reading method, monitor and server cluster |
-
2019
- 2019-06-19 CN CN201910531925.1A patent/CN112115206B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109074362A (en) * | 2016-05-02 | 2018-12-21 | 谷歌有限责任公司 | Fractionation and moving range in distributed system |
| CN108319602A (en) * | 2017-01-17 | 2018-07-24 | 广州市动景计算机科技有限公司 | Data base management method and Database Systems |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112115206A (en) | 2020-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11379428B2 (en) | Synchronization of client machines with a content management system repository | |
| CN113297166B (en) | Data processing system, method and device | |
| CN109189835B (en) | Method and device for generating data wide table in real time | |
| CN111061680B (en) | A method and device for data retrieval | |
| US10817472B2 (en) | Storage organization system with associated storage utilization values | |
| CN103002027B (en) | Data-storage system and the method for tree directory structure is realized based on key-value pair system | |
| CN113051221B (en) | Data storage method, device, medium, equipment and distributed file system | |
| US10019452B2 (en) | Topology aware distributed storage system | |
| CN111917834A (en) | A data synchronization method, device, storage medium and computer equipment | |
| CN110858194A (en) | Method and device for expanding database | |
| CN108629029A (en) | A kind of data processing method and device applied to data warehouse | |
| CN114741335B (en) | Cache management method, device, medium and equipment | |
| US10275430B2 (en) | Multimodal sharing of content between documents | |
| CN113760966A (en) | Data processing method and device based on heterogeneous database system | |
| CN112115206B (en) | A method and device for processing object storage metadata | |
| CN112783887A (en) | Data processing method and device based on data warehouse | |
| CN113760600B (en) | Database backup method, database restoration method and related devices | |
| CN111694801A (en) | Data deduplication method and device applied to fault recovery | |
| CN113760860B (en) | Data reading method and device | |
| WO2025097963A1 (en) | Data processing method and apparatus, computer device, and computer readable storage medium | |
| CN114995764B (en) | A method and device for data storage based on streaming computing | |
| CN117389475A (en) | A data processing method and device | |
| CN112783904B (en) | A method and device for updating index data | |
| CN117493355A (en) | Data reading and writing methods, devices, electronic equipment and readable storage media | |
| CN110851192B (en) | Method and device for responding to degraded switch configuration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |