CN110532228B - Method, system and equipment for reading block chain data and readable storage medium - Google Patents

Abstract

The present application discloses a method for reading blockchain data, including: receiving a data reading request; determining a keyword of the data to be read according to the data reading request; calling the underlying LSM storage engine to sequentially store data in an in-memory database and a temporary database The data item corresponding to the keyword is searched; if the data item corresponding to the keyword is not found, the data item corresponding to the keyword is searched in parallel in each level of the file system. In the present application, when the data item corresponding to the keyword is not found in the in-memory database and the temporary database, the data item corresponding to the keyword is searched in parallel in each level of the file system, so as to reduce the time overhead of the blockchain read operation, greatly reducing the time overhead of the blockchain read operation. It reduces the problem of slow reading and writing caused by too many LSM layers, so that the LSM read performance has been significantly improved. The present application also provides a system, device and computer-readable storage medium for reading blockchain data, which have the above beneficial effects.

Description

A method, system, device and readable storage medium for reading blockchain data

technical field

The present application relates to the field of blockchain, and in particular, to a method, system, device and computer-readable storage medium for reading blockchain data.

Background technique

The blockchain stores data through the LSM Tree (Log Structure Merge Tree) structure. The traditional LSM Tree can be stored through level db (a very efficient kv database implemented by google) and rocksdb (embedded Key-value storage). System) to realize the storage of data, the advantage of which is to achieve the optimal write performance by converting the random write of the disk into a sequential write, so as to quickly build an index.

However, as the amount of data increases and the level increases, the problem of LSM read amplification becomes more obvious, and the overhead of LSM compaction surges, especially in a stressful environment, which affects the overall performance of the blockchain process.

Therefore, how to improve the overall read performance of the blockchain is a technical problem that needs to be solved by those skilled in the art.

SUMMARY OF THE INVENTION

The purpose of this application is to provide a method, system, device and computer-readable storage medium for reading blockchain data, which are used to improve the overall reading performance of the blockchain.

In order to solve the above technical problems, the present application provides a method for reading blockchain data, the method comprising:

Receive data read requests;

Determine the keyword of the data to be read according to the data read request;

Invoke the underlying LSM storage engine to find the data item corresponding to the keyword in the memory database and the temporary database in turn;

If the data item corresponding to the keyword is not found, the data item corresponding to the keyword is searched in parallel in each level of the file system.

Optionally, after the parallel search for the data item corresponding to the keyword in each level of the file system, the method further includes:

Determine whether the data item corresponding to the keyword is found;

If not, return an error message;

If yes, then judge whether the found data item is unique;

When the data item is not unique, the data item with the lowest level is returned.

Optionally, also include:

Receive data write request;

Invoke the underlying LSM storage engine to write the data to be written corresponding to the data write request into the log file, and write the data to be written into the in-memory database;

When the data stored in the in-memory database reaches a predetermined threshold, a secondary merge operation is performed to persist the data stored in the in-memory database to the file system.

Optionally, after the performing the secondary merge operation, the method further includes:

Get the status information of the blockchain process;

When the state of the blockchain process is idle, the old data in the log file is recovered.

Optionally, the storage medium of the file system is a solid-state hard disk;

The size of the SSTable file in the file system is at least one of 1mb, 2mb, and 4mb.

Optionally, the parallel search for the data item corresponding to the keyword in each level of the file system includes:

mapping data items in the zero-level of the file system to the in-memory database by means of a memory-mapped file;

Search the data item corresponding to the keyword in the memory database.

The application also provides a system for reading blockchain data, the system comprising:

a first receiving module, configured to receive a data read request;

a determining module, configured to determine the keyword of the data to be read according to the data read request;

The first calling module is used to call the underlying LSM storage engine to search the data item corresponding to the keyword in the memory database and the temporary database in turn;

The parallel search module is configured to search for the data item corresponding to the keyword in parallel in each level of the file system when the data item corresponding to the keyword is not found.

Optionally, also include:

a first judgment module, used for judging whether the data item corresponding to the keyword is found;

The first return module is used to return error prompt information when the data item corresponding to the keyword is not found;

a second judgment module, configured to judge whether the found data item is unique when the data item corresponding to the keyword is found;

The second returning module is configured to return the data item with the lowest level when the data item is not unique.

The application also provides a blockchain data reading device, the blockchain data reading device includes:

memory for storing computer programs;

The processor is configured to implement the steps of the method for reading blockchain data according to any one of the above when executing the computer program.

The present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method for reading blockchain data according to any one of the above A step of.

The method for reading blockchain data provided by this application includes: receiving a data reading request; determining the keyword of the data to be read according to the data reading request; calling the underlying LSM storage engine to sequentially search for the key in the memory database and the temporary database If the data item corresponding to the keyword is not found, the data item corresponding to the keyword is searched in parallel in each level of the file system.

The technical solution provided by the present application, when the data item corresponding to the keyword is not found in the memory database and the temporary database, the data item corresponding to the keyword is searched in parallel in each level of the file system, so as to reduce the block chain read operation The time overhead of LSM is provided to the blockchain in the form of a plug-in, which greatly reduces the slow reading and writing problem caused by too many LSM layers, and greatly improves the LSM read performance. The present application also provides a system, device and computer-readable storage medium for reading blockchain data, which have the above beneficial effects, and will not be repeated here.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only It is an embodiment of the present application. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without any creative effort.

1 is a flowchart of a method for reading blockchain data provided by an embodiment of the present application;

Figure 2 is a schematic diagram of the LSM storage engine performing a secondary merge operation;

Fig. 3 is the schematic diagram that LSM storage engine performs main merge operation;

4 is a flowchart of another method for reading blockchain data provided by an embodiment of the present application;

5 is a structural diagram of a system for reading blockchain data provided by an embodiment of the present application;

FIG. 6 is a structural diagram of another blockchain data reading system provided by an embodiment of the present application;

FIG. 7 is a structural diagram of a block chain data reading device provided by an embodiment of the present application.

Detailed ways

The core of this application is to provide a method, system, device and computer-readable storage medium for reading blockchain data, which are used to improve the overall reading performance of the blockchain.

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Please refer to FIG. 1 , which is a flowchart of a method for reading blockchain data provided by an embodiment of the present application.

It specifically includes the following steps:

S101: Receive a data read request;

The blockchain stores data through the LSM Tree structure. LSM Tree is a hard disk-based data structure. Compared with B+tree, it can significantly reduce the overhead of hard disk and disk seek, and can provide files for a long time. high-speed insertion (deletion). However, LSM-tree performs poorly in some cases, especially when queries require fast responses. Usually LSM-tree is suitable for applications where index insertion is more frequent than retrieval.

When the size of the data stored in the in-memory database reaches a preset predetermined threshold, the LSM storage engine will perform a minor consolidation operation (Minor Compaction) to persist the data in the in-memory database to the file system, please refer to Figure 2, Figure 2 A schematic diagram of performing a secondary merge operation for the LSM storage engine, as shown in Figure 2, the secondary merge operation is to select one or more small, adjacent dump SSTable files and zero or more data stored in in-memory databases Merged into a larger SSTable file, the result of a Minor Compaction is fewer and larger SSTable files;

The secondary merge operation is triggered when there are three conditions as follows:

1) When the number of files at level 0 exceeds the predetermined upper limit;

2) When the total file size of a layer exceeds the maximum capacity limit of the layer;

3) When a certain file is read invalidly too many times;

Please refer to FIG. 3. FIG. 3 is a schematic diagram of the LSM storage engine performing the main merge operation. As shown in FIG. 3, the major merge operation (Major Compaction) is to merge multiple low-level SSTable files into a high-level SSTable file. Under normal circumstances, the main merge operation time will last for a long time, and the whole process will consume a lot of system resources, which will have a relatively large impact on the upper-layer business. When the amount of data increases, the level level will increase accordingly, resulting in LSM read amplification problem It is more obvious, so the present application provides a method for reading blockchain data, which is used to solve the above problems.

S102: Determine the keyword of the data to be read according to the data read request;

S103: Invoke the underlying LSM storage engine to sequentially search the in-memory database and the temporary database for the data item corresponding to the keyword;

The value based on the data to be read may exist in an in-memory database, a temporary database, or a file system. Therefore, in this embodiment of the present application, after the keyword of the data to be read is determined according to the data read request, the underlying LSM storage engine is called to store the data in the memory in turn. The data item corresponding to the keyword is searched in the database and the temporary database. If the data item is found, the data item can be directly returned to complete the data read operation.

S104: If the data item corresponding to the keyword is not found, search for the data item corresponding to the keyword in each hierarchy of the file system in parallel.

Optionally, the parallel search for data items corresponding to keywords in each level of the file system mentioned here may specifically be:

Map the data items in the zero-level file system to the in-memory database by means of memory-mapped files;

Find the data item corresponding to the keyword in the memory database;

Since the level 0 layer is a special layer, its SSTable file is directly obtained by the Minor Compaction operation performed by memtable, so the keys in the level 0 layer are out of order, and an LSM read operation may read multiple files in level 0. , in extreme scenarios, we will serially read all SSTable files in the level 0 layer, and the read operations in level 0 can also perform concurrent read operations. However, the storage space of the SSTable file in level 0 is small, so in the embodiment of the present application, the file in the level 0 layer is mapped to the memory by mmap to improve the read performance;

Optionally, after the data items corresponding to the keywords are searched in parallel in each level of the file system, the following steps may also be performed:

Determine whether the data item corresponding to the keyword is found;

If not, return an error message;

If yes, then judge whether the found data item is unique;

When the data item is not unique, the data item with the lowest level is returned.

Optionally, when the data item is unique, just return the data item directly;

The purpose of returning the data item with the lowest level when the data item is not unique mentioned here is that the data with the lower level in the file system is newer, so the data item with the lowest level is returned;

Preferably, because the Major Compaction operation of LSM will be triggered when a certain file has too many invalid reads, and the concurrent read feature will undoubtedly magnify the impact of invalid file reads, the embodiment of the present application may also include files in the concurrent read scheme. Remove the condition that invalid read triggers Major Compaction of LSM to avoid performance loss caused by Major Compaction operation caused by excessive number of invalid file reads;

Preferably, since solid-state drives have the advantage of random read performance, blockchain nodes using solid-state drives can use this solution to increase the read performance of LSM; on this basis, the size of the SSTable file stored in the blockchain LSM can be changed from 64kb adjusted to 1mb, 2mb or 4mb to make SSTable files more suitable for SSD storage media;

Since the common solid-state hard disk storage media currently on the market can support a read and write capability of 100,000 iops, which can fully meet the level-based concurrent read characteristics, the technical solution provided in this application can make the random read performance of the solid-state hard disk be utilized. The time that originally needed to read N layers is uniformly optimized to the time to read 1 layer. Because solid-state drives have high iops capability, and because the blockchain reads a small amount of db data per second, the LSM concurrent read solution is It can be well displayed in the blockchain project;

Based on the above technical solutions, a method for reading blockchain data provided by the present application, by searching for the key in each level of the file system in parallel when the data item corresponding to the keyword is not found in the memory database and the temporary database The data item corresponding to the word can reduce the time overhead of the blockchain read operation, and provide it to the blockchain in the form of a plug-in. Performance has been significantly improved.

On the basis of the above embodiment, when a data write command is received, the steps shown in FIG. 4 may also be included, which will be described below with reference to FIG. 4 .

Please refer to FIG. 4 , which is a flowchart of another method for storing blockchain data provided by an embodiment of the present application.

It specifically includes the following steps:

S401: Receive a data write request;

S402: Invoke the underlying LSM storage engine to write the data to be written corresponding to the data write request into the log file, and write the data to be written into the in-memory database;

S403: When the data stored in the in-memory database reaches a predetermined threshold, perform a secondary merge operation to persist the data stored in the in-memory database to the file system.

The minor merge operation mentioned here is the Minor Compaction mentioned above;

Optionally, after the secondary merge operation is performed, the following steps may also be performed:

Get the status information of the blockchain process;

When the state of the blockchain process is idle, the old data in the log file is recycled.

In the embodiment of the present application, old data in the log file is recovered when the blockchain process is idle, so as to ensure that the storage overhead is not infinitely amplified.

Please refer to FIG. 5 , which is a structural diagram of a system for reading blockchain data provided by an embodiment of the present application.

The system can include:

a first receiving module 100, configured to receive a data read request;

A determination module 200, configured to determine the keyword of the data to be read according to the data read request;

The first calling module 300 is used for calling the underlying LSM storage engine to search the data item corresponding to the keyword in the in-memory database and the temporary database in turn;

The parallel search module 400 is configured to search for the data item corresponding to the keyword in parallel in each level of the file system when the data item corresponding to the keyword is not found.

Please refer to FIG. 6 , which is a structural diagram of another system for reading blockchain data provided by an embodiment of the present application.

The system may also include:

The first judgment module is used for judging whether the data item corresponding to the keyword is found;

The first return module is used to return an error message when the data item corresponding to the keyword is not found;

The second judgment module is used for judging whether the found data item is unique when the data item corresponding to the keyword is found;

The second return module is used to return the data item with the lowest level when the data item is not unique.

The system may also include:

a second receiving module, configured to receive a data write request;

The second calling module is used to call the underlying LSM storage engine to write the data to be written corresponding to the data write request into the log file, and to write the data to be written into the in-memory database;

The execution module is configured to perform a secondary merge operation when the data stored in the in-memory database reaches a predetermined threshold, so as to persist the data stored in the in-memory database to the file system.

The system may also include:

The acquisition module is used to acquire the status information of the blockchain process;

The recycling module is used to recycle the old data in the log file when the state of the blockchain process is idle.

Optionally, the storage medium of the file system may be a solid-state hard disk;

The size of the SSTable file in the file system may be at least one of 1mb, 2mb, and 4mb.

The parallel search module 400 may include:

The mapping submodule is used to map the data items in the zero-level file system to the in-memory database by means of memory-mapped files;

The lookup submodule is used to look up the data item corresponding to the keyword in the memory database.

Each component in the above system can be practically applied in the following embodiments:

The first receiving module receives the data reading request; the determining module determines the keyword of the data to be read according to the data reading request; the first calling module invokes the underlying LSM storage engine to sequentially search the in-memory database and the temporary database for the data item corresponding to the keyword ; When the data item corresponding to the keyword is not found, the parallel search module searches for the data item corresponding to the keyword in parallel in each level of the file system.

The first judgment module judges whether the data item corresponding to the keyword is found; when the data item corresponding to the keyword is not found, the first return module returns an error message; when the data item corresponding to the keyword is found, the second judgment The module judges whether the found data item is unique; when the data item is not unique, the second return module returns the data item with the lowest level.

The second receiving module receives the data write request; the second calling module calls the underlying LSM storage engine to write the data to be written corresponding to the data write request into the log file, and writes the data to be written into the memory database; when the memory When the data stored in the database reaches a predetermined threshold, the execution module performs a secondary merge operation to persist the data stored in the in-memory database to the file system; the acquisition module obtains the status information of the blockchain process; When the status is idle, the recycling module recycles the old data in the log file.

Please refer to FIG. 7 , which is a structural diagram of a blockchain data reading device provided by an embodiment of the present application.

The blockchain data reading device 700 may vary greatly due to different configurations or performances, and may include one or more central processing units (CPU) 722 (eg, one or more processors) and memory 732, one or more storage media 730 (eg, one or more mass storage devices) that store applications 742 or data 744. Among them, the memory 732 and the storage medium 730 may be short-term storage or persistent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instructions to operate on the apparatus. Furthermore, the central processing unit 722 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the blockchain data reading device 700 .

The blockchain data reading device 700 may also include one or more power supplies 727, one or more wired or wireless network interfaces 750, one or more input and output interfaces 758, and/or, one or more operating systems 741, Such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM and so on.

The steps in the method for reading blockchain data described above in FIGS. 1 to 4 are implemented by the blockchain data reading device based on the structure shown in FIG. 7 .

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and module described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed apparatuses, devices and methods may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or components may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or modules, and may be in electrical, mechanical or other forms.

Modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical modules, that is, may be located in one place, or may be distributed to multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically alone, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules.

The integrated modules, if implemented in the form of software functional modules and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , which includes several instructions to cause a computer device (which may be a personal computer, a function invocation device, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

The method, system, device, and computer-readable storage medium for reading blockchain data provided by the present application have been described in detail above. Specific examples are used herein to illustrate the principles and implementations of the present application, and the descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can also be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

It should also be noted that, in this specification, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities or operations. There is no such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article, or device that includes the element.

Claims (7)

1. A method for reading block chain data, comprising: Receive data read requests; Determine the keyword of the data to be read according to the data read request; Invoke the underlying LSM storage engine to find the data item corresponding to the keyword in the memory database and the temporary database in turn; If the data item corresponding to the keyword is not found, map the data item in the zero level of the file system to the memory database by means of a memory-mapped file, and search the memory database for the data item corresponding to the keyword. data item; Determine whether the data item corresponding to the keyword is found; If not, return an error message, and remove the condition that the invalid read of the file triggers the Major Compaction of LSM; If yes, then judge whether the found data item is unique; When the data item is not unique, the data item with the lowest level is returned. 2. The method of claim 1, further comprising: Receive data write request; Invoke the underlying LSM storage engine to write the data to be written corresponding to the data write request into the log file, and write the data to be written into the in-memory database; When the data stored in the in-memory database reaches a predetermined threshold, a secondary merge operation is performed to persist the data stored in the in-memory database to the file system. 3. The method according to claim 2, wherein after the performing the secondary merge operation, further comprising: Get the status information of the blockchain process; When the state of the blockchain process is idle, the old data in the log file is recovered. 4. The method according to claim 1, wherein the storage medium of the file system is a solid-state hard disk; The size of the SSTable file in the file system is at least one of 1mb, 2mb, and 4mb. 5. A system for reading blockchain data, comprising: a first receiving module, configured to receive a data read request; a determining module, configured to determine the keyword of the data to be read according to the data read request; The first calling module is used to call the underlying LSM storage engine to search the data item corresponding to the keyword in the memory database and the temporary database in turn; A parallel search module, configured to map the data items in the zero-level file system to the memory database by means of a memory-mapped file when the data item corresponding to the keyword is not found, and search in the memory database the data item corresponding to the keyword; The first judgment module is used to judge whether the data item corresponding to the keyword is found; the first return module is used to return an error message when the data item corresponding to the keyword is not found, and remove the invalid file Read the condition that triggers the Major Compaction of LSM; a second judgment module, configured to judge whether the found data item is unique when the data item corresponding to the keyword is found; The second returning module is configured to return the data item with the lowest level when the data item is not unique. 6. A block chain data reading device, characterized in that it comprises: memory for storing computer programs; The processor is configured to implement the steps of the method for reading blockchain data according to any one of claims 1 to 4 when executing the computer program. 7. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the blocks described in any one of claims 1 to 4 are implemented Chain data read method steps.

Images