CN108874310A - The read buffer method, apparatus and computer readable storage medium of distributed storage - Google Patents

Abstract

The invention discloses a read cache method and device for distributed storage and a computer-readable storage medium. The method includes: after obtaining the current read request, extracting the current read offset of the file corresponding to the current read request and Record; if the file corresponding to this read request is the same as the file corresponding to the last read request, determine whether the read offset is within the threshold range corresponding to the last read offset; if so, initiate pre-reading; The present invention expands the judgment conditions for triggering pre-reading by judging whether the read offset is within the threshold range corresponding to the last read offset, and can perform normal pre-reading when there is an incomplete sequential read scene. Reading, which enhances the stability of front-end reading performance and improves user experience.

Description

Distributed storage read cache method, device and computer-readable storage medium

technical field

The present invention relates to the field of distributed storage applications, in particular to a distributed storage read cache method, device and computer-readable storage medium.

Background technique

With the development of science and technology in modern society, distributed storage systems that store data in multiple independent devices based on the network have been widely used. In a distributed storage system, in order to improve the read and write performance of the front-end client, the concept of cache is introduced in the system design, and an area in the memory is divided to accommodate the issued read and write request data, especially for read operations. The data pre-read from the back-end storage can improve the cache hit rate and greatly improve the read performance.

In the existing technology, the traditional read cache mechanism in the distributed storage system is only designed for general situations. If the read files are not read in complete order, the pre-read will be invalid, affecting the front-end reading performance and not conducive to user experience. Therefore, how to improve the existing read cache mechanism, enhance the stability of front-end read performance, and improve user experience is an urgent problem to be solved today.

Contents of the invention

The purpose of the present invention is to provide a distributed storage read cache method, device and computer-readable storage medium, to expand the judgment conditions for triggering pre-reading on the basis of the current pre-reading mechanism, and enhance the stability of front-end reading performance , improve user experience.

In order to solve the above technical problems, the present invention provides a read cache method for distributed storage, including:

After obtaining the current read request, extract and record the current read offset of the file corresponding to the current read request;

If the file corresponding to the read request this time is the same as the file corresponding to the last read request, determine whether the read offset is within the threshold range corresponding to the last read offset;

If so, initiate read-ahead.

Optionally, after obtaining the read request, it also includes:

The current read length of the file corresponding to the current read request is extracted and recorded.

Optionally, the judging whether the read offset is within the threshold range corresponding to the last read offset includes:

Judging whether the current read offset is greater than or equal to the difference between the last read offset and the first threshold, and is less than or equal to the difference between the last read offset, the last read length, and the second threshold and;

If yes, execute the step of initiating pre-reading.

Optionally, after extracting and recording the previous read offset and the current read length, it also includes:

If the file corresponding to the current read request is different from the file corresponding to the last read request, send the current read request to the backend storage to obtain the data corresponding to the current read request.

Optionally, the method also includes:

Calculate and generate or update the read code stream of the file corresponding to the current read request according to the recorded current read offset and the current read length;

According to the ratio of the read code stream of the file corresponding to the current read request to the read code stream of all the files corresponding to all the read requests, assign the corresponding file to the file corresponding to the current read request in the cache of the preset capacity capacity.

The present invention also provides a read cache device for distributed storage, including:

The extracting record module is used to extract and record the current reading offset of the file corresponding to the current reading request after obtaining the current reading request;

A judging module, configured to judge whether the current read offset is within the threshold range corresponding to the last read offset if the file corresponding to the current read request is the same as the file corresponding to the last read request;

A read-ahead module, configured to initiate a read-ahead if the current read offset is within the threshold range.

Optionally, the extracting record module includes:

The extract record submodule is configured to extract and record the current read offset and the current read length of the file corresponding to the current read request after obtaining the current read request.

Optionally, the judging module includes:

A judging submodule, used to judge whether the read offset is greater than or equal to the difference between the last read offset and the first threshold, and is less than or equal to the last read offset, the last read The sum of the length and the second threshold; if yes, send an activation signal to the pre-reading module.

Optionally, the device also includes:

A calculation module, configured to calculate, generate or update the read code stream of the file corresponding to the current read request according to the recorded current read offset and the current read length;

The cache allocation module is configured to, according to the ratio of the read code stream of the file corresponding to the current read request to the read code stream of all files corresponding to all read requests, provide the current read file in a cache with a preset capacity. Request the corresponding file to allocate the corresponding capacity.

In addition, the present invention also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the distributed storage as described in any one of the above-mentioned The steps of the read cache method.

A read cache method for distributed storage provided by the present invention includes: after acquiring the read request, extracting and recording the current read offset of the file corresponding to the read request; The file is the same as the file corresponding to the last read request, and judge whether the read offset is within the threshold range corresponding to the last read offset; if so, initiate pre-reading;

It can be seen that the present invention expands the judgment conditions for triggering pre-reading by judging whether the read offset is within the threshold range corresponding to the last read offset, and can perform normal reading when incomplete sequential reading occurs. Pre-reading enhances the stability of front-end reading performance and improves user experience. In addition, the present invention also provides a read cache device for distributed storage and a computer-readable storage medium, which also have the above beneficial effects.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

FIG. 1 is a flow chart of a read cache method for distributed storage provided by an embodiment of the present invention;

FIG. 2 is a flow chart of another read cache method for distributed storage provided by an embodiment of the present invention;

Fig. 3 is a trigger pre-reading display diagram of another distributed storage read cache method provided by an embodiment of the present invention;

Fig. 4 is a cache adjustment flow chart of another distributed storage read cache method provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of a memory structure of another read cache method for distributed storage provided by an embodiment of the present invention;

FIG. 6 is a structural diagram of a read cache device for distributed storage provided by an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to FIG. 1 . FIG. 1 is a flow chart of a read cache method for distributed storage provided by an embodiment of the present invention. The method can include:

Step 101: After obtaining the current read request, extract and record the current read offset of the file corresponding to the current read request.

Wherein, this step may be that the front-end client (processor) connected to the back-end storage in the distributed storage system extracts the read offset of the file corresponding to the read request after receiving the read request sent by the front-end.

It can be understood that the current read request in this step may be a read request currently received by the front-end client from the front-end. That is to say, in this step, after the front-end client receives each read request, it can extract and record the read offset of the file corresponding to each read request. For the specific method of extracting and recording the read offset of the file corresponding to the read request in this step, the designer can set it up according to the practical scenario and user needs, for example, it can be implemented in a similar way to the existing technology . As long as the front-end client can extract the read offset of the file corresponding to the read request after receiving the read request. This embodiment does not impose any limitation on this.

It should be noted that the purpose of this step can be to prepare for judging whether the read offset is within the threshold range corresponding to the last read offset in step 102 through the extracted current read offset

Step 102: If the file corresponding to the current read request is the same as the file corresponding to the last read request, determine whether the read offset is within the threshold range corresponding to the last read offset; if so, proceed to step 103 .

Wherein, the last read request in this step may be a read request received by the front-end client before receiving this read request. Correspondingly, the last read offset may be the read offset of the file corresponding to the last read request extracted and recorded by the front-end client through step 101 after receiving the last read request.

It can be understood that the purpose of this step can be to determine whether the read offset is higher than the last read offset when the file corresponding to the current read request is the same as the file corresponding to the last read request. Within the corresponding threshold range, determine whether to trigger pre-reading, so as to expand the judgment conditions for triggering pre-reading, tolerate small jitters when reading files, so that the front-end client can perform normal pre-reading when incomplete sequential reading occurs. read.

Specifically, the threshold range corresponding to the last read offset in this step can be set by the designer according to practical scenarios and user needs. For example, the threshold range can be set as the last read offset minus A threshold and the last read offset plus the threshold, that is, to determine whether the read offset is within the threshold range corresponding to the last read offset, which can be used to determine the current read offset and Whether the absolute value of the difference between the last read offsets is less than or equal to this threshold; the threshold range can also be set to last read offset minus one threshold and last read offset plus another Threshold; the threshold range can also be set between the difference between the last read offset and the first threshold and the sum of the last read offset, last read length, and the second threshold, that is, to judge this time Whether the read offset is within the threshold range corresponding to the last read offset can be judged whether the current read offset is greater than or equal to the difference between the last read offset and the first threshold, and less than or equal to The sum of the last read offset, the last read length, and the second threshold. This embodiment does not impose any limitation on this.

Wherein, the last read length may be the read length of the file corresponding to the last read request extracted and recorded by the front-end client after receiving the last read request. The acquisition method of the last read length can be set by the designer himself. For example, when extracting and recording the read offset of the file corresponding to each read request in step 101, extract and record the offset of the file corresponding to each read request. Read length. As long as it can be judged whether the read offset is greater than or equal to the difference between the last read offset and the first threshold, and is less than or equal to the sum of the last read offset, the last read length, and the second threshold Before, the length of the last read is obtained, which is not limited in this embodiment.

It should be noted that this step is to judge when the file corresponding to this read request is the same as the file corresponding to the last read request, whether the file corresponding to this read request is the same as the file corresponding to the last read request The specific judgment process and judgment flow are set by the designer. If it can be realized in a similar way to the existing technology, as long as it can be guaranteed that this step is performed when the last read request and this read request correspond to the same file Judging, this embodiment does not impose any limitation on this.

Correspondingly, for the situation that the file corresponding to this read request is different from the file corresponding to the last read request, a method similar to the existing technology can be adopted, and the front-end client obtains the corresponding file from the back-end storage according to the current read request. The data. If the file corresponding to this read request is different from the file corresponding to the last read request, the front-end client can send this read request to the back-end storage to obtain the data corresponding to this read request.

Specifically, if the read offset is not within the threshold range corresponding to the last read offset, it can be considered as a jumpy read, which is not suitable for initiating read-ahead to occupy bandwidth and space, and does not trigger read-ahead , the front-end client can directly obtain the corresponding data from the back-end storage according to the current read request in a manner similar to the prior art.

Step 103: Initiate pre-reading.

Among them, the purpose of this embodiment can be to expand the judgment conditions for triggering pre-reading through step 102, and to tolerate the slight jitter that occurs when the file is read, so that the front-end client can perform normal pre-reading when there is an incomplete sequential reading scene. . For the specific process and method of initiating pre-reading in this step after pre-reading is triggered, the designer can set it up in a similar way to the existing technology according to practical scenarios and user needs. Read ahead with the data within the threshold after the read length. This implementation does not impose any restrictions on this.

In this embodiment, the embodiment of the present invention expands the judgment conditions for triggering pre-reading by judging whether the read offset this time is within the threshold range corresponding to the last read offset, so that incomplete sequential reads can occur Perform normal pre-reading in certain scenarios, which enhances the stability of front-end reading performance and improves user experience.

Please refer to FIG. 2 . FIG. 2 is a flowchart of another distributed storage read caching method provided by an embodiment of the present invention. The method can include:

Step 201: After obtaining the current read request, extract and record the current read offset and current read length of the file corresponding to the current read request.

Specifically, as shown in Figure 3, this step can be for the front-end client (processor) connected to the back-end storage to extract the read file corresponding to the read request after receiving the read request sent by the front-end this time. Offset (offset') and read length (len').

Step 202: If the file corresponding to this read request is the same as the file corresponding to the last read request, determine whether the read offset is greater than or equal to the difference between the last read offset and the first threshold, and is less than or equal to the above The sum of the second read offset, the last read length and the second threshold; if yes, go to step 103.

Wherein, as shown in FIG. 3 , the last read offset and the last read length may be the read offset of the file corresponding to the read request extracted by the front-end client after receiving the read request sent by the front-end last time. Shift (offset) and read length (len).

It can be understood that the setting of the specific values of the first threshold and the second threshold in this step can be set by the designer according to practical scenarios and user needs, for example, both the first threshold and the second threshold can be set to 1MB . This embodiment does not impose any limitation on this.

Specifically, as shown in FIG. 3, when the first threshold and the second threshold are both 1MB, offset-1MB<offset'<offset+len+1MB satisfies the pre-reading trigger judgment in this step, which can be performed through step 203 read ahead.

Step 203: Pre-read the data within the third threshold after the sum of the current read offset and the current read length.

Specifically, the setting of the specific numerical value of the third threshold in this step can be set by the designer according to practical scenarios and user needs. For example, the third threshold can be set to 4MB, as shown in FIG. 3 , for offset' The 4MB file data after +len' initiates pre-reading, so that subsequent data requests can be directly hit and improve performance. This embodiment does not impose any limitation on this.

In this embodiment, the embodiment of the present invention judges whether the current read offset is greater than or equal to the difference between the last read offset and the first threshold, and is less than or equal to the last read offset, the last read The sum of the length and the second threshold further expands the judgment conditions for triggering pre-reading, limits the range of tolerance for small jitters when reading files, enhances the stability of front-end reading performance, and improves user experience.

Based on any of the above-mentioned embodiments, in order to ensure full utilization of the cache, prevent unbalanced memory usage, and affect front-end business performance, the method provided in this embodiment may also include the process of dynamically adjusting the cache as shown in Figure 4 . Specifically, please refer to FIG. 4 . FIG. 4 is a cache adjustment flow chart of another read cache method for distributed storage provided by an embodiment of the present invention. The method can include:

Step 301: Calculate, generate or update the read code stream of the file corresponding to the current read request according to the recorded current read offset and the current read length.

Wherein, this step can be that when the file corresponding to this read request does not have a read code stream, that is, when the file corresponding to the previous read request is different from the file corresponding to this read request, the processor will Shift and the length of this read, calculate and generate the read code stream of the file corresponding to this read request; when the file corresponding to this read request has a read code stream, that is, the file corresponding to this read request is different from the previous When the files corresponding to the read requests are the same, the processor calculates and updates the read stream of the files corresponding to the read requests according to the current read offset and the current read length.

Specifically, the specific way of calculating and generating or updating the read code stream of the file corresponding to this read request in this step can be set by the designer himself. For example, it can be set in a way similar to the prior art. There are no restrictions on this.

Step 302: According to the ratio of the read code stream of the file corresponding to this read request to the read code stream of all files corresponding to all read requests, assign the corresponding file to the file corresponding to this read request in the cache with a preset capacity. capacity.

It can be understood that the read code streams of all files corresponding to all read requests in this step may be the read code streams of all different files corresponding to all read requests, that is, the number of read code streams corresponds to all read requests The number of all different files is the same, and the same file corresponding to multiple read requests corresponds to only one read code stream.

Specifically, as shown in FIG. 5, this step can be that the processor counts the read code streams (v1+v2+... vn, n is the number of all files), allocate a cache with a preset capacity (memory_cache) in proportion, for example, in the cache with a preset capacity, the cache size allocated for the file corresponding to this read request can be memory_cache*v1/(v1 +v2+...vn).

It should be noted that the division of the specific cache size (capacity) of the cache with the preset capacity can be set by the designer or the user, and this embodiment does not impose any limitation on this.

Correspondingly, this step may also include a step of adjusting the cache size of all files except the file corresponding to this read request. This embodiment does not impose any limitation on this.

In this embodiment, according to the ratio of the read code stream of the file corresponding to this read request to the read code stream of all files corresponding to all read requests, the embodiment of the present invention provides the current read file in the cache with a preset capacity. Request the corresponding file to allocate the corresponding capacity, dynamically allocate the cache space occupied by each file, realize the dynamic adjustment of the cache, maximize the efficiency of the cache usage, and avoid the occurrence of unbalanced memory usage, which in turn affects the performance of the front-end business.

Please refer to FIG. 6 . FIG. 6 is a structural diagram of a read cache device for distributed storage provided by an embodiment of the present invention. The device can include:

The extracting record module 100 is used to extract and record the read offset of the file corresponding to the read request after obtaining the read request;

The judging module 200 is used to determine whether the read offset is within the threshold range corresponding to the last read offset if the file corresponding to the read request is the same as the file corresponding to the last read request;

The read-ahead module 300 is configured to initiate a read-ahead if the read offset is within a threshold range.

Optionally, extracting the recording module 100 may include:

The extract record sub-module is used to extract and record the current read offset and current read length of the file corresponding to the current read request after obtaining the current read request.

Optionally, the judging module 200 includes:

The judging sub-module is used to judge whether the read offset is greater than or equal to the difference between the last read offset and the first threshold, and is less than or equal to the last read offset, the last read length and the second threshold. The sum of the thresholds; if so, send a start signal to the pre-reading module.

Optionally, the device may also include:

A sending module, configured to send the current read request to the backend storage if the file corresponding to the current read request is different from the file corresponding to the last read request, so as to obtain the file corresponding to the current read request data.

Optionally, the device may also include:

A calculation module, configured to calculate, generate or update the read code stream of the file corresponding to the read request according to the recorded current read offset and the current read length;

The cache allocation module is used to assign the file corresponding to this read request in the cache with a preset capacity according to the ratio of the read code stream of the file corresponding to this read request to the read code stream of all files corresponding to all read requests Allocate the corresponding capacity.

In this embodiment, the embodiment of the present invention uses the judging module 200 to judge whether the read offset is within the threshold range corresponding to the last read offset, and expands the judgment conditions for triggering pre-reading. Normal pre-reading is performed in sequential reading scenarios, which enhances the stability of front-end reading performance and improves user experience.

The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the steps of the read-cache method for distributed storage provided by any of the above-mentioned embodiments can be realized. The storage medium may include various media capable of storing program codes such as a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk.

Each embodiment in the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the device and the computer-readable storage medium disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the relevant details, please refer to the description of the method part.

Professionals can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the possible For interchangeability, in the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The read caching method, device and computer-readable storage medium for distributed storage provided by the present invention have been introduced in detail above. In this paper, specific examples are used to illustrate the principle and implementation of the present invention, and the descriptions of the above embodiments are only used to help understand the method and core idea of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (10)

1. A read cache method for distributed storage, comprising: After obtaining the current read request, extract and record the current read offset of the file corresponding to the current read request; If the file corresponding to the read request this time is the same as the file corresponding to the last read request, determine whether the read offset is within the threshold range corresponding to the last read offset; If so, initiate read-ahead. 2. The read caching method for distributed storage according to claim 1, wherein, after obtaining the read request, further comprising: The current read length of the file corresponding to the current read request is extracted and recorded. 3. The read cache method for distributed storage according to claim 2, wherein the judging whether the read offset is within the threshold range corresponding to the last read offset comprises: Judging whether the current read offset is greater than or equal to the difference between the last read offset and the first threshold, and is less than or equal to the difference between the last read offset, the last read length, and the second threshold and; If yes, execute the step of initiating pre-reading. 4. The read cache method of distributed storage according to claim 2, characterized in that, after extracting the previous read offset and this read length and recording, further comprising: If the file corresponding to the current read request is different from the file corresponding to the last read request, send the current read request to the backend storage to obtain the data corresponding to the current read request. 5. The read cache method for distributed storage according to any one of claims 2 to 4, further comprising: Calculate and generate or update the read code stream of the file corresponding to the current read request according to the recorded current read offset and the current read length; According to the ratio of the read code stream of the file corresponding to the current read request to the read code stream of all the files corresponding to all the read requests, assign the corresponding file to the file corresponding to the current read request in the cache of the preset capacity capacity. 6. A read cache device for distributed storage, comprising: The extracting record module is used to extract and record the current reading offset of the file corresponding to the current reading request after obtaining the current reading request; A judging module, configured to judge whether the current read offset is within the threshold range corresponding to the last read offset if the file corresponding to the current read request is the same as the file corresponding to the last read request; A read-ahead module, configured to initiate a read-ahead if the current read offset is within the threshold range. 7. The read cache device for distributed storage according to claim 6, wherein the extracting and recording module comprises: The extract record submodule is configured to extract and record the current read offset and the current read length of the file corresponding to the current read request after obtaining the current read request. 8. The read cache device for distributed storage according to claim 7, wherein the judging module comprises: A judging submodule, used to judge whether the read offset is greater than or equal to the difference between the last read offset and the first threshold, and is less than or equal to the last read offset, the last read The sum of the length and the second threshold; if yes, send an activation signal to the pre-reading module. 9. The read cache device for distributed storage according to any one of claims 7 to 9, further comprising: A calculation module, configured to calculate, generate or update the read code stream of the file corresponding to the current read request according to the recorded current read offset and the current read length; The cache allocation module is configured to, according to the ratio of the read code stream of the file corresponding to the current read request to the read code stream of all files corresponding to all read requests, provide the current read file in a cache with a preset capacity. Request the corresponding file to allocate the corresponding capacity. 10. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the distribution according to any one of claims 1 to 5 is realized. The steps of the read caching method for storage.