CN109033328A - A kind of access request processing method, device, equipment and readable storage medium storing program for executing - Google Patents

Abstract

The invention discloses a kind of access request processing methods, method includes the following steps: acquisition pair/dev/nbdx access request；Access request is sent to reader process by nsock using the nbdkernel thread in nbdkernel module；Reader process calls librbd that access request is issued distributed file system, so that distributed file system handles access request；The response that distributed file system returns is received using librbd process, and will be responded using writer process and nbdclient is sent to by dsock, so that nbdclient processing responds.The performance of rbd is improved, rbd stability is enhanced, promotes user experience.The invention also discloses a kind of access request processing unit, equipment and readable storage medium storing program for executing, have corresponding technical effect.

Description

Access request processing method, device, device and readable storage medium

technical field

The present invention relates to the field of storage technology, in particular to an access request processing method, device, device and readable storage medium.

Background technique

rbd is a tool for manipulating rados block device (RBD) images. QEMU/KVM is used through the Linuxrbd kernel driver and rbd storage driver. Among them, the RBD image is a simple block device, which is striped into small objects and stored in the RADOS (Reliable, Autonomic Distributed Object Store) object storage cluster in the distributed storage system (ceph).

At present, the rbd of the kernel is used to access the ceph storage. When the ceph is relatively new and the kernel is relatively old, there may be some images (image is the label for storing pictures, in the form of <image src="the relative or absolute path of the picture" />) If the kernel does not support it, it cannot map (a collection class of java). Distributed storage needs to implement all functions of librbd (librbd is the block storage interface provided by ceph) in the kernel (kernel), which cannot meet user needs.

To sum up, how to effectively improve rbd performance and other issues is a technical issue urgently needed to be solved by those skilled in the art.

Contents of the invention

The object of the present invention is to provide an access request processing method, device, device and readable storage medium, so as to improve the performance of rbd.

In order to solve the above technical problems, the present invention provides the following technical solutions:

A method for processing an access request, comprising:

Obtain access requests to /dev/nbdx;

Utilize the nbdkernel thread in the nbdkernel module to send the access request to the reader process through nsock;

The reader process calls librbd to send the access request to the distributed file system, so that the distributed file system processes the access request;

The librbd process is used to receive the response returned by the distributed file system, and the writer process is used to send the response to nbdclient through dsock, so that the nbdclient processes the response.

Preferably, before obtaining the access request to /dev/nbdx, it also includes:

Configure the kvm virtualization environment, and use virt-manager or virt-install to install a virtual machine;

Write the default rbd-nbd driver and start the libvirtd daemon;

Modify the xml file of the virtual machine, and find the block device mapped by the distributed file system in the added device project.

Preferably, the modification of the xml file of the virtual machine includes:

Use the libvirt api interface to write the storage mode of rbd into the virtual xml file and make it take effect;

Write the default rbd-nbd configuration mode into the xml file.

Preferably, after installing the virtual machine, before modifying the xml file of the virtual machine, it also includes:

Use the virsh command line to dump the xml file of the virtual machine.

Preferably, before obtaining the access request to /dev/nbdx, further include:

When nbdclient starts, create a socketpair; wherein, the socketpair includes nsock, rsock;

Use the NBD_SET_SOCK ioctl interface to pass the nsock to the nbdkernel module;

Use the nbdkernel module to create nbd kernel thread;

Use the nbdclient to create nbdserver;

The nbd server creates a reader and a writer; wherein, the reader monitors the rsock, and the nbdclient invokes the NBD_DO_IT ioct l interface to enter the kernel to monitor the nsock.

An access request processing device, comprising:

An access request acquisition module, configured to acquire an access request to /dev/nbdx;

The first transfer module is used to send the access request to the reader process through nsock by using the nbdkernel thread in the nbdkernel module;

The second transfer module is used for the reader process to call librbd to send the access request to the distributed file system, so that the distributed file system processes the access request;

The processing module is configured to use the librbd process to receive the response returned by the distributed file system, and use the writer process to send the response to nbdclient through dsock, so that the nbdclient can process the response.

Preferably, it also includes:

The configuration module is used to configure the kvm virtualization environment before obtaining the access request to /dev/nbdx, and install a virtual machine using virt-manager or virt-install; write the preset rbd-nbd driver, start The libvirtd daemon process; modify the xml file of the virtual machine to find the block device mapped by the distributed file system using the added device item.

Preferably, the configuration module is specifically configured to use the libvirt api interface to write the storage mode of rbd into a virtual xml file, and make it take effect; write the default rbd-nbd configuration mode into the xml file.

An access request processing device, comprising:

memory for storing computer programs;

A processor configured to implement the steps of the access request processing method when executing the computer program.

A readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of the above access request processing method are implemented.

Apply the method provided by the embodiment of the present invention to obtain an access request to /dev/nbdx; utilize the nbdkernel thread in the nbdkernelmodule to send the access request to the reader process through nsock; the reader process calls librbd to send the access request to the distributed file system, In order for the distributed file system to process access requests; use the librbd process to receive the response returned by the distributed file system, and use the writer process to send the response to nbdclient through dsock, so that nbdclient can process the response.

When an access request to /dev/nbdx is obtained, the access request is first sent to the distributed file system, and the response returned by the distributed file system is sent to nbdclient. That is to say, combining the rbd tool with nbd to process the access requests of the distributed storage system can realize functions that cannot be realized by the original rbd tool with nbd. It is equivalent to mapping the RBD image to an NBD (NetworkBlock Device, network block device) device, so that the RBD can be used as a regular local block device. Combined with the block driver framework, the distributed system in the kernel can realize all the functions in librdb, improve the performance of rbd, enhance the stability of rbd, and improve the user experience.

Correspondingly, the embodiments of the present invention also provide an access request processing device, device, and readable storage medium corresponding to the above access request processing method, which have the above technical effects, and will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is an implementation flowchart of an access request processing method in an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of an access request processing device in an embodiment of the present invention;

Fig. 3 is a schematic structural diagram of an access request processing device in an embodiment of the present invention.

Detailed ways

The core of the present invention is to provide a kind of access request processing method, this method combines rbd-nbd, finally realizes that RBD image is mapped to NBD (Network Block Device, network block device) device, is about to use RBD as a conventional local block device up. Specifically, rbd uses the rbd module of the kernel to access the distributed storage system (ceph) storage. When ceph is newer and the kernel is older, there may be some image feature kernels that do not support it, so they cannot map; nbd uses librbd to access ceph storage , basically there will be no situation where the feature is not supported. Among them, nbd exporting a blockdevice on the host is a relatively mature and traditional mode. All access requests to the device are forwarded to user space programs for processing. nbd provides a block device driver framework that supports the implementation of block devices in user space. Similar to FUSE support for implementing file systems in user space. That is to say, by combining rbd and nbd, all the functions in librdb can be realized in the distributed storage system in the kernel (kernel), which improves the performance of rbd, enhances the stability of rbd, and improves the user experience.

Another core of the present invention is an access request processing device, device and readable storage medium corresponding to the above access request processing method, which has the above technical effect.

In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Embodiment one:

Please refer to FIG. 1. FIG. 1 is a flowchart of a method for processing an access request in an embodiment of the present invention. The method includes the following steps:

S101. Obtain an access request to /dev/nbdx.

In the embodiment of the present invention, a virtual machine can be set, and the IO request of the virtual machine is an access request to /dev/nbdx.

Wherein, dev in /dev/nbdx is a device file system, nbd is an NBD (NetworkBlock Device, network block device) device in the device file system, and x is an NBD device label. For example, x can be any value from 0 to 32. You can use the pre-set monitoring process/monitor for /dev/nbdx to monitor /dev/nbdx to obtain NBD access requests.

S102. Utilize the nbdkernel thread in the nbdkernel module to send the access request to the reader process through nsock.

Among them, nbdkernel module is the nbd kernel, and nbdkernel thread is the nbd process.

In the embodiment of the present invention, the socket nsock can be created in advance, and nsock is a socket for communication between the nbdkernel thread in the nbdkernel module and the reader process. In actual implementation, in order to facilitate the operation, you can use the nbdkernel module to put the access request into queue 1 first, then wake up the nbdkernel thread, and the nbdkernel thread sends the access request to the reader process through nsock.

S103. The reader process invokes librbd to send the access request to the distributed file system, so that the distributed file system can process the access request.

Among them, librbd is the block storage interface provided by ceph. The librbd interface is specifically c/c++ two types of interfaces. The header files are respectively in ceph-master\ceph-master\src\include\rbd\librbd.hpp and source\ceph -master\ceph-master\src\include\rbd\librbd.h. Use this interface to perform operations such as image creation, deletion, clone, and read and write in ceph. That is to say, through the librbd interface, the access request can be sent to ceph, and the access request is processed in ceph. Specifically, because librbd provides a namspace externally. In this way, using librbd can use the functions provided by linrbd. In libedb, two classes are provided, namely rbd and image, and the corresponding functions can be realized in ceph-master\ceph-master\src\librbd\librbd.cc.

In actual application, for the convenience of operation, the Reader process can put the access request into queue 2, and call librbd to send the request to ceph. After Ceph receives the access request, it processes the access request and returns the response to librbd.

S104. Use the librbd process to receive the response returned by the distributed file system, and use the writer process to send the response to nbdclient through dsock, so that nbdclient can process the response.

In the embodiment of the present invention, a writer process can be created in advance, and the process is connected to nbdclient through a dsock socket.

After receiving the response corresponding to the access request, the writer process can be used to send the response to nbdclient through dsock. So that nbdclient can deal with the response accordingly. Specifically, the response may be an operation request sent to the nbdclient, that is to say, the nbdclient needs to be used to finally complete the processing of the access request.

In practical applications, for the convenience of operation, the response can be placed in queue three, and the writer process can be woken up. The writer process sends the response to nbdclient (user space, nbd client) through dsock, and nbdclient processes the response. After nbdclient finishes processing, the final processing result can be returned to rbd, so as to provide information feedback for the access request, such as outputting prompt information.

Apply the method provided by the embodiment of the present invention to obtain an access request to /dev/nbdx; utilize the nbdkernel thread in the nbdkernelmodule to send the access request to the reader process through nsock; the reader process calls librbd to send the access request to the distributed file system, In order for the distributed file system to process access requests; use the librbd process to receive the response returned by the distributed file system, and use the writer process to send the response to nbdclient through dsock, so that nbdclient can process the response.

When an access request to /dev/nbdx is obtained, the access request is first sent to the distributed file system, and the response returned by the distributed file system is sent to nbdclient. That is to say, combining the rbd tool with nbd to process the access requests of the distributed storage system can realize functions that cannot be realized by the original rbd tool with nbd. It is equivalent to mapping the RBD image to an NBD (Network Block Device, network block device) device, so that the RBD can be used as a regular local block device. Combined with the block driver framework, the distributed system in the kernel can realize all the functions in librdb, improve the performance of rbd, enhance the stability of rbd, and improve the user experience.

It should be noted that, based on the first embodiment above, the embodiment of the present invention also provides a corresponding improvement solution. In subsequent embodiments, the same steps as in the first embodiment above or corresponding steps may be referred to each other, and the corresponding beneficial effects may also be referred to each other, and will not be repeated in the improved embodiments below.

Embodiment two:

In order for those skilled in the art to understand the technical solution provided by the embodiment of the present invention, the technical solution provided by the embodiment of the present invention will be described in detail below in conjunction with performing environment construction and configuration before obtaining an access request. The environment construction and configuration process includes:

Step 1. Configure the kvm virtualization environment, and use virt-manager or virt-install to install a virtual machine;

Step 2, write the default rbd-nbd driver, and start the libvirtd daemon process;

Step 3: Modify the xml file of the virtual machine, and find the block device mapped by the distributed file system in the added device project.

For ease of description, the above three steps will be described in combination below.

First, configure the kvm virtualization environment, and install a virtual machine through virt-manager or virt-install. For how to configure the kvm virtualization environment and how to install the virtual machine, please refer to the common processing flow, and will not repeat them here. Preferably, after the virtual machine is installed, before the xml file of the virtual machine is modified, the xml file of the virtual machine can also be dumped using the virsh command line. That is to say, backup and save the xml file, so that when the xml file is improperly modified, it can be restored to the original state.

The steps of the write access request processing method provided by the first embodiment above can be written into the rbd-nbd driver, then the rbd-nbd driver is written into the virtual, and then the libvirtd daemon process is started.

Then, modify the xml file so as to realize the access request processing method provided by the embodiment of the present invention. When modifying the xml file of the virtual machine, the rbd storage mode can be written into the virtual xml file. Specifically, use the libvirt api interface to write the rbd storage mode into the virtual xml file and make it take effect; write the default rbd-nbd configuration mode into the xml file. The storage mode of rbd, specifically, on linux, the disk required by the upper layer virtual machine application, that is, the block device mapped by rbd is operated through the kernel layer of the host to obtain the block device in the osd of the distributed file system. By virtualizing the block-driven device of qemu-kvm, the request is passed to qmeu-rbd to be converted into the IO queue of the host.

The modified xml file includes rbd storage mode and rbd-nbd configuration mode. For example, the xml file of the modified virtual machine can be:

Specifically, when modifying the xml file of the virtual machine, in order to reduce the modification workload, you can find the block device mapped by the distributed file system using the added device project. That is to say, it is necessary to map the RBD image as an NBD (Network Block Device, network block device) device to be found first, so that the found RBD image can be used as a conventional local block device in the application. In practical applications, after adding the rbd-nbd processing module, the upper-layer virtual machine io request can be processed by the nbd module, and then passed to librbd, and then fed back to the distributed file system storage. In other words, the functions in librbd that rbd cannot implement can be implemented by using nbd.

Preferably, in order to better realize the technical solutions provided by the embodiments of the present invention, after the virtual environment is set up and the xml file is configured, each process required to realize the technical solutions provided by the embodiments of the present invention can be carried out Create, and create the corresponding socket to establish a communication connection for the process. In order to process the request after receiving the access request. Make rbd realize more functions. Specifically, before obtaining access requests to /dev/nbdx, execute:

Step 1. When nbdclient starts, create a socket pair; where, the socket pair includes nsock and rsock;

Step 2. Use the NBD_SET_SOCK ioctl interface to pass nsock to the nbdkernel module;

Step 3: Use nbdkernel module to create nbd kernel thread;

Step 4: Use nbdclient to create nbdserver;

Step 5: nbd server creates reader and writer; among them, reader listens to rsock, and nbdclient calls NBD_DO_IT ioct l interface to enter kernel to listen to nsock.

For the convenience of description, the above five steps are combined for description below.

Implement an rbd-nbd program between user spaces, that is, nbdclient, create a socketpair when starting, and the two sockets are nsock and rsock respectively. Pass nsock to nbdkernelmodule using NBD_SET_SOCK ioctl interface, nbdkernel module creates nbd kernel thread in queue one, and nbdclient creates nbdserver. Nbd server creates two threads: reader and writer. Among them, the reader listens to rsock, the writer is in queue 2, and nbdclient calls the NBD_DO_IT ioct l interface to enter the kernel to listen to nsock.

Corresponding to the above method embodiments, an embodiment of the present invention further provides an access request processing device, and the access request processing device described below and the access request processing method described above may refer to each other correspondingly.

Referring to shown in Figure 2, the device includes the following modules:

An access request obtaining module 101, configured to obtain an access request to /dev/nbdx;

The first transfer module 102 is used for utilizing the nbdkernel thread in the nbdkernel module to send the access request to the reader process through nsock;

The second transfer module 103 is used for the reader process to call librbd to send the access request to the distributed file system, so that the distributed file system processes the access request;

The processing module 104 is configured to use the librbd process to receive the response returned by the distributed file system, and use the writer process to send the response to nbdclient through dsock, so that nbdclient can process the response.

Among them, the first transfer module and the first transfer module are only used to distinguish the transmission of access requests between different processes and/or sockets, and do not indicate that in actual applications, there must be a first transfer module and a second Two transfer modules. That is, in practical applications, the actions performed by the above-mentioned first transfer module and the second transfer module may be exchanged, or implemented by only one module.

Apply the device provided by the embodiment of the present invention to obtain an access request to /dev/nbdx; use the nbdkernel thread in the nbdkernelmodule to send the access request to the reader process through nsock; the reader process calls librbd to send the access request to the distributed file system, In order for the distributed file system to process access requests; use the librbd process to receive the response returned by the distributed file system, and use the writer process to send the response to nbdclient through dsock, so that nbdclient can process the response.

When an access request to /dev/nbdx is obtained, the access request is first sent to the distributed file system, and the response returned by the distributed file system is sent to nbdclient. That is to say, combining the rbd tool with nbd to process the access requests of the distributed storage system can realize functions that cannot be realized by the original rbd tool with nbd. It is equivalent to mapping the RBD image to an NBD (NetworkBlock Device, network block device) device, so that the RBD can be used as a regular local block device. Combined with the block driver framework, the distributed system in the kernel can realize all the functions in librdb, improve the performance of rbd, enhance the stability of rbd, and improve the user experience.

In a specific embodiment of the present invention, it also includes:

The configuration module is used to configure the kvm virtualization environment before obtaining access requests to /dev/nbdx, and install a virtual machine using virt-manager or virt-install; write the default rbd-nbd driver, and start the libvirtd daemon Process; modify the xml file of the virtual machine, and find the block device mapped by the distributed file system using the added device project.

In a specific embodiment of the present invention, the configuration module is specifically used to write the storage mode of rbd into a virtual xml file using the libvirt api interface, and make it take effect; write the default rbd-nbd configuration mode into the xml file .

In a specific embodiment of the present invention, it also includes:

The backup module is used to dump the xml file of the virtual machine using the virsh command line after installing the virtual machine and before modifying the xml file of the virtual machine.

In a specific embodiment of the present invention, it also includes:

The process setting module is used to perform the following steps before obtaining the access request to /dev/nbdx:

When nbdclient starts, create a socketpair; among them, socketpair includes nsock, rsock;

Use the NBD_SET_SOCK ioctl interface to pass nsock to the nbdkernel module;

Use nbdkernel module to create nbd kernel thread;

Use nbdclient to create nbdserver;

nbd server creates reader and writer; among them, reader listens to rsock, and nbdclient calls NBD_DO_IT ioct l interface to enter kernel to listen to nsock.

Corresponding to the above method embodiments, an embodiment of the present invention also provides an access request processing device, and the access request processing device described below and the access request processing method described above may refer to each other correspondingly.

Referring to Fig. 3, the access request processing equipment includes:

memory D1 for storing computer programs;

The processor D2 is configured to implement the steps of the access request processing method in the foregoing method embodiment when executing the computer program.

Corresponding to the above method embodiment, the embodiment of the present invention further provides a readable storage medium, and a readable storage medium described below and a method for processing an access request described above may refer to each other correspondingly.

A readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of the access request processing method in the foregoing method embodiments are implemented.

Specifically, the readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like that can store program codes. readable storage media.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

Professionals can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the possible For interchangeability, in the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

In this paper, specific examples are used to illustrate the principles and implementation methods of the present invention, and the descriptions of the above embodiments are only used to help understand the technical solutions and core ideas of the present invention. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (10)

1. a kind of access request processing method characterized by comprising

Acquisition pair/dev/nbdx access request；

The access request is sent to by nsock using the nbdkernel thread in nbdkernel module Reader process；

The reader process calls librbd that the access request is issued distributed file system, so as to the distributed text The part system processing access request；

The response that the distributed file system returns is received using the librbd process, and will be described using writer process Response is sent to nbdclient by dsock, so as to the nbdclient processing response.

2. access request processing method according to claim 1, which is characterized in that the acquisition pair/dev/nbdx's Before access request, further includes:

Kvm virtualized environment is configured, and one empty machine is installed using virt-manager or virt-install；

Default rbd-nbd driver is written, starts libvirtd finger daemon；

The xml document for modifying the empty machine, is found addition device project and is set using the block that distributed file system maps out It is standby.

3. access request processing method according to claim 2, which is characterized in that the xml text of the modification empty machine Part, comprising:

Memory module using libvirt api interface write-in rbd enters virtual xml document, and it is made to come into force；

Default rbd-nbd configuration mode is written and enters xml document.

4. access request processing method according to claim 2, which is characterized in that after installing empty machine, modifying Before the xml document of the void machine, further includes:

In the xml document using virsh order line dump void machine.

5. access request processing method according to any one of claims 1 to 4, which is characterized in that in acquisition pair/dev/ Before the access request of nbdx, further includes:

In nbdclient starting, a socketpair is created；Wherein, the socketpair includes nsock, rsock；

The nsock is passed into nbdkernel module using NBD_SET_SOCK ioctl interface；

Nbd kernel thread is created using the nbdkernel module；

Nbdserver is created using the nbdclient；

The nbd server creates reader and writer；Wherein, the reader monitors the rsock, described Nbdclient calls NBD_DO_IT ioctl interface to enter kernel and monitors the nsock.

6. a kind of access request processing unit characterized by comprising

Access request obtains module, for obtaining pair/the access request of dev/nbdx；

First shift module, for being led to the access request using the nbdkernel thread in nbdkernel module It crosses nsock and is sent to reader process；

Second shift module calls librbd that the access request is issued distributed field system for the reader process System, so as to the distributed file system processing access request；

Processing module for receiving the response that the distributed file system returns using the librbd process, and utilizes The response is sent to nbdclient by dsock by writer process, so as to the nbdclient processing response.

7. access request processing unit according to claim 6, which is characterized in that further include:

Configuration module, for configuring kvm virtualized environment, and utilize before the access request of the acquisition pair/dev/nbdx Virt-manager or virt-install installs an empty machine；Default rbd-nbd driver is written, starts libvirtd Finger daemon；The xml document for modifying the empty machine, finds what addition device project was mapped out using distributed file system Block device.

8. access request processing unit according to claim 7, which is characterized in that the configuration module is specifically used for benefit Enter virtual xml document with the memory module of libvirt api interface write-in rbd, and it is made to come into force；Default rbd- is written Nbd configuration mode enters xml document.

9. a kind of access request processing equipment characterized by comprising

Memory, for storing computer program；

Processor realizes the access request processing side as described in any one of claim 1 to 5 when for executing the computer program The step of method.

10. a kind of readable storage medium storing program for executing, which is characterized in that be stored with computer program, the meter on the readable storage medium storing program for executing It is realized when calculation machine program is executed by processor as described in any one of claim 1 to 5 the step of access request processing method.