CN111522514A - Cluster file system, data processing method, computer equipment and storage medium - Google Patents

Abstract

The present disclosure provides a cluster file system, a data processing method, a computer device, and a storage medium, wherein the cluster file system includes: an object storage server, a plurality of object storage devices, and a plurality of first virtual block devices; the object storage server is connected with a plurality of object storage devices; each object storage device is connected with at least one first virtual block device; the object storage server is used for receiving a first input/output (IO) instruction sent by the client and accessing the first virtual block device through the object storage device based on the first IO instruction; target data is stored in the first virtual block device; wherein the target data is stored as at least three copies of data in the first virtual block device.

Description

Cluster file system, data processing method, computer equipment and storage medium

technical field

The present disclosure relates to the technical field of computer storage, and in particular, to a cluster file system, a data processing method, a computer device, and a storage medium.

Background technique

A cluster file system refers to a file system that runs on multiple computers and communicates with each other in a certain way to integrate and virtualize all storage space resources in the cluster and provide file access services to the outside world.

Data reliability is an important indicator to measure the stability of a cluster file system; as a cluster file system, Lustre usually adopts a dynamic file system (Zettabyte File System, ZFS) and hardware redundancy to ensure data in order to prevent data loss. reliability. Among them, ZFS is a kernel file system, which combines multiple disks into a floppy disk array (Redundant Arrays of Independent Disks, RAID) to provide data storage disks for the Lustre cluster.

This data storage method has the problem of poor scalability.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide at least a cluster file system, a data processing method, a computer device, and a storage medium.

In a first aspect, an embodiment of the present disclosure provides a cluster file system, including: an object storage server, a plurality of object storage devices, and a plurality of first virtual block devices; wherein the object storage server and a plurality of the objects storage device connection; each object storage device is connected to at least one of the first virtual block devices; the object storage server is configured to receive the first input and output IO instruction sent by the client, and based on the first IO instruction , access the first virtual block device through the object storage device; target data is stored in the first virtual block device; wherein, the target data is stored in the first virtual block device as at least three pieces of data copy.

In this way, the first virtual block device can aggregate multiple physical disks together to form a large-capacity virtual device, thus having stronger scalability. Meanwhile, the target data can be stored as at least three copies in the first virtual block device, which can greatly improve the reliability of the data.

In a possible implementation manner, it further includes: a metadata server, multiple metadata storage devices, and multiple second virtual block devices; wherein the metadata server is connected to multiple metadata storage devices; each Each of the metadata storage devices is connected to at least one of the second virtual block devices; the metadata server is configured to receive a second IO instruction sent by the client, and based on the second IO instruction, The metadata storage device accesses the second virtual block device; the second virtual block device stores metadata corresponding to the target data; wherein the metadata is stored in the second virtual block device as At least three copies of metadata.

In this way, since the second virtual block device can aggregate multiple physical disk Disks together to form a large-capacity virtual device, and the number of Disks contained in the second virtual block device is larger, it has a stronger capacity. Extensibility.

At the same time, the storage capacity of the second virtual block device is larger, because the metadata can be stored as at least three copies in the first virtual block device, which can greatly improve the reliability of the data.

In a possible implementation manner, the target data includes: data blocks aggregated from multiple files; and the metadata corresponding to the data blocks includes location information of each of the files in the data blocks.

In this way, since the target data includes data blocks aggregated by multiple files; for the metadata server, the size of the file directory that needs to be maintained will be significantly reduced, and the cluster file system provided by the embodiments of the present disclosure can be applied to a large number of small file application.

In a possible implementation manner, the first virtual block device is further configured to, according to a preset first reading sequence, determine from at least three data copies corresponding to the first IO instruction and send the data to the The object storage device returns a copy of the target data.

In this way, the smooth reading of the target data is ensured.

In a second aspect, an embodiment of the present disclosure provides a data processing method, including: receiving an input/output IO instruction sent by a client; and based on the IO instruction, controlling a storage device to write data corresponding to the IO instruction into the virtual machine The data corresponding to the IO instruction is read from the block device or from the virtual block device; wherein, the data is stored as at least three data copies in the virtual block device.

In a possible implementation manner, the storage device includes: an object storage device; the virtual block device includes: a first virtual block device; the IO instruction includes: a first IO instruction; the data includes target data; Described, based on the IO instruction, controlling the storage device to write the data corresponding to the IO instruction into the virtual block device, or to read the data corresponding to the IO instruction from the virtual block device, including: based on The first IO instruction controls the object storage device to write the target data corresponding to the first IO instruction into the first virtual block device, or read the target data corresponding to the first virtual block device from the first virtual block device. Target data corresponding to an IO instruction.

In a possible implementation manner, the storage device includes: a metadata storage device; the virtual block device includes: a second virtual block device; the IO instruction includes: a second IO instruction; the data includes metadata; The controlling the storage device to write data corresponding to the IO instruction into the virtual block device based on the IO instruction, or to read data corresponding to the IO instruction from the virtual block device, including: Based on the second IO instruction, the metadata storage device is controlled to write the metadata data corresponding to the second IO instruction into the second virtual block device, or to read the metadata from the second virtual block device. metadata corresponding to the second IO instruction.

In a possible implementation manner, the reading data corresponding to the IO instruction from the virtual block device includes: sending the IO instruction to the virtual block device, so that the virtual block device According to a preset reading order, the target data copy is determined from at least three data copies corresponding to the IO instruction and returned to the object storage device.

In a third aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor and a memory connected to each other, the memory stores machine-readable instructions executable by the processor, and when the computer device runs, the Machine-readable instructions are executed by the processor to implement the second aspect above, or the steps of the data processing method in any possible implementation manner of the second aspect.

In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the second aspect, or any one of the second aspect. Steps of a data processing method in one possible implementation.

In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings required in the embodiments, which are incorporated into the specification and constitute a part of the specification. The drawings illustrate embodiments consistent with the present disclosure, and together with the description serve to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. Other related figures are obtained from these figures.

FIG. 1 shows a schematic diagram of a cluster file system provided by an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of a data reading process in a cluster file system provided by an embodiment of the present disclosure;

3 shows a schematic diagram of a process of OST reading target data from VBD in a cluster file system provided by an embodiment of the present disclosure;

FIG. 4 shows a flowchart of a data processing method provided by an embodiment of the present disclosure;

FIG. 5 shows a schematic diagram of a computer device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only These are some, but not all, embodiments of the present disclosure. The components of the disclosed embodiments generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure as claimed, but is merely representative of selected embodiments of the disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present disclosure.

After research, it is found that in the cluster file system Lustre, disk arrays are usually used to store data. The number of Disks that can be installed in an object storage device is limited. To expand the storage capacity of Lustre, the number of object storage devices must be increased accordingly; however, the cost of physical devices is high, which leads to poor scalability of Lustre.

At the same time, due to the limited number of Disks that can be installed on object storage devices, currently Lustre can only provide data redundancy at the highest level of two copies, resulting in poor data reliability.

In addition, the cluster and parallel architecture of Lustre is more suitable for many clients to send and write files concurrently; but it is not suitable for small file applications; especially for massive small file applications, both the metadata server and the object storage server need to maintain huge the file directory, causing Lustre's response speed to drop.

Based on the above research, the present disclosure provides a cluster file system and a data processing method, including an object storage server, a plurality of object storage devices, and a plurality of first virtual block devices; wherein the object storage server and a plurality of all The object storage device is connected; each of the object storage devices is connected to at least one of the first virtual block devices; the object storage server is configured to receive a first input/output (Input/Output, IO) instruction sent by the client, And based on the first IO instruction, the first virtual block device is accessed through the object storage device; the first virtual block device stores target data; the first virtual block device (Virtual Block Disk, VBD) can Disk multiple physical disks. Aggregated together to form a large-capacity virtual device, which is more scalable. Meanwhile, the target data can be stored as at least three copies in the first virtual block device, which can greatly improve the reliability of the data.

The defects existing in the above solutions are all the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions to the above problems proposed by the present disclosure hereinafter should be the inventors Contributions made to this disclosure during the course of this disclosure.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

In order to facilitate understanding of this embodiment, a cluster file system disclosed by an embodiment of the present disclosure is first introduced in detail.

The Luster cluster includes: a metadata server (Meta Data Server, MDS) and an object storage server (Object Storage Server, OSS); MDS is connected to multiple metadata storage devices (Meta Data Target, MDT), and OSS is connected to multiple Object Storage Target (OST). Each MDT and each OST are connected to a physical storage (Disk), which provides data storage services for the MDT and SOT through the Disk.

Among them, MDS is responsible for providing metadata services for the cluster and managing the namespace of the entire cluster; one MDT is shared among multiple MDSs; each MDT saves file metadata objects, such as file name, directory structure and access rights; customers The client (Client) reads the metadata stored on the MDT through the MDS.

OSS is responsible for the interaction between Client and Disk and data storage, and provides data input/output (I/O) interfaces for Client; each OSS manages one or more OSTs, and each OST is used for storage file data object.

Referring to FIG. 1, which is a schematic structural diagram of a cluster file system according to an embodiment of the present disclosure, the cluster file system includes:

an object storage server 11, a plurality of object storage devices 12, and a plurality of first virtual block devices 13;

Wherein, the object storage server 11 is connected to a plurality of the object storage devices 12; each of the object storage devices 12 is connected to at least one of the first virtual block devices 13;

The object storage server 11 is configured to receive the first input/output IO instruction sent by the client, and access the first virtual block device 13 through the object storage device 12 based on the first IO instruction;

The first virtual block device 13 stores target data;

Wherein, the target data is stored as at least three data copies in the first virtual block device.

In a specific implementation, after receiving the first IO instruction sent by the client, the object storage server (Object Storage Server, OSS) 11, in the case that the first IO instruction is a data writing instruction, the first IO instruction also carries There is target data to be written corresponding to the first IO instruction. The object storage server 11 determines a target object storage device from a plurality of object storage devices (Object Storage Target, OST) 12 according to the first IO instruction, and transmits the first IO instruction to the target object storage device; the target object The storage device is connected with a first virtual block device (Virtual BlockDisk, VBD) 13, and the target object storage device transfers the target data carried in the first IO instruction to the first virtual block device 13 connected to it; the first virtual block device 13 will The target data is stored as at least three copies of the data.

In the case where the first IO instruction is a read data instruction, the first IO instruction carries the metadata of the target data to be read; the object storage server 11, according to the first IO instruction, from the plurality of object storage devices 12, Determine the target object storage device that stores the target data corresponding to the metadata, and then pass the first IO instruction to the target object storage device; the target object storage device, according to the first IO instruction, from the first virtual block connected to it The device 13 reads the target data corresponding to the metadata carried in the first IO instruction; here, when the first virtual block device 13 returns the target data to the target object storage device, according to the preset first reading order, Read the target data copy from the saved at least three data copies; specifically, if the first data copy in the reading order cannot be found, the corresponding second data copy is read in the reading order; If the data copy cannot be found, the corresponding third data copy is read according to the reading order. After reading the target data from the first virtual block device connected to it, the target object storage device returns the target data to the object storage server 11, and the object storage server returns the target data to the client.

The cluster file system provided by another embodiment of the present disclosure further includes: a metadata server 14, multiple metadata storage devices 15, and multiple second virtual block devices 16;

Wherein, the metadata server 14 is connected to a plurality of the metadata storage devices 15; each of the metadata storage devices 15 is connected to at least one of the second virtual block devices 16;

The metadata server 14 is configured to receive the second IO instruction sent by the client, and based on the second IO instruction, access the second virtual block device 16 through the metadata storage device 15;

The second virtual block device 16 stores metadata corresponding to the target data;

Wherein, the metadata is stored in the second virtual block device 16 as at least three copies of the metadata.

In a specific implementation, after the metadata server (Meta Data Server, MDS) 14 receives the second IO instruction sent by the client, if the second IO instruction is a data writing instruction, in the second IO instruction Also carries the metadata to be written corresponding to the second IO instruction. The metadata server 14 determines a target metadata storage device from a plurality of metadata storage devices (Meta Data Target, MDT) 15 according to the second IO instruction, and transmits the second IO instruction to the determined target metadata storage device The target metadata storage device is connected with the second virtual block device 16, and the target metadata storage device transfers the metadata carried in the second IO instruction to the second virtual block device 16; the second virtual block device 16 stores the metadata as At least three copies of metadata.

In the case where the second IO instruction is a read data instruction, the second IO instruction carries relevant information of the metadata to be read, such as a file name, etc.; A target metadata storage device is determined among the connected multiple metadata storage devices 15, and the second IO instruction is transmitted to the target metadata storage device; The metadata related to the carried metadata is read from the second virtual block device 16 connected to the metadata corresponding to the second IO instruction. Here, when returning the metadata to the target metadata storage device, the second virtual block device 16 reads the target metadata copy from the at least three saved metadata copies according to the preset second reading order; specifically , if the first metadata copy in the reading order cannot be found, the corresponding second metadata copy is read in the reading order; if the second metadata copy cannot be found, it is read in the reading order The corresponding third metadata copy... until the metadata copy is found; if the metadata copy cannot be found, the read failure information is returned to the target metadata storage device. After reading the metadata from the second virtual block device connected to it, the target metadata storage device returns the metadata to the metadata server 14; the metadata server returns the metadata to the client, so that the client can According to the obtained metadata, a first IO instruction is initiated to the object storage server to read target data corresponding to the obtained metadata.

In the above process, each time the client initiates a data reading process, it first needs to access the metadata server to obtain the metadata of the file to be read, and then access the object storage server to read the file according to the metadata of the file to be read. target data.

Similarly, every time the client initiates a data writing process, it first needs to access the metadata server. After the metadata server generates metadata, it then accesses the object storage server to store the target data corresponding to the metadata to the object storage device. in the first virtual block device.

In a possible implementation manner, the metadata includes: the file name, directory structure and access rights of the target data, and the like.

In another possible implementation manner, the target data includes: data blocks aggregated from multiple files;

The metadata corresponding to the data block includes location information of each of the files in the data block.

In this way, multiple files can be aggregated into data blocks; when the client accesses a certain file in the data block, the client can determine the storage location of the file to be accessed in the second virtual block device based on the location information.

As shown in FIG. 2 , an embodiment of the present disclosure provides a complete process for reading data, including:

The client Client requests metadata from the metadata server MDS; the MDS initiates data reading from the specific metadata storage device MDT based on the client's request; the MDT reads the metadata from the second virtual block device VBD2 and returns the metadata to MDT; MDT returns metadata to MDS; MDS returns metadata to Client.

The client finds the corresponding object storage server OSS according to the obtained metadata information to obtain the target data; the OSS transfers the IO request to the specific object storage device OST; the OST obtains the target data from the first virtual block device VBD1 connected to it, and Return the acquired target data to the OST; the OST returns the target data to the OSS; the OSS returns the target data to the Client.

Through the above process, one data read is completed.

Referring to FIG. 3 , an embodiment of the present disclosure further provides a specific data processing process in which the first virtual block device 13 provides a data storage service for an OST, wherein the first virtual block device 13 virtualizes at least one physical disk Disk Then, the obtained virtual storage device; and the object storage device 12 are deployed with a virtual block device (Virtual Block Disk, VBD) service; in addition, the VBD service can also be independently set in another specific device, called a VBD server.

The OST requests the secondary metadata from the VBD service, and the VBD service obtains the specific secondary metadata from the manager node Monitor in the VBD cluster based on the request.

Here, the metadata includes the file name of the target data, the directory structure and access rights in the virtual block device, etc.

The secondary metadata includes the file name of the target data, the directory structure and access rights of the target data in at least one physical disk constituting the virtual block device, and the like.

Monitor returns secondary metadata to VBD service; VBD service returns secondary metadata to OSD; OSD obtains target data corresponding to secondary metadata from specific physical disk Disk according to secondary metadata copy.

Here, it should be noted that different target data copies are, for example, deployed in different Disks that constitute the first virtual block device; in this example, the Disks that constitute the first virtual fast device include: Disk1, Disk2 and Disk3; and each Each Disk stores a target data copy of the target data; the preset first reading order is: Disk1→Disk2→Disk3; when the OSD cannot read the target data copy from Disk1 according to the preset first reading order Then, read the target data copy from Disk2; after the target data copy cannot be read from Disk2, read the target data copy from Disk3.

It is extremely unlikely that the three Disks that constitute the virtual block device will lose the target data copy at the same time, thereby ensuring the reliability of the data.

In addition, when data is written, the OST will also send a write request to the VBD service; the VBD service will send the write request to the Monitor to generate secondary metadata; and the VBD service will write the corresponding target data to the different Disks that constitute the VBD. , to store as different target data copies in multiple Disks.

The cluster file system provided by the embodiments of the present disclosure includes an object storage server, multiple object storage devices, and multiple first virtual block devices; wherein, the object storage server is connected to the multiple object storage devices; The object storage device is connected to at least one of the first virtual block devices; the object storage server is configured to receive a first input/output (IO) instruction sent by the client, and based on the first IO instruction, through the The object storage device accesses the first virtual block device; the first virtual block device stores target data; the first virtual block device can aggregate a plurality of physical disk Disks together to form a large-capacity virtual device, so it has stronger of scalability.

At the same time, the target data can be stored as at least three copies in the first virtual block device, which can greatly improve the reliability of the data

In addition, the target data of the embodiments of the present disclosure includes data blocks that are aggregated by multiple files; for the metadata server, the scale of the file directory that needs to be maintained will be significantly reduced, and the cluster file system provided by the embodiments of the present disclosure can Applicable to massive small file applications.

Referring to FIG. 4 , an embodiment of the present disclosure further provides a data processing method, including:

S401: Receive an input and output IO instruction sent by the client.

S402: Based on the IO instruction, control the storage device to write data corresponding to the IO instruction into the virtual block device, or read data corresponding to the IO instruction from the virtual block device.

Wherein, the data is stored as at least three data copies in the virtual block device.

In a possible implementation manner, the storage device includes: an object storage device; the virtual block device includes: a first virtual block device; the IO instruction includes: a first IO instruction; the data includes target data;

The controlling the storage device to write data corresponding to the IO instruction into the virtual block device based on the IO instruction, or to read data corresponding to the IO instruction from the virtual block device, including:

Based on the first IO instruction, the object storage device is controlled to write the target data corresponding to the first IO instruction into the first virtual block device, or to read the target data corresponding to the first virtual block device from the first virtual block device Target data corresponding to the first IO instruction.

Here, for the specific data reading method and data writing method of the target data, reference may be made to the above-mentioned embodiment corresponding to FIG. 1 , which will not be repeated here.

In another possible implementation manner, the storage device includes: a metadata storage device; the virtual block device includes: a second virtual block device; the IO instruction includes: a second IO instruction; the data includes metadata data;

The controlling the storage device to write data corresponding to the IO instruction into the virtual block device based on the IO instruction, or to read data corresponding to the IO instruction from the virtual block device, including:

Based on the second IO instruction, the metadata storage device is controlled to write the metadata data corresponding to the second IO instruction into the second virtual block device, or to read the metadata from the second virtual block device. metadata corresponding to the second IO instruction.

Here, for the specific way of reading and writing metadata, reference may be made to the above-mentioned embodiment corresponding to FIG. 1 , which will not be repeated here.

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.

An embodiment of the present disclosure further provides a computer device 10. As shown in FIG. 5, a schematic structural diagram of the computer device 10 provided by an embodiment of the present disclosure includes:

The processor 11 and the memory 12 are interconnected, the memory 12 stores machine-readable instructions executable by the processor, and when the computer device is running, the machine-readable instructions are executed by the processor 11 to achieve the following: Describe the steps: .

Receive input and output IO commands sent by the client;

Based on the IO instruction, controlling the storage device to write data corresponding to the IO instruction into the virtual block device, or to read data corresponding to the IO instruction from the virtual block device;

Wherein, the data is stored as at least three data copies in the virtual block device.

For the specific execution process of the above instruction, reference may be made to the steps of the data processing method described in the embodiments of the present disclosure, and details are not repeated here.

Embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the data processing method described in the foregoing method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

The computer program product of the data processing method provided by the embodiments of the present disclosure includes a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the steps of the data processing methods described in the above method embodiments. , for details, refer to the foregoing method embodiments, which will not be repeated here.

Embodiments of the present disclosure also provide a computer program, which implements any one of the methods in the foregoing embodiments when the computer program is executed by a processor. The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. .

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of software products in essence, or the parts that contribute to the prior art or the parts of the technical solutions. The computer software products are stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure rather than limit them. The protection scope of the present disclosure is not limited thereto, although referring to the foregoing The embodiments describe the present disclosure in detail. Those of ordinary skill in the art should understand that: any person skilled in the art can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present disclosure. Changes can be easily thought of, or equivalent replacements are made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered in the present disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims (10)

1. A cluster file system, comprising: an object storage server, multiple object storage devices, and multiple first virtual block devices; Wherein, the object storage server is connected to a plurality of the object storage devices; each of the object storage devices is connected to at least one of the first virtual block devices; the object storage server, configured to receive the first input/output IO instruction sent by the client, and access the first virtual block device through the object storage device based on the first IO instruction; Target data is stored in the first virtual block device; Wherein, the target data is stored as at least three data copies in the first virtual block device. 2. The cluster file system according to claim 1, further comprising: a metadata server, multiple metadata storage devices, and multiple second virtual block devices; Wherein, the metadata server is connected to a plurality of the metadata storage devices; each of the metadata storage devices is connected to at least one of the second virtual block devices; the metadata server, configured to receive a second IO instruction sent by the client, and access the second virtual block device through the metadata storage device based on the second IO instruction; metadata corresponding to the target data is stored in the second virtual block device; Wherein, the metadata is stored as at least three metadata copies in the second virtual block device. 3. The cluster file system according to claim 2, wherein the target data comprises: a data block aggregated by a plurality of files; The metadata corresponding to the data block includes location information of each of the files in the data block. 4. The cluster file system according to any one of claims 1-3, wherein the first virtual block device is further configured to, according to a preset first reading order, start from and the first IO The target data copy is determined from the at least three data copies corresponding to the instruction and returned to the object storage device. 5. A data processing method, characterized in that, comprising: Receive input and output IO commands sent by the client; Based on the IO instruction, controlling the storage device to write data corresponding to the IO instruction into the virtual block device, or to read data corresponding to the IO instruction from the virtual block device; Wherein, the data is stored as at least three data copies in the virtual block device. 6. The data processing method according to claim 5, wherein the storage device comprises: an object storage device; the virtual block device comprises: a first virtual block device; the IO instruction comprises: a first IO instruction ; the data includes target data; The controlling the storage device to write data corresponding to the IO instruction into the virtual block device based on the IO instruction, or to read data corresponding to the IO instruction from the virtual block device, including: Based on the first IO instruction, the object storage device is controlled to write the target data corresponding to the first IO instruction into the first virtual block device, or to read the target data corresponding to the first virtual block device from the first virtual block device Target data corresponding to the first IO instruction. 7. The data processing method according to claim 5 or 6, wherein the storage device comprises: a metadata storage device; the virtual block device comprises: a second virtual block device; the IO instruction comprises: a first Two IO instructions; the data includes metadata; The controlling the storage device to write data corresponding to the IO instruction into the virtual block device based on the IO instruction, or to read data corresponding to the IO instruction from the virtual block device, including: Based on the second IO instruction, the metadata storage device is controlled to write the metadata data corresponding to the second IO instruction into the second virtual block device, or to read and metadata corresponding to the second IO instruction. 8. The data processing method according to any one of claims 5-8, wherein the reading data corresponding to the IO instruction from the virtual block device comprises: Sending the IO instruction to the virtual block device, so that the virtual block device determines from at least three data copies corresponding to the IO instruction according to a preset reading order and sends the data to the object storage device Returns a copy of the target data. 9. An electronic device, comprising: a processor and a memory connected to each other, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device runs, the machine-readable instructions Executed by the processor to implement the data processing method according to any one of claims 5-8. 10. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the data processing method according to any one of claims 5-8 is executed.

Images