CN110535692B - Fault handling method, device, computer equipment, storage medium and storage system - Google Patents

Abstract

The invention discloses a fault processing method, device, computer equipment, storage medium and storage system of a distributed storage system, belonging to the technical field of fault processing. The method determines the fault state of the distributed storage system according to the faulty storage node of at least one of the plurality of storage nodes, so that it is not necessary to determine the fault state of the distributed storage system only when all the storage nodes are faulty, After the fault status is determined, the fault status can be sent to each storage node in the distributed storage system immediately, so that each storage node can perform fault processing according to the determined fault status, thereby reducing the time taken for the distributed storage system to recover to normal. .

Description

Fault handling method, device, computer equipment, storage medium and storage system

technical field

The present invention relates to the technical field of fault handling, in particular to a method, device, computer equipment, storage medium and distributed storage system for fault handling of a distributed storage system.

Background technique

With the development of big data technology, in order to store more data and prevent data loss, distributed storage systems are more and more favored by enterprises. In the event of a failure, when all storage nodes in the distributed storage system fail, in order to ensure that the failed storage nodes do not affect normal services, the computing nodes that provide services to users can perform fault handling on the failures in the distributed storage system.

The fault handling may be the following process: in the distributed storage system, the client sends a small computer system interface (SCSI) request to multiple storage nodes; when all storage nodes in the distributed storage system appear In the event of a failure, if the storage node has a fault that can be repaired in a short time, the storage node will not respond to the SCSI request. It is determined to be in the all path down (APD) state. The APD state is a fault state of a storage node defined by the VMWare virtual machine. It is used to indicate that all paths of the back-end storage node cannot respond to host requests, and the client is not suspended. The processed SCSI request is waiting for the technician to repair the fault in the distributed storage system; when the storage node has an irreparable fault within a short period of time, the storage node returns a storage exception message to the client, and the client receives the storage exception. After the message, the fault state of the distributed storage system is determined as a permanent device lost (permanent device lost, PDL) state, wherein the PDL state is a fault state of a storage node defined by the VMWare virtual machine, which is used to represent the back-end storage Long-term or permanent node failure. The long-term failure of the storage node will cause the file system in the distributed storage system to be damaged. When the failure state determined by the client is the PDL state, the client powers off the file system and waits for the technician. Fix bugs.

In the above fault handling process, the client will only perform the above fault handling process when all storage nodes in the distributed storage system fail. However, when some storage nodes in the distributed storage system fail, the client will The terminal will not perform the above troubleshooting process. However, it is a relatively common phenomenon that some storage nodes fail in a distributed storage system. Once some storage nodes fail, if the fault diagnosis of the storage nodes in the distributed storage system is not performed, the client cannot determine the distributed storage nodes. Whether the storage nodes in the storage system are faulty, so that the technicians cannot timely learn from the client that the storage nodes are faulty, and they cannot repair the faulty storage nodes immediately, thus prolonging the time required for the distributed storage system to return to normal use. time.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method, device, computer equipment, storage medium and storage system for fault handling of a distributed storage system, which can reduce the time taken for the distributed storage system to recover to normal. The technical solution is as follows:

In a first aspect, a method for handling faults in a distributed storage system is provided, where the distributed storage system includes multiple storage nodes; the method includes:

A fault state of the distributed storage system is determined according to at least one faulty storage node in the plurality of storage nodes; the fault state is used to indicate whether the at least one faulty storage node can be All repaired within the set time;

The failure status is sent to each of the plurality of storage nodes.

Based on the above implementation manner, the fault state of the distributed storage system is determined according to a storage node in which at least one of the plurality of storage nodes is faulty, so that it is not necessary to determine the fault status of the distributed storage system only when all storage nodes are faulty. When the fault status is determined, the fault status can be sent to each storage node in the distributed storage system immediately, so that each storage node can perform fault processing according to the determined fault status, thereby reducing the recovery time of the distributed storage system. time used.

In a possible implementation, the method further includes:

Perform fault processing according to the fault state of the distributed storage system.

In a possible implementation manner, the determining of the fault state of the distributed storage system according to at least one of the multiple storage nodes in which the faulty storage node is faulty includes:

determining at least one storage node that has failed in the distributed storage system and target data that cannot be accessed in the at least one storage node;

The fault state is determined according to the at least one storage node and the target data.

In a possible implementation manner, the determining the fault state according to the at least one storage node and the target data includes:

When the number of the at least one storage node is greater than the redundancy of the distributed storage system, and the data volume of the target data meets a preset condition, the fault state is determined to be a first fault state, and the first fault state is determined. A fault state is used to indicate that the at least one storage node can be completely repaired within the first preset time period.

Based on the above possible implementation manners, in the first fault state, the target device will not power off the file system. Once the at least one storage node can be repaired within the first preset time, the file system can also be prevented from being powered off. , which can reduce the time to repair the file system, so that the distributed storage system can resume business as soon as possible to ensure the quality of service.

In a possible implementation manner, the determining the fault state according to the at least one storage node and the target data includes:

When the number of the at least one storage node is greater than the redundancy of the distributed storage system, and the data volume of the target data meets a preset condition, determine the failure according to the failure scenario of the distributed storage system status, and the failure scenario is used to indicate whether the at least one storage node fails simultaneously.

In a possible implementation, the preset condition includes any of the following:

The ratio between the data volume of the target data and the first preset data volume is greater than the preset ratio, and the first preset data volume is the total data volume of all data stored in the distributed storage system;

The data volume of the target data is greater than the second preset data volume.

In a possible implementation manner, after the fault state is determined according to the at least one storage node and the target data, the method further includes:

When the fault state is the first fault state, if the at least one storage node is not all repaired within the first preset time period, the fault state is updated from the first fault state to the second fault state A failure state, where the second failure state is used to indicate that the at least one storage node cannot be fully repaired within the first preset time period.

In a possible implementation manner, before determining the fault state of the distributed storage system according to the fault scenario of the distributed storage system, the method further includes:

The failure scenario is determined according to the time when the at least one storage node fails.

In a possible implementation manner, the determining of the failure scenario according to the time when the at least one storage node fails includes:

When the at least one storage node fails within the target duration, the failure scenario is determined as the first failure scenario; otherwise, the failure scenario is determined as the second failure scenario, and the first failure scenario is used for Indicates that the at least one storage node fails simultaneously, and the second failure scenario is used to indicate that the at least one storage node fails at different times.

In a possible implementation manner, the determining the fault state of the distributed storage system according to the fault scenario of the distributed storage system includes:

If the failure scenario is the first failure scenario, the failure state is determined according to the failure type of each storage node in the at least one storage node, and the first failure scenario is used to indicate that the at least one storage node occurs simultaneously failure, the failure type is used to indicate whether the failure of a storage node can be repaired within the second preset time period;

If the failure scenario is a second failure scenario, the failure state is determined according to the failure type of the first storage node that fails the latest among the at least one storage node, and the second failure scenario is used to indicate the at least one storage node. A storage node fails at different times.

In a possible implementation manner, the determining the fault state according to the fault type of each storage node in the at least one storage node includes:

When the fault type of each storage node in the at least one storage node is the first fault type, the fault state is determined as the first fault state, and the first fault type is used to indicate the fault energy of one storage node Repaired within the second preset time period, the first fault state is used to indicate that the at least one storage node can be completely repaired within the first preset time period;

When the failure type of a target number of storage nodes in the at least one storage node is the second failure type, and if the target number is less than or equal to the redundancy of the distributed storage system, determine the failure state is the first fault state, otherwise, the fault state is determined to be the second fault state, and the second fault type is used to indicate that the fault of one storage node cannot be repaired within the second preset time period, so The second fault state is used to indicate that the at least one storage node cannot be completely repaired within the first preset time period.

In a possible implementation manner, the determining the fault state according to the fault type of the first storage node that fails the latest among the at least one storage node includes:

When the fault type of the first storage node is the first fault type, the fault state is determined to be the first fault state, and the first fault type is used to indicate that the fault of one storage node can be being repaired within a preset time period, and the first fault state is used to indicate that the at least one storage node can be completely repaired within the first preset time period;

When the fault type of the first storage node is the second fault type, the fault state is determined to be the second fault state, and the second fault type is used to indicate that the fault of one storage node cannot be repaired within a preset time period, and the second fault state is used to indicate that the at least one storage node cannot be fully repaired within the first preset time period.

In a possible implementation manner, before determining the fault state of the distributed storage system according to the fault scenario of the distributed storage system, the method further includes:

For any storage node in the at least one storage node, when a preset network failure, a preset abnormal power failure, a preset misoperation failure, a preset hardware failure or a preset failure occurs on the any storage node When the software is faulty, the fault type of any storage node is determined as the first fault type, otherwise, the fault type of any storage node is determined as the second fault type, and the first fault type is used for Indicates that the failure of one storage node can be repaired within the second preset time period, and the second failure type is used to indicate that the failure of one storage node cannot be repaired within the second preset time period.

In a possible implementation manner, after the fault processing is performed according to the fault state of the distributed storage system, the method further includes:

When the repair of the at least one storage node is completed, a repair completion response is sent to each device in the distributed storage system, where the repair completion response is used to indicate that there is no faulty device in the distributed storage system.

In a second aspect, a method for handling faults in a distributed storage system is provided, where the distributed storage system includes multiple storage nodes; the method includes:

sending an access request to the target storage node in the distributed storage system;

Receive a response returned by the target storage node; the response includes the fault status of the distributed storage system; the fault status is used to indicate whether at least one faulty storage node can be completely destroyed within a first preset time period repair.

In a possible implementation manner, after receiving the response returned by the target storage node, the method further includes:

Fault handling is performed based on the fault status contained in the response.

In a possible implementation manner, the fault identifier of the fault state includes any one of a first fault identifier or a second fault identifier, wherein the first fault identifier is used to indicate a first fault state, and the second fault identifier The failure identifier is used to indicate a second failure state, the first failure state is used to indicate that the at least one storage node can be completely repaired within a first preset time period, and the second failure state is used to indicate that the at least one storage node is All cannot be repaired within the first preset time period, and the storage node is a faulty storage node in the distributed storage system.

In a possible implementation manner, the performing fault processing based on the fault state included in the response includes:

When the access request is sent by the target client in the distributed storage system based on the target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is the first fault state, no request is sent to the target virtual machine. The target virtual machine responds to the access request, and the first fault state is used to indicate that at least one storage node can be completely repaired within a first preset time period;

When the access request is sent by the target client based on the target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is the second fault state, return to the target virtual machine An abnormal message is stored, and the second fault state is used to indicate that the at least one storage node cannot be completely repaired within the first preset time period.

In a possible implementation manner, the performing fault processing based on the fault state included in the response includes:

When the access request is sent by the target client in the distributed storage system based on the target virtual machine, and the target virtual machine is not a VMWare virtual machine, if the fault state is the first fault state, send the request to the The target virtual machine sends a retry request, where the retry request is used to instruct to re-issue the access request, and the first fault state is used to indicate that at least one storage node can be fully repaired within a first preset duration;

When the access request is sent by the target client based on the target virtual machine, and the target virtual machine is not a VMWare virtual machine, if the fault state is the second fault state, return the information to the target virtual machine. A target error identifiable by the target virtual machine, where the target error is used to indicate a storage medium failure, and the second failure state is used to indicate that the at least one storage node cannot be completely repaired within the first preset time period.

Based on the above possible implementation manners, different fault handling manners are provided, thereby making the fault handling manner provided by this embodiment of the present invention more universal.

In a possible implementation manner, before the sending an access request to any storage node in the distributed storage system, the method further includes:

receiving a target access request sent by a target client in the distributed storage system;

The sending an access request to the target storage node in the distributed storage system includes:

Based on the target access request, the access request is sent to the target storage node in the distributed storage system.

In a possible implementation manner, after receiving the response returned by the target storage node, the method further includes:

A repair completion response returned by the target storage node is received, where the repair completion response is used to indicate that there is no faulty device in the distributed storage system.

In a third aspect, a distributed storage system is provided, the distributed storage system includes a supervisory node and a plurality of storage nodes;

The supervisory node is used to:

A fault state of the distributed storage system is determined according to at least one faulty storage node among the plurality of storage nodes; the fault state is used to indicate whether the at least one faulty storage node can All repaired in time;

sending the fault status to each of the plurality of storage nodes;

Each storage node in the plurality of storage nodes is configured to receive the fault status.

In a possible implementation manner, the fault identifier of the fault state includes any one of a first fault identifier and a second fault identifier, the first fault identifier is used to indicate the first fault state, and the first fault identifier is used to indicate the first fault state. The second fault identifier is used to indicate a second fault state, the first fault state is used to indicate that the at least one storage node can be completely repaired within the first preset time period, and the second fault state is used to indicate that all The at least one storage node cannot all be repaired within the first preset time period.

In a possible implementation manner, each storage node in the plurality of storage nodes is further configured to suspend the access request if the access request is received again after receiving the fault state, based on Receive the fault status and perform fault processing.

In a fourth aspect, a fault processing apparatus is provided, which is used for executing the above-mentioned fault processing method of a distributed storage system. Specifically, the fault processing apparatus includes a functional module for executing the fault processing method provided in the above-mentioned first aspect or any optional manner of the above-mentioned first aspect.

In a fifth aspect, a fault processing apparatus is provided for executing the above-mentioned fault processing method for a distributed storage system. Specifically, the fault processing apparatus includes a functional module for executing the fault processing method provided in the above second aspect or any optional manner of the above second aspect.

In a sixth aspect, a computer device is provided, the computer device includes a processor and a memory, the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above-mentioned method for handling a fault in a distributed storage system. operation.

In a seventh aspect, a storage medium is provided, and at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the operations performed by the above-mentioned method for handling a fault in a distributed storage system.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a schematic diagram of a distributed storage system according to an embodiment of the present invention;

2 is a schematic diagram of a network environment of a distributed storage system provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of interaction between various devices in a distributed storage system according to an embodiment of the present invention

4 is a schematic structural diagram of a computer device provided by an embodiment of the present invention;

5 is a flowchart of a method for handling faults in a distributed storage system provided by an embodiment of the present invention;

6 is a flowchart of a method for handling faults in a distributed storage system provided by an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a fault processing apparatus provided by an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a fault processing apparatus provided by an embodiment of the present invention;

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

FIG. 1 is a schematic diagram of a distributed storage system according to an embodiment of the present invention. Referring to FIG. 1 , the distributed storage system includes at least one client 101 , multiple storage nodes 102 , and a supervisory node 103 . The client 101 is used to provide users with data storage and data reading services, that is, the client 101 can store the data uploaded by the user in the storage node 102 or read data from the storage node 102 .

The storage node 102 is used to store the data written by the client 101, and is also used to return data to the client 101. The returned data can be the data requested by the client 101 to read, and can also be returned to the client 101 under the supervision node 103. The fault state of the distributed storage system is sent, so that the client 101 can handle the fault according to the fault state of the distributed storage system.

The supervisory node 103 is used to monitor whether each storage node 102 in the distributed storage system is faulty. When the number of faulty storage nodes in the distributed storage system is higher than the redundancy of the distributed storage system, it may affect the Therefore, the supervisory node 103 can determine the fault status of the distributed storage system according to the faulty storage node in the distributed storage system, and send the determined fault status to all the storage nodes 102, and the storage The node 102 can inform the client 101 of the fault status of the distributed storage system, so that the client 101 or the storage node 102 can perform fault processing according to the fault status of the distributed storage system.

The baseboard management controller (BMC) of each storage node 102 can monitor in real time whether the respective storage node is faulty, when the BMC of any storage node in the at least one storage node 102 monitors that any storage node occurs In the event of a failure, the BMC can store the reason for the failure of any storage node and the time of failure, and can also send the reason for the failure of any storage node and the time of failure to the supervisory node 103, so that the supervisory node 103 can Find out if any of the storage nodes is faulty.

When the BMC of the storage node does not send the reason for the failure of the storage node and the time of the failure to the supervisory node 103, the supervisory node 103 can access each storage node 102 to learn whether each storage node 102 is faulty. Then, the cause of the failure and the time of the failure can be obtained from the failed storage node.

Regardless of whether the supervisory node 103 receives the cause of the failure and the time of the failure sent by the failed storage node, or actively obtains the cause of the failure and the time of the failure sent by the failed storage node, the supervisory node 103 can The reason for the failure of the faulty storage node and the time of the failure are stored, so that the failure type and the failure scenario can be determined according to the cause of the failure and the time of the failure sent by the failed storage node. The description of the failure type Refer to step 603 below, and refer to step 602 below for the description of the failure scenario.

The supervisory node 103 may store the failure reason and failure time of the failed storage node, which may include: storing the failure reason and failure time sent by the failed storage node in a failure table, and the failure table may Storage number, storage node identifier, failure time, and failure cause, where the number is used to indicate the number of storage nodes that have failed, and the identifier of a storage node is used to uniquely indicate a storage node, which can be the storage node's Internet Protocol address (internet protocol address, IP), and may also be the media access control address (media access control address, MAC) of the storage node, and may also be the number of the storage node in the distributed storage system. The identifier of the storage node is not specifically limited. In addition, the failure time is the time when the storage node fails, and the failure cause is the reason for the failure of the storage node.

For example, in the fault table shown in Table 1, it can be seen from Table 1 that there are two faulty storage nodes in the current distributed storage system, which are the storage node marked X and the storage node marked Y, where the identifier is X The time of failure of the storage node of Y is A, and the reason of failure is D; the time of failure of the storage node marked as Y is B, and the reason of failure is C.

Table 1

Numbering ID of the storage node downtime cause of issue 01 X A D 02 Y C E

It should be noted that after the storage node recorded in the fault table is repaired, the supervisory node can delete the relevant information of the repaired storage node in the fault table, so that the supervisory node can store the data according to the last storage node in the fault table. The number of the node, which determines the number of failed storage nodes in the distributed storage system.

When the storage node in the distributed storage system fails, the file system in the distributed storage system may be damaged. When the file system is damaged, the metadata in the file system may be wrong. Because the metadata is used to describe System data of a file's characteristics, such as access rights, file owner, and distribution information of data blocks in the file, etc. When there is an error in the metadata, the data blocks in the file indicated by the metadata may not be accessible.

In some implementations, when a client fails to access a data block in any storage node, the client can send the data volume of the data block and a data identifier that uniquely identifies the data block to the supervisory node, and the supervisory node receives the After the data volume and data identification, the received data amount and data identification are stored. Specifically, the data can be stored in a data table. The data table can be used to store the total amount of data, the data identification and the corresponding data identification. The amount of data, where the total amount of data is the amount of all data that is currently inaccessible in the distributed storage system.

For example, in the data table shown in Table 2, it can be seen from Table 2 that the currently inaccessible data in the distributed storage system is the data in the data block indicated by the data identifier M and the data in the data block indicated by the data identifier N, Among them, the total amount of data that cannot be accessed currently is 30 kilobytes (kilobyte, KB), and the currently inaccessible data includes 10 KB of data in the data block indicated by the data identifier M and the data indicated by the data identifier N. 20KB of data within a block.

Table 2

It should be noted that when the client accesses the data block that cannot be accessed again, if the client can access the data block successfully, the client sends the access data success response carrying the data identifier of the data block to the supervision node. The data success response is used to indicate that the data in the data block can be accessed. After receiving the access data success response, in the data table, the supervisory node can delete the data volume corresponding to the data identifier and update the data in the data table. the total amount of data. For example, if the access data successfully responds with the data identifier M, the supervisory node deletes the information related to the data identifier M in the data table, and updates the total amount of data to 20KB. It should be noted that, the data table may also store the identifier of the storage node corresponding to the data identifier, so as to indicate which data block in which storage node cannot be accessed.

In some embodiments, since the distributed storage system is responsible for a relatively large amount of business, the number of clients 101 and storage nodes 102 may be relatively large. The application layer of the system may be provided with at least one service switch for interaction between the client 101 and the storage node 102 . In order to facilitate data transmission between the storage nodes 102 , at least one storage switch may be set at the storage layer where the storage nodes 102 are located, so as to realize the interaction between the storage nodes 102 . In order to facilitate data transmission between the supervisory node 103 and the storage node 102 , a supervisory switch may be provided to realize the interaction between the supervisory node 103 and the storage node 102 .

As can be seen from the above description, in the distributed storage system, in addition to business services, monitoring services also need to be provided, and different services can be implemented through different networks. In order to realize the network connection between the client, the storage node and the supervisory node, at least one network port can be installed in the client, the storage node and the supervisory node, and the at least one network port can be used to connect different networks. The network can transmit data of different services, and the at least one network port may be a service network port connected to the service network, a supervision network port connected to the supervision network, and a BMC network port connected to the BMC network.

In order to illustrate the network environment in the distributed storage system, refer to the schematic diagram of the network environment of a distributed storage system provided by an embodiment of the present invention shown in FIG. 2 . In the distributed storage system, the network may include a service network, a supervision network, and a and the BMC network.

The service network is the network used for heartbeat, data synchronization and mirroring between storage nodes. For example, when the data block 1 stored in the storage node 1 is synchronized to the storage node 2, the storage node 1 can use the service network In this service network, data block 1 is sent to storage node 2, then storage node 2 can receive data block 1 through its own service network port, and store data block 1 in storage node 2.

The supervisory network is a network used to monitor whether a storage node is faulty and to query information. In the supervisory network, the fault status of the distributed storage system issued by the supervisory node can be transmitted, and the faulty storage node can also be queried. In some possible implementations, the supervisory node can send the fault status of the distributed storage system to the supervisory network port of the storage node through the supervisory network port. Receive the fault status of the supervisory node. When the storage node receives the client's service request (that is, the SCSI request below), it can directly return the delivered fault status to the client without processing the received service request, so that the client can According to the fault status, the corresponding fault processing can be carried out.

The BMC network is the network that manages the BMC. The supervisory node can monitor the status of the BMC by accessing the BMC network port of the BMC network. Based on the monitored status of the BMC, it can determine whether the storage node is faulty. It should be noted that the BMC network is an optional network. In some embodiments, the BMC may not be used to monitor whether the storage node is faulty, but other methods may be used to monitor whether the storage node is faulty. Therefore, the distribution The BMC network can also not be set up in the storage system, and the monitoring can be realized directly through the supervision network.

When the storage nodes in the distributed storage system are all connected to the service network, the supervisory network, and the BMC network, the supervisory node can receive real-time storage node status information from the three networks, such as information about whether it is faulty.

In order to further illustrate the interaction process between the client, the storage node, and the supervisory node, refer to the schematic diagram of interaction between various devices in a distributed storage system provided by an embodiment of the present invention shown in FIG. 3 . In FIG. 3 , a storage At least one object storage device (OSD) process, a SCSI processing process, and a node monitor service (node monitor service, NMS) agent process may be installed in the node.

One OSD process may correspond to one or more storage media used to store data in the storage node, and the storage media may be a hard disk. The OSD process is used to manage access requests for one or more storage media, and the access request is used to indicate Processing the data to be processed, wherein the processing of the data to be processed may include reading data blocks stored in the at least one or more storage media, the data blocks to be processed include the data to be processed, and performing processing on the data to be processed. Processing may further include writing the data to be processed into the at least one or more storage media, and when the access request may be sent using SCSI, the access request may be regarded as a SCSI request.

The SCSI processing process is used to obtain the SCSI request sent by the client from the service network, convert and decompose the SCSI request, obtain multiple SCSI sub-requests, and deliver the multiple SCSI sub-requests to the corresponding OSD process. For example, the logical block address (LBA) of the SCSI request to carry the data to be read is 100-200. Since the storage location indicated by the LBA 100-150 is in the storage medium 1 of the storage node 1, the LBA 151- The storage location indicated by 200 is in the storage medium 2 of the storage node 1, then the SCSI processing process can convert and decompose the SCSI request into two SCSI sub-requests, wherein the SCSI sub-request 1 is used to indicate the request to read the storage medium 1. The data stored in the LBAs 100-150, and the SCSI sub-request 2 is used to indicate a request to read the data stored in the LBAs 151-200 in the storage medium 2, so that the SCSI processing process can send the SCSI sub-request 1 to the OSD corresponding to the storage medium 1. process, send SCSI sub-request 2 to the OSD process corresponding to storage medium 2.

An NMS agent process is used to receive the fault status of the distributed storage system delivered by the supervisory node, and deliver the received fault status to all OSD processes of a storage node. For example, the client sends the fault status of the distributed storage system to the storage node through the supervision network, and the NMS agent process in the storage node obtains the fault status sent by the supervision node from the supervision network, and sends the obtained fault status to the storage node. In each OSD process in the system, when any OSD process receives the fault status, if it receives the SCSI sub-request or SCSI request sent by the SCSI processing process, it will directly send the received fault status to the SCSI processing process, so that the SCSI processing process can be installed. The device of the process performs fault processing according to the received fault status.

It should be noted that, in some embodiments, the SCSI processing process is not installed in the storage node, but is installed in the client, and the embodiment of the present invention does not specifically limit the device on which the SCSI processing process is installed. For example, the LBA of the SCSI request carrying the data to be read in the SCSI processing process of the client is 0-100. Since the storage location indicated by LBA 0-50 is in storage node 1, the storage location indicated by LBA 51-100 is in In the storage node 2, the SCSI processing process can convert and decompose the SCSI request into 2 SCSI sub-requests, where the SCSI sub-request 1 is used to instruct the request to read the data stored in the LBA 0-50 in the storage node 1, and the SCSI sub-request Request 2 is used to indicate a request to read the data stored at LBA 51-100 in storage node 2, so that the SCSI processing process can send SCSI sub-request 1 to the OSD process in storage node 1, and send SCSI sub-request 2 to the storage node. 2 within the OSD process.

The client, the storage node, and the supervisory node can all be computer equipment. For further description, for the hardware structure of the computer equipment, refer to the schematic structural diagram of a computer equipment provided by an embodiment of the present invention shown in FIG. 4. The computer equipment 400 includes a Large differences in configuration or performance may include one or more processors (central processing units, CPU) 401 and one or more memories 402, wherein the memory 402 stores at least one instruction, and the at least one instruction is stored in the memory 402. An instruction is loaded and executed by the processor 401 to implement the method provided by the following fault handling method embodiment. Of course, the computer device 400 may also have components such as a wired or wireless network interface, a keyboard, and an input/output interface for input and output, and the computer device 400 may also include other components for implementing device functions, which will not be repeated here.

In an exemplary embodiment, a computer-readable storage medium, such as a memory including instructions, is also provided, and the instructions can be executed by a processor in the terminal to complete the fault handling method in the following embodiments. For example, the computer-readable storage medium may be read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape , floppy disk and optical data storage nodes, etc.

In this embodiment of the present invention, the supervisory node may determine the fault state of the distributed storage system according to the faulty storage node, and the supervisory node sends the determined fault state to all storage nodes in the distributed storage system; the client After receiving the user's read request or write request, a SCSI request can be sent to the storage nodes in the distributed storage system to complete the user's read request or write request; when any storage node receives the SCSI request, any The storage node temporarily does not process the received SCSI request, and returns the received fault status to the client, so that the client can perform fault processing based on the fault status returned by any storage node. In some embodiments, any storage node may also perform fault processing based on the fault status. In a possible implementation manner, when any storage node receives the fault status sent by the supervisory node, if it receives the client If the SCSI request is sent, any storage node can perform fault processing according to the fault state.

To further illustrate the above process, refer to the flowchart of a method for handling faults in a distributed storage system provided by an embodiment of the present invention as shown in FIG. 5 , the method specifically includes:

501. The supervisory node determines at least one faulty storage node in the distributed storage system and target data that cannot be accessed by the at least one storage node.

The target data may also be inaccessible data stored in the distributed storage system, and the target data may also be only inaccessible data stored by the at least one storage node. The embodiments of the present invention are described by taking the target data as data that cannot be accessed in the at least one storage node as an example.

The supervisory node may determine the at least one storage node and the target data by querying. In a possible implementation manner, the supervisory node may query the fault table and the data table every time the eighth preset time period elapses, and retrieve the data from the data table. The faulty storage node is determined from the fault table, and the data that cannot be accessed is determined from the data table. The eighth preset duration may be 10 minutes or 1 hour, and the embodiment of the present invention does not specifically limit the eighth preset duration.

It should be noted that the method of determining the faulty storage node from the fault table and the method of determining the inaccessible data from the data table have been described above, and will not be repeated here.

After the supervisory node has performed this step 501, it can also determine the fault state according to the at least one storage node and the target data. Then, it is determined that the at least one storage node in the distributed storage system is faulty and the at least one storage node cannot The accessed target data; the process of determining the fault state according to at least one storage node and the target data, that is, determining the distributed storage node according to at least one faulty storage node in the plurality of storage nodes The process of the failure state of the system. Wherein, the process of determining the fault state of the distributed storage system according to the faulty storage node in at least one of the plurality of storage nodes may be implemented through the process shown in the following step 502 .

502. When the number of the at least one storage node is greater than the redundancy of the distributed storage system, and the data volume of the target data meets a preset condition, the supervisory node determines the fault state of the distributed storage system as the first a fault condition.

The redundancy is the redundancy of the data stored in the distributed storage system, that is, the number of copies of the data stored in the distributed storage system, and the fault status of the distributed storage system is used to indicate that the at least one storage node can No all are repaired within the first preset time period, the fault state may include any one of the first fault state and the second fault state, wherein the second fault state is used to indicate that the at least one storage node cannot operate in the first preset time period. All repaired within the set time. That is, when the at least one storage node can be repaired within the first preset time period, it can be considered that the fault in the distributed storage system can be repaired in a short time, the distributed storage system is in the first fault state, and the first A fault state is also a transient node down (transient node down, TND) state. When the at least one storage node can be repaired within the first preset time period, it is considered that the fault in the distributed storage system cannot be repaired in a short time, the distributed storage system is in the second fault state, and the second fault state is also That is, the long-term node down (permanent node down, PND) state. The first preset duration may be 20 minutes or two or two hours, and the embodiment of the present invention does not specifically limit the first preset duration.

When the data stored in the distributed storage system is large and the number of storage nodes is also large, if the number of faulty storage nodes in the distributed storage system is less than the redundancy of the distributed storage system, the The non-faulty storage nodes can reconstruct the data that cannot be accessed in the faulty storage node according to the data that can be accessed. Therefore, when the number of faulty storage nodes in the distributed storage system is less than the redundancy of the distributed storage system. In the case of redundancy, the normal business of the distributed storage system will not be affected, so there is no need to repair the faulty storage node. However, if the number of faulty storage nodes is greater than the redundancy of the distributed storage system, the non-faulty storage nodes in the distributed storage system cannot reconstruct the failed storage nodes according to the data that can be accessed. The accessed data may affect the normal business in the distributed storage system. And considering whether the data stored in the distributed storage system can be accessed is also an important factor affecting the business, when there are too many target data that cannot be accessed in the distributed storage system, the impact on the business provided by the distributed storage system will also be It is relatively large and may affect the normal operation of the business. Therefore, the supervisory node can first determine the fault status of the distributed storage system, so that the fault can be quickly handled according to the fault status to minimize the impact on the business. When the target data is small, the impact on the business is relatively small, and the normal operation of the business may not be affected. In order to continuously provide services for users, the supervisory node may not perform fault processing for the time being.

The supervisory node can determine whether the data volume of the target data can affect the normal operation of the business through preset conditions, and the preset conditions can include any of the following: the difference between the data volume of the target data and the first preset data volume. The ratio is greater than a preset ratio, and the first preset data volume is the total data volume of all data stored in the distributed storage system; the data volume of the target data is greater than the second preset data volume. When the ratio between the data volume of the target data and the first preset data volume is greater than the preset ratio, or when the data volume of the target data is greater than the second preset data volume, it means that the data volume of the target data is relatively large. It may affect the normal operation of the business, so when the data volume of the target data meets the preset conditions, the supervisory node can first set the fault state of the distributed storage system to the first fault state, so that the Troubleshooting to reduce business impact. If the at least one storage node is not fully repaired after the first preset time, the fault state may be updated to the second fault state. It should be noted that the preset ratio may be 0.4, 0.5, or 0.6, and the embodiment of the present invention does not specifically limit the preset ratio, the first preset data amount, and the second preset data amount.

In order to be able to compare the number of the at least one storage node with the redundancy of the distributed storage system, the supervisory node may query a fault table stored by the supervisory node, and determine the number of the at least one storage node from the fault table. In order to be able to determine whether the data volume of the target data meets the preset condition, the supervisory node may query the stored data table, and determine the data volume of the target data currently inaccessible in the distributed storage system from the data table. For example, the supervisory node may determine that the target data includes 10KB of data in the data block indicated by the data identifier M and 20KB of data in the data block indicated by the data identifier N by looking up the table 2.

It should be noted that when the fault table was introduced in the foregoing, the process of determining the number of faulty storage nodes in the distributed storage system from the fault table was described. The process of determining the number of faulty storage nodes in the distributed storage system will not be described in detail.

It should be noted that the process shown in this step 502 is also a process of determining the fault state of the distributed storage system according to the at least one storage node and the target data.

503. The supervisory node sends a first fault identifier for indicating a first fault state to all storage nodes in the distributed storage system.

The fault identifier of the fault state includes any one of the first fault identifier and the second fault identifier, wherein the first fault identifier is used to indicate the first fault state, and the second fault identifier is used to indicate the second fault state, and the first fault identifier is used to indicate the second fault state. The identifier and the second fault identifier may be different. For example, the first fault identifier may be s, and the second fault identifier may be t. The embodiments of the present invention do not specifically limit the representations of the first fault identifier and the second fault identifier.

The supervisory node may send the first fault identifier to the NMS agent process of each storage node, so as to send the first fault identifier to all storage nodes, so as to inform all storage nodes that the current fault state of the distributed storage system is the first fault state.

It should be noted that the process shown in this step 503 is also the process of the supervisory node sending the fault status to each of the multiple storage nodes included in the distributed storage system.

It should be noted that, in some embodiments, after determining the fault state, the supervisory node may also perform fault processing according to the fault state of the storage system. The embodiment of the present invention does not specify a process for the supervisory node to perform fault processing. limited.

504. The target storage node in the distributed storage system receives the first fault identifier.

The target storage node is any storage node in the distributed storage system, and each OSD process in the target storage node can obtain the first fault identifier from the NMS proxy process of the target storage node, so that the Each OSD process can acquire the first fault identifier. It should be noted that, each storage node in the distributed storage system can perform this step 504, and the faulty storage node may be able to receive the first fault identifier, or may not receive the first fault identifier.

505. The target device sends an access request to the target storage node,

The access request is used to instruct to read data stored in the target storage node or write data to the target storage node. The target device is a device installed with a SCSI processing process, and may be the target client or a target storage node, wherein the target client is any client in the distributed storage system. This step 506 may be implemented by a SCSI processing process in the target device.

Before 505 in this step, the target client in the distributed storage system may send a target access request to the target device, where the target access request is used to perform processing on the first target data, wherein the first target data includes the data of the access request. The indicated data, the target access request may carry a target storage address, and the target storage address may be the storage address of the first target data. The target access request is sent by the target virtual machine that the target client can install. Specifically, the target virtual machine may send the target access request to a SCSI processing process in the target device. The target virtual machine sends a target access request to the SCSI processing process can be triggered by user actions. For example, when the user enters the storage address of the data to be read in the client interface and clicks the read button, the The target virtual machine sends a target access request to the SCSI processing process, so as to request to read the data stored at the storage address input by the user.

Then, the target device receives the target access request sent by the target client in the distributed storage system, and this step 506 can be implemented in the following manner: the target device sends a request to the target storage node in the distributed storage system based on the target access request. Send an access request. Specifically, after receiving the target access request, the SCSI processing process transforms and decomposes the target access request according to the target address, and obtains multiple access requests. Each access request can carry part of the address in the target address. The address can be an offset address in the storage medium managed by any OSD process in the target storage node, so that the SCSI processing process sends the corresponding access request to the OSD process in the target storage node, and the process of converting the access request to the access request That is, the aforementioned process of converting the SCSI request.

506. After the target storage node in the distributed storage system receives the first fault identifier, if the target storage node receives the access request again, the target storage node suspends the access request and sends the first fault to the target device. logo.

This step 506 may be implemented by the OSD process that receives the access request in the target storage node. When the target storage node receives the first fault identifier, it means that the target storage node already knows that there is a faulty storage node in the distributed storage system, and the current fault state is the first fault state, because the current distributed storage system If the storage node in the storage node fails, the target storage node can suspend the access request, temporarily suspend the access request, and wait for the faulty storage node to be repaired automatically or manually.

In order for the target device to also know the fault state of the distributed storage system, when the target storage node receives the access request sent by the target device, the target storage node can output the first fault identifier to the target device. Specifically, after any OSD process in the target storage node receives the access request sent by the SCSI processing process of the target device, any OSD process sends the first fault identifier to the SCSI processing process. Of course, in some embodiments, after the supervisory node has determined the fault status of the distributed storage system, it can also directly send the fault status to the target device, so that the target device can obtain the fault status of the distributed storage system, so there is no need to pass The storage node sends a fault status to the target device.

It should be noted that the process shown in this step 506 is also the process of outputting the fault identifier when the target storage node receives the access request after the target storage node in the distributed storage system receives the fault identifier. .

507. The target device receives the first fault identifier returned by the target storage node based on the access request.

This step 507 is implemented by the SCSI processing process in the target device, and the process shown in this step 507 is also the process of receiving the fault identifier returned by the target storage node based on the access request. When any OSD process sends the first fault status to the SCSI processing process, the SCSI processing process can receive the first fault identification. It should be noted that the process shown in this step 507, that is, receiving the response returned by the target storage node; the response includes the fault status of the distributed storage system; the fault status is used to indicate at least The process of whether a faulty storage node can be completely repaired within a first preset time period, wherein the response includes the fault status of the distributed storage system, that is, the fault identifier.

508. The target device performs fault processing based on the received first fault identifier.

This step 508 may be performed by a SCSI processing process installed in the target device. After the SCSI processing process receives the first fault identifier, the SCSI processing process may perform fault processing based on the first fault identifier, so that the target client can be processed accordingly.

In some embodiments, the target virtual machine connected to the SCSI processing process in the target client may be a VMWare virtual machine or may not be a VMWare virtual machine. Since different virtual machine target devices have different fault handling methods, this step 508 can be implemented in any of the following ways 1-2.

Mode 1. When the access request is sent by the target client in the distributed storage system based on the target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is the first fault state, the SCSI processing process does not work. Respond to the access request to the target virtual machine.

When the SCSI processing process receives the first fault identifier, it indicates that the distributed storage system is in the first fault state. For the VMWare virtual machine, the first fault state is also the ADP state. And since the access request received by the SCSI processing process is sent by the target virtual machine, in order for the VMWare virtual machine to perceive that the distributed storage system is in the APD state defined by itself, the SCSI processing process does not respond to the target virtual machine, and because the distribution All SCSI processing processes in the storage system will receive the first fault identifier as long as they issue access requests to the OSD process, and each SCSI processing process processing access requests will not respond to the target virtual machine, so that it is possible to simulate distributed All links in the storage system are unresponsive (DOWN). For the target virtual machine, if no response from the SCSI processing process is received, the target virtual machine will continue to send access requests, so that even if the storage nodes in the distributed storage system do not have all the In the event of a fault, you can also perform fault processing according to the fault state defined by the VMWare virtual machine.

Mode 2: When the access request is sent by the target client based on the target virtual machine, and the target virtual machine is not a VMWare virtual machine, if the fault state is the first fault state, the SCSI processing process sends the target virtual machine to the Retry request, the retry request is used to instruct to resend the access request.

The detection keyword carried in the retry request can be the Unit Attention (0x6) error code, and the Unit Attention (0x6) error code can indicate that the storage medium or link status in the storage node has changed, that is, a fault has occurred, the target After the virtual machine receives the retry request, the target virtual machine re-issues an access request to the SCSI processing process, so as to implement the processing method in the above-mentioned method 1.

For different virtual machines, the embodiments of the present invention provide different fault handling methods, so that the fault handling methods provided by the embodiments of the present invention are more universal.

It should be noted that the process shown in this step 508 is also a process of performing fault processing based on the fault state included in the response.

509. If the at least one storage node is all repaired within the first preset time period, the supervisory node sends a repair completion response to each device in the distributed storage system, and the repair completion response is used to indicate that there is no storage in the distributed storage system. faulty equipment.

The respective devices include storage nodes and clients. When all the at least one storage node is repaired within the first preset time period, it means that there is no faulty device in the distributed storage system at this time, and if the supervisory node stores the first fault identifier used to identify the first fault state , the supervisory node can delete the first fault identifier, and send a repair completion response to each device in the distributed storage system, and has informed each device that there is no faulty device in the distributed storage system and can work normally, then when each device After the repair completion response is received, the previously received fault identifier is deleted, and normal work can be started. It should be noted that the process shown in step 509 is also a process of sending a repair completion response to each device in the distributed storage system when the at least one storage node is repaired.

In the prior art, when the client does not obtain a response from any storage node, it directly considers that the fault state of the distributed storage system is the APD state, and only considers the fault of the distributed storage system to be the fault of the distributed storage system when the storage node clearly returns a message of storage exception. The state is determined to be the PDL state. If the storage node has a long-term failure, it may not be able to return a storage exception message to the client, so the client may determine the failure state as the APD state. Therefore, in the prior art, the failure of the distributed storage system is determined. The status is imprecise, and if the PDL status is mistaken for the APD status, the repair work on the service side will not be carried out, and eventually the fault repair time will be prolonged. In the embodiment provided by the present invention, each storage node in the distributed storage system knows the fault status of the distributed storage system, and for the storage node that does not have a fault, the fault status can be returned to the target device based on the SCSI request. , so that the target device can clearly know the fault state of the distributed storage system, thereby improving the accuracy of the target device in determining the fault state.

510. When the fault state is the first fault state, if the at least one storage node is not all repaired within the first preset time period, the supervisory node updates the fault state from the first fault state to the second fault state .

Since the first preset duration is only a preset duration, within the first preset duration, the at least one storage node may not be fully repaired, then, when the at least one storage node may not be fully repaired, the It may take longer to repair unrepaired storage nodes. Since the time period for repairing an unrepaired storage node is uncertain and may be relatively long, the supervisory node can directly update the fault state from the first fault state to the second fault state.

511. When the fault state is updated from the first fault state to the second fault state, the supervisory node sends a second fault identifier for indicating the second fault to all storage nodes in the distributed storage system.

The manner in which the supervisory node sends the second fault identifier to all storage nodes is the same as the manner in which all storage nodes send the first fault identifier in step 503 , and this step 511 is not repeated in this embodiment of the present invention. The process shown in this step 511 is also the process of sending the fault status to each of the multiple storage nodes.

512. The target storage node receives the second fault identifier.

The manner in which the target storage node receives the second fault identifier is the same as the manner in which the first fault identifier is received in step 504, and this step 512 is not repeated in this embodiment of the present invention.

513. The target device sends an access request to the target storage node.

The manner in which the target device sends an access request to the target storage node is described in step 505 , and this step 513 is not repeated in this embodiment of the present invention.

514. When the target storage node in the distributed storage system receives the second fault identifier, if the target storage node receives the access request again, the target storage node suspends the access request and outputs the second fault identifier.

The manner in which any storage node suspends the access request and outputs the second fault identifier is the same as the manner in which the target storage node suspends the access request and outputs the first fault identifier in step 506 . 514 does not go into details.

515. The target device receives the second fault identifier returned by the target storage node based on the access request.

The manner in which the target device receives the second fault identifier is the same as the manner in which the first fault identifier is received in step 507, and details are not described herein in this embodiment of the present invention.

516. The target device performs fault processing based on the received second fault identifier.

This step 516 may be performed by the SCSI processing process of the target device. In some embodiments, the target virtual machine connected to the SCSI processing process in the client may be a VMWare virtual machine, or may not be a VMWare virtual machine, because different virtual machines The target device performs fault processing in different ways. Therefore, this step 516 can be implemented in any of the following ways 3-4.

Mode 3. When the access request is sent by the target client based on the target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is the second fault state, the SCSI processing process returns a storage exception to the VMWare virtual machine news.

When the fault is marked with the second fault, it means that at least one storage node cannot be repaired within the first preset time, and it takes a longer time. The target virtual machine can perform fault processing in the PDL state, so that the target virtual machine can Perceiving the PDL state, the abnormal storage message can carry SCSI errors such as SK 0x0, ASC&ASCQ0x0200 or SK 0x5, ASC&ASCQ 0x2500 customized by the VMWare virtual machine, and the SCSI error can indicate that the state in the distributed storage system is the PDL state, so that the target virtual machine The PDL status can be sensed after receiving the message of the storage exception. Then the target virtual machine can power off the file system in the distributed storage system and wait for the technician to repair the fault in the distributed storage system, or customize it according to the user-defined function. Select the optimal fault handling method to deal with the faulty storage node. For example, power off the failed storage node.

It should be noted that when the storage node in the distributed storage system has a fault that cannot be repaired in a short time, the file system may be abnormal. If you want to use it, you need to power off the file system first. When the repair is complete, power up the file system and repair the file system.

Mode 4. When the access request is sent by the target client based on the target virtual machine, and the target virtual machine is not a VMWare virtual machine, if the fault state is the second fault state, the target device returns the target virtual machine. A target error recognized by the target virtual machine that indicates a storage media failure.

The target error can be a Sense key 0x3 error, that is, a storage medium error (Medium Error), which can be identified by general virtual machines. When the target virtual machine receives the target error, it means that the state of the distributed storage system is the first. In the second fault state, the target device can power off the distributed file system and wait for technicians to repair the fault in the distributed storage system, or select a better fault processing method to deal with the faulty storage node according to the user-defined fault processing method. .

It should be noted that the process shown in this step 516 is also a process of performing fault processing on the target device based on the received fault identifier, that is, a process of performing fault processing based on the fault status included in the response.

517. When the at least one storage node is repaired, send a repair completion response to each device in the distributed storage system.

This step 517 is the same as that of step 509, and this step 517 is not described repeatedly in this embodiment of the present invention. It should be noted that the fault of each storage node may be repaired by itself or by a technician, and the embodiment of the present invention does not specifically limit the repairing method of the storage node.

It should be noted that, when the client receives the repair completion response, if the file system has been powered off, the client powers on the file system and repairs the file system. Since a large amount of metadata is stored in the file system, when repairing the file system, it is necessary to scan all the metadata in the file system and modify the scanned error metadata. Generally, the process of repairing the file system needs to consume part of the metadata. In the first fault state, the client will not power off the file system. Once the at least one storage node can be repaired within the first preset time, the file system can be prevented from being powered off, thereby reducing the need for repairs. The time of the file system enables the distributed storage system to resume business as soon as possible to ensure the quality of service.

In the method shown in the embodiment of the present invention, the fault state of the distributed storage system is determined according to the faulty storage node of at least one of the plurality of storage nodes, so that it is not necessary to determine the distributed storage system only when all storage nodes are faulty. The fault status of the distributed storage system can be sent to each storage node in the distributed storage system immediately after the fault status is determined, so that each storage node can perform fault processing according to the determined fault status, thereby reducing the distributed storage system. The time it takes for the storage system to return to normal. Moreover, for different virtual machines, the embodiments of the present invention provide different fault handling methods, so that the fault handling methods provided by the embodiments of the present invention are more universal. In addition, each storage node in the distributed storage system knows the fault status of the distributed storage system. For storage nodes that have not failed, the fault status can be returned to the target device based on the access request, so that the target device can clearly know the fault status. The fault state of the distributed storage system can further improve the accuracy of determining the fault state of the target device. Moreover, in the first fault state, the target device will not power off the file system. Once the at least one storage node can be repaired within the first preset time, the file system can also be avoided from powering off, and when the at least one storage node can be repaired After a storage node is restored, the file system and services can be restored immediately, which can reduce the time for repairing the file system and enable the distributed storage system to restore services as soon as possible to ensure service quality.

Since the failure of a storage node may be repairable in a short time, it may also take a long time to repair, and the repair time of each storage node may affect the repair time of the entire distributed storage system. Therefore, in some embodiments , the fault state of the distributed storage system can also be determined according to the fault type of each storage node. To further illustrate this process, refer to the flowchart of a method for handling faults in a distributed storage system provided by an embodiment of the present invention as shown in FIG. 6 . The flowchart of the method may include the following steps.

601. The supervisory node determines at least one faulty storage node in the distributed storage system and target data that cannot be accessed by the at least one storage node.

This step 601 is the same as that of step 501, and the embodiment of the present invention does not describe step 601 in detail here.

602. The supervisory node determines a failure scenario of the distributed storage system according to the time when the at least one storage node fails, where the failure scenario is used to indicate whether the at least one storage node fails simultaneously.

The failure scenario may include any one of a first failure scenario and a second failure scenario, wherein the first failure scenario is used to indicate that the at least one storage node fails simultaneously, and the second failure scenario is used to indicate that the at least one storage node fails Nodes fail at different times.

The supervisory node can determine the failure time of each storage node from the stored fault table, so that the supervisory node can determine the failure scenario according to whether the failure time of at least one storage node is the same.

In a possible implementation manner, when the at least one storage node fails within the target duration, the supervisory node determines the failure scenario as the first failure scenario, otherwise, determines the failure scenario as the second failure scenario . It should be noted that the process shown in this step 602 is also the process of determining the failure scenario according to the time when the at least one storage node fails.

603. For any storage node in the at least one storage node, when a preset network failure, a preset abnormal power failure, a preset misoperation failure, a preset hardware failure, or a preset failure occurs in the any storage node; When the set software fails, the supervisory node determines the failure type of any storage node as the first failure type, otherwise, determines the failure type of any storage node as the second failure type.

The fault type of the storage node is used to indicate whether the fault of a storage node can be repaired within the second preset time period, the fault type may include any one of the first fault type and the second fault type, and the second fault type is used to indicate A failure of a storage node cannot be repaired within the second preset duration, the second preset duration may be less than or equal to the first preset duration, and the second preset duration is not specifically limited in this embodiment of the present invention.

Wherein, the preset network fault may include any of the following items 1.1-1.7:

Item 1.1. The service network port of any storage node cannot be accessed, but the supervision network port of any storage node can be accessed. The service network port is the service used for heartbeat, data synchronization and mirroring between storage nodes. The network port of the network, the supervision network port is the network port of the supervision network used to monitor whether the storage node is faulty and perform information query.

The supervisory node can send an Internet packet explorer (ping) request to the service network port of any storage node through the service network port of the supervisory node. The ping request is used to request the establishment of a connection. If the connection is successful, it is considered that any storage node If the service network port of the node can be accessed, otherwise, it is considered that the service network port of any storage node cannot be accessed. Similarly, the supervisory node can send a ping request to the supervisory network port of any storage node through the supervisory network port of the supervisory node. If the connection is successful, the supervisory network port of any storage node is considered to be accessible; otherwise, the supervisory network port of any storage node is considered to be accessible. The supervision network port of a storage node cannot be accessed.

If the supervisory node cannot access any of the storage nodes through the service network port, it means that any storage node has a network failure, but can access any of the storage nodes through the supervisory network port, it means that the failure of any of the storage nodes can be If it is repaired within the time limit, the preset network failure occurs on any storage node.

Item 1.2. When the service network and the supervisory network are the same target network, the data packets transmitted by any storage node in the target network have a first preset number of packet loss or a second preset number of malformations package, and the service network port, supervision network port, and baseboard management controller BMC network port of any storage node are inaccessible, and the BMC network port is the network port of the BMC network that manages the BMC.

The supervisory node can send a ping request to the BMC network port of any storage node through the BMC network port of the supervisory node. If the connection is successful, the BMC network port of any storage node is considered to be accessible; otherwise, it is considered that any storage node can be accessed. The BMC network port of the node cannot be accessed.

In the delivery stage, technicians can configure the service network and the supervision network as the same network, that is, the target network. When the service network and the supervision network are the same target network, if any storage node is on the target network The first preset number of lost packets or the second preset number of malformed packets in the data packets transmitted within the network indicate that any storage node has a network failure. If the supervisory node cannot access any storage node, it indicates that the If the failure of a storage node can be repaired in a short time, then any storage node has a preset network failure.

Item 1.3. When the service network and the supervisory network are the same target network, the data packets transmitted by any storage node in the target network have packet loss greater than the first preset number or greater than the second preset number. malformed packets, and the delay for any storage node to transmit data in the target network is greater than the third preset duration.

It should be noted that when the supervisory node sends a ping request to any storage node, if the connection is unsuccessful, any of the storage nodes will send a connection failure response to the supervisory node, and the connection failure response is used to indicate that the connection fails. And the connection failure response can carry delay information, the delay information is used to indicate the delay of any storage node transmitting data in the target network, so that the supervision node can judge whether the delay indicated by the delay information is greater than or equal to The third preset duration.

Item 1.4. When the service network and the supervisory network are the same target network, the flow control PFC packets with the priority of any storage node in the target network are greater than the third preset number, and any storage node Node is not accessible.

The supervisory node can detect flow control (priority-based flowcontrol, PFC) packets of each priority in the target network, so as to determine whether the number of PFC packets sent by any storage node is greater than a third preset number, the supervisory The node can then determine whether any of the storage nodes meets the preset conditions. This embodiment of the present invention does not specifically limit the third preset number.

Item 1.5. When the service network and the supervision network are the same target network, any storage node in the target network sends a third preset number of priority flow control PFC packets, and any storage node appears in the target network. The delay for the node to transmit data in the target network is greater than the fourth preset duration.

It should be noted that, the embodiment of the present invention does not specifically limit the four preset durations.

Item 1.6. When the service network and the supervision network are the same target network, a broadcast storm caused by any storage node occurs in the target network, and the service network port, supervision network port and BMC network port of any storage node mouth is not accessible.

It should be noted that when any storage node sends a large number of broadcast packets in the target network, a broadcast storm may occur in the target network.

Item 1.7. When the service network and the supervision network are the same target network, a broadcast storm caused by any storage node occurs in the target network, and the delay of any storage node in the target network is greater than the Five preset durations.

It should be noted that, the embodiment of the present invention does not specifically limit the five preset durations.

The preset abnormal power failure can include any of the following items 2.1-2.2:

Item 2.1. The service network ports, supervisory network ports, and BMC network ports of all storage nodes in the chassis are inaccessible, and the chassis includes any of the storage nodes.

A chassis can include at least one storage node. When all the storage node service network ports, supervisory network ports, and BMC network ports are inaccessible, it can be considered that all storage nodes in the chassis are powered off. If any storage node is in the chassis, it means that any storage node is also powered off. As long as the chassis is powered on, the fault of any storage node can be repaired, and it is considered that any storage node has a preset fault. abnormal power failure.

Item 2.2. Within the seventh preset time period, the service network ports, supervisory network ports and BMC network ports of the storage nodes of the first target number are all inaccessible, and the storage nodes of the first target number include any storage node of the storage node. node.

When the service network ports, supervisor network ports and BMC network ports of the storage nodes of the first target number are all inaccessible within the seventh preset time period, the storage nodes of the first target number can be considered to have the preset number of storage nodes. Abnormal power failure. It should be noted that, the embodiment of the present invention does not specifically limit the seventh preset duration.

The preset misoperation fault may include: any storage node is actively powered off. For example, when a user clicks the shutdown button or restart button of any storage node, the storage node considers it to be powered off actively, and sends the information about the active power off to the supervisory node, so that the supervisory node determines that any storage node has a preset misoperation failure.

The preset hardware failure includes: any storage node exits abnormally, the BMC network port of any storage node can be accessed, and any storage node has loose components.

When any storage node exits abnormally, it can send abnormal exit information to the supervisory node to indicate that it has exited abnormally. The abnormal exit may be caused by loose internal components. The loose parts can be memory sticks and card sticks, etc. The loose parts can be recovered immediately by plugging and unplugging, that is, a short-term fault occurs. It should be noted that when any storage node detects that any component is poorly connected, it means that any storage node has loose components, and then any storage node can send loose component information to the supervisory node, so that the supervisory node can According to the information of the loose parts, it can be determined that a preset misoperation fault occurs in any storage node.

The preset software fault may include any of the following items 3.1-3.3:

Item 3.1: An abnormality of the operating system of any storage node causes the abnormal reset of any storage node.

When the memory of any storage node is insufficient, the operating system of any storage node cannot continue to run and needs to be reset, or any storage node is reset abnormally triggered by the watchdog, etc. During reset, any storage node can send a message of abnormal reset to the supervisory node, so that the supervisory node can know that any of the storage nodes is abnormally reset, indicating that a preset software fault occurs in any of the storage nodes.

Item 3.2, the software exception of any storage node causes the target process of any storage node to exit.

The target process may be an OSD process. When any storage node is abnormally reset, any storage node can send a message that the target process has exited to the supervisory node, so that the supervisory node can learn that the target process of any storage node has exited , it means that any storage node has a preset software failure.

Item 3.3, the software exception of any storage node causes the operating system of any storage node to reset.

Due to a software exception, when the operating system of any storage node is reset, any storage node can send a message of operating system reset to the supervisory node, so that the supervisory node can learn that the operating system of any storage node is reset. A preset software failure has occurred on either storage node.

It should be noted that, when each storage node in the at least one storage node is faulty, the supervisory node can determine whether the fault type of each storage node is the first fault type or the second fault type through this step 603, and determine whether the fault type of each storage node is the first fault type or the second fault type. The fault type of each storage node is stored in the fault table, so that when the supervisory node needs the fault type of any storage node, it can be directly obtained from the fault table.

It should be noted that, the various fault type discrimination methods embodied in this step 603 can accurately determine the fault type of each storage node, and then according to the exact fault type of each storage node, the distributed storage can be more accurately determined. The fault state of the system.

604. When the number of the at least one storage node is greater than the redundancy of the distributed storage system, and the data volume of the target data meets a preset condition, if the failure scenario is the first failure scenario, the supervisory node will The failure type of each storage node in the at least one storage node determines the failure state.

Since the first failure scenario is used to indicate that the at least one storage node occurs simultaneously, the supervisory node may determine the failure state of the distributed storage system according to each storage node.

In a possible implementation manner, the supervisory node determines the fault state according to the fault type of each storage node in the at least one storage node, which may include: when the fault type of each storage node in the at least one storage node is In the case of the first failure type, the supervisory node determines the failure state as the first failure state, and the first failure state is used to indicate that the at least one storage node can be completely repaired within the first preset time period; when the first failure state is When the failure type of a target number of storage nodes in at least one storage node is the second failure type, if the target number is less than or equal to the redundancy of the distributed storage system, the supervisory node determines the failure state as the failure state. The first failure state, otherwise, the failure state is determined as the second failure state, and the second failure state is used to indicate that the at least one storage node cannot be fully repaired within the first preset time period.

When the fault type of each storage node in the at least one storage node is the first fault type, it can be considered that the fault in the distributed storage system can be repaired in a short time, and the fault state can be determined as the first fault state. Although the failure type of the target number of storage nodes is the second failure type, when the target number is less than or equal to the redundancy of the distributed storage system, it means that the target number of storage nodes has little impact on the distributed storage system , the fault state can be determined as the first fault state. Once the target number is greater than the redundancy of the distributed storage system, indicating that the target number of storage nodes has a greater impact on the distributed storage system, the fault state can be determined as the second fault state, so as to quickly repair the fault.

It should be noted that the description of whether the number of the at least one storage node is greater than the redundancy of the distributed storage system and whether the data volume of the target data meets the preset conditions is reflected in the foregoing description. The embodiment will not be repeated.

It should be noted that, when the fault scenario is the first fault scenario, the supervisory node can use any preset fault among the preset network fault in step 603, the preset abnormal power failure fault or the preset misoperation fault, Determine the fault type of each storage node.

605. When the number of the at least one storage node is greater than the redundancy of the distributed storage system, and the data volume of the target data meets the preset condition, if the failure scenario is the second failure scenario, the supervisor node will The failure type of the last first storage node in one storage node that fails, and the failure state is determined.

Since the second failure scenario is used to indicate that the at least one storage node fails at different times, the supervisory node may determine the distributed storage system according to the failure type of the last failed storage node among the at least one storage node. The fault state is the first storage node, where the last faulty storage node is also the first storage node.

In a possible implementation manner, when the fault type of the first storage node is the first fault type, the supervisory node determines the fault state as the first fault state, and the first fault state is used to indicate the at least one The storage nodes can all be repaired within the first preset time period; when the failure type of the first storage node is the second failure type, the supervisory node determines the failure state as the second failure state, and the second failure state is the second failure state. The fault state is used to indicate that the at least one storage node cannot be fully repaired within the first preset time period.

It should be noted that the processes shown in steps 604 and 605 are also processes of determining the fault state of the distributed storage system according to the fault scenario of the distributed storage system.

It should be noted that, when the failure scenario is the second failure scenario, the supervisory node only needs to determine the failure type of the first storage node, and does not need to determine the failure type of all storage nodes. The network failure preset in step 603 can be used. The fault type of the first storage node is determined based on any one of a preset abnormal power failure fault, a preset misoperation fault, a preset hardware fault, or a preset software fault.

606. The supervisory node sends a fault identifier for indicating a fault state to all storage nodes in the distributed storage system.

When the fault state is the first fault state, the supervisory node sends the first fault identifier to all storage nodes in the distributed storage system, and when the fault state is the second fault state, sends the supervisory node to the distributed storage system. All storage nodes in the storage system send the second fault identifier, and the specific execution process is the same as that of step 503, which is not repeated here. It should be noted that, the process shown in this step 606 is also the process of sending the fault status to each of the plurality of storage nodes.

607. The target storage node in the distributed storage system receives the fault identifier.

When the fault identifier is the first fault identifier, the target storage node receives the first fault identifier. When the fault identifier is the second fault identifier, the target storage node receives the second fault identifier. The specific execution process is the same as step 504. , which will not be repeated here.

608. The target device sends an access request to the target storage node.

This step 608 is the same as the process shown in step 505, and this step 608 is not described repeatedly in this embodiment of the present invention.

609. After the target storage node in the distributed storage system receives the fault identifier, if the target storage node receives the access request again, the target storage node suspends the access request and outputs the fault identifier.

When the fault identifier is the first fault identifier, the target storage node outputs the first fault identifier to the target device. When the fault identifier is the second fault identifier, the target storage node outputs the second fault identifier to the target device. The specific execution process is as follows: The same is true for step 506, which is not repeated here.

610. The target device receives the fault identifier returned by the target storage node based on the access request.

This step 610 is the same as the process shown in step 505, and this step 610 is not described repeatedly in this embodiment of the present invention. It should be noted that the process shown in this step 610 is also to receive the response returned by the target storage node; the response includes the fault status of the distributed storage system; the fault status is used to indicate that at least one occurrence of The process of whether all the faulty storage nodes can be repaired within the first preset time period. Wherein, the fault state included in the response is also the fault identifier.

611. The target device performs fault processing based on the received fault identifier.

When the fault identifier is the first fault identifier, the target device performs fault processing based on the received first fault identifier, and the specific execution process is the same as the process shown in step 508 . When the fault identifier is the second fault identifier, the target device performs fault processing based on the received second fault identifier. The specific execution process is the same as the process shown in step 516. Here, this embodiment of the present invention does not perform step 611. Repeat.

It should be noted that each storage node in the distributed storage system knows the fault status of the distributed storage system. For storage nodes that have not failed, the fault status can be returned to the target device based on the access request, so that the target device can The fault state of the distributed storage system is clearly known, thereby improving the accuracy of the target device in determining the fault state.

It should be noted that, for different virtual machines, the embodiments of the present invention provide different fault processing methods, so that the fault processing methods provided by the embodiments of the present invention are more universal.

It should be noted that the process shown in this step 611 is also a process of performing fault processing based on the fault state included in the response.

612. When the repair of the at least one storage node is completed, the supervisory node sends a repair completion response to each device in the distributed storage system.

This step 612 is the same as that of step 509, and this step 612 is not described repeatedly in this embodiment of the present invention. It should be noted that, when the fault state is the first fault state, if the at least one storage node is completely restored within the first preset time period, this step 612 can be directly executed. If the at least one storage node is not all repaired within the time period, the supervisory node updates the fault state from the first fault state to the second fault state, and jumps to step 606 . It should be noted that, in the first fault state, the client will not power off the file system. Once the at least one storage node can be repaired within the first preset time, the file system can be avoided from powering off, thereby It can reduce the time to repair the file system, so that the distributed storage system can resume business as soon as possible to ensure the quality of service.

In the method shown in the embodiment of the present invention, the fault state of the distributed storage system is determined according to the faulty storage node of at least one of the plurality of storage nodes, so that it is not necessary to determine the distributed storage system only when all storage nodes are faulty. The fault status of the distributed storage system can be sent to each storage node in the distributed storage system immediately after the fault status is determined, so that each storage node can perform fault processing according to the determined fault status, thereby reducing the distributed storage system. The time it takes for the storage system to return to normal. Moreover, for different virtual machines, the embodiments of the present invention provide different fault handling methods, so that the fault handling methods provided by the embodiments of the present invention are more universal. In addition, each storage node in the distributed storage system knows the fault status of the distributed storage system. For storage nodes that have not failed, the fault status can be returned to the target device based on the access request, so that the target device can clearly know the fault status. The fault state of the distributed storage system can further improve the accuracy of determining the fault state of the target device. In addition, in the first fault state, the target device will not power off the file system. Once the at least one storage node can be repaired within the first preset time, the file system can be prevented from being powered off, thereby reducing the need for repairs. The time of the file system enables the distributed storage system to resume business as soon as possible to ensure the quality of service. In addition, the various fault type discrimination methods embodied in this step 603 can accurately determine the fault type of each storage node, and then more accurately determine the fault of the distributed storage system according to the exact fault type of each storage node. state.

7 is a schematic structural diagram of a fault processing apparatus provided by an embodiment of the present invention, which is applied to a distributed storage system, where the distributed storage system includes a plurality of storage nodes, and the apparatus includes:

A determination module 701, configured to determine a fault state of the distributed storage system according to at least one faulty storage node in the plurality of storage nodes; the fault state is used to indicate the at least one faulty storage node Can all be repaired within the first preset duration;

A sending module 702, configured to send the fault status to each of the multiple storage nodes.

Optionally, the device further includes:

A processing module, configured to send the fault status to each of the plurality of storage nodes.

Optionally, the determining module 701, the determining module includes:

a first determining unit, configured to perform the above step 501;

A second determining unit, configured to determine the fault state according to the at least one storage node and the target data.

Optionally, the second determining unit is used for:

When the number of the at least one storage node is greater than the redundancy of the distributed storage system, and the data volume of the target data meets a preset condition, the fault state is determined to be a first fault state, and the first fault state is determined. A fault state is used to indicate that the at least one storage node can be completely repaired within the first preset time period.

Optionally, the second determining unit is used for:

When the number of the at least one storage node is greater than the redundancy of the distributed storage system, and the data volume of the target data meets a preset condition, determine the failure according to the failure scenario of the distributed storage system status, and the failure scenario is used to indicate whether the at least one storage node fails simultaneously.

Optionally, the preset conditions include any of the following:

The ratio between the data volume of the target data and the first preset data volume is greater than the preset ratio, and the first preset data volume is the total data volume of all data stored in the distributed storage system;

The data volume of the target data is greater than the second preset data volume.

Optionally, the device further includes:

An update module, configured to change the fault state from the first fault state to the first fault state if the at least one storage node is not all repaired within the first preset time period when the fault state is the first fault state The state is updated to a second failure state, where the second failure state is used to indicate that the at least one storage node cannot be fully repaired within the first preset time period.

Optionally, the second determining unit is configured to perform the above step 602 .

Optionally, the second determining unit is configured to determine the failure scenario as the first failure scenario when the at least one storage node fails within the target duration, otherwise, determine the failure scenario as the failure scenario. A second failure scenario, where the first failure scenario is used to indicate that the at least one storage node fails simultaneously, and the second failure scenario is used to indicate that the at least one storage node fails at different times.

Optionally, the second determining unit includes:

a first determination subunit, configured to perform the above step 604;

The second determination subunit is used to perform the above step 605 .

Optionally, the first determination subunit is used for:

When the fault type of each storage node in the at least one storage node is the first fault type, the fault state is determined as the first fault state, and the first fault type is used to indicate the fault energy of one storage node Repaired within the second preset time period, the first fault state is used to indicate that the at least one storage node can be completely repaired within the first preset time period;

When the failure type of a target number of storage nodes in the at least one storage node is the second failure type, and if the target number is less than or equal to the redundancy of the distributed storage system, determine the failure state is the first fault state, otherwise, the fault state is determined to be the second fault state, and the second fault type is used to indicate that the fault of one storage node cannot be repaired within the second preset time period, so The second fault state is used to indicate that the at least one storage node cannot be completely repaired within the first preset time period.

Optionally, the second determination subunit is used for:

When the fault type of the first storage node is the first fault type, the fault state is determined to be the first fault state, and the first fault type is used to indicate that the fault of one storage node can be being repaired within a preset time period, and the first fault state is used to indicate that the at least one storage node can be completely repaired within the first preset time period;

When the fault type of the first storage node is the second fault type, the fault state is determined to be the second fault state, and the second fault type is used to indicate that the fault of one storage node cannot be repaired within a preset time period, and the second fault state is used to indicate that the at least one storage node cannot be fully repaired within the first preset time period.

Optionally, the determining module 701 is further configured to perform step 603 .

Optionally, the sending module 702 is further configured to perform the above step 509 .

8 is a schematic structural diagram of a fault processing apparatus provided by an embodiment of the present invention, where the distributed storage system includes a plurality of storage nodes; the apparatus includes:

A sending module 801, configured to perform the above step 608;

A receiving module 802, configured to receive a response returned by the target storage node; the response includes a fault status of the distributed storage system; the fault status is used to indicate whether at least one faulty storage node can All have been fixed within the preset duration.

Optionally, the device also includes:

The processing module is configured to perform fault processing based on the fault status included in the response.

Optionally, the fault identifier of the fault state includes any one of a first fault identifier or a second fault identifier, wherein the first fault identifier is used to indicate the first fault state, and the second fault identifier is used to Indicates a second failure state, the first failure state is used to indicate that at least one storage node can be fully repaired within a first preset time period, and the second failure state is used to indicate that the at least one storage node cannot operate in the All are repaired within the first preset time period, and the storage node is a faulty storage node in the distributed storage system.

Optionally, the processing module is used to:

When the access request is sent by the target client in the distributed storage system based on the target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is the first fault state, no request is sent to the target virtual machine. The target virtual machine responds to the access request, and the first fault state is used to indicate that at least one storage node can be completely repaired within a first preset time period;

When the access request is sent by the target client based on the target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is the second fault state, return to the target virtual machine An abnormal message is stored, and the second fault state is used to indicate that the at least one storage node cannot be completely repaired within the first preset time period.

Optionally, the processing module is used to:

When the target access request is sent by the target client in the distributed storage system based on the target virtual machine, and the target virtual machine is not a VMWare virtual machine, if the fault state is the first fault state, send the request to the The target virtual machine sends a retry request, where the retry request is used to instruct to re-issue the access request, and the first fault state is used to indicate that at least one storage node can be completely repaired within a first preset time period ;

When the target access request is sent by the target client based on the target virtual machine, and the target virtual machine is not a VMWare virtual machine, if the fault state is the second fault state, return to the target virtual machine A target error identifiable by the target virtual machine, where the target error is used to indicate a storage medium failure, and the second failure state is used to indicate that the at least one storage node cannot be completely repaired within the first preset time period .

Optionally, the receiving module 802 is further configured to receive a target access request sent by a target client in the distributed storage system, where the target access request is used to instruct the processing of the first target data, and the first target data is processed. a target data includes the target data;

The sending module 801 is configured to send the access request to the target storage node in the distributed storage system based on the target access request.

Optionally, the receiving module 802 is configured to receive a repair completion response returned by the target storage node, where the repair completion response is used to indicate that there is no faulty device in the distributed storage system.

The embodiment of the present invention also provides a distributed storage system, the distributed storage system includes a supervisory node and a plurality of storage nodes;

The supervisory node is used to:

A fault state of the distributed storage system is determined according to at least one faulty storage node among the plurality of storage nodes; the fault state is used to indicate whether the at least one faulty storage node can All repaired in time;

sending the fault status to each of the plurality of storage nodes;

Each storage node in the plurality of storage nodes is configured to receive the fault status.

Optionally, the fault identifier of the fault state includes any one of a first fault identifier and a second fault identifier, the first fault identifier is used to indicate the first fault state, and the second fault identifier is used for Indicates a second fault state, the first fault state is used to indicate that the at least one storage node can be completely repaired within the first preset time period, and the second fault state is used to indicate that the at least one storage node is All cannot be repaired within the first preset time period.

Optionally, each storage node in the plurality of storage nodes is further configured to suspend the access request if the access request is received again after receiving the fault identifier, and based on the received fault status, Troubleshoot.

It should be noted that each device in the distributed storage system provided above may be the device in Embodiments 5 and 6.

All the above-mentioned optional technical solutions can be combined arbitrarily to form optional embodiments of the present disclosure, which will not be repeated here.

It should be noted that when the fault handling device provided in the above-mentioned embodiments handles faults, only the division of the above-mentioned functional modules is used as an example for illustration. The internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the embodiments of the method for handling faults in a distributed storage system provided by the above embodiments belong to the same concept, and the specific implementation process thereof is detailed in the method embodiments, which will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, etc.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection of the present invention. within the range.

Claims (19)

1. A distributed storage system fault processing method is characterized in that the distributed storage system comprises a plurality of storage nodes; the method comprises the following steps:

determining at least one storage node with a fault in the distributed storage system and target data which cannot be accessed in the at least one storage node;

determining a fault state of the distributed storage system according to the at least one storage node and the target data; the fault state is used for indicating whether the at least one storage node can be completely repaired within a first preset time length;

transmitting the fault status to each of the plurality of storage nodes.

2. The method of claim 1, further comprising:

and carrying out fault processing according to the fault state of the storage system.

3. The method of claim 1, wherein determining the fault condition based on the at least one storage node and the target data comprises:

when the number of the at least one storage node is greater than the redundancy of the distributed storage system and the data volume of the target data meets a preset condition, determining the fault state as a first fault state, wherein the first fault state is used for indicating that the at least one storage node can be completely repaired within the first preset time.

4. The method of claim 1, wherein determining the fault condition based on the at least one storage node and the target data comprises:

when the number of the at least one storage node is larger than the redundancy of the distributed storage system and the data volume of the target data meets a preset condition, determining the fault state according to a fault scene of the distributed storage system, wherein the fault scene is used for indicating whether the at least one storage node simultaneously breaks down.

5. A distributed storage system fault processing method is characterized in that the distributed storage system comprises a plurality of storage nodes; the method comprises the following steps:

sending an access request to a target storage node in the distributed storage system;

receiving a response returned by the target storage node; the response comprises a fault state of the distributed storage system, wherein the fault state is determined according to at least one storage node with a fault in the distributed storage system and target data which cannot be accessed in the at least one storage node; the fault state is used for indicating whether at least one storage node can be completely repaired within a first preset time length.

6. The method of claim 5, wherein after receiving the response returned by the target storage node, the method further comprises:

and processing the fault based on the fault state contained in the response.

7. The method of claim 6, wherein performing fault handling based on the fault status included in the response comprises:

when the access request is sent by a target client in the distributed storage system based on a target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is a first fault state, the access request is not responded to the target virtual machine, and the first fault state is used for indicating that at least one storage node can be completely repaired within a first preset time length;

when the access request is sent by the target client based on a target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is a second fault state, returning a message of abnormal storage to the target virtual machine, wherein the second fault state is used for indicating that the at least one storage node cannot be completely repaired within the first preset time.

8. The method of claim 6, wherein performing fault handling based on the fault status included in the response comprises:

when the access request is sent by a target client in the distributed storage system based on a target virtual machine and the target virtual machine is not a VMWare virtual machine, if the fault state is a first fault state, sending a retry request to the target virtual machine, wherein the retry request is used for indicating to resend the access request, and the first fault state is used for indicating that at least one storage node can be completely repaired within a first preset time length;

when the access request is sent by the target client based on a target virtual machine and the target virtual machine is not a VMWare virtual machine, if the fault state is a second fault state, returning a target error which can be identified by the target virtual machine to the target virtual machine, wherein the target error is used for indicating a storage medium fault, and the second fault state is used for indicating that the at least one storage node cannot be completely repaired within the first preset time length.

9. A distributed storage system, comprising a supervisory node and a plurality of storage nodes;

the supervisory node is configured to:

determining at least one storage node with a fault in the distributed storage system and target data which cannot be accessed in the at least one storage node;

determining a fault state of the distributed storage system according to the at least one storage node and the target data; the fault state is used for indicating whether the at least one storage node can be completely repaired within a first preset time length;

transmitting the fault status to each of the plurality of storage nodes;

each storage node of the plurality of storage nodes to receive the fault condition.

10. The system of claim 9, wherein each of the plurality of storage nodes is further configured to suspend the access request if an access request is received after receiving the failure status, and perform failure handling based on the received failure status.

11. A fault handling apparatus, applied to a distributed storage system including a plurality of storage nodes, the apparatus comprising a determining module and a sending module:

wherein the determination module comprises a first determination unit and a second determination unit;

the first determining unit is configured to determine at least one storage node in the distributed storage system that has a failure and target data that cannot be accessed in the at least one storage node;

the second determining unit is configured to determine a fault state of the distributed storage system according to the at least one storage node and the target data; the fault state is used for indicating whether the at least one storage node can be completely repaired within a first preset time length;

the sending module is configured to send the failure status to each of the plurality of storage nodes.

12. The apparatus of claim 11, wherein the second determining unit is configured to:

when the number of the at least one storage node is greater than the redundancy of the distributed storage system and the data volume of the target data meets a preset condition, determining the fault state as a first fault state, wherein the first fault state is used for indicating that the at least one storage node can be completely repaired within the first preset time.

13. The apparatus of claim 11, wherein the second determining unit is configured to:

when the number of the at least one storage node is larger than the redundancy of the distributed storage system and the data volume of the target data meets a preset condition, determining the fault state according to a fault scene of the distributed storage system, wherein the fault scene is used for indicating whether the at least one storage node simultaneously breaks down.

14. A fault handling apparatus, applied to a distributed storage system, the distributed storage system including a plurality of storage nodes, the apparatus comprising:

a sending module, configured to send an access request to a target storage node in the distributed storage system, where the distributed storage system includes a plurality of storage nodes;

the receiving module is used for receiving a response returned by the target storage node; the response comprises a fault state of the distributed storage system, wherein the fault state is determined according to at least one storage node with a fault in the distributed storage system and target data which cannot be accessed in the at least one storage node; the fault state is used for indicating whether at least one storage node can be completely repaired within a first preset time length.

15. The apparatus of claim 14, further comprising:

and the processing module is used for processing the fault based on the fault state contained in the response.

16. The apparatus of claim 15, wherein the processing module is configured to:

when the access request is sent by a target client in the distributed storage system based on a target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is a first fault state, the access request is not responded to the target virtual machine, and the first fault state is used for indicating that at least one storage node can be completely repaired within a first preset time length;

when the access request is sent by the target client based on a target virtual machine, and the target virtual machine is a VMWare virtual machine, if the fault state is a second fault state, returning a message of abnormal storage to the target virtual machine, wherein the second fault state is used for indicating that the at least one storage node cannot be completely repaired within the first preset time.

17. The apparatus of claim 15, wherein the processing module is configured to:

when the access request is sent by a target client in the distributed storage system based on a target virtual machine and the target virtual machine is not a VMWare virtual machine, if the fault state is a first fault state, sending a retry request to the target virtual machine, wherein the retry request is used for indicating to resend the access request, and the first fault state is used for indicating that at least one storage node can be completely repaired within a first preset time length;

when the access request is sent by the target client based on a target virtual machine and the target virtual machine is not a VMWare virtual machine, if the fault state is a second fault state, returning a target error which can be identified by the target virtual machine to the target virtual machine, wherein the target error is used for indicating a storage medium fault, and the second fault state is used for indicating that the at least one storage node cannot be completely repaired within the first preset time length.

18. A computer device comprising a processor and a memory, the memory having stored therein at least one instruction that is loaded and executed by the processor to perform operations performed by the distributed storage system fault handling method of any of claims 1 to 8.

19. A storage medium having stored therein at least one instruction that is loaded and executed by a processor to perform an operation performed by the distributed storage system fault handling method of any one of claims 1 to 8.

Images