HK40036252B - Method, device and system for detecting fault, apparatus and storage medium - Google Patents

Description

A fault detection method, apparatus, system, device, and storage medium

Technical Field

This application relates to the field of communication technology, and in particular to a fault detection method, apparatus, system, device and storage medium.

Background Technology

As the first line of defense for external business requests entering the internal business network system, the gateway plays a crucial role in the continuous access of internet services. With the rapid development and widespread adoption of internet services, the business traffic passing through the gateway is typically enormous (e.g., exceeding 10T). In this context, even minor and transient quality fluctuations can have immeasurable impacts on internet services.

As the essential pathway for external business requests to reach internal business servers, the gateway is a fundamental network infrastructure and typically has the following characteristics: 1) High business sensitivity: For sensitive internet services, even a 1-2 minute failure is unacceptable, and faults must be detected and handled within seconds; 2) Difficult fault location: In general, a problem in any part of the gateway system can affect the entire network link. However, due to the complexity of the network environment and the gateway system architecture, it is often difficult to accurately pinpoint the problematic link after detecting an impact on the network link.

It is evident that how to quickly and accurately locate faults in a gateway system has become an urgent problem to be solved.

Summary of the Invention

This application provides a fault detection method, apparatus, system, device, and storage medium that can quickly and accurately detect whether a gateway server is faulty.

In view of the above, the first aspect of this application provides a fault detection method, the method comprising:

During the fault detection period, the network devices that receive the probe request packet report the logging information they receive; the logging information is the record information generated by the network devices when they operate on the probe request packet; the network devices include at least one of a client, a gateway server, and a service server.

The marker information containing the same request packet identifier is used as the marker information of the probe request packet corresponding to the request packet identifier;

Based on the tracking information of the probe request packet, the network device is used to detect gateway faults.

A second aspect of this application provides a fault detection device, the device comprising:

The information acquisition module is used to receive, during the fault detection period, the tracking information reported by each network device through which the probe request packet passes; the tracking information is the record information generated by the network device when it operates on the probe request packet; the network device includes at least one of a client, a gateway server, and a service server;

The information acquisition module is further configured to use the marker information containing the same request packet identifier as the marker information of the probe request packet corresponding to the request packet identifier;

The fault detection module is used to perform gateway fault detection on the network device based on the logging information of the probe request packet.

A third aspect of this application provides a fault detection system, the system comprising: a client, a gateway server, and a fault detection server;

The client is configured to generate tracking information based on its packet sending and receiving operations for probe request packets, and report the tracking information to the fault detection server.

The gateway server is configured to generate tracking information based on its packet sending and receiving operations for the probe request packet, and report the tracking information to the fault detection server.

The fault detection server is used to execute the fault detection method as described in the first aspect above.

A fourth aspect of this application provides an apparatus comprising a processor and a memory:

The memory is used to store computer programs;

The processor is configured to perform the steps of the fault detection method as described in the first aspect above, according to the computer program.

The fifth aspect of this application provides a computer-readable storage medium for storing a computer program for performing the steps of the fault detection method described in the first aspect.

A sixth aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the fault detection method described in the first aspect.

As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

This application provides a fault detection method that draws on medical techniques of using isotope or fluorescent staining to locate the cause of a disease. It performs end-to-end staining and marking of probe request packets passing through network devices on the Internet. Specifically, it utilizes at least one of the client, gateway server, and business server to generate marking information for the probe request packets based on their operations on the packets. Then, based on the marking information of the probe request packets acquired within a fault detection period, gateway fault detection is performed on the network devices to accurately detect whether a fault exists and, if a fault is detected, to locate the fault location and cause. Thus, based on the transmission link path of the probe request packets, gateway fault detection and location are achieved in complex network environments.

Attached Figure Description

Figure 1 is a schematic diagram of the working architecture of the fault detection system provided in an embodiment of this application;

Figure 2 is a schematic diagram illustrating the principle of recording dot information provided in an embodiment of this application;

Figure 3 is a flowchart illustrating the fault detection method provided in an embodiment of this application;

Figure 4 is a schematic diagram of an exemplary network architecture provided in an embodiment of this application;

Figure 5 is a schematic diagram of the forwarding process of the main forwarding program in the gateway server provided in the embodiment of this application;

Figure 6 is a schematic diagram illustrating the principle of communication between the control client and the business server provided in an embodiment of this application;

Figure 7 is a schematic diagram of the working principle of the staining system provided in the embodiment of this application;

Figure 8 is a schematic diagram of the log forwarding program and log analysis program provided in the embodiments of this application;

Figure 9 is a performance curve of a gateway cluster provided in an embodiment of this application;

Figure 10 is a performance curve of another gateway cluster provided in an embodiment of this application;

Figure 11 is a schematic diagram of the structure of the first type of fault detection device provided in the embodiment of this application;

Figure 12 is a schematic diagram of the structure of the second type of fault detection device provided in the embodiment of this application;

Figure 13 is a schematic diagram of the structure of the third type of fault detection device provided in the embodiment of this application;

Figure 14 is a schematic diagram of the fourth type of fault detection device provided in the embodiments of this application;

Figure 15 is a schematic diagram of the server structure provided in an embodiment of this application.

Detailed Implementation

To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

To address the problem of how to quickly and accurately detect gateway server faults, this application provides a fault detection method. This method draws inspiration from medical techniques that use isotope or fluorescent staining to locate the cause of a disease. It involves staining and marking probe request packets passing through network devices on the Internet along the entire transmission chain. Specifically, at least one of the client, gateway server, and business server generates marking information for the probe request packets based on their operations on the packets. Then, based on the marking information of the probe request packets acquired within the fault detection period, gateway fault detection is performed on the network devices to accurately detect whether a fault exists. If a fault is detected, the location and cause of the fault are identified. Thus, based on the transmission path of the probe request packets, gateway fault detection and location are achieved in complex network environments.

It should be noted that the fault detection method provided in this application is generally applied to servers with data processing capabilities. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. This application does not impose any restrictions on these services.

To facilitate understanding of the fault detection method provided in the embodiments of this application, the fault detection system applied by the fault detection method provided in the embodiments of this application will be introduced below in conjunction with the working architecture of the gateway server.

Referring to Figure 1, Figure 1 is a schematic diagram of the working architecture of the fault detection system provided in the embodiment of this application. As shown in Figure 1, the fault detection system provided in the embodiment of this application includes a client 110, a gateway server 120, and a fault detection server 130. This fault detection system is typically deployed in an Internet communication architecture, which also includes a business server 140, an external network switch 150, and an internal network switch 160.

In this application, client 110 generates a first request packet and sends it to business server 140 via gateway server 120. During this process, client 110 can generate tracking information based on its packet sending operation for the first request packet and upload the generated tracking information to fault detection server 130. In practical applications, client 110 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. This application does not impose any limitations on client 110.

In this system, gateway server 120 receives a first request packet sent by client 110, processes the packet accordingly, and then sends it to business server 140. During this process, gateway server 120 can generate tracking information based on its packet sending and/or receiving operations for the first request packet, and upload the generated tracking information to fault detection server 130. In practical applications, gateway server 120 can be any gateway server in a gateway cluster; that is, gateway server 120 can be deployed in a gateway cluster that includes multiple gateway servers 120.

The business server 140 is used to receive the first request packet forwarded by the gateway server 120, generate a second request packet corresponding to the first request packet, and return the second request packet to the client 110 through the gateway server 120.

The gateway server 120 is also used to receive the second request packet sent by the business server 140, process the second request packet accordingly, and send it to the client 110. During this process, the gateway server 120 can generate the tracking information based on its packet sending and receiving operations for the second request packet, and upload the generated tracking information to the fault detection server 130.

The client 110 is also used to receive the second request packet sent by the gateway server 120, and generate point information based on its packet receiving operation of the second request packet, and upload the generated point information to the fault detection server 130.

It should be noted that in the technical solutions provided in the embodiments of this application, the first request packet and the second request packet mentioned above both belong to the probe request packets in the embodiments of this application. After the service server 140 receives the first request packet sent by the client through the gateway server 120, it generates a second request packet for the first request packet and sends the second request packet back to the client through the gateway server 120. In this process, the first request packet and the second request packet actually belong to the same probe request packet, that is, the first request packet and the second request packet include the same request packet identifier. For example, the first request packet and the second request packet belonging to the same probe request packet may include the same quadruple identifier composed of source IP address, source port, destination IP address and destination port.

The fault detection server 130 is used to execute the fault detection method provided in this application embodiment. During the fault detection period, it receives the tracking information reported by each of the network devices (including at least one of client 110, gateway server 120, and service server 140) through which the probe request packet passes, and uses the tracking information containing the same request packet identifier as the tracking information of the probe request packet corresponding to the request packet identifier. Then, it performs fault detection on the gateway server 120 in the gateway cluster according to the obtained tracking information of the probe request packet. The specific process of fault detection performed by the fault detection server 130 will be described in detail in the method embodiment below.

The external network switch 150 is used to forward request packets from the external network (such as the first request packet sent by client 110) to the gateway server 120, and to forward request packets from the gateway server 120 to client 110. The internal network switch 160 is used to forward request packets from the internal network (such as the second request packet sent by business server 140) to the gateway server 120, and to forward request packets from the gateway server 120 to business server 140.

As shown in Figure 1, in practical applications, the first request packet sent by client 110 arrives at external network switch 150 after network transmission. External network switch 150 can send the first request packet to a gateway server 120 in the gateway cluster. After receiving the first request packet, gateway server 120 selects a business server 140 according to a certain selection strategy and sends the first request packet, which is encapsulated by Generic Routing Encapsulation (GRE) protocol, to internal network switch 160. Internal network switch 160 then sends the first request packet to business server 140.

After receiving the first request packet, the business server 140 generates a second request packet corresponding to the first request packet and sends the second request packet to the intranet switch 160. The intranet switch 160 then sends the second request packet to a gateway server 120 in the gateway cluster. After receiving the second request packet, the gateway server 120 disassembles the GRE header encapsulated by the GRE protocol and then returns the disassembled second request packet to the client 110 through the external network switch 150.

During the request packet transmission process shown in Figure 1, client 110 and gateway server 120 can generate corresponding marker information for the probe request packet. Specifically, as shown in Figure 2, after client 110 sends the first request packet, it can record the first marker information TP1 corresponding to the probe request packet, which can also be understood as recording the first point of a colored link; when gateway server 120 receives the first request packet, it can record the second marker information TP2 corresponding to the probe request packet, which can also be understood as recording the second point of a colored link; when gateway server 120 completes the processing of the first request packet and sends it to business server 140, it can record the third marker information TP3 corresponding to the probe request packet, which can also be understood as recording the third point on ... After receiving the second request packet returned by the business server 140 in response to the first request packet, the server 120 can record the fourth marker information TP4 corresponding to the probe request packet, which can also be understood as the fourth point on a colored link. After the gateway server 120 completes the processing of the second request packet and sends it to the client 110, it can record the fifth marker information TP5 corresponding to the probe request packet, which can also be understood as the fifth point on a colored link. After the client 110 receives the second request packet, it can record the sixth marker information TP6 corresponding to the probe request packet, which can also be understood as the sixth point on a colored link.

After the client 110 and the gateway server 120 complete the recording of the logging information corresponding to the probe request packet, they need to send the logging information corresponding to the probe request packet they recorded to the fault detection server 130 so that the fault detection server 130 can detect whether the gateway server 120 in the gateway cluster has a fault based on this logging information, and locate the cause of the fault in the gateway server 120 if a fault is detected.

It should be understood that, under normal circumstances, in order to accurately detect whether the gateway server 120 is faulty, the client 110 and the business server 140 need to exchange a large number of different probe request packets during the fault detection period. Accordingly, the fault detection server 130 can obtain the tracking information recorded by the client 110 and the gateway server 120 for these probe request packets, and perform fault detection on the gateway server 120 based on the tracking information corresponding to each of the large number of probe request packets.

It should be noted that in practical applications, in addition to the client 110 and gateway server 120 reporting the tracking information of the probe request packet to the fault detection server 130, the gateway server 120 can also report the tracking information of the probe request packet to the fault detection server 130 on its own. Alternatively, the client 110, gateway server 120 and business server 140 can also collaboratively report the tracking information of the probe request packet to the fault detection server 130. This application does not impose any restrictions on the source of the tracking information received by the fault detection server 130.

The fault detection method provided in this application will be described in detail below through method embodiments.

Referring to Figure 3, which is a flowchart illustrating the fault detection method provided in this embodiment, the following embodiment uses a fault detection server as the executing entity. As shown in Figure 3, the fault detection method includes the following steps:

Step 301: During the fault detection period, receive the tracking information reported by each network device through which the probe request packet passes; the tracking information is the record information generated by the network device when it operates on the probe request packet; the network device includes at least one of a client, a gateway server, and a service server.

Step 302: Use the logging information containing the same request packet identifier as the logging information of the probe request packet corresponding to the request packet identifier.

Since steps 301 and 302 are closely related, the following text will integrate steps 301 and 302 and introduce the overall implementation process of steps 301 and 302.

During the fault detection period, the client can interact with the business server through the gateway server to exchange a large number of probe request packets. During the transmission of each probe request packet, the gateway server can generate the corresponding point information for the probe request packet based on its own packet sending and/or packet receiving operations, and then send the generated point information to the fault detection server.

To ensure that the fault detection server can obtain a more complete transmission link path corresponding to the probe request packet, in practical applications, the client used to send the probe request packet can also generate the corresponding tracking information based on its own packet sending and receiving operations, and send the generated tracking information to the fault detection server.

It should be noted that the tracking information typically includes a request packet identifier. This identifier corresponds to the probe request packet used to generate the tracking information; that is, the request packet identifier contained in the tracking information is actually the request packet identifier corresponding to the probe request packet. Tracking information corresponding to the same probe request packet should contain the same request packet identifier, while tracking information corresponding to different probe request packets should contain different request packet identifiers. This allows the fault detection server to identify the specific probe request packet corresponding to a tracking information based on the request packet identifier, thereby facilitating the server to determine all tracking information corresponding to a probe request packet and, based on this, determine the transmission path of that probe request packet.

Since the various markers corresponding to a probe request packet can usually identify the transmission link path corresponding to that probe request packet, the fault detection server can detect whether the gateway server is faulty by analyzing the transmission link paths corresponding to a large number of probe request packets. This method of using marker information to identify the transmission link path corresponding to the probe request packet is similar to the medical method of using isotopes or fluorescent staining to locate the cause of disease. Therefore, the method of locating gateway server faults based on marker information in this application can also be called staining.

It should be noted that the above-mentioned packet sending operation for the probe request packet specifically includes: sending the first request packet and sending the second request packet; the above-mentioned packet receiving operation for the probe request packet specifically includes: receiving the first request packet and receiving the second request packet. The first request packet is a request packet sent by the client to the business server through the gateway server. The second request packet is a request packet generated by the business server in response to the received first request packet. The business server needs to send the second request packet back to the client through the gateway server. The second request packet and the first request packet have a corresponding relationship, and the corresponding first and second request packets belong to the same probe request packet, containing the same request packet identifier.

More specifically, for the gateway server, it needs to generate a logging record corresponding to a probe request packet based on its operation of receiving the first request packet sent by the client, generating another logging record corresponding to a probe request packet based on its operation of sending the first request packet to the business server, generating another logging record corresponding to a probe request packet based on its operation of receiving the second request packet sent by the business server, and generating yet another logging record corresponding to a probe request packet based on its operation of sending the second request packet to the client. For the client, it needs to generate a logging record corresponding to a probe request packet based on its operation of sending the first request packet, and also generate another logging record corresponding to a probe request packet based on its operation of receiving the second request packet.

Assuming the gateway server is functioning correctly, for a probe request packet, the fault detection server should receive at least six tagged information entries corresponding to that packet. These six entries should represent the complete transmission link path of the probe request packet. Conversely, if the fault detection server does not receive six tagged information entries for a probe request packet, it indicates that the packet was lost during transmission, further suggesting a potential fault in the gateway server.

For example, as shown in Figure 2, this application defines six nodes that record and upload tracking information: TP_CLIENT_SEND (TP1), TP_LD_FRONT_RCV (TP2), TP_LD_FRONT_SND (TP3), TP_LD_BACK_RCV (TP4), TP_LD_BACK_RCV (TP5), and TP_CLIENT_RCV (TP6). The "inbound" direction refers to the direction of transmitting the first request packet, and the "outbound" direction refers to the direction of transmitting the second request packet corresponding to the first request packet.

Specifically, after the client sends the first request packet, it can record the first trace point corresponding to the probe request packet (trace_point = TP1), which can also be understood as recording the first point of a colored link. When the gateway server receives the first request packet, it can record the second trace point corresponding to the probe request packet (trace_point = TP2), which can also be understood as recording the second point of a colored link. After the gateway server completes the processing of the first request packet and sends it to the business server, it can record the third trace point corresponding to the probe request packet (trace_point = TP3), which can also be understood as recording the third point on a colored link. After the server receives the second request packet returned by the business server in response to the first request packet, it can record the fourth trace point information corresponding to the probe request packet, trace_point=TP4, which can also be understood as the fourth point on a colored link. When the gateway server completes the processing of the second request packet and sends it to the client, it can record the fifth trace point information corresponding to the probe request packet, trace_point=TP5, which can also be understood as the fifth point on a colored link. When the client receives the second request packet, it can record the sixth trace point information corresponding to the probe request packet, trace_point=TP6, which can also be understood as the sixth point on a colored link.

It should be understood that in practical applications, to ensure that the fault detection server can obtain a more complete transmission link path corresponding to the probe request packet, the business server can also generate tracking information based on its own packet sending and/or receiving operations for the probe request packet, and report the generated tracking information to the fault detection server. Specifically, after receiving the first request packet from the client forwarded by the gateway server, the business server can generate a corresponding tracking information and report it to the fault detection server. After the business server generates a second request packet for the first request packet and sends the second request packet to the gateway server (so that the gateway server can forward the second request packet to the client), it can generate a corresponding tracking information and report it to the fault detection server. In this scenario, if the gateway server is not faulty, for a probe request packet (including a first request packet and a second request packet with a corresponding relationship), the fault detection server should receive eight tracking information corresponding to the probe request packet. These eight tracking information can represent the complete transmission link path corresponding to the probe request packet.

Optionally, in order for the fault detection server to more accurately analyze whether the gateway server is faulty based on the transmission link path corresponding to the probe request packet, it is usually necessary to ensure that the multiple probe request packets sent by the client within a fault detection cycle are different, so as to avoid the fault detection server being confused by the tracking information it receives and not knowing the specific probe request packet it corresponds to.

Specifically, a quadruple identifier can be used to uniquely represent a probe request packet. That is, a quadruple identifier is used as the request packet identifier for each logging information corresponding to the same probe request packet. The quadruple identifier includes the source IP address cip, the source port cport, the destination IP address vip, and the destination port vport. Accordingly, within a fault detection cycle, the probe request packets with different quadruple identifiers can be controlled to interact between the client and the business server. Thus, within the fault detection cycle, the fault detection server will receive multiple logging information containing different quadruple identifiers.

The (cip, cport, vip, vport) quadruple can be used as a unique identifier for probe request packets, which are then transmitted to the gateway server and the business server via the Transmission Control Protocol (TCP). In actual design, since there are more than 60,000 cports available, and the client can probe every 5 seconds for a specific business rule (vip, vport), the probe request packets using the above quadruple identifier will usually only be repeatedly interacted with for several days (i.e., probe request packets corresponding to the same identifier quadruple). Since a fault detection cycle is usually on the order of minutes, this ensures that the fault detection server can obtain the marking information corresponding to probe request packets with different quadruple identifiers within a fault detection cycle, thus achieving the purpose of coloring.

Step 302: Based on the logging information of the probe request packet, perform gateway fault detection on the network device.

After obtaining the tracking information corresponding to the probe request packet, the fault detection server can perform multi-dimensional fault detection on the gateway server based on the obtained tracking information, such as detecting whether the gateway server is faulty, whether the gateway server has jittery packet loss faults, whether the gateway server has abnormal business rules, etc.

In some embodiments, the fault detection server can detect whether the gateway server in the gateway cluster is faulty based on the tracking information. That is, when the forwarding success rate of the gateway cluster drops and an alarm is triggered, the fault detection server can locate the faulty gateway server in the gateway cluster based on the tracking information it has obtained.

Specifically, the logging information generated by the gateway server based on its packet reception operations related to probe request packets can be divided into front-end packet reception information and back-end packet reception information. Front-end packet reception information is the record information generated by the gateway server when it completes the packet reception operation for the first request packet, while back-end packet reception information is the record information generated by the gateway server when it completes the packet reception operation for the second request packet. The first request packet is the request packet sent by the client to the business server through the gateway server, and the second request packet is the request packet sent back to the client by the business server after receiving the first request packet. Both the first and second request packets contain the same request packet identifier. Furthermore, the fault detection server can, for each gateway server in the gateway cluster, count the number of front-end packets received and the number of back-end packets received for that gateway server based on the front-end and back-end packet reception information uploaded by that gateway server, and determine the faulty gateway server in the gateway cluster based on the front-end and back-end packet reception counts of each gateway server in the cluster.

For example, when the client and gateway server need to upload tracking information (TP1 to TP6) to the fault detection server, TP2 belongs to front-end packet reception information, and TP4 belongs to back-end packet reception information. For each gateway server in the gateway cluster, the fault detection server needs to count the number of TP2 uploaded by that gateway server (i.e., the number of front-end packets received) and the number of TP4 uploaded by that gateway server (i.e., the number of back-end packets received); then, based on the number of TP2 and TP4 uploaded by each gateway server in the gateway cluster, the faulty gateway server in the gateway cluster is determined.

In one possible implementation, the fault detection server can determine the faulty gateway server based on whether the number of packets received at the front end and/or the number of packets received at the back end of the gateway server has reached its limit. That is, the fault detection server can check whether the number of packets received at the front end and the number of packets received at the back end of each gateway server in the gateway cluster has reached its limit. If any one or more of the number of packets received at the front end and the number of packets received at the back end have reached their limit, then the gateway server can be determined to be a faulty gateway server.

Specifically, the fault detection server can plot a front-end packet reception count curve for the gateway cluster based on the front-end packet reception count of each gateway server in the cluster. Then, based on this curve, it can determine if any gateway server in the cluster has a front-end packet reception count that has reached its lowest point. If so, the gateway server with the lowest front-end packet reception count is directly identified as a faulty gateway server. Similarly, the fault detection server can plot a back-end packet reception count curve for the gateway cluster based on the back-end packet reception count of each gateway server in the cluster. Then, based on this curve, it can determine if any gateway server in the cluster has a back-end packet reception count that has reached its lowest point. If so, the gateway server with the lowest back-end packet reception count is directly identified as a faulty gateway server.

In another possible implementation, the fault detection server can determine a faulty gateway server by judging whether the number of front-end packets received and/or the number of back-end packets received by the gateway server is significantly lower than the number of front-end packets received and/or the number of back-end packets received by other gateway servers in the gateway cluster. Specifically, the fault detection server can determine the average front-end packet reception threshold and the average back-end packet reception threshold based on the number of front-end packets received and the number of back-end packets received by each gateway server in the gateway cluster. Then, for each gateway server in the gateway cluster, it determines a first difference between the average front-end packet reception threshold and the number of front-end packets received by that gateway server, and determines whether the first difference exceeds a preset front-end difference threshold. If so, the gateway server is determined to be a faulty gateway server. Furthermore, for each gateway server in the gateway cluster, it determines a second difference between the average back-end packet reception threshold and the number of back-end packets received by that gateway server, and determines whether the second difference exceeds a preset back-end difference threshold. If so, the gateway server is determined to be a faulty gateway server.

Specifically, the fault detection server can plot a front-end packet reception curve for the gateway cluster based on the number of packets received by each gateway server in the cluster. This front-end packet reception curve reflects the average front-end packet reception threshold of the gateway cluster. Furthermore, the fault detection server can use the front-end packet reception curve to determine whether there are gateway servers whose front-end packet reception count is significantly lower than that of other gateway servers. In other words, it can determine whether there are gateway servers whose difference between their front-end packet reception count and the average front-end packet reception threshold exceeds a preset front-end difference threshold. If such a gateway server exists, it can be directly identified as a faulty gateway server.

Similarly, the fault detection server can plot a backend packet reception curve for the gateway cluster based on the number of packets received by each gateway server in the gateway cluster. This backend packet reception curve can reflect the average backend packet reception threshold of the gateway cluster. Furthermore, the fault detection server can use the backend packet reception curve for the gateway cluster to determine whether there are gateway servers whose backend packet reception count is significantly lower than that of other gateway servers. In other words, it can determine whether there are gateway servers whose difference between their backend packet reception count and the average backend packet reception threshold exceeds a preset backend difference threshold. If such a gateway server exists, it can be directly identified as a faulty gateway server.

Thus, when an alarm indicating a drop in the forwarding success rate of the gateway cluster is detected, the faulty gateway server in the gateway cluster can be quickly and accurately located based on the tracking information uploaded by the gateway server (i.e., front-end packet reception information and back-end packet reception information). This means locating the gateway server in the gateway cluster with a forwarding performance failure. Compared to the existing technology where maintenance personnel manually check the traffic of each gateway server to locate the faulty gateway server, the method provided in this application is faster and more efficient.

In some embodiments, the fault detection server can also detect whether the gateway server has a jitter packet loss fault. Specifically, when a gateway server has a jitter packet loss fault, the number of packets received by each gateway server in the gateway cluster is actually not significantly different. Therefore, the above method alone cannot detect whether the gateway server has a jitter packet loss fault. In order to detect the jitter packet loss fault of the gateway server, this application embodiment also provides a method for detecting jitter packet loss fault.

The following analysis uses the network architecture shown in Figure 4 as an example. Taking the client -> gateway server -> service server direction as an example, the external network switch and multiple gateway servers in the gateway cluster form a neighbor network through the OSPF (Open Shortest Path First) routing protocol. Each device in this neighbor network can obtain an equivalent route. Similarly, in the service server -> gateway server -> client direction, the internal network switch and multiple gateway servers in the gateway cluster also form a neighbor network through the OSPF routing protocol.

In the client-to-gateway-server-to-service-server direction, when the first request packet arrives at the external network switch, the external network switch will distribute the first request packet to each gateway server with approximately equal probability using a certain hash strategy. For example, the hash strategy used could be a binary hash, that is, hashing based on the source IP and destination IP in the first request packet. Similarly, in the service-to-gateway-server-to-client direction, when the second request packet arrives at the internal network switch, the internal network switch will also distribute the second request packet to each gateway server with approximately equal probability using a certain hash strategy.

Based on the above principles, it can be determined that in most neighbor networks based on the OSPF routing protocol, if the neighbor relationships are stable, request packets with the same source IP address and destination IP address will always end up on a fixed gateway server. Therefore, the fault detection server can maintain a mapping relationship in the data analysis program between each IP address combination (including the source IP address and destination IP address) and the front-end gateway server (i.e., the gateway server through which the first request packet passes) and the back-end gateway server (i.e., the gateway server through which the second request packet passes).

That is, the fault detection server can obtain the tracking information corresponding to historical probe request packets; based on the tracking information corresponding to the historical probe request packets, it determines the front-end gateway server and back-end gateway server through which the historical probe request packets pass. The historical probe request packets include a first historical request packet and a second historical request packet. The first historical request packet is a request packet sent by the client to the business server through the front-end gateway server, and the second historical request packet is a request packet sent by the business server to the client through the back-end gateway server after receiving the first historical request packet; then, it constructs a mapping relationship between the IP address combination included in the historical probe request packets and the front-end gateway server and the back-end gateway server. The IP address combination includes the source IP address and the destination IP address.

In other words, the fault detection server can automatically learn the mapping relationship between the IP address combination (cip, vip) included in the probe request packet and the front-end gateway server through which the first request packet passes, as well as the mapping relationship between the IP address combination (cip, vip) and the back-end gateway server through which the second request packet (corresponding to the first request packet) passes, based on the tracking information corresponding to the probe request packets uploaded by the client and the gateway server.

Furthermore, considering that when the business server sends a second request packet to the gateway server, it will encapsulate the second request packet with a GRE header and place it before the network layer of the second request packet (as shown in Figure 4), this results in the internal network switch containing two types of hashes: inner hash (corresponding to request packets without GRE headers) and outer hash (corresponding to request packets with GRE headers). This introduces a new problem: for inner hashes, the internal network switch can directly determine the specific gateway server through the IP address combination included in the request packet; however, for outer hashes, the internal network switch needs to use the IP address combination (rs_ip, tsv) encapsulated in the GRE header for hashing. In this case, for request packets containing the same IP address combination, the backend gateway servers they pass through may be different.

To address the aforementioned issues, it's possible to control the forwarding of the first historical request from the target client to the target business server, establishing a correspondence between the target client and the target business server. Specifically, dedicated probing rules can be established within the gateway cluster, and these rules can be configured for session persistence. When the gateway server forwards the first request packet, it can control the forwarding of first request packets from the same client (including those with the same CIP) to the same business server. In other words, the first request packet from the target client is always forwarded to the target business server, maintaining a correspondence between the target client and the target business server. Since VIP and TSV are fixedly associated, requests with the same (CIP, VIP) can still land on the same backend gateway server.

After establishing the mapping relationship between IP address combinations (CIP, VIP) and front-end and back-end gateway servers, the fault detection server can perform jitter fault detection on the gateway servers in the gateway cluster based on this mapping relationship. Specifically, when the fault detection server determines that the transmission link corresponding to the probe request packet is incomplete based on the tracking information of the probe request, it can determine the destination gateway server corresponding to the probe request packet based on the missing tracking information in the transmission link, the IP address combinations included in the probe request packet, and the aforementioned mapping relationship, and update the packet loss count corresponding to that destination gateway server accordingly. When the packet loss count corresponding to the destination gateway server reaches a preset packet loss threshold, it is determined that the destination gateway server has a jitter packet loss fault.

For example, taking the transmission link shown in Figure 2 as an example, assuming that the trace point information corresponding to the probe request packet is missing trace_point=TP4, the fault detection server can determine that the transmission link corresponding to the probe request packet is incomplete. Since the missing information is the trace point information that the gateway server should upload after receiving the second request packet, it can be determined that the packet loss belongs to the backend gateway server. In this case, the fault detection server can first determine the IP address combination (cip, vip) included in the probe request packet, then determine the mapping relationship including the IP address combination, and determine the backend gateway server in the mapping relationship as the destination gateway server corresponding to the probe request packet. That is, it means that the second request packet should be transmitted to the destination gateway server, but in fact, the destination gateway server did not receive the second request packet, resulting in jitter packet loss. Furthermore, the fault detection server can increment the packet loss count corresponding to the destination gateway server by 1. If the packet loss count of a gateway server in the gateway cluster reaches the preset packet loss count threshold, it can be determined that the gateway server has a jitter packet loss fault.

It should be noted that, because the network environment changes in real time, if a gateway server in the gateway cluster suddenly goes offline, or if a gateway server suddenly comes online, the mapping relationship between the IP address combinations previously learned by the fault detection server and the front-end and back-end gateway servers will change accordingly. In other words, the previously learned mapping relationship can no longer be applied to the current network environment. If this mapping relationship continues to be applied, jitter-induced packet loss and false detections may occur. Therefore, this application also provides a scheme for detecting whether the mapping relationship is valid.

In other words, the fault detection server determines the front-end gateway server and back-end gateway server that the probe request packet passes through based on the tracking information corresponding to the received probe request packet. The probe request packet includes a first request packet and a second request packet. The first request packet is a request packet sent by the client to the business server through the front-end gateway server, and the second request packet is a request packet sent by the business server to the client through the back-end gateway server after receiving the first request packet. Then, based on the target IP address combination included in the probe request packet, the target mapping relationship including the target IP address combination is determined. Furthermore, it is determined whether the front-end gateway server that the probe request packet passes through is consistent with the front-end gateway server in the target mapping relationship, and whether the back-end gateway server that the probe request packet passes through is consistent with the back-end gateway server in the target mapping relationship. If any one or more inconsistencies exist, the target mapping relationship can be determined to be invalid and needs to be updated.

Specifically, after receiving the tracking information corresponding to the probe request packet, the fault detection server can determine the front-end gateway server that the first request packet in the probe request packet passed through, and the back-end gateway server that the second request packet in the probe request packet passed through. Then, based on the target IP address combination (cip, vip) included in the probe request packet, it determines the target mapping relationship including the (cip, vip). Furthermore, it determines whether the front-end gateway server included in the target mapping relationship is consistent with the front-end gateway server that the first request packet actually passed through, and whether the back-end gateway server included in the target mapping relationship is consistent with the back-end gateway server that the second request packet actually passed through. If any of these are inconsistent, it means that the target mapping relationship is no longer applicable to the current network environment. Accordingly, the fault detection server will no longer continue to detect jitter packet loss faults based on the target mapping relationship, but will relearn the mapping relationship based on the tracking information corresponding to the probe request packets received thereafter, until the learned mapping relationship reaches stability, and then detect jitter packet loss faults based on the newly learned mapping relationship.

It should be understood that in practical applications, in order to achieve fault tolerance, the fault detection server can also determine a failure threshold n (n is an integer greater than 1) for the mapping relationship. That is, if the front-end gateway server or back-end gateway server in the n-times mapping relationship is inconsistent with the front-end gateway server that the first request packet actually passed through or the back-end gateway server that the second request packet actually passed through, then the mapping relationship is determined to be invalid.

In some embodiments, the fault detection server can also detect whether there are abnormal business rules in the gateway server. Specifically, the tracking information reported by the gateway server may include: packet loss record information generated by the gateway server when performing packet loss operation on probe request packets, which includes the reason for the packet loss; after obtaining such packet loss record information, the fault detection server can analyze whether there are abnormal business rules in the gateway server based on this packet loss record information.

The forwarding process of the main forwarding program umod in the gateway server will be described below with reference to Figure 5. As shown in Figure 5, after the first request packet arrives at the network card in the gateway server, it is evenly distributed to multiple RX queues of the network card via RSS (Receive Side Scaling). Each RX queue corresponds to an RX thread in the umod program. After receiving the first request packet, the RX thread hashes the first request packet to the TX thread responsible for business processing based on the (cip, vip) included in the first request packet. The TX thread performs legality checks, forwarding rule matching, overload rate limiting security group checks, packet encapsulation, and other processing on the first request packet. Then, it sends the processed first request packet to the TX queue bound to the TX thread, and then sends the first request packet to the business server through the TX queue, thus completing the forwarding of the first request packet. It should be understood that the gateway server's processing method for the second request packet is actually similar to the processing method for the first request packet, with only the transmission direction being different.

Based on the method provided in the embodiments of this application, the gateway server can record the processing operations of the detection request packets (including the first request packet and the second request packet), similar to the use of isotopes and fluorescent staining in medicine to locate pathogens. Based on the clear transmission link information corresponding to the request packet, the fault location can be clearly and directly located.

Specifically, in the method provided in this application embodiment, the umod program in the gateway server can be modified to record each processing stage of the request packet. If a probe request packet is lost, the packet loss record information is recorded via drop_point, and a data collection thread is started to transmit the recorded packet loss record information to the fault detection server. Furthermore, the fault detection server can periodically analyze the collected packet loss record information to achieve comprehensive detection and localization of abnormal business rules.

For example, the following packet loss record information can be defined in the forwarding logic of the gateway server:

//RX

DP_PORT_RX_MBUF_EMPTY = 100, //RXmbuf is empty

DP_PORT_RX_ENQUEUE_FAIL = 105, // RX dequeue failed

//TX before sched

DP_VIP_NOT_EXIST = 200, //VIP does not exist

DP_RULE_NOT_EXIST = 205, // Rule does not exist

DP_INVALID_VLANID = 210, // VLANID is invalid

DP_MARTIAN_SOURCE = 215, // Illegal message

DP_NOT_SYN_SCHED = 220, //TPC, no connection table, received non-SYN packets

DP_INVALID_RS_TYPE = 225, // RS type error

DP_DEST_UNAVAILABLE = 230, //RS is unavailable

DP_DEST_OFFLINE = 235, //RS is offline

DP_SESSION_CREATE_FAIL = 240, //Session creation failed

//TX limit and flow control

DP_FLOW_OVERLOAD = 300, //Flow overload

DP_CONN_OVERLOAD = 305, // Connection limit exceeded

DP_CONN_CREATE_FAIL = 310, // Connection creation failed

DP_IN_BLACKLIST = 315, // Request to be added to the blacklist

DP_NOT_IN_WHITHLIST = 320, // The request is not in the whitelist

DP_FLOW_LIMIT = 325, // Traffic rate limiting

DP_CTRL_DROP = 330, // Packet loss in auxiliary processes, deprecated, will be removed in later versions.

DP_CTRL_FREE_CONN = 335, // Auxiliary processes drop connections, deprecated, will be removed in later versions.

//TX packet encap related

DP_MANGLE_INNER_FAIL = 400, // Message modification failed.

DP_TUNNEL_ENCAP_FAIL = 405, // Encapsulation of tunnel header failed.

DP_ROUTE_LOOKUP_FAIL = 410, // Route lookup failed

//TX xmit packet distribution related

DP_XMIT_FAIL = 500, // Packet sending failed

It should be understood that in practical applications, in addition to defining the above-mentioned packet loss record information, other packet loss record information can be defined according to actual needs. This application does not impose any limitations on the defined packet loss record information.

Compared to related technologies that rely on deploying packet capture programs on gateway servers to locate abnormal business rules based on the packet capture results, the method for locating abnormal business rules provided in this application can comprehensively detect business rules based on packet loss tracking information uploaded by the gateway server, effectively improving the efficiency and accuracy of locating abnormal business rules.

Optionally, considering that the client and the business server usually communicate based on the TCP protocol, after the client and the business server complete a three-way handshake, the business server will be aware of the establishment of communication and allocate processing resources to the client accordingly. However, in the process of fault detection of the gateway server, the processing resources allocated by the business server to the client will not be actually used by the client. Therefore, in order to avoid the business server allocating useless processing resources to the client and causing unnecessary waste of the business server's processing resources, this application embodiment also provides a method to make the business server unaware of the fault detection process of the gateway server.

This means controlling the firewall to block the third handshake feedback information sent by the client to the business server, and controlling the kernel to send a response failure message to the business server.

Specifically, as shown in Figure 6, the client can initiate a socket connection request from the application layer program, using the Linux kernel protocol stack's TCP three-way handshake process for probing. After completing the first two handshakes with the business server, the firewall blocks the third handshake (ACK), thus preventing the TCP connection from being established successfully. Therefore, the business server will not be aware of this probing request packet and will not allocate processing resources to the client. Finally, to prevent the aforementioned half-open connection state from consuming server resources, the client can notify the kernel to send an RST to the business server after the probing ends, thus terminating the entire process. In this way, the entire probing process involves only one complete data interaction with the gateway server, and the business server remains unaware of it, making it very lightweight.

The fault detection method provided in the above-described embodiments of this application draws on the medical approach of using isotope or fluorescent staining to locate the cause of disease. It performs end-to-end staining and marking on probe request packets passing through a gateway server on the Internet. Specifically, the client and/or gateway server generate marking information based on packet sending and receiving operations related to the probe request packets, obtaining the marking information corresponding to each probe request packet. Then, based on the marking information corresponding to multiple probe request packets obtained within the fault detection period, fault detection is performed on the gateway server to accurately detect whether a fault exists, and if a fault is detected, to locate the fault location and cause. Thus, based on the transmission link path of the probe request packets, fault detection and location of gateway servers in complex network environments are achieved.

The following describes an exemplary fault detection system used in the aforementioned fault detection method, also known as a coloring system. As shown in Figure 7, this coloring system includes a probe client (which can be a server acting as the probe client), a log collection and storage system, a log forwarding server (gw_probe_proxy), a log analysis server (gw_probe_brain), and an operations system. The log collection and storage system collects and stores the logging information recorded by the gateway server. The log forwarding server (gw_probe_proxy) forwards the logging information recorded by the probe client and stored in the log collection and storage system to the log analysis server (gw_probe_brain), which is essentially the fault detection server mentioned above, used to execute the aforementioned fault detection method. The operations system is for maintenance personnel. This coloring system can provide real-time monitoring of the gateway server and trace gateway server faults, serving the current network environment of the gateway system.

Regarding the selection of probe clients and probe points, since the external network switches and internal network switches are all load-balanced with the multiple gateway servers in the gateway cluster, it is necessary to ensure that the probes to the gateway cluster can cover all the gateway servers in the gateway cluster, whether in the direction of incoming packets or outgoing packets. Typically, the gateway cluster can include 4 to 8 gateway servers.

Based on this, for inbound packets, 30 probe servers can be deployed across 3 locations, with 10 probe servers deployed in each location as a group. All servers should use a CAP (Consistency, Availability, Partition tolerance) network to eliminate the impact of carrier network quality and single-point probe server failures on the probe test results. For outbound packets, each gateway cluster should have dedicated rules for probe testing, and 20 to 30 service servers should be connected behind these rules, distributed across multiple data centers, to ensure that outbound packets from the service servers can cover all gateway servers in the gateway cluster.

For data aggregation and analysis, a key function of the coloring system is to monitor the health of the online gateway cluster, thus requiring high real-time performance and accuracy of the data. Coloring records are initially stored locally as logs, then aggregated to a Kafla cache using the cloud architecture platform's log collection function. Since there are two data sources (one from the client and one from the gateway server), analysis requires converging both sources in one operation, meaning the aggregation must be completed within the memory of a single machine. Considering the massive data volume, a single machine may not be able to handle it; therefore, a distributed approach is necessary. Two specific design schemes are available:

Option 1: Use Spark for real-time analysis. However, this approach has the following problems: the logging information recorded by the client and gateway server belongs to multiple Kafla sources. Forwarding all logging information from the same test to the same machine in the Spark cluster is quite complex. Furthermore, Spark, as a large data analysis system, requires huge maintenance costs and dedicated manpower support. Once a problem occurs, it is extremely difficult to locate the problem.

Option 2: Develop two programs: `gw_probe_proxy` for log forwarding and `gw_probe_brain` for log analysis. As shown in Figure 8, the log forwarding program `gw_probe_proxy` only performs hash forwarding, ensuring that the logging information for the same entire testing link is forwarded to the same log analysis program `gw_probe_brain` based on the identifier four-tuple. This ensures that multiple log analysis programs `gw_probe_brain` are completely independent, allowing for parallel scaling. Plugin 1 within the log analysis program `gw_probe_brain` can perform data analysis based on the logging information and provide the results to the health center's decision-making process. This completes data processing within seconds, ensuring the real-time nature and availability of online monitoring.

Furthermore, Plugin 2 in the log analysis program gw_probe_brain can notify the feedback/retransmission module in the log forwarding program gw_probe_proxy to resend the unreceived log information when receiving log points fails. Plugin 2 in the log analysis program gw_probe_brain can also send the aggregated complete link data to the Quefang platform responsible for offline processing.

In addition to online monitoring, the coloring system can also be used as a network problem analysis tool. When locating issues, operations and maintenance personnel may need to review historical data points. Due to the massive volume of historical data (up to 10TB+ per day) and its numerous dimensions, traditional database indexing methods are insufficient to meet time requirements. To address this issue, this application integrates with the Quefang platform. As shown in Figures 7 and 8, it utilizes Elasticsearch's inverted index to achieve fast offline data retrieval; terabytes of data can be retrieved within 1 second.

In practical applications, the staining system shown in Figure 7 above can achieve the following effects:

1. Automatic fault tolerance—second-level monitoring of cluster issues

The tracking information corresponding to the probe request packets is statistically reported every 20 seconds. The fault detection server can make a decision within 1 minute, playing a crucial role in automatic fault tolerance on the live network and significantly reducing the number of gateway issues. Taking a gateway cluster consisting of two gateway servers (x.x.x.101 and x.x.x.102) as an example, the fault detection performance under two different scenarios is as follows:

When a gateway server fails to forward packets, as shown in Figure 9, the overall forwarding success rate of the gateway cluster decreases. The number of packets received by the front end, calculated based on the front end packet receiving information uploaded by the gateway server, and the number of packets received by the back end, calculated based on the back end packet receiving information uploaded by the gateway server, both show a significant decrease. This indicates that the gateway server has malfunctioned.

When a gateway server experiences jittery packet loss, as shown in Figure 10, the overall forwarding success rate of the gateway cluster decreases. However, in terms of the number of packets received by the two gateway servers, there is no parallel difference. But the fault detection server can successfully count that all the lost packets belong to the server x.x.x.101 through IP route learning. Therefore, it can be determined that the server has a jittery packet loss fault.

2. Operational Tools—Offline Statistics to Assist in Daily Problem Locating

In addition to enabling real-time monitoring of gateway servers based on the tracking information corresponding to probe request packets, it can also help locate log issues. For example, operations and maintenance personnel can analyze the full-link information of specified probe request packets based on the offline data analysis interface provided by the coloring system, thereby locating abnormal issues reported by the business.

In response to the fault detection method described above, this application also provides a corresponding fault detection device to enable the practical application and implementation of the above fault detection method.

Referring to Figure 11, which is a structural schematic diagram of a fault detection device 1100 corresponding to the fault detection method shown in Figure 3 above, the fault detection device 1100 includes:

The information acquisition module 1101 is used to receive, during the fault detection period, the tracking information reported by each network device through which the probe request packet passes; the tracking information is the record information generated by the network device when it operates on the probe request packet; the network device includes at least one of a client, a gateway server, and a service server.

The information acquisition module 1101 is further configured to use the tracking information containing the same request packet identifier as the tracking information of the probe request packet corresponding to the request packet identifier;

The fault detection module 1102 is used to perform gateway fault detection on the network device based on the logging information of the probe request packet.

Optionally, based on the fault detection device shown in Figure 11, the network device includes the gateway server, which is deployed in a gateway cluster, and the gateway cluster includes multiple gateway servers. The tracking information reported by the gateway server includes: front-end packet reception information and back-end packet reception information. The front-end packet reception information is the record information generated by the gateway server when it completes the packet reception operation for the first request packet, and the back-end packet reception information is the record information generated by the gateway server when it completes the packet reception operation for the second request packet. The first request packet is a request packet sent by the client to the service server through the gateway server, and the second request packet is a request packet fed back by the service server to the client through the gateway server after receiving the first request packet. The second request packet contains the same request packet identifier as the first request packet. Referring to Figure 12, Figure 12 is a structural schematic diagram of another fault detection device 1200 provided in this application embodiment. As shown in Figure 12, the fault detection module 1102 includes:

The packet reception count counting unit 1201 is used to count the front-end packet reception count and the back-end packet reception count of each gateway server in the gateway cluster, based on the front-end packet reception information and the back-end packet reception information uploaded by the gateway server.

The fault determination unit 1202 is used to determine the faulty gateway server in the gateway cluster based on the number of packets received at the front end and the number of packets received at the back end of each gateway server in the gateway cluster.

Optionally, based on the fault detection device shown in Figure 12, the fault determination unit 1202 is specifically used for:

For each gateway server in the gateway cluster, determine whether the number of packets received at the front end and the number of packets received at the back end of the gateway server have dropped to zero. If any one or more of the number of packets received at the front end and the number of packets received at the back end have dropped to zero, then the gateway server is determined to be the faulty gateway server.

Optionally, based on the fault detection device shown in Figure 12, the fault determination unit 1202 is specifically used for:

Based on the number of packets received by the front end and the number of packets received by the back end of each gateway server in the gateway cluster, the average threshold for front end packet reception and the average threshold for back end packet reception are determined respectively.

For each gateway server in the gateway cluster, a first difference is determined between the average front-end packet reception threshold and the number of front-end packets received by the gateway server. It is then determined whether the first difference exceeds a preset front-end difference threshold. If so, the gateway server is determined to be the faulty gateway server.

For each gateway server in the gateway cluster, a second difference is determined between the average backend packet reception threshold and the number of backend packets received by the gateway server. It is then determined whether the second difference exceeds a preset backend difference threshold. If so, the gateway server is determined to be the faulty gateway server.

Optionally, based on the fault detection device shown in Figure 11, the network device includes at least the client and the gateway server; the gateway server is deployed in a gateway cluster, and the gateway cluster includes multiple gateway servers. Referring to Figure 13, Figure 13 is a schematic diagram of another fault detection device 1300 provided in an embodiment of this application. As shown in Figure 13, the device further includes:

The mapping relationship construction module 1301 is used to obtain the tracking information of historical probe request packets; based on the tracking information of the historical probe request packets, determine the front-end gateway server and back-end gateway server through which the historical probe request packets pass; the historical probe request packets include a historical first request packet and a historical second request packet, the historical first request packet being a request packet sent by the client to the business server through the front-end gateway server, and the historical second request packet being a request packet fed back by the business server to the client through the back-end gateway server after receiving the historical first request packet; constructing a mapping relationship between the IP address combinations included in the historical probe request packets and the front-end gateway server and the back-end gateway server; the IP address combinations include a source IP address and a destination IP address.

Optionally, based on the fault detection device shown in Figure 13, the mapping relationship construction module 1301 is further used for:

The target client is controlled to send the historical first request packet to the target business server; there is a correspondence between the target client and the target business server.

Optionally, based on the fault detection device shown in Figure 13, the fault detection module 1102 is specifically used for:

When it is determined that the transmission link corresponding to the probe request packet is incomplete based on the puncturing information of the probe request packet, the destination gateway server corresponding to the probe request packet is determined based on the missing puncturing information in the transmission link, the IP address combination included in the probe request packet, and the mapping relationship, and the packet loss count corresponding to the destination gateway server is updated.

When the number of packet losses corresponding to the destination gateway server reaches a preset packet loss threshold, it is determined that the destination gateway server has a jitter packet loss fault.

Optionally, based on the fault detection device shown in Figure 13, the mapping relationship construction module 1301 is further used for:

Based on the tracking information of the probe request packet, the front-end gateway server and back-end gateway server through which the probe request packet passes are determined; the probe request packet includes a first request packet and a second request packet, wherein the first request packet is a request packet sent by the client to the business server through the front-end gateway server, and the second request packet is a request packet sent by the business server to the client through the back-end gateway server after receiving the first request packet;

Based on the target IP address combination included in the probe request packet, determine the target mapping relationship including the target IP address combination;

Determine whether the front-end gateway server through which the probe request packet passes is consistent with the front-end server in the target mapping relationship, and whether the back-end gateway server through which the probe request packet passes is consistent with the back-end server in the target mapping relationship. If any one or more of these are inconsistent, the target mapping relationship is determined to be invalid and needs to be updated.

Optionally, based on the fault detection device shown in Figure 11, the network device includes the gateway server, and the logging information reported by the gateway server includes: packet loss record information generated by the gateway server when performing packet loss operation on the probe request packet, and the packet loss record information includes the reason for packet loss of the probe request packet; then the fault detection module 1102 is specifically used for:

Based on the packet loss record information reported by the gateway server, analyze whether the gateway server has any abnormal business rules.

Optionally, based on the fault detection device shown in Figure 11, the request packet identifier is a four-tuple identifier, which includes the source IP address, source port, destination IP address, and destination port; then the information acquisition module 1101 is specifically used for:

During the fault detection period, multiple dot information messages containing different quadruple identifiers are received.

Optionally, based on the fault detection device shown in Figure 11, when the client and the service server communicate based on the Transmission Control Protocol (TCP). Referring to Figure 14, which is a schematic diagram of another fault detection device 1400 provided in an embodiment of this application. As shown in Figure 14, the device further includes:

The communication control module 1401 is used to control the firewall to intercept the third handshake feedback information sent by the client to the business server; and to control the kernel to send a response failure message to the business server.

The fault detection device provided in the above-described embodiments of this application, drawing on the medical approach of using isotopes or fluorescent staining to locate the cause of disease, performs end-to-end staining and marking on probe request packets passing through gateway servers in the Internet. Specifically, the gateway server and/or client generate marking information based on packet sending and receiving operations related to the probe request packets, obtaining the marking information corresponding to each probe request packet. Then, based on the marking information corresponding to each of the multiple probe request packets obtained within the fault detection period, fault detection is performed on the gateway server to accurately detect whether a fault exists, and if a fault is detected, to locate the fault location and cause. Thus, based on the transmission link path of the probe request packets, fault detection and location of gateway servers in complex network environments are achieved.

This application also provides a device for detecting gateway server faults. Specifically, the device can be a server. The server provided in this application will be described below from the perspective of hardware implementation.

Referring to Figure 15, which is a schematic diagram of the structure of a server 1500 provided in an embodiment of this application, the server 1500 can vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 1522 (e.g., one or more processors) and memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server. Furthermore, the CPU 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the server 1500.

Server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input/output interfaces 1558, and/or one or more operating systems 1541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

The steps performed by the server in the above embodiments can be based on the server structure shown in Figure 15.

CPU 1522 is used to perform the following steps:

During the fault detection period, the network devices that receive the probe request packet report the logging information they receive; the logging information is the record information generated by the network devices when they operate on the probe request packet; the network devices include at least one of a client, a gateway server, and a service server.

The marker information containing the same request packet identifier is used as the marker information of the probe request packet corresponding to the request packet identifier;

Based on the tracking information of the probe request packet, the network device is used to detect gateway faults.

Optionally, the CPU 1522 can also be used to execute the steps of any implementation of the fault detection method provided in the embodiments of this application.

This application also provides a computer-readable storage medium for storing a computer program that executes any one of the implementation methods of the fault detection method described in the foregoing embodiments.

This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the implementation methods of the fault detection method described in the foregoing embodiments.

Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing computer programs.

It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And/or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and/or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character "/" generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims (24)

1. A fault detection method, characterized in that the method comprises: During the fault detection period, the network devices that receive the probe request packet report the logging information they receive; the logging information is the record information generated by the network devices when they operate on the probe request packet; the network devices include at least one of a client, a gateway server, and a service server. The marker information containing the same request packet identifier is used as the marker information of the probe request packet corresponding to the request packet identifier; Based on the dot information of the probe request packet, the network device is used to detect gateway faults. The network device includes the gateway server, which is deployed in a gateway cluster, and the gateway cluster includes multiple gateway servers. The tracking information reported by the gateway server includes front-end packet reception information and back-end packet reception information. The front-end packet reception information is the record information generated by the gateway server when it completes the packet reception operation for a first request packet. The back-end packet reception information is the record information generated by the gateway server when it completes the packet reception operation for a second request packet. The first request packet is a request packet sent by the client to the business server through the gateway server. The second request packet is a request packet fed back by the business server to the client through the gateway server after receiving the first request packet. The second request packet contains the same request packet identifier as the first request packet. The step of performing gateway fault detection on the network device based on the logging information of the probe request packet includes: For each gateway server in the gateway cluster, based on the front-end packet receiving information and the back-end packet receiving information uploaded by the gateway server, the front-end packet receiving count and the back-end packet receiving count of the gateway server are counted respectively. The faulty gateway server in the gateway cluster is determined based on the number of packets received at the front end and the number of packets received at the back end of each gateway server in the gateway cluster. 2. The method according to claim 1, characterized in that, determining the faulty gateway server in the gateway cluster based on the number of front-end packets received and the number of back-end packets received by each gateway server in the gateway cluster includes: For each gateway server in the gateway cluster, determine whether the number of packets received at the front end and the number of packets received at the back end of the gateway server have dropped to zero. If any one or more of the number of packets received at the front end and the number of packets received at the back end have dropped to zero, then the gateway server is determined to be the faulty gateway server. 3. The method according to claim 1, characterized in that, determining whether there is a faulty gateway server in the gateway cluster based on the number of front-end packets received and the number of back-end packets received by each gateway server in the gateway cluster includes: Based on the number of packets received by the front end and the number of packets received by the back end of each gateway server in the gateway cluster, the average threshold for front end packet reception and the average threshold for back end packet reception are determined respectively. For each gateway server in the gateway cluster, a first difference is determined between the average front-end packet reception threshold and the number of front-end packets received by the gateway server. It is then determined whether the first difference exceeds a preset front-end difference threshold. If so, the gateway server is determined to be the faulty gateway server. For each gateway server in the gateway cluster, a second difference is determined between the average backend packet reception threshold and the number of backend packets received by the gateway server. It is then determined whether the second difference exceeds a preset backend difference threshold. If so, the gateway server is determined to be the faulty gateway server. 4. The method according to claim 1, wherein the network device includes at least the client and the gateway server; the gateway server is deployed in a gateway cluster, and the gateway cluster includes multiple gateway servers; the method further includes: Obtain the location information of historical probe request packets; Based on the tracking information of the historical probe request packets, the front-end gateway server and back-end gateway server through which the historical probe request packets pass are determined; the historical probe request packets include a first historical request packet and a second historical request packet, the first historical request packet is a request packet sent by the client to the business server through the front-end gateway server, and the second historical request packet is a request packet sent by the business server to the client through the back-end gateway server after receiving the first historical request packet; Construct a mapping relationship between the IP address combinations included in the historical probe request packets and the front-end gateway server and the back-end gateway server; the IP address combinations include source IP addresses and destination IP addresses. 5. The method according to claim 4, characterized in that the method further comprises: The target client is controlled to send the historical first request packet to the target business server; there is a corresponding relationship between the target client and the target business server. 6. The method according to claim 4, wherein the step of performing gateway fault detection on the network device based on the logging information of the probe request packet includes: When it is determined that the transmission link corresponding to the probe request packet is incomplete based on the puncturing information of the probe request packet, the destination gateway server corresponding to the probe request packet is determined based on the missing puncturing information in the transmission link, the IP address combination included in the probe request packet, and the mapping relationship, and the packet loss count corresponding to the destination gateway server is updated. When the number of packet losses corresponding to the destination gateway server reaches a preset packet loss threshold, it is determined that the destination gateway server has a jitter packet loss fault. 7. The method according to claim 4, characterized in that the method further comprises: Based on the tracking information of the probe request packet, the front-end gateway server and back-end gateway server through which the probe request packet passes are determined; the probe request packet includes a first request packet and a second request packet, wherein the first request packet is a request packet sent by the client to the business server through the front-end gateway server, and the second request packet is a request packet sent by the business server to the client through the back-end gateway server after receiving the first request packet; Based on the target IP address combination included in the probe request packet, determine the target mapping relationship including the target IP address combination; Determine whether the front-end gateway server through which the probe request packet passes is consistent with the front-end server in the target mapping relationship, and whether the back-end gateway server through which the probe request packet passes is consistent with the back-end server in the target mapping relationship. If any one or more of these are inconsistent, the target mapping relationship is determined to be invalid and needs to be updated. 8. The method according to claim 1, wherein the tracking information reported by the gateway server includes: packet loss record information generated by the gateway server when performing packet loss operation on the probe request packet, the packet loss record information including the reason for packet loss of the probe request packet; The step of performing gateway fault detection on the network device based on the logging information of the probe request packet includes: Based on the packet loss record information reported by the gateway server, analyze whether the gateway server has any abnormal business rules. 9. The method according to claim 1, wherein the request packet identifier is a four-tuple identifier, the four-tuple identifier including a source IP address, a source port, a destination IP address, and a destination port; and the step of receiving the tracking information reported by each network device through which the probe request packet passes during the fault detection period includes: During the fault detection period, multiple dot information messages containing different quadruple identifiers are received. 10. The method according to claim 1, wherein when the client and the service server communicate based on Transmission Control Protocol (TCP), the method further comprises: The control firewall intercepts the third handshake feedback information sent by the client to the business server; the control kernel sends a response failure message to the business server. 11. A fault detection device, characterized in that the device comprises: The information acquisition module is used to receive, during the fault detection period, the tracking information reported by each network device through which the probe request packet passes; the tracking information is the record information generated by the network device when it operates on the probe request packet; the network device includes at least one of a client, a gateway server, and a service server; The information acquisition module is further configured to use the marker information containing the same request packet identifier as the marker information of the probe request packet corresponding to the request packet identifier; The fault detection module is used to perform gateway fault detection on the network device based on the logging information of the probe request packet; The network device includes the gateway server, which is deployed in a gateway cluster, and the gateway cluster includes multiple gateway servers. The tracking information reported by the gateway server includes front-end packet reception information and back-end packet reception information. The front-end packet reception information is the record information generated by the gateway server when it completes the packet reception operation for a first request packet. The back-end packet reception information is the record information generated by the gateway server when it completes the packet reception operation for a second request packet. The first request packet is a request packet sent by the client to the business server through the gateway server. The second request packet is a request packet fed back by the business server to the client through the gateway server after receiving the first request packet. The second request packet contains the same request packet identifier as the first request packet. The fault detection module includes: The packet reception count unit is used to count the front-end packet reception count and the back-end packet reception count of each gateway server in the gateway cluster, based on the front-end packet reception information and the back-end packet reception information uploaded by the gateway server. The fault determination unit is used to determine the faulty gateway server in the gateway cluster based on the number of packets received at the front end and the number of packets received at the back end of each gateway server in the gateway cluster. 12. The apparatus according to claim 11, wherein the fault determination unit is specifically used for: For each gateway server in the gateway cluster, determine whether the number of packets received at the front end and the number of packets received at the back end of the gateway server have dropped to zero. If any one or more of the number of packets received at the front end and the number of packets received at the back end have dropped to zero, then the gateway server is determined to be the faulty gateway server. 13. The apparatus according to claim 11, wherein the fault determination unit is specifically used for: Based on the number of packets received by the front end and the number of packets received by the back end of each gateway server in the gateway cluster, the average threshold for front end packet reception and the average threshold for back end packet reception are determined respectively. For each gateway server in the gateway cluster, a first difference is determined between the average front-end packet reception threshold and the number of front-end packets received by the gateway server. It is then determined whether the first difference exceeds a preset front-end difference threshold. If so, the gateway server is determined to be the faulty gateway server. For each gateway server in the gateway cluster, a second difference is determined between the average backend packet reception threshold and the number of backend packets received by the gateway server. It is then determined whether the second difference exceeds a preset backend difference threshold. If so, the gateway server is determined to be the faulty gateway server. 14. The apparatus according to claim 11, wherein the network device includes at least the client and the gateway server; the gateway server is deployed in a gateway cluster, the gateway cluster including multiple gateway servers; the apparatus further includes: A mapping relationship construction module is used to obtain the tracking information of historical probe request packets; based on the tracking information of the historical probe request packets, determine the front-end gateway server and back-end gateway server through which the historical probe request packets pass; the historical probe request packets include a historical first request packet and a historical second request packet, the historical first request packet being a request packet sent by the client to the business server through the front-end gateway server, and the historical second request packet being a request packet sent by the business server to the client through the back-end gateway server after receiving the historical first request packet; constructing a mapping relationship between the IP address combinations included in the historical probe request packets and the front-end gateway server and the back-end gateway server; the IP address combinations include source IP address and destination IP address. 15. The apparatus according to claim 14, wherein the mapping relationship construction module is further configured to: The target client is controlled to send the historical first request packet to the target business server; there is a corresponding relationship between the target client and the target business server. 16. The apparatus according to claim 14, wherein the fault detection module is specifically used for: When it is determined that the transmission link corresponding to the probe request packet is incomplete based on the puncturing information of the probe request packet, the destination gateway server corresponding to the probe request packet is determined based on the missing puncturing information in the transmission link, the IP address combination included in the probe request packet, and the mapping relationship, and the packet loss count corresponding to the destination gateway server is updated. When the number of packet losses corresponding to the destination gateway server reaches a preset packet loss threshold, it is determined that the destination gateway server has a jitter packet loss fault. 17. The apparatus according to claim 14, wherein the mapping relationship construction module is further configured to: Based on the tracking information of the probe request packet, the front-end gateway server and back-end gateway server through which the probe request packet passes are determined; the probe request packet includes a first request packet and a second request packet, wherein the first request packet is a request packet sent by the client to the business server through the front-end gateway server, and the second request packet is a request packet sent by the business server to the client through the back-end gateway server after receiving the first request packet; Based on the target IP address combination included in the probe request packet, determine the target mapping relationship including the target IP address combination; Determine whether the front-end gateway server through which the probe request packet passes is consistent with the front-end server in the target mapping relationship, and whether the back-end gateway server through which the probe request packet passes is consistent with the back-end server in the target mapping relationship. If any one or more of these are inconsistent, the target mapping relationship is determined to be invalid and needs to be updated. 18. The apparatus according to claim 11, wherein the tracking information reported by the gateway server includes: packet loss record information generated by the gateway server when performing packet loss operation on the probe request packet, the packet loss record information including the reason for packet loss of the probe request packet; The fault detection module is specifically used for: Based on the packet loss record information reported by the gateway server, analyze whether the gateway server has any abnormal business rules. 19. The apparatus according to claim 11, wherein the request packet identifier is a four-tuple identifier, the four-tuple identifier including a source IP address, a source port, a destination IP address, and a destination port; the information acquisition module is specifically used for: During the fault detection period, multiple dot information messages containing different quadruple identifiers are received. 20. The apparatus according to claim 11, wherein when the client and the service server communicate based on Transmission Control Protocol (TCP), the apparatus further comprises: The communication control module is used to control the firewall to intercept the third handshake feedback information sent by the client to the business server; and to control the kernel to send a response failure message to the business server. 21. A fault detection system, characterized in that the system comprises: a client, a gateway server, and a fault detection server; The client is configured to generate tracking information based on its packet sending and receiving operations for probe request packets, and report the tracking information to the fault detection server. The gateway server is configured to generate tracking information based on its packet sending and receiving operations for the probe request packet, and report the tracking information to the fault detection server. The fault detection server is used to execute the fault detection method according to any one of claims 1 to 10. 22. A device, characterized in that the device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the fault detection method according to any one of claims 1 to 10 according to the computer program. 23. A computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program, the computer program being used to execute the fault detection method according to any one of claims 1 to 10. 24. A computer program product, characterized in that the computer program product includes computer instructions, wherein a processor of a computer device executes the computer instructions, causing the computer device to perform the fault detection method according to any one of claims 1 to 10.