WO2025055769A1 - Latency measurement method and apparatus, and device and computer-readable storage medium - Google Patents

Abstract

The present application belongs to the technical field of communications. Disclosed are a latency measurement method and apparatus, and a device and a computer-readable storage medium. The method comprises: a first device sending a first message to a second device, wherein the transmission path of the first message is a first path; and the first device acquiring a first latency of the first path, which first latency is determined on the basis of a first timestamp and a second timestamp, wherein the first timestamp is a timestamp of when the first message is sent which is recorded by a network interface card on the first device, and the second timestamp is a timestamp of when the first message is received which is recorded by a network interface card on the second device. In the present application, the first latency, which is determined by means of hardware timestamps recorded by network interface cards, serves as a network latency on a first path between a first device and a second device, and thus a long-tail latency caused by an on-device processing latency of the first device and the second device can be avoided, and the binding of a path and a latency can be realized.

Description

Delay measurement method, device, equipment and computer readable storage medium

This application claims priority to Chinese patent application No. 202311172026.X filed on September 12, 2023, with invention name “Time Delay Measurement Method, Device, Equipment and Computer-Readable Storage Medium”, the entire contents of which are incorporated by reference into this application.

Technical Field

The present application relates to the field of communication technology, and in particular to a delay measurement method, device, equipment and computer-readable storage medium.

Background Art

In a communication system, when two devices exchange messages, multiple network devices between the two devices need to forward the messages, resulting in a delay between the time the message is sent and the time the message is received. Therefore, it is necessary to measure the delay between the two devices to accurately estimate the time it takes for a message to reach another device from one device.

Summary of the invention

The present application provides a delay measurement method, apparatus, device, and computer-readable storage medium to accurately measure the network delay on a path. The technical solution is as follows:

In a first aspect, a delay measurement method is provided, the method comprising: a first device sends a first message to a second device, the transmission path of the first message is a first path; the first device obtains a first delay of the first path determined based on a first timestamp and a second timestamp, the first timestamp is a timestamp of sending the first message recorded by a network card on the first device, and the second timestamp is a timestamp of receiving the first message recorded by the network card on the second device.

The timestamp recorded by the network card in the present application is a hardware timestamp. The first delay determined according to the timestamp is the network delay on the first path between the first device and the second device. This can avoid the long-tail delay caused by the delay in processing messages by the first device and the second device, and can realize the binding of delay and path.

In a possible implementation, before the first device sends the first message to the second device, it also includes: the first device sends a path detection message to the second device, the path detection message includes first indication information, and the first indication information indicates that a detection result message is returned based on the path detection message; the first device receives at least one detection result message returned based on the path detection message, and the detection result message includes information about the physical port and virtual port of the network device that sends the detection result message; the first device determines at least one path between the first device and the second device based on the information about the physical port and virtual port of at least one network device included in the at least one detection result message; the first device selects a first path from the at least one path.

The first device clarifies each path between the first device and the second device through a path detection message and a detection result message, and selects a first path among each hop path between the first device and the second device, so that the first message can be transmitted along the first path, and the delay on the first path is measured to achieve binding of the delay and the path.

In a possible implementation, the first message includes second indication information, and the second indication information indicates that the second message is returned based on the first message; before the first device obtains the first delay of the first path determined based on the first timestamp and the second timestamp, it also includes: the first device receives at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and the first virtual port of the first network device that sends any second message; the first device determines the first path based on the information about the first physical port and the first virtual port of at least one first network device included in the at least one second message, and the first path includes the first device, the second device, and at least one first network device through which the first message passes.

The first device sends a first message carrying second indication information, receives a second message returned by the first network device based on the first message, clarifies the first network devices passed by the first message, and further clarifies the first path for transmitting the first message, so that the first device can bind the acquired first delay to the first path.

In a possible implementation, the first device obtains a first delay of a first path determined based on a first timestamp and a second timestamp, including: the first device receives a message including a second timestamp sent by a centralized device or a second device based on a first message, and obtains the second timestamp from the message including the second timestamp; the first device determines the first delay of the first path according to the first timestamp and the second timestamp. The first device obtains the second timestamp and autonomously calculates the first delay according to the first timestamp and the second timestamp, thereby improving the efficiency of obtaining the first delay.

In a possible implementation, the first device obtains the first delay of the first path determined based on the first timestamp and the second timestamp, including: the first device sends a message including the first timestamp to the second device or the centralized device, the first timestamp is used by the second device or the centralized device to determine the first delay of the first path; the first device receives the message including the first delay sent by the second device or the centralized device, the first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp; the first device obtains the first delay from the message including the first delay. The first device obtains the first delay by sending the first timestamp to the second device or the centralized device and receiving the first delay determined and sent by the second device or the centralized device based on the first timestamp and the second timestamp, without the need for the first device to autonomously calculate the first delay, thereby reducing the calculation pressure of the first device.

In a possible implementation, after the first device obtains the first delay of the first path determined based on the first timestamp and the second timestamp, it also includes: the first device receives the third message sent by the second device based on the first message, and the transmission path of the third message is the second path; the first device obtains the second delay of the second path determined based on the third timestamp and the fourth timestamp, the third timestamp is the timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is the timestamp of sending the third message recorded by the network card on the second device. The first device can also obtain the second delay of the third message returned by the second device on the second path based on a similar method to obtaining the first delay. The second delay is the network delay between the first device and the second device, which can avoid the long tail delay caused by the end processing delay of the first device and the second device, and realize the binding of delay and path.

In a possible implementation, the first device obtains the second delay of the second path determined based on the third timestamp and the fourth timestamp, including: the first device receives a message including a fourth timestamp sent by the centralized device or the second device based on the third message, and obtains the fourth timestamp; the first device determines the second delay of the second path according to the third timestamp and the fourth timestamp. The first device obtains the fourth timestamp and autonomously calculates the second delay according to the third timestamp and the fourth timestamp, thereby improving the efficiency of obtaining the second delay.

In a possible implementation, the first device obtains the second delay of the second path determined based on the third timestamp and the fourth timestamp, including: the first device sends a message including the third timestamp to the second device or the centralized device, the third timestamp is used by the second device or the centralized device to determine the second delay of the second path; the first device receives the message including the second delay sent by the second device or the centralized device, the second delay is determined by the second device or the centralized device based on the third timestamp and the fourth timestamp; the first device obtains the second delay from the message including the second delay. The first device obtains the second delay by sending the third timestamp to the second device or the centralized device and receiving the second delay determined and sent by the second device or the centralized device based on the third timestamp and the fourth timestamp, without the need for the first device to autonomously calculate the second delay, thereby reducing the calculation pressure of the first device.

In a possible implementation, the method further includes: for any one of the at least one first network devices, the first device determines the information of the second virtual port corresponding to the first physical port of any one of the first network devices; the first device updates the first message based on the information of the second virtual port to obtain an updated message, and the updated message includes the information of the second virtual port; the first device sends the updated message to the second device, and the transmission path of the updated message is the first path; the first device obtains the third delay of the first path determined based on the fifth timestamp and the sixth timestamp, the fifth timestamp is the timestamp of sending the updated message recorded by the network card on the first device, and the sixth timestamp is the timestamp of receiving the updated message recorded by the network card of the second device. The first device can update the first message based on the information of the second virtual port, so that the first device and the second device re-establish a connection according to the updated message. When the first path remains unchanged, the third delay of the first path when transmitting based on the new virtual port is re-detected.

In a second aspect, a delay measurement device is provided, which includes: a transceiver module for sending a first message to a second device, the transmission path of the first message being a first path; a processing module for obtaining a first delay of the first path determined based on a first timestamp and a second timestamp, the first timestamp being the timestamp of sending the first message recorded by a network card on the first device, and the second timestamp being the timestamp of receiving the first message recorded by the network card on the second device.

In one possible implementation, the transceiver module is also used to send a path detection message to the second device, the path detection message includes first indication information, and the first indication information indicates that a detection result message is returned based on the path detection message; receive at least one detection result message returned based on the path detection message, the detection result message includes information about the physical port and virtual port of the network device that sent the detection result message; the processing module is also used to determine at least one path between the first device and the second device based on information about the physical port and virtual port of at least one network device included in at least one detection result message; and select a first path from the at least one path.

In one possible implementation, the first message includes second indication information, and the second indication information indicates that the second message is returned based on the first message; the transceiver module is also used to receive at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and the first virtual port of the first network device that sends any second message; the processing module is also used to determine the first path based on the information about the first physical port and the first virtual port of at least one first network device included in the at least one second message, and the first path includes the first device, the second device, and at least one first network device through which the first message passes.

In a possible implementation, the processing module is configured to receive a message including a second time and a second time sent by the centralized device or the second device based on the first message. The message including the time stamp is obtained by obtaining the second time stamp from the message including the second time stamp; and the first delay of the first path is determined according to the first time stamp and the second time stamp.

In one possible implementation, a processing module is used to send a message including a first timestamp to a second device or a centralized device, where the first timestamp is used by the second device or the centralized device to determine a first delay of a first path; receive a message including a first delay sent by the second device or the centralized device, where the first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp; and obtain the first delay from the message including the first delay.

In one possible implementation, the transceiver module is also used to receive a third message sent by the second device based on the first message, and the transmission path of the third message is the second path; the processing module is also used to obtain a second delay of the second path determined based on a third timestamp and a fourth timestamp, the third timestamp is the timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is the timestamp of sending the third message recorded by the network card on the second device.

In a possible implementation, the processing module is used to receive a message including a fourth timestamp sent by the centralized device or the second device based on the third message, obtain the fourth timestamp; and determine the second delay of the second path according to the third timestamp and the fourth timestamp.

In one possible implementation, the processing module is used to send a message including a third timestamp to a second device or a centralized device, where the third timestamp is used by the second device or the centralized device to determine a second delay of a second path; receive a message including a second delay sent by the second device or the centralized device, where the second delay is determined by the second device or the centralized device based on the third timestamp and a fourth timestamp; and obtain the second delay from the message including the second delay.

In a possible implementation, the processing module is also used to determine, for any one of the at least one first network devices, information of a second virtual port corresponding to a first physical port of any one of the first network devices; update the first message based on the information of the second virtual port to obtain an updated message, wherein the updated message includes information of the second virtual port; the transceiver module is also used to send the updated message to the second device, wherein the transmission path of the updated message is the first path; the processing module is also used to obtain a third delay of the first path determined based on a fifth timestamp and a sixth timestamp, wherein the fifth timestamp is the timestamp of sending the updated message recorded by the network card on the first device, and the sixth timestamp is the timestamp of receiving the updated message recorded by the network card of the second device.

In a third aspect, another communication device is provided, the device comprising: a network interface, a memory, and a processor. The network interface, the memory, and the processor communicate with each other through an internal connection path, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to control the network interface to receive signals and control the network interface to send signals, and when the processor executes the instructions stored in the memory, the processor executes the method in the first aspect or any possible implementation of the first aspect.

Optionally, there are one or more processors and one or more memories.

Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

According to a fourth aspect, a communication system is provided, the system comprising the device according to the second aspect or any possible implementation manner of the second aspect.

In a fifth aspect, a computer program (product) is provided, which includes: computer program code, which, when executed by a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program or instruction. When the program or instruction runs on a computer, the method in the above-mentioned first aspect or any possible implementation of the first aspect is executed.

In a seventh aspect, a chip is provided, comprising a processor for calling and executing instructions stored in a memory, so that a computer equipped with the chip executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

In an eighth aspect, another chip is provided, comprising: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, a computer equipped with the chip executes the method in the first aspect or any possible implementation of the first aspect.

It should be understood that the beneficial effects achieved by the technical solutions of the second to eighth aspects of the present application and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a data center network topology structure provided in an embodiment of the present application;

FIG2 is a schematic diagram of a path of a data center network provided in an embodiment of the present application;

FIG3 is a schematic diagram of another path of a data center network provided in an embodiment of the present application;

FIG4 is a schematic diagram of an implementation scenario provided in an embodiment of the present application;

FIG5 is a schematic diagram of a flow chart of a delay measurement method provided in an embodiment of the present application;

FIG6 is a schematic diagram of a partial structure of a path detection message provided in an embodiment of the present application;

FIG7 is a schematic diagram of a path detection process provided by an embodiment of the present application;

FIG8 is a schematic diagram of another path detection process provided by an embodiment of the present application;

FIG9 is a schematic diagram of another path detection process provided by an embodiment of the present application;

FIG10 is a schematic diagram of another implementation scenario provided in an embodiment of the present application;

FIG11 is a schematic diagram of a ping network software architecture provided in an embodiment of the present application;

FIG12 is a schematic diagram of a data matrix of a measurement provided in an embodiment of the present application;

FIG13 is a schematic diagram of the structure of a delay measurement device provided in an embodiment of the present application;

FIG14 is a schematic diagram of the structure of a delay measurement device provided in an embodiment of the present application;

FIG. 15 is a schematic diagram of the structure of another delay measurement device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

The data center deployed in the data communication network (DCN) includes a large number of servers, which are interconnected through various types of media, such as network cards, switches, routers, cables, and optical fibers.

Referring to FIG1 , a schematic diagram of a data center network topology is shown. The topology shows three different DCs, namely, data center (DC) 1, DC2 and DC3, which are connected to each other through an inter-DC network. The structure of the intra-DC network can refer to the internal structure of DC1.

The internal network of DC1 may include multiple network devices and multiple servers. The network devices may include devices such as switches or routers that can be used for data forwarding. The connection between any two network devices represents that any two network devices can achieve communication connection. Multiple network devices can be divided into network devices of different levels. Taking the network device as a switch as an example, the multiple switches in the DC can be divided into spine switches, leaf switches and top of rack (TOR) switches. The spine switch can also be called a spine switch. The spine switches in DC1 constitute the spine layer of the internal network of DC1. The leaf switch can also be called a leaf switch. The leaf layer switches in DC1 constitute the leaf layer of the internal network of DC1. The top of rack switch can achieve direct communication connection with the server. A top of rack switch and multiple servers that can achieve direct communication connection with the top of rack switch can be deployed in the same point of delivery (POD). The distribution point is a physical partition of resources. Multiple PODs and leaf switches that can achieve direct communication connection with the multiple PODs can be deployed in the same Podset.

With the development of cloud computing technology, the amount of data that needs to be interacted and calculated is increasing, and the size of data centers is getting larger and larger. Based on the physical facilities of data centers, large-scale distributed services are running, such as search, distributed file systems, distributed storage, distributed computing (Mapreduce), and other distributed services.

Due to the large scale of distributed services, including more software systems and a large number of components, the coupling relationship between different distributed services is relatively complex, and the interaction between components of different distributed services needs to be completed through the network, including but not limited to the network within the same data center and the network across different data centers.

In large-scale digital communication systems, software and hardware failures are relatively frequent. However, due to the composition characteristics of distributed services, many software and hardware failures are manifested as increased latency. For example, intermittent connectivity of some components, a sharp increase in end-to-end latency, or a sharp drop in network throughput of servers in a data center are not caused by network anomalies. These service anomalies are usually manifested as increased latency, causing users to mistakenly believe that they are network problems. Therefore, a latency measurement method is urgently needed to determine whether a service anomaly is caused by a network anomaly.

In addition, when the service-level agreement (SLA) of the network is damaged (broken) due to various network failures, it will cause live-site failures, where live-site failures are any events that may affect customers, partners or revenue. Live-site failures need to be quickly detected, migrated and resolved. However, data center networks may include hundreds of thousands or even millions of servers, hundreds of thousands of switches, millions of cables and optical fibers, and in data center networks, there can be multiple paths for traffic transmission between two servers, and different paths can be used to share different traffic interacting between the two servers. When a path between two servers is abnormal, the traffic transmission task of the abnormal path can be assumed by other paths between the two servers, or the traffic on the abnormal path can be switched to other paths between the two servers for transmission. Therefore, the multi-path feature of the data center network can ensure network reliability and traffic load balancing.

For example, see Figure 2, which is a path diagram of a data center network. The data center network includes spine switches 1 to 4 and multiple leaf switches. Among them, leaf switch 1 can achieve direct communication connection with server 1, and leaf switch 2 can achieve direct communication connection with server 2. There are four paths between server 1 and server 2, which are path 1: server 1-leaf switch 1-spine switch 1-leaf switch 2-server 2; path 2: server 1-leaf switch 1-spine switch 2-leaf switch 2-server 2; path 3: server 1-leaf switch 1-spine switch 3-leaf switch 2-server 2; path 4: server 1-leaf switch 1-spine switch 4-leaf switch 2-server 2. Each path only limits the devices on the path but not the direction of the path. Each path is a path that can communicate in both directions. Taking path 1 as an example, data can be transmitted from server 1 to server 2 through path 1, and can also be transmitted from server 2 to path 1 through path 1. When path 1 is abnormal, the traffic transmitted by path 1 can be switched to path 2, path 3 or path 1 to ensure the normal transmission of the traffic.

Alternatively, see also the path diagram of another data center network shown in FIG3. The data center network includes core layer network devices, network devices in an aggregation, and edge devices, network devices 11 and 12 are core layer network devices, network devices 211 to 214 and network devices 221 to 224 are network devices in an aggregation, and edge devices can be servers, that is, edge devices 31 to 34 can be servers. There are multiple different paths between two different edge devices.

Because there can be multiple paths between two devices in a data center network, there is an urgent need for a method that can quickly detect and locate faults. For example, there is an urgent need for a method that can bind each device and the path formed by the devices to the delay to quickly determine the location of the network fault, so as to further realize fault detection and fault location.

The embodiment of the present application provides a delay measurement method that can accurately measure the end-to-end network delay, determine whether the service anomaly is caused by the network anomaly, and bind the delay to each device and each path to determine the location where the network failure occurs.

Referring to FIG. 4 , a schematic diagram of an implementation scenario provided by an embodiment of the present application is shown. The implementation scenario includes a first device 41, a network device 42, and a second device 43. The first device 41 and the second device 43 are connected in communication via the network device 42. The first device 41 may also be referred to as a source device. The first device 41 may, for example, be a storage service node in a server, a server cluster, or a cloud storage. The storage service node includes a storage client, a storage controller, a storage array, and the like. The second device may also be referred to as a destination device. The second device 43 may also be a storage service node in a server, a server cluster, or a cloud storage. The network device 42 is a device used for message forwarding in a communication network. For example, it may be a switch, a router, or other forms of forwarding devices. The number of network devices 42 may be one or more, and the embodiment of the present application does not limit this.

Optionally, the implementation scenario may also include a centralized device not shown in FIG4 , which may respectively realize communication connection with the first device 41, the network device 42, and the second device 43, and the centralized device may also be referred to as a centralized collector or analyzer. In addition, the scenario shown in FIG4 is not limited to the above data center, and may also be applied to network topologies using other topologies, such as a torus network, a dragonfly network, a mesh network, and CLOS (a new generation of data center network architecture).

Referring to FIG. 5 , a schematic flow chart of a delay measurement method provided in an embodiment of the present application is shown. The method can be applied to the implementation scenario shown in FIG. 4 . The method includes but is not limited to the following S501 and S502 .

S501: A first device sends a first message to a second device, where a transmission path of the first message is a first path.

The embodiment of the present application does not limit the type of the first message. For example, the first message may be a message obtained by extending the precision time protocol (PTP) based on 1588 version 2, or may be a message of other types. If the first message is a message obtained by extending the PTP based on 1588v2, the first message may be sent by a PTP agent deployed on the first device.

Optionally, the first path may be determined by the first device before sending the first message, or may be determined after sending the first message. Below, taking case A1 and case A2 as examples, different cases in which the first device determines the first path are described.

In case A1, the first device determines the first path before sending the first message. There are multiple network devices for forwarding messages between the first device and the second device, so there can be at least one path between the first device and the second device. In this implementation, before the first device sends the first message to the second device, the first device can perform path detection or path perception on at least one path between the first device and the second device, determine at least one path between the first device and the second device, and select the first path from the at least one path between the first device and the second device.

Exemplarily, the first device may send a path detection message to the second device, the path detection message including first indication information, the first indication information indicating that a detection result message is returned based on the path detection message, and the first indication information is the first mark for path perception. Afterwards, the first device receives at least one detection result message returned based on the path detection message, the detection result message including information about the physical port and virtual port of the network device that sent the detection result message. Among them, the first indication information may be an indicator (flag), such as characters such as letters or numbers, or the first indication information may also be information about the virtual port of at least one network device or an identifier of at least one network device. The first indication information may be added to a reserved identification bit of the path detection message, the reserved identification bit may be located in the message header of the path detection message, the reserved identification bit may be a field extended by the message, or may reuse an existing field in the message.

Before the first device sends a path detection message to the second device, a matching or identification rule can be issued to the network device in advance, so that when the network device matches or identifies the path detection message, it can generate and return a detection result message to the first device. If the information of the virtual port in the first indication information in the text includes the information of the virtual port of network device A, the path detection message is a unicast message, and after receiving the path detection message, network device A recognizes that the information of the virtual port in the first indication information includes the information of the virtual port of the device, then it can be considered that the path detection message matches network device A, and thus network device A can generate and return a detection result message to the first device. If the information of the virtual port in the first indication information in the path detection message does not include the information of the virtual port of network device B, after receiving the path detection message, network device B determines that the information of the virtual port in the first indication information does not include the information of the virtual port of the device, then it is considered that the path detection message does not match network device B, and thus network device B can discard the path detection message. In a possible implementation, the network device can copy the received path detection message, that is, mirror the path detection message, and then the network device can add the information of the physical port on the network device that receives the path detection message to the mirrored message obtained by mirroring, complete the encapsulation and second marking of the mirrored message, and obtain the detection result message, wherein the information of the physical port can be a physical port identifier (identifier, ID).

The path detection message also includes an equal-cost multipath routing (ECMP) hash field (Hash Fields) and a payload field corresponding to the path detection message. Exemplarily, a partial structure of the path detection message can be seen in FIG6, where the ECMP Hash Fields include a source Internet Protocol (Internet Protocol, IP) address field, a destination IP address field, a protocol field, a source port information field, and a destination port information field of the path detection message. Optionally, the ECMP Hash Fields corresponding to the path detection message may also include a source media access control address (MAC) address and a destination MAC address of the message.

The source port information in the path detection message can also be called the source port number, which corresponds to the information of the virtual port in the embodiment of the present application, and is used to identify the virtual port that the path detection message needs to pass through. Since the detection result message includes the same mirror message as the path detection message, and the path detection message includes the information of the virtual port through which the path detection message passes, the detection result message also includes the information of the virtual port corresponding to the network device. Optionally, the network device can also add information about the network device to the mirror message, such as the ID of the network device. After encapsulation is completed, the network device can return the detection result message to the first device. In addition, after completing the copying of the path detection message, the network device can forward the path detection message to the next hop, and the next hop is determined based on the ECMP Hash Fields in the path detection message.

The following is an explanation of the process by which a network device determines the next hop to which a path detection message is sent and forwards it. After receiving a path detection message, the network device can use the information in all or part of the fields in the ECMP Hash Fields corresponding to the path detection message (e.g., the source port information in the source port information field that matches the network device) as a hash key value (HASH KEY) and generate a bucket through a hash algorithm. A bucket can be an array with an index that stores multiple data generated by a hash algorithm. The network device then performs a modulo (MOD) operation on the data in the bucket to obtain the next hop and outbound interface information of the message, and forwards the path detection message to the determined next hop based on the determined outbound interface.

Each network device that the path detection message passes through can determine the next hop and outbound interface information of the path detection message through the above method, until the next hop determined by the last network device that the path detection message passes through is the destination device. Since different network devices extract different hash key values, the next hops determined by different network devices are also different. Therefore, it is possible to determine a path among multiple paths based on the information of different virtual ports to transmit the path detection message.

Optionally, the process in which the first device receives at least one detection result message returned based on the path detection message can be divided into different situations. For example, the first device can receive a detection result message sent directly to the first device by a network device. Alternatively, after generating the detection result message, the network device can send the detection result message to the centralized device. After receiving all detection result messages corresponding to the same path detection message, the centralized device can send all detection result messages corresponding to the same path detection message to the first device, so that the first device obtains all detection result messages corresponding to the path detection message. The method in which each network device through which the path detection message passes directly sends the detection result message to the first device can be called distributed path detection, and the method in which the centralized device concentrates each detection result message and then sends each detection result message to the first device can be called centralized path detection.

Referring to FIG. 7 , a schematic diagram of a path detection process is shown. The first device and the second device include network device 21, network device 22, network device 23, network device 24, network device 11 and network device 12, and the connection relationship between each network device can refer to the connection line in FIG. 7 . As shown in FIG. 7 or FIG. 8 , there are multiple paths based on different network devices between the first device and the second device. After the first device sends a path detection message to the second device, the path detection message can be transmitted based on one of the multiple paths. The path detection message determines a path in the multiple paths to transmit the path detection message. The process can refer to the above description based on FIG. 6 . Taking the transmission of the path detection message along the path shown by the arrow in FIG. 7 as an example, the first device first sends the path detection message to the network device 21. The network device 21 mirrors the received path detection message, and determines that the next hop is the network device 11 according to the information of the virtual port in the path detection message, and forwards the path detection message to the network device 11. The network device 21 adds the information of the physical port of the received path detection message and the information of the network device 21 to the mirror message, completes the encapsulation of the mirror message, obtains the detection result message, and sends the detection result message to the first device. Alternatively, as shown in Figure 9, the detection result message can be sent to the centralized device.

After receiving the path detection message, the network device 11 performs the same steps as the network device 21 to send the determined next hop network device to the network device 21. The network device 12 forwards the path detection message and returns the detection result message to the first device. Afterwards, the network device 12 forwards the path detection message to the determined next-hop network device 24 and returns the detection result message to the first device. After receiving the path detection message, the network device 24 forwards the path detection message to the determined next-hop second device and returns the detection result message to the first device.

In a distributed path detection scenario, the path of the path detection message and the path of the detection result message may be asymmetric, that is, the path detection message and the detection result message may pass through the same or different network devices and port sets. For example, in FIG7, the path detection message reaches network device 24 via the forwarding of network device 21, network device 11, and network device 12, and the detection result message generated by network device 24 may be returned to the first device along the forwarding path of the path detection message, or may be returned to the first device along a path different from that of the path detection message. For example, the detection result message generated by network device 24 may reach the first device via the forwarding of network device 12, network device 11, and network device 22.

The first device may send at least one path detection message to receive at least one detection result message returned by at least one network device based on the at least one path detection message, so as to realize path detection of each path between the first device and the second device. Afterwards, the first device may determine at least one path between the first device and the second device based on the information of the physical port and the virtual port of at least one network device included in the at least one detection result message; the first device selects a first path from the at least one path.

Exemplarily, the first device can determine the physical ports of each network device between the first device and the second device, and the communication connection relationship between the physical ports of each network device, based on the information of the physical ports included in each acquired detection result message, so as to determine at least one path between the first device and the second device. In addition, the first device can also determine the correspondence between the information of each virtual port and the information of the physical port based on at least one detection result message.

After the first device selects the first path from at least one path, the network device on the first path is determined as the first network device, information of the virtual ports corresponding to the physical ports of each of the first network devices is determined and added to the first message, and then the first message is sent, so that the first message can be forwarded according to the first path based on the information of the virtual ports corresponding to the physical ports of each of the first network devices.

In case A1, the process of performing path detection based on the path detection message can be called the initial connection establishment process or the initial connection establishment stage. The first device selects multiple candidate virtual ports and adds different virtual port information to different path detection messages, so that different path detection messages can be transmitted along different paths. If there are N paths between the first device and the second device, path detection can be carried out by selecting information of more than N candidate virtual ports to achieve path detection with full coverage of the paths between the first device and the second device.

In case A2, the first device determines the first path after sending the first message. In a possible implementation, the first message may include second indication information, and the second indication information indicates that the second message is returned based on the first message.

In this case, the first device can receive at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and information about the first virtual port of the first network device that sends any second message; the first device determines the first path based on the information about the first physical port and information about the first virtual port of at least one first network device included in the at least one second message, and the first path includes the first device, the second device, and at least one first network device through which the first message passes.

The second indication information may be the same as the first indication information in case A1, and the second indication information may also include an identifier with an indication function, or be information including the first virtual port of at least one first network device on the first path. In this implementation, the second message is equivalent to the detection result message in case A1. In addition to being used for delay measurement, the first message can also perform path detection, that is, the first message is given the ability of path detection to achieve the synchronization of path detection and delay measurement. If the second indication information is the information of the first virtual port of at least one first network device, the first message may be a unicast message forwarded based on the first path.

The process of the first device receiving at least one second message can refer to the process of the first device receiving at least one detection result message in situation A1, which will not be repeated here. After the first device receives at least one second message, it can determine each first network device that the first message passes through, that is, determine the first path of the first message, and determine the correspondence between the information of the physical port of each network device and the information of the virtual port.

In a possible implementation, situation A1 can be combined with situation A2, that is, before sending the first message, the first device performs path detection to determine the paths between the first device and the second device, and actively adds the information of the first virtual port corresponding to the first physical port of each first network device and the second indication information to the first message, completes the first path marking, and makes the first message a unicast message. The first message is transmitted along the first path, and after receiving the first message, each first network device on the first path checks whether the virtual port information in the first message includes the information of its own virtual port. If it does, the first network device mirrors the first message and encapsulates it to obtain the second message, completes the second path marking, and sends the mirror message back to the first device, so that the first device determines again whether the network device passing through is the first network device on the first path.

Optionally, in this implementation, the first message may also include an inband network telemetry header (INT header) or, after receiving the first message, the first network device may also combine INT or in-situ flow information telemetry (IFIT) technology to directly add metadata such as information about the physical port of the first network device, the ID of the first network device, information about the physical port of the first network device receiving the first message, and information about the physical port of the first network device sending the first message to the first message. (metadata, MD) and timestamp (timestamp, TS) are used to implement secondary marking of the path and marking of the hop-by-hop delay. The first network device can send the second-marked first message to the determined next hop until the first message is transmitted to the second device. The second device then returns the first message to the first device, so that the first device determines whether the network device through which the first message passes is a network device on the first path, and further clarifies the transmission path of the first message. Alternatively, the second device, as a tail hop server, can report the first message that has undergone secondary marking to a centralized device, and the centralized device analyzes each mark in the first message.

S502, the first device obtains a first delay of a first path determined based on a first timestamp and a second timestamp, where the first timestamp is a timestamp of sending a first message recorded by a network card on the first device, and the second timestamp is a timestamp of receiving the first message recorded by a network card on the second device.

In the embodiment of the present application, different devices may determine the first delay according to the first timestamp and the second timestamp. Depending on the different devices determining the first delay, the situation in which the first device obtains the first delay may be divided into multiple different situations, including but not limited to the following situation B1 and situation B2.

In case B1, the first delay is calculated autonomously by the first device.

In this case, the first device receives a message including a second timestamp sent by the centralized device or the second device based on the first message, obtains the second timestamp from the message including the second timestamp, and determines the first delay of the first path according to the first timestamp and the second timestamp.

After receiving the first message, the second device records the second timestamp of receiving the first message through the network card on the second device. After the second device generates the message including the second timestamp, the second device can directly send the message including the second timestamp to the first device, or send the message including the second timestamp to the centralized device, and the centralized device forwards the message including the second timestamp to the first device.

After receiving the message including the second timestamp, the first device may obtain the second timestamp from the message including the second timestamp, and determine the first path of the first delay according to the first timestamp and the second timestamp.

Optionally, the first delay may be a difference between the second timestamp and the first timestamp. The first timestamp and the second timestamp may be standard timestamps or non-standard timestamps. If the first timestamp and the second timestamp are standard timestamps, the first device may determine the difference between the second timestamp and the first timestamp as the first delay of the first path. If at least one of the first timestamp and the second timestamp is a non-standard timestamp, the standard timestamp corresponding to the non-standard timestamp may be calculated based on the difference between the non-standard timestamp and the standard timestamp, and then the first delay may be determined based on the standard timestamp corresponding to the non-standard timestamp.

Alternatively, in an embodiment of the present application, it is also possible to ignore whether the first timestamp and the second timestamp are standard timestamps. Since the difference between the timestamp recorded by the network card on the first device and the standard timestamp is certain, the difference between the timestamp recorded by the network card on the second device and the standard timestamp is also certain. Therefore, for different paths, the difference between the delay directly determined according to the first timestamp and the second timestamp and the delay determined according to the standard timestamp is the same, that is, the relative size of the delays corresponding to different paths will not change due to whether the first timestamp and the second timestamp are standard, and the accuracy of path selection directly based on the delay determined based on the first timestamp and the second timestamp will not be affected.

Case B2: the first delay is calculated by the second device or the centralized device.

In this case, the first device sends a message including a first timestamp to the second device or the centralized device. The first timestamp is used by the second device or the centralized device to determine the first delay of the first path and receive the message including the first delay sent by the second device or the centralized device. The first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp. The first device obtains the first delay from the message including the first delay.

After sending the first message to the second device, the first device may send a message including the first timestamp to the second device, so that the second device can determine the first delay of the first path according to the second timestamp and the first timestamp recorded by the network card on the second device. Alternatively, after sending the first message to the second device, the first device may send a message including the first timestamp to the centralized device, and after receiving the first message, the second device may send a message including the second timestamp to the centralized device, so that the centralized device can obtain both the first timestamp and the second timestamp, and determine the first delay of the first path according to the first timestamp and the second timestamp.

In a possible implementation, the message including the first timestamp sent by the first device to the second device may also be the first message itself, that is, after recording the first timestamp, the network card on the first device may mark the first timestamp in the first message, so that when the second device receives the first message, the second device can obtain the first timestamp. Afterwards, the second device may send the first timestamp and the second timestamp to the centralized device, so that the centralized device can obtain the first timestamp and the second timestamp, and then the centralized device determines the first delay according to the first timestamp and the second timestamp.

After the central device or the second device determines the first delay, a message including the first delay may be sent to the first device, so that the first device can acquire the first delay from the message including the first delay.

Optionally, after the first device obtains the first delay of the first path determined based on the first timestamp and the second timestamp, the first device may also receive a third message sent by the second device based on the first message, where the transmission path of the third message is the second path.

The process of the second device sending the third message to the first device and the process of the first device determining the second path can refer to the process of the first device sending the first message to the second device and the process of the first device determining the first path in the above S501, which will not be repeated here.

The first device obtains a second delay of the second path determined based on a third timestamp and a fourth timestamp, where the third timestamp is a timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is a timestamp of sending the third message recorded by the network card on the second device.

When receiving the third message, the first device may record a third timestamp of receiving the third message through the network card on the first device.

In the embodiment of the present application, different devices can determine the second delay according to the third timestamp and the fourth timestamp. According to different devices for determining the second delay, the situation in which the first device obtains the second delay can be divided into multiple different situations, including but not limited to the following situation C1 and situation C2.

In case C1, the second delay is calculated autonomously by the first device.

In this case, the first device receives a message including a fourth timestamp sent by the centralized device or the second device based on the third message, obtains the fourth timestamp, and determines the second delay of the second path according to the third timestamp and the fourth timestamp.

Among them, the process of the second device or centralized device sending a message including the fourth timestamp to the first device can refer to the process of the second device or centralized device sending a message including the second timestamp to the first device in situation B1, and the process of the first device determining the second delay according to the third timestamp and the fourth timestamp can refer to the process of the first device determining the first delay according to the first timestamp and the second timestamp in situation B1, which will not be repeated here.

In case C2, the second delay is calculated by the second device or the centralized device.

In this case, the first device sends a message including a third timestamp to the second device or the centralized device, and the third timestamp is used by the second device or the centralized device to determine the second delay of the second path; and receives a message including the second delay sent by the second device or the centralized device, and the second delay is determined by the second device or the centralized device based on the third timestamp and the fourth timestamp; the first device obtains the second delay from the message including the second delay.

In this case, the process of the first device acquiring the second delay can refer to the process of the first device acquiring the first delay in case B1, which will not be repeated here.

In an embodiment of the present application, N unicast virtual ports can be selected, virtual port information can be added to each first message, and unicast normalized delay measurement can be performed on each path between the first device and the second device according to the above-mentioned situation B1, situation B2, situation C1 and situation C2, so as to achieve full coverage of accurate delay measurement. The normalized delay measurement can be a periodic measurement of a fixed frequency. The embodiment of the present application does not limit the size of the fixed frequency. The size of the fixed frequency can be statically configured or dynamically adjusted according to the status of each device.

In a possible implementation, the correspondence between the information of the physical port and the information of the virtual port of the network device on each path between the first device and the second device may change. Taking the change in the correspondence between the information of the physical port and the information of the virtual port of the first network device A on the first path as an example, the first device again sends the first message for measuring the first delay of the first path to the second device. Since the information of the virtual port in the first message is the information of the virtual port of the first network device A before the change in the correspondence, the first message cannot be transmitted along the first path. Therefore, it is necessary to update the first message according to the changed virtual port information so that the updated message can continue to be transmitted along the first path. In addition, the first device can also determine whether it needs to be re-established based on the connection correlation between the path to be newly connected and the existing path. If necessary, the virtual port information is replaced to re-establish the connection with the second device to achieve adaptive reconstruction.

Exemplarily, for any one of the at least one first network devices, the first device determines the information of the second virtual port corresponding to the first physical port of any one of the first network devices. The embodiment of the present application does not limit the manner in which the first device determines the information of the second virtual port. Exemplarily, the first device may first determine the information of the virtual port that is close or similar to the information of the first virtual port corresponding to the first physical port of the first network device. For example, if the information of the first virtual port is A1, the information of the virtual port that is close to the information of the first virtual port may be A2 or A3, etc. Afterwards, the first device may update the first message according to the information of the virtual port that is close to the information of the first virtual port, and send the updated first message to the second device to implement the full coverage path detection of a small range corresponding to the first network device, and determine the information of the second virtual port that enables the updated first message to be transmitted based on the first path based on the result of the path detection. Optionally, the information of the second virtual port may be selected from the information of multiple virtual ports that enable the updated first message to be transmitted based on the first path based on the correlation between the information of each virtual port and the information of the first virtual port, and the correlation between the information of the second virtual port and the information of the first port is less than the correlation threshold. Among them, the correlation threshold may be set based on experience.

Afterwards, when the connection used by the service corresponding to the first message is abnormal, that is, because the correspondence between the first virtual port and the first physical port of the first network device on the first path changes, the first message cannot be forwarded normally along the first path, the first device can update the first message based on the information of the second virtual port to obtain an updated message, and the updated message includes the information of the second virtual port. The first device sends the updated message to the second device, so that the transmission path of the updated message is the first path. Then the first device can obtain the information based on the fifth time. The third delay of the first path is determined by the fifth timestamp and the sixth timestamp, wherein the fifth timestamp is the timestamp of sending the updated message recorded by the network card on the first device, and the sixth timestamp is the timestamp of receiving the updated message recorded by the network card of the second device.

The process in which the first device acquires the third delay determined based on the fifth timestamp and the sixth timestamp may refer to the process in which the first device acquires the first delay determined based on the first timestamp and the second timestamp, which will not be described in detail here.

The first device sends the updated message to the second device to switch from the old connection to the new connection, thereby switching the flow path. The new connection is a connection selected based on the correlation with the old connection, bypassing the abnormal point or selecting a connection with low correlation, that is, selecting the information of the second virtual port with low similarity to the information of the first virtual port.

Referring to FIG. 10 , another implementation scenario of the delay measurement method provided in an embodiment of the present application is shown, and the implementation scenario is used for clock synchronization. The first device may be a master device, and the second device may be a slave device. The first device sends a synchronization (synchronization, sync) message to the second device, the synchronization message is the above-mentioned first message, and records the first timestamp t1' of sending the synchronization message through the network card. Afterwards, the first device sends a follow-up message to the second device, and the follow-up message is the message including the first timestamp described above.

The second device receives the synchronization message and records the second timestamp t2’ through the network card. The second device receives the follow-up message to obtain the first timestamp t1’. The second device sends a delay request message to the first device, and records the fourth timestamp t3’ of sending the delay request message through the network card. The delay request message is the third message mentioned above. The first device records the third timestamp t4’ of receiving the delay request message through the network card, and sends a delay response message to the second device. The delay response message is the message including the third timestamp mentioned above.

After acquiring the four timestamps, the second device calculates the round trip delay including the first delay and the second delay, wherein the difference between t1' and the standard timestamp t1 is Δtm, the difference between t2' and the standard timestamp t2 is Δts, the difference between t3' and the standard timestamp t3 is Δts, and the difference between t4' and the standard timestamp t4 is Δtm.

Round trip delay: (t2’-t1’)+(t4’-t3’)=(t2-t1)+(Δts-Δtm)+(t4-t3)+(Δtm-Δts)=(t2-t1)+(t4-t3).

In addition, the delay measured by the delay measurement method provided in the embodiment of the present application can also be used in scenarios such as troubleshooting, high performance computing (HPC), high performance distributed storage, big data development, artificial intelligence (AI), and traffic scheduling. For example, in a traffic scheduling scenario, the average or variance of the delay of each path can be calculated based on the delay obtained by the first device, and a path recommendation can be made based on a set threshold.

Below, the beneficial effects of the embodiments of the present application are described in combination with related technologies.

Related technology 1 is a method for measuring latency based on a ping network. Referring to FIG11 , a schematic diagram of a ping network software architecture is shown. The software architecture includes a ping network controller (pingmesh controller), a ping network agent (pingmesh agent) and a data storage and analysis (DSA). The ping network controller is connected to the resource management system, and the ping network generator (pingmesh generator) generates a ping list (pinglist file) based on the network graph obtained from the resource management system, so that the servers in the data center perform network probing based on the ping list to ensure the normal provision of network services (web services). The ping list includes a probing list of each server in the data center and a virtual internet protocol address (VIP) of each server. The format of the ping list can be, for example, an extensible markup language (XML) format.

The ping network agent runs on each server in the data center. For example, the ping network agent runs on both server 1 and server 2 shown in FIG11. Taking the ping network agent running on server 2 as an example, the ping network agent obtains the ping list from the ping network controller, determines the servers that need to detect each other with server 2 and the VIPs of the servers that need to detect each other with server 2 based on the obtained ping list, and detects according to the server list. For example, if server 2 needs to detect each other with server 1, server 2 can send a detection request to server 1 based on the VIP of server 1. The detection request can be a detection request based on the transmission control protocol (TCP) or hypertext transfer protocol (HTTP). Server 1 and server 2 send detection requests to each other to measure the delay and packet loss between servers. The ping network agent running on the server generates a log based on the detection results and uploads the log to DSA. DSA stores the log in blockchain according to the VIP of each server. DSA stores the log stored in the blockchain in the database through range operations. In addition, the ping network agent can also report the detection results to the performance counter aggregator in the DSA, so that the performance counter aggregator can monitor the network status. The performance counter aggregator can also run big data analysis algorithms to analyze and process the measurement data for generating data sheets or for fault detection and fault location, realize network status visualization, and can provide timely alerts when network failures occur.

Since there are many servers in a data center, if any two servers in the data center detect each other, it will bring great pressure to the ping network agent. Therefore, the related technology is to measure the latency through hierarchical control. For example, within the POD, servers detect each other; between PODs within the DC, servers within PODm or PODn are sorted, and servers at corresponding positions in two PODs detect each other; between DCs, a detection relationship is established between the selected servers within the DC; N servers are selected in each DC and sorted. In the DC, at least two machines can be selected in each POD. Finally, the controller controls one server to establish a detection relationship with up to 5,000 servers to prevent the server where the Ping network agent is located from being overloaded.

The global operation status of the DC network is shown in Figure 12: the horizontal and vertical axes are servers, and Figure 12 is the data matrix measured between servers. The blank rectangle represents that the detection result is empty, the rectangle with a check mark represents that the detection result is normal, and the rectangle with a cross represents that the detection result is abnormal.

(a) in Figure 12 is the data matrix under normal operating conditions of the DC; (b) is an overall failure of a Podset (for example, power off), resulting in data loss; (c) is a Podset failure that causes the delay of a Podset to exceed the standard, which is generally manifested as packet loss or broadcast storm in the leaf switch, resulting in packet loss and delay; (d) is a spine switch failure that causes spine switch packet loss, affecting the entire DC, but the internal communication of the Podset is normal.

Since the delay measured by the related technology is not only the end-to-end network delay, but also includes the processing delay on the stack and the delay caused by the CPU parsing the message, it is impossible to judge whether the service anomaly is caused by the delay increase caused by the network anomaly based on the delay measured by the related technology. In addition, the reliability monitoring of related technology one relies on the controller and a large number of analyzers. The network is invisible to the service nodes and is a black box for the service. It only relies on a large number of probes to improve the coverage, and the actual coverage is unknown. A large amount of measurement data is used to infer the fault, and the analysis is complex. It is impossible to accurately determine the fault and there are misjudgments; the fault location time is long. Related technology one This dial-up test solution, because the network is treated as a black box, blind testing requires a lot of overhead (bandwidth and computing resources) to construct more probes to improve the coverage of paths, links and forwarding behaviors, and it is difficult to find problems in time.

Related Technology 2 believes that input-output (IO) latency is a key factor affecting storage services, and is also a high-performance service requirement for customers, distinguishing between high, medium and low storage service core indicators. Related Technology 2 perceives the end-to-end round-trip time (RTT) on the storage service node, and continuously measures the RTT of the end-to-end connection through the service layer. The end side distinguishes the load of the path based on the RTT, and distributes more traffic to the low RTT path, thereby ensuring the average latency and tail latency of IO. The RTT measured between nodes includes the protocol stack latency, which has large errors and inaccurate adjustments, resulting in inaccurate service path switching and affecting service performance.

Related technology 3: By deploying INT on network nodes, the network nodes measure the delay of each hop and sense the path of the service flow in the network. The end side distinguishes the load of the path based on the measured end-to-end network delay, and distributes more traffic to the low-latency path, thereby ensuring the average delay and tail delay of IO. However, the in-band measurement technology INT is used, and the existing devices do not support INT. INT also requires the identification and mirroring of service or measurement traffic, which occupies a large network overhead and is actually difficult to deploy in the existing network.

In the embodiment of the present application, the first delay determined by the hardware timestamp recorded by the network card is the network delay on the first path between the first device and the second device, which can avoid the long tail delay caused by the end processing delay of the first device and the second device, and can realize the binding of the delay and the path, and realize the visualization of the delay.

Exemplarily, the embodiment of the present application can sense multiple paths between the first device and the second device, and perform normalized delay measurement on each path based on the hardware timestamp recorded by the network card, without relying on the hop-by-hop delay generated by the network device, and can achieve end-to-end accurate delay measurement. Therefore, the ability to record timestamps through the network card in the embodiment of the present application only requires the support of the network card on the server, and does not require the modification of the network device. It has high deployability and can eliminate the delay caused by the protocol stack and the central processing unit (CPU) parsing the message on the end. In addition, the embodiment of the present application combines the two capabilities of network delay measurement and path perception, and can achieve full coverage delay measurement of network paths at a relatively low cost through multiple unicast streams. In addition, it can more accurately allocate network traffic, improve service performance, and reduce long-tail delays caused by network traffic queuing.

The above describes the delay measurement method provided by the embodiment of the present application. Corresponding to the above method, the embodiment of the present application also provides a delay measurement device. The device is applied to a first device. The device is used to execute the delay measurement method performed by the first device in FIG. 5 through the modules shown in FIG. 13. As shown in FIG. 13, the delay measurement device provided by the embodiment of the present application includes the following modules.

The transceiver module 1301 is used to send a first message to a second device, and the transmission path of the first message is the first path; the processing module 1302 is used to obtain a first delay of the first path determined based on a first timestamp and a second timestamp, the first timestamp is the timestamp of sending the first message recorded by the network card on the first device, and the second timestamp is the timestamp of receiving the first message recorded by the network card on the second device.

In one possible implementation, the transceiver module 1301 is also used to send a path detection message to the second device, the path detection message includes first indication information, and the first indication information indicates that a detection result message is returned based on the path detection message; receive at least one detection result message returned based on the path detection message, the detection result message includes information about the physical port and virtual port of the network device that sent the detection result message; the processing module 1302 is also used to determine at least one path between the first device and the second device based on information about the physical port and virtual port of at least one network device included in at least one detection result message; and select a first path from the at least one path.

In a possible implementation, the first message includes second indication information, and the second indication information indicates that the second message is returned based on the first message; the transceiver module 1301 is further used to receive at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and the first virtual port of the first network device that sends any second message; the processing module 1302 is also used to receive at least one second message sent by at least one first network device through which the first message passes, and any second message includes information about the first physical port and the first virtual port of the first network device that sends any second message. The first path is determined based on the information of the virtual port, where the first path includes the first device, the second device, and at least one first network device through which the first message passes.

In a possible implementation, the processing module 1302 is used to receive a message including a second timestamp sent by a centralized device or a second device based on the first message, obtain the second timestamp from the message including the second timestamp; and determine a first delay of the first path according to the first timestamp and the second timestamp.

In one possible implementation, the processing module 1302 is used to send a message including a first timestamp to a second device or a centralized device, where the first timestamp is used by the second device or the centralized device to determine a first delay of a first path; receive a message including a first delay sent by the second device or the centralized device, where the first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp; and obtain the first delay from the message including the first delay.

In one possible implementation, the transceiver module 1301 is also used to receive a third message sent by the second device based on the first message, and the transmission path of the third message is the second path; the processing module 1302 is also used to obtain a second delay of the second path determined based on a third timestamp and a fourth timestamp, the third timestamp is the timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is the timestamp of sending the third message recorded by the network card on the second device.

In a possible implementation, the processing module 1302 is configured to receive a message including a fourth timestamp sent by the centralized device or the second device based on the third message, obtain the fourth timestamp, and determine the second delay of the second path according to the third timestamp and the fourth timestamp.

In one possible implementation, the processing module 1302 is used to send a message including a third timestamp to a second device or a centralized device, where the third timestamp is used by the second device or the centralized device to determine a second delay of a second path; receive a message including a second delay sent by the second device or the centralized device, where the second delay is determined by the second device or the centralized device based on the third timestamp and a fourth timestamp; and obtain the second delay from the message including the second delay.

In one possible implementation, the processing module 1302 is also used to determine, for any one of the at least one first network devices, information of a second virtual port corresponding to a first physical port of any one of the first network devices; update the first message based on the information of the second virtual port to obtain an updated message, and the updated message includes information of the second virtual port; the transceiver module 1301 is also used to send the updated message to the second device, and the transmission path of the updated message is the first path; the processing module 1302 is also used to obtain a third delay of the first path determined based on a fifth timestamp and a sixth timestamp, the fifth timestamp being the timestamp of sending the updated message recorded by the network card on the first device, and the sixth timestamp being the timestamp of receiving the updated message recorded by the network card of the second device.

It should be understood that the beneficial effects of the device provided in FIG. 13 when implementing its functions are the same as the beneficial effects of the method provided in FIG. 5, which will not be described in detail here. In addition, when the device provided in FIG. 13 implements its functions, only the division of the above-mentioned functional modules is used as an example. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

Referring to FIG. 14 , FIG. 14 shows a schematic structural diagram of an exemplary delay measurement device 1400 of the present application. The delay measurement device 1400 includes at least one processor 1401 , a memory 1403 , and at least one network interface 1404 .

Processor 1401 is, for example, a general-purpose central processing unit, a digital signal processor (DSP), a network processor (NP), a GPU, a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs) for implementing the solution of the present application, a programmable logic device (PLD), other general-purpose processors or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, or any combination thereof. PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the advanced RISC machines (ARM) architecture. It may implement or execute various logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

Optionally, the delay measurement device 1400 further includes a bus 1402. The bus 1402 is used to transmit information between the components of the delay measurement device 1400. The bus 1402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 1402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, FIG. 14 is represented by only one line, but it does not mean that there is only one bus or one type of bus.

The memory 1403 is, for example, a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache.

By way of example and not limitation, many forms of ROM and RAM are available. For example, ROM is a compact disc read-only memory (CD-ROM). RAM includes, but is not limited to, static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).

The memory 1403 may also be other types of storage devices that can store static information and instructions. Or it may be other types of dynamic storage devices that can store information and instructions. Or it may be other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, blue-ray disk, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by the computer, but is not limited to this. The memory 1403, for example, exists independently and is connected to the processor 1401 through the bus 1402. The memory 1403 may also be integrated with the processor 1401.

The network interface 1404 uses any transceiver-like device to communicate with other devices or communication networks, and the communication network can be Ethernet, radio access network (RAN) or wireless local area network (WLAN), etc. The network interface 1404 may include a wired network interface and a wireless network interface. Specifically, the network interface 1404 may be an Ethernet interface, such as a Fast Ethernet (FE) interface, a Gigabit Ethernet (GE) interface, an Asynchronous Transfer Mode (ATM) interface, a WLAN interface, a cellular network interface or a combination thereof. The Ethernet interface may be an optical interface, an electrical interface or a combination thereof. In some embodiments of the present application, the network interface 1404 may be used for the delay measurement device 1400 to communicate with other devices.

In a specific implementation, as some embodiments, the processor 1401 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG14. Each of these processors may be a single-core processor or a multi-core processor. The processor here may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

In a specific implementation, as some implementation methods, the delay measurement device 1400 may include multiple processors, such as the processor 1401 and the processor 1405 shown in FIG14. Each of these processors may be a single-core processor or a multi-core processor. The processor here may refer to one or more devices, circuits, and/or processing cores for processing data (such as computer program instructions).

In some implementations, the memory 1403 is used to store program instructions 1410 for executing the solution of the present application, and the processor 1401 can execute the program instructions 1410 stored in the memory 1403. That is, the delay measurement device 1400 can implement the method provided by the method embodiment, that is, the method of FIG. 5, through the processor 1401 and the program instructions 1410 in the memory 1403. The program instructions 1410 may include one or more software modules. Optionally, the processor 1401 itself may also store program instructions for executing the solution of the present application.

In the specific implementation process, the delay measurement device 1400 of the present application may correspond to a first network element device for executing the above method. The processor 1401 in the delay measurement device 1400 reads the instructions in the memory 1403, so that the delay measurement device 1400 shown in Figure 14 can execute all or part of the steps in the method embodiment.

Among them, each step of the method shown in Figure 5 is completed by the hardware integrated logic circuit or software instructions in the processor of the delay measurement device 1400. The steps of the method embodiment disclosed in the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method embodiment in combination with its hardware. To avoid repetition, it will not be described in detail here.

Referring to FIG. 15 , FIG. 15 shows a schematic diagram of the structure of an exemplary delay measurement device 1300 of the present application. The delay measurement device 1300 includes: a main control board 1310 and an interface board 1330. The delay measurement device 1300 shown in FIG. 15 is used to perform the operations involved in the delay measurement method shown in FIG. 5 above. The delay measurement device 1300 is, for example, a switch, a router, a controller, etc.

The main control board 1310 is also called a main processing unit (MPU) or a route processor card. The main control board 1310 is used to control and manage various components in the delay measurement device 1300, including route calculation, device management, device maintenance, and protocol processing functions. The main control board 1310 includes: a central processing unit 1311 and a memory 1312.

The interface board 1330 is also called a line processing unit (LPU), a line card or a service board. The interface board 1330 is used to provide various service interfaces and implement data packet forwarding. The service interfaces include but are not limited to Ethernet interfaces, POS (Packet over Ethernet), and The interface board 1330 includes: a central processor 1331, a network processor 1332, a forwarding table memory 1334 and a physical interface card (PIC) 1333.

The central processor 1331 on the interface board 1330 is used to control and manage the interface board 1330 and communicate with the central processor 1311 on the main control board 1310 .

The network processor 1332 is used to implement the forwarding processing of the message. The network processor 1332 can be in the form of a forwarding chip. Specifically, the network processor 1332 is used to forward the received message based on the forwarding table stored in the forwarding table entry memory 1334. If the destination address of the message is the address of the delay measurement device 1300, the message is sent to the CPU (such as the central processor 1311) for processing; if the destination address of the message is not the address of the delay measurement device 1300, the next hop and the output interface corresponding to the destination address are found from the forwarding table according to the destination address, and the message is forwarded to the output interface corresponding to the destination address. Among them, the processing of the uplink message includes: processing of the message input interface, forwarding table search; processing of the downlink message: forwarding table search, etc.

The physical interface card 1333 is used to implement the connection function of the physical layer, and the original traffic enters the interface board 1330 from it, and the processed message is sent from the physical interface card 1333. The physical interface card 1333 is also called a daughter card, which can be installed on the interface board 1330 and is responsible for converting the optical signal into a message and forwarding the message to the network processor 1332 for processing after checking the legitimacy of the message. In some embodiments, the central processor can also perform the functions of the network processor 1332, such as implementing software forwarding based on a general-purpose CPU, so that the network processor 1332 is not required in the physical interface card 1333.

Optionally, the delay measurement device 1300 includes multiple interface boards. For example, the delay measurement device 1300 further includes an interface board 1340 . The interface board 1340 includes a central processor 1341 , a network processor 1342 , a forwarding table entry memory 1344 , and a physical interface card 1343 .

Optionally, the delay measurement device 1300 further includes a switching fabric board 1320. The switching fabric board 1320 may also be referred to as a switch fabric unit (SFU). In the case where the delay measurement device has multiple interface boards 1330, the switching fabric board 1320 is used to complete data exchange between the interface boards. For example, the interface board 1330 and the interface board 1340 may communicate through the switching fabric board 1320.

The main control board 1310 and the interface board 1330 are coupled. For example, the main control board 1310, the interface board 1330, the interface board 1340, and the switching network board 1320 are connected to the system backplane through the system bus to achieve intercommunication. In a possible implementation, an inter-process communication (IPC) channel is established between the main control board 1310 and the interface board 1330, and the main control board 1310 and the interface board 1330 communicate through the IPC channel.

Logically, the delay measurement device 1300 includes a control plane and a forwarding plane. The control plane includes a main control board 1310 and a central processing unit 1331. The forwarding plane includes various components for performing forwarding, such as a forwarding table entry memory 1334, a physical interface card 1333, and a network processor 1332. The control plane performs functions such as a router, generating a forwarding table, processing signaling and protocol messages, and configuring and maintaining the status of the device. The control plane sends the generated forwarding table to the forwarding plane. On the forwarding plane, the network processor 1332 forwards the message received by the physical interface card 1333 based on the forwarding table sent by the control plane. The forwarding table sent by the control plane can be stored in the forwarding table entry memory 1334. In some embodiments, the control plane and the forwarding plane can be completely separated and not on the same device.

It is worth noting that there may be one or more main control boards, and when there are multiple boards, they may include a primary main control board and a backup main control board. There may be one or more interface boards. The stronger the data processing capability of the delay measurement device, the more interface boards are provided. There may also be one or more physical interface cards on the interface board. There may be no switching network board, or there may be one or more switching network boards. When there are multiple switching network boards, they can jointly realize load sharing and redundant backup. In a centralized forwarding architecture, the delay measurement device may not need a switching network board, and the interface board is responsible for the processing function of the service data of the entire system. In a distributed forwarding architecture, the delay measurement device may have at least one switching network board, and the switching network board is used to realize data exchange between multiple interface boards, providing large-capacity data exchange and processing capabilities. Therefore, the data access and processing capabilities of the delay measurement device with a distributed architecture are greater than those of the device with a centralized architecture. Optionally, the delay measurement device may also be in the form of only one board, that is, without a switching network board, and the functions of the interface board and the main control board are integrated on the board. In this case, the central processor on the interface board and the central processor on the main control board can be combined into one central processor on the board to perform the functions of the two superimposed. The data exchange and processing capabilities of this type of device are relatively low (for example, low-end switches or communication devices such as routers). The specific architecture to be adopted depends on the specific networking deployment scenario, and no limitation is made here.

In an exemplary embodiment, a delay measurement system is provided. The system includes a first device, a second device, and a network device. The first device is used to execute the method executed by the first device in FIG. 5 .

In an exemplary embodiment, a computer program (product) is provided. The computer program (product) includes: computer program code. When the computer program code is executed by a computer, the computer executes the method shown in FIG. 5 .

In an exemplary embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program or an instruction. When the program or the instruction is executed on a computer, the computer executes the method shown in FIG. 5 .

In an exemplary embodiment, a chip is provided, including a processor for calling from a memory and executing instructions stored in the memory, so that a computer equipped with the chip executes the method shown in FIG. 5 .

In an exemplary embodiment, another chip is provided, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, a computer equipped with the chip executes the method shown in FIG. 5 .

In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk), etc.

In this application, the terms "first", "second", etc. are used to distinguish the same or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there a limit on the quantity and execution order. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another element.

It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

The term "at least one" in this application means one or more, and the term "multiple" in this application means two or more, for example, multiple second devices means two or more second devices. The terms "system" and "network" are often used interchangeably herein.

It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should also be understood that the term "and/or" used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "and/or" is a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character "/" in this application generally indicates that the associated objects before and after are in an "or" relationship.

It should also be understood that the terms "if" and "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

The above description is only an embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims (13)

A delay measurement method, characterized in that the method comprises: The first device sends a first message to the second device, where a transmission path of the first message is a first path; The first device obtains a first delay of the first path determined based on a first timestamp and a second timestamp, wherein the first timestamp is a timestamp of sending the first message recorded by the network card on the first device, and the second timestamp is a timestamp of receiving the first message recorded by the network card on the second device. The method according to claim 1, characterized in that before the first device sends the first message to the second device, it also includes: The first device sends a path detection message to the second device, where the path detection message includes first indication information, and the first indication information indicates to return a detection result message based on the path detection message; The first device receives at least one detection result message returned based on the path detection message, wherein the detection result message includes information about a physical port and information about a virtual port of a network device that sends the detection result message; The first device determines at least one path between the first device and the second device based on information about a physical port and a virtual port of at least one network device included in the at least one detection result message; The first device selects the first path among the at least one path. The method according to claim 1, characterized in that the first message includes second indication information, and the second indication information indicates that a second message is returned based on the first message; Before the first device acquires the first delay of the first path determined based on the first timestamp and the second timestamp, the method further includes: The first device receives at least one second message sent by at least one first network device through which the first message passes, wherein any second message includes information about a first physical port and information about a first virtual port of the first network device that sends the any second message; The first device determines the first path based on information about a first physical port and a first virtual port of at least one first network device included in the at least one second message, wherein the first path includes the first device, the second device and the at least one first network device through which the first message passes. The method according to any one of claims 1 to 3, characterized in that the first device obtains a first delay of the first path determined based on the first timestamp and the second timestamp, comprising: The first device receives a message including the second timestamp sent by the centralized device or the second device based on the first message, and obtains the second timestamp from the message including the second timestamp; The first device determines a first delay of the first path according to the first timestamp and the second timestamp. The method according to any one of claims 1 to 3, characterized in that the first device obtains a first delay of the first path determined based on the first timestamp and the second timestamp, comprising: The first device sends a message including the first timestamp to the second device or the centralized device, where the first timestamp is used by the second device or the centralized device to determine a first delay of the first path; The first device receives a message including the first delay sent by the second device or the centralized device, where the first delay is determined by the second device or the centralized device based on the first timestamp and the second timestamp; The first device obtains the first delay from the message including the first delay. The method according to any one of claims 1 to 5, characterized in that after the first device obtains the first delay of the first path determined based on the first timestamp and the second timestamp, it also includes: The first device receives a third message sent by the second device based on the first message, where a transmission path of the third message is a second path; The first device obtains a second delay of the second path determined based on a third timestamp and a fourth timestamp, wherein the third timestamp is a timestamp of receiving the third message recorded by the network card on the first device, and the fourth timestamp is a timestamp of sending the third message recorded by the network card on the second device. The method according to claim 6, characterized in that the first device obtains the second delay of the second path determined based on the third timestamp and the fourth timestamp, comprising: The first device receives a message including the fourth timestamp sent by the centralized device or the second device based on the third message, and acquires the fourth timestamp; The first device determines a second delay of the second path according to the third timestamp and the fourth timestamp. The method according to claim 6, characterized in that the first device obtains the second delay of the second path determined based on the third timestamp and the fourth timestamp, comprising: The first device sends a message including the third timestamp to the second device or the centralized device, where the third timestamp is used by the second device or the centralized device to determine the second delay of the second path; The first device receives a message including the second delay sent by the second device or the centralized device, where the second delay is determined by the second device or the centralized device based on the third timestamp and the fourth timestamp; The first device obtains the second delay from the message including the second delay. The method according to any one of claims 1 to 8, characterized in that the method further comprises: For any one of the at least one first network devices, the first device determines information of a second virtual port corresponding to a first physical port of the any one of the first network devices; The first device updates the first message based on the information of the second virtual port to obtain an updated message, where the updated message includes the information of the second virtual port; The first device sends the updated message to the second device, and the transmission path of the updated message is the first path; The first device obtains a third delay of the first path determined based on a fifth timestamp and a sixth timestamp, wherein the fifth timestamp is a timestamp recorded by the network card on the first device for sending the updated message, and the sixth timestamp is a timestamp recorded by the network card of the second device for receiving the updated message. A delay measurement device, characterized in that the device comprises: A transceiver module, configured to perform the reception and/or transmission related operations performed by the first device in the method according to any one of claims 1 to 9; A processing module, used to perform other operations other than the reception and/or transmission related operations performed by the first device in any one of the methods described in claims 1-9. A delay measurement device, characterized in that the device includes a processor, which is coupled to a memory; the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor, so that the delay measurement device implements the delay measurement method described in any one of claims 1-9. A computer-readable storage medium, characterized in that at least one instruction is stored in the computer-readable storage medium, and the instruction is loaded and executed by a processor to implement the delay measurement method described in any one of claims 1-9. A computer program product, characterized in that the computer program product comprises a computer program/instructions, and the computer program/instructions are executed by a processor so that a computer implements the delay measurement method described in any one of claims 1-9.