CN119728803A

CN119728803A - RDMA-based message sending and processing method, system and electronic device

Info

Publication number: CN119728803A
Application number: CN202411917494.XA
Authority: CN
Inventors: 朱向阳; 喻敬海
Original assignee: ZTE Corp
Current assignee: ZTE Corp
Filing date: 2024-12-24
Publication date: 2025-03-28

Abstract

The application provides a message sending and processing method, a message sending and processing system and electronic equipment based on RDMA. The message sending method comprises the steps of sending a first message to a receiving end, wherein the first message carries indication information and identification information of a message to which the first message belongs, and the indication information is used for indicating whether disordered reception among the messages is allowed or not. Because the sending end carries the identification information of the message and the indication information for indicating whether to allow the disordered reception of the message in the first message, for the receiving end, when the indication information carried in the first message indicates that the disordered reception of the message is allowed, the receiving end only needs to keep the order of the message in the message, and the message is not required to keep the order of the message, that is, the receiving end can not trigger retransmission under the disordered condition of the message, and only trigger retransmission under the condition that the message in the message is lost or disordered, thereby reducing the retransmission range (namely limiting the retransmission range in the message), and improving the transmission efficiency of the network.

Description

RDMA (remote direct memory Access) -based message sending and processing method, system and electronic equipment

Technical Field

The present application relates to the field of network communications technologies, and in particular, to a method, a system, and an electronic device for sending and processing a message based on RDMA.

Background

Remote direct memory access (Remote Direct Memory Access, RDMA) is a network communication technology that allows computers to directly transfer data between memories over a network. RDMA technology can provide multiple transmission modes at the transport layer, and is commonly used in order-preserving transmission modes, such as a Reliable Connection (RC) mode used by a converged ethernet-based remote direct memory access RoCEv, a Reliable order-preserving transmission (Reliable Ordered Delivery, ROD) mode used by the super ethernet alliance (Ultra Ethernet Consortium, UEC), and the like. In the related art, when data transmission is performed in a sequence preserving transmission mode provided by using an RDMA technology, a transmitting end may split data to be transmitted into messages, a receiving end may check whether the received messages are continuous, and if discontinuous, it is considered that packet loss or disorder occurs and retransmission is triggered, so that sequence preserving transmission of the messages may be ensured. However, in practical application, when packet loss or disorder occurs in the transmission mode, the retransmission cost is high, and the transmission efficiency of the network is reduced.

Disclosure of Invention

The application provides a message sending and processing method, a message sending and processing system and electronic equipment based on RDMA (remote direct memory access), which are used for solving the problem of lower transmission efficiency when a message is transmitted by using a sequence preserving transmission mode provided by an RDMA technology.

In order to solve the technical problems, the application is realized as follows:

In a first aspect, an RDMA-based packet sending method is provided, applied to a sending end, and includes:

and sending a first message to a receiving end, wherein the first message carries indication information and identification information of a message to which the first message belongs, and the indication information is used for indicating whether to allow out-of-order reception between the messages.

In a second aspect, an RDMA-based packet processing method is provided, applied to a receiving end, and includes:

Receiving a first message sent by a sending end, wherein the first message carries indication information and identification information of a message to which the first message belongs, and the indication information is used for indicating whether to allow out-of-order reception between the messages;

and processing the first message according to the indication information and the identification information, wherein the processing comprises the steps of carrying out message-to-message disorder receiving or message-to-message order-preserving receiving on the message to which the first message belongs.

In a third aspect, an RDMA-based packet sending and processing system is provided, including a sending end and a receiving end, where;

The sending terminal sends a first message to the receiving terminal, wherein the first message carries indication information and identification information of a message to which the first message belongs, and the indication information is used for indicating whether to allow out-of-order reception between the messages;

The receiving end receives the first message, processes the first message according to the indication information and the identification information, and the processing comprises message disorder receiving or message preservation receiving of messages to which the first message belongs.

In a fourth aspect, there is provided an electronic device comprising:

A processor;

a memory for storing the processor-executable instructions;

Wherein the processor is configured to execute the instructions to implement the method according to the first or second aspect.

In a fifth aspect, there is provided a computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the method of the first or second aspect.

In a sixth aspect, there is provided a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of the method as described in the first aspect, or to perform some or all of the steps of the method as described in the second aspect.

In the embodiment of the application, when the sending end and the receiving end transmit the message based on RDMA, the message transmission can be performed based on the message, specifically, when the sending end sends the first message to the receiving end, the sending end can carry the identification information of the message to which the first message belongs and the indication information for indicating whether to allow the disordered reception between the messages in the first message, and after the receiving end receives the first message, the receiving end can process the message based on the identification information and the indication information carried in the first message. In this way, under the condition that the first message sent by the sending end indicates that the disordered reception of the messages is allowed, the receiving end only needs to perform the order preservation on the messages in the messages, and the order preservation between the messages is not needed, that is, the receiving end can trigger no retransmission under the condition that the messages are disordered, and only trigger retransmission under the condition that the messages in the messages are lost or disordered, thereby reducing the retransmission range (namely, limiting the retransmission range in the messages), and further improving the transmission efficiency.

Drawings

In order to more clearly illustrate the application or the technical solutions of the prior art, the drawings that are used in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments described in the present application, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.

FIG. 1 is a schematic diagram of an embodiment of the application in which the application layer does not require inter-message order preservation;

FIG. 2 is a flow diagram of an RDMA-based messaging method according to one embodiment of the present application;

FIG. 3 is a schematic diagram of an application layer sending a first parameter over an extended programming interface in accordance with one embodiment of the application;

FIG. 4 is a schematic diagram of load balancing performed by a sender according to an embodiment of the present application;

Fig. 5 is a schematic diagram of a retransmission message at a transmitting end according to an embodiment of the present application;

FIG. 6 is a flow diagram of an RDMA-based message processing method according to one embodiment of the present application;

FIG. 7 is a flow chart of processing a message according to an embodiment of the present application;

FIG. 8 is a schematic diagram of message granularity based load balancing in accordance with an embodiment of the application;

FIG. 9 is a diagram illustrating a basic encapsulation format of a message according to an embodiment of the present application;

FIG. 10 is a schematic diagram of expanding the basic encapsulation format of a message according to one embodiment of the present application;

FIG. 11 is a schematic diagram of expanding the semantic header encapsulation format of a message according to one embodiment of the present application;

FIG. 12 is a schematic diagram of expanding the encapsulation format of NACK messages in accordance with an embodiment of the present application;

FIG. 13 is a schematic diagram of the architecture of an electronic device in accordance with one embodiment of the application;

FIG. 14 is a schematic diagram of an RDMA-based messaging device according to one embodiment of the present application;

FIG. 15 is a schematic diagram of the architecture of an electronic device in accordance with one embodiment of the application;

FIG. 16 is a schematic diagram of an RDMA-based message processing apparatus according to one embodiment of the present application.

Detailed Description

RDMA is a network communication technology, and adopts mechanisms such as '0 copy', kernel bypass, hardware unloading and the like, so that a computer is allowed to directly conduct data transmission between memories on a network. By using RDMA technology for data transfer, end-to-end communication latency can be reduced while improving throughput, which makes RDMA one of the important technologies in high-performance computing and intelligent data center networks.

RDMA technology provides multiple transport modes at the transport layer. For example, roCEv is widely used, and includes modes such as Reliable Connection (RC), unreliable datagram (Unreliable Datagram, UD), and the like. The RC is the most commonly used mode, and can provide reliable order-preserving transmission at a message level, for example, the sending end segments data to be sent into messages, each message carries a sequence number (Packet Sequence Number, PSN), if the RC mode is adopted, the receiving end can check whether the received PSN is continuous, and if the discontinuous is regarded as packet loss by default, retransmission is triggered. Other RDMA technologies are similar, for example, in the current hot UEC, modes such as reliable order-preserving transmission (Reliable Ordered Delivery, ROD) and reliable non-order-preserving transmission (Reliable Unordered Delivery, RUD) are provided at the transmission layer, when the ROD mode is adopted, the transmission layer at the receiving end can preserve the order of the received message, when the RUD mode is adopted, the receiving end directly delivers the message into the host memory, at this time, the message in the same message can arrive out of order, and when the transmission layer judges that the packet loss occurs (typically, whether the packet loss occurs or not is judged based on an out-of-order threshold), a selective retransmission mechanism is adopted. Due to the shortcomings of the selective retransmission mechanism, current commercial RDMA network cards still use the go-back-N retransmission mechanism in the production environment.

In the related art, when a packet is transmitted in a sequence preserving transmission mode (such as the RC mode used in RoCEv or the ROD mode used in UE C), the packet needs to be preserved, so that once the packet is lost or disordered, the receiving end triggers retransmission, resulting in higher retransmission cost and affecting the transmission efficiency of the network.

In some application scenarios, the transport layer provides a programming interface (Verbs, libfabric, etc.) to the application layer, which may require intra-message order preservation, but not inter-message order preservation. As shown in fig. 1, the application layer program at the transmitting end needs to transmit 2 high-performance computing messages, each message is split into 4 messages, each message carries a matrix data, and the application layer program at the receiving end needs to receive the two messages and perform matrix addition operation. In this case, the order of the messages is required to be preserved by the internal report Wen Bixu, otherwise, the carried data will have errors, and the arrival sequence of the messages does not affect the result of calculation because of the exchange law of matrix addition, so the order of the messages may not need to be preserved. Other various scenes such as mathematical multiplication operation, linear equation solution and the like meet the exchange law, and also the order preservation among messages is not needed. Therefore, if the transmission layer can provide a processing mode, only the internal order of the messages is kept, but the inter-message order is not kept, the range of the traditional reliable connection packet loss or the disordered retransmission can be greatly reduced while the application scene is met, and the transmission efficiency of the network is improved. However, current RDMA techniques also lack such fine-grained transport handling mechanisms.

The embodiment of the application provides a message sending and processing method, a message sending and processing system and electronic equipment based on RDMA, wherein a sending end and a receiving end can carry out message transmission based on messages when messages are transmitted based on RDMA, and particularly, when the sending end sends a first message to the receiving end, the sending end can carry identification information of the message to which the first message belongs and indication information for indicating whether disordered reception between the messages is allowed or not in the first message, and after the receiving end receives the first message, the receiving end can process the message based on the identification information and the indication information carried in the first message. In this way, under the condition that the first message sent by the sending end indicates that the disordered reception of the messages is allowed, the receiving end only needs to perform the order preservation on the messages in the messages, and the order preservation between the messages is not needed, that is, the receiving end can trigger no retransmission under the condition that the messages are disordered, and only trigger retransmission under the condition that the messages in the messages are lost or disordered, thereby reducing the retransmission range (namely, limiting the retransmission range in the messages), and further improving the transmission efficiency.

It should be noted that the technical solution provided by the embodiment of the present application may be applied to a router or a switch and a corresponding configuration unit.

In order that those skilled in the art will better understand the present application, a more particular and complete description of the same will be rendered by reference to the appended drawings, wherein it is to be understood that the present application is illustrated in one or more embodiments thereof. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, shall fall within the scope of the application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate, such that the application may be practiced otherwise than as specifically illustrated or described herein. Furthermore, in the present application and in the claims, "and/or" means at least one of the connected objects, and the character "/" generally means that the associated object is an or relationship.

The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

FIG. 2 is a flow diagram of an RDMA-based messaging method according to an embodiment of the present application. The message transmission method shown in fig. 2 is applied to the transmitting end, that is, the message transmission method shown in fig. 2 may be performed by software or hardware installed at the transmitting end. In some embodiments, the method for sending a message shown in fig. 2 may be specifically applied to a transmission layer of a sending end. The method of sending the message shown in fig. 2 is as follows.

S202, a first message is sent to a receiving end, wherein the first message carries indication information and identification information of a message to which the first message belongs, and the indication information is used for indicating whether to allow out-of-order reception between the messages.

When the sending end transmits the message based on RDMA, the message transmission can be performed based on the message. For example, when the sending end shown in fig. 1 sends msg1 and msg2 to the receiving end, msg1 and msg2 may be split into 4 messages respectively, and then these messages are sent to the receiving end in sequence. In the embodiment of the application, when the sending end sends the message to the receiving end, taking sending the first message as an example, the indication information and the identification information can be carried in the first message. The indication information is used for indicating whether to allow out-of-order reception between the messages, for example, when the sending end sends msg1 and msg2 to the receiving end, the sent message carries the indication information, and the indication information is used for indicating whether to allow the receiving end to receive msg1 and msg2 out-of-order. The identification information is the identification information of the message to which the first message belongs, for example, the first message is a message obtained by splitting msg1, and then the identification information is the identification information of msg1, for example, the ID of msg 1.

In this way, the sending end carries the identification information of the message and the indication information for indicating whether to allow the disordered reception of the message in the first message, so that for the receiving end, when the indication information carried in the first message indicates that the disordered reception of the message is allowed, the receiving end only needs to keep the order of the message in the message, and the message is not required to be kept in order, that is, the receiving end can not trigger retransmission under the disordered condition of the message, and only trigger retransmission under the condition that the message in the message is lost or disordered, thereby reducing the retransmission range (namely limiting the retransmission range in the message), and improving the transmission efficiency of the network.

In some embodiments, the sending end sends the first message to the receiving end, which may include:

Receiving a first parameter sent by an application layer, wherein the first parameter is used for indicating whether to allow out-of-order reception between messages;

and sending the first message to the receiving end according to the first parameter.

Specifically, the application layer of the transmitting end may transmit a first parameter to the transmission layer of the transmitting end according to whether to perform the order-preserving requirement between the messages, where the first parameter may indicate whether to allow the out-of-order receiving between the messages. For example, if the application layer has no requirement for inter-message order preservation, the first parameter may indicate that the inter-message order reception is allowed, and if the application layer has the requirement for inter-message order preservation, the first parameter may indicate that the inter-message order reception is not allowed.

To facilitate the application layer to pass in the first parameter, in some embodiments, the programming interface provided by the transport layer to the application layer may be extended, such that the application layer may send the first parameter to the transport layer via the extended programming interface, and correspondingly, the transport layer may receive the first parameter sent by the application layer based on the extended programming interface. Wherein the extended programming interface may include an extended first field therein, which may be used to carry a first parameter. That is, when expanding the programming interface, a first field may be expanded in the programming interface so that the application layer may pass the requirement of whether to perform inter-message order preservation to the transport layer through the first field.

It should be noted that different RDMA technologies use different programming interfaces, e.g., roCEv uses libverbs, UEC uses libfabric, intel uses PSM2, cisco uses usNIC, etc., and the different programming interfaces differ in details but extend in a similar manner. For example, in IB libverbs interfaces, when the interfaces are extended, new transport sub-mode delivery application layer requirements may be extended by extending the ibjcreate_qp interface parameter ibv _qp_init_attr. As shown in fig. 3, when qp_type is RC and qp_sub_type=true, it may indicate that the application layer has no requirement for the order preservation between messages, that is, the application layer allows the message to be received out of order.

After receiving the first parameter sent by the application layer, the sending end may send the first message to the receiving end according to the first parameter when sending the first message to the receiving end. The sending of the first message to the receiving end according to the first parameter may be that whether the indication information carried in the first message allows the out-of-order receiving between the messages is set according to the first parameter. For example, if the first parameter indicates that the message is allowed to be received in a disorder mode, the indication information carried in the first message is specifically used for indicating that the message is allowed to be received in a disorder mode, and if the first parameter indicates that the message is not allowed to be received in a disorder mode, the indication information carried in the first message is specifically used for indicating that the message is not allowed to be received in a disorder mode.

In some embodiments, in order to carry the identification information of the message and the indication information for indicating whether to allow out-of-order reception between the messages in the first message, the first message may be extended as required. For example, for the identification information of the message, since the identification information carrying the message is not defined in the service message of RoCEv, and the identification information carrying the message is defined in the UEC transport layer message, the service message of RoCEv needs to be correspondingly expanded, and the UEC transport layer message does not need to be correspondingly expanded. For the indication information for indicating whether to allow out-of-order reception between messages, since such indication information is not defined in the service message RoCEv and the UEC transport layer message currently, the service message RoCEv and the UEC transport layer message need to be correspondingly expanded so that the service message RoCEv2 and the UEC transport layer message can carry the indication information.

When expanding the message, the message format of the RDMA technology used can be correspondingly expanded. Wherein, for the indication information for indicating whether or not the out-of-order reception between messages is allowed, alternatively, a flag bit of 1 bit may be used for the indication. For example, in the case where the flag bit is true (or where the flag bit has a value of 1), the instruction information is used to instruct that the inter-message out-of-order reception is permitted, and in the case where the flag bit is false (or where the flag bit has a value of 0), the instruction information is used to instruct that the inter-message out-of-order reception is not permitted.

In some embodiments, the first packet may further carry an operation code, where the operation code is used to characterize a current transmission mode, such as a reliable mode or an unreliable mode. When the transmitting end transmits the first message to the receiving end, load balancing can be performed according to the identification information in the first message under the condition that the first condition is met. The first condition includes that the operation code is in a reliable mode (such as RC 0), the first message carries identification information and indication information of the message, and the indication information is used for indicating that the out-of-order reception between the messages is allowed. That is, in the case that the operation code carried in the first message is in a reliable mode, the operation code carries the identification information and the indication information of the message, and the indication information is used for indicating that the out-of-order reception between the messages is allowed, the sending end is allowed to perform load balancing based on the granularity of the message.

The current widely applied flow level hash routing algorithm is based on the service flow IP quintuple, and has a plurality of problems such as hash conflict polarization and the like. When the sending end determines that load balancing can be performed based on the granularity of the message, the identification information of the message can be used as input of a hash algorithm, at the moment, messages belonging to the same message (i.e. messages with the same identification information) can be hashed to one path, so that the sequence of the messages in the message is ensured, and messages belonging to different messages (i.e. messages with different identification information) can be sent to different paths, so that the load balancing effect is improved. That is, when the transmitting end performs load balancing according to the identification information, the transmitting end may specifically transmit the messages with the same identification information to the same transmission path, and transmit the messages with different identification information to the same or different transmission paths.

Referring to fig. 4, when the transmitting end sends msg1, msg2, msg3 and msg4 to the receiving end, and each message includes a plurality of messages (these messages are not shown in fig. 4), the transmitting end may send each message in a manner as shown in fig. 4 when determining that load balancing can be performed based on the granularity of the message. In fig. 4, when the sending end performs load balancing, all the messages belonging to the same message are sent to one transmission path, and the messages belonging to different messages are sent to the same or different transmission paths. Taking the first switch connected to the transmitting end in fig. 4 as an example, when msg1, msg2, msg3 and msg4 are forwarded, each message is forwarded through one transmission path, so that the messages in the same message are all sent to one transmission path and are not sent separately, different messages are forwarded through the same or different transmission paths, that is, msg1 and msg4 are sent to one transmission path, msg2 and msg3 are sent to the other transmission path, and the messages in different messages can be sent to the same or different transmission paths.

When the sending end performs load balancing based on the granularity of the message, the load balancing effect is related to the size of the message, and in general, the smaller the message is, the better the load balancing effect is. The selection of the message size may be set according to different application characteristics, which is not specifically limited herein.

In some embodiments, when the transmitting End sends the first packet to the receiving End, the first packet may further carry a packet sequence number (Packet Sequence Number, PSN), a Start Of last Msg, SOM, end Of last Msg, EOM, an identifier Of a source queue pair (Source queue pair, SQP), and an identifier Of a destination queue pair (Destination queue pair, DQP), where these information may be used for the processing Of the first packet by the receiving End, and the specific processing flow may be referred to an example shown in fig. 6, and will not be described in detail herein. The packet sequence number can represent the sequence of the messages and can be used for the receiving end to store the messages. The first packet flag is used to indicate whether the message is the first message in the belonging message, alternatively, the first packet flag may use 1 bit to indicate that the message is the first message in the belonging message, and a bit value of 1 is used to indicate that the message is not the first message in the belonging message, and a bit value of 0 is used to indicate that the message is not the first message in the belonging message. The tail packet flag is used to indicate whether the message is the last message in the belonging message, alternatively, the tail packet flag may also use 1 bit to indicate that the message is the last message in the belonging message, and a bit value of 1 is used to indicate that the message is not the last message in the belonging message, and a bit value of 0 is used to indicate that the message is not the last message in the belonging message. The identification of the source queue pair may be an ID of the source queue pair. The identification of the destination queue pair may be an ID of the destination queue pair.

In some embodiments, when the indication information carried in the first packet is used to indicate that the out-of-order reception between the messages is allowed, the sending end may further perform the following operations:

Receiving a NACK message sent by a receiving end, wherein the NACK message is sent by the receiving end under the condition that the receiving end determines that the message in the message to which the first message belongs loses packets or is disordered;

And retransmitting the message according to the NACK message.

Specifically, under the condition that the indication information carried by the first message is used for indicating that the disordered reception of the messages is allowed, for the receiving end, the message internal order preservation can be only performed, the message internal order preservation is not performed, namely, a plurality of messages in the message to which the first message belongs need to be received in order, and the message to which the first message belongs can be received in disorder. When the receiving end performs the in-message order preservation, the receiving end can trigger retransmission and send a NACK message to the sending end under the condition that packet loss or disorder possibly exists for a plurality of messages in the message to which the first message belongs. After receiving the NACK message, the transmitting end can retransmit the message according to the NACK message.

In some embodiments, in order to make the sender limit the retransmission range of the message within the message, the NACK message may carry the identification information of the message to which the retransmission message belongs (i.e. the identification information of the message to which the first message belongs) and the sequence number of the retransmitted initial packet. In this way, when the transmitting end retransmits the message according to the NACK message, the transmitting end can only transmit the message from the retransmitted initial packet sequence number to the receiving end in the message corresponding to the identification information according to the identification information carried in the NACK message and the retransmitted initial packet sequence number, thereby limiting the retransmission range in the message, reducing the number of the retransmitted message, reducing the retransmission cost and improving the transmission efficiency of the network.

In order to make the NACK message carry the identification information of the message, in some embodiments, the NACK message may be extended according to needs, that is, if the identification information carrying the message is not defined in the current NACK message, the NACK message needs to be extended, so that the NACK message may carry the identification information of the message. Optionally, the NACK message may further carry indication information for indicating that the out-of-order reception between the messages is allowed, and at this time, the NACK message needs to be correspondingly expanded, so that the NACK message may carry the indication information.

Fig. 5 is a schematic diagram of a retransmission message at a transmitting end according to an embodiment of the present application.

In fig. 5, a transmitting end transmits msg1 and msg2 to a receiving end, wherein msg1 includes 4 messages, PSNs are 1, 2, 3 and 4 respectively, msg2 includes 5 messages, PSNs are 5, 6, 7, 8 and 9 respectively, and an application layer of the transmitting end allows out-of-order reception between messages. When receiving a message, a receiving end discovers that the message with PSN=3 is lost, triggers retransmission and sends a NACK message to a sending end, wherein the NACK message carries a message identifier msg1, a retransmission starting packet sequence number PSN=3 and indication information msg_order_flag=true for indicating that disordered reception between messages is allowed. After receiving the NACK message, the transmitting end may retransmit only the messages with psn=3 and psn=4 in msg1 when retransmitting the message, i.e. the retransmission range is psn= { 3-4 }. It should be noted that, since out-of-order reception is allowed between messages, msg2 may arrive at the receiving-end transport layer before the {3,4} message of msg1 and be delivered to the memory of the receiving-end host.

Assuming that the application layer of the transmitting end does not allow the out-of-order reception between the messages, in fig. 5, when the message with psn=3 is lost, the retransmission range of the transmitting end is psn= { 3-9 }, and when the out-of-order reception between the messages is allowed, the retransmission range of the transmitting end is psn= { 3-4 }, and therefore, when the application layer of the transmitting end allows the out-of-order reception between the messages, the retransmission range of the transmitting end can be reduced, and the transmission efficiency is improved.

In the embodiment of the application, the sending end carries the identification information of the message and the indication information for indicating whether to allow the disordered reception of the message in the first message, so that for the receiving end, the receiving end only needs to keep the order of the message in the message when the indication information carried in the first message indicates that the disordered reception of the message is allowed, and the message is not required to be kept in order, that is, the receiving end can not trigger retransmission under the disordered condition of the message, and only trigger retransmission under the condition that the message in the message is lost or disordered, thereby reducing the retransmission range (namely, limiting the retransmission range in the message), and further improving the transmission efficiency of the network. In addition, under the condition of allowing out-of-order reception among messages, the sending end can perform load balancing based on the granularity of the messages, so that message blocking can be avoided, and the transmission efficiency and the overall network utilization rate of the network are further improved.

FIG. 6 is a flow diagram of an RDMA-based message processing method according to one embodiment of the present application. The message sending method shown in fig. 2 is applied to the receiving end, that is, the message processing method shown in fig. 6 may be performed by software or hardware installed at the receiving end. In some embodiments, the method for processing a message shown in fig. 6 may be specifically applied to a transmission layer of a receiving end. The message processing method shown in fig. 6 is as follows.

S602, receiving a first message sent by a sending end, wherein the first message carries indication information and identification information of a message to which the first message belongs, and the indication information is used for indicating whether to allow out-of-order reception between the messages.

After the sending end sends the first message to the receiving end based on RDMA, the receiving end can receive the first message. The first message carries indication information for indicating whether to allow out-of-order reception between the messages and identification information of the message to which the first message belongs. The specific implementation manner of the sending end to send the first message to the receiving end may refer to the corresponding content in the embodiment shown in fig. 2, which is not described in detail herein.

S604, processing the first message according to the indication information and the identification information, wherein the processing comprises the steps of carrying out message disorder receiving or message order keeping receiving on the message to which the first message belongs.

After receiving the first message, the receiving end can process the first message according to the indication information and the identification information carried in the first message, and the process can include performing message disorder receiving or message preservation receiving on the message to which the first message belongs. If the receiving end performs message disorder receiving on the message to which the first message belongs, that is, the receiving end does not perform message disorder preserving on the message to which the first message belongs, the receiving end may not trigger retransmission under the condition of message disorder, thereby reducing retransmission events and improving transmission efficiency. In addition, the processing of the first message may further include order-preserving receiving the first message, that is, order-preserving a plurality of messages in the message to which the first message belongs. Therefore, under the condition that the receiving end performs order preservation on the first message and does not perform inter-message order preservation on the message to which the first message belongs, the receiving end can trigger retransmission only under the condition that packet loss or disorder occurs on the message in the message, and does not trigger retransmission under the condition that the message is disorder, so that the retransmission range (namely, the retransmission range is limited in the message) can be reduced, and the transmission efficiency is improved.

In some embodiments, the processing, by the receiving end, the first packet according to the indication information and the identification information carried in the first packet may include:

Under the condition that the indication information indicates that disordered reception between messages is allowed, carrying out first processing on the first message according to the identification information, wherein the processing principle of the first processing comprises the steps of preserving the sequence of a plurality of messages in the message to which the first message belongs and not preserving the sequence between the messages to which the first message belongs;

And under the condition that the indication information indicates that the out-of-order reception of the messages is not allowed, carrying out second processing on the first message, wherein the processing principle of the second processing comprises the steps of preserving the order of a plurality of messages in the message to which the first message belongs and preserving the order of the messages to which the first message belongs.

The content indicated by the indication information carried in the first message includes two cases, one is that the indication information indicates that the out-of-order reception between the messages is allowed, and the other is that the indication information indicates that the out-of-order reception between the messages is not allowed. Under the condition that the indication information indicates that the disordered reception of the messages is allowed, the receiving end needs to perform the intra-message order preservation when the messages are processed, and the inter-message order preservation can not be performed, namely the receiving end can perform the first processing on the first message according to the identification information, and the processing principle of the first processing comprises the order preservation of a plurality of messages in the message to which the first message belongs and the non-inter-message order preservation of the message to which the first message belongs. When the indication information indicates that the disordered reception of the messages is not allowed, the receiving end needs to perform the message internal order preservation when processing the messages, namely the receiving end can perform the second processing on the first message, and the processing principle of the second processing comprises the order preservation of a plurality of messages in the message to which the first message belongs and the message to which the first message belongs. The specific implementation manner of the receiving end in the second line processing of the first message may refer to the specific implementation in the related art, and will not be described in detail herein.

In some embodiments, the first packet may further carry a packet sequence number, a first packet flag, a tail packet flag, an identifier of a source queue pair, and a destination queue pair, and a detailed description of these information may refer to corresponding contents in the embodiment shown in fig. 2, which will not be repeated here. Under the condition that the first message carries a packet sequence number, a first packet mark, a tail packet mark, an identifier of a source queue pair and an identifier of a destination queue pair, the receiving end performs first processing on the first message according to the identifier information, which may be that the first message is subjected to first processing according to the identifier information, the packet sequence number, the first packet mark, the tail packet mark, the identifier of the source queue pair and the identifier of the destination queue pair carried in the first message. Specifically, it may include:

Judging whether the first message is the first message or the last message in the belonged message according to the first packet mark and the tail packet mark;

and carrying out first processing on the first message according to the judging result, the identification information, the packet serial number, the identification of the source queue pair and the identification of the destination queue pair.

The first packet flag is used to indicate whether the first packet is the first packet in the belonging message, for example, the first packet flag may use 1 bit, where a bit value of 1 is used to indicate that the packet is the first packet in the belonging message, and a bit value of 0 is used to indicate that the packet is not the first packet in the belonging message. The tail packet flag is used to indicate whether the first message is the last message in the belonging message, and the tail packet flag may also use 1 bit to indicate that the bit value is 1 to indicate that the message is the last message in the belonging message, and 0 to indicate that the message is not the last message in the belonging message. When the receiving end judges whether the first message is the first message and the last message in the affiliated message according to the first packet mark and the last packet mark, the receiving end can judge according to the value of the first packet mark and the value of the last packet mark. For example, if the first packet flag is set to1, it may be determined that the first packet is the first packet in the belonging message, if the last packet flag is set to1, it may be determined that the first packet is the last packet in the belonging message, and if the first packet flag is set to 0, the last packet flag is set to 0, it may be determined that the first packet is neither the first packet nor the last packet in the belonging message, but other packets except the first packet and the last packet in the belonging message.

After judging whether the first message is the first message or the last message in the belonged message, the first message can be subjected to first processing according to a judging result, identification information carried in the first message, a packet serial number, an identification of a source queue pair and an identification of a destination queue pair.

As described above, the determination result may include three cases, where the first message is the first message in the belonging message, the second case is the first message in the belonging message and not the first message in the belonging message, and the third case is the last message in the belonging message. For different judging results, the receiving end can execute different processing operations when processing the first message. In the following, for different judging results, how to process the first message will be described in detail.

In the case that the determination result includes that the first packet is the first packet in the belonging message, in some embodiments, performing the first processing on the first packet according to the determination result, the identification information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair may include:

Generating a first variable and a second variable according to the identification information, the identification of the source queue pair and the identification of the destination queue pair, wherein the first variable is used for recording the packet sequence number of the last message received in a sequence-preserving manner in the message, and the second variable is used for indicating whether the first message in the message is successfully received;

Initializing a first variable and a second variable, wherein the value of the first variable after initialization is the packet sequence number of a first message, and the value of the second variable after initialization is used for indicating that the first message in a message is successfully received;

and delivering the first message to a host memory.

Specifically, in the case that the first message is the first message in the belonging message, two variables may be generated for the message, one for recording the packet sequence number of the last message received in the message in order, i.e. the first variable described above (alternatively may be denoted as msg_psn_cur), and the other for indicating whether the first message in the message has been successfully received, i.e. the second variable described above (alternatively may be denoted as som _received). In the embodiment of the application, the first variable and the message identifier, the identifier of the source queue pair and the identifier of the destination queue pair may be one-to-one mapping relationship, and optionally, the mapping relationship may be stored as a table, and the key is the message identifier, the identifier of the source queue pair and the identifier of the destination queue pair, and the value is the first variable. Similarly, the second variable may also have a one-to-one mapping relationship with the message identifier, the identifier of the source queue pair, and the identifier of the destination queue pair.

After the first variable and the second variable are generated, both variables may be initialized. Since the first message is the first message in the affiliated message and has been successfully received by the receiving end, after initializing the first variable and the second variable, the value of the first variable is the packet sequence number of the first message, and the value of the second variable is used for indicating that the first message in the message has been successfully received, for example, the second variable is represented by 1bit, and then, after initializing the second variable, the value of the second variable may be true or 1.

After initializing the first variable and the second variable, the first message may be delivered to the host memory.

In the case that the determination result includes that the first message is not the first message or the last message in the belonging message, in some embodiments, performing the first processing on the first message according to the determination result, the identification information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair may include:

Searching a second variable corresponding to the identification information, the identification of the source queue pair and the identification of the destination queue pair;

Searching for a first variable corresponding to the identification information, the identification of the source queue pair and the identification of the destination queue pair in the case that the second variable exists and is used for indicating that the first message in the message has been successfully received;

delivering the first message to a host memory and updating the value of the first variable to the packet sequence number of the first message under the condition that the packet sequence number of the first message is equal to the sum of the value of the first variable and 1;

and under the condition that the packet sequence number of the first message is not equal to the sum of the value of the first variable and 1, determining that the message in the message to which the first message belongs is lost or disordered, and sending a NACK message to the sending end.

Specifically, in the case that the first message is neither the first message in the belonging message nor the last message in the belonging message, the corresponding second variable (for indicating whether the first message in the message has been successfully received) may be searched according to the identification information carried in the first message, the identification of the source queue pair, and the identification of the destination queue pair. If the second variable or the value of the second variable is not found and is used to indicate that the first message in the message is not successfully received (for example, the value of the second variable is false), it may be indicated that the receiving end does not successfully receive the first message in the message to which the first message belongs, and at this time, the first message may be processed according to a processing method in the related art, which is not described in detail herein. If the second variable exists and the value of the second variable is used to indicate that the first message in the message has been successfully received (for example, the value of the second variable is true), it may be indicated that the receiving end has successfully received the first message in the message to which the first message belongs, and at this time, the subsequent operation may be continuously performed, that is, according to the identification information carried in the first message, the identification of the source queue pair and the identification of the destination queue pair, the corresponding first variable (for recording the packet sequence number of the last message that has been transmitted in order in the message) is searched. The first variable and the second variable are generated according to the identification information, the identification of the source queue pair and the identification of the destination queue pair when the sending end receives the first message in the message, so that the first variable can be searched under the condition that the second variable is searched.

After the first variable is found, it may be determined whether the packet sequence number of the first packet is equal to the sum of the value of the first variable and 1. If the packet sequence number of the first message is equal to the sum of the value of the first variable and 1, it can be stated that the packet sequence number of the first message is continuous with the packet sequence number of the last message received by the receiving end in a guaranteed sequence, no packet loss or disorder occurs, at this time, the first message can be delivered to the host memory, and updating the value of the first variable into the packet sequence number of the first message so that when the receiving end receives the next message, whether the packet sequence number of the first message is continuous with the packet sequence number of the next message or not can be judged according to the updated first variable, and further whether packet loss or disorder occurs or not is judged. If the packet sequence number of the first packet is not equal to the sum of the value of the first variable and 1, it can be stated that the packet sequence number of the first packet is discontinuous with the packet sequence number of the last packet received by the receiving end in a sequence-preserving manner, and the packet in the message to which the first packet belongs is lost or disordered, at this time, the receiving end can trigger retransmission and send a NACK message to the sending end.

When the transmitting end transmits the NACK message to the receiving end, in order to limit the message retransmission range of the transmitting end to the message, in some embodiments, the NACK message may carry the identification information of the message to which the retransmission message belongs (i.e., the identification information of the message to which the first message belongs) and the sequence number of the retransmitted initial packet, where the sequence number of the retransmitted initial packet may be equal to the sum of the value of the first variable and 1, which indicates that packet loss or disorder occurs from the sequence number of the initial packet, and the message from the sequence number of the initial packet needs to be retransmitted in the message. For the transmitting end, when receiving a NACK message and retransmitting the message, based on the identification information carried in the NACK message and the retransmitted initial packet sequence number, the transmitting end can only transmit the message from the retransmitted initial packet sequence number to the receiving end in the message corresponding to the identification information, thereby limiting the retransmission range in the message, reducing the number of the retransmitted messages, reducing the retransmission cost and improving the transmission efficiency of the network. Reference is made in particular to the embodiment shown in fig. 5, which is not illustrated here.

In the case that the determination result includes that the first packet is the last packet in the belonging message, in some embodiments, performing, according to the determination result, the identification information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair, the first processing on the first packet includes:

Searching a first variable and a second variable corresponding to the identification information, the identification of the source queue pair and the identification of the destination queue pair;

Releasing the first variable and the second variable;

Delivering the first message to a host memory;

and sending the ACK message to the sending end.

In order to facilitate understanding of the processing flow of the message by the receiving end, a more specific embodiment shown in fig. 7 will be described below as an example. As shown in fig. 7, when a certain message is processed, the following steps may be included.

And 1, extracting fields such as identification information msg_id, a packet sequence number PSN, a first packet mark SOM, a last packet mark EOM, indication information msg_order_flag for indicating whether to allow out-of-order reception between messages, a source queue pair SQP, a destination queue pair DQP and the like from the message.

And 2, checking the value of the msg_order_flag, and if the msg_order_flag=false, that is, the message is not allowed to be received in disorder, processing the message according to the existing flow, and turning to the step 8. If msg_order_flag=true, i.e. allowing out-of-order reception between messages, step 3 is performed.

Step 3, checking the value of SOM, if som=1, maintaining a variable msg_psn_cur for the msg_id, wherein the initial value of msg_psn_cur is the PSN of the message, and simultaneously maintaining a variable SOM _received=true, and then turning to step 5. If som=0, it is determined whether SOM _received corresponding to the msg exists and is true, if yes, the corresponding msg_psn_cur is searched according to the msg_id, step 4 is executed, if no, the message is processed according to the existing flow, and step 8 is transferred.

And 4, judging whether the PSN of the message is equal to msg_psn_cur+1, if so, updating the msg_psn_cur to the PSN of the message, then executing the step 5, and if not, executing the step 7.

And 5, delivering the message to a host memory, judging whether the message is a tail packet, if so, executing the step 6, and if not, executing the step 8.

Step 6, if the message eom=1, releasing the current msg_psn_cur variable, sending an ACK to the source (i.e. the sender), generating a completion queue entry (Complete Queue Entry, CQE) (the process is the same as the existing flow), and proceeding to step 8.

And 7, if the PSN of the message is not equal to the msg_psn_cur+1, triggering retransmission, namely sending a NACK message to the source end and indicating that the retransmission starting sequence number is msg_psn_cur+1, and then turning to the step 8.

And 8, ending the current message processing.

In the embodiment of the application, the first message received by the receiving end carries the identification information of the message to which the first message belongs and the indication information for indicating whether to allow the disordered reception of the messages, so that for the receiving end, the receiving end only needs to keep the order of the messages in the message under the condition that the disordered reception of the messages is indicated in the first message sent by the sending end, that is, the receiving end can not trigger retransmission under the condition that the messages are disordered, and only trigger retransmission under the condition that the messages in the message are lost or disordered, thereby reducing the retransmission range (namely limiting the retransmission range to the message), and further improving the transmission efficiency.

In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, three more specific implementation manners will be described below as examples.

First embodiment this embodiment illustrates how load balancing can be based on message granularity when the sender application layer allows for message-to-message misordering. In this embodiment, a typical spine-leaf architecture is taken as an example, assuming that a switch device has 4 ports, and the port bandwidth is 25Gbps, when there is no bandwidth convergence networking, each leaf switch has 2 uplinks available, and assuming that the remaining available bandwidth from a spine1 node to a leaf2 node is 10Gbps.

An elephant flow1 is sent to the server 1 and the server 2, assuming that the elephant flow rate is 16Gbps. The hash routing based on the ID (typically five-tuple) of the flow on the leaf switch is random, and cannot be optimized by WCMP according to the spine egress link, so that the flow may be hashed to the first uplink, and the elephant flow is sent to the spine1 node, and since only 10Gbps remain from spine1 to leaf2, congestion is caused at the spine1- > leaf2 egress, congestion control, flow control and other mechanisms are triggered, and finally the sending rate of flow1 is limited to be less than 10Gbps, and meanwhile, the flow tail delay is increased, as shown in fig. 8 (a).

If the message carries an msg_order_flag=true flag, the leaf switch can identify the service type to which the message belongs and flexibly perform hash routing based on the msg_id+flow ID. Because of the randomness of the hash, in this way, each uplink of leaf1 shares 8Gbps traffic, thus not causing the congestion of the egress port of spline 1- > leaf2 to occur. The final flow1 can be sent at a full rate of 16Gbps without loss of throughput, as shown in fig. 8 (b).

When the message arrives at the server 2, the receiving network card transmission layer judges that the message is not required to be subjected to the inter-message order keeping based on the msg_order_flag carried by the message, and the disordered message is not required to be rearranged, so that the message disordered order rearrangement overhead is not brought.

Second embodiment this embodiment illustrates the overall process flow of using a message-based fine-grained sink transport layer. Assuming that a certain service flow1 does not need order preservation among messages, a message carries an msg_order_flag=true mark, when a receiving end transmission layer analyzes the message, transmission layer receiving processing is performed based on the granularity of the message, and the embodiment illustrates advantages brought by a fine granularity processing mode based on the granularity of the message by two cases.

Case 1: flow1 contains two msg, msg1 and msg2, respectively, msg1, msg2 are each split into 4 messages. Wherein the message psn of msg1 is 1-4, and the message psn of msg2 is 5-8. The messages are sent in the order of psn1- > psn8 at the source end. Suppose that two msg messages arrive out of order (still in the message) but no message is lost because of the load balancing policy based on msg granularity. Assume that the PSN sequence of the message reaching the receiving end is 1,2,5,3,6,4,7,8. The processing procedure of the message by the receiving end is as follows:

1) When the PSN1 message arrives, the msg number is 1, the SOM mark is 1, a variable msg1_psn_cur=1 and msg1_som_received=true is created, the PSN1 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

2) When the PSN2 message arrives, the msg number is 1, the SOM mark is 0, the msg1_psn_cur=1 is searched according to the msg1, the msg1_som_received=true, and because the PSN of the PSN2 message is 2 and is equal to the msg1_psn_cur+1, the current message sequence is correct, the msg1_psn_cur is updated to 2, the PSN2 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

3) When the PSN5 message arrives, the msg number is 2, the SOM mark is 1, a variable msg2_psn_cur=5 and msg2_som_received=true is created, the PSN5 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

4) When the PSN3 message arrives, the msg number is 1, the SOM mark is 0, the msg1_psn_cur=2 is searched according to the msg1, the msg2_som_received=true, and because the PSN of the PSN3 message is 3 and is equal to the msg1_psn_cur+1, the current message sequence is correct, the msg1_psn_cur is updated to 3, the PSN3 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

5) When the PSN6 message arrives, the msg number is 2, the SOM mark is 0, the msg2_psn_cur=5 is searched according to the msg2, the msg2_som_received=true, and because the PSN of the PSN6 message is 6 and is equal to the msg2_psn_cur+1, the current message sequence is correct, the msg2_psn_cur is updated to 6, the PSN6 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

6) When the PSN4 message arrives, the msg number is 1, the SOM mark is 0, msg1_psn_cur=3 is searched according to msg1, and msg1_som_received=true, because the PSN of the PSN4 message is 4 and is equal to msg1_psn_cur+1, the current message sequence is correct, and the PSN4 message is delivered to the host memory. Because the EOM mark is 1, the message is processed to complete the related processing, and the msg1_psn_cur variable is cleared;

7) When the PSN7 message arrives, the msg number is 2, the SOM mark is 0, the msg2_psn_cur=6 is searched according to the msg2, the msg2_som_received=true, and as the PSN of the PSN7 message is 7 and is equal to the msg2_psn_cur+1, the current message sequence is correct, the msg2_psn_cur is updated to 7, the PSN7 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

8) When the PSN8 message arrives, the msg number is 2, the SOM mark is 0, msg2_psn_cur=7 is searched according to msg2, and the PSN of the PSN8 message is 8 and equal to msg2_psn_cur+1, the current message sequence is correct, and the PSN8 message is delivered to the host memory. With EOM marked 1, do the message to complete the correlation and clear the msg2_psn_cur variable.

In this case, although the packet PSN arrives at the receiving end out of order, since no order preservation is required between messages, it is only determined whether or not the packet is out of order in msg, and therefore, packet loss and retransmission flows are not triggered.

Case 2 flow1 contains four msg, msg1, msg2, msg3 and msg4, each of msg1, msg2 and msg4 is split into 4 messages, and msg3 is split into one message. Wherein the message psn of msg1 is 1-4, the message psn of msg2 is 5-8, the message psn of msg3 is 9, and the message psn of msg4 is 10-13. The messages are sent in the order of psn1- > psn13 at the source end. Assuming that the packet PSN3, PSN9 and PSN10 are lost due to error code in transmission, the PSN sequence of the receiving end receiving the packet is 1,2,4,5,6,7,8,11,12,13. The processing procedure of the message by the receiving end is as follows:

2) When the PSN2 message arrives, the msg number is 1, the SOM mark is 0, the msg1_psn_cur=1 is searched according to the msg1, the msg1_som_received=true, and because the PSN of the PSN2 message is 2 and is equal to the msg1_psn_cur+1, the current message sequence is correct, the msg1_psn_cur is updated to 2, the PSN1 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

3) When the PSN4 message arrives, the msg number is 1, the SOM mark is 0, msg1_psn_cur=2 is searched according to msg1, msg1_som_received=true, and because the PSN of the PSN4 message is equal to 4 and is not equal to msg1_psn_cur+1 (namely 3), the current message sequence is incorrect, the current PSN4 message is discarded, and NACK messages are returned to the transmitting end, wherein the information of msg1, PSN3 and msg_order_flag=true is carried;

4) When the PSN5 message arrives, the msg number is 2, the SOM mark is 1, a variable msg2_psn_cur=5 and msg2_som_received=true is created, the PSN5 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

6) When the PSN7 message arrives, the msg number is 2, the SOM mark is 0, the msg2_psn_cur=6 is searched according to the msg2, the msg2_som_received=true, and as the PSN of the PSN7 message is 7 and is equal to the msg2_psn_cur+1, the current message sequence is correct, the msg2_psn_cur is updated to 7, the PSN7 message is delivered to a host memory, the EOM mark is 0, and the relevant processing is completed without the message;

7) When the PSN8 message arrives, the msg number is 2, the SOM mark is 0, msg2_psn_cur=7 is searched according to msg2, and msg2_som_received=true, because the PSN of the PSN8 is 8 and is equal to msg2_psn_cur+1, the current message sequence is correct, and the PSN8 message is delivered to the host memory. Because the EOM mark is 1, the message is processed to complete the related processing, and the msg2_psn_cur variable is cleared;

8) The PSN11 message arrives, the msg number is 4, the SOM mark is 0, the msg4 is extracted, the msg4_ som _received information is not found, the first packet is lost, and the original flow is taken away;

9) The PSN12 message arrives, the msg number is 4, the SOM mark is 0, the msg4 is extracted, the msg4_ som _received information is not found, the first packet is lost, and the original flow is taken away;

10 PSN13 message arrives, msg number is 4, SOM mark is 0, msg4 is extracted, msg4_ som _received information is not found, the first packet is lost, and the original flow is taken away;

11 If the PSN3 message arrives, msg is numbered 1, SOM is marked as 0, msg1_psn_cur=2 is searched according to msg1, msg1_som_received=true, because PSN of PSN3 is 3 and is equal to msg1_psn_cur+1, the current message sequence is correct, msg1_psn_cur is updated to 3, PSN3 message is delivered to a host memory, EOM is marked as 0, and relevant processing is not completed by the message;

12 If the PSN4 message arrives, msg is numbered 1, SOM is marked as 0, msg1_psn_cur=3 is searched according to msg1, msg1_som_received=true, and because PSN of PSN4 is 4 and is equal to msg1_psn_cur+1, the current message sequence is correct, and the PSN4 message is delivered to the host memory. Because the EOM mark is 1, the message is processed to complete the related processing, and the msg1_psn_cur variable is cleared;

13 The message of PSN9 retransmitted over time arrives, msg number is 3, and the message is delivered to the memory of the host computer for processing the message to finish the related processing because of carrying SOM and EOM marks;

14 The processing flow of the packet arrival of PSN 10-PSN 13 of the overtime retransmission is identical to msg2, and is not repeated.

In this case, although packet loss occurs, since the retransmission range of the packet loss is limited within msg, only 2 packets need to be retransmitted, the retransmission rate of the packet loss is reduced, and thus the transmission efficiency is improved.

In the third embodiment, the present embodiment illustrates a packaging extension mode that the service packet carries msg_id (if needed) and msg_order_flag information, and the NACK packet carries msg_id (if needed) and msg_order_flag, taking the current relatively popular Infiniband (IB) and UEC protocols as examples.

For the Infiniband protocol, the basic encapsulation format of the message is shown in FIG. 9. The Opcode is used for indicating the type of the current message, the Opcode has 8 bits in total, when the Opcode starts with 000, the current message is indicated to be an RC service message or an ACK/NACK message, and the last 5 bits are used for indicating the specific type of the message.

In the basic encapsulation format, there are two Reserved fields which can be multiplexed, wherein 1bit can be used to represent msg_order_flag information, and the remaining 12 bits can be used to identify msg_id information.

The new RC message type may be defined by modifying Opcode. In the Opcode field of RC operation, values of 00011000-00011111 are reserved values, which can be used to define a new encapsulation format. For example, the present invention takes 00011000 to indicate that the message is a Write message and does not require inter-message ordering, and similarly, 00011111 may be taken to indicate that the message is a NACK message, which is also the subject of the present invention. The NACK message is followed by a AETH header (Ack Extension Transport Header), which is defined unchanged.

For the UEC protocol, the extension based on the semantic header encapsulation format is shown in fig. 11, and the message already carries a 16bit message id field, so that the extension is not needed. In addition, there are 3 Reserved fields in the message, 13 bits in total, which can be used to carry msg_order_flag, in the figure, the msg_order_flag is carried by 1bit of the first Reserved field. The Opcode indicates in the reserved value the message encapsulation format described in the present invention, e.g., 0x11.

In the NACK message, the extended encapsulation format is shown in fig. 12. Wherein messge _id is carried in the semantic header, which is a generic return header encapsulation. When the Code value is not equal to 0, the PSN is indicated to be a NACK message currently, and an index generation field in a universal semantic header of the NACK message is an invalid field and can be used for carrying msg_order_flag. The Opcode redefines the encapsulation type in the reserved value indicating the semantic header and delivery sublayer message, e.g., the value 0x12.

The three embodiments described above are only preferred embodiments of the present application and are not intended to limit the present application, and various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

The foregoing describes certain embodiments of the present application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Fig. 13 is a schematic structural view of an electronic device according to an embodiment of the present application. Referring to fig. 13, at the hardware level, the electronic device includes a processor, and optionally an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (PERIPHERAL COMPONENT INTERCONNECT, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 13, but not only one bus or type of bus.

And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.

The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs, and forms the RDMA-based message sending device on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:

The method performed by the RDMA-based messaging device disclosed in the embodiment of FIG. 13 of the present application may be implemented in or by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The Processor may be a general-purpose Processor including a central processing unit (Central Processing Unit, CPU), a network Processor (Network Processor, NP), etc., or may be a digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components. The methods, steps and logic blocks disclosed in the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present application may be embodied directly in a hardware decoding processor for execution, or in a combination of hardware and software modules in a decoding processor for execution. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

The electronic device may also execute the method of fig. 2 and implement the functions of the RDMA-based messaging device in the embodiment shown in fig. 2, which is not described herein.

Of course, other implementations, such as a logic device or a combination of hardware and software, are not excluded from the electronic device of the present application, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or a logic device.

The present application also proposes a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a portable electronic device comprising a plurality of application programs, enable the portable electronic device to perform the method of the embodiment of fig. 2, and in particular to perform the operations of:

Fig. 14 is a schematic structural diagram of an RDMA-based messaging device 140 according to an embodiment of the present application. Referring to fig. 14, in a software implementation, the RDMA-based packet sending device 140 may include a sending module 141, where:

The sending module 141 sends a first message to the receiving end, where the first message carries indication information and identification information of a message to which the first message belongs, where the indication information is used to indicate whether to allow out-of-order reception between messages.

In some embodiments, the sending module 141 sends the first message to the receiving end, including:

sending a first message to the receiving end according to the first parameter;

And when the first parameter indicates that the message is not allowed to be received in disorder, the indication information carried in the first message is used for indicating that the message is not allowed to be received in disorder.

In some embodiments, the sending module 141 receives the first parameter sent by the application layer, including:

And receiving the first parameter sent by the application layer based on an extended programming interface, wherein the extended programming interface comprises an extended first field, and the first field is used for bearing the first parameter.

In some embodiments, the first message further carries an operation code, and the sending module 141 performs load balancing according to the identification information when the first condition is satisfied;

The first condition comprises that the operation code is in a reliable mode and the indication information is used for indicating that the disordered reception of the messages is allowed, and the load balancing according to the identification information comprises that the messages with the same identification information are sent to the same transmission path, and the messages with different identification information are sent to the same or different transmission paths.

In some embodiments, in the case that the indication information is used to indicate that the out-of-order reception between messages is allowed, the sending module 141 further includes:

Receiving a NACK message sent by the receiving end, wherein the NACK message is sent by the receiving end under the condition that the receiving end determines that the message in the message to which the first message belongs is lost or disordered;

And retransmitting the message according to the NACK message.

In some embodiments, the NACK message carries the identification information and a retransmitted start packet sequence number, and the sending module 141 performs a message retransmission according to the NACK message, including:

And sending a message from the initial packet sequence number in a message corresponding to the identification information to the receiving end according to the identification information and the initial packet sequence number.

The RDMA-based messaging device 140 provided in the present application may further execute the method of fig. 2, and implement the functions of the RDMA-based messaging device 140 in the embodiment shown in fig. 2, which are not described herein.

Fig. 15 is a schematic structural view of an electronic device according to an embodiment of the present application. Referring to fig. 15, at the hardware level, the electronic device includes a processor, and optionally an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.

The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (PERIPHERAL COMPONENT INTERCONNECT, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in fig. 15, but not only one bus or one type of bus.

The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs, and forms the message processing device based on RDMA on the logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:

The method performed by the RDMA-based message processing apparatus disclosed in the embodiment of FIG. 15 of the present application may be implemented in or by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The Processor may be a general-purpose Processor including a central processing unit (Central Processing Unit, CPU), a network Processor (Network Processor, NP), etc., or may be a digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components. The methods, steps and logic blocks disclosed in the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present application may be embodied directly in a hardware decoding processor for execution, or in a combination of hardware and software modules in a decoding processor for execution. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.

The electronic device may also execute the method of fig. 6 and implement the functions of the RDMA-based message processing apparatus in the embodiment shown in fig. 6, which is not described herein.

The present application also proposes a computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a portable electronic device comprising a plurality of application programs, enable the portable electronic device to perform the method of the embodiment of fig. 6, and in particular to perform the following operations:

FIG. 16 is a schematic diagram of an RDMA-based message processing apparatus 160 according to an embodiment of the present application. Referring to FIG. 16, in a software implementation, the RDMA-based message processing apparatus 160 may include a receiving module 161 and a processing module 162, where:

A receiving module 161, configured to receive a first message sent by a sending end, where the first message carries indication information and identification information of a message to which the first message belongs, where the indication information is used to indicate whether to allow out-of-order reception between messages;

The processing module 162 processes the first message according to the indication information and the identification information, where the processing includes performing out-of-order reception between messages or performing out-of-order reception between messages to which the first message belongs.

In some embodiments, the processing module 162 processes the first packet according to the indication information and the identification information, including:

under the condition that the indication information indicates that disordered reception is allowed between messages, carrying out first processing on the first message according to the identification information, wherein the processing principle of the first processing comprises the steps of preserving the sequence of a plurality of messages in the message to which the first message belongs and not preserving the sequence between messages to which the first message belongs;

And under the condition that the indication information indicates that the disordered reception of the messages is not allowed, carrying out second processing on the first message, wherein the processing principle of the second processing comprises the steps of preserving the sequence of a plurality of messages in the message to which the first message belongs and preserving the sequence of the messages to which the first message belongs.

In some embodiments, the first packet further carries a packet sequence number, a first packet flag, a last packet flag, an identifier of a source queue pair, and an identifier of a destination queue pair, and the processing module 162 performs a first process on the first packet according to the identifier information, where the first process includes:

judging whether the first message is the first message or the last message in the affiliated message according to the first packet mark and the tail packet mark;

And performing first processing on the first message according to the judging result, the identification information, the packet sequence number, the identification of the source queue pair and the identification of the destination queue pair.

In some embodiments, the determining that the first message is a first message in the belonging message includes the processing module 162 performing a first processing on the first message according to the determining result, the identification information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair, including:

Initializing the first variable and the second variable, wherein the value of the first variable after initialization is the packet sequence number of the first message, and the value of the second variable after initialization is used for indicating that the first message in the message is successfully received;

and delivering the first message to a host memory.

In some embodiments, the determining result includes that the first message is not the first message or the last message in the belonging message, and the processing module 162 performs a first process on the first message according to the determining result, the identification information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair, including:

Searching a first variable corresponding to the identification information, the identification of the source queue pair and the identification of the destination queue pair under the condition that the second variable exists and is used for indicating that the first message in the message is successfully received;

delivering the first message to a host memory and updating the value of the first variable to the packet sequence number under the condition that the packet sequence number is equal to the sum of the value of the first variable and 1;

And under the condition that the packet sequence number is not equal to the sum of the value of the first variable and 1, determining that packet loss or disorder occurs in the message to which the first message belongs, and sending a NACK message to the sending end.

In some embodiments, the NACK message carries the identification information and a retransmitted starting packet sequence number, where the starting packet sequence number is equal to a sum of a value of the first variable and 1.

In some embodiments, the determining that the first message is the last message in the message includes the processing module 162 performing a first process on the first message according to the determining result, the identification information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair, including:

releasing the first variable and the second variable;

delivering the first message to a host memory;

and sending an ACK message to the sending end.

The RDMA-based message processing apparatus 160 provided in the present application may further execute the method of fig. 6, and implement the functions of the RDMA-based message processing apparatus 160 in the embodiment shown in fig. 6, which are not described herein.

The embodiment of the application also provides a message sending and processing system based on RDMA, which comprises a sending end and a receiving end. The sending terminal sends a first message to the receiving terminal, wherein the first message carries indication information and identification information of a message to which the first message belongs, and the indication information is used for indicating whether disordered reception between the messages is allowed or not. The receiving end receives the first message, and processes the first message according to the indication information and the identification information, wherein the processing comprises the steps of carrying out message disorder receiving or message sequence preserving receiving on the message to which the first message belongs.

In the embodiment of the present application, the specific implementation manner of each step executed by the transmitting end may refer to the specific implementation manner of the corresponding step in the embodiment shown in fig. 2, and the same technical effects can be achieved, which will not be described in detail herein. The specific implementation of each step performed by the receiving end may refer to the specific implementation of the corresponding step in the embodiment shown in fig. 6, and the same technical effect can be achieved, which is not described in detail herein.

The present application also proposes a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of the above-described RDMA-based messaging method embodiments or to perform some or all of the steps of the above-described RDMA-based message processing method embodiments.

In summary, the foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.

The embodiments of the present application are described in a progressive manner, and the same and similar parts of the embodiments are all referred to each other, and each embodiment is mainly described in the differences from the other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

Claims

1. A message sending method based on RDMA is applied to a sending end, and comprises the following steps:

2. The method of claim 1, wherein the sending the first message to the receiving end includes:

sending a first message to the receiving end according to the first parameter;

3. The method of claim 2, the receiving the first parameter sent by the application layer, comprising:

4. The method of claim 1, wherein the first message further carries an operation code, and wherein the method further comprises:

Under the condition that the first condition is met, carrying out load balancing according to the identification information;

5. The method of claim 1, wherein in the case where the indication information is used to indicate that out-of-order reception between messages is allowed, the method further comprises:

And retransmitting the message according to the NACK message.

6. The method of claim 5 wherein the NACK message carries the identification information and a retransmitted starting packet sequence number, and wherein the retransmitting the message according to the NACK message comprises:

7. A message processing method based on RDMA is applied to a receiving end, and comprises the following steps:

8. The method of claim 7, wherein the processing the first message according to the indication information and the identification information comprises:

9. The method of claim 8, wherein the first message further carries a packet sequence number, a first packet flag, a last packet flag, an identifier of a source queue pair, and an identifier of a destination queue pair, and wherein the performing the first processing on the first message according to the identifier information comprises:

10. The method of claim 9, wherein the determining that the first message is a first message in the belonging message comprises performing a first process on the first message according to the determining that the first message is a first message in the belonging message, the identification information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair, including:

and delivering the first message to a host memory.

11. The method of claim 9, wherein the determining that the first message is not a first message or a last message in the message comprises performing a first process on the first message according to the determining that the first message is not the first message or the last message in the message, the identifying information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair, including:

12. The method of claim 11, wherein the NACK message carries the identification information and a retransmitted starting packet sequence number, the starting packet sequence number being equal to a sum of a value of the first variable and 1.

13. The method of claim 9, wherein the determining that the first message is the last message in the message comprises performing a first process on the first message according to the determining that the first message is the last message in the message, the identification information, the packet sequence number, the identification of the source queue pair, and the identification of the destination queue pair, including:

releasing the first variable and the second variable;

delivering the first message to a host memory;

and sending an ACK message to the sending end.

14. An RDMA-based message sending and processing system comprises a sending end and a receiving end, wherein the sending end and the receiving end are connected through a network;

15. An electronic device, comprising:

A processor;

a memory for storing the processor-executable instructions;

Wherein the processor is configured to execute the instructions to implement the method of any one of claims 1 to 13.

16. A computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the method of any of claims 1 to 13.

17. A computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of the method of any of claims 1 to 13.