CN118266203A - Intelligent NIC grouping - Google Patents

Abstract

Some embodiments provide a method for a first smart NIC of a plurality of smart NICs of a host computer. Each of the smart NICs executes a smart NIC operating system that performs virtual network operations for a set of data computing machines executing on the host computer. The method receives a data message sent by one of the data computing machines executing on the host computer. The method performs virtual network operations on the data message to determine that the data message is to be transmitted from a port of a second smart NIC of the plurality of smart NICs. The method delivers the data message to the second smart NIC via a dedicated communication channel connecting the plurality of smart NICs.

Description

Smart NIC Teaming

Background technique

Many operations typically associated with a host computer are pushed to a programmable intelligent network interface controller (NIC). Some of the operations pushed to these intelligent NICs include virtual network processing for data messages of the computing machine. In some cases, the host computer will have multiple such intelligent NICs that perform network processing or other operations. It is desirable to enable these intelligent NICs to work together while performing operations that were previously performed on the host computer (e.g., by the host computer's hypervisor).

Summary of the invention

Some embodiments provide methods for enabling multiple smart NICs of the same host computer to operate as a single entity (e.g., as a group of smart NICs). In some embodiments, the smart NICs each execute a smart NIC operating system that performs virtual network operations (and/or other operations, such as virtual storage operations) for a group of data computing nodes (e.g., virtual machines (VMs), containers, etc.) executing on the host computer. In some embodiments, the smart NICs are connected by a dedicated communication channel to share dynamic state information, share configuration data (so that one of the smart NICs can act as a single point of contact for a network management and control system), and/or pass data messages between each other that are sent to and from a data computing node (DCN) that requires virtual network processing.

By executing the Smart NIC operating system, the Smart NICs are able to perform various tasks that would otherwise be performed by host computer software (e.g., the host computer's hypervisor). These tasks may include virtual network processing for data messages (i.e., performing virtual switching and/or routing, firewall operations, etc.), virtual storage operations, etc. In order for multiple Smart NICs to perform these operations that would otherwise be performed entirely by a single entity (e.g., a hypervisor), communication between the Smart NICs may be required.

As mentioned, in some embodiments, a dedicated communication channel is established between the smart NICs to enable communication between the smart NICs. In some embodiments, the dedicated communication channel is a physically separate channel. For example, in some embodiments, the smart NICs are connected via a set of physical cables that carry only communication between the smart NICs. In different such embodiments, the smart NICs may be connected in series (so that each smart NIC is directly connected to two other smart NICs, except for the smart NICs at the ends of the connection that are only connected to one other smart NIC), connected in a loop (similar to a serial connection, but where all smart NICs are connected to two other smart NICs), or connected via a separate physical switch so that each smart NIC can communicate directly with any other smart NIC through the physical switch. If there are enough available ports, the cables may be connected to the Ethernet ports of the smart NICs (thereby occupying these ports so that the network traffic of the host computer does not use these ports) or to the management ports of the smart NICs (usually lower bandwidth ports). In some embodiments, the smart NICs use separate specially constructed channels that are designed to connect the smart NICs to each other, rather than occupying ports that can be used for other purposes.

In other embodiments, the smart NICs communicate via a logical dedicated communication channel using existing physical connections. For example, if all smart NICs are connected to the same data link layer (layer 2) network, then a dedicated virtual local area network (VLAN) can be used as a dedicated communication channel for the smart NICs. However, if this existing layer 2 network has many other host computers (the host computers have their own smart NIC groups that require separate VLANs) and also carry data messages for the DCN on the host computers, then the maximum number of VLANs can be reached. Some embodiments instead use an overlay network based on encapsulation (e.g., Virtual Extensible LAN (VXLAN) or Generic Network Virtualization Encapsulation (Geneve)) as the logical dedicated communication channel. Such an overlay network is not as constrained in number as VLANs and also has the benefit of enabling the smart NICs to communicate across multiple layer 2 networks when necessary (i.e., as long as the smart NICs are all on the same layer 3 network).

In other embodiments, the intelligent NIC of the host computer communicates via a dedicated communication channel through the host computer. For example, the intelligent NIC is typically connected to the host computer's Peripheral Component Interconnect Express (PCIe) subsystem, which can be used for the dedicated communication channel. In various embodiments, the intelligent NIC uses the standard point-to-point transfer feature of PCIe, utilizes the PCIe switch fabric, or uses other enhancements over PCIe (e.g., Compute Express Link (CXL)).

As mentioned, one use of the dedicated communication channel is for a first smart NIC to pass a data message (e.g., a data message sent to or from a host computer or a DCN executing on a host computer) to a second smart NIC. The smart NICs operate as a single entity because their smart NIC operating systems collectively implement a set of virtual network operations (e.g., implement logical switches and/or routers, firewalls, etc.). However, each smart NIC has its own interfaces (e.g., physical functions and virtual functions) and its own physical network ports to which the host computer's DCN is bound.

Thus, the first smart NIC will receive data messages from the DCN bound to the port of the smart NIC. If the smart NICs collectively implement virtual network operations, then this first smart NIC processes these data messages. However, based on this processing, the data messages may need to be transmitted to the second smart NIC via a dedicated communication channel so that the second smart NIC can output the data messages. For example, if the destination is another DCN bound to the second smart NIC on the host computer, then the first smart NIC will need to pass the data message to the second smart NIC so that the data message can be output via the correct interface. In addition, if all ports of the smart NICs are grouped in a link aggregation group (LAG), the connections for a single DCN will be load balanced across these ports, so some of the data messages sent to the first smart NIC from a specific DCN bound to the interface of the first smart NIC will be output to the physical network via other smart NICs. In contrast, data messages received at the physical network port of the first smart NIC will be processed by the first smart NIC, but may need to be sent to the second smart NIC for delivery to the destination DCN bound to the second smart NIC. In another scenario, if all of the physical network ports of a first smart NIC have failed, but the smart NIC itself is still operational, the smart NIC can still perform virtual network operations on data messages, but will need to send those data messages to other smart NICs for output to the physical network, regardless of whether the ports are operating in a LAG.

In many cases, the smart NICs receive configuration data for virtual network operations from a network management and control system. Each of the smart NICs has its own set of ports (possibly including a management port) with its own network address, but many network management and control systems treat each host computer as a single entity (e.g., communicating with an agent in a hypervisor of a host computer that does not use the smart NICs for network virtualization operations). The network management and control system uses a single management network address for each host computer and therefore should not communicate directly with all of the multiple smart NICs of a host computer.

In some embodiments, the SmartNICs use clustering techniques in order to appear as a single entity to the host computer to the network management and control system. For example, in some embodiments, the SmartNICs of the host computer perform leader election to determine a single one of the SmartNICs to communicate with the network management and control system. In some such embodiments, each of the SmartNIC operating systems runs a deterministic algorithm that selects one of the SmartNICs as the point of contact. Any messages required for this leader election are delivered over a dedicated communication channel.

The selected smart NIC receives configuration data (e.g., logical switch and logical router configuration data) from the network management and control system and spreads this data to other smart NICs via a dedicated communication channel so that all smart NICs can perform virtual network operations on data messages sent to and from the DCN executing on the host computer. In some embodiments, the network management and control system includes a management plane (MP) and a central control plane (CCP), which perform different functions and provide different configuration data to the host computer (in addition to receiving different data from the host computer). In some cases, the smart NIC elects two different leaders, one leader for communicating with the MP and one leader for communicating with the CCP.

In addition to propagating configuration data from the network management and control system, the leader SmartNIC also receives information from other SmartNICs via dedicated communication channels, some of which is reported to the network management and control system. This information may include runtime statistics (e.g., data message processing statistics), status information, etc., and may be used by the network management and control system and/or the leader SmartNIC to monitor the host computer and/or SmartNIC. The network management and control system may also use this information to modify the virtual network configuration for the SmartNIC.

For various purposes, in some embodiments, the smart NICs also use a dedicated communication channel to synchronize dynamic state information. For example, in the event that a leader smart NIC fails, monitoring data retrieved by the selected leader smart NIC may be synchronized with at least one backup smart NIC. In addition, when performing virtual network processing, the smart NICs may need to store dynamic state information and share the data with each other. In many cases, the smart NIC operating system stores connection tracking information that indicates open connections and congestion windows for each open connection. This connection tracking information is used by firewall operations to determine whether to allow or drop/block data messages. If a smart NIC becomes inoperable and has not yet shared any state with other smart NICs, then all connections managed by the smart NIC will be transferred to other smart NICs, and the other smart NICs will not have any record of the connection. In this way, the smart NICs share this connection tracking state information with each other, enabling seamless handling of failover between smart NICs.

This state sharing can also be used by SmartNICs that are performing operations other than virtual networking (or SmartNICs that are performing multiple types of operations that use state sharing). If storage virtualization operations are handled by the SmartNICs, then in some embodiments, the storage virtualization functions include running a network stack to manage layer 4 connections to the storage devices. In this case, if a failover occurs, the connection information should be shared again between the SmartNICs so that if one of the SmartNICs fails, the connections are not reset.

The foregoing summary is intended to serve as a brief introduction to some embodiments of the present invention. This is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The following detailed description and the drawings referenced in the detailed description will further describe the embodiments described in the summary as well as other embodiments. Therefore, in order to understand all the embodiments described in this document, a comprehensive review of the summary, the detailed description, and the drawings is required. In addition, the claimed subject matter is not limited by the illustrative details in the summary, the detailed description, and the drawings, but will be defined by the appended claims, because the claimed subject matter may be embodied in other specific forms without departing from the spirit of the subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appended claims.For purposes of illustration, however, several embodiments of the invention are set forth in the following figures.

FIG. 1 conceptually illustrates a host computer with multiple physical Smart NICs performing network virtualization operations.

FIG. 2 conceptually illustrates the collective operation of a group of intelligent NICs of a single host computer of some embodiments.

FIG. 3 conceptually illustrates some embodiments in which the smart NICs of a host computer are connected in series.

4 conceptually illustrates some embodiments in which each intelligent NIC of a host computer is directly connected to each other intelligent NIC of the host computer.

FIG. 5 conceptually illustrates an example of a host computer's SmartNICs connected through a single physical switch.

6 conceptually illustrates two host computers, each having three smart NICs, connected to a data center network and using an overlay on the data center network as their respective dedicated communication channels.

FIG. 7 conceptually illustrates a host computer with three Smart NICs communicating over the host computer's PCIe bus.

8 conceptually illustrates a process of some embodiments for processing a data message at a smart NIC, which is one of a plurality of smart NICs of a host computer.

FIG. 9 conceptually illustrates the paths of two data messages received at a smart NIC via a physical port of the smart NIC.

FIG. 10 conceptually illustrates the path of two data messages received at a smart NIC from a VM via a VF to which the VM is bound.

11 conceptually illustrates the path of a data message sent from a first VM operating on a host computer to a second VM.

12 conceptually illustrates the path of a multicast data message received at a first smart NIC of a host computer via a physical port of the smart NIC.

13 conceptually illustrates the path of a data message for one of the connections shown in FIG. 10 after a physical port of the intelligent NIC has failed.

FIG. 14 conceptually illustrates a process of some embodiments for configuring multiple Intelligent NICs to perform network virtualization operations.

FIG. 15 illustrates that each of a pair of smart NICs executes a smart NIC operating system having a leader election module.

FIG. 16 illustrates the distribution of configuration data to multiple intelligent NICs.

17 conceptually illustrates a selected leader Intelligent NIC that collects statistics from itself and another Intelligent NIC and reports the statistics to a network management and control system.

FIG. 18 conceptually illustrates three Smart NICs of a host computer, each of which operates a Smart NIC operating system having a leader election module.

FIG. 19 conceptually illustrates two Smart NICs sharing a connection state.

20 illustrates that the first smart NIC in FIG. 19 has become inoperable and the VMs previously bound to the first smart NIC are now bound to the interface of the second smart NIC while the connection remains open.

FIG. 21 conceptually illustrates an electronic system utilized to implement some embodiments of the present invention.

Detailed ways

In the following detailed description of the invention, numerous details, examples and embodiments of the invention are stated and described. However, it will be clear and apparent to those skilled in the art that the invention is not limited to the embodiments stated and that the invention can be practiced without some of the specific details and examples discussed.

Some embodiments provide methods for enabling multiple smart NICs of the same host computer to operate as a single entity (e.g., as a group of smart NICs). In some embodiments, the smart NICs each execute a smart NIC operating system that performs virtual network operations (and/or other operations, such as virtual storage operations) for a group of data computing nodes (e.g., virtual machines (VMs), containers, etc.) executing on the host computer. In some embodiments, the smart NICs are connected by a dedicated communication channel to share dynamic state information, share configuration data (so that one of the smart NICs can act as a single point of contact for a network management and control system), and/or pass data messages between each other that are sent to and from a data computing node (DCN) that requires virtual network processing.

By executing the Smart NIC operating system, the Smart NICs are able to perform various tasks that would otherwise be performed by host computer software (e.g., the host computer's hypervisor). These tasks may include virtual network processing for data messages (i.e., performing virtual switching and/or routing, firewall operations, etc.), virtual storage operations, etc. In order for multiple Smart NICs to perform these operations that would otherwise be performed entirely by a single entity (e.g., a hypervisor), communication between the Smart NICs may be required.

1 conceptually illustrates a host computer 100 having multiple physical smart NICs 105 and 110 that perform network virtualization operations. As shown, the host computer 100 includes multiple DCNs (in this case, virtual machines) 115-125 that are connected to the smart NICs 105 and 110 in a pass-through mode (i.e., without applying any kind of network virtualization processing within the virtualization software 130 of the host computer 100). Each of the VMs 115-125 has an associated virtual NIC (vNIC) 135-145 that is connected to a different virtual function (VF) 161-164 of one of the smart NICs 105 and 110 via a peripheral component interconnect express (PCIe) fabric 165 (a motherboard-level interconnect that connects the physical processors of the host computer 100 to the physical interfaces of the smart NICs 105 and 110).

Each vNIC 135-145, and therefore each VM 115-125, is bound to a different VF of one of the smart NICs 105 or 110. In some embodiments, VFs 161-164 are virtualized PCIe functions that are exposed as interfaces to the smart NICs. Each VF is associated with a physical function (PF), which is a physical interface of the smart NIC that is treated as a unique PCIe resource. In this case, smart NIC 105 has one PF 170 and smart NIC 110 has one PF 175, but in many cases each smart NIC will have more than one PF. PF 170 is virtualized to provide at least VFs 161-162, and PF 175 is virtualized to provide at least VFs 163-164.

In some embodiments, VFs are provided in order to provide different VMs with different virtual interfaces of a smart NIC to which they can each connect. In some embodiments, VF drivers 150-160 are executed in each of the VMs 115-125 to manage the respective connections of the VMs to the VFs. As shown, in some embodiments, each VM 115-125 is associated with a vNIC 135-145 provided by the virtualization software 130 as a software emulation of a NIC. In different embodiments, the VMs 115-125 access the VFs through their respective vNICs 135-145 or directly in a pass-through mode (where the virtualization software 130 does not participate in most network communications). In other embodiments, the VMs 115-125 can switch between this pass-through mode and accessing the VFs via their respective vNICs 135-145. In either case, virtualization software 130 participates in assigning VFs 161 - 164 to VMs 115 - 125 and enables access to the VFs from VF drivers 150 - 160 .

It should also be noted that while in this scenario all network virtualization operations have been transferred from the host computer's virtualization software 130 to the smart NICs 105 and 110, in other embodiments, the virtual switch provided by the virtualization software 130 may be directly connected to the PFs 170 and 175. In some such embodiments, data traffic is sent from the VM via the vNIC to the virtual switch, which provides the traffic to the PF. In this scenario, the virtual switch performs basic switching operations, but leaves the network virtualization operations to the smart NIC.

Intelligent NICs 105 and 110 also include physical network ports 181-184. In different embodiments, the intelligent NICs may each include only a single physical network port or multiple (e.g., 2, 3, 4, etc.) physical network ports. These physical network ports 181-184 provide physical communication for the host computer 100 to the data center network. In addition, a dedicated communication channel 180 is shown between the two intelligent NICs 105 and 110 that allows these intelligent NICs to communicate. As further described below, this communication channel 180 can take various forms (e.g., a direct physical connection, a logical connection via an existing network, a connection via PCIe messages).

Finally, Figure 1 illustrates that smart NICs 105 and 110 perform network virtualization operations 185. In some embodiments, these operations may include logical switching and/or routing operations, distributed firewall operations, encapsulation, and other network operations that are typically performed in the virtualization software of the host computer. In some embodiments, all smart NICs of a given host computer have the same virtual network configuration.

Although not shown in the figure, in some embodiments, each smart NIC is a NIC that includes (i) packet processing circuitry, such as an application specific integrated circuit (ASIC), (ii) a general purpose central processing unit (CPU), and (iii) memory. In some embodiments, the packet processing circuitry is an I/O ASIC that handles the processing of data messages forwarded to and from the DCN in the host computer and is controlled at least in part by the CPU. In other embodiments, the packet processing circuitry is a field programmable gate array (FPGA) configured to perform packet processing operations or a firmware programmable processing core specialized for network processing (which differs from a general purpose CPU in that the processing core is specialized and therefore more efficient at packet processing). In some embodiments, the CPU executes a NIC operating system that controls the packet processing circuitry and may run other programs. In some embodiments, the CPU configures the packet processing circuitry to implement network virtualization operations by configuring flow entries that the packet processing circuitry uses to process data messages.

When a data message is sent by one of the VMs 115 to 125, the data message is sent (in the software of the host computer 100) via the corresponding vNIC 135 to 145. The data message is passed to the corresponding VF 161 to 164 of the appropriate intelligent NIC over the PCIe bus 165. The intelligent NIC ASIC processes the data message to apply the configured network virtualization operation 185 and then (as long as the data message does not need to be sent to another intelligent NIC of the host computer and the destination of the data message is external to the host computer) sends the data message out of one of its physical ports 181 to 184.

It should be noted that although FIG. 1 illustrates a host computer with virtualization software on which various VMs operate, the discussion of smart NICs herein also applies to host computers hosting other types of virtualized DCNs (e.g., containers) as well as bare metal computing devices (i.e., the computer does not execute virtualization software on the bare metal computing device). In the latter case, the bare metal computing device will typically directly access the PFs of multiple smart NICs, rather than any VFs. That is, the smart NICs are used to provide network virtualization (or other operations, such as storage virtualization), without the software on the computing device being aware of these operations.

2 conceptually illustrates the collective operation of a set of smart NICs 205-210 of a single host computer of some embodiments. Each of these smart NICs includes multiple VFs for communicating with the host computer's VMs and multiple physical ports for communicating with a data center network (e.g., to which other host computers that may or may not use the smart NICs are also connected).

Each of the smart NICs runs (i.e., on the CPU of the respective smart NIC) a smart NIC operating system 215-220. Each smart NIC operating system 215-220 controls the ASIC of the smart NIC and performs additional operations, such as network virtualization operations 225 and storage virtualization operations 230. These operations 225 and 230 (and in other embodiments, other types of operations) are spread across the various smart NICs 215-220 of the host computer so that the smart NICs appear to operate as a single entity (i.e., in the same way that the virtualization software of the host computer is a single entity). As indicated above, the network virtualization operations 225 include performing logical switching and/or routing of data messages for one or more logical forwarding elements, applying distributed firewall rules, performing network address translation, and other network features. If each of the smart NICs 205-210 is configured to perform the same network virtualization operations, any of the smart NICs can receive a data message directed to or sent from one of the DCNs executing on the host computer and properly process such a data message.

Similarly, if storage virtualization operations 230 are configured across all smart NICs, a VM can bind to any of the smart NICs and can handle I/O requests from the VM to the virtual storage network. The VM binds to the smart NIC network adapter VF for network operations, while the VF to which the VM binds for storage virtualization purposes is a storage VF (e.g., a non-volatile memory express (NVMe) device or a small computer system interface (SCSI) device).

In order for multiple intelligent NICs to perform these operations as if they were a single entity (similar to the hypervisor of a host computer), communication may be required between the intelligent NICs. Therefore, in some embodiments, a dedicated communication channel is established between the intelligent NICs to enable communication between the intelligent NICs.

In some embodiments, the dedicated communication channel is a physically separate channel. For example, in some embodiments, the smart NICs are connected via a set of physical cables that carry only communications between the smart NICs. FIG. 3 conceptually illustrates some embodiments in which the smart NICs of a host computer 300 are connected in series. As shown, each of smart NICs 305-320 is connected to two other smart NICs. That is, smart NIC 305 is connected to smart NICs 320 and 310, smart NIC 310 is connected to smart NICs 305 and 315, smart NIC 315 is connected to smart NICs 310 and 320, and thus smart NIC 320 is connected to smart NICs 315 and 305. Depending on the physical arrangement of the smart NICs, some embodiments do not directly connect the smart NICs at the ends (i.e., in the example, smart NICs 305 and 320 will not be connected).

Having a full ring connection (as shown in FIG. 3 ) allows any of the smart NICs 305-320 to communicate with any of the other smart NICs in the event that one of the communication links (or one of the smart NICs itself) fails. For example, if the link between smart NIC 310 and smart NIC 315 fails, smart NIC 310 can still reach smart NIC 315 via the other two smart NICs 305 and 320. Similarly, if smart NIC 310 itself fails, smart NIC 305 can still reach smart NIC 315 via smart NIC 320.

To achieve more robust fault protection, some embodiments include a dedicated communication channel link between each pair of smart NICs (i.e., a fully meshed connection). FIG. 4 conceptually illustrates some embodiments in which each smart NIC of a host computer 400 is directly connected to each other smart NIC of the host computer. As shown, each of smart NICs 405-420 is directly connected to each of the other three smart NICs 405-420. In this setup, if the host computer has N smart NICs, then each smart NIC requires N-1 direct connections to the other smart NICs. This setup is reasonable for host computers with a reasonably small number of smart NICs (e.g., 3 to 5 smart NICs), but becomes more difficult for larger numbers of smart NICs.

In some embodiments, these connections may use separate specially constructed channels for inter-NIC communication. In other embodiments, if the smart NIC has enough physical ports, the connections may repurpose the NIC's physical network ports (e.g., using an Ethernet cable—but if there are more than two smart NICs, this may require two network ports). Other embodiments use the management ports of the smart NICs, if those ports are available and if the bandwidth of the management ports is high enough to handle the intended communication between the smart NICs. In some embodiments, the smart NIC components that enable the dedicated communication channel are isolated from the other components of the smart NIC. In this case, the smart NICs are still able to at least relay traffic between the smart NICs even if the other smart NIC components are inoperable (e.g., due to firmware or software errors, hardware failures, etc.).

In other embodiments, these smart NICs are connected via separate physical switches, so that each smart NIC can communicate directly with any other smart NIC through the physical switch, rather than having the smart NICs directly connected to each other (whether serially or meshed). FIG. 5 conceptually illustrates an example of smart NICs of a host computer 500 connected via a separate physical switch 505. As shown, each of the smart NICs 510 to 520 of the host computer 500 is connected to an isolated physical switch 505. In some embodiments, this physical switch 505 only handles inter-smart NIC communications (i.e., the physical switch is not part of the data center network that handles data messages and/or management and control traffic between DCNs). In fact, this physical switch may not even use the same switching technology (e.g., Ethernet or Infiniband) used to carry network traffic within the data center. As with the previous examples, these connections may use separate specially constructed channels, physical network ports, or management ports. In addition, for redundancy, some embodiments use two (or more than two) separate isolated switches, with each smart NIC 510 to 520 connected to each of these isolated switches.

In some embodiments, the Smart NICs communicate via a logical dedicated communication channel that uses an existing physical connection, rather than using a separate physical channel for dedicated communication between the Smart NICs (e.g., if a separate purpose-built channel does not exist and the network port cannot be used for this purpose). For example, all Smart NICs of a host computer will typically be connected to the same physical data center network, so the dedicated communication channel can be overlaid on that network.

6 conceptually illustrates two host computers 605 and 610 each having three smart NICs connected to a data center network 600 and using an overlay on the data center network as their respective dedicated communication channels. As shown, the first host computer 605 includes three smart NICs 615-625 connected to the data center network 600, while the second host computer 610 also includes three smart NICs 630-640 connected to the data center network 600.

Each of these respective groups of smart NICs uses a different overlay network (e.g., using encapsulation) as a dedicated communication channel. The first group of smart NICs 615-625 uses a first overlay network 645 and the second group of smart NICs 630-640 uses a second overlay network 650. These overlay networks used as dedicated communication channels can be VXLAN networks, Geneve networks, etc. In some embodiments, the encapsulation network addresses used are those associated with the physical network ports of the smart NICs (i.e., the same network addresses used to encapsulate data traffic between DCNs on their respective host computers), while the underlying overlay network addresses are logical addresses associated with the smart NIC operating system (in fact, the first group of smart NICs 615-625 can use the same set of overlay network addresses as the second group of smart NICs 630-640).

The use of an overlay network requires only that all of the Smart NICs of a host computer be attached to the same layer 3 network (but not necessarily the same subnet). Thus, if one of the Smart NICs is connected only to a physically separate management network, but the other Smart NICs are connected to a data network within the data center (and not to the management network), then the Smart NICs cannot communicate via this overlay network. If all of the Smart NICs of a host computer are connected to the same data link layer (layer 2) network, then some other embodiments use a dedicated VLAN as a dedicated communication channel. However, if this existing physical layer 2 network has numerous other host computers (which have their own group of Smart NICs that require separate VLANs) and also carries data messages for the DCN on these host computers, then the maximum number of VLANs available on a single network (4094) may be reached.

In other embodiments, the smart NICs of a host computer communicate via a dedicated communication channel through the host computer. As described above, the smart NICs are typically connected to a PCIe subsystem of a host computer, which can be used for the dedicated communication channel. FIG. 7 conceptually illustrates a host computer 700 with three smart NICs 705-715 communicating through a PCIe fabric 720 of the host computer. Communicating through the PCIe subsystem typically allows any smart NIC to talk directly to any of the other smart NICs. In various embodiments, the smart NICs use the standard point-to-point transfer features of PCIe, utilize a PCIe switch fabric, or use other enhancements on top of PCIe (e.g., Compute Express Link (CXL)).

As mentioned, one use of the dedicated communication channel is for a first smart NIC to pass a data message (e.g., a data message sent to or from a host computer or a DCN executing on a host computer) to a second smart NIC. The smart NICs operate as a single entity because their smart NIC operating systems collectively implement a set of virtual network operations (e.g., implement logical switches and/or routers, firewalls, etc.). However, each smart NIC has its own interfaces (e.g., physical functions and virtual functions) and its own physical network ports to which the host computer's DCN is bound.

FIG. 8 conceptually illustrates a process 800 of some embodiments for processing a data message at a smart NIC, which is one of a plurality of smart NICs of a host computer. As described above, each of the smart NICs has one or more interfaces, and the DCNs operating on the host computer are each bound to a different interface. In addition, the virtual network operations performed on the data message have been pushed into the smart NIC operating system (rather than being performed by a forwarding element executing in the hypervisor of the host computer). Process 800 will be described in part with reference to FIGS. 9-11, which illustrate examples of data messages being processed by the smart NIC.

As shown, process 800 begins with receiving (at 805) a data message at a smart NIC. This data message may be received from a data center network through a physical port of the smart NIC (e.g., as in FIG. 9 ) or from a host computer (e.g., from a DCN executing on a host computer) through an interface of the smart NIC bound to a host computer or one or more DCNs on the host computer (e.g., as in FIGS. 10 and 11 ). It should be understood that the terms data message, packet, data packet, or message are used herein to refer to various formatted collections of bits that may be sent between network endpoints (e.g., between DCNs in a host and/or across a physical network), such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Although the examples herein refer to data messages, packets, data packets, or messages, it should be understood that the present invention should not be limited to any particular format or type of data message.

The process 800 then applies (at 810) network virtualization operations to the received data message based on the data message header. As described above, these operations may include logical switching (e.g., based on the logical destination MAC address of the data message), logical routing (e.g., based on the logical destination IP address of the data message), distributed firewall operations (e.g., based on the connection 5-tuple of the data message, including source and destination IP addresses, transport layer protocol, and source and destination transport layer ports), network address translation, encapsulation (if necessary), and other operations typically performed by the hypervisor of the host computer. If the smart NICs jointly implement virtual network operations, the smart NIC that first receives the data message performs this processing. When the first smart NIC receives the data message from the second smart NIC over the dedicated communication channel, the second smart NIC will typically have performed the required network virtualization operations (or most of these operations), and the first smart NIC can determine the destination of the data message with minimal additional processing.

Based on these network virtualization operations, the smart NIC is able to determine the destination of the data message. It should be understood that process 800 is a conceptual process and does not necessarily reflect specific operations performed by the smart NIC. For example, the smart NIC will typically only identify a matching record (e.g., a flow record) for the data message and perform the action specified by the matching record, rather than performing a series of determinations (i.e., those shown in operations 815, 825, and 840) as to whether the destination is of a particular type. It should also be noted that this process does not cover the full range of data message processing options. For example, in some embodiments, the smart NIC may block and/or drop the data message due to firewall rules, congestion, etc.

Process 800 determines (at 815) whether the destination of the data message is a DCN bound to the current smart NIC (i.e., the smart NIC executing process 800). This may be the case for data messages received from an external network or from other DCNs on the host computer (which may be bound to any of the smart NICs). When the destination is this DCN bound to the current smart NIC, the process is output from the smart NIC via the interface to which the destination DCN is bound (at 820). In some embodiments, the data message is then handled by the host computer (e.g., sent to the DCN via a vNIC or directly to a VF driver executing on the DCN without requiring additional network virtualization processing in the hypervisor of the host computer).

When the destination of the data message is not a DCN bound to the current smart NIC, the process 800 determines (at 825) whether the destination is a DCN bound to a different smart NIC bound to the host computer. This may be the case for data messages received from an external network or from other DCNs on the host computer that are bound to the current smart NIC. In addition, if the dedicated communication channel does not have direct communication between each pair of smart NICs, then the first smart NIC may receive a data message from the second smart NIC and need to send the data message to a third smart NIC (e.g., in the example shown in FIG. 3). When the destination is this DCN bound to another smart NIC, the process 800 sends (at 830) the data message to the other smart NIC (or an intermediate smart NIC if the NICs are connected in series) via the dedicated communication channel between the smart NICs.

FIG. 9 conceptually illustrates the path of two data messages 910 and 915 received at a first smart NIC 900 via a physical port 905 of the smart NIC. In this example, each of the smart NICs 900 and 920 of a host computer 925 each has a single physical port. At least two VMs 930 and 935 execute on the host computer 925, and for simplicity, only the VFs bound to these VMs are shown. The first smart NIC 900 provides a VF 940 to which the first VM 930 is bound (e.g., via its vNIC, not shown), while the second smart NIC 920 provides a VF 945 to which the second VM 935 is bound (also via its vNIC). Each of the smart NICs 900 and 920 performs network virtualization operations 950, and a dedicated communication channel 955 connects the two smart NICs. This dedicated communication channel may be any of the types described above (eg, a separate physical channel, a VLAN or overlay network on the physical network to which the intelligent NIC's physical ports 905 and 960 are connected, or a connection through a PCIe subsystem).

The smart NIC 900 performs a network virtualization operation 950 on each of the data messages 910 and 915. Since the destination address of the first data message 910 is the address of VM1 930 bound to the smart NIC 900, the smart NIC 900 outputs the data message 910 to the VM 930 via the VF 940. On the other hand, the network virtualization operation 950 applied to the second data message 915 recognizes that the destination address of this data message 915 is the address of VM2 935 bound to the second smart NIC 920. As such, the first smart NIC 900 delivers this data message 915 to the second smart NIC 920 via the dedicated communication channel 955. In some embodiments, the first smart NIC 900 also provides the second smart NIC 920 with context information about the processing of the data message by the network virtualization operation 950 so that this processing does not need to be fully repeated at the second smart NIC 920. In some embodiments, the second smart NIC 920 applies network virtualization operations 950 to evaluate this context and determine that the data message 915 should be sent to VM2 935. As such, the smart NIC 920 outputs the data message 915 to VM 935 via the VF 945.

Returning to FIG. 8 , if the destination of the data message is not a DCN on the host computer, then (assuming that the data message will not be dropped or blocked) the destination is external to the host computer. Thus, process 800 identifies (at 835) the physical network output port for the data message. In some cases, all ports of all intelligent NICs are grouped in a link aggregation group (LAG) or other grouping mechanism. In this case, the connection for a single DCN bound to a specific intelligent NIC is load balanced across all physical output ports of all intelligent NICs, and is not only output by the intelligent NIC receiving the data message. In other cases, different intelligent NIC ports may have different connectivity, so that data messages for a specific destination need to be output from one intelligent NIC, and data messages for other destinations need to be output from another intelligent NIC (regardless of any load balancing operation).

As such, process 800 determines (at 840) whether the identified physical network output port is on another smart NIC or on the current smart NIC. If the output port for the data message is a port of another smart NIC, process 800 sends (at 830) the data message to the other smart NIC (or an intermediate smart NIC if the NICs are connected in series) via a dedicated communication channel between smart NICs. On the other hand, if the identified output port is a port of the current smart NIC, process 800 outputs (at 845) the data message to the physical network via the identified output port. After outputting the data message to the DCN (via the interface of the current smart NIC), the physical network, or another smart NIC via the dedicated communication channel, process 800 ends.

FIG. 10 conceptually illustrates the path of two data messages 1005 and 1010 received from a first VM 930 via a VF 940 to which the first VM is bound at a first smart NIC 900. As shown, the first data message 1005 is directed to a first destination (Dest1), while the second data message 1010 is directed to a second destination (Dest2). The smart NIC 900 processes the two data messages according to the configured network virtualization operation 950, which in this case (i) determines that both data messages should be output to the physical network and (ii) includes load balancing across multiple output ports (e.g., in a LAG). In some embodiments, only the first data message in a connection has the load balancing operation applied, and for subsequent operations, the cached results direct the data messages to the same physical output port.

Based on these operations, the smart NIC 900 outputs the first data message 1005 to the physical network via its own physical port 905. However, the second data message 1010 is sent to the second smart NIC 920 via the dedicated communication channel 955. In some embodiments, the first smart NIC 900 also provides context information indicating that a network virtualization operation has been performed on the data message 1010 and that the data message should be output via the physical port 960 of the second smart NIC 920. The second smart NIC 920 receives the second data message 1010 via the dedicated communication channel 955 and outputs the data message 1010 to the physical network via its physical port 960.

As described above with reference to Figure 8, a dedicated communication channel is also used when a DCN bound to an interface of one smart NIC sends a data message to a DCN bound to an interface of another smart NIC. The dedicated communication channel allows data messages to be sent directly between smart NICs, rather than the first smart NIC outputting the data message onto the physical network to be switched and/or routed back to the host computer via the second smart NIC.

11 conceptually illustrates the path of a data message 1100 sent from a first VM 930 operating on a host computer 925 to a second VM 935. Here, the first smart NIC 900 receives the data message 1100 via the VF 940 to which the source VM 930 is bound. The smart NIC 900 applies a network virtualization operation 950 to the data message 1100 to determine that the destination of the data message is a VM bound to the second smart NIC 920. Based on this determination, the smart NIC 900 sends the data message 1100 to the second smart NIC 920 via a dedicated communication channel 955 between smart NICs. As in other examples, in some embodiments, the first smart NIC 900 also provides context information indicating that the network virtualization operation has been performed on the data message 1100 and that the data message is directed to a DCN bound to the smart NIC 920 (although it is not necessary to indicate which interface the DCN is bound to). The second smart NIC 920 receives the data message 1100 via the dedicated communication channel 955 and outputs the data message to the VM 935 via the VF interface 945 .

The process 800 described above and the examples shown in Figures 9 to 11 involve unicast data messages (i.e., data messages with a single destination). In some embodiments, a single data message (e.g., a broadcast or multicast data message) can be sent along multiple paths through network virtualization operations performed within one intelligent NIC.

12 conceptually illustrates the path of a multicast data message 1200 received at a first smart NIC 900 via a physical port 905 of the smart NIC. In this example, the first smart NIC 900 applies network virtualization operations 950 to the multicast data message 1200 and determines that both the first VM 930 and the second VM 935 are in the multicast group to which the data message 1200 is sent. Based on this, the smart NIC 900 (i) outputs a first copy of the multicast data message 1200 to the first VM 930 via a VF 940, and (ii) delivers a second copy of the data message 1200 to the second smart NIC 920 via a dedicated communication channel 955. In some embodiments, the first smart NIC 900 also provides context information to the second smart NIC 920 regarding the processing of the data message by the network virtualization operations 950 so that such processing does not need to be fully repeated at the second smart NIC 920. In some embodiments, the second smart NIC 920 applies network virtualization operations 950 to evaluate this context and determines that the multicast data message 1200 should be sent to the second VM 935. As such, the smart NIC 920 outputs the data message 1200 to the VM 935 via the VF 945. In some embodiments, if multiple destinations for the multicast data message are bound to the second smart NIC 920, only one copy of the data message is delivered via the communication channel 955, allowing the second smart NIC 920 to generate and output the necessary copies of the data message. Similarly, if one of the VMs attached to the first smart NIC sends a broadcast or multicast data message, the receiving smart NIC may process the data message and generate any copies needed to send the data message to other VMs attached to the first smart NIC, output the data message via its physical ports, and/or pass the data message to other smart NICs (sending the data message to the VMs bound to those smart NICs, outputting the data message via their physical ports, or a combination thereof).

Another situation where a dedicated communication channel may need to be used to pass data messages between smart NICs occurs if all of the physical network ports of a smart NIC are inoperable, but the smart NIC itself is still operational. In this case, the smart NIC can still perform virtual network operations on data messages sent from the DCN bound to the smart NIC, but will need to send those data messages to other smart NICs for output to the physical network, regardless of whether the ports are operating in a LAG. When the ports are operating in a LAG or the smart NICs are configured in a NIC team using another teaming mechanism, the connection previously assigned to the inoperable physical port will be moved to another physical port (e.g., on another smart NIC).

FIG. 13 conceptually illustrates the path of a data message for one of the connections shown in FIG. 10 after a physical port 905 of the first smart NIC 900 has failed. This may occur due to a problem with the smart NIC itself, a physical cable connecting port 905 to the data center network being disconnected, etc. As shown, another data message 1300 directed to Dest1 is sent from VM 930 to smart NIC 900 via VF 940. Smart NIC 900 processes the data message according to the configured network virtualization operation 950, which determines that the data message should be output to the physical network, but the physical port 905 previously used for this connection is no longer available. As such, the connection is now rebalanced to use another physical port 960 of the second smart NIC 920. Therefore, data message 1300 is sent to the second smart NIC 920 via dedicated communication channel 955. In some embodiments, the first smart NIC 900 also provides context information indicating that a network virtualization operation has been performed on the data message 1300 and that the data message should be output via the physical port 960 of the second smart NIC 920. The second smart NIC 920 receives the second data message 1300 via the dedicated communication channel 955 and outputs the data message 1300 to the physical network via its physical port 960.

In many cases, the smart NIC receives configuration data for virtual network operations from a network management and control system. In some embodiments, such a network management and control system receives data defining network operations (e.g., defining logical networks), security operations, etc. from a user (e.g., a network and/or security administrator), and uses this definition data to generate configuration data for various network elements (e.g., forwarding elements such as virtual switches and routers, middlebox elements such as distributed firewalls, etc.), and provides the configuration data to the network elements so that the network elements can implement various network and security operations. Such network elements include smart NICs that perform network virtualization operations.

Each of the ports of the different smart NICs, possibly including the management port, has its own network address, but many network management and control systems treat each host computer as a single entity. For example, for host computers that do not use smart NICs for network virtualization operations, the network management and control system of some embodiments communicates with an agent in the hypervisor of the host computer. The network management and control system uses a single management network address for each host computer and therefore should not communicate directly with all of the multiple smart NICs of the host computer.

In some embodiments, the Smart NICs use clustering techniques in order to appear as a single entity to the host computer to the network management and control system. For example, in some embodiments, the Smart NICs of the host computer perform leader election to determine a single one of the Smart NICs to communicate with the network management and control system. In some such embodiments, each of the Smart NIC operating systems runs a deterministic algorithm that selects one of the Smart NICs as the point of contact. Any messages required for this leader election are communicated via a dedicated communication channel.

FIG. 14 conceptually illustrates a process 1400 of some embodiments for configuring multiple smart NICs to perform network virtualization operations. In some embodiments, process 1400 is independently performed by each of a group of smart NICs of a host computer when the smart NICs are online (e.g., when the host computer boots up). Process 1400 may also be performed (again, independently performed by each smart NIC of the host computer) in response to a change in smart NIC team membership (e.g., a smart NIC is added to a NIC team or a smart NIC is removed from a NIC team, whether due to an external action or a NIC failure). In some embodiments, the smart NICs monitor team membership using beacons or keep-alive messages (e.g., sent over a dedicated communication channel). Process 1400 is described in part with reference to FIGS. 15 and 16, which illustrate the operation of a pair of smart NICs 1500 and 1505 of a host computer (not shown).

15 illustrates that each of the smart NICs 1500 and 1505 executes a smart NIC operating system 1510 and 1515, respectively. The smart NIC operating system includes multiple modules, such as network virtualization operations 1520 and 1525, control agents 1530 and 1535, and leader election modules 1540 and 1545. A dedicated communication channel 1550 connects the two smart NICs and allows communication between the smart NICs (e.g., for sending data messages, configuration data, etc.).

In some embodiments, the control agents 1530 and 1535 communicate with a network management and control system that configures network virtualization operations on numerous host computers in a data center (e.g., by setting up those host computers to perform switching and/or routing to implement a logical network). The control agents 1530 and 1535 receive configuration data from this network management and control system and use the configuration data to appropriately configure their respective network virtualization operations 1520 and 1525. The control agents 1530 and 1535 are able to communicate with each other via a dedicated communication channel 1550.

Leader election modules 1540 and 1545 perform leader election to designate one of the smart NICs as the leader for a particular task (e.g., communicating with a network management and control system). Leader election modules 1540 and 1545 may communicate via a dedicated communication channel 1550 to confirm leader election for a task, share identification information so that each leader election module is aware of all smart NICs in the host computer that may be selected as task leaders, etc.

As shown, process 1400 begins with using (at 1405) a leader election algorithm to determine which smart NIC is the single point of communication for the network management and control system. In some embodiments, this leader election algorithm is a deterministic algorithm that is executed separately on each individual smart NIC of a group of smart NICs of a host computer. That is, if there are five smart NICs, each of the five smart NICs runs the leader election algorithm to arrive at the same elected leader. An example of this algorithm is a hash-based decision that hashes the identifiers of the five smart NICs and calculates the resulting hash modulo 5 (the number of smart NICs) to determine the leader. In other embodiments, the leader election algorithm involves communication and/or negotiation between smart NICs to arrive at a elected leader smart NIC that is designated as communicating with the network management and control system.

Once this election has been completed, process 1400 determines (at 1410) whether the current smart NIC (i.e., the smart NIC executing this process) is elected as the contact point. It should be understood that process 1400 is a conceptual process and that each smart NIC does not necessarily make this particular determination. Rather, the smart NIC elected as the leader performs a first set of operations, while the other smart NICs perform a different set of operations after the leader election. In the example of FIG. 16 (which illustrates the distribution of configuration data to multiple smart NICs), leader election module 1540 is bolded to indicate that smart NIC 1500 has been elected as the leader to communicate with network management and control system 1600.

For the Smart NIC that is not the selected point of contact with the network management and control system, the process 1400 eventually receives (at 1415) configuration data from the selected Smart NIC via the dedicated communication channel. It should be noted that this will not occur before the selected Smart NIC receives this configuration data from the network management and control system and disseminates the data to other Smart NICs.

At the smart NIC selected as the point of contact with the network management and control system, the process establishes (at 1420) communications with the network management and control system using the management IP address assigned for the host computer. In some embodiments, the network management and control system treats each host computer as a single entity, which may not involve internal network implementations on each host computer. To establish communications, in some embodiments, the selected smart NIC sends a message or a set of messages from the management IP address to the network management and control system. In some embodiments, the network management and control system will automatically use the assigned IP address, but the selected smart NIC needs to notify the data center network that messages sent to the IP address should be directed to a specific one of its ports that uses the IP address.

Once communication is established, the process receives (at 1425) configuration data from the network management and control system. In some embodiments, this configuration data specifies how the smart NIC should handle data messages. The configuration data may include routing tables, virtual switch configurations, firewall rules, network address translation rules, load balancing rules, etc. In some embodiments, the configuration data is in a specific format for a specific type of network virtualization software running on the smart NIC operating system. In other embodiments, the configuration data is in a generic format and the controller agent on each smart NIC is responsible for converting the data to a specific format for the network virtualization software. FIG. 16 illustrates that the network management and control system 1600 provides configuration data 1605 to the control agent 1530 of the smart NIC 1500, which has been selected as the contact point for the network management and control system 1600.

Next, the process shares (at 1430) the received configuration data with other intelligent NICs (i.e., those intelligent NICs that are not in direct communication with the network management and control system). This data is provided to the other intelligent NICs via a dedicated communication channel between the intelligent NICs. Also at this point, the other intelligent NICs reach operation 1415 in their own processes because they are now able to receive configuration data.

Process 1400 (whether executed on the selected smart NIC or on one of the other smart NICs) then configures (at 1435) network virtualization operations on the smart NIC based on the configuration data. As mentioned, in some embodiments, the control agent uses the configuration data received from the network management and control system (e.g., as a first set of data tuples) to generate configuration data for network virtualization operations (e.g., as a second set of data tuples). In some embodiments, the network virtualization operations and/or control agent in the smart NIC operating system also programs the data message processing ASIC of the smart NIC based on this configuration data. Process 1400 then ends, but in practice, the selected smart NIC will periodically receive updates from the network management and control system as configuration changes are provided to the system.

16 shows that the control agent 1530 on the selected smart NIC 1500 provides configuration data 1605 to the control agent 1535 on the second smart NIC 1505 (e.g., via a dedicated communication channel 1550). The control agent 1530 also uses this configuration data 1605 to configure the network virtualization operation 1520 on its smart NIC 1500, while the control agent 1535 on the second smart NIC 1505 uses the configuration data 1605 to configure its corresponding network virtualization operation 1525.

In addition to propagating configuration data from the network management and control system, in some embodiments, the leader smart NIC also receives information from other smart NICs via a dedicated communication channel. In some embodiments, this information includes statistics (e.g., data message processing statistics), status/monitoring information, and other data. In some embodiments, the selected leader smart NIC performs various monitoring tasks based on this information (e.g., ensuring that the various smart NICs are currently operational and sending messages to other smart NICs if one of the smart NICs fails).

In some embodiments, some of the shared information is reported to the network management and control system. FIG. 17 conceptually illustrates a selected leader Smart NIC 1500 collecting statistics from itself and another Smart NIC 1505 and reporting the statistics to the network management and control system 1600. As shown, control agents 1530 and 1535 collect statistics from their respective sets of network virtualization operations 1520 and 1525. The control agent 1535 provides these statistics to the control agent 1530 at the leader Smart NIC 1500 via a dedicated communication channel 1550. At least some of these statistics from both Smart NICs are sent from the control agent 1530 to the network management and control system 1600. In some embodiments, the control agent 1530 or another module on the selected leader Smart NIC 1500 aggregates the statistics so that the network management and control system 1600 is provided with information that appears to be from a single entity.

This collected information can be used by the network management and control system 1600 to monitor the host computer and/or individual Smart-NICs. The network management and control system can also use this information to modify the virtual network configuration of the Smart-NICs, in which case the network management and control system provides configuration updates to the leader Smart-NIC, which in turn spreads these updates to the other Smart-NICs via a dedicated communication channel.

In some embodiments, the network management and control system includes multiple components that perform different functions and provide different configuration data to host computers (in addition to receiving different data from host computers). For example, the network management and control system of some embodiments includes both a management plane (MP) and a central control plane (CCP). The MP receives configuration data from administrators, saves this data, and provides specific configuration information to host computers. In addition, in some embodiments, the host computers provide statistics, status, and other real-time data to the MP. In some embodiments, the CCP receives network configuration data from the MP, determines the host computers (and other forwarding elements, such as gateways) that need each portion of the network configuration data, and provides this data to agents on these host computers.

In some embodiments, the Smart NIC elects multiple different leaders for multiple different tasks. For example, some embodiments elect one leader for receiving configuration data, another leader for collecting traffic statistics, a third leader for collecting monitoring data, etc. In some embodiments, one leader is elected for communicating with the MP and a second leader is elected for communicating with the CCP. These leader elections may use different hash functions or different inputs to the same hash function in order to derive different Smart NICs as the elected leaders. In some embodiments, if a Smart NIC is elected for communicating with the MP, then the Smart NIC is not considered for communicating with the CCP in order to ensure that the load is shared.

18 conceptually illustrates three smart NICs 1805-1815 of a host computer (not shown) that respectively operate smart NIC operating systems 1820-1830. As in the previous figures, each of the smart NIC operating systems 1820-1830 includes a respective control agent 1835-1845 and a leader election module 1850-1860. In addition, each of the smart NICs 1805-1815 is connected to the other smart NICs via a dedicated communication channel 1865.

Additionally, the network management and control system 1800, which includes both the MP 1870 and the CCP 1875, communicates with the smart NICs 1805-1815. Here, the leader election modules 1850-1860 have designated the first smart NIC 1805 as the point of contact for the MP 1870, and have designated the third smart NIC 1815 as the point of contact for the CCP 1875. As such, the control agent 1835 on the first smart NIC 1805 communicates with the MP 1870, and the control agent 1840 on the third smart NIC 1815 communicates with the CCP 1875. In some embodiments, each of the smart NIC operating systems actually runs a separate MP agent and CP agent, with the selected MP agent communicating with the MP 1870 and the selected CP agent communicating with the CCP 1875.

For various purposes, in some embodiments, the smart NICs also use a dedicated communication channel to synchronize dynamic state information. That is, when a first smart NIC receives or creates a set of dynamic state information, the first smart NIC uses a dedicated communication channel to provide the same set of dynamic state information to one or more of the other smart NICs. Different types of states may be shared with a single other smart NIC or multiple (or all) other smart NICs of a given host computer. If one of the smart NICs fails, synchronization of the dynamic state information allows the information to be preserved, rather than lost. A smart NIC may fail due to an electrical short, a disconnect, overheating, etc.

As mentioned, a selected leader smart NIC among a group of smart NICs of a host computer can collect monitoring data from all other smart NICs. This collected data or data generated from the collected data can include dynamic status information synchronized with at least one backup smart NIC. Thus, if the leader smart NIC fails, the next leader can retrieve the monitoring status information.

In addition, when performing virtual network processing, the smart NICs may need to store dynamic state information and share that data with each other. FIG. 19 conceptually illustrates two smart NICs 1905 and 1910 that share connection state. As in the previous figures, each of the smart NICs (e.g., within a smart NIC operating system) performs network virtualization operations 1915 and 1920. These operations include switching and routing 1925 and 1930 and firewall engines 1935 and 1940 that perform firewall operations (i.e., determine whether to allow, block, or drop data messages based on their headers). In some embodiments, the firewall operations are stateful and therefore use information from respective connection trackers 1945 and 1950.

The connection trackers 1945 and 1950 store information about open connections handled by the smart NICs. As shown, some embodiments store at least a 5-tuple (source and destination IP addresses, source and destination transport layer ports, transport layer protocol), the current state of the connection, and the congestion window of the connection for each open connection. This connection information is a dynamic state that the connection trackers 1945 and 1950 synchronize via a dedicated communication channel 1955 between the smart NICs.

As shown, the connection tracker 1945 on the first smart NIC 1905 stores information for two open connections (cxn1 and cxn2), as well as the congestion windows for these open connections. Other embodiments may also store additional data (e.g., a receiver window). The firewall engines 1935 and 1940 use this dynamic connection state information from their respective connection trackers to process data messages sent to and from the DCN on their host computers. Information about whether a particular connection is open (e.g., whether a three-way handshake has been completed) allows the firewall engines 1935 and 1940 to determine whether the data message should be allowed. The congestion window is a dynamic state variable determined by the connection endpoints (and learned by the smart NICs) that limits the amount of data that can be sent to the network (i.e., from the physical port of one of the smart NICs) for a particular connection, and typically starts small and increases to a maximum value (which may be set by the receiver window).

If the connection state of an ongoing connection is lost (e.g., due to a failure of the smart NIC to store the connection state in its connection tracker), then, depending on the firewall engine settings, all traffic for the connection will be blocked by the firewall engine of the smart NIC that picked up the connection, or the firewall engine on the smart NIC will need to relearn the connection state from the endpoint. In the first option, not only will the connection need to be reestablished, but the congestion window will start small again, limiting the amount of data that can be transmitted. The latter option avoids disconnection, but at the expense of a lax window for security enforcement.

Thus, connection trackers 1945 and 1950 share their dynamic state information with each other to avoid the need for either of these options. At this point, the state information for cxn1 and cxn2 has been shared; these connections can be handled by either of smart NICs 1905 and 1910. At this point, VM 1900 is in the process of opening a new connection (cxn3) and sending a data message 1960 for this connection to network virtualization operation 1915 on the first smart NIC 1905 (i.e., the smart NIC to which VM 1900 is bound). Therefore, connection tracker 1945 also synchronizes this connection state data 1965 to connection tracker 1950. In some embodiments, each smart NIC synchronizes its connection state data (or other state data) to only one other smart NIC, while in other embodiments, each smart NIC synchronizes its connection state data (or other state data) to all other smart NICs.

Different embodiments synchronize dynamic state information at different intervals. Some embodiments synchronize each change through a dedicated communication channel, while other embodiments synchronize state data at regular time intervals (e.g., every 1 ms, every 100 ms, every second, every 5 seconds, etc.). If the dedicated communication channel is a specially constructed channel, this can achieve very fast synchronization (e.g., every 1 ms or so). In addition, some embodiments use mechanisms in the smart NIC to write the connection state (or other synchronized data) to a specific storage area in the smart NIC, where this write is automatically mirrored to a peer storage area on another smart NIC, thereby achieving faster synchronization (e.g., less than 10 μs of latency). If the synchronization interval is longer (higher latency) so that the congestion window cannot be accurately synchronized, some embodiments only synchronize basic connection status (i.e., whether the connection is open and allowed). In the event that the first smart NIC handling a particular connection fails, the new smart NIC that begins handling the connection allows traffic for the connection until the new smart NIC has learned the congestion window for the connection.

When VM 1900 is bound to first Smart NIC 1905 (and assuming this connection is sent to and from the physical port of this first Smart NIC 1905), second Smart NIC 1910 does not actually have any use for this information. However, FIG. 20 illustrates that while these connections remain open, first Smart NIC 1905 has become inoperable, and therefore VM 1900 is now bound to the interface of second Smart NIC 1910. This does not mean that the VM needs to restart all of its connections, as this information has been synchronized from first Smart NIC 1905. Depending on the configuration, if there are more than two Smart NICs, then in different embodiments, all VMs bound to the now inoperable Smart NIC are either transferred to the same Smart NIC or balanced across all remaining Smart NICs.

As shown, VM 1900 continues to send data messages 2000 for cxn3 (now to the second smart NIC 1910). Since the current state of this connection is that it is now open with a congestion window of 3 (before the first smart NIC 1905 failed), the firewall engine 1940 is able to process these data messages without requiring the connection or its congestion window to be restarted.

This state sharing can also be used by smart NICs that are performing operations other than virtual networking (or smart NICs that are performing multiple types of operations that use state sharing). If storage virtualization operations are handled by the smart NICs, then in some embodiments, the storage virtualization functions include running a network stack to manage transport layer (e.g., TCP) connections to the storage devices. In this case, if a failover occurs, the connection information should be shared again between the smart NICs so that if one of the smart NICs fails, the connections are not reset.

FIG. 21 conceptually illustrates an electronic system 2100 utilized to implement some embodiments of the present invention. The electronic system 2100 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, a server computer, a mainframe, a blade computer, etc.), a phone, a PDA, or any other kind of electronic device. This electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 2100 includes a bus 2105, a processing unit 2110, a system memory 2125, a read-only memory 2130, a permanent storage device 2135, an input device 2140, and an output device 2145.

The bus 2105 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system 2100. For example, the bus 2105 communicatively connects the processing unit 2110 with the read-only memory 2130, the system memory 2125, and the permanent storage device 2135.

From these various memory units, processing unit 2110 retrieves instructions to execute and data to process in order to perform the processes of the present invention. In various embodiments, the processing unit may be a single processor or a multi-core processor.

The read-only memory (ROM) 2130 stores static data and instructions required by the processing unit 2110 and other modules of the electronic system. On the other hand, the permanent storage device 2135 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 2100 is turned off. Some embodiments of the present invention use a large-capacity storage device (such as a magnetic disk or optical disk and its corresponding magnetic disk drive) as the permanent storage device 2135.

Other embodiments use removable storage devices (e.g., floppy disks, flash drives, etc.) as permanent storage devices. Like permanent storage device 2135, system memory 2125 is a read-write memory device. However, unlike storage device 2135, system memory is a volatile read-write memory, such as random access memory. System memory stores some instructions and data that the processor needs at run time. In some embodiments, the processes of the present invention are stored in system memory 2125, permanent storage device 2135, and/or read-only memory 2130. From these various memory units, processing unit 2110 retrieves instructions to be executed and data to be processed in order to perform the processes of some embodiments.

The bus 2105 is also connected to input devices 2140 and output devices 2145. The input devices enable a user to communicate information and select commands to the electronic system. The input devices 2140 include an alphanumeric keyboard and a pointing device (also called a "cursor control device"). The output devices 2145 display images generated by the electronic system. Output devices include printers and display devices, such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs). Some embodiments include devices that function as both input devices and output devices, such as touch screens.

Finally, as shown in FIG21, bus 2105 also connects electronic system 2100 to a network 2165 through a network adapter (not shown). In this way, the computer can be part of a computer network, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all components of electronic system 2100 may be used in conjunction with the present invention.

Some embodiments include electronic components, such as microprocessors, storage devices, and memories, that store computer program instructions in machine-readable or computer-readable media (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), read-only digital versatile disks (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and/or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as that produced by a compiler, and files containing higher-level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

Although the above discussion mainly refers to a microprocessor or multi-core processor that executes software, some embodiments are performed by one or more integrated circuits such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

As used in this specification, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the term "display" or "displaying" means displaying on an electronic device. As used in this specification, the terms "computer-readable medium (medium)", "computer-readable medium (media)" and "machine-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient signals.

Reference is made throughout this specification to computing and network environments that include virtual machines (VMs). However, a virtual machine is only one instance of a data computing node (DCN) or data computing end node (also referred to as an addressable node). A DCN may include non-virtualized physical hosts, virtual machines, containers that run on top of a host operating system (without a hypervisor or separate operating system), and a hypervisor kernel network interface module.

In some embodiments, the VM uses the resources of the host virtualized by the virtualization software (e.g., a hypervisor, a virtual machine monitor, etc.) to operate on the host using its own guest operating system. The tenant (i.e., the owner of the VM) can choose which applications to operate on the guest operating system. On the other hand, some containers are structures that run on top of the host operating system without the need for a hypervisor or a separate guest operating system. In some embodiments, the host operating system uses namespaces to isolate containers from each other and thus provide operating system-level separation of different application groups operating in different containers. This separation is similar to the VM separation provided in the hypervisor virtualization environment of the virtualized system hardware, and can therefore be regarded as a form of virtualization that isolates different application groups operating in different containers. Such containers are more lightweight than VMs.

In some embodiments, the hypervisor kernel network interface module is a non-VM DCN that includes a network stack with a hypervisor kernel network interface and receive/transmit threads. An example of a hypervisor kernel network interface module is the vmknic module, which is part of VMware, Inc.'s ESXi™ hypervisor.

It should be understood that although this specification refers to VMs, the examples given may be any type of DCN, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, in some embodiments, the example network may include a combination of different types of DCNs.

Although the present invention has been described with reference to numerous specific details, those skilled in the art will recognize that the present invention may be embodied in other specific forms without departing from the spirit of the present invention. In addition, several figures (including Figures 8 and 14) conceptually illustrate the process. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a series of continuous operations, and different specific operations may be performed in different embodiments. In addition, the process may be implemented using several sub-processes, or implemented as part of a larger macro process. Therefore, those skilled in the art will understand that the present invention is not limited by the aforementioned illustrative details, but will be defined by the appended claims.

Claims (42)

1. A method comprising: At a first intelligent network interface controller (NIC) of a host computer, each of the intelligent NICs executes an intelligent NIC operating system that performs virtual network operations for a set of data computing machines executing on the host computer: receiving a data message sent by one of the data computing machines executing on the host computer; performing a virtual network operation on the data message to determine that the data message is to be transmitted from a port of a second smart NIC of the plurality of smart NICs; and The data message is communicated to the second intelligent NIC via a dedicated communication channel connecting the plurality of intelligent NICs. 2. The method of claim 1 , wherein performing a virtual network operation on the data message to determine that the data message is to be transmitted from a port of the second intelligent NIC comprises: determining, based on the virtual network operation, a destination for the data message, the destination being external to the host computer and reachable through a first physical port of the first smart NIC; and It is determined that the first physical port of the first smart NIC is currently inoperable and that the destination is reachable through a second physical port of the second smart NIC. 3. The method of claim 2, wherein all physical ports of the first smart NIC that connect the host computer to a physical network are inoperable. 4. The method of claim 2, wherein the data computing machine that sends the data message is bound to a port of the first smart NIC. 5. The method of claim 1 , wherein performing a virtual network operation on the data message comprises: determining a destination of the data message external to the host computer based on the virtual network operation; and Load balancing is performed to assign the data messages to the ports of the second intelligent NIC. 6. The method according to claim 5, wherein: A set of physical ports of the first intelligent NIC and a set of physical ports of the second intelligent NIC are in the same link aggregation group, and data messages are load balanced across the link aggregation group; and The data computing machine that sends the data message is bound to an interface of the first intelligent NIC. 7. The method according to claim 1, wherein: The data computing machine is a first data computing machine bound to the first smart NIC; and Performing a virtual network operation on the data message includes determining, based on the virtual network operation, that a destination for the data message is a second data computing machine executing on the host computer and bound to the second smart NIC. 8. The method of claim 1, wherein the virtual network operations include logical switching operations. 9. The method of claim 8, wherein the virtual network operations further include logical routing operations. 10. The method of claim 1, wherein the data message is a first data message, the method further comprising: receiving a second data message from the second smart NIC via the dedicated communication channel, wherein the second smart NIC performs a virtual network operation on the second data message to determine that the second data message is destined for the data computing machine; and The second data message is sent to the data computing machine through the port of the first intelligent NIC to which the data computing machine is bound. 11. The method according to claim 10, wherein the data message is a first data message, the method further comprising: receiving a second data message sent by the data computing machine; performing a virtual network operation on the data message to determine that the data message will be transmitted from a physical port of the first smart NIC; and The data message is transmitted from the physical port. 12. A method for synchronizing states between a plurality of intelligent network interface controllers (NICs) of a host computer that use dynamic state information to perform operations, the method comprising: storing, at a first smart NIC of the plurality of smart NICs, a set of dynamic state information; and The set of dynamic state information is synchronized across a communication channel connecting the plurality of intelligent NICs such that each of the intelligent NICs in the plurality of intelligent NICs also stores the set of dynamic state information. The method of claim 12 , wherein the dynamic state information comprises connection tracking data. 14. The method of claim 13, wherein each of the smart NICs executes a smart NIC operating system that uses the connection tracking data to perform virtual network operations on data messages sent to and from a set of data computing machines executing on the host computer. 15. The method of claim 14, wherein the virtual network operation comprises a firewall operation that determines whether to allow the data message. 16. The method of claim 13, wherein the connection tracking data comprises a list of active transport layer connections. 17. The method of claim 16, wherein the connection tracking data further comprises a congestion window for each of at least a subset of the active transport layer connections. 18. The method of claim 12, wherein the dynamic state information comprises tracked storage connection information used to perform virtualized storage operations. 19. The method of claim 12, wherein each of the plurality of intelligent NICs uses a subset of the dynamic state information to perform operations, the method further comprising: Upon failure of the second smart NIC, operations previously performed by the second smart NIC are performed using dynamic state information received from the second smart NIC in a synchronization operation. 20. The method of claim 12, wherein the communication channel is a dedicated communication channel with high bandwidth and low latency. 21. The method of claim 12, wherein the dedicated communication channel is a set of physical cables that carry only communications between the smart NICs. 22. The method of claim 21, wherein for each corresponding pair of smart NICs in the plurality of smart NICs, the set of physical cables includes a corresponding physical cable connecting the pair of smart NICs. 23. The method of claim 21, wherein each of the plurality of intelligent NICs is directly connected to two other intelligent NICs via the set of physical cables. 24. The method of claim 21, wherein the set of physical cables is connected to a set of one or more physical switches to carry communications between the smart NICs. 25. The method of claim 12, wherein: The smart NIC is connected to a physical network via a physical port to transmit and receive data to and from other host computers; and The dedicated communication channel is a logical dedicated communication channel utilizing the physical network. 26. The method of claim 25, wherein the logical dedicated communication channel is an overlay network. 27. The method of claim 12, wherein the dedicated communication channel uses a Peripheral Component Interconnect Express (PCIe) subsystem of the host computer. 28. A method comprising: At a first intelligent network interface controller (NIC) of a host computer, each of the intelligent NICs is configured to perform virtual network operations for a set of data computing machines executing on the host computer: Determining to select the first intelligent NIC to communicate with a network management and control system that configures the virtual network operation; receiving a set of configuration data for operation of the virtual network from the network management and control system; and The received set of configuration data is provided to other intelligent NICs of the host computer. 29. The method of claim 28, wherein determining to select the first intelligent NIC to communicate with the network management and control system comprises executing a deterministic selection algorithm to determine to select the first intelligent NIC. 30. The method of claim 29, wherein each of the other smart NICs in the plurality of smart NICs executes a same deterministic algorithm to determine selection of the first smart NIC. 31. The method of claim 28, wherein determining to select the first smart NIC to communicate with the network management and control system comprises exchanging messages with the other smart NICs of the plurality of smart NICs via a dedicated communication channel connecting the smart NICs. 32. The method of claim 28, wherein: The network management and control system includes a management plane and a control plane; Determining to select the first intelligent NIC to communicate with the network management and control system includes: determining to select the first intelligent NIC to communicate with the management plane; and The set of configuration data is received from the management plane. 33. The method of claim 32, further comprising determining to select a second smart NIC of the plurality of smart NICs to communicate with the control plane. 34. The method of claim 33, wherein: The set of configuration data is a first set of configuration data; and The second intelligent NIC receives a second set of configuration data from the control plane and provides the second set of configuration data to the first intelligent NIC and the other intelligent NICs of the host computer. 35. The method of claim 28, wherein the received set of configuration data is provided to the other intelligent NICs via a dedicated communication channel connecting the plurality of intelligent NICs. 36. The method of claim 28, further comprising monitoring the other intelligent NIC to determine whether the other intelligent NIC is operational. 37. The method of claim 28, wherein upon determining that the first intelligent NIC is selected to communicate with a network management and control system, a network address used by the network management and control system to communicate with the host computer is assigned to an interface of the first intelligent NIC. 38. The method of claim 28, further comprising collecting runtime statistics from the other intelligent NICs. 39. A machine-readable medium storing a program which, when executed by at least one processing unit, implements the method according to any one of claims 1 to 38. 40. An electronic device comprising: a set of processing units; and A machine-readable medium storing a program which, when executed by at least one of the processing units, implements the method according to any one of claims 1 to 38. 41. A system comprising means for implementing the method according to any one of claims 1 to 38. 42. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 38.