JP2005506726A - Virtual network system and method in processing system - Google Patents

Abstract

Systems and methods for virtual networking (100). Switched Ethernet local area network semantics are provided across the underlying point-to-point network. Computer processor nodes may communicate directly across the switch structure via a virtual interface, or they may communicate via an Ethernet switch emulator (115a, 115b). ARP (Address Resolution Protocol) logic (135) is used to associate a virtual interface with an IP address while allowing a computer processor to answer an ARP request with a virtual MAC address.
[Selection] Figure 1

Description

【Technical field】
[0001]
The present invention relates to a computer system for enterprises and application service providers, and more particularly to a processing system having a virtual communication network.
[Background]
[0002]
In today's enterprise computing and application service provider environments, employees from multiple information technology (IT) functions (electrical, networking, etc.) must participate to deploy processing and network resources. Thus, it may take weeks or months to deploy a new computer server due to scheduling and other obstacles in coordinating activities from multiple parts. This long and manual process increases both human and equipment costs and delays the start of the application.
[0003]
In addition, it is difficult to predict how much processing power an application will require, so managers tend to oversupply the amount of computing power. As a result, the computing resources of the data center are often not used or have a low utilization rate.
[0004]
If more processing power is ultimately needed than the original supply, various IT functions can deploy more or improved servers and connect them to a network of communications and storage devices. Such activities will need to be adjusted again. As the system grows, this task becomes increasingly difficult.
[0005]
Deployment is also a problem. For example, if you deploy 24 conventional servers, you may need more than 100 individual connections to form the entire system. Management of these cables is an ongoing challenge and each symbolizes a point of failure. Attempting to reduce the risk of failure by adding redundancy can double the wiring and exacerbate the problem, while increasing complexity and cost.
[0006]
Providing high usability with today's technology is a difficult and expensive proposition. In general, a failover server must be deployed for every primary server. In addition, complex management software and professional services are typically required.
[0007]
In general, it is not possible to adjust processing power or upgrade the CPU on an existing server. Instead, increasing processor capacity and / or moving to vendor next-generation architectures often means adding more hardware / software systems, necessitating new connections, etc. Request a “forklift upgrade”.
DISCLOSURE OF THE INVENTION
[Problems to be solved by the invention]
[0008]
Accordingly, there is a need for a system and method that provides an enterprise and ASP computing platform that addresses the above shortcomings.
[Means for Solving the Problems]
[0009]
The invention features a platform and method for computer processing in which a virtual processing area network is to be formed and deployed.
[0010]
In accordance with certain aspects of the present invention, a method and system for emulating a switched Ethernet local area network is provided. Multiple computer processors, switch structures and point-to-point links to those processors are provided. Virtual interface logic establishes a virtual interface across the switch structure and point-to-point links. Each virtual interface defines a software communication path from one computer processor to another by a switch structure. The Ethernet driver emulation logic performs tasks on at least two computer processors, and the switch emulation logic performs tasks on at least one of the computer processors. The switch emulation logic establishes a virtual interface between the switch emulation logic and each computer processor having Ethernet driver emulation logic executing thereon and allows software communication therebetween. In addition, it receives a message from one of the virtual interfaces to the computer processor that has Ethernet driver emulation logic that performs the tasks on it and responds to the information associated with that message as well. The message is sent to another computer processor having Ethernet driver emulation logic that performs the task. In addition, it also establishes a virtual interface between each computer processor having Ethernet driver emulation logic that performs tasks on it and every other computer processor that has Ethernet driver emulation logic that performs tasks on it. . The Ethernet driver emulation logic unicast is via the virtual interface that defines the software communication path when the virtual interface is operating satisfactorily, and via the switch emulation logic when the virtual interface is not operating satisfactorily. Communicate with another computer processor in the emulated Ethernet network.
[0011]
According to another aspect of the invention, a method and system for implementing an Address Resolution Protocol (ARP) is provided. The computing platform has a plurality of processors connected by an underlying physical network. Logic executable on one of the processors defines the topology of the Ethernet network to be emulated on a computing platform. The topology includes processor nodes and switch nodes. The logic that can be executed on one of the processors is assigned a set of processors from a plurality of ones so as to serve as a processor node. Logic executable on one of those processors assigns a virtual MAC address to each processor node of the emulated Ethernet network. Logic executable on one of those processors assigns a virtual interface to the underlying physical network to provide direct software communication from each processor node to every other processor node. . Each virtual interface has a corresponding identification. Each processor node has ARP request logic to communicate the ARP request to the switch node where the ARP request includes an IP address. The switch node includes ARP request broadcast logic to communicate ARP requests to all other processor nodes in the emulated Ethernet network. Each processor node has ARP response logic to determine if it is the processor node associated with the IP address in the ARP request and, if so, to issue an ARP response to that switch node, where The ARP response contains the virtual MAC address of the processor node associated with that IP address. The switch node includes ARP response logic for receiving the ARP response and modifying the ARP response including a virtual interface identification for the ARP requesting node.
[0012]
In accordance with another aspect of the invention, a platform and method for computer processing is provided to support processor failover. A plurality of computer processors are connected to the internal communication network. A virtual local area communication network on the internal network is defined and established. Each computer processor in the virtual local area communication network has a corresponding virtual MAC address, and the virtual local area network provides communication between a set of computer processors, but the defined set Processors from a plurality other than the inside are excluded. The virtual storage space is defined and established by a defined correspondence to the storage network address space. In response to a failure by the computer processor, one of the plurality of computer processors is allocated for replacement with the failed processor. The MAC address of the failed processor is assigned to the processor that replaces the failed processor. The virtual memory space of the failed processor and the defined correspondence is assigned to the processor that replaces the failed processor. A virtual local area network is re-established to include the processor that replaces the failed processor and to exclude the failed processor.
[0013]
According to another aspect of the invention, a system and method for providing a service addressed by an IP address is provided. At least two computer processors each contain logic to provide that service. The cluster logic receives the service request message. The message has an IP address. The cluster logic distributes the request to one of at least two computer processors having logic to provide the service.
[0014]
In accordance with another aspect of the invention, a computer processing platform includes a plurality of computer processors connected to an internal communication network. At least one control node is in communication with an external communication network and an external storage device network having an external storage device address space. At least one control node is connected to the internal network, thereby communicating with the multiple computer processors. The configuration logic provides a virtual processing area network having a set of corresponding computer processors from multiple processors and communication between the set of computer processors, but excludes multiple processors that are not in the defined set. Defining and establishing a virtual local area communication network and a virtual storage space with a defined correspondence to the address space of the storage network.
BEST MODE FOR CARRYING OUT THE INVENTION
[0015]
Preferred embodiments of the present invention provide a processing platform in which virtual systems are deployed by configuration commands. The platform provides a large group of processors from which subgroups are selected by software commands and can be deployed to serve a given set of applications and customers (a “processing area network” or “processor cluster”). )) Virtual network is formed. The virtualized processing area network (PAN) can then be used to run customer specific applications such as web-based server applications. The virtualization may include local area network (LAN) virtualization or input / output storage device virtualization. By providing such a platform, processing resources can be quickly delivered via software via configuration commands from the administrator rather than physically supplying servers, connecting networks and storage devices and providing power to each server, etc. And can be easily deployed.
[0016]
(Overview of the platform and its behavior)
As shown in FIG. 1, a preferred hardware platform 100 includes a set of processing nodes 105a-105n connected to switch structures 115a, b via high speed interconnects 110a, b. The switch structure 115a, b is also connected to an external IP network 125 (or other data communication network) and at least one control node 120a, b in the communication path to the storage device network (SAN) 130. . For example, a management application 135 that performs tasks remotely will access one or more of the control nodes via the IP network 125 to support configuring the platform 100 and deploying a virtual PAN.
[0017]
Under certain embodiments, approximately 24 processing nodes 105a-105n, two control nodes 120, and two switch structures 115a, 115b are housed in a single chassis to provide a point-to-point (PtP) link. Interconnected with a fixed pre-wired network. Each processing node 105 includes one or more (e.g. four) processors 106J-106l, one or more LAN connection cards (NICs) that contain some BIOS firmware for booting and initial configuration among others. A board containing 107 and local memory (eg, greater than 4 Gbytes). There is no local disk for the processor 106; instead, all storage devices, including those required for paging, are handled by the SAN storage device 130.
[0018]
Each control node 120 includes one or more (e.g., four) processors, local memory, and a boot image and initial file system used to boot operating system software for processing node 105 and control node 106. It is a single board containing a local magnetic disk unit for holding independent copies. Each control node communicates with SAN 130 via a 100 Mbyte / s Fiber Channel adapter card 128 connected to Fiber Channel links 122, 124, and one connected to Gigabit Ethernet links 121, 123 It communicates with the Internet (or other external network) 125 via the external network interface 129 having the above Gigabit Ethernet NIC. (Many other technologies and hardware can be used for SAN and external network connectivity.) Each control node can be used by management application 135 as a dedicated management port that can be used remotely instead of web-based management. Includes a slow Ethernet port (not shown).
[0019]
Its switch structure consists of one or more 30-port Giganet switches 115, such as NIC-CLAN 1000 and clan 5300 switches, and various processing and control nodes have corresponding NICs in such structural modules Used for communication with. The Giganet switch structure has the non-broadcast multiple connectivity (NBMA) network semantics. All inter-node communication is via a switch structure. Each link is formed as a series connection between the NIC 107 and a port in the switch structure 115. Each link operates at 112 megabytes / second.
[0020]
In some implementations, multiple cabinets or chassis may be interconnected to form a larger platform. Also, in other embodiments, the configuration may be different; for example, redundant connections, switches, and control nodes may be removed.
[0021]
Under software management, the platform supports multiple simultaneous independent processing area networks (PANs). Each PAN is formed by a software command to have a corresponding subset of processors 106 communicating via a virtual local area network emulated over a PtP network. Each PAN is formed to have a corresponding virtual I / O subsystem. Physical deployment or wiring is not necessary to establish a PAN. Under certain preferred embodiments, software logic that performs tasks on processor nodes and / or control nodes emulates switched Ethernet semantics; performs tasks on processor nodes and / or control nodes Other software logic follows the SCSI semantics and provides a virtual storage subsystem function that provides an independent I / O address space for each PAN.
[0022]
(Network architecture)
One preferred embodiment uses the virtual components, interfaces and connections to allow an administrator to build a virtual emulated LAN. Each virtual LAN may be internal and dedicated to the platform 100, or multiple processors may be formed in a processor cluster that appears externally as a single IP address.
[0023]
Under certain embodiments, the physical underlying network is a PtP network, but the virtual network so created emulates a switched Ethernet network. Virtual networks use IEEE MAC addresses, and processing nodes support IETF ARP processing to associate and associate IP addresses with MAC addresses. Thus, a given processor node will consistently answer ARP requests whether the ARP request came from a node that is internal or external to the platform.
[0024]
FIG. 2A shows a typical network deployment that can be modeled or emulated. The first subnet 202 is a processing node PN that communicates with each other via the switch 206 ₁ , PN ₂ And PN _k It is formed by. The second subnet 204 is a processing node PN that communicates with each other via the switch 208 _k And PN _m It is formed by. Under switched Ethernet semantics, one node on a subnet may communicate directly with another node on that subnet; for example, PN ₁ Is PN ₂ Send a message to Semantics also allows one node to communicate with another set of nodes; for example, PN ₁ May send broadcast messages to other nodes. PN _m Are on different subnets, so the processing node PN ₁ And PN ₂ Is PN _m Cannot communicate directly with. PN ₁ And PN ₂ Is PN _m To communicate with, higher layer network software with a better understanding of both subnets will need to be utilized. Although not shown, a given switch may also be able to communicate via an “uplink” to another switch or the like. As can be seen in the following description, the need for such an uplink is different when the switch is physical. Specifically, since the switches are virtual and modeled in software, they can be expanded horizontally by the required width. (In contrast, physical switches have a fixed number of physical ports, and occasionally uplinks are required to provide horizontal scalability.)
[0025]
FIG. 2B illustrates exemplary software channels and logic used under certain embodiments to model subnets 202 and 204 of FIG. 2A. The communication path 212 is the processing node PN ₁ , PN ₂ , PN _k And PN _m In particular, connect their corresponding processor-side network communication logic 210, and furthermore they connect the processing node to the control node. (Listed as a single instance of logic for clarity, but PN _k May have multiple instances of corresponding processor logic, eg, one per subnet. Under the preferred embodiment, the management logic and control node logic are responsible for establishing, managing and destroying the communication path. Individual processing nodes are not allowed to establish such a path.
[0026]
As described in detail below, both processor logic and control node logic emulate switched Ethernet semantics on such a communication path. For example, the control node has control node side virtual switch logic 214 to emulate some (but not all) in the Ethernet switch semantics, and the processor logic is Contains logic to emulate (though not all).
[0027]
Within a subnet, one processor node may communicate directly with another via the corresponding virtual interface 212. Similarly, one processor node may communicate with control node logic via another virtual interface. Under certain embodiments, the underlying switch structure and associated logic (e.g., switch structure administrator logic (not shown)) establishes such virtual interfaces (VIs) over a point-to-point network. And provide the ability to manage. In addition, these virtual interfaces are established in a reliable and redundant manner, referred to herein as RVI. In this aspect of the description, the choice between VI vs. RVI is highly dependent on the cost incurred as system resources and the amount of reliability desired by the system, so it is called virtual interface (VI) and reliable virtual interface (RVI). The terms are used interchangeably.
[0028]
Referring to FIGS. 2A and 2B combined, node PN ₁ Is node PN ₂ If it should communicate with the virtual interface 212 _1-2 It is done normally. However, for example VI 212 _1-2 Is not working satisfactorily, in a preferred embodiment, PN ₁ And PN ₂ Communication between the two can occur via the switch emulation logic. In this case the message is VI 212 _{1 # switch206} Via, also VI 212 _switch206-2 Can be sent via. PN ₁ If it is to broadcast or multicast a message to other nodes in subnet 202, it does so by sending the message to control node side logic 214 via virtual interface 2121-switch 206. The control node side logic 214 then emulates the broadcast or multicast functionality by using the relevant VIs to clone and send messages to other relevant nodes. . The same or similar VI may be used to convey other messages that require control node side logic. For example, as described below, the control node side logic includes logic to support Address Resolution Protocol (ARP), and the VI is used to communicate ARP responses and requests to the control node. Although the above description suggests only one VI between the processor logic and the control logic, many implementations use multiple such connections. Furthermore, although the figure suggests symmetry in the software channel, its architecture actually allows asymmetric communication. For example, for a cluster communication service, the packet will be sent via the control node, as discussed below. However, return communication may be direct between nodes.
[0029]
Node PN, as in the network of Figure 2A ₂ And PN _m It should be noted that there is no communication mechanism in between. Furthermore, because the communication path is centrally managed and created (not via the processing node), such a path cannot be created by the processing node, and the connection of the defined subnet is broken by the processor. Absent.
[0030]
FIG. 2C illustrates an exemplary physical connection of an embodiment for implementing the subnets of FIGS. 2A and B. In particular, each instance of the processing network logic 210 communicates with the switch structure 115 via the PtP link 216 of the interconnect 110. Similarly, the control node has multiple instances of switch logic 214 and each communicates over a PtP connection 216 to the switch structure. The virtual interface of FIG. 2B includes logic to communicate information across these physical links, as described further below.
[0031]
To create and form such a network, the administrator defines the PAN's network topology and specifies the MAC address assignments of the various nodes (eg, via a utility in the management software 135). The MAC address is virtual and identifies the virtual interface and is not tied to any particular physical node. Under certain embodiments, the MAC address follows the IEEE 48-bit address format, but its contents are the "locally processed" bit (set to 1), the virtual interface was first defined (details below) It includes the serial number of the control node 120 and the count value from the persistent sequential counter on the control node maintained in NVRAM at the control node. These MACs will be used to identify nodes (as conventional) at the layer 2 level. For example, when answering an ARP request (whether from a node internal to PAN or on an external network), these MACs will be included in the ARP response.
[0032]
The network logic on the control node side maintains a data structure that includes information reflecting the LAN connection (for example, which node communicates with which other node). The control node logic also distributes and assigns VI (or RVI) mappings to specified MAC addresses, and assigns and assigns VIs (or RVIs) between control nodes and between control nodes and processing nodes. In the example of FIG. 2A, the logic would allocate and assign VI 212 of FIG. 2B. (The VI and RVI designations in some embodiments are the result of the switch structure used and the switch structure administrator logic.)
[0033]
As each processor boots, the BIOS-based boot logic initializes each processor 106 of node 105 and, among other things, establishes (or discovers) VI 212 to the control node logic. The processor node then obtains the relevant data link information such as the processor node's MAC address from the control node and the MAC authentication of other devices in the same data link configuration. Each processor then registers its IP address with the control node, which binds that IP address to the node and RVI (eg, the RVI on which the registration arrived). In this way, the control node will be able to bind an IP address for each virtual MAC for each node on the subnet. In addition to the above, the processor node also obtains RVI or VI related information for its connection to other nodes or to the control node network logic.
[0034]
Thus, after boot and initialization, the various processor nodes must understand their layer 2 data link connections. As explained below, layer 3 (IP) connectivity and in particular layer 3 and layer 2 coupling is determined during the normal processing of the processor as a result of the address resolution protocol.
[0035]
FIG. 3A details the network logic 210 on the processor side, and FIG. 3B details the network 310 logic on the control node side of one embodiment. The processor side logic 210 includes an IP stack 305, a virtual network driver 310, an ARP logic 350, an RCLAN layer 315, and redundant Giganet drivers 320a, b. The control node side logic 310 includes redundant Giganet drivers 325a, b, RCLAN layer 330, virtual cluster proxy logic 360, virtual LAN server 335, AJRP server logic 355, virtual LAN proxy 340, and physical LAN driver 345.
[0036]
(IP stack)
The IP stack 305 is a communication protocol stack provided with an operating system (for example, Linux) used by the processing node 106. The IP stack provides a layer 3 interface to applications and operating systems that perform tasks on the processor 106 to communicate with the simulated Ethernet network. The IP stack supplies a packet of information to the virtual Ethernet layer 310 and, at the same time, provides a layer 3 IP address as the destination of the packet. The IP stack logic is conventional except that certain implementations avoid checksum computation and logic.
[0037]
(Virtual Ethernet driver)
A virtual Ethernet driver 310 will appear in the IP stack 305 like a “true” Ethernet driver. In this regard, the virtual Ethernet driver 310 receives an IP packet or datagram from the IP stack for subsequent transmission on the network, and it receives packet information from the network so that it can be delivered to that stack as an IP packet. obtain.
[0038]
The stack creates a MAC header. “Normal” Ethernet code in the stack may be used. The virtual Ethernet driver receives a packet with the MAC header already created and the exact MAC address already in the header.
[0039]
In the data part and with respect to FIGS. 4A-C, the virtual Ethernet driver 310 dequeues the 405 output IP datagram so that the packet is transmitted over the network. Standard IP stack ARP logic is used. As explained below, the driver blocks all ARP packets entering and leaving the system to modify them so that the appropriate information ends in each node's ARP table. The normal ARP logic puts the correct MAC address in the link layer header of the outgoing packet before the packet is queued to the Ethernet driver. The driver then only checks the link layer header and destination MAC to determine how to send the packet. The driver does not manipulate the ARP table directly (except for invalidity that can occur during ARP entry).
[0040]
The driver 310 determines 415 whether the ARP logic 350 has MAC address information (details below) associated with the IP address in the dequeued packet. If the ARP logic 350 has information, that information is used to send the packet accordingly (420). If the ARP logic 350 does not have the information, the driver needs to determine such information, and in some preferred embodiments, this information is the ARP protocol, as discussed with respect to FIGS. 4B and 4C. As a result of the execution of.
[0041]
If the ARP logic 350 has MAC address information, the driver analyzes the information returned from the ARP logic 350 to determine where and how to send the packet. In particular, the driver looks at the address to determine if the MAC address is in a valid format or a specially invalid format. For example, in some implementations, an internal node (i.e., a PAN node within that platform) has a locally-managed bit, a multicast bit, and another predetermined bit pattern in the first byte of the MAC address. A signal is given in the combination to set. The dominant pattern is unlikely to be a valid pattern.
[0042]
If the MAC address returned from the ARP logic is in a valid format, the IP address associated with that MAC address is at least for nodes that are external to the associated subnet, and in a preferred embodiment, the IP address External to the platform. To convey such a packet, the driver prepends the packet with a TLV (type-length-value) header. The logic then sends the packet to the control node over the pre-established VI. In addition, the control node handles the remainder of the transmission appropriately.
[0043]
If the MAC address information returned from the ARP logic 350 is in a specially invalid format, the invalid format signals that the node at that IP address is for an internal node, and then the MAC That information in the address information is used to help identify the VI (or RVI) that directly connects the two processing nodes. For example, an ARP table entry may hold information identifying the RVI 212 used to send a packet, eg 212i-2, to another processing node. The driver prepends a packet with a TLV header. It then puts address information in the header as well as information identifying the Ethernet protocol type. The logic then selects the appropriate VI (or RVI) to send the encapsulated packet. If that VI (or RVI) is working satisfactorily, it is used to carry the packet; if it is not working satisfactorily, the switch logic can send it to the appropriate node , The packet is sent to the control node switch logic (detailed below). The ARP table may contain information for specifying the RVI to actually use, but many other techniques may be used. For example, the information in the table may provide such information indirectly, for example by pointing to the information or by revealing the information through the absence of the information.
[0044]
For any multicast or broadcast type message, the driver sends those messages to the control node on the specified VI. The control node then clones the packet and sends it to all nodes (except the sending node) and the uplink.
[0045]
Without ARP mapping, the upper layer would never send the packet to the driver. If no data link layer mapping is available, the packet is shelved until ARP decomposition is complete. Once the ARP layer finishes ARPing, packets that have held the pending ARP are built with their data link headers, which are then sent to the driver.
[0046]
If the ARP logic does not have a mapping for the IP address of the IP packet from the IP stack, and therefore the driver 310 cannot determine the relevant addressing information (i.e. MAC address or RVI related information), the driver Get such information by following the ARP protocol. Referring to FIGS. 4B and 4C, the driver creates an ARP request packet that includes an associated IP address for which the MAC mapping is not in the local ARP table (425). The node then adds an ARP packet with a TLV type header to the beginning (430). Next, an ARP request is sent to the network logic on the control node side, specifically, to the virtual LAN server 335 via dedicated RVI.
[0047]
As discussed in more detail below, the ARP request packet is processed (435) by the control node and broadcast 440 to the associated node. For example, the control node will flag whether the requesting node is part of an IP service cluster.
[0048]
The associated node's Ethernet driver logic 310 receives the ARP response (445), and further compares the target IP address with a locally formed list of IP addresses by making a call to that node's IP stack. , Determine if it is the target of the ARP request (450). If it is not the target, it will forgo the packet without modification. If it is the target, the driver creates a local MAC header from the TLV header (460) and updates the local ARP table to create an ARP response (465). The driver modifies the information in the ARP request (primarily the source MAC) and then usually forgets the ARP request for higher layers to handle. It is the upper layer that forms the ARP response when necessary. The response in the others contains the MAC address of the responding node and has a bit set in the TLV header indicating that the response is from the local node. In this respect, the node responds according to the EETF type ARP semantics (as opposed to the ATM ARP protocol, where ARP responses are handled centrally). The response is then sent to 470.
[0049]
As described in more detail below, control node logic 335 receives the response and modifies it (473). For example, the control node may substitute the MAC address of the responding internal node having information identifying the source cabinet, processing node number, RVI connection number, channel, virtual interface number, and virtual LAN name. Once the ARP response has been modified, the control node logic then sends the ARP response to the appropriate node, ie the node that sent the ARP request, or, in the example, the load balancer in the JDP service cluster discussed below (475 ).
[0050]
Finally, an encapsulated ARP response is received (480). If the replying node is an external node, the ARP response contains the MAC address of the replying node. If the responding node is an internal node, the ARP response instead contains information identifying the associated RVI communicating with the node. In either case, the local table is updated (485).
[0051]
The pending datagram is removed from the queue (487) and the appropriate RVI is selected (493). As discussed above, the appropriate RVI is selected based on whether the target node is internal or external. The TLV header is added to the beginning of the packet and transmitted (495).
[0052]
For communication within a virtual LAN, the maximum transfer unit (MTU) is formed as 16,896 bytes. Even if the formed MTU is 16,896 bytes, the Ethernet driver 310 recognizes when the packet is being sent to the external network. Through the use of path MTU discovery, ICMP and IP stack changes, the path MTU is changed at the source node 105. This mechanism is used to trigger a packet checksum.
[0053]
Certain embodiments of the present invention support promiscuous mode by a combination of logic within the virtual LAN server 335 and the virtual LAN driver 310. When the virtual LAN driver 310 receives a promiscuous mode message from the virtual LAN server 335, the message includes information regarding the authentication of the receiver who wishes to enter promiscuous mode. This information includes the location of the receiver (cabinet, node, etc.), the interface number of the promiscuous virtual interface 310 on that receiver (required for packet demultiplexing), and the name of the virtual LAN to which the receiver belongs . This information is then used by driver 310 to determine how to send promiscuous packets to the receiver (which RVI or other mechanism to use to send those packets). The virtual interface 310 maintains a promiscuous listener list on the same virtual LAN. When the sending node receives a promiscuous mode message, it will update its promiscuous list accordingly.
[0054]
If the packet is sent through the virtual Ethernet driver 310, this list will be examined. If the list is not empty, virtual Ethernet interface 310 will do the following:
[0055]
Promiscuous copies will not be sent if the transmitted packet is broadcast or multicast. Normal broadcast operations will send packets to the promiscuous listener.
[0056]
If the packet is a unicast packet with a destination other than the promiscuous listener, the packet will be cloned and sent to the promiscuous listener.
[0057]
The header TLV includes additional information that can be used by the destination to demultiplex and validate the incoming packet. Part of this information is the destination virtual Ethernet interface number (the destination device number on the receiving node). Since these can differ between the actual packet destination and the promiscuous destination, this header cannot simply be cloned. Thus, memory would have to be allocated for each header of each packet clone to individual promiscuous listeners. If the packet header for a promiscuous packet is shaped, the packet type will be set to indicate that the packet was a promiscuous transmission rather than a unicast transmission.
[0058]
Furthermore, the virtual Ethernet driver 310 is responsible for handling redundant control node connections. For example, a virtual Ethernet driver will periodically test end-to-end connectivity by sending a TLV heartbeat to each connection RVI. This will allow the virtual Ethernet driver to determine if the node has stopped responding or if the stopped node has started responding again. If it is determined that the RVI or control node 120 is down, the Ethernet driver will send traffic through the remaining control node. If both control nodes are functioning, driver 310 will attempt to load balance traffic between the two nodes.
[0059]
Certain embodiments of the present invention provide improved performance. For example, with changes to the IP stack 305, packets sent only within the platform 100 are guaranteed to be sent without checksum because all elements of the platform 100 provide error detection.
[0060]
Further, for communication within the PAN (or even within the platform 100), the RVI may be configured such that the packet is larger than the maximum size allowed by the Ethernet. Thus, while some models emulate Ethernet behavior in certain embodiments, the maximum packet size may be broken through to improve performance. The actual packet size will be handled well as part of the data link layer.
[0061]
A control node failure is detected by notification from the RCLAN layer or by a TLV heartbeat failure. If the control node fails, the Ethernet driver 310 will send traffic only to the remaining control nodes. The Ethernet driver 310 will recognize the recovery of the control node via notification from the RCLAN layer or notification from the resumption of the TLV heartbeat. Once the control node has recovered, the Ethernet driver 310 will resume load balancing.
[0062]
If it detects that a node cannot communicate with another node via direct RVI (as outlined above), it will act as a switch and attempt to communicate via the control node. Such a fault may be signaled by the lower RCLAN layer, for example, from a virtual interface response signal reception fault or from a fault detected through a heartbeat mechanism. In this example, the driver can mark a bit in the TLV header to indicate that the message should be unicast and send the packet to the desired node (e.g., based on IP address if necessary). Send the packet to the control node so that it can.
[0063]
(RCLAN layer)
The RCLAN layer 315 is responsible for handling the redundancy, failover, and load balancing logic of the redundant interconnected NIC 107. This includes failure detection, traffic path reconfiguration on redundant connections in case of failure, load balancing and reporting that traffic cannot be returned to the virtual network driver 310. If there is a critical error that disables RVI on any RVI, or if RVI fails for any reason, the virtual Internet driver 310 should be notified asynchronously.
[0064]
Under normal circumstances, the virtual network driver 310 on each processor will attempt to balance the load of transmitted packets among the available control nodes. This can be done via a simple round-robin alternation between available control nodes, or by tracking how many bytes have been sent to each and sending them to the control node that sent the least bytes. Can be done.
[0065]
RCLAN provides reliable asynchronous fixed communication with high bandwidth between kernels (224 MB / sec per route), low latency, and low latency. If the data cannot be communicated, the data sender will be notified and the best effort will be made to convey it. RCLAN uses two Giganet clan 1000 cards to provide redundant communication paths between kernels. It recovers without interruption to a single failure in a clan 1000 card or Giganet switch. It detects data loss and data errors and retransmits the data as needed. Communication will not be interrupted unless one of the connections is partially working, eg, the error rate exceeds 5%. RCLAN clients include an RFC mechanism, a remote SCSI mechanism, and a remote Ethernet. RCLAN also provides simple shape flow control. Short latency and high parallelism means that multiple simultaneous requests for each device can be transferred to the device as soon as possible, or the device as much as possible against queuing all requests for processor nodes By being sent by the processor node to the control node so that it can be queued for completion near.
[0066]
The RCLAN layer 330 on the control node side operates in the same manner as described above.
[0067]
(Giganet driver)
The Giganet driver logic 320 is the logic responsible for providing an interface to the Giganet NIC 107 on the processor 106 or control node 120. In short, the Giganet driver logic establishes a VI connection that is related by a VI indicator so that a higher layer, for example RCLAN 315, Ethernet driver 310, simply needs to understand the semantics of the VI.
[0068]
The Giganet driver logic 320 is responsible for allocating memory to each buffer and queue node for VIs and conditioning NIC 107 to know about connections and their storage allocations. One embodiment uses the VI connection provided by the Giganet driver. The Giganet NIC driver code establishes a virtual interface pair (ie VI) and assigns it to the corresponding virtual interface indicator.
[0069]
Each VI is established between one Giganet port and another, or more precisely, between a memory buffer and a memory queue on one node for the buffer and a queue on another node Bidirectional connection. Port and memory allocation is handled by the NIC driver as described above. Data is sent by triggering an action by putting it in a buffer known to the NIC and writing to a specific memory map register. At the receiving end, the data appears in the buffer and the completion status appears in the queue. If the sending and receiving programs can generate and consume messages in the connection's buffer, there is no need to copy the data. If the operating system memory maps the buffers and instruction registers of the connection to the application address space, the transmission can be directly from the application program to the application program. Each Giganet port supports 1024 simultaneous VI connections on it, allowing you to keep them separated from each other with hardware protection, so the operating system is as secure as disparate applications One port can be shared. Under certain embodiments of the invention, 14 VI connections are established simultaneously from every other port every other port.
[0070]
In the preferred embodiment, the NIC driver establishes a VI connection in a pair of redundant connections, one connection in the pair going through one of the two switch structures 115a, 115b and the other connection in the other Go through the switch. Further, in the preferred embodiment, data is transmitted alternately on the two legs in pairs, making the load on the switches equal. Alternatively, redundant pairs may be used in a failover manner.
[0071]
As long as the operating system remains, all connection pairs established by that node will survive. Establishing a connection pair to simulate an Ethernet connection is similar to physically plugging in cables between network interface cards and is intended to be persistent as well. If the defined configuration of a node changes while the operating system is running, the applicable redundant virtual interface connection pair will be destroyed when it is established or changed.
[0072]
The Giganet driver logic 325 on the control node side operates similar to the above.
[0073]
(Virtual LAN server)
Virtual LAN server logic 335 facilitates emulation of an Ethernet network over a basic NBMA network. Virtual LAN server logic is:
1. Manage membership to the corresponding virtual LAN;
2. Provide RVI mapping and management;
3. ARP processing and IP mapping to RVI;
4. Provide broadcast and multicast services;
5. Facilitates bridging and routing to other areas;
Also
6. Manage service clusters.
[0074]
(1. Virtual LAN membership management)
Administrators use the management application 135 to configure a virtual LAN. The assignment and configuration of the IP address on the virtual LAN may be performed in the same manner as on the “normal” subnet. The selection of the IP address to use depends on the external visibility of the node on the virtual LAN. If the virtual LAN is not visible globally (either visible outside the platform 100 or not visible from the Internet), a private IP address should be used. Otherwise, the IP address must consist of a range provided by the Internet service provider (ISP) that provides the Internet connection. In general, virtual LAN IP address assignment must be handled in the same way as normal LAN IP address assignment. The configuration file stored on the local disk of the control node 120 defines the IP address in the virtual LAN. For a virtual network interface, an IP alias simply creates another IP for RVI mapping on the virtual LAN server logic 335. Each processor may form a plurality of virtual interfaces as necessary. The main limitation on the creation and configuration of virtual network interfaces is the assignment and configuration of IP addresses.
[0075]
Each virtual LAN has a corresponding instance of server logic 335 that performs the task on both the control node 120 and the many nodes that execute the task on the processor node 105. The topology is defined by the administrator.
[0076]
Each virtual LAN server 335 is formed to manage exactly one broadcast area, and any number of layer 3 (IP) subnets may exist in the layer 2 broadcast area. The server 335 is formed and created in response to an administrator command for creating a virtual LAN.
[0077]
When the processor 106 boots and configures its virtual network, it connects to the virtual LAN server 335 via a special management RVI. The processor then obtains the virtual MAC address assigned to it and their data link configuration information such as virtual LAN membership information. The virtual LAN server 335 determines and verifies that the processor attempting to connect to it is appropriately a member of the virtual LAN that the server 335 is serving. If the processor is not a virtual LAN member, connection to the server is refused. If it is a member, the virtual network driver 310 registers its IP address with the virtual LAN server. (If the driver 310 is configured, the IP address is provided by the IP stack 305.) The virtual LAN server then binds the IP address to the RVI on which the registration arrived. This allows the virtual LAN server to find the processor associated with a particular IP address. Further, the association between the processor and the IP address may be performed via the virtual LAN management interface 135. The latter method is necessary to properly configure a cluster IP address, or a special handling IP address discussed below.
[0078]
(2. RVI mapping and management)
As outlined above, one embodiment uses RVI to connect nodes at the data link layer to form a control connection. Some of these connections are created and assigned as part of the control node boot and initialization. Data link layer connections are used for the reasons described above. Control connections are used to exchange management, configuration and health information.
[0079]
Some RVI connections are between nodes for unicast traffic (eg 212i.2). Other RVI connections are in the virtual LAN server logic 335 so that the server can handle its requests, eg ARP traffic, broadcasts, etc. To create the RVI, the virtual LAN server 335 creates and deletes the RVI through a call to the Giganet switch manager 360 (supplied with the switch structure and Giganet NIC). The switch manager may execute a task on the control node 120 and create an RVI in cooperation with the Giganet driver.
[0080]
Since the nodes register with the virtual LAN server 335 for processor connection, the virtual LAN server creates and assigns virtual MAC addresses to those nodes as described above. Along with this, the virtual LAN server logic maintains a data structure that reflects the topology and MAC assignments for the various nodes. The virtual LAN server logic then creates a corresponding RVI for the unicast path between the nodes. These RVIs are then allocated and made known to the node during the boot of the node. RVI is also associated with an IP address during virtual LAN server ARP traffic handling. If the node is removed from the topology, the RV1 connection is disconnected.
[0081]
If a node 106 at one end of an established RVI connection is rebooted, the two operating systems at each end of the connection and the RVI management logic reestablish the connection. Software that uses a connection on the remaining processing node will not be aware of what happened to the connection itself. Whether the software notices or notices that the software at the other end has been rebooted is what the connection it is using for, and the rebooted terminal persists Depending on the extent to which the state can be re-established from the storage device. For example, any software that communicates via the Transfer Control Protocol (TCP) will notice that all TCP sessions are closed by a reboot. On the other hand, if it occurs within the allowed timeout period, Network File System (NFS) access is unaffected and is not affected by reboot.
[0082]
Should a node fail to send a packet over direct RVI at any time, it can always attempt to send the packet to the destination via the virtual LAN server 335. Since the virtual LAN server 335 is connected to all the virtual Ethernet driver 310 interfaces on the virtual LAN via the control connection, the virtual LAN server 335 further serves as a packet relay mechanism as a last resort.
[0083]
With respect to connecting to a virtual LAN server 335, one embodiment uses a virtual Ethernet driver 310 that algorithmically determines the RVI it should use to connect to its associated virtual LAN server 335. Implementation dependent algorithms will need to consider identification information such as cabinet number to identify the RVI.
[0084]
(3. ARP processing and IP mapping to RVI)
As described above, one embodiment of the virtual Ethernet driver 310 supports ARP. In these implementations, ARP processing is effectively used to create unicast traffic, including IP packets, between nodes, and to create mappings at nodes between IP addresses and RVI.
[0085]
To do this, the virtual Ethernet driver 310 sends an ARP packet request and response to the virtual LAN server 335 via dedicated RVI. The virtual LAN server 335, and in particular the ARP server logic 355, processes the packet by adding information to the packet header. As explained above, this information facilitates source and target identification and identifies the RVI used between nodes.
[0086]
The ARP server logic 355 accepts the ARP request, processes the TLV header, and broadcasts the request to all relevant nodes on the internal platform and external network, if appropriate. Among other things, the server logic 355 determines who receives the ARP response due to the request. For example, if the source is a cluster IP address, the response should be sent to the cluster load balancer, not necessarily the source of the ARP request. Server logic 355 indicates this by including information in the TLV header of the ARP request, so that the ARP target responds accordingly. Server 335 will process the ARP packet by including more detailed information in the added header and broadcast the packet to nodes in the associated region. For example, the modified header contains information identifying the source cabinet, processing node number, RYE connection number, channel, virtual interface number and virtual LAN name (some of which are known only to server 335). You may go out.
[0087]
That ARP response is received by the server logic 355, which then maps the MAC information to the response to the corresponding RVI related information. The RVI related information is placed in the target MAC entry of the response and to the appropriate source node (e.g. it may be the sender of the request, but in instances such as with cluster IP address, it may be a different node) Sent.
[0088]
(4. Broadcast and multicast services)
As outlined above, broadcasts are handled by receiving packets on dedicated RVI. The packet is then cloned by server 335 and unicast to all virtual interfaces 310 in the associated broadcast area.
[0089]
The same approach can be used for multicast. All multicast packets will be reflected on the virtual LAN server. Under some alternative implementations, the virtual LAN server relies on IP filtering on each node to handle multicast as well as broadcast and further filter out unwanted packets.
[0090]
If an application wants to send or receive a multicast address, it must first join the multicast group. If the process performs multicast binding on the processor, the processor virtual network driver 310 sends a binding request to the virtual LAN server 335 via dedicated RVI. The virtual LAN server then configures a specific multicast MAC address on the interface and notifies the LAN proxy 340 when necessary, as discussed below. Proxy 340 needs to keep track of usage counts on a particular multicast group, so the multicast address is only removed if no processor belongs to that multicast group.
[0091]
(5. Bridging and routing to other areas)
From a perspective view of the system 100, the external network 125 can operate in one of two modes: filtered or unfiltered. In filtered mode, a single MAC address for the entire system is used for all transmitted packets. This hides the virtual MAC address of the processing node 107 after the virtual LAN proxy 340 and causes the system to appear as a single node on the network 125 (or as a plurality of nodes after bridging or proxying). Since this does not expose the unique link layer information of each internal node 107, some other unique identifier is required to properly convey incoming packets. When operating in filter mode, the destination IP address of each incoming packet is used to uniquely identify the intended recipient since the MAC address identifies only that system. In the unfiltered mode, the virtual MACs of nodes 107 are visible outside the system as they are used to direct incoming traffic. That is, the filtered mode orders layer 3 switching, while the unfiltered mode allows layer 2 switching. Filtered mode requires a basic element (in this case virtual LAN proxy 340) to replace the address with the MAC address of the external network 125 on all outgoing packets from the node virtual MAC.
[0092]
Some implementations support the ability of a virtual LAN to be connected to an external network. The virtual LAN will therefore have to handle IP addresses that are not locally configured. One embodiment for addressing this imposes the restriction that each virtual LAN so connected is limited to one external broadcast area. The IP address and subnet assignment for the internal node of the virtual LAN will have to be adapted to the external area.
[0093]
Virtual LAN server 335 serves external connections by effectively acting as a data link layer bridge in that it moves packets between external Ethernet driver 345 and the internal processor and does not perform IP processing. . However, unlike something like a data link layer bridge, a server may not necessarily rely on a unique layer 2 address from the external network to the internal node, and instead the connection makes a bridging decision. Layer 3 (IP) information may be used to To do this, the external connection software extracts the IP address information from the incoming packet, and it uses this information to identify the correct node 106 so that it moves the packet to that node.
[0094]
Virtual LAN server 335 with attached external broadcast area blocks packets from and to external areas so that external nodes have a consistent view of the subnet of the broadcast area, Must be processed.
[0095]
When a virtual LAN server 335 having an attached external broadcast area receives an ARP request from an external node, it will relay the request to all internal nodes. The correct node will then compose the response and send the response through the virtual LAN server 335 to the requester. The virtual LAN server cooperates with the virtual LAN proxy 340 so that the proxy handles any necessary MAC address translation on the outgoing request. All ARP responses and ARP advertisements from external sources will be relayed directly to the target node.
[0096]
The virtual Ethernet interface 310 will send all unicast packets with external destinations to the virtual LAN server 335 on the control connection RVI. (The external destination may be recognized by the driver in the MAC address format.) The virtual LAN server will then move the packet to the external network 125 accordingly.
[0097]
When the virtual LAN server 335 receives a broadcast or multicast packet from the internal node, it relays the packet to the external network in addition to the relay of the packet to all the internal virtual LAN members. When the virtual LAN server 335 receives a broadcast or multicast packet from an external source, it relays the packet to all attached internal nodes.
[0098]
Under certain embodiments, interconnecting virtual LANs through the use of IP routers or firewalls is accomplished using a similar mechanism to that used to interconnect physical LANs. The One processor is configured on both LANs, and the Linux kernel on that processor must have enabled routing (and perhaps IP impersonation). Normal IP subnetting and routing semantics will always be preserved for two nodes on the same platform.
[0099]
The processor could be configured as a router between two external subnets, between and between external and internal subnets, and between two internal subnets. If the internal node is sending packets via a router, there is no problem because of the point-to-point topology of the internal network. The sender will send directly to the router (ie a processor configured with routing logic) without the intervention of a virtual LAN server (ie the typical processor-to-processor communication discussed above).
[0100]
When the external node sends a packet to the internal router and the external network 125 is operating in filtered mode, the destination MAC address of the incoming packet will be that of the platform 100. Therefore, the MAC address cannot be used to uniquely identify the packet destination node. For packets whose destination is an internal node on the virtual LAN, the destination IP address in the IP header is used to direct the packet to the appropriate destination node. However, since the router is not the final destination, the destination IP address in the IP header is that of the final destination, not that of the next hop (which is an internal router). Thus, there is nothing in the incoming packet that can be used to direct it to the correct internal node. In order to handle this situation, one implementation imposes the limitation of at most one router exposed to an external network on the virtual LAN. This router is registered by the virtual LAN server 335 as a default destination so that incoming packets that do not have a valid destination are directed to this default node.
[0101]
When the external node sends a packet to the internal router and the external network 125 is operating in the unfiltered mode, the destination MAC address of the incoming packet will be the virtual MAC address of the internal destination node. LAN server 335 will then use this virtual MAC to send the packet directly to the destination internal node. In this case, the router as the MAC address of the incoming packet may uniquely identify the destination node and any number of internal nodes may function as the router.
[0102]
If a configuration requires multiple routers on a subnet, one router can be taken as an exposed router. This router should be able to route to other routers in order when needed.
[0103]
Under certain embodiments, router redundancy is provided by making the router cluster service and load balancing or failing over in an unaffiliated manner (ie, per IP packet, not per TCP connection).
[0104]
Certain embodiments of the present invention provide promiscuous mode by providing switch semantics in which a given port is designated as a promiscuous port so that all traffic passing through the switch is repeated on the promiscuous port. Support functionality. Nodes that are allowed to listen in promiscuous mode will be administratively assigned by the virtual LAN server.
[0105]
When the virtual Ethernet interface 310 enters promiscuous receive mode, it will send a message to the virtual LAN server 335 over the management RVI. This message will contain all information about the virtual Ethernet interface 310 entering promiscuous mode. When a virtual LAN server receives a promiscuous mode message from a node, it will check its configuration information to determine whether the node is allowed to listen without regard to the other party. If not allowed, the virtual LAN server will withdraw its promiscuous mode message without further processing. If the node is allowed to enter promiscuous mode, the virtual LAN server will broadcast a promiscuous mode message to all other nodes on the virtual LAN. The virtual LAN server will also mark the node as promiscuous so that a copy of the incoming external packet can be forwarded to that node. When a promiscuous listener detects any change in its RVI configuration, it will send a promiscuous mode message to the virtual LAN to update the state of all other nodes on the associated broadcast area. This will update any node that enters or exits the virtual LAN. When the virtual Ethernet interface 310 exits promiscuous, it will send a message to the virtual LAN server notifying that the interface is out of promiscuous mode. The virtual LAN server will then send this message to all other nodes on the virtual LAN. Promiscuous configuration will allow external connections to be placed in promiscuous mode when any internal virtual interface is a promiscuous listener. This will make traffic that is external to that platform (but on the same virtual LAN) available to promiscuous listeners.
[0106]
(6. Service cluster to be managed)
A service cluster is a set of services that can be used with one or more IP addresses (or host names). Examples of these services are HTTP, FTP, Telnet, NFS, etc. The IP address and port number pair represents a particular service type (but not a service instance) presented by the cluster to clients, including clients on external network 125.
[0107]
FIG. 5 illustrates how an embodiment provides virtual cluster 505 services as a single virtual host to the Internet or other external network 125 via a cluster IP address. All services in cluster 505 are addressed by a single IP address through different ports of that IP address. In the example of FIG. 5, service B is a load balance service.
[0108]
With reference to FIG. 3B, the virtual cluster is supported by the intervention of virtual cluster proxy (VCP) logic 360 that cooperates with the virtual LAN server 335. In short, VCP 360 is responsible for handling the distribution of incoming connections, port filters and true server connections to each configured virtual IP address. There will be one VCP configured for each cluster IP address.
[0109]
When a packet arrives on the virtual cluster IP address, virtual LAN proxy logic 340 will send the packet to VCP 360 for processing. The VCP then determines where to send the packet based on the packet content, its internal connection state cache, any load balancing algorithm applied to incoming traffic, and the configured service uptime. Will. VCP will relay incoming packets based on both the destination IP address as well as the TCP or UDP port number. In addition, it will only distribute packets destined for known port numbers to the VCP (or for existing TCP connections). It is the configuration of these ports and the mapping of port numbers to one or more processors that create a virtual cluster and make a particular service instance available in that cluster. When multiple instances of the same service from multiple application processors are configured, the VCP can balance the load among the service instances.
[0110]
VCP 360 maintains a cache of all active connections that exist at the cluster IP address. Some load balancing decision will only be made when a new connection is established between the client and the service. Once the connection is set up, the VCP verifies that the source in the incoming packet header and The destination information will be used. Without the ability to determine a client session (eg HTTP session), the actual connection / load balancing mapping cache sends packets based on the client address, so that subsequent connections from the same client are the same Go to the processor (make the client session persistent or "sticky"). Since only certain types of services require session persistence, session persistence must be selectable in a service port number scheme.
[0111]
Responses to ARP requests and routing of ARP responses are handled by VCP. If the processor sends an arbitrary ARP packet, it will send it out by the virtual Ethernet driver 310. The packet will then be sent to the virtual LAN server 335 for normal ARP processing. As usual, the virtual LAN server will broadcast the packet, but will make sure it is not broadcast to any member of the cluster (not just a sender). It will also put in the packet header TLV information indicating to the ARP target that the ARP source can only be reached through that virtual LAN server, specifically only through the load balancer. The ARP target, whether internal or external, will normally process the ARP request and send the response back through the virtual LAN server. Since the ARP source was a cluster IP address, the virtual LAN server would not be able to determine which processor sent the original request. Therefore, the virtual LAN server will send its response to each cluster member so that it can be handled appropriately. When an ARP packet is sent as a target by a source with a cluster IP address, the virtual LAN server will send the request to all cluster members. Each cluster member will receive the ARP request and process it normally. They will then construct an ARP response and send it to the source via the virtual LAN server. If the virtual LAN server receives any ARP response from a cluster member, it will stop responding, but the virtual LAN server will construct and send an ARP response to the ARP source. Therefore, the virtual LAN server will respond to all ARPs for the cluster IP address. The ARP response will contain the information necessary for the ARP source to send all packets to the cluster IP address to the VCP. For an external ARP source, this would simply be an ARP response with the external MAC address as the source hardware address. For internal ARP sources, this will be the information needed to instruct that source to send packets to the cluster IP address under Virtual LAN Management RVI, not through directly connected RVI. Let's go. Any gratuitous ARP packets received will be forwarded to all cluster members. Any gratuitous ARP packets sent by cluster members will also be sent normally.
[0112]
(Virtual LAN proxy)
The virtual LAN proxy 340 performs basic coordination of physical network resources in all processors that have a virtual interface to the external physical network 125. It bridges the virtual LAN server 335 to the external network 125. When the external network 125 is operating in filtered mode, the virtual LAN proxy 340 will translate the internal virtual MAC address from each node to a single external MAC assigned to the system 100. If the external network 125 is operating in unfiltered mode, such MAC conversion is not necessary. Virtual LAN proxy 340 also inserts and removes demultiplexed packets based on IEEE 802.1Q virtual LAN ID tagging information and their VLAN indicators. It also aligns access to the physical Ethernet interface 129 and coordinates the assignment and removal of MAC addresses such as multicast addresses on the physical network.
[0113]
When the external network 125 is operating in filtered mode and the virtual LAN proxy 340 receives a transmission packet (ARP or other) from the virtual LAN server 335, it physically uses the internal format MAC address as the source MAC address. Replace with the MAC address of a typical Ethernet device 129. If the external network 125 is operating in the unfiltered mode, such replacement is not necessary.
[0114]
When the virtual LAN proxy 340 receives an incoming ARP packet, it processes the packet and moves the packet to a virtual LAN server 335 that relays the packet to the correct destination. If the ARP packet is a broadcast packet, the packet is relayed to all internal nodes on the virtual LAN. If the packet is a unicast packet, the packet is sent only to the destination node. By the IP address in the ARP packet when the external network 125 is operating in filtered mode, or by the MAC address in the Ethernet header of the ARP packet (the one that is not that MAC address) The destination node is determined.
[0115]
(Physical LAN driver)
Under certain embodiments, the connection to the external network 125 is via a Gigabit or 100/10 base-T Ethernet link connected to the control node. The physical LAN driver 345 is responsible for interfacing with such links. Packets being sent on that interface will be queued to the device in the usual way, including putting those packets in a socket buffer. The queue used to queue the packet is that used by the protocol stack to queue the packet to the device's transfer routine. For incoming packets, the socket buffer that contains the packet will be passed through and the packet data will not be copied (although it will be cloned if needed for multicast operation). Under these implementations, a generic Linux network device driver may be used in an unaltered control node. This facilitates the addition of new equipment to the platform without requiring additional device driver work.
[0116]
The physical network interface 345 is in a communication state only with the virtual LAN proxy 340. This prevents the control node from using external connections in any way that interferes with the operation of the virtual LAN, and further improves the security and isolation of user data, i.e. the administrator `` scents '' on any user packet It is not necessary.
[0117]
(Load balancing and failover)
Under some embodiments, redundant connections to the external network 125 will be used alternately to balance the packet transmission between the two redundant interfaces to the external network 125. Other embodiments balance the load by configuring individual virtual network interfaces with respect to alternating control nodes so that the virtual interfaces are evenly distributed between the two control nodes. Another embodiment transmits through one control node and receives through another control node.
[0118]
When in filtered mode, there will be one externally visible MAC address to which external nodes will send packets for a set of virtual network interfaces. If that adapter goes down, not only does the virtual network interface have to fail over to another control node, but the MAC address also allows the external node to continue sending packets to the MAC address already in the ARP cache. Need to file over. Under certain embodiments of the invention, when a failed control node recovers, a single MAC address is manipulated and that MAC address does not need to be remapped upon recovery.
[0119]
Under another embodiment of the invention, load balancing is performed by allowing transmissions on both control nodes, but only through one. In case of failover, both transmission and reception are performed through the same control node. In the case of recovery, it is a transmission through the recovered control node since it does not require a MAC operation.
[0120]
The control node that performs reception has IP information for filtering and multicast address information for multicast MAC configuration. This information is necessary for processing an incoming packet, and should be failed over if the reception control node fails. If the transmission control node fails, the virtual network driver needs to start sending transmission packets only to the reception control node. No special failover process is required other than the recognition that the transmission control node has failed. If the failed control node recovers, the virtual network driver may resume sending transmission packets to the recovered control node without additional special recovery processing. If the reception control node fails, the transmission control node must assume the role of the reception interface. In order to do this, it must configure all MAC addresses on that physical interface to allow packet reception. Alternately, both control nodes should be able to have the same MAC address configured on their interfaces, but the reception will be Ethernet by the device driver until the control node is ready to receive the packet. Should be physically impossible on the device. A failover would then simply allow reception on that device.
[0121]
When any processor joins a multicast group, its interface must be configured with a multicast MAC address, so multicast information must be shared between control nodes so that failover is transparent to that processor . Since the virtual network driver must keep track of multicast group membership anyway, this information is always available to the LAN proxy via the virtual LAN server as needed. Thus, receive failover will result in multicast group membership being queried from the virtual network driver to rebuild the local multicast group membership table. This operation is low overhead, requires no special processing except between failover and recovery, and does not require any special replication of data between control nodes. If the reception has failed over and the failed control node recovers, only the transmission will be transferred to the recovered control node. Thus, the recovery algorithm on the virtual network interface always moves the transmission to the recovered control node and leaves the receiving process where it is.
[0122]
Virtual service clusters may also use load balancing and failover.
[0123]
(Multi-cabinet platform)
Some embodiments allow the cabinets to be connected together to form a larger platform. Each cabinet will have at least one control node used for inter-cabinet connections. Each control node will include a virtual LAN server 335 to handle local connections and traffic. One of the servers is configured to be a master, such as one installed at a control node that has an external connection for the virtual LAN. Another virtual LAN server will act as a proxy server or slave so that local processors in those cabinets can participate. The master maintains all virtual LAN state and control, while the proxy relays packets between the processor and the master.
[0124]
Each virtual LAN server proxy maintains RVI to each master virtual LAN server. Each local processor will connect to the virtual LAN server proxy server as if it were the master. If the processor connects and registers the IP and MAC addresses, the proxy will register the IP and MAC addresses with the master. This will cause the master to bind its address to RVI from the proxy. Thus, the master will include RVI bindings for all internal nodes, but the proxy will only include node bindings for the same cabinet.
[0125]
When a processor at a location in a multi-cabinet virtual LAN sends an arbitrary packet to the virtual LAN server, the packet will be relayed to the master for processing. The master will then perform normal processing on the packet. As necessary for multicast and broadcast, the master will relay the packet to the proxy. The master will also relay unicast packets based on the destination IP address and registered IP address of the unicast packet on the proxy. Note that on the master, the proxy connection is very similar to a node with many configured IP addresses.
[0126]
(Network management logic)
While there is no operating system running on a processing node, such as booting or kernel debugging, the serial console traffic and boot image of that node are controlled by the processing node's kernel debugging software or the switch driver code in the BIOS. Sent to management software running on (not shown). From there, console traffic can again be accessed from the high speed external network 125 or through the management port of the control node. The boot image request can be satisfied from the local disk of the control node or from a partition on the external SAN 130. Preferably, the control node 120 is booted and operating normally before anything can be done on the processing node. The control node is itself booted or debugged from its management port.
[0127]
Some customers may want to limit controller boot and debugging to local access only by plugging their management port into the field computer when needed. Others choose to enable remote booting and debugging by establishing a secure network segment for management purposes that is properly insulated from the Internet by capping their management port Might do. Once the control unit has been booted and is operating normally, all other management functions for the control unit and for the rest of the platform are also fast externally from the management port if authorized by the administrator. It can also be accessed from the network 125.
[0128]
Serial console traffic to and from each processing node 105 is sent to management software running on the control node 120 by an operating system kernel driver on the switch structure 115. From there, the console traffic of any node can be accessed either through the normal high speed external network 125 or the management port of the control node.
[0129]
(Storage device architecture)
One embodiment follows the SCSI type of storage device. Each virtual PAN has its own virtual I / O space and also issues SCSI commands and status within such space. The control node logic translates or transforms addresses and commands as needed from the PAN and sends them to the SAN 130 that handles those commands. From the perspective of the SAN, the client is the platform 100, and the actual PAN that issued the command is hidden and anonymous. Since the SAN address space becomes virtual, one PAN running on platform 100 may have devices that number starting with device number 1, and another PAN may also have device number 1. . However, each of device numbers 1 will correspond to a different and unique part of the SAN storage device.
[0130]
Under the preferred embodiment, the administrator can create a virtual storage device. Each PAN will have its own independent view of mass storage. Thus, as described below, the first PAN may have a given device / LUN address map for the first location of that SAN, and another PAN may have a second different for that SAN. May have the same given device / LUN map for a location. Each processor maps device / LUN addresses to primary and secondary device numbers, for example, to identify disks and partitions. Their primary and secondary device numbers are considered physical addresses by the PAN and the processors within the PAN, but in fact they are treated by the platform as virtual addresses for the mass storage devices provided by the SAN. That is, the main and secondary device numbers of each processor are mapped to the corresponding SAN location.
[0131]
FIG. 6 illustrates the software components used to implement an embodiment storage architecture. Typically, the building block 605 that runs on the control node 120 is in communication with the external SAN 130. The management interface primitive 610 provides an interface to its constituent primitives 605 and is in communication with the IP network 125 and thus with the remote management logic 135 (see FIG. 1). Each processor 106 of system 100 includes an instance of processor-side storage logic 620. Each such instance 620 communicates via two RV1 connections 625 to the corresponding instance of the control node side storage logic 615.
[0132]
In short, the building block 605 and interface 610 allow the administrator to discover the portion of the SAN storage that is distributed to the platform 100 and to further subdivide a portion to a particular PAN or processor 106. responsible. Furthermore, the storage device placement logic 605 is also responsible for communicating SAN storage allocation to the control node side logic 615. The processor-side storage logic 620 is responsible for communicating storage device requests on the processor's internal interconnect 110 and storage structure 115 to the control node-side logic 615 via a dedicated RVI 625. The request will, under certain embodiments, include a virtual storage address and a SCSI command. The control node side logic identifies the corresponding real address for the SAN, and includes, but is not limited to, commands and protocols, such as Fiber Channel (Gigabit Ethernet with iSCSI is another typical concatenation), Responsible for receiving and processing such commands by converting them to the appropriate format for the SAN.
[0133]
(Structural basic elements)
The configuration primitive 605 determines which elements in the SAN 130 are visible to each individual processor 106. It provides a mapping function that translates device numbers used by the processor (eg, SCSI targets and LUNs) into device numbers that are visible to the control node through their attached SCSI and Fiber Channel I / O interfaces 128. In addition, it provides an access management function that prevents the processor from accessing external storage devices that are attached to the control node but are not included in the processor configuration. The model presented to a processor (and to system administrators and applications / users on that processor) has it's own mass storage device with each processor attached to an interface on that processor Show as if.
[0134]
Among other things, this functionality allows software on processor 106 to be easily moved to another processor. For example, in one embodiment, a control node via software (without physical re-cabling) may change the PAN configuration to allow a new processor to access the required devices. Thus, the new processor may be able to inherit the personality of another storage device.
[0135]
In some implementations, the control node appears as a host on the SAN, but alternative implementations allow the processor to operate as such.
[0136]
As outlined above, the configuration logic discovers SAN storage devices distributed to platform 100 (eg, during platform boot), and this pool is later allocated by the administrator. If discovery is triggered later, the control node performing the discovery operation compares the new appearance with the previous appearance. Newly available storage devices are added to a pool of storage devices that can be allocated by the administrator. The partition that makes it invisible is not allocated and is removed from the available pool of storage that can be distributed to the PAN. The partition that makes it invisible will be assigned a trigger error message.
[0137]
(Management interface basic elements)
The configuration primitive 605 allows management software to access and update information describing the device mapping between devices visible to the control node 120 and virtual devices visible to the individual processor 106. It also allows access to control information. The assignment can be identified by the processing node associated with the simulated SCSI disk identification, eg, by the name of the simulated controller, cable, unit, or logical unit number (LUN).
[0138]
Under certain embodiments, interface primitive 610 cooperates with its constituent primitives to collect and monitor information and statistics such as:
Total number of I / O operations performed
Total number of bytes transferred
The total number of read operations performed
Total number of write operations performed
Total amount of I / O time that was in progress
[0139]
(Processor side memory logic)
The processor side logic 620 of the protocol is implemented as a host adapter module that emulates the SCSI subsystem by providing a low level virtual interface to the operating system on the processor 106. The processor 106 uses this virtual interface to send SCSI I / O commands to the control node 120 for processing.
[0140]
Under embodiments that use redundant control nodes 120, the processing node 105 will include one instance of logic 620 per control node 120. Under certain embodiments, the processor refers to the storage device using physical device numbering rather than logical. That is, to identify the LUN, SCSI target, channel, host adapter, and control node 120 (eg, node 120a or 120b), the address is specified as the device name. As shown in FIG. 8, one embodiment places the target (T) and LUN (L) on the host adapter (H), channel (C), mapped target (mT), and mapped LUN (mL). Map.
[0141]
FIG. 7 shows an exemplary architecture for the processor side logic 720. The logic 720 includes a device type specific driver (eg, disk driver) 705, an intermediate level SCSI I / O driver 710, and wrapper and interconnect logic 715.
[0142]
The device type dedicated driver 705 is a conventional driver for which an operating system is prepared and associated with a specific device type.
[0143]
The intermediate level SCSI I / O driver 710 is a conventional intermediate level driver that is called by the driver 705 dedicated to the device type once the driver 705 determines that the device is a SCSI device.
[0144]
The wrapper and interconnect logic 715 is invoked by the mid-level SCSI I / O driver 710. This logic provides a SCSI subsystem interface, thereby emulating the SCSI subsystem. In one implementation using the Giganet structure, logic 715 obscures SCSI commands when needed, and causes the NIC to send packets to the control node via dedicated RVI to the control node, as described above. Is responsible for interacting with the Giganet and RCLAN interfaces. The header information for the Giganet packet is modified to indicate that this is a storage packet, described below in the context, and contains other information. Although not illustrated in FIG. 7, the wrapper logic 715 may use the RCLAN layer to support and utilize redundant interconnects 110 and structures 115.
[0145]
For embodiments using the Giganet structure 115, the RVI of connection 725 is assigned a virtual interface (VI) number from a range of 1024 available VIs. For the two endpoints to communicate, switch 115 is programmed with a bidirectional path between the pair (control node switch port, control node VI number) and (processor node 105 switch port, processor node VI number). Is done.
[0146]
A separate RVI is used for each type of message sent in one direction. Thus, there is always a pending receive buffer on each RVI for messages that can be sent from the other side of the protocol. In addition, since only one type of message is sent in one direction on each RVI, the receive buffer posted on each of the RVI channels is limited to the maximum message length that the protocol uses for that type of message. Can be appropriately sized. Under other embodiments, all possible message types are multiplexed on a single RVI rather than using two VIs. Protocols and message formats do not specifically require the use of two RVIs, and messages themselves have message type information in their headers so that they can be demultiplexed.
[0147]
One of the two channels is used to exchange SCSI command (CMD) and status (STAT) messages. Another channel is used to exchange buffer (BUF) and transfer (TRAN) messages. This channel is used to process the data payload of the SCSI command.
[0148]
The CMD message also includes control information, the SCSI command to be executed, and the virtual address and size of the I / O buffer for the node 105. The STAT message contains control information and a completion status code that reflects any errors that may occur while processing the SCSI command. The BUF message includes control information in the control node 120 and the virtual address and size of the I / O buffer. The TRAN message includes control information and is used to confirm the successful data transmission from the node 105 to the control node 120.
[0149]
The processor-side wrapper logic 715 examines the SCSI command that is sent to determine if the command requires data transfer and in which direction it is. By analysis, the wrapper logic 715 sets the appropriate flag information in the message header accordingly. The section describing the control node side logic describes how the flag information is used.
[0150]
Under certain embodiments of the present invention, the link 725 between the processor side storage logic 720 and the control node side storage logic 715 is not part of the SCSI protocol and is not communicated to the SAN 130. May be used to communicate control messages. Instead, these control messages are to be processed by the control node side logic 715.
[0151]
Protocol control messages are always generated on the processor side of the protocol and are connected to the control node side of the protocol in one of two virtual interfaces (VIs) that connect the processor side logic 720 to the control node side storage logic 715. Sent. The message header used for the protocol control operation is the same as the command message header, except that a different flag bit is used to identify the message as a protocol control message. The control node 120 performs the requested operation and corresponds to RVI with a message header that is the same as that used by the status message. This method does not require a separate RVI for protocol control operations that are rarely used.
[0152]
Under certain embodiments using redundant control nodes, the processor-side logic 720 detects an error from the issued command and reissues the command to another control node accordingly. This retry will be implemented in an intermediate level driver 710.
[0153]
(Control node side storage device logic)
Under certain embodiments, the control node side storage logic 715 is implemented as a device driver module. Logic 715 provides a device level interface to the operating system on control node 120. This device level interface is also used to access the building blocks 705. When this device driver module is initialized, it responds to protocol messages from all processors 106 of the platform 100. All configuration activities are introduced through the device level interface. All I / O activity is introduced through messages that are transmitted and received through interconnect 110 and switch structure 115. At control node 120 there will be one instance of logic 715 per processor node 105 (although it is simply shown as one box in FIG. 7). Under certain implementations, the control node side logic 715 can be configured via the FCP or FCP-2 protocol, or iSCSI, or other protocols that use the SCSI-2 or SCSI-3 command set across various media. Communicate with 130.
[0154]
As described above, the processor-side logic sets a flag in the RVI message header that indicates whether the data flow is associated with a command and, if so, in which direction. The control node side storage logic 715 receives the message from the processor side logic and then analyzes the header information to determine how to operate, eg, to allocate a buffer or the like. The logic further converts the address information contained in the message from the processor to a corresponding and mapped SAN address and issues a command to the SAN 130 (eg, via FCP or FCP-2).
[0155]
SCSI commands such as the TEST UNIT READY command that do not require a SCSI data transfer phase are sent by the processor-side logic 720 that sends a single command on the RVI used for the command message, and a single status message on the same RVI. Processed by the control node side logic to send back. More specifically, the processor side of the protocol builds a message with a standard message header, a new sequence number for this command, the desired SCSI target and LUN, the SCSI command to be executed, and a list size of zero. To do. The control node side of the logic receives the message, extracts the SCSI command information, and transmits it to the SAN 130 via the interface 128. After the control node receives the command completion callback, it uses the standard message header, the sequence number for this command, the status of the completed command, and optionally the request sense data if the command is completed with a check condition status It then builds a status message to the processor.
[0156]
A SCSI command, such as a READ command that requires the SCSI data transfer phase to transfer data from the SCSI device to the host memory, is one or more RDMA WRITE operations to the logic 715 on the control node side and the memory of the processor node 105T , And by a processor-side logic that sends a command message to a control node that responds with a single status message from the control node. More specifically, the processor side logic 720 stores the standard message header, the new sequence number for this command, the desired SCSI target and LUN, the SCSI command to be executed, and the data from the command. Construct a command message with a list of the regions of memory that are. The control node side logic 715 allocates a temporary storage buffer to store data from the SCSI operation while the SCSI command performs a task on the control node. The control node side logic 715 sends a SCSI command to the SAN 130 for processing, and after the command is completed, it sends data back to the processor 105 memory that has a sequence of one or more RDMA WRITE operations. It then builds a standard message header, a sequence number for this command, the status of the completed command, and a status message with optional REQUEST SENSE data if the command is completed with SCSI CHECK CONDITION status.
[0157]
A SCSI command, such as a WRITE command, that requires the SCSI data transfer stage to transfer data from the host memory to the SCSI device, is a single command message to the control node side logic 715, from the control node side logic 715 to the processor side logic. One or more BUF messages, one or more RDMA WRITE operations from processor-side storage logic to memory in the control node, one or more TRAN messages from processor-side logic to control node-side logic, and control node-side logic to processor Processed by processor-side logic 720 that sends a single status message to the side logic. The use of a BUF message to communicate the location of the control node's temporary buffer memory to the processor side storage logic, and the use of a TRAN message to indicate the completion of the RDMA WRITE data transfer Due to lack of READ ability. If the underlying structure supports RDMA READ operations, a different sequence of corresponding actions will be used. More specifically, the processor side logic 720 builds a CMD message with a standard message header, a new sequence number for this command, the desired SCSI target and LUN, and the SCSI command to be executed. The control node side logic 715 allocates a temporary storage buffer to store data from the SCSI operation while the SCSI command is executing a task on the control node. Next, the control node side of the protocol builds a BUF message with a standard message header, a sequence number for this command, and a list of areas of virtual memory used for temporary storage buffers on the control node. Next, the processor-side logic 720 sends data to the control node memory in a series of one or more RDMA WRITE operations. It then builds a TRAN message with a standard message header and a sequence number for this command. After the control node side logic sends the SCSI command to SAN 130 for processing and receives further command completion, the standard message header, the sequence number for this command, the status of the completed command, and the command is in CHECK CONDITION status If completed with, optionally construct a STAT message with REQUEST SENSE data.
[0158]
Under some implementations, the CMD message includes a list of areas of virtual memory in which data for the command is stored. The BUF and TRAN messages also include an index field that allows the protocol control node to send an individual BUF message to each entry in the region list in the CMD message. The processor side of the protocol responds to such a message by performing an RDMA WRITE operation for the amount of data described in the BUF message, followed by a TRAN message to indicate the completion of a single segment of data transfer. Will.
[0159]
The protocol between the processor side logic 720 and the control node side logic 715 enables scatter-gather input / output operations. This function allows the data contained in the I / O request to be read or written from several separate areas of virtual and / or physical memory. This allows multiple non-contiguous buffers to be used for the request on the control node.
[0160]
As mentioned above, the configuration logic 705 is responsible for discovering SAN storage devices distributed to the platform and for interacting with the interface logic 710 to allow administrators to suballocate storage devices to specific PANs. is there. As part of this assignment, the building block 705 creates and maintains a storage data structure 915 that includes information that identifies a match between the processor address and the actual SAN address. FIG. 7 shows such a structure. The match will be between the processing node and the simulated SCSI disk identification, eg, by the name of the simulated controller, cable, unit, or logical unit number (LUN).
[0161]
(Management logic)
Management logic 135 is used to interface to the control node software for supplying the PAN. Among other things, logic 135 allows the administrator to establish the PAN's virtual network topology, its visibility to external networks (eg as a service cluster), and the type of device on the PAN, eg bridge and routing Makes it possible to establish
[0162]
Logic 135 also interfaces with storage management interface logic 710 so that the administrator may define storage for the PAN during or after the initial assignment. Its configuration definition includes the storage agreement (SCSI to SAN) discussed above, and access control permissions.
[0163]
As described above, each PAN and each processor will have a personality defined by its virtual networking (including virtual MAC addresses) and virtual storage. To implement processor clustering, a structure that records such personality will be accessed by management logic, as described below. Furthermore, they will be accessed by an administrator as described above or with an agent administrator. For example, the agent may be used to reconfigure the PAN in response to certain events such as time or year, or in response to certain loads on the system.
[0164]
The processor operating system software includes serial console driver code to send console I / O traffic for the node through the Giganet switch 115 to management software running on the control node. From there, the management software can route any node's console I / O stream through the control node's management port (its low-speed Ethernet port and its emergency management port), or via the high-speed external network 125, as authorized by the administrator. Can be made accessible. Console traffic can be recorded for auditing and historical purposes.
[0165]
(Cluster management logic)
FIG. 9 illustrates the cluster management logic of an embodiment. The cluster management logic 905 accesses the data structure 910 that records the network information, such as the PAN network topology, MAC address assignment in the PAN, and the like. In addition, cluster management logic 905 also accesses a data structure 915 that records the storage device correspondence of the various processors 106. In addition, the cluster management logic 905 also accesses a data structure 920 that records free resources such as unassigned processors in the platform 100.
[0166]
In response to a processor error event or administrator command, the cluster management logic 905 can change the data structure to “move” the storage and network identity of a given processor to a new processor. In this way, the new processor “inherits” the personality of the previous processor. The cluster management logic 905 can be responsible for doing this to replace a new processor in the PAN to replace what is faulty.
[0167]
The new processor will inherit the previous processor's MAC address and behave like the previous one. When a new processor boots, the control node will communicate connectivity information and update connectivity information for non-failed processors as needed. For example, in some implementations, RVI connections for other processors are updated transparently; that is, software on other processors need not be involved in establishing a connection to a newly replaced processor. In addition, the new processor will inherit the storage support of the previous one, and thus will inherit the surviving state of the previous processor.
[0168]
Among other advantages, this allows a free pool of resources, including processors, to be shared across all platforms rather than across a given PAN. Thus free resources (which would be maintained as such to improve system reliability and fault tolerance) would be used more efficiently.
[0169]
When a new processor is "swapped" it will need to ARP again to learn the association between IP address and MAC address.
[0170]
(Option)
Since each Giganet port in switch structure 115 can support 1024 simultaneous virtual interface connections on it and keep them separated from each other with hardware protection, the operating system Application programs and node Giganet ports can be shared securely. This will allow direct connection of application programs without having to work through the entire stack of driver code. When doing this, the operating system call will establish a virtual interface channel and memory map its buffers and queues to the application address space. In addition, a library encapsulating the low-level details that interface to that channel will facilitate the use of such virtual interface connections. The library will automatically establish redundant virtual interface channel pairs and be able to manage sharing and failures between them without requiring any effort or awareness from the calling application. .
[0171]
The above embodiment emulated Ethernet internally for an ATM-like structure. This design may be modified to use an internal Ethernet structure that simplifies much of the architecture, for example, removing the need for emulation capabilities. If the external network communicates in response to ATM, another variation would use ATM internally without Ethernet emulation, and ATM would communicate externally to the external network when so addressed Would be able to. Another variation would enable ATM internally to the platform (ie without Ethernet emulation) and only external communications would be converted to Ethernet. This will streamline internal communication but will require emulation logic at the controller.
[0172]
Some embodiments deploy PANs based on software configuration commands. It will be appreciated that deployment will be based on program control. For example, more processors can be deployed under software management at the peak of operation for that PAN, or more or less storage space can be accommodated for that PAN under the control of the software algorithm. May be.
[0173]
The scope of the invention is not limited to the embodiments described above, but is defined by the appended claims; and these claims will encompass modifications and improvements of what has been described Will be recognized.
[Brief description of the drawings]
[0174]
FIG. 1 is a system diagram illustrating an embodiment of the present invention.
FIG. 2A is a diagram illustrating a communication link established in accordance with an embodiment of the present invention.
FIG. 2B is a diagram illustrating a communication link established in accordance with an embodiment of the present invention.
FIG. 2C is a diagram illustrating a communication link established in accordance with an embodiment of the present invention.
FIG. 3A is a diagram illustrating the network software architecture of an embodiment of the present invention.
FIG. 3B is a diagram illustrating the network software architecture of an embodiment of the present invention.
FIG. 4A is a flow diagram illustrating driver logic in accordance with an embodiment of the present invention.
FIG. 4B is a flow diagram illustrating driver logic in accordance with an embodiment of the present invention.
FIG. 4C is a flow diagram illustrating driver logic according to an embodiment of the present invention.
FIG. 5 illustrates a service cluster according to an embodiment of the present invention.
FIG. 6 illustrates the storage device software architecture of an embodiment of the present invention.
FIG. 7 illustrates the processor side storage logic of an embodiment of the present invention.
FIG. 8 illustrates storage address mapping logic of an embodiment of the present invention.
FIG. 9 illustrates the cluster management logic of an embodiment of the present invention.
[Explanation of symbols]
[0175]
105 processing node
115 Switch mechanism
120 control node
135 Management Logic
206 switch
208 switch
210 processor logic
214 Virtual switch logic
305 IP stack
310 Virtual network driver
320 Giganet driver
325 Giganet driver
335 Virtual LAN server
340 Virtual LAN proxy
350 ARP logic,
505 clusters
605 Storage device configuration logic
610 management interface,
615 Control node side storage device logic
905 cluster management logic,
910 Network data structure
915 Storage device data structure
920 Free resource data structure

Claims (58)

A method of emulating a switched Ethernet local area network in a platform having a plurality of computer processors, a switch structure, and a point-to-point link to the processors:
Providing Ethernet driver emulation logic to perform tasks on at least two computer processors;
Providing switch emulation logic to perform tasks on at least one of the computer processors;
Establishing a virtual interface between the switch emulation logic and each computer processor having Ethernet driver emulation logic that performs tasks on it to enable software communication therebetween, The interface establishes a virtual interface defining a software communication path from a computer processor to another computer processor via the switch structure;
Establishing a virtual interface between each computer processor having Ethernet driver emulation logic executing a task thereon and every other computer processor having Ethernet driver emulation logic executing a task thereon;
When the virtual interface is operating between one computer processor and another so as to satisfy a predetermined evaluation criterion, the Ethernet driver emulation logic of the one computer processor is in software communication between them. Unicast communication with other computer processors via a virtual interface defining the path; and
When the virtual interface is operating between one computer processor and another computer without satisfying a predetermined evaluation standard, the Ethernet driver emulation logic of the one computer processor is connected to the other computer. Unicast communication with the other computer processor via a virtual interface to the switch emulation logic that transmits the unicast communication to a processor;
A method comprising: The method of claim 1, wherein each of the computer processors having Ethernet driver emulation logic on which a task is performed is associated with a virtual MAC address, the MAC address being connected to an external network. The method, wherein the method is formed according to a rule for identifying one of the plurality of computer processors different from the MAC address of the computer. The method of claim 2, wherein the platform is connected to an external network via interface logic to communicate with an external network, the external network interface logic being associated with its own MAC address; And the method wherein the message is communicated on the external network using the MAC address of the external network interface logic. The method of claim 1, wherein a first computer processor uses a first virtual interface to unicast communication with a second computer processor, but the second computer processor The method of using a different virtual interface to communicate to a first computer processor. The method of claim 1, wherein each computer processor includes switch structure driver logic for communicating over a point-to-point link, which also includes a checksum capability, the Ethernet driver emulation logic. Includes a checksum capability, but if the switch structure driver logic has already checksummed the message, the method inhibits such checksum functionality. 6. The method of claim 5, wherein the switch structure driver logic implements a reliable communication protocol to ensure receipt of messages on the switch structure. The method according to claim 1, wherein the switch structure and the point-to-point link are arranged in a redundant configuration. The method of claim 1, wherein the Ethernet driver emulation logic broadcasts the message by sending a message to the switch emulation logic via a virtual interface, the switch emulation logic including a virtual interface. The method of receiving and cloning broadcast messages from and sending the cloned messages to other computer processors in the network. The method of claim 1, wherein the switch emulation logic defines and maintains computer processor membership to an emulated network. The method of claim 1, wherein the Ethernet driver emulation logic transmits a message that is larger than a maximum transfer unit (MTU) size. A system for emulating a switched Ethernet local area network:
With multiple computer processors;
A switch structure and a point-to-point link to the processor;
Virtual interface logic for establishing virtual interfaces across the switch structure and point-to-point links, each virtual interface defining a software communication path from one computer processor to another via the switch structure With virtual interface logic;
Ethernet driver emulation logic that performs tasks on at least two computer processors;
Switch emulation logic that performs tasks on at least one computer processor,
Logic to establish a virtual interface between the switch emulation logic and each computer processor having Ethernet driver emulation logic that performs tasks thereon to enable software communication therebetween;
An ethernet driver that receives a message from one of the virtual interfaces to a computer processor having ethernet driver emulation logic on which the task is executed and executes the task on it in response to address information associated with the message Each computer processor having logic for transmitting a message to another computer processor having emulation logic and Ethernet driver emulation logic executing a task thereon, and Ethernet driver emulation logic executing a task thereon Logic for establishing a virtual interface with every other computer processor;
Said switch emulation logic comprising:
With
When the virtual interface is operating so that the virtual interface satisfies a predetermined evaluation criterion, the Ethernet driver emulation logic passes through a virtual interface that defines a software communication path therebetween, and the virtual interface satisfies a predetermined evaluation criterion. If not, the switched Ethernet local area network that contains logic for unicast communication with another computer processor in the emulated Ethernet network is emulated via the switch emulation logic. A system to rate. 12. The system of claim 11, wherein each of the computer processors having Ethernet driver emulation logic on which a task is performed is associated with a virtual MAC address, and the MAC address connects the computer processor to an external network. The system formed according to a rule for identifying one of a plurality of computer processors different from a MAC address. 13. The system of claim 12, further comprising external network interface logic for communicating with an external network, wherein the external network interface logic is associated with its own MAC address and the switch emulation logic is Logic for sending a message to the external network interface logic for communication over an external network, such message using the MAC address of the external network interface logic, on the external network The system communicated in. 12. The system of claim 11, wherein the first computer processor uses a first virtual interface for unicast communication with a second computer processor, wherein the second computer processor is the first computer processor. A system that uses different virtual interfaces to communicate to one computer processor. 12. The system of claim 11, wherein each computer processor includes switch structure driver logic for communicating over the point-to-point link, the switch structure driver logic including checksum capabilities. In addition, when the Ethernet driver emulation logic includes a checksum capability and the switch structure driver logic performs a checksum of a message, a logic for suppressing a checksum function in the Ethernet driver emulation logic is provided. Including said system. 16. The system of claim 15, wherein the switch structure driver logic implements a reliable communication protocol to ensure receipt of messages across the switch structure. 12. The system according to claim 11, wherein the switch structure and the point-to-point link are arranged in a redundant configuration. 12. The system of claim 11, wherein the Ethernet driver emulation logic includes logic for broadcasting a message by sending the message to the switch emulation logic via a virtual interface, the switch emulation logic. The system includes broadcast logic for receiving and cloning broadcast messages from a virtual interface and transmitting the cloned messages to other computer processors in the network. 12. The system of claim 11, wherein the switch emulation logic includes logic for defining and maintaining computer processor membership to an emulated network. 12. The system of claim 11, wherein the Ethernet driver emulation logic includes logic for transmitting a message that is larger than a maximum transmission unit (MTU) size. A method of implementing an Address Resolution Protocol (ARP) on a computing platform with multiple processors:
Defining the topology of an Ethernet network to be emulated on the computing platform, including processor nodes and switch node nodes;
Assigning a set of processors from a plurality of processors to serve as the processor node;
Assigning a processor to act as the switch node;
Assigning a virtual MAC address to each processor node of the emulated Ethernet network;
Allocating virtual interfaces on the underlying physical network to provide direct software communication from every processor node to every other processor node, each virtual interface having a corresponding identification Steps and;
A processor node communicating an ARP request to the switch node, wherein the ARP request includes an IP address;
The switch node communicates an ARP request to all other processor nodes in the emulated Ethernet network;
A processor node, which is associated with the IP address that grants the switch node an ARP response that includes the virtual MAC address of the processor node associated with the IP address;
The switch node receives the ARP response and includes a virtual interface identification for a virtual interface that the processor node issuing the ARP request should use for subsequent communication with the processor node associated with the IP address. Modifying the ARP response;
ARP implementation method with The method of claim 21, wherein the underlying physical network is a point-to-point network connecting the plurality of processors. 22. The method of claim 21, wherein the subset of processors is organized as a cluster, one of the processors in the cluster is a load balancing processor node, and any processor in the cluster is A method in which the switch node modifies the ARP response so that upon granting an ARP request, the virtual interface identification for the load balancing processor node is included. The method of claim 21, wherein the switch node is in communication with an external IP network and the act of communicating an ARP response includes identifying the ARP response from a processor node of the platform. An Address Resolution Protocol (ARP) system:
A computing platform having a plurality of processors connected by an underlying physical network;
Logic that is executable on one of the processors and that defines the topology of an Ethernet network to be emulated on the computing platform, the topology comprising processor nodes and switch nodes When;
Logic for assigning a set of processors from the plurality to be executable on one of the processors to be a processor for acting as the processor node;
Logic that is executable on one of the processors and assigns a virtual MAC address to each processor node of the emulated Ethernet network;
Logic that is executable on one of the processors and assigns a virtual interface to the underlying physical network to provide direct software communication from each processor node to every other processor node. Logic with each virtual interface having a corresponding identification;
Each processor node having ARP request logic for communicating the ARP request to the switch node, wherein the ARP request includes an IP address;
The switch node including ARP request broadcast logic to communicate the ARP request to all other processor nodes in the emulated Ethernet network;
Each processor node with ARP response logic to determine whether a response is the processor node associated with the IP address in an ARP request, and if so, provide an ARP response to the switch node; Each processor node wherein the ARP response includes the virtual MAC address of the processor node associated with the IP address;
The switch node including ARP response logic, comprising: ARP response logic for modifying the ARP response to include receiving the ARP response and including a virtual interface identification for the ARP request node Switch node,
An Address Resolution Protocol (ARP) system. 26. The system of claim 25, wherein the underlying physical network is a point-to-point network connecting the plurality of processors. 26. The system of claim 25, wherein the subset of processors is organized as a cluster, one of the processors in the cluster is a load balancing processor node, and the switch node is an ARP from a processor node. Including logic to detect whether the response is from any processor in the cluster and, if so, to modify the ARP response to include the virtual interface identification for the load balancing processor node System. 26. The system of claim 25, wherein the switch node is in communication with an external IP network, and the processor node ARP response logic includes logic to identify that the ARP response is from a processor node of the platform. . A computer processing platform,
A plurality of computer processors connected to an internal communication network;
At least one control node communicating with an external communication network and an external storage device network having an external storage device address space, wherein the at least one control node is connected to the internal network, thereby At least one control node in communication with a plurality of computer processors;
Configuration logic that defines and establishes a virtual processing area network, which provides a set of corresponding computer processors from the plurality of processors and communications within the set of computer processors, Configuration logic having a virtual storage area with a defined correspondence to the address space of the storage network, and a virtual local area communication network that excludes the processor from the plurality of none
A computer processing platform comprising: The control node receives a communication message addressed to an entity on the external communication network via the internal communication network, and the control node provides a message related to the external communication network corresponding to the received message 30. The platform of claim 29, comprising logic. Logic for the control node to receive a communication message addressed to an entity on the platform via the external communication network, and for the control node to provide a message to the addressed entity corresponding to the received message 30. The platform of claim 29, comprising: 30. The platform of claim 29, wherein the computer processor and the control node include network emulation logic for emulating an Ethernet function on the internal communication network. The platform according to claim 32, wherein the internal communication network is a point-to-point exchange structure. 30. The platform of claim 29, wherein the internal communication network comprises redundant interconnections connecting the computer processor and at least one control node to a redundant switch structure. 35. The platform of claim 34, comprising at least one other control node connected to the interconnection to form a redundant control node. The defined response in the external storage device address space for the control node to receive a storage device message from the computer processor via an internal communication network and for the control node to extract an address from the received storage device message 30. The platform of claim 29, further comprising logic for identifying an address and for providing a message on an external storage device network corresponding to the received storage device message and having the corresponding address. For the control node to buffer data corresponding to writing a message received from a computer processor and to provide buffered data in the corresponding message provided to the external storage device network; 38. The platform of claim 36, comprising logic. The control node receives a storage device message from the external storage device network, the control node identifies a corresponding computer processor or control node to which the received message responds, and corresponds to the identified processor or control node 38. The platform of claim 36, including logic for providing a message to perform. A virtual processing area network deployment method:
A platform having a plurality of computer processors and at least one control node connected to an internal communication network, wherein the control node is in communication with an external communication network and an external storage device network having an external storage device address space. The act of providing,
Defining a corresponding set of computer processors for the virtual processing network;
Providing a communication within the set of computer processors, but establishing a virtual local area communication network excluding processors from a plurality not in the defined set;
An act of defining a correspondence between the virtual storage space of the virtual processing network having the defined one and the address space of the storage device network;
A virtual processing area network deployment method comprising: The control node receives a communication message addressed to an entity on the external communication network via the internal communication network, and the control node provides a message on the external communication network corresponding to the received message. Item 40. The method according to Item 39. 40. The control node receives a communication message addressed to an entity on the platform via the external communication network, and the control node provides a message to the addressed entity corresponding to the received message. The method described in 1. 40. The method of claim 39, wherein the computer processor and the control node emulate an Ethernet function on the internal communication network. 43. The method of claim 42, wherein the internal communication network is a, a point-to-point exchange structure, and the emulation of Ethernet functionality is provided across the internal point-to-point exchange structure. 40. The method of claim 39, wherein the computer processor communicates over a redundant interconnection connecting the computer processor and the at least one control node. 45. The platform of claim 44, comprising at least one other control node connected to the interconnection, forming a redundant control node. The control node receives a storage device message from the computer processor via the internal communication network, the control node extracts an address from the received storage device message, and a prescribed corresponding address in the external storage device address space 40. The method of claim 39, wherein the message is provided on an external storage network corresponding to the received storage message and having the corresponding address. 47. The control node buffers data corresponding to writing a message received from a computer processor and provides the buffered data in the corresponding message provided to the external storage device network. the method of. The control node receives a storage device message from the external storage device network, the control node identifies a corresponding computer processor or control node to which the received message responds, and a message corresponding to the identified processor or control node 47. The method of claim 46, wherein: A computer processing platform connectable to an external communication network and a storage device network,
A plurality of computer processors connected to an internal communication network;
Configuration logic for defining and establishing the following (a) and (b):
Each computer processor in the virtual local area communication network has a corresponding virtual MAC address, and the virtual local area network provides communication between a set of computer processors, The virtual local area communication network on the internal network, excluding processors from not a plurality,
Configuration logic for defining and establishing a virtual storage space having a defined contact to the address space of the storage device network;
Failover logic for allocating a computer processor from the plurality to replace the failed processor in response to a failure by the computer processor, the MAC address of the failed processor being assigned to the processor that replaces the failed processor Including logic to allocate, the virtual storage space of the failing processor and a defined correspondence to the processor to replace the failing processor, and the processor to replace the failing processor, and excluding the failing processor Failover logic including logic to re-establish the virtual local area network,
A computer processing platform comprising: The configuration logic establishes a virtual interface to define a software communication path in the processor of the virtual network, and the failover logic replaces the failed processor from the processor in the virtual network. 50. The platform of claim 49, including logic for establishing a virtual interface to the processor. 50. The platform of claim 49, wherein the configuration logic comprises: a second set of computer processors; and a second virtual storage space that maintains prescribed communication to the storage device network address space. A second virtual local area network is established, and the failover logic causes the processor to replace the failed processor to inherit the individuality of the virtual local area network and the virtual storage of the failed processor. Said platform. A computer processing method on a platform having a plurality of computer processors connected to an internal communication network comprising:
Each computer processor in the virtual local area communication network has a corresponding virtual MAC address, and the virtual local area network provides communication between a set of computer processors, but within its defined set Defining and establishing a virtual local area communication network on the internal network excluding the processor from a plurality of
Defining and establishing a virtual storage space having a specified contact to the address space of the storage device network;
Allocating the MAC address to the processor to replace the failed processor in the step of allocating a plurality of computer processors to replace the failed processor in response to a failure by the computer processor; Assigning the virtual storage space and the default communication of the failed processor to the processor to replace, and including the processor to replace the failed processor and excluding the failed processor from the virtual local Said step comprising re-establishing an area network;
The computer processing method comprising: When establishing a virtual local area network, a virtual interface is established to define a software communication path between the processors of the virtual network, and when a processor replaces a failed processor, the virtual interface 54. The method of claim 52, established for the processor to replace. A second virtual local area network is established by a second set of computer processors and a second virtual storage space having a defined contact with the storage device network address space, and if the processor fails, the failure 53. The method of claim 52, wherein the processor that replaces the processor inherits the virtual storage device personality of the virtual local area network and the failed processor. Each of the at least two computer processors includes logic to provide a service;
Cluster logic for receiving a request message for the service, wherein the message has an IP address and distributes the request to one of at least two computer processors having logic to provide the service Cluster logic for
A system for providing the service comprising: 56. The system of claim 55, wherein the logic for distribution includes logic for analyzing the source information in an input message in determining which processor should service the message. A method for providing a service addressed by an IP address comprising:
Including logic for servicing each of at least two computer processors;
Receiving a request message for the service, the message having an IP address, and further distributing the request to one of the at least two computer processors having logic to provide the service; about,
How to provide a service that has 58. The method of claim 57, wherein source information of the incoming message is analyzed to determine which processor will service the message.

Images