CN106576073B - Method and system for transmitting data through aggregated connections - Google Patents

Abstract

Methods and systems for processing data packets received at a first network node and processing encapsulation packets received at a second network node are disclosed. The first network node receives a data packet from its network interface. It then selects the first tunnel according to a selection policy and does not select or selects the at least one second tunnel. The Original Encapsulation Packet (OEP) is transmitted through the first tunnel to the second network node and the at least one Duplicate Encapsulation Packet (DEP) is transmitted through the at least one second tunnel. The second network node receives an encapsulated packet with a Global Sequence Number (GSN) over the aggregated connection. The second network node determines whether one or more data packets corresponding to the encapsulation packet have been previously received. Subsequently, the second network node may determine whether to forward the one or more data packets.

Description

Method and system for transmitting data over an aggregated connection

technical field

The present invention relates generally to the field of computer networks. More particularly, the present invention relates to methods and systems for handling data packets, encapsulated packets, and replicating encapsulated packets transmitted over an aggregated connection having multiple tunnels.

Background technique

A multi-wide area network (WAN) site-to-site VPN router is a router that supports aggregating the bandwidth of multiple interconnections (eg, WAN connections used to access one or more remote private networks). In some implementations, each TCP/IP session is routed to only one WAN. In this configuration, a single TCP file transfer session can only utilize the bandwidth of one WAN connection on each end. For example, in a session-based site-to-site virtual private network (VPN) connection, VPN traffic is routed to multiple WAN connections between two sites (eg, Site A and Site B).

In one embodiment, M x N tunnels are initially formed between the WAN connections, where M and N are the number of WAN network connections for Site A and Site B, respectively. Subsequently, application TCP/IP sessions are routed over different tunnels. However, it is worth noting that while a session-based site-to-site VPN can use different tunnels for different sessions, a single download session in this type of connection can only utilize one tunnel.

In wireless communications, the quality of packet transmission can be unpredictable, and the packet loss rate can change frequently. This degrades the quality of the entire package transfer. Even with a high per-tunnel bandwidth limit, the packet loss rate may not improve. A solution is needed to utilize multiple tunnels to improve the odds of successfully transferring data, which can be achieved by using replicated packets.

Contents of the invention

The present invention discloses a method and system for processing data packets received at a first network node. When the first network node receives a data packet from the network interface of the first network node, the first network node selects the first tunnel according to the selection strategy and also does not select or selects at least one second tunnel according to the selection strategy. Subsequently, the first network node transmits an original encapsulating packet (OEP) through the first tunnel. The OEP encapsulates the data packet, and each OEP has an original encapsulated packet global sequence number (OEP-GSN). OEP-GSN is stored in a field of each OEP. When at least one second tunnel is selected, the first network node also transmits at least one duplicate encapsulation packet (DEP) via the at least one second tunnel. At least one of the at least one DEP-encapsulated packets. Each of the at least one DEP has a duplicate encapsulating packet global sequence number (DEP-GSN) stored in a field of each of the at least one DEP.

The first tunnel and at least one second tunnel may be included in the aggregated connection. The selection policy is based on one or more of the following criteria: user selection, performance of multiple tunnels, service provider, usage restrictions, location, time of day, usage price, security, user identity, Internet protocol address range, communication protocol, communication Technologies, Applications and Devices. According to one of the embodiments, the performance of the first tunnel is determined to be better than the performance of the second tunnel, and the performance is substantially based on the delay and bandwidth of the tunnel.

According to one of the embodiments, the OEP-GSN of the OEP is the same as the DEP-GSN of at least one DEP.

According to one of the embodiments, said at least one DEP comprises a list of OEP-GSNs. The list of OEP-GSNs contains at least one OEP-GSN.

According to one of the embodiments, when multiple DEPs are transmitted for each OEP, each of the multiple DEPs is transmitted over a different tunnel of the aggregated connection.

The invention also discloses a method and system for processing at a second network node an encapsulated packet received from a first network node over an aggregated connection. The second network node receives the encapsulated packet through one of the tunnels of the aggregated connection. The encapsulation package can be OEP or DEP. When the encapsulating packet is OEP, it encapsulates a data packet. Alternatively, when the encapsulating packet is DEP, its encapsulation is substantially based on the packet information of the packet. The data packet may originate from the first network node, or be received by the first network node. The second network node determines whether the data packet has been previously received over the aggregated connection. This determination is essentially based on missing Global Sequence Number (GSN) records. When the second network node determines to forward the data packet, if the encapsulated packet is an OEP, the second network node decapsulates the data packet from the encapsulated packet. If the encapsulated packet is DEP, the second network node essentially recreates the data packet based on the data packet information. Subsequently, the second network node forwards the data packet to its destination. Indicates the destination in the header of the packet. Subsequently, the second network node may update the records of the missing GSNs.

According to one of the embodiments, when the encapsulation package is DEP and the data package information stores multiple encapsulated packages or error correction information, the encapsulation package includes a list of OEP-GSNs.

According to one of the embodiments, the second network node determines whether the GSN corresponding to the data packet is in the record of the missing GSN. If the GSN is not in the record of the missing GSN, the second network node determines not to forward the data packet, instead, if the GSN is in the record of the missing GSN, the second network node determines to forward the data packet.

According to one of the embodiments, the second network node may further update the expected global sequence number after determining whether to forward the data packet.

Description of drawings

FIG. 1A shows an overall system for optimizing the throughput of multiple variable bandwidth connections according to an embodiment of the present invention;

Figure 1B shows a network environment according to various embodiments of the present invention;

FIG. 1C illustrates a system 100 adapted in accordance with an embodiment configured to optimize throughput of bonded multiple variable bandwidth connections;

FIG. 2A shows a flowchart depicting a method for increasing throughput of a bonded connection according to an embodiment of the present invention;

FIG. 2B shows a flowchart depicting a method for increasing the throughput of a bonded connection according to an embodiment of the present invention;

Figure 3 is an example embodiment illustrating the types of information that may be encapsulated in transmitted IP packets according to an embodiment of the invention;

FIG. 4A is an example embodiment illustrating the types of information that may be encapsulated in a feedback packet according to an embodiment of the invention;

Figure 4B is a chart showing possible values for the fields of the feedback packet of Figure 4A;

Figure 5 depicts a block diagram of a processing system suitable for implementing the present invention;

FIG. 6 is a flowchart illustrating a process for transmitting encapsulated packets over an aggregated connection, according to various embodiments of the invention;

FIG. 7 is a flowchart illustrating a process for processing a received encapsulated packet;

Figure 8A illustrates the structure of an original encapsulation packet (OEP) according to various embodiments;

Figure 8B shows the structure of a duplicate encapsulation package (DEP) according to one of the embodiments of the present invention;

Figure 8C shows the structure of a DEP according to one of the embodiments of the present invention;

FIG. 9 is a flowchart showing a process for transmitting DEP according to one of the embodiments of the present invention;

Figure 10 is a flowchart showing processing according to one of the embodiments of the present invention;

FIG. 11 is a flowchart illustrating the process of processing encapsulated packets received over an aggregated connection;

FIG. 12A shows the contents of a DEP according to an exemplary embodiment;

Figure 12B shows the contents of a DEP according to one of the embodiments;

Figure 12C shows the contents of a DEP according to one of the embodiments; and

Fig. 13 is a flowchart illustrating a process for handling DEP according to one of the embodiments.

Detailed ways

FIG. 1A illustrates a system 101 adapted according to an embodiment configured to optimize throughput of bonded multiple variable bandwidth connections by adjusting tunnel bandwidth weighting patterns during a data transfer session. System 101 includes a plurality of sites 102 and 104 each including at least one network node. A network node may be referred to as a communication router. However, the scope of the invention is not limited to communication routers, so that the invention can be performed at a gateway, router, server or any other type of network node. For simplicity, FIG. 1A shows that site 102 includes communication router 106 and site 104 includes communication router 108 . Communications router 106 and communications router 108 may be implemented as multi-WAN routers that support aggregating the bandwidth of multiple Internet connections. Communication router 106 and communication router 108 are connected by network 110 . Network 110 may include a LAN, MAN, WAN, wireless network, PSTN, Internet, Intranet, Extranet, and the like.

Site 102 and router 106 may include M connections 112 and site 104 and router 108 may include N connections 114 . Connection 112 and connection 114 are data connections used to communicate information in network 110 between site 102 and site 104 . In the illustrated embodiment, M is equal to 3 and N is equal to 2; however, these values may vary depending on the desired router and configuration. Connection 112 and connection 114 may have similar or different bandwidth capabilities. Additionally, connection 112 and connection 114 may include different types of WAN connections, such as WiFi, cable, DSL, T1, 3G, 4G, satellite connections, and the like. It should also be noted that both site 102 and site 104 may be considered senders or receivers, and a discussion of the functionality of either site may be implemented at the other site. In other words, system 101 may be implemented as a symmetric network.

FIG. 1B shows a network environment according to one of the embodiments of the present invention. Tunnels 103A, 103B and 103C are established between communication router 106 and communication router 108 . Tunnels 103A, 103B, and 103C may be bonded to form an aggregated connection.

According to one of the embodiments, communications router 106 and communications router 108 may have multiple network interfaces. Communications router 106 establishes tunnels 103A, 103B, and 103C with one or more network interfaces of communications router 108 via one or more of its plurality of network interfaces.

Communications device 106 and communications device 108 may function as gateways, routers, switches, access points, hubs, bridges, and the like.

FIG. 1C illustrates a system 100 adapted in accordance with an embodiment configured to optimize throughput of a bonded plurality of variable bandwidth connections. System 100 is similar to system 101 except that there are M×N virtual tunnels 116 . When a bonded connection is created between site 102 and site 104, such as by implementing a bonded site-to-site VPN connection, MxN tunnels 116 may be created. Tunnel 116 corresponds to a unique arrangement of site 102's network connections and site 104's network connections. An aggregated connection may be formed between communications router 106 and communications router 108 . Tunnel 116 may be a virtual tunnel.

Multiple established tunnels 116 may be aggregated, bonded or bound together to form an aggregated connection. Those skilled in the art will appreciate that there are countless ways to aggregate, combine or bind multiple established tunnels to form one aggregated tunnel. An aggregated connection is considered a tunnel by the session or application that is using it. Aggregated connections can be peer-to-peer, virtual private network, or non-connection-oriented. For example, the aggregate connection can be a TCP connection or a UDP connection. In another example, the aggregated connection is an aggregate of multiple tunnels, and each tunnel is linked between communications router 106 and communications router 108 . In another example, the aggregated connection may be a VPN tunnel that includes multiple established tunnels, and each established tunnel is linked between communications router 106 and communications router 108 .

FIG. 2A shows a high-level flowchart depicting the operation of the system 100 of a method 200 for increasing throughput of a bonded connection. It should be understood that the specific functions, order of functions, etc., presented in FIG. 2A are intended as exemplary operations in accordance with the inventive concept. Thus, the concepts herein can be implemented in various ways other than those of the illustrated embodiments.

At block 201 of the illustrated embodiment, when a bonded connection is established between site 102 and site 104, such as by implementing a bonded site-to-site VPN connection, MxN virtual tunnels 116 as shown in FIG. 1C may be created . Virtual tunnel 116 corresponds to a unique arrangement of site 102's network connections and site 104's network connections.

At block 202 of the illustrated embodiment, a default weight for the tunnel is determined and/or assigned. To determine the default weights, an embodiment exchanges uplink bandwidth data and downlink bandwidth data for connection 112 and connection 114 between site 102 and site 104 . Using this bandwidth data, the default weights can be calculated according to the following method: Assume that the downlink bandwidth of connection 1 to connection m of site 102 is d1,d2...dm, and the uplink bandwidth of connection 1 to connection n of site 104 is The road bandwidth is U1, U2,...Un; the default weight of the tunnel between connection X of site 102 and connection Y of site 104 can be defined as DW(x,y), where DW(x,y)=dx .dy. Using the above method to calculate the default weight, if connection 112-1 to connection 112-3 are WAN connections of multi-WAN routers with corresponding uplink/downlink bandwidths of 10M/6M, 8M/4M and 6M/6M, and connection 114-1 to connection 114-2 are WAN connections of multi-WAN routers with corresponding uplink/downlink bandwidths of 7M/5M and 9M/3M, then the respective default weights for each tunnel will be as follows Show:

【Table 001】

site 102 site 104 DW(1,1)=6*7=42 DW(1,1)=5*10=50 DW(1,2)=6*9=54 DW(1,2)=5*8=40 DW(2,1)=4*7=28 DW(1,3)=5*6=30 DW(2,2)=4*9=36 DW(2,1)=3*10=30 DW(3,1)=6*7=42 DW(2,2)=3*8=24 DW(3,2)=6*9=54 DW(2,3)=3*6=18

Note that other ways to calculate default weights are contemplated, so the above are merely implementation examples of embodiments of the invention. Note that many different weighting schemes can be used to define the initial bandwidth of the tunnel. For example, it may be desirable to weight the tunnel in one direction using only the downlink capacity of the receiving site and the uplink capacity of the sending site. Any weighting scheme for characterizing the capacity of a tunnel when establishing a bonded connection may be used for the purposes of the present invention.

When routing packets from site 102 to site 104 according to an embodiment, the packets are assigned to tunnels in a ratio according to the effective weight EW(x,y). Initially, the effective weight of an embodiment is set equal to the default weight, EW(x,y)=DW(x,y), and is The best. However, if one or more tunnels lose packets while a user is downloading a file over a bonded network connection in a TCP session, the overall throughput for that session will drop significantly. To some extent, this is because packet loss will cause TCP to continue to retransmit. Although the tunnel without packet loss is not fully occupied, TCP's flow control will also maintain a low throughput.

An effective way to increase throughput is to avoid this packet loss. To this end, embodiments of the present invention identify at block 203 of the illustrated embodiment when a tunnel is experiencing an increase or decrease in packet loss rate. Embodiments also provide for modifying the effective weight of tunnels that are experiencing or have experienced a change in packet loss rate at block 204 . The packet loss rate information may be monitored continuously, or based on a specific time period. Once it is determined that the tunnel is experiencing an unacceptable rate of packet loss (block 204-1), the illustrated embodiment blocks at 204-2 the effective weight of the tunnel. In some embodiments, unacceptable may mean that the packet loss rate is a non-zero amount, while other embodiments may determine an unacceptable rate as any rate that exceeds a predefined threshold. Embodiments implement effective weight reduction in a step-by-step manner, in a continuous manner, with a one-time reduction proportional to the increase in packet loss rate, and the like. When reductions are made in a stepwise manner, embodiments may continue to monitor the tunnel in order to optimize the amount of reduction achieved.

Tunnels 116 may be established or monitored by sending heartbeat packets from router 106 or router 108 through each tunnel. In some embodiments, when the receiver fails to receive a heartbeat packet from a tunnel for a period of time, then it considers the tunnel closed and the tunnel will not be used for routing traffic. If heartbeat packets start being received again, the tunnel can be re-established and weighted along with the other tunnels. Thus, embodiments may utilize heartbeat packets to monitor and re-establish connections in the event that all packets are lost in a tunnel and the tunnel's effective weight is reduced to zero.

Furthermore, when a tunnel restores all or part of its respective bandwidth, e.g., it is determined that the packet loss rate has dropped (block 204-3), then the illustrated embodiment acts to increase the effective weight of such tunnel (block 204-4) so that Fully or more fully utilize bandwidth. Some embodiments increase the tunnel's effective weight using a predetermined step size until the exact effective weight is regained. Other embodiments may increase the effective weight in proportion to the remeasured bandwidth corresponding to the remeasured packet loss rate. Furthermore, embodiments may increase the effective weight of a tunnel based on a predetermined linear or exponential scale.

After the tunnel's effective weights are adjusted, or when it is determined that no adjustment is necessary, the system's weighting scheme is updated at block 205 of the illustrated embodiment. This updating may include storing any processed information, using such information in further processing, causing the system to take no action, and the like. For example, the processing performed with respect to block 205 may be operable to average the weighted patterns over a period of time, eg, to reduce errors associated with highly transient anomalies. In addition, the updated information can be used on the system 100 to modify packet allocations for data transfer sessions, as discussed with respect to FIG. 2B. The system 100 may continue to implement steps 203 to 205 continuously or periodically during the data transfer session.

FIG. 2B illustrates an embodiment where after weighting method 200 is implemented, packets are allocated based at least in part on the modified tunnel weights. Specifically, block 206 of the illustrated embodiment operates to distribute packets traversing the tunnel according to a weighting pattern determined by the operations of method 200 . In some embodiments, this allocation will change during the data transfer session, therefore, the steps of Figure 2B are shown as loops. Some embodiments change the packet allocation whenever the system is updated at block 205 . Additionally, block 205 may change thresholds, etc. such that changes are implemented periodically in response to certain loss rates. It should be appreciated that determining weights by the operations of method 200 and applying the determined weights to packet allocations at block 206 may have a different period. For example, the method 200 is operable to use relatively short iteration periods to provide updates of weighting pattern information, while using longer iteration periods to change the allocation of packets based on such weighting pattern information.

In order to monitor the bandwidth of each tunnel 116, some embodiments of the present invention enable each transmitted IP packet to be encapsulated with various information. FIG. 3 illustrates an example embodiment illustrating the types of information 300 that may be encapsulated in transmitted IP packets. The transmitted IP packets may be called encapsulated packets, ie original encapsulated packets or duplicate encapsulated packets. Details of the original and duplicate capsules are shown in Figures 8A, 8B and 8C. Version field 302 may contain information about the protocol version being used, and protocol type field 303 may contain the protocol type of the payload packet. Typically, the value of this field will correspond to the Ethernet protocol type of the packet. However, additional values can be defined in other documents. Tunnel ID field 304 may be a 32-bit field and may contain an identifier for identifying the current tunnel of the IP packet. Advanced Encryption Standard (AES) initialization vector field 306 may be a 32-bit field and may contain an initialization vector for AES encryption. The global sequence number field 308 may be a 32-bit field and may contain sequence numbers that are used to reorder the individual packets of the various sessions into the proper order as they emerge from their respective tunnels. A per tunnel sequence number field 310 may be a 32-bit field that may represent a sequence number assigned to each packet routed to a particular tunnel. AES encrypted payload field 312 may be used to transport the payload of the IP packet. For higher security of the payload, AES encryption can be applied to prevent attacks from third parties.

The per-tunnel sequence numbers discussed above can be used to monitor lost packets in the tunnel. In one embodiment, the router at the receiving end calculates the packet loss rate DR(x,y) of each tunnel every f seconds by monitoring the sequence number of each tunnel in the received packet. DR(x,y) can be characterized as missing sequence numbers divided by sequence number growth during time period f. The length of the time period f is variable, and in one embodiment f is equal to 5 seconds.

Other methods can also be used to monitor lost packets, for example: the sender can periodically notify the receiver how many packets it has sent, the sender sends a heartbeat packet to the receiver every constant period of time and the receiver can pass the monitoring Heartbeat packet loss rate to estimate the overall loss rate, by obtaining loss rate data from physical interface/device/layer, etc.

The receiving end can feed back the loss rate, effective weight, or other bandwidth indicator for a particular tunnel to the sending router. When the sender receives information about packet loss, some embodiments reduce the tunnel's effective weight EW(x,y) by EW(x,y)·DR(x,y). Other metrics can be used to modify the effective weight of the tunnel. In some embodiments, the sender may receive feedback and the effective weight may be reduced by an amount greater or less than the packet loss rate. This difference can be configured according to the specific needs of the communication system. The above example presents a metric that attempts to reduce the effective weight of the tunnel to a weight that maximizes the amount of bandwidth available to the tunnel while preventing further packet loss. Any metric that finds this balance may be preferred.

FIG. 4A illustrates an example embodiment of the type of information 400 that may be encapsulated in a feedback packet sent to a transmitting router to report packet loss or other bandwidth-related data received at a receiving router. Type field 402 may include data related to the type of data to be included in Data 1 field 404 and Data 2 field 406 . Data 1 field 404 and Data 2 field 406 may contain any information that may be used to assist the router in determining tunnel information regarding the number of tunnels, tunnel bandwidth, number of packets lost in the tunnel, and the like. An example of possible values for type field 402 in data fields 404 and 406 is shown in the diagram of FIG. 4B.

Information encapsulated in transmitted IP packets such as those shown in Figures 3 and 4 can also be used for packet buffering and reordering. Because the delay for each tunnel can be different, when two consecutive packets of the same TCP session are sent to the VPN peer through the bound VPN tunnel, since the two consecutive packets are routed via two different tunnels packets, so they may arrive out of order. If a TCP session receives out-of-sequence packets from the VPN, that TCP session will be slowed down due to TCP retransmissions. Therefore, the receiver SHOULD buffer packets arriving too early until slower packets arrive or the expiration time has passed. With this buffering, delayed packets arriving before the expiration time will be forwarded in-order to the target device. This buffering helps optimize end-to-end throughput.

Note that the embodiments described herein are sometimes discussed in the context of a VPN connection. These discussions are presented to illustrate example embodiments of bonded connections. The inventive concepts claimed herein are not limited to this connection. In fact, any connection in which sufficient data is available and exchanged to dynamically monitor the bandwidth of the multiple communication paths used in a data transfer session can be implemented with embodiments of the present invention.

As mentioned above, each packet may be assigned two different sequence numbers, a Global Sequence Number (GSN) and a Per Tunnel Sequence Number (PTSN). These numbers can be used to assist in packet buffering and reordering operations. After the packet is delivered to the upper layer, the receiver can update the next expected per-tunnel sequence number (NE-PTSN) and the next expected global sequence number (NE-GSN)

One method of how a packet may be buffered or forwarded to a target device after it has been received and decrypted will be described below.

1. If the packet's GSN is equal to 0, it is immediately forwarded to the target device.

2. Check if the PTSN of the packet is equal to NE-PTSN. If not, all packets with a GSN smaller than the GSN of the packet are dequeued (forwarded to the target device) in order. Leave said packet unprocessed.

3. Update NE-PTSN (ie, set NE-PTSN to PTSN+1).

4. If GSN is less than NE-GSN, forward to target device.

5. If the GSN of the packet is equal to the NE-GSN, update the NE-GSN (ie set NE-GSN to GSN+1) and forward to the target device. If the GSN of the header is equal to the new NE-GSN, the NE-GSN is updated repeatedly and the buffered header is dequeued from the buffer.

6. Otherwise (GSN is greater than NE-GSN), enqueue the packet in the order of GSN.

7. If the packet is in the queue for longer than a fixed amount of time, set the NE-GSN to the packet's GSN+1 and dequeue the packet and all packets with a GSN less than the packet's GSN in order.

Thus, the encapsulated packet information discussed in FIGS. 2 and 3 may include information that optimizes the tunnel bandwidth by assisting in the optimization of tunnel bandwidth in response to monitoring packet loss rates and by assisting in the efficient ordering of packets received in a data transfer session. The overall throughput of a data transmission system (eg, system 100).

FIG. 5 illustrates an exemplary processor-based system 500 that may be used to implement systems, apparatuses, and methods according to certain embodiments. Processor-based system 500 may represent the architecture of communications router 106 and communications router 108 . A central processing unit (CPU) 501 is coupled to a system bus 502 . CPU 501 may be any general purpose CPU, or may be a special purpose CPU designed to implement the above teachings. The present disclosure is not limited by the architecture of CPU 501 (or other components of exemplary system 500), so long as CPU 501 (and other components of system 500) support the inventive operations described herein. CPU 501 can execute various logic instructions described herein. For example, CPU 501 may execute machine-level instructions according to the exemplary operation flow described above in conjunction with FIG. 2 . When executing instructions representing the operational steps shown in FIG. 2, CPU 501 becomes a special-purpose processor of a special-purpose computing platform specifically configured to operate in accordance with various embodiments of the teachings described herein.

System 500 also includes random access memory (RAM), which may be SRAM, DRAM, SDRAM, or the like. The RAM 503 may be an auxiliary memory storing program instructions executable by the CPU 501 . System 500 includes read only memory (ROM) 504 which may be PROM, EPROM, EEPROM, or the like. RAM 503 and ROM 504 hold user and system data and programs, as is known in the art.

System 500 also includes input/output (I/O) adapter 505 , communications adapter 511 , user interface adapter 508 , and display adapter 509 . In some embodiments, I/O adapter 505, user interface adapter 508, and/or communication adapter 511 may enable a user to interact with system 500 to enter information.

I/O adapter 505 connects storage device 506 , such as one or more of a hard disk drive, compact disk (CD) drive, floppy disk drive, tape drive, etc., to system 500 . For storage requirements associated with performing the operations discussed in the above embodiments, storage devices are utilized in addition to the RAM 503 . Communications adapter 511 is adapted to couple system 500 to a network 512, which may enable communications via such network 512 (e.g., the Internet or other wide area network, local area network, public or private switched telephone network, wireless network, any combination of the foregoing) to System 500 inputs information and/or outputs information from system 500 .

Communications adapter 511 may be considered a network interface, and system 500 may include multiple communications adapters 511 . User interface adapter 508 couples user input devices (eg, keyboard 513 , pointing device 507 , and microphone 514 ) and/or output devices (eg, speakers 515 ) to system 500 . The display adapter 509 is driven by the CPU 501 to control display on the display device 510 . Display adapter 509 transmits instructions for switching or manipulating the state of various numbers of pixels used by display device 510 to visually present desired information to a user. Such instructions include instructions for changing the state from on to off, setting a specific color, intensity, duration, and the like. Each such instruction constitutes a rendering instruction that controls how and what is displayed on the display device 510 .

6 is a flow diagram illustrating a process for transmitting encapsulated packets from communications router 106 to communications router 108 over an aggregated connection, according to various embodiments of the invention. An aggregated connection can include multiple tunnels. In step 601, the CUP 501 of the communications router 106 selects a tunnel among multiple tunnels for transmitting an original encapsulation packet (OEP). The selected tunnel may be referred to as the first tunnel. Communications router 106 then transmits the OEP through the first tunnel. In step 602, the CPU 501 selects another tunnel among the plurality of tunnels for transmitting a duplicate encapsulation packet (DEP). The tunnel selected in step 602 may be referred to as a second tunnel. Communications router 106 then transmits the DEP through the second tunnel.

In a variant, the process of Figure 6 may be performed periodically. This processing can preferably be performed every few seconds. Instead, the process of FIG. 6 is performed upon receipt of a trigger. For example, a trigger is received when the CPU 501 determines that an acknowledgment has not been received or received delayed through the first tunnel and/or the second tunnel. This may indicate that the latency of the first tunnel and/or the second tunnel has become very high. Subsequently, the CPU 501 executes the steps of FIG. 6 again to select the first tunnel and the second tunnel again. In another example, the trigger may be received when the process of FIG. 6 is manually initiated by a user or administrator.

The first tunnel and the second tunnel may be selected according to a selection policy. In a variant, the selection strategy may be based on one or more of the following criteria: tunnel performance, service provider, usage restrictions, location, time, usage price, security, user identity, Internet protocol address range, communication protocol, Communication technologies, applications and devices. When selecting a policy based on tunnel performance, it can be based on factors such as throughput, error rate, packet delay, packet jitter, symbol jitter, quality of service, bandwidth, bit error rate, packet error rate, frame error rate, packet loss rate, queuing Performance metrics such as delay, cycle time, capacity, signal level, interference level, bandwidth delay product, switching delay time, signal-to-interface ratio, and signal-to-noise ratio are used to perform the selection.

A user or administrator can configure this selection policy. For example, a user may configure the selection policy to be based only on the bandwidth of each of the plurality of tunnels. Therefore, the tunnel with the best bandwidth is selected as the first tunnel through which the OEP is transmitted. Another tunnel with the next best bandwidth is chosen as the second tunnel over which to transport the DEP. In another embodiment, a user may configure the selection strategy to be based only on the delay of each of the plurality of tunnels. Therefore, the tunnel with the lowest delay is selected as the first tunnel through which the OEP is transmitted. Another tunnel with the next lowest latency is chosen as the second tunnel over which to transport the DEP.

In another example, a user may configure selection policies to be based on usage limits and packet loss rates. The tunnel with the highest usage limit and/or lowest packet loss rate is chosen as the first tunnel. The tunnel with the next highest usage limit and/or the next lowest packet loss rate is selected as the second tunnel.

In another variant, the selection strategy is based on user selection. A user or administrator may configure communications router 106 to use a certain tunnel as the first tunnel and another tunnel as the second tunnel.

A user or administrator can configure the selection policy of the communications router 106 by sending the configuration locally or remotely via a web interface, application programming interface (API), command line interface, or console.

Those skilled in the art will appreciate that in wireless communications, packet transmission quality may be unpredictable, and delays may be high in some cases. There may also be a higher rate of packet loss. Transmitting the DEP increases the chances that communications router 108 will receive the data because the OEP may be lost. However, bandwidth usage can be significantly improved through the transmission of DEP. DEP can also be used for forward error correction (FEC), which will be explained in more detail below.

Referring to FIGS. 8A , 8B, and 8C, types of information contained in the OEP 800 and the DEPs 810 and 820 may be similar to those shown in FIG. 3 .

The encapsulation package encapsulates the data packet. Therefore, when a data packet is encapsulated by an encapsulation packet, the data packet becomes an encapsulation packet. Data packets may be received by communications router 106 through one or more of its network interfaces from hosts or nodes in site 102 accessible through communications router 106 . The hosts or nodes may be located in the LAN of communications router 106 . The source and destination of the data packet can be indicated in the header of the data packet.

Each encapsulation packet may be assigned two sequence numbers, a global sequence number and a per-tunnel sequence number. Global sequence numbers can be used to aid in packet buffering and reordering operations. When the package is assigned a global serial number, the designated host or node arranges the package arriving at the designated host or node according to the global serial number corresponding to the package. Each tunnel sequence number indicates which of the multiple tunnels the encapsulated packet is transmitted through.

Figure 8A shows the structure of an OEP according to various embodiments. OEP 800 includes IP header 801 , original encapsulated packet global sequence number (OEP-GSN) 802 , per-tunnel sequence number 803 , other information 804 and encapsulated packet 805 . The OEP-GSN 802 stores the OEP-GSN of the OEP 800 . The IP header 801 may include a version field 302 and a protocol type field 303 . Each tunnel sequence number 803 may be optional. In one variation, each tunnel sequence number 803 and tunnel ID field 304 can be used to compare tunnel performance and reallocate packets based on tunnel performance. In another variation, when the per-tunnel sequence number 803 and tunnel ID fields 304 are not included, it may be necessary to maintain a database to store which tunnel is used to transport the encapsulated packets of which GSN. Tunnel performance can be compared by performing a lookup against the database. Thus, including each tunnel sequence number 803 and tunnel ID field 304 can assist in determining delays without the need to maintain and lookup a database of all GSNs. Other information 804 may include an AES initialization vector field 306 and a tunnel ID field 304 . In one variation, the OEP-GSN 802 , per-tunnel sequence number 803 and other information 804 are included in the options field of the IP header 801 and the encapsulated packet 805 is included in the AES encrypted payload field 312 . The advantage of including the OEP-GSN 802, per-tunnel sequence number 803 and other information 804 in the optional fields of the IP header 801 is that the information in these fields can be obtained without decapsulating the OEP 800, which can significantly reduce processing time. However, since routers and devices through which the OEP 800 passes may change the IP header 801 when the OEP 800 is tunneled, information in the optional fields of the IP header 801 may be lost if they are included in these fields. Therefore, instead, OEP-GSN 802, per-tunnel sequence number 803, other information 804, and encapsulated packet 805 can all be included in AES encrypted payload field 312, so that when OEP 800 passes through routers and Information will not be lost during installation.

FIG. 8B shows the structure of a DEP according to one of the embodiments of the present invention. DEP 810 includes IP header 811 , duplicate encapsulated packet global sequence number (DEP-GSN) 812 , per-tunnel sequence number 813 , other information 814 and encapsulated packet 815 . The IP header 811 may include a version field 302 and a protocol type field 303 . Each tunnel sequence number 813 and other information 814 may be optional. Other information 814 may include AES initialization vector field 306 and tunnel ID field 304 . In one variation, the DEP-GSN 812 , per-tunnel sequence number 813 and other information 814 are included in the options field of the IP header 811 , and the encapsulated packet 815 is included in the AES encrypted payload field 312 . Instead, the DEP-GSN 812 , per-tunnel sequence number 813 , other information 814 and encapsulated packet 815 are all included in the AES encrypted payload field 312 . Since DEP 810 includes the same encapsulated packets as OEP 800, DEP 810 can be used to recreate the information in OEP 800 in the event that OEP 800 is lost, lost, or delayed during transmission, and DEP 810 is received before OEP 800 is received .

OEP-GSN 802 and DEP-GSN 812 may be identical. Encapsulated packets 805 and 815 may be identical, such that OEP 800 and DEP 810 encapsulate the same data packet, which becomes encapsulated packets 805 and 815, respectively. When communications router 106 generates DEP 810 for transmission, its payload (ie, encapsulated packet) is the same as that of OEP 800 . Since OEP 800 and DEP 810 have the same content, their global serial numbers are also the same.

FIG. 8C shows the structure of a DEP according to one of the embodiments of the present invention. DEP 820 includes IP header 821 , DEP-GSN 822 , per-tunnel sequence number 823 , list of m OEP-GSNs 826 , other information 824 and encapsulated packet 825 . Since the payload of DEP 820 contains information of the last few OEPs that have been transmitted over the aggregated connection, it can be used to recreate missing encapsulated packets corresponding to OEPs. This is illustrated in more detail in Figures 12A, 12B and 12C.

DEP 820 may be transmitted periodically. In one variation, the time period between transmissions of each DEP 820 may be predefined. When the number of OEPs transmitted within the time period is n, the DEP 820 may include m encapsulated packets corresponding to n OEPs, where m is less than or equal to n, and the value of m is at least 1. The m encapsulated packets are the same as the encapsulated packets in m of the n OEPs. m encapsulated packets such as encapsulated packets 825 - 1 , 825 - 2 through 825 -m may be included in the AES encrypted payload field 312 of the DEP 820 . The list of m OEP-GSNs 826 includes the OEP-GSNs of the OEPs corresponding to the m encapsulated packets. For example, 10 OEPs are transmitted in a predefined time period. The first DEP 820 may include 6 encapsulated packets corresponding to 6 OEPs among the 10 OEPs, and the second DEP 820 may include 4 encapsulated packets corresponding to the remaining 4 OEPs. After a predefined period of time, the first DEP 820 and the second DEP 820 are transmitted. The first DEP 820 and the second DEP 820 can transmit through the same tunnel or different tunnels. Alternatively, the first DEP 820 may include 10 encapsulated packets respectively corresponding to 10 OEPs. After a predefined period of time, the first DEP 820 is transmitted. DEP 820 cannot include more than 10 encapsulated packets, since each encapsulated packet must correspond to each OEP.

In another variant, the number of OEPs before transmitting a DEP 820 is predefined. Therefore, the predefined number is n. DEP 820 may include m encapsulated packets corresponding to n OEPs, where m is less than or equal to n. The m encapsulated packets may be the same as the encapsulated packets in m of the n OEPs. m encapsulated packets such as encapsulated packets 825 - 1 , 825 - 2 through 825 -m may be included in the AES encrypted payload field 312 of the DEP 820 . For example, when n is defined as 5, after every 5 OEPs are transmitted, the DEP 820 is transmitted, and the DEP 820 includes 5 encapsulated packets corresponding to the 5 OEPs. The time period between transmissions of DEP 820 may not be predefined. The list of m OEP-GSNs 826 includes the OEP-GSNs of the 5 OEPs corresponding to the encapsulated packets. Alternatively, when n is defined to be 5, the first DEP 820 and the second DEP 820 are transmitted after every 5 OEPs are transmitted. The first DEP 820 may include encapsulated packets corresponding to 2 OEPs of the 5 OEPs, and the second DEP 820 may include encapsulated packets corresponding to the remaining 3 OEPs of the 5 OEPs.

DEP 820 can be used to correct any errors that may have occurred while transmitting the n OEPs, since encapsulated packets 825 contain information from the data packets encapsulated in the OEPs. In a variation, the DEP-GSN 822, the sequence number per tunnel 823, the list of m OEP-GSNs 826, and other information 824 may be included in the options field of the IP header 821, and the encapsulated packet 825 included in the AES encrypted valid In the payload field 312. Instead, a DEP-GSN 822 , a per-tunnel sequence number 823 , a list of m OEP-GSNs 826 , other information 824 , and an encapsulated packet 825 are all included in the AES encrypted payload field 312 . It is possible that one or more of the OEPs transmitted over the aggregated connection have been lost, lost or delayed. Even when an OEP is received, there may be errors in the OEP. DEP 820 can be used to recreate the encapsulated packets in the OEP, and also to check for any errors. Details regarding the transport of DEP 820 and the reconstruction of encapsulated packets are shown in Figures 12A, 12B and 12C.

OEP 800 is transported through a different tunnel than the tunnel used to transport DEP 810 or DEP 820 . Therefore, each tunnel serial number 803 may be different from each tunnel serial number 813 or each tunnel serial number 823 . If OEP 800 and DEP are transmitted over the same tunnel, and the performance of that tunnel is degraded, it is possible that neither OEP 800 nor DEP can be successfully received by communications router 108 . Therefore, to increase the chance of at least one of OEP 800 or DEP being received, OEP 800 is transmitted over a different tunnel than the tunnel used to transmit DEP 810 or DEP 820 .

The source address indicated in IP headers 801 , 811 , and 812 may be the IP address of one of communications router 106 's network interfaces. The destination address indicated in IP headers 801 , 811 , and 812 may be the IP address of one of the network interfaces of communications router 108 . The source address of encapsulated packets 805 , 815 , and 825 may be the IP address of a host or node in site 102 accessible through communications router 106 . The destination address of encapsulated packets 805 , 815 , and 825 may be the IP address of a host or node in site 104 accessible through communications router 108 .

OEP-GSN 802 , DEP-GSN 812 , and DEP-GSN 822 may correspond to global sequence number field 308 . Each tunnel sequence number 803 , 813 , and 823 may correspond to each tunnel sequence number field 310 .

Because encapsulated packets may arrive out of order, one of the purposes of encapsulating data packets within an encapsulated packet is to reorder data packets when they are received at the other end of the aggregated connection. The data packet can also have a different protocol and can be encapsulated in an encapsulation packet to meet the protocol requirements of the aggregated connection.

FIG. 9 is a flowchart illustrating a process for transmitting a DEP according to one of the embodiments of the present invention. The CPU 501 of the communication router 106 determines in step 901 the number of DEPs to be transmitted. In step 902, the CPU 501 determines which tunnel(s) have not been used for transmitting OEP or DEP, and selects the determined tunnel for transmitting DEP. In step 903, CPU 501 determines whether the bandwidth limit of the selected tunnel is approaching, and in step 904 communications router 106 does not transmit DEP through the tunnel that is approaching its bandwidth limit. If the bandwidth limit is not approached, then in step 905 communications router 106 transmits the DEP through the selected tunnel. In step 906, the process ends. Monitoring when bandwidth limits are approaching can help reduce congestion and prevent tunnel flooding. Those skilled in the art should know that as the bandwidth limit of the tunnel approaches, more and more packets may be lost, and the performance of the tunnel is degraded. Therefore, it is preferable to stop transmitting DEP over a tunnel that is approaching its bandwidth limit.

According to one of the embodiments of the present invention, communication router 106 may transmit multiple DEPs for one OEP. Each of the multiple DEPs travels through a different tunnel. The number of DEPs to be transmitted may be defined by a user or administrator of communications router 106 . However, the number of DEPs being transmitted cannot be higher than the number of tunnels available for transporting DEPs. This is because it may not be beneficial to tunnel multiple DEPs per OEP. When the performance of the tunnel is degraded, most of the plurality of DEPs may not be successfully received. Therefore, this consumes more bandwidth unnecessarily without increasing the chance of successfully receiving at least one of the DEPs. When the user or administrator defines the number of DEPs to be higher than the number of tunnels available to transmit DEPs, communications router 106 does not transmit the number of DEPs defined by the user or administrator. Instead, communications router 106 transmits a number of DEPs equal to the number of available tunnels, ie, one DEP per tunnel. At least one tunnel may be reserved for transporting OEP, and the remaining tunnels may be used for transporting DEP. For illustration purposes, the user defines the number of DEPs to be transmitted as five. The communications router 106 has only four tunnels established with the communications router 108, namely a first tunnel, a second tunnel, a third tunnel and a fourth tunnel. Each OEP is transmitted through the first tunnel. One DEP corresponding to each OEP is transmitted through each of the second tunnel, the third tunnel, and the fourth tunnel, respectively. Therefore, only 3 DEPs are transmitted for each OEP instead of 5.

In one example, if a user or administrator defines the number of DEPs to be less than the number of available tunnels, the DEPs may be transmitted according to the capabilities of the tunnels. For illustration purposes, the user defines the number of DEPs to be transmitted as two. The communications router 106 has four tunnels established with the communications router 108 , namely, a first tunnel, a second tunnel, a third tunnel, and a fourth tunnel. OEP is transmitted through the first tunnel with the best performance. If the second tunnel and the third tunnel have better performance than the fourth tunnel, then one DEP is transmitted through the second tunnel and the other DEP is transmitted through the third tunnel.

In another example, when there are more than two tunnels available, the OEP is transmitted through the first tunnel with the best performance. DEP is transported through the remaining tunnels using load balancing techniques. Alternatively, DEPs can also be transmitted through the remaining tunnels in a round-robin manner, especially when the number of DEPs to be transmitted is less than the number of remaining tunnels.

FIG. 7 is a flowchart illustrating a process for processing a received encapsulated packet. First, the existing processing environment is a processing environment in which an aggregated connection including a plurality of tunnels is established between the communication router 106 and the communication router 108, and the communication router 106 sends the encapsulated packet to the communication router 108 through the aggregated connection. Communications router 108 receives the encapsulated packet from communications router 106 over the aggregated connection. The encapsulated packet may include an encapsulated packet destined for a host or node accessible through communications router 108 .

Communication router 108 may receive a plurality of encapsulated packets encapsulated in OEP and/or DEP. OEP may arrive earlier than DEP and vice versa. Therefore, it is necessary to know whether an encapsulated packet, whether encapsulated in OEP or DEP, has been received before.

In step 701 , communications router 108 receives an encapsulated packet from communications router 106 . In step 702, the CPU of communications router 108 determines whether an encapsulated packet encapsulated in an encapsulated packet has been previously received. If the global sequence number of the encapsulated packet is the same as another global sequence number of another encapsulated packet that has been received before, it is determined that the encapsulated packet has been received before. If an encapsulated packet has been previously received, then in step 703, communications router 108 does not forward the encapsulated packet inside the encapsulated packet to the destination, and discards the encapsulated packet. If the encapsulated packet has not been received before, then in step 704, the encapsulated packet is forwarded to the destination. In step 705, the process ends. When it is determined that the encapsulated packet has been received before, it may be assumed that the encapsulated packet has been forwarded. For this reason, the communication router 108 does not forward the encapsulated packet in the encapsulated packet again, and discards the encapsulated packet.

The encapsulation package can be OEP 800 or DEP 810. Referring to step 702, there may be various methods of determining whether an encapsulated packet has been received. As mentioned above, one way to be sure is by checking the GSN of the encapsulating packet. Communications router 108 determines that an encapsulated packet has been received if any other encapsulated packet with the same global sequence number has been previously received over the same aggregated connection. Subsequently, there is no need to forward that same encapsulated packet.

Another way to determine if an encapsulated packet has been received is by checking the hash code. When the encapsulated packet arrives at communications router 108, the CPU of communications router 108 may apply a hash function to the payload (ie, the encapsulated packet) after decapsulating the encapsulated packet. A hash code generated by applying a hash function may be stored in a storage medium. The storage medium may be a local storage medium (eg, RAM 503 ) or a remote storage medium (eg, a remote server). When applying the hash function to the same encapsulated packet that has been received before, the hash code will be the same. The CPU of communications router 108 may then determine whether the generated hash code is unique, or whether it has been previously stored. If the hash code is unique, it is determined that the encapsulated packet has not been received before. If the hash code has been previously stored, it is determined that the encapsulated packet has been received. Although the hash function or seed can change, it is more likely to remain the same for a few minutes. Therefore, the uniqueness of the hash code can be used to determine whether an encapsulated packet has been received within the past few minutes.

FIG. 10 is a flowchart showing processing performed at communication router 108 when communication router 108 receives an encapsulated packet from communication router 106 . The encapsulated packet is assigned a global sequence number and is then transmitted by communications router 106 to communications router 108 over the aggregation connection. The encapsulation package can be OEP 800 or DEP 810.

In step 1000, communications router 108 receives an encapsulated packet assigned a first global sequence number (GSN-1). In step 1001, the CPU of communications router 108 determines an expected global sequence number (E-GSN). Then, in step 1002, GSN-1 is compared with E-GSN. E-GSN can be used to estimate what the GSN of the next encapsulated packet should be.

If it is determined in step 1002 that E-GSN is larger than GSN-1, it is considered that the encapsulated packet has arrived late. In step 1003, the CPU of communications router 108 determines whether GSN-1 is identified in the record for the missing GSN. If GSN-1 is identified in the missing GSN's record, then in step 1005, the missing GSN's record is updated to remove GSN-1 from the missing GSN's record. Communication router 108 then decapsulates the encapsulated packet to obtain the encapsulated packet, and in step 1009 forwards the encapsulated packet to its destination. The destination may be a host or node in site 104 and accessible through communications router 108 . Alternatively, if GSN-1 is not identified in the missing GSN's record, then in step 1004, the encapsulated packet is discarded and not forwarded to the destination. An encapsulated packet is lost because it is considered to contain an encapsulated packet that has been received before. Since GSN-1 is not found in the missing GSN's record, this indicates that another encapsulated packet with the same GSN-1 has been received before and may have been forwarded to the destination.

In step 1002, if E-GSN is not determined to be greater than GSN-1, then in step 1006 the CPU of communications router 108 determines whether E-GSN is equal to GSN-1. If E-GSN is equal to GSN-1, then in step 1007, E-GSN may be added if necessary. The encapsulated packet is decapsulated to obtain the encapsulated packet, and then, in step 1009, the encapsulated packet is forwarded to the destination. If E-GSN is not equal to GSN-1 (ie, E-GSN is smaller than GSN-1), it is considered that the encapsulated packet has arrived early. Therefore, in step 1008, the encapsulated packet is stored in the buffer for a predefined period of time. When the predefined time period has expired, the encapsulated packet may then be removed from the buffer and decapsulated to obtain the encapsulated packet. Then, in step 1009, the encapsulated packet is forwarded to the destination. Alternatively, when E-GSN has become equal to GSN-1, then the encapsulated packet may be removed from the buffer and decapsulated. Then, in step 1009, the encapsulated packet is forwarded to the destination. E-GSN may become equal to GSN-1 because the E-GSN is increased by the CPU of the communication router 108 based on the GSN of the received encapsulated packet. In a variant, the encapsulated packet may first be decapsulated to obtain the encapsulated packet, and then, in step 1008, the encapsulated packet is stored in a buffer. When the predefined time period has expired or when the E-GSN becomes equal to GSN-1, then the encapsulated packet can be removed from the buffer and forwarded to the destination. In step 1010, the process ends.

The advantage of storing the encapsulated packets in the buffer is that the encapsulated packets do not need to be decapsulated before being stored, thus saving computing resources for decapsulating. However, encapsulated packets may consume more space in the buffer than encapsulated packets, which may not be desirable. When the encapsulated packet is stored in the buffer, a table indicating the corresponding GSN is also stored together with the encapsulated packet.

E-GSN can be added even when several encapsulated packets with smaller global sequence numbers may be missing (ie delayed, lost or lost). The E-GSN continues to increase because the buffer space for storing encapsulated packets is limited, and once the GSN of the encapsulated packet becomes equal to the E-GSN, the encapsulated packet should be removed from the buffer.

The detailed processing corresponding to step 1007 and step 1008 and the method for calculating E-GSN are published on April 11, 2013, and the international publication number is W02013/049960Al, entitled "METHOD AND SYSTEM FORREDUCTION OF TIME VARIANCE OF PACKETS RECEIVED FROM BONDED COMMUNICATIONLINKS" ("Method and System for Reducing Time Variation of Packets Received from Bonded Communication Links"). In one variation, step 1008 may be performed according to the process of FIG. 13 .

Communications router 108 stores the GSN of encapsulated packets received over the aggregated connection. The missing GSN can be recorded in the missing GSN record. For example, communications router 108 has received 5 encapsulated packets with GSN 01, GSN 02, GSN 03, GSN 05, and GSN 06 within a period of time. The package with GSN 04 is determined to be missing. Therefore, GSN 04 is recorded in the missing GSN record.

In one variation, communications router 108 stores the GSN of each encapsulated packet whose corresponding encapsulated packet has been forwarded to the destination host or node. The GSN may be recorded in the forwarding GSN's record. Encapsulation package can be OEP 800 or DEP810. When determining whether to forward an encapsulated packet corresponding to the encapsulated packet, communications router 108 checks the forwarded GSN's record. If the GSN that encapsulates the packet is found in the forwarded GSN's record, the encapsulated packet is not forwarded to the target host or node because it has already been forwarded before.

FIG. 11 is a flowchart illustrating the process of processing encapsulated packets received over an aggregated connection. For example, communications router 108 receives encapsulated packets from communications router 106 over an aggregated connection that includes multiple tunnels. An encapsulated packet may include one or more encapsulated packets destined for a host or node accessible through communications router 108 .

In step 1100, communications router 108 receives the encapsulated packet. In step 1101, the CPU of communications router 106 determines whether the encapsulation packet includes a list of OEP-GSNs. If the encapsulation packet includes a list of OEP-GSNs, it may be a DEP 820, and if the encapsulation packet does not include a list of OEP-GSNs, it may be an OEP 800.

When it is determined that the encapsulated packet is OEP 800, in step 1110, the CPU of communications router 108 determines whether the GSN of the encapsulated packet (ie, OEP-GSN 802) is recorded in the missing GSN record. If the GSN is recorded in the missing GSN's record, then in step 1112 the encapsulated packet (eg, encapsulated packet 805 ) encapsulated in the encapsulated packet is forwarded to the target host or node. If the GSN is not recorded in the record of the missing GSN, it is determined that the encapsulated packet has been received before, and the encapsulated packet has been forwarded, therefore, in step 1111, the encapsulated packet is no longer forwarded to the destination land.

When it is determined that the encapsulated packet is DEP 820, in step 1120, the CPU of communication router 108 determines whether at least one OEP-GSN in list 826 of m OEP-GSNs is recorded in the missing GSN record. If not, then in step 1121 none of the encapsulated packets 825 are forwarded to the target host or node. If at least one OEP-GSN is found in the record of the missing GSN, then in step 1122, the communication router 106 forwards the at least one OEP-GSN with the list of m OEP-GSNs recorded in the record of the missing GSN The corresponding encapsulated package. For example, if only the OEP-GSN corresponding to the encapsulated packets 825-2 and 825-3 is found in the record of the missing GSN, then in step 1122 only the encapsulated packets 825-2 and 825-3 are forwarded to the destination, while the remaining encapsulated packets are not forwarded to the destination. In step 1130, the process ends.

According to one of the embodiments, when the communication router 108 receives the encapsulated packet, it may determine whether the encapsulated packet is OEP or DEP by checking the tunnel ID. Communications router 106 may notify communications router 108 to select the first tunnel for transporting OEP and select the second tunnel for transporting DEP. Accordingly, communications router 108 may determine that any packets received through the first tunnel are OEP and any packets received through the second tunnel are DEP. However, in some cases, the second tunnel can also be used to transmit other packets, such as management packets, health check packets, and the like. In this case, the encapsulation packet may have an additional field indicating whether the encapsulation packet is OEP or DEP. Alternatively, information indicating whether the encapsulated packet is OEP or DEP may be included in the other information field of the encapsulated packet. Instead of performing step 1101, checking the tunnel ID or checking additional fields may be performed to determine whether the encapsulated packet is OEP or DEP.

FIG. 12A shows the contents of a DEP 820 created based on n OEPs according to an exemplary embodiment. For ease of viewing, the IP header, per-tunnel sequence numbers, and other information are not shown in OEP 800-1, OEP 800-2, OEP 800-3, OEP 800-4, and DEP 820-1. Although the IP header, per-tunnel sequence numbers, and other information are not shown, each of OEP and DEP contains those fields.

Communication router 106 first receives packet 1 , packet 2 , packet 3 , and packet 4 from hosts or nodes in site 102 . These packets are encapsulated in OEP. Data Packet 1 becomes Encapsulated Packet 1 805-1 encapsulated in OEP 800-1. Data packet 2 becomes encapsulated packet 2 805-2 encapsulated in OEP 800-2. Data packet 3 becomes encapsulated packet 3 805-3 encapsulated in OEP 800-3. Data packet 4 becomes encapsulated packet 4 805-4 encapsulated in OEP 800-4. Fields OEP-GSN 802-1 , OEP-GSN 802-2 , OEP-GSN 802-3 and OEP-GSN 802-4 contain OEP-GSN 1 , OEP-GSN 2 , OEP-GSN 3 and OEP-GSN 4 respectively. Communications router 106 transmits OEP 800-1, OEP 800-2, OEP 800-3, and OEP 800-4 to communications router 108 through the first tunnel of the aggregated connection. OEP 800-1, OEP 800-2, OEP 800-3, and OEP800-4 are n OEPs to be transmitted, and their GSNs are OEP-GSN 1, OEP-GSN 2, OEP-GSN 3, and OEP-GSN 4. Communication router 106 then transmits DEP 820-1 to communication router 108 through the second tunnel for four OEPs 800-1, OEP 800-2, OEP 800-3, and OEP 800-4. The list 827 of m OEP-GSNs includes OEP-GSN 1 , OEP-GSN 2 , OEP-GSN 3 and OEP-GSN 4 . As shown in FIGS. 12B and 12C , information (packet information) 1225 is based on encapsulated packets 1 to 4 of OEP-GSN. One of the purposes of transmitting DEP 820-1 is to recover any information that may have been lost during transmission of the OEP. If any of the four OEPs are not received by communications router 108, or if there is an error in any of the four OEPs, DEP 820-1 can be used to recreate any of the four OEPs because of its Contains information from four OEPs. Information 1225 may or may not contain encapsulated packages 1-4. However, information 1225 can be used to recreate encapsulated packages 1-4.

FIG. 12B shows the contents of DEPs with respect to the four OEPs of FIG. 12A according to one of the embodiments. For ease of viewing, the IP header, per-tunnel sequence numbers, and other information are not shown in DEP 820-2. DEP 820-2 may be transmitted to communications router 108 through a second tunnel for the four OEPs 800-1, OEP 800-2, OEP 800-3, and OEP 800-4. In DEP 820-2, information 1225 includes encapsulated packet 1 805-1, encapsulated packet 2 805-2, encapsulated packet 3 805-3, and encapsulated packet 4 805-4. In this way, information 1225 is based on encapsulated packets 1-4, and therefore, encapsulated packets 1-4 can be recreated by communications router 108 using information 1225 in DEP 820-2.

Figure 12C shows the content of a DEP with respect to the four OEPs of Figure 12A, according to an alternative embodiment. For ease of viewing, the IP header, per-tunnel sequence numbers, and other information are not shown in DEP 820-3. DEP 820-3 may be transmitted to communications router 108 through a second tunnel for the four OEPs 800-1, OEP 800-2, OEP 800-3, and OEP 800-4. In DEP 820-3, message 1225 contains message 805-12 and message 805-34. Information 805-12 and information 805-34 may be generated by applying operations to encapsulated packets 1-4. For example, the operation is an exclusive OR (XOR) operation. Information 805-12 is generated by applying an XOR operation to Encapsulated Packet 1 and Encapsulated Packet 2. Accordingly, information 805-12 may be used to recreate encapsulated package 1 and/or encapsulated package 2. Information 805-34 is generated by applying an XOR operation to encapsulated packet 3 and encapsulated packet 4. Accordingly, information 805-34 may be used to recreate encapsulated package 3 and/or encapsulated package 4. When communications router 108 receives DEP 820-3, it can use information 805-12 and information 805-34 to recreate any missing encapsulated packets 1-4 or to correct any of any of encapsulated packets 1-4. mistake. Different combinations of encapsulated packets 1 to 4 can be used together to generate other types of information in the information 1225 of DEP 820-3 by performing an XOR operation. The scope of the invention is not limited to operations that are XOR operations, such that it can be any operation suitable for performing forward error correction. One of the benefits of transmitting DEP 820-3 instead of DEP 820-2 is that the transmission of DEP 820-3 may consume less bandwidth due to the reduced overhead.

FIG. 13 is a flowchart illustrating a process for processing a DEP (eg, DEP 820 - 3 ) received at communications router 108 . In step 1301, communications router 108 receives DEP 820-3 over an aggregated connection. In step 1302, the CPU of the communications router 108 determines the OEP-GSN in the list 827 of OEP-GSNs. In step 1303, the CPU of the communications router 108 determines whether an OEP corresponding to at least one of the OEP-GSNs in the list 827 of OEP-GSNs has been received. More precisely, in step 1303, it is determined whether at least one of OEP 800-1, OEP 800-2, OEP 800-3, and OEP 800-4 has been received. If none of OEP 800-1, OEP 800-2, OEP 800-3, and OEP 800-4 have been received, then in step 1304, information 805-12 and information 805-34 may be stored in a buffer for later post-processing. If at least one of OEP 800-1, OEP 800-2, OEP 800-3, and OEP 800-4 has been received, and after the delay in step 1309, if it is determined in step 1305 that all OEP 800-1 , OEP 800-2, OEP 800-3, and OEP 800-4, then in step 1308, DEP 820-3 is discarded. If at least one but not all of OEP 800-1, OEP 800-2, OEP 800-3, and OEP 800-4 have been received, Then in step 1306, if possible, communications router 108 may use stored information 805-12 and/or information 805-34 to recreate one or more data packets corresponding to one or more OEPs not previously received . In step 1307, the recreated data packet may be forwarded to its destination.

A delay is introduced between step 1303 and step 1305 because it may take some time for all OEPs to reach communication router 108 . Therefore, all OEPs may not be received immediately after one of the OEPs is received.

For purposes of illustration, if it is determined in step 1303 that OEP 800-1 was previously received, but OEP 800-2 was not received, then in step 1306, packet 2 may be recreated using information 805-12. This is possible because both information 805-12 and packet 1 are available, and there is sufficient information to create packet 2. If both OEP 800-3 and OEP 800-4 have not been received before, then in step 1306, DEP 820-3 and information 805-34 may not be used to recreate packet 3 or packet 4 because even before OEP 800-1 and/or OEP800-2 have been received, but also not enough information.

In another example, when the information included in DEP 820-3 is generated by performing an XOR operation on three packets, and an OEP corresponding to only one of the three packets was previously received, then it may not be possible Recreate the other two packets.

In one variation, in step 1304, the entire DEP 820-3 is stored in the buffer rather than just the information 805-12 and the information 805-34. An advantage of storing the entire DEP 820-3 is that no decapsulation is required before storing the DEP 820-3. Computational resources for decapsulation may be saved if DEP 820-3 is eventually discarded. However, DEP 820-3 may consume more space in the buffer than message 805-12 and message 805-34, which may not be desirable.

In a variant, before forwarding the data packet at step 1307, the communications router 108 stores the data packet in a buffer for a predefined period of time. When the predefined time period expires, then the packet is forwarded to the destination. Alternatively, the data packet may be stored in a buffer until the E-GSN becomes equal to the OEP-GSN corresponding to the data packet, after which the data packet is forwarded to the destination.

In one example, communications router 106 establishes an aggregated connection with communications router 108 that includes three tunnels (ie, a first tunnel, a second tunnel, and a third tunnel). The first tunnel is selected to transport the OEP, and the second and third tunnels are selected to transport the DEP. A user or administrator may configure communications router 106 to transport either type of DEP 810 and DEP 820 , or to transport both types of DEP 810 and DEP 820 . When communications router 106 is configured to transmit DEPs 810 , then at least one DEP 810 is transmitted for each OEP 800 . As discussed above, the number of DEPs 810 corresponding to each OEP 800 to be transmitted is configurable.

When communications router 106 is configured to transmit DEP 820 , one DEP 820 may be transmitted for n OEPs 800 . The number n can be configured. For purposes of illustration, n is 5, and communications router 106 is configured to transmit one DEP 820 for every 5 OEPs 800 transmitted through the first tunnel. Therefore, the communication router 106 transmits one DEP 820 through the second tunnel or the third tunnel after transmitting five OEPs 800 through the first tunnel. If the first OEP 800, the second OEP 800, the third OEP 800, the fourth OEP 800, and the fifth OEP 800 are transmitted, and the DEP 820 includes the The information corresponding to the encapsulated packet of the fifth OEP 800, the list 826 of m OEP-GSNs includes the OEP-GSN of the first OEP 800, the OEP-GSN of the second OEP 800, and the OEP-GSN of the third OEP 800 , the OEP-GSN of the fourth OEP 800 and the OEP-GSN of the fifth OEP 800 . DEP-GSN 822 is not identical to OEP-GSN 802 of any OEP 800 . The encapsulated package 825 includes an encapsulated package 825-1, an encapsulated package 825-2, an encapsulated Packet 825-3, Encapsulated Packet 825-4, and Encapsulated Packet 825-5.

When communications router 106 is configured to transport both DEP 810 and DEP 820, it may or may not use the same tunnel for transport. For purposes of illustration, communications router 106 is configured to transmit two DEPs 810 for each OEP 800 and one DEP 820 for every five OEPs 800 . Two DEPs 810 corresponding to the same OEP 800 are not tunneled through the same. The first DEP 810 corresponding to each OEP 800 may be transmitted through the second tunnel, and the second DEP 810 corresponding to each OEP 800 may be transmitted through the third tunnel. DEP 820 may be transported through the second tunnel or the third tunnel. DEP 820 may transmit through the second tunnel and the third tunnel in a polling manner. Alternatively, when communications router 106 is configured not to transmit DEP 810 and DEP 820 through the same tunnel, it may select the second tunnel to transmit one DEP 810 for each OEP 800 and the third tunnel to transmit one DEP 810 for every 5 OEPs 800 Transmit a DEP 820.

In some cases, it may be preferable not to transmit the DEP as it may have an adverse effect on the transmission of the OEP. For example, when the first tunnel and the second tunnel are established using the same network interface of communication device 106 and/or using the same network interface of communication device 108, transmitting the DEP through the second tunnel may extend the delay experienced by the OEP through the first tunnel. The throughput of OEP can also be reduced. Similar effects may be observed when the first and second tunnels are established over the same WAN in site 102 and/or the same WAN in site 104 . A similar effect can also be observed when the first and second tunnels are established using a network provided by the same operator, since the tunnels may be connected to the same base station. In addition, the computing resources consumed by generating and transmitting DEP may slow down the transmission of OEP. Therefore, in these cases, it may be preferable not to transmit a DEP, or to transmit a large number of DEPs corresponding to each OEP.

The process described above is also applicable to encapsulating packets transmitted from communications router 108 to communications router 106 .

It should be understood that the present disclosure is not limited to the architecture of system 500 . For example, any suitable processor-based device may be used to implement the above teachings, including but not limited to routers, personal computers, laptops, computer workstations, multi-processor servers, and even mobile phones. Additionally, certain embodiments may be implemented on an Application Specific Integrated Circuit (ASIC) or Very Large Scale Integration (VLSI). In fact, one of ordinary skill in the art may utilize any number of suitable structures capable of performing logical operations in accordance with the embodiments.

Although the embodiments of the present invention and its advantages have been described in detail, it should be understood that various changes, modifications and substitutions can be made herein without departing from the spirit and scope of the invention as defined by the appended claims. Furthermore, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. Those of ordinary skill in the art will readily understand from the disclosure of the present invention that currently existing or later developed devices that perform substantially the same functions as the corresponding embodiments described herein or implement corresponding embodiments described herein can be utilized according to the present invention. Process, machine, manufacture, composition of matter, means, method or steps that result in substantially the same effect. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

Claims (21)

1. A method for processing data packets received at a network node, comprising the steps of: Step (a): receiving a data packet from a network interface of the network node; Step (b): selecting the first tunnel according to the selection strategy; Step (c): not selecting or selecting at least one second tunnel according to the selection strategy; Step (d): Transmitting an Original Encapsulation Packet (OEP) through the first tunnel, wherein the OEP encapsulates the data packet, wherein each of the OEPs has an Original Encapsulation Packet Global Sequence Number (OEP-GSN); Step (e): When at least one second tunnel is selected, transmitting at least one duplicate encapsulation packet (DEP) through the at least one second tunnel, wherein the at least one DEP encapsulation is based on at least one of the data packets data packet information, wherein each of the at least one DEP has a duplicate encapsulation packet global sequence number (DEP-GSN); Wherein, the OEP-GSN is stored in a field of each of the OEPs, and the DEP-GSN is stored in a field of each of the at least one DEP. 2. The method of claim 1, wherein the first tunnel and the at least one second tunnel are included in an aggregated connection. 3. The method of claim 1, wherein the selection policy is based on one or more of the following criteria: user selection, performance of multiple tunnels, service provider, usage restrictions, location, time, usage price, Security, User Identity, Internet Protocol Address Ranges, Communication Protocols, Communication Technologies, Applications and Devices. 4. The method of claim 1, wherein the OEP-GSN is the same as the DEP-GSN of the at least one DEP. 5. The method of claim 1, wherein the at least one DEP comprises a list of OEP-GSNs, wherein the list of OEP-GSNs contains at least one OEP-GSN. 6. The method of claim 1, wherein performance of the first tunnel is better than performance of the second tunnel, wherein the performance is substantially based on bandwidth and/or latency. 7. The method of claim 2, wherein when a plurality of DEPs are transmitted through the network node for each OEP, each of the plurality of DEPs is transmitted through a different tunnel of the aggregated connection. 8. A method for processing at a second network node an encapsulated packet received from a first network node over an aggregated connection, comprising the steps of: Step (a): receiving an encapsulated packet from the first network node through one of the tunnels of the aggregated connection, wherein the encapsulated packet is an original encapsulated packet (OEP) or a duplicate encapsulated packet (DEP), wherein the OEP for encapsulating a data packet, wherein DEP is used for encapsulating data packet information, wherein the data packet information is substantially based on the data packet, wherein the data packet originates from the first network node or passes through the first network node a network node receiving the data packet; Step (b): determining whether the data packet has previously been received over the aggregated connection, substantially based on a missing Global Sequence Number (GSN) record; Step (c): When it is determined that the data packet has not been received before: (i) when the encapsulated packet is the OEP, decapsulate the data packet from the OEP; (ii) when the encapsulated packet is the DEP, substantially recreating the packet based on the packet information; (iii) forwarding the data packet to a destination, wherein the destination is indicated in a header of the data packet; and (iv) Updating the record of said missing GSN. 9. The method according to claim 8, wherein, when the encapsulation package is DEP and the packet information preserves a plurality of encapsulated packages or error correction information, the encapsulation package includes the original encapsulation package global serial number ( OEP-GSN) list. 10. The method of claim 8, further comprising updating the expected global sequence number after step (b). 11. The method of claim 8, wherein step (b) comprises: determining whether the GSN corresponding to the data packet is in the record of the missing GSN; If the GSN corresponding to the data packet is not in the record of the missing GSN, then determining not to forward the data packet; and If the GSN corresponding to the data packet is in the record of the missing GSN, it is determined to forward the data packet. 12. A system comprising a first network node and a second network node, wherein both the first network node and the second network node comprise: multiple network interfaces; at least one main memory; Wherein, the first network node further includes at least one first processing unit and at least one first auxiliary memory, The at least one first auxiliary memory stores program instructions executable by the at least one first processing unit to process data packets received at the first network node, comprising the steps of: Step (a): receiving a data packet from a network interface of the first network node; Step (b): selecting the first tunnel according to the selection strategy; Step (c): not selecting or selecting at least one second tunnel according to the selection strategy; Step (d): Transmitting an Original Encapsulation Packet (OEP) through the first tunnel, wherein the OEP encapsulates the data packet, wherein each of the OEPs has an Original Encapsulation Packet Global Sequence Number (OEP-GSN); Step (e): When at least one second tunnel is selected, transmitting at least one duplicate encapsulation packet (DEP) through the at least one second tunnel, wherein the at least one DEP encapsulation is based on at least one of the data packets wherein each of the at least one DEP has a duplicate encapsulation package global sequence number (DEP-GSN); wherein the OEP-GSN is stored in a field of each of the OEPs, and the DEP-GSN is stored in a field of each of the at least one DEP; Wherein, the second network node further includes at least one second processing unit and at least one second auxiliary memory, and the second auxiliary memory stores the at least one second processing unit executable to execute at the second network node Program instructions for processing encapsulated packets received from said first network node over an aggregated connection, comprising the steps of: Step (f): receiving an encapsulated packet from the first network node through one of the tunnels of the aggregated connection, wherein the encapsulated packet is OEP or DEP, wherein OEP is used to encapsulate a data packet, wherein DEP is used for encapsulating data packet information, wherein the data packet information is substantially based on the data packet, wherein the data packet originates from the first network node or through receiving the data packet via the first network node; Step (g): determining whether said packet has been previously received over said aggregated connection, substantially based on a missing Global Sequence Number (GSN) record; Step (h): When it is determined that the data packet has not been received before: (i) when the encapsulated packet is the OEP, decapsulate the data packet from the OEP; (ii) when the encapsulated packet is the DEP, substantially recreating the packet based on the packet information; (iii) forwarding the data packet to a destination, wherein the destination is indicated in a header of the data packet; and (iv) Updating the record of said missing GSN. 13. The system of claim 12, wherein the first tunnel and the at least one second tunnel are included in the aggregated connection. 14. The system of claim 12, wherein the selection policy is based on one or more of the following criteria: user selection, performance of multiple tunnels, service provider, usage restrictions, location, time, usage price, Security, User Identity, Internet Protocol Address Ranges, Communication Protocols, Communication Technologies, Applications and Devices. 15. The system of claim 12, wherein the OEP-GSN is the same as the DEP-GSN of the at least one DEP. 16. The system of claim 12, wherein the at least one DEP comprises a list of OEP-GSNs, wherein the list of OEP-GSNs contains at least one OEP-GSN. 17. The system of claim 12, wherein performance of the first tunnel is better than performance of the second tunnel, wherein the performance is substantially based on bandwidth and/or latency. 18. The system of claim 12, wherein when a plurality of DEPs are transmitted through the first network node for each OEP, each of the plurality of DEPs is transmitted through a different tunnel of the aggregated connection . 19. The system of claim 12, wherein when the encapsulated packet is DEP and the data packet information holds a plurality of encapsulated packets or error correction information, the encapsulated packet includes a list of OEP-GSNs. 20. The system of claim 12, further comprising updating the expected global sequence number after step (g). 21. The system of claim 12, wherein step (g) comprises: determining whether the GSN corresponding to the data packet is in the record of the missing GSN; If the GSN corresponding to the data packet is not in the record of the missing GSN, then determining not to forward the data packet; and If the GSN corresponding to the data packet is in the record of the missing GSN, it is determined to forward the data packet.