CN102075423A - Hardware multi-level table-based method for controlling output traffic - Google Patents
Hardware multi-level table-based method for controlling output traffic Download PDFInfo
- Publication number
- CN102075423A CN102075423A CN201110003670.5A CN201110003670A CN102075423A CN 102075423 A CN102075423 A CN 102075423A CN 201110003670 A CN201110003670 A CN 201110003670A CN 102075423 A CN102075423 A CN 102075423A
- Authority
- CN
- China
- Prior art keywords
- flow control
- priority
- message
- output
- flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
本发明公开了一种基于硬件多级表的输出流量控制方法,要解决的技术问题是在保证流的完整性的前提下进行流量控制。技术方案是先构建由输入卡、输出卡、控制主机、后端分析系统组成的高速网络内容监控系统,输入卡中增加优先级模块,输出卡增加流量控制模块,控制主机上运行流量控制软件;优先级模块在优先级表中查找优先级号,将带有优先级号的报文送给输出卡;流量控制模块查找报文的优先级号对应优先级的流控表,决定对报文发送或丢弃;流量控制软件统计流量,设置丢弃标志位,控制流量控制模块进行流量控制。采用本发明能在保证流的完整性的前提下进行流量控制,且硬件逻辑简单,处理报文速度快,网络抖动小。
The invention discloses an output flow control method based on a hardware multilevel table, and the technical problem to be solved is to control the flow on the premise of ensuring the integrity of the flow. The technical solution is to build a high-speed network content monitoring system composed of input card, output card, control host, and back-end analysis system, add a priority module to the input card, add a flow control module to the output card, and run flow control software on the control host; The priority module looks up the priority number in the priority table, and sends the message with the priority number to the output card; the flow control module looks up the flow control table corresponding to the priority number of the message, and decides to send the message or discard; the flow control software counts the flow, sets the discard flag bit, and controls the flow control module to perform flow control. By adopting the invention, the flow control can be carried out on the premise of ensuring the integrity of the flow, and the hardware logic is simple, the message processing speed is fast, and the network jitter is small.
Description
技术领域technical field
本发明涉及高速网络流量监控领域,尤其是高速网络上超负载流量的控制方法。The invention relates to the field of high-speed network traffic monitoring, in particular to a method for controlling overloaded traffic on the high-speed network.
背景技术Background technique
高速网络内容监控系统一般由前端数据捕获分流设备和后端分析系统组成,总体结构如图1所示。前端数据捕获分流设备完成网络接入、数据捕获和分流、数据过滤和分析等功能,主要由包含硬件DPI(DPI:深度报文检测)模块的输入卡、输出卡组成。输入卡接收网络数据流,其中的硬件DPI模块进行深度报文检测处理(例如规则匹配),处理后的报文从输出卡的端口输出至后端分析系统,输出卡根据后端分析系统的负载能力进行负载均衡。后端分析系统对前端设备输出的报文进行进一步分析,实现网络行为审计、网络内容审计和入侵检测。随着骨干网链路速度的不断提升,高速网络内容监控系统面临的主要问题是:巨大流量经常超出后端分析系统的处理能力,因此前端设备需要对输出流量进行控制,通过丢弃部分报文使流量符合后端系统的处理能力。由于后端分析系统通常基于流(TCP或UDP流,包括通信双方一个会话中的所有报文)进行数据分析,所以要求流量控制时尽可能保持“流完整性”,即丢弃的报文集中在少量流中,而使尽可能多的流不丢弃报文,保持流完整性。因此,这就需要一种新的能够保持流完整性的流量控制方法。A high-speed network content monitoring system generally consists of a front-end data capture and distribution device and a back-end analysis system. The overall structure is shown in Figure 1. The front-end data capture and distribution equipment completes functions such as network access, data capture and distribution, data filtering and analysis, and is mainly composed of an input card and an output card including a hardware DPI (DPI: Deep Packet Inspection) module. The input card receives the network data flow, and the hardware DPI module in it performs in-depth message detection processing (such as rule matching), and the processed message is output from the port of the output card to the back-end analysis system, and the output card is based on the load of the back-end analysis system capacity for load balancing. The back-end analysis system further analyzes the packets output by the front-end equipment to realize network behavior audit, network content audit and intrusion detection. With the continuous improvement of the link speed of the backbone network, the main problem faced by the high-speed network content monitoring system is: the huge traffic often exceeds the processing capacity of the back-end analysis system, so the front-end equipment needs to control the output traffic by discarding some packets to make the Traffic conforms to the processing capabilities of the backend system. Since the back-end analysis system usually performs data analysis based on streams (TCP or UDP streams, including all packets in a session of the communication parties), it is required to maintain "flow integrity" as much as possible during flow control, that is, discarded packets are concentrated in In a small number of flows, as many flows as possible do not discard packets, maintaining flow integrity. Therefore, there is a need for a new flow control method that can maintain flow integrity.
现有的流量控制方法主要有: The existing flow control methods mainly include:
1.出口负载均衡方法。其基本的思想是将流量尽可能平均的分配给多个出口,以减少报文的丢弃。但是考虑到各个出口所连接的后端系统的处理能力可能不同,实际流量大小也不同,在实现中很难做到平衡、灵活的分流。更为重要的是,出口负载均衡不能保证流完整性。1. Egress load balancing method. The basic idea is to distribute traffic as evenly as possible to multiple egresses to reduce packet discarding. However, considering that the processing capabilities of the back-end systems connected to each outlet may be different, and the actual flow size is also different, it is difficult to achieve a balanced and flexible distribution in the implementation. More importantly, egress load balancing cannot guarantee flow integrity.
2.拥塞控制方法。大多数网络设备都有拥塞控制机制,当报文的等待队列已满时,拥塞控制机制将丢弃多余的报文,但是通常这种丢弃是随机的,不保持流的完整性。2. congestion control method. Most network devices have a congestion control mechanism. When the packet waiting queue is full, the congestion control mechanism will discard redundant packets, but usually this discard is random and does not maintain the integrity of the flow.
3.QoS(Quality of Service,服务质量)方法。根据应用协议确定报文的优先级,当网络发生拥塞时,优先级高的报文不会被丢弃,而对于低优先级的应用不能保持流的完整性。同时,由于优先级只根据应用协议确定,流量控制的灵活性差。总结以上几种方法,其共有的最大缺陷是不保证流的完整性,难以满足高速网络内容监控系统的需要。现有的网络设备大多采用以上方法的一种或几种,综合起来进行流量控制,但是对于流的完整性仍无法保证。3. QoS (Quality of Service, quality of service) method. The priority of the packets is determined according to the application protocol. When the network is congested, the packets with high priority will not be discarded, and the integrity of the flow cannot be maintained for applications with low priority. At the same time, because the priority is only determined according to the application protocol, the flexibility of flow control is poor. Summarizing the above methods, the biggest defect in common is that they do not guarantee the integrity of the flow, and it is difficult to meet the needs of the high-speed network content monitoring system. Most of the existing network devices use one or more of the above methods to control the flow in combination, but the integrity of the flow is still not guaranteed.
发明内容Contents of the invention
本发明要解决的技术问题是在保证流的完整性的前提下进行流量控制。The technical problem to be solved by the invention is to perform flow control on the premise of ensuring the integrity of the flow.
为解决上述具体技术问题,技术方案包括以下步骤:In order to solve the above-mentioned specific technical problems, the technical solution includes the following steps:
第一步,构建高速网络内容监控系统,高速网络内容监控系统由输入卡、输出卡、控制主机、后端分析系统组成。输入卡中增加优先级模块,优先级模块与硬件DPI模块和输入卡的输出端口相连,它从硬件DPI接收经深度报文检测处理的报文,确定报文的优先级,然后将确定了优先级的报文传送给输出卡;输出卡与输入卡、控制主机、后端分析系统相连,一个输出卡有一个输入端口和若干输出端口,在输出卡各输出端口增加流量控制模块,各输出端口的流量控制模块与输出卡的输入端口、控制主机和后端分析系统相连,流量控制模块决定对报文发送或丢弃;控制主机与输出卡相连,其上运行流量控制软件,流量控制软件统计流量,并根据流控策略设置丢弃标志位,从而控制流量控制模块进行流量控制。 The first step is to build a high-speed network content monitoring system. The high-speed network content monitoring system consists of an input card, an output card, a control host, and a back-end analysis system. A priority module is added to the input card, and the priority module is connected with the hardware DPI module and the output port of the input card. Level messages are sent to the output card; the output card is connected to the input card, the control host, and the back-end analysis system. An output card has an input port and several output ports. A flow control module is added to each output port of the output card. Each output port The flow control module of the output card is connected with the input port of the output card, the control host and the back-end analysis system, and the flow control module decides to send or discard the message; the control host is connected with the output card, and the flow control software runs on it, and the flow control software counts the flow , and set the discard flag bit according to the flow control policy, so as to control the flow control module to perform flow control. the
优先级模块由控制逻辑和优先级表组成。控制逻辑分为报文取头逻辑和加报文头逻辑,报文取头逻辑接收硬件DPI模块输出的头部带有规则ID号的报文,将其头部的规则ID号取出,并查找优先级表,获得规则ID与优先级号的对应关系,找出该规则ID对应的优先级号,再由加报文头逻辑将优先级号加在报文头部。优先级表的每个表项包括规则ID值和优先级号两个域,规则ID值是此报文匹配的规则号,优先级号是此规则ID值对应的优先级,一个规则ID值对应一个优先级号,优先级号越小表示优先级越高。这两个域的项数均为P个,P为正整数。Priority module consists of control logic and priority table. The control logic is divided into the message header logic and the message header logic. The message header logic receives the message with the rule ID number in the head output by the hardware DPI module, takes out the rule ID number in the head, and searches The priority table obtains the corresponding relationship between the rule ID and the priority number, finds out the priority number corresponding to the rule ID, and then adds the priority number to the message header by adding the message header logic. Each entry in the priority table includes two fields: the rule ID value and the priority number. The rule ID value is the rule number that this packet matches, and the priority number is the priority corresponding to the rule ID value. A priority number, the smaller the priority number, the higher the priority. The number of items in these two fields is P, and P is a positive integer.
输出卡的每一个输出端口对应一个流量控制模块。每一个流量控制模块由流量控制逻辑和P个流控表组成,即每一个优先级对应一个流控表。流控表的每一项由序号(ID)、字节数(bytes)和丢弃标志位(discard)三个域构成。序号ID由源IP的最后n位和目的IP的最后k位共n+k位(n、k均为大于等于1小于等于32的正整数)来确定,因此一个流控表一共有2(n+k)项。字节数表示某一个表项在规定时间内的流量大小,丢弃标志表示此表项是丢弃还是转发,0为转发,1为丢弃。每个输出端口的流量控制模块的流量控制逻辑与输出卡的输入端口、P个流控表、后端分析系统相连,它从输出卡的输入端口接收输入的报文并根据该报文的优先级确定该报文对应的优先级的流控表,然后根据该报文的源IP地址和目的IP地址查找此流控表的表项,读取丢弃标志位discard以决定对报文转发还是丢弃,并对转发出去的报文更新其所在表项的字节数bytes。Each output port of the output card corresponds to a flow control module. Each flow control module is composed of flow control logic and P flow control tables, that is, each priority corresponds to a flow control table. Each item in the flow control table consists of three fields: serial number (ID), byte count (bytes) and discard flag (discard). The serial number ID is determined by the last n digits of the source IP and the last k digits of the destination IP, a total of n+k digits (n and k are both positive integers greater than or equal to 1 and less than or equal to 32), so a flow control table has a total of 2 (n +k) item. The number of bytes indicates the traffic size of a certain entry within the specified time, and the discard flag indicates whether the entry is discarded or forwarded, 0 for forwarding, 1 for discarding. The flow control logic of the flow control module of each output port is connected with the input port of the output card, P flow control tables, and the back-end analysis system. level to determine the flow control table corresponding to the priority of the message, and then look up the entry of the flow control table according to the source IP address and destination IP address of the message, and read the discard flag bit discard to decide whether to forward or discard the message , and update the number of bytes in the entry of the forwarded message.
第二步,硬件DPI根据流入输入卡的报文的五元组进行规则匹配,每条规则都有自己的规则ID号,当一个报文匹配上某条规则时,就将此规则的ID加在报文头部,转发给优先级模块,优先级模块根据报文头部的ID值在优先级表中查找相应的优先级号,并加在报文头部,然后将带有优先级号的报文送给输出卡。In the second step, the hardware DPI performs rule matching according to the quintuple of packets flowing into the input card. Each rule has its own rule ID number. When a packet matches a certain rule, the ID of this rule is added to the In the header of the message, it is forwarded to the priority module, and the priority module looks up the corresponding priority number in the priority table according to the ID value of the message header, and adds it to the header of the message, and then adds the priority number The message is sent to the output card.
第三步,输出卡的流量控制模块根据每个报文头部携带的优先级号确定该报文对应的优先级的流控表,然后根据报文源IP地址的最后n位和目的IP地址的最后k位共(n+k)位,查找此流控表的第2(n+k) 个表项,然后查看这个表项的丢弃标志位,若丢弃标志位为0,则转发该报文;若丢弃标志位为1,则丢弃该报文。并对转发出去的报文更新其对应表项的字节数。用 表示输出端口Ai(1≤i≤Z)的优先级为j(1≤j≤M)的流控表,用表示输出端口Ai的优先级为j的流控表的第m项的字节数的值;表示输出端口Ai的优先级为j的流控表的第m项的丢弃标志位的值;用表示流控表中丢弃标志位为0的表项的集合。流量控制模块的流量控制逻辑按以下步骤对流量进行控制: In the third step, the flow control module of the output card determines the flow control table corresponding to the priority of the message according to the priority number carried in the head of each message, and then according to the last n bits of the source IP address of the message and the destination IP address (n+k) bits in total, look for the second (n+k) entry of the flow control table, and then check the discard flag of this entry, if the discard flag is 0, forward the report If the discard flag is 1, the message will be discarded. And update the byte count of the corresponding table entry for the forwarded message. use Indicates the flow control table with the priority j (1≤j≤M) of the output port A i (1≤i≤Z), using Represents the value of the number of bytes of the mth item of the flow control table whose priority of output port A i is j; Indicates the value of the discard flag bit of the mth item of the flow control table whose priority is j at the output port A i ; use Indicates flow control table A collection of entries whose flag bit is 0 is discarded. The flow control logic of the flow control module controls the flow according to the following steps:
3.1根据输入报文的优先级号确定该报文所属优先级的流控表。3.1 Determine the flow control table of the priority of the message according to the priority number of the input message.
3.2按输入报文的源IP地址的最后n位与目的IP地址的最后k位共(n+k)位查找3.1所述流控表的第2(n+k)项。3.2 Find the 2nd (n+k) item of the flow control table described in 3.1 according to the last n bits of the source IP address of the input message and the last k bits of the destination IP address (n+k) bits altogether.
3.3判断3.2所述流控表的第2(n+k)项的是否为1,如果为1,则丢弃报文;否则,将报文发送出去。3.3 Judgment of item 2 (n+k) of the flow control table described in 3.2 Whether it is 1, if it is 1, discard the packet; otherwise, send the packet.
3.4将发送出去的报文的大小加在该报文对应的表项的字节数一项上,进行统计。3.4 Add the size of the sent message to the number of bytes of the entry corresponding to the message to make statistics.
第四步,流量控制软件定时读取输出卡各输出端口的流量控制模块的所有流控表,对所有丢弃标志位为0的表项的字节数域求和,计算当前输出流量,如果当前输出流量超过设定的端口流量限制,则以优先级从低到高的顺序将流控表中的若干表项的丢弃标志位置1,增加硬件丢弃的流量。如果当前流量小于端口流量限制,则以优先级从高到低的顺序将流控表中的若干表项的丢弃标志位置0,减少硬件丢弃的流量。In the fourth step, the flow control software regularly reads all the flow control tables of the flow control module of each output port of the output card, sums the byte count fields of all entries whose discard flag is 0, and calculates the current output flow. If the output traffic exceeds the set port traffic limit, the discard flags of several entries in the flow control table will be set to 1 in order of priority from low to high, increasing the traffic discarded by the hardware. If the current traffic is less than the port traffic limit, the discard flags of several entries in the traffic control table will be set to 0 in order of priority from high to low to reduce the traffic discarded by the hardware.
对于有A1、A2、……Az等Z个报文输出端口的输出卡,A1、A2、……Az的限定流量分别为B1、B2、……Bz,A1、A2、……Az分别维护P个不同优先级的流控表。流量控制软件的流程是:For an output card with Z message output ports such as A 1 , A 2 , ... A z , the limited flows of A 1 , A 2 , ... A z are B 1 , B 2 , ... B z , A 1 , A 2 , ... A z respectively maintain P flow control tables with different priorities. The flow control software flow is:
4.1经过时间间隔T(T根据实际的流量控制需要来设定,一般小于等于60秒) ,流量控制软件从输出端口Ai(1≤i≤Z)的P个优先级的流控表中读取丢弃标志位为0的各表项中字节数的值并求和(即),Si是当前输出端口Ai实际输出总流量。若Si>Bi,则转步骤4.2,否则转步骤4.3。4.1 After a time interval T (T is set according to the actual flow control needs, generally less than or equal to 60 seconds), the flow control software reads from the flow control tables of P priorities at the output port A i (1≤i≤Z) Take the value of the number of bytes in each entry whose discard flag is 0 and sum them up (that is, ), S i is the actual output total flow of the current output port A i . If S i >B i , go to step 4.2, otherwise go to step 4.3.
4.2此时输出端口Ai的总输出流量超过了其负载限制,采取快丢弃的策略。执行步骤如下:4.2 At this time, the total output flow of the output port A i exceeds its load limit, and the strategy of fast discarding is adopted. The execution steps are as follows:
4.2.1 将优先级j置为1。4.2.1 Set priority j to 1.
4.2.2设超过部分的流量(待丢弃流量)为=Si-Bi。在流控表中选择若干表项的丢弃位置1,使得这些表项的字节数之和为,从而恰好将超过部分的流量丢弃。如何选择表项是一个子集合问题。子集合问题可以表示为一个对偶(G,t),其中G是正整数的一个集合{X1,X2,……Xn},t是一个正整数,要求找出G的一个子集,其和既要尽可能大但又不能大于t。由于子集合问题是一个NP完全问题,即只有指数时间的精确解,这里采用算法导论(INTRODUCTION TO ALGORITHMS,高等教育出版社,2002年版,第1046页)公布的多项式时间的近似算法APPROX_SUBSET_SUM(G,t)来求解:算法输入为(G,t),G为流控表表项的集合,t是超负载流量,输出为G的一个子集,使得G中表项的字节数之和尽可能大但不超过t。如果APPROX_SUBSET_SUM(,)返回空集,则转4.2.2.1;如果APPROX_SUBSET_SUM(,)返回,即全部丢弃也不能满足要求,则转4.2.2.3;否则转4.2.2.2。4.2.2 Let the excess traffic (traffic to be discarded) be =S i -B i . In the flow control table, select the discarding position of several entries to be 1, so that the sum of the bytes of these entries is , thus discarding exactly the excess traffic. How to select table items is a subset problem. The subset problem can be expressed as a dual (G, t), where G is a set {X 1 , X 2 ,...X n } of positive integers, t is a positive integer, and it is required to find a subset of G whose and must be as large as possible but not larger than t. Since the subset problem is an NP-complete problem, that is, there is only an exact solution in exponential time, the polynomial time approximation algorithm APPROX_SUBSET_SUM (G, t) to solve: the input of the algorithm is (G, t), G is the set of flow control table entries, t is the overload traffic, and the output is a subset of G, so that the sum of the bytes of the entries in G is as small as possible. May be as large as but not larger than t. If APPROX_SUBSET_SUM( , ) returns an empty set, then turn to 4.2.2.1; if APPROX_SUBSET_SUM( , )return , even discarding all of them can not meet the requirements, then go to 4.2.2.3; otherwise, go to 4.2.2.2.
4.2.2.1 不需要再丢弃流量,转4.2.3;4.2.2.1 No need to discard traffic anymore, go to 4.2.3;
4.2.2.2判定当前优先级的流控表是否能够完成流控要求,即对所有丢弃标志位为0的表项的字节数一项求和,设这个和为Q。再将Q和超负载的流量作比较。若Q≥,将APPROX_SUBSET_SUM(,)返回的集合(表项的集合)中所有表项的丢弃标志位置1;转4.2.3;否则,执行4.2.2.3;4.2.2.2 Determine whether the current priority flow control table can meet the flow control requirements, that is, sum the number of bytes of all entries whose discard flag is 0, and set this sum as Q. Q and the overloaded traffic compared to. If Q≥ , will APPROX_SUBSET_SUM( , ) in the returned collection (a collection of entries), the discarding flag position of all entries is 1; go to 4.2.3; otherwise, execute 4.2.2.3;
4.2.2.3当前优先级的流控表不能完成流量丢弃要求,首先将当前优先级流控表的所有表项的丢弃标志置1,从而丢弃掉此表中的全部流量。剩余的待丢弃流量需要在更高级流控表中进行丢弃,令=-Q,Q为当前优先级的流控表中丢弃标志位为0的所有表项的字节数之和。j=j+1;转4.2.2。4.2.2.3 If the current priority flow control table cannot fulfill the traffic discarding requirements, first set the discard flags of all entries in the current priority flow control table to 1, thereby discarding all the traffic in the table. The remaining traffic to be discarded needs to be discarded in the higher-level flow control table, so that = -Q, Q is the sum of the number of bytes of all entries in the flow control table of the current priority with the discard flag bit set to 0. j=j+1; turn to 4.2.2.
4.2.3 完成本次流控,转4.4。4.2.3 Complete this flow control, go to 4.4.
4.3此时输出端口Ai的总输出流量小于其负载能力,采取慢恢复的策略,设此时输出端口Ai有丢弃标志位被置1的所有流控表中优先级最高的为H,并设这个优先级为H的流控表中有F1、F2、……FL个表项丢弃标志位被置1(1≤L≤2(n+k))。则选取这些表项中字节数最少的一项,即min{, ,……},设为(1≤Y≤L),令此项的丢弃标志位为0,即=0。转4.4。4.3 At this time, the total output flow of the output port A i is less than its load capacity, and the strategy of slow recovery is adopted. At this time, the output port A i has the discard flag bit set to 1 in all flow control tables with the highest priority as H, and Assume that in the flow control table with priority H, there are F 1 , F 2 , ... F L entry discard flags set to 1 (1≤L≤2 (n+k) ). Then select the item with the least number of bytes among these entries, that is, min{ , ,... }, set to (1≤Y≤L), let the discard flag of this item be 0, namely =0. Go to 4.4.
4.4清空所有流控表表项的字节数,重新开始统计,转4.1。4.4 Clear the bytes of all flow control table entries, restart the statistics, and go to 4.1.
采用本发明可以达到以下技术效果:Adopt the present invention can reach following technical effect:
(a)硬件逻辑简单。本发明优先级模块和流量控制模块由硬件实现,且只需要分别维护优先级表和流控表,查表过程实现简单,所以处理报文速度快,符合高速网络内容监控系统的需要。(a) The hardware logic is simple. The priority module and the flow control module of the present invention are realized by hardware, and only need to maintain the priority table and the flow control table respectively, and the table look-up process is simple to realize, so the message processing speed is fast, which meets the needs of the high-speed network content monitoring system.
(b)保持流的完整性。为了保证内容安全应用,在丢弃和恢复报文时以优先级和源IP地址、目的IP地址查找表项进行操作,即以流为单位进行,一个流要么全部被丢弃,要么全部被保留。克服了其他方法不保持流的完整性的缺点。(b) Preserve the integrity of the stream. In order to ensure the application of content security, when discarding and restoring packets, the operations are performed based on the priority, source IP address, and destination IP address lookup table entries, that is, it is performed in flow units, and a flow is either completely discarded or completely reserved. The disadvantage of other methods not maintaining the integrity of the stream is overcome.
(c)根据硬件DPI的结果确定一个报文所属的优先级的流控表,用户感兴趣的报文可以使其拥有较高优先级,以尽量不丢包。(c) According to the result of the hardware DPI, determine the flow control table of the priority of a message. The message that the user is interested in can be given a higher priority to avoid packet loss as much as possible.
(d)网络抖动小。若总流量超过限定值,则采取快丢弃策略,即在下一个时间段将超负载的流量全部丢弃,使实际流量迅速减少;若实际总流量小于限定值,则采取慢恢复策略,即一次只恢复一个流,避免了短时间内实际流量再次超过限定值。(d) The network jitter is small. If the total traffic exceeds the limit value, the fast discard strategy is adopted, that is, all the overloaded traffic is discarded in the next time period, so that the actual traffic decreases rapidly; if the actual total traffic is less than the limit value, the slow recovery strategy is adopted, that is, only one recovery A flow to avoid the actual flow exceeding the limit again in a short period of time.
附图说明Description of drawings
图1是背景技术所述现有的高速网络内容监控系统总体结构图。FIG. 1 is a general structural diagram of the existing high-speed network content monitoring system described in the background art.
图2是本发明第一步设计的高速网络内容监控系统的总体结构图。Fig. 2 is the overall structural diagram of the high-speed network content monitoring system designed in the first step of the present invention.
图3是本发明优先级模块结构图。Fig. 3 is a structural diagram of the priority module of the present invention.
图4是本发明流量控制模块结构图。Fig. 4 is a structural diagram of the flow control module of the present invention.
图5是本发明流量控制逻辑决定对报文发送或丢弃的流程图。Fig. 5 is a flow chart of the flow control logic decision of the present invention to send or discard the message.
图6是本发明流量控制软件流程图。Fig. 6 is a flowchart of the flow control software of the present invention.
图7是本发明总体流程图。Fig. 7 is an overall flowchart of the present invention.
图8是本发明高速网络内容监控系统的一个实例。Fig. 8 is an example of the high-speed network content monitoring system of the present invention.
具体实施方式:Detailed ways:
图1是背景技术现有的高速网络内容监控系统总体结构图。前端数据捕获分流设备完成网络接入、数据捕获和分流、数据过滤和分析等功能,主要由包含硬件DPI(DPI:深度报文检测)模块的输入卡、输出卡组成。输入卡接收网络数据流,其中的硬件DPI模块进行深度报文检测处理(例如规则匹配),处理后的报文从输出卡的端口输出至后端分析系统,输出卡根据后端分析系统的负载能力进行负载均衡。后端分析系统对前端设备输出的报文进行进一步分析,实现网络行为审计、网络内容审计和入侵检测。FIG. 1 is a general structure diagram of an existing high-speed network content monitoring system in the background technology. The front-end data capture and distribution equipment completes functions such as network access, data capture and distribution, data filtering and analysis, and is mainly composed of an input card and an output card including a hardware DPI (DPI: Deep Packet Inspection) module. The input card receives the network data flow, and the hardware DPI module in it performs in-depth message detection processing (such as rule matching), and the processed message is output from the port of the output card to the back-end analysis system, and the output card is based on the load of the back-end analysis system capacity for load balancing. The back-end analysis system further analyzes the packets output by the front-end equipment to realize network behavior audit, network content audit and intrusion detection.
图2是本发明第一步实现的高速网络内容监控系统的总体结构图。高速网络内容监控系统由输入卡、输出卡、控制主机、后端分析系统组成。输入卡中增加优先级模块,优先级模块与硬件DPI模块和输入卡的输出端口相连,它从硬件DPI接收经深度报文检测处理的报文,确定报文的优先级,然后将确定了优先级的报文传送给输出卡;输出卡与输入卡、控制主机、后端分析系统相连,一个输出卡有一个输入端口和若干输出端口,在输出卡各输出端口增加流量控制模块,各输出端口的流量控制模块与输出卡的输入端口、控制主机、后端分析系统相连,流量控制模块决定对报文发送或丢弃;控制主机与输出卡相连,其上运行流量控制软件,流量控制软件统计流量,并根据本发明实现的流控策略设置丢弃标志位,从而控制流控模块进行流量控制。 Fig. 2 is an overall structural diagram of the high-speed network content monitoring system implemented in the first step of the present invention. The high-speed network content monitoring system consists of an input card, an output card, a control host, and a back-end analysis system. A priority module is added to the input card, and the priority module is connected with the hardware DPI module and the output port of the input card. Level messages are sent to the output card; the output card is connected to the input card, the control host, and the back-end analysis system. An output card has an input port and several output ports. A flow control module is added to each output port of the output card. Each output port The flow control module of the output card is connected to the input port of the output card, the control host, and the back-end analysis system. The flow control module decides to send or discard the message; the control host is connected to the output card, and the flow control software runs on it, and the flow control software counts the flow. , and set the discarding flag bit according to the flow control policy realized in the present invention, so as to control the flow control module to perform flow control. the
图3是优先级模块结构图。优先级模块由控制逻辑和优先级表组成。控制逻辑分为报文取头逻辑和加报文头逻辑,报文取头逻辑接收硬件DPI模块输出的头部带有规则ID号的报文,将其头部的规则ID号取出,并查找优先级表,找出对应的优先级号,再由加报文头逻辑将优先级号加在报文头部。优先级表的每个表项包括规则ID值和优先级号两个域,规则ID值是此报文匹配的规则号,优先级号是此规则号对应的优先级,越小表示优先级越高。这两个域的项数根据实际需要取非负整数M。Figure 3 is a structural diagram of the priority module. Priority module consists of control logic and priority table. The control logic is divided into the message header logic and the message header logic. The message header logic receives the message with the rule ID number in the head output by the hardware DPI module, takes out the rule ID number in the head, and searches Priority table, find out the corresponding priority number, and then add the priority number to the message header by adding the message header logic. Each entry in the priority table includes two fields: the rule ID value and the priority number. The rule ID value is the rule number that this packet matches, and the priority number is the priority corresponding to the rule number. The smaller the priority, the higher the priority. high. The number of items in these two domains takes a non-negative integer M according to actual needs.
图4是流量控制模块结构图。流量控制模块由流量控制逻辑和流控表组成。流控表的每一项由序号(ID)、字节数(bytes)和丢弃标志(discard)三个域构成。每一个优先级对应一个流控表,ID由源IP的最后n位和目的IP的最后k位共n+k位来确定,因此流控表一共有2n+k项。字节数表示某一个表项在规定时间内的流量大小,丢弃标志表示此表项是丢弃还是转发,0为转发,1为丢弃。流量控制逻辑接收输入的报文并根据其优先级确定该报文对应的优先级的流控表,然后根据该报文的源IP地址和目的IP地址查找此流控表的表项,读取丢弃标志位以决定对报文转发还是丢弃,并对转发出去的报文更新其所在表项的字节数。Figure 4 is a structural diagram of the flow control module. The flow control module is composed of flow control logic and flow control table. Each item in the flow control table consists of three fields: serial number (ID), byte count (bytes) and discard flag (discard). Each priority corresponds to a flow control table, and the ID is determined by n+k bits of the last n bits of the source IP and the last k bits of the destination IP, so the flow control table has a total of 2 n+k items. The number of bytes indicates the traffic size of a certain entry within the specified time, and the discard flag indicates whether the entry is discarded or forwarded, 0 for forwarding, 1 for discarding. The flow control logic receives the input message and determines the priority flow control table corresponding to the message according to its priority, and then searches the entry of the flow control table according to the source IP address and destination IP address of the message, and reads The discard flag is used to decide whether to forward or discard the packet, and to update the byte count of the entry for the forwarded packet.
图5是流量控制逻辑决定对报文发送或丢弃的流程图流量控制逻辑根据每个报文的优先级、源IP地址的最后n位和目的IP地址的最后k位共(n+k)位(n、k均为大于等于1小于等于32的正整数),查找相应优先级的流控表的第2n+k个表项,查看该表项的丢弃标志位,若丢弃标志位为0,则转发该报文;若丢弃标志位为1,则丢弃该报文。然后更新与该报文对应的流控表的表项的字节数的值。Figure 5 is a flow chart of the flow control logic deciding whether to send or discard the message. The flow control logic is based on the priority of each message, the last n bits of the source IP address and the last k bits of the destination IP address, a total of (n+k) bits (n and k are both positive integers greater than or equal to 1 and less than or equal to 32), find the 2nd n+k entry in the flow control table of the corresponding priority, check the discard flag of the entry, if the discard flag is 0 , the packet is forwarded; if the discard flag is 1, the packet is discarded. Then update the value of the number of bytes in the entry of the flow control table corresponding to the message.
图6是流量控制软件流程图。Figure 6 is a flowchart of the flow control software.
首先,流量控制软件定时读取输出卡各输出端口的流量控制模块的所有流控表,对所有丢弃标志位为0的表项的字节数域求和,计算当前输出流量。First, the flow control software regularly reads all the flow control tables of the flow control module of each output port of the output card, sums the byte count fields of all entries whose discard flag is 0, and calculates the current output flow.
然后进行判断:如果当前输出流量超过设定的端口流量限制,则以优先级从低到高的顺序采用子集合问题近似算法求解,将流控表中的若干表项的丢弃标志位置1,增加硬件丢弃的流量。如果当前流量小于端口流量限制,则以优先级从高到低的顺序采用慢恢复策略,将多级流控表中的若干表项的丢弃标志位置0,减少硬件丢弃的流量。Then make a judgment: if the current output flow exceeds the set port flow limit, use the subset problem approximation algorithm in order of priority from low to high to solve, set the discard flag position of several entries in the flow control table to 1, increase Traffic dropped by hardware. If the current traffic is less than the port traffic limit, the slow recovery strategy is adopted in order of priority from high to low, and the discard flags of several entries in the multi-level flow control table are set to 0 to reduce the traffic discarded by the hardware.
最后,清空字节数表项,以便进行下一次统计。Finally, the byte count entry is cleared for the next statistics.
图7是本发明总体流程图。Fig. 7 is an overall flowchart of the present invention.
第一步,构建高速网络内容监控系统。The first step is to build a high-speed network content monitoring system.
第二步,硬件DPI根据流入输入卡的报文的五元组进行规则匹配,每条规则都有自己的规则ID号,当一个报文匹配上某条规则时,就将此规则的ID加在报文头部,转发给优先级模块,优先级模块根据报文头部的ID值在优先级表中查找相应的优先级号,并加在报文头部。将带有优先级号的报文送给输出卡。In the second step, the hardware DPI performs rule matching according to the quintuple of packets flowing into the input card. Each rule has its own rule ID number. When a packet matches a certain rule, the ID of this rule is added to the In the header of the message, it is forwarded to the priority module, and the priority module looks up the corresponding priority number in the priority table according to the ID value of the header of the message, and adds it to the header of the message. Send the message with the priority number to the output card.
第三步,输出卡的流量控制模块根据每个报文头部携带的的优先级号、报文源IP地址的最后n位和目的IP地址的最后k位共(n+k)位,查找此优先级对应的流控表的第2(n+k)个表项,然后查看这个表项的丢弃标志位,若丢弃标志位为0,则转发该报文;若丢弃标志位为1,则丢弃该报文。In the third step, the flow control module of the output card searches for (n+k) bits according to the priority number carried by each message header, the last n bits of the message source IP address, and the last k bits of the destination IP address. The second (n+k) entry of the flow control table corresponding to this priority, and then check the discarding flag of this table entry, if the discarding flag is 0, then forward the message; if the discarding flag is 1, then discard the message.
第四步,流量控制软件定时读取输出卡各输出端口的流量控制模块的所有流控表,对所有丢弃标志位为0的表项的字节数域求和,计算当前输出流量,如果当前输出流量超过设定的端口流量限制,则以优先级从低到高的顺序将流控表中的若干表项的丢弃标志位置1,增加硬件丢弃的流量。如果当前流量小于端口流量限制,则以优先级从高到低的顺序将流控表中的若干表项的丢弃标志位置0,减少硬件丢弃的流量。In the fourth step, the flow control software regularly reads all the flow control tables of the flow control module of each output port of the output card, sums the byte count fields of all entries whose discard flag is 0, and calculates the current output flow. If the output traffic exceeds the set port traffic limit, the discard flags of several entries in the flow control table will be set to 1 in order of priority from low to high, increasing the traffic discarded by the hardware. If the current traffic is less than the port traffic limit, the discard flags of several entries in the traffic control table will be set to 0 in order of priority from high to low to reduce the traffic discarded by the hardware.
图8是国防科大采用本发明设计的高速网络内容监控系统的一个实例:Fig. 8 is an example of a high-speed network content monitoring system designed by the National Defense University using the present invention:
第一步,构建高速网络内容监控系统。在本发明中,输入卡使用的是OC768(40G)线卡,输出卡采用OC192(10G)线卡。OC768线卡的优先级模块采用Stratix EP2SGX130F1508C4的FPGA实现,流量控制模块和流量控制表在OC192(10G)线卡的Altera StratixⅡGX EP2S60GX的FPGA中实现,流量控制软件用C语言在linux操作系统环境下实现。输出卡的各端口与IDS,内容审计等后端分析系统相连。OC192(10G)线卡有四个输出端口,每个端口的限定输出流量为10G。The first step is to build a high-speed network content monitoring system. In the present invention, the input card uses an OC768 (40G) line card, and the output card uses an OC192 (10G) line card. The priority module of the OC768 line card is realized by the FPGA of Stratix EP2SGX130F1508C4, the flow control module and the flow control table are realized by the FPGA of Altera StratixⅡGX EP2S60GX of the OC192 (10G) line card, and the flow control software is realized by C language in the environment of the linux operating system . Each port of the output card is connected with back-end analysis systems such as IDS and content auditing. The OC192 (10G) line card has four output ports, and the limited output traffic of each port is 10G.
第二步,报文流经OC768(40G)线卡,线卡的优先级模块根据硬件DPI中用户设置的规则对报文进行五元组过滤来确定其优先级。In the second step, the message flows through the OC768 (40G) line card, and the priority module of the line card performs quintuple filtering on the message according to the rules set by the user in the hardware DPI to determine its priority.
第三步,确定了优先级的报文流经OC192(10G)线卡,OC192(10G)线卡有四个输出端口,每个输出端口都有一个流量控制模块。线卡的流量控制模块对每个输出端口都维护多个不同优先级的流控表,根据每个报文的源IP地址的后四位(n=4)和目的IP地址的后五位(k=5)共九位,将报文映射到相应的优先级的流控表中,每个流控表有29即512个表项。流量控制模块根据各端口流控表中丢弃标志位的设置对报文进行转发或丢弃操作。In the third step, the packets with determined priority flow through the OC192 (10G) line card. The OC192 (10G) line card has four output ports, and each output port has a flow control module. The flow control module of the line card maintains multiple flow control tables with different priorities for each output port, according to the last four bits of the source IP address (n=4) and the last five bits of the destination IP address ( k=5) has nine bits in total, and maps the message to the flow control table of the corresponding priority, and each flow control table has 29 or 512 entries. The flow control module forwards or discards the packets according to the setting of the discarding flag bit in the flow control table of each port.
第四步,流量控制软件每30秒按照4.1~4.4所述的流程对各端口的流控表进行统计字节数,设置标志位等操作。In the fourth step, the flow control software counts the number of bytes in the flow control table of each port, sets flags and other operations according to the procedures described in 4.1 to 4.4 every 30 seconds.
Claims (1)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201110003670.5A CN102075423B (en) | 2011-01-10 | 2011-01-10 | Hardware multi-level table-based method for controlling output traffic |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201110003670.5A CN102075423B (en) | 2011-01-10 | 2011-01-10 | Hardware multi-level table-based method for controlling output traffic |
Publications (2)
Publication Number | Publication Date |
---|---|
CN102075423A true CN102075423A (en) | 2011-05-25 |
CN102075423B CN102075423B (en) | 2013-01-02 |
Family
ID=44033775
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201110003670.5A Expired - Fee Related CN102075423B (en) | 2011-01-10 | 2011-01-10 | Hardware multi-level table-based method for controlling output traffic |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN102075423B (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103139093A (en) * | 2013-02-22 | 2013-06-05 | 桂林电子科技大学 | High speed network data flow load balancing scheduling method based on field programmable gate array (FPGA) |
CN104717101A (en) * | 2013-12-13 | 2015-06-17 | 中国电信股份有限公司 | Deep packet inspection method and system |
CN106961445A (en) * | 2017-04-28 | 2017-07-18 | 中国人民解放军信息工程大学 | Message parsing method and its device based on FPGA hardware parallel pipeline |
CN107147585A (en) * | 2017-03-31 | 2017-09-08 | 北京奇艺世纪科技有限公司 | A kind of flow control methods and device |
CN107995199A (en) * | 2017-12-06 | 2018-05-04 | 锐捷网络股份有限公司 | The port speed constraint method and device of the network equipment |
CN111200561A (en) * | 2019-12-31 | 2020-05-26 | 奇安信科技集团股份有限公司 | Data packet transmission method and apparatus, computer system and readable storage medium |
CN112087395A (en) * | 2020-08-28 | 2020-12-15 | 浪潮云信息技术股份公司 | Service type hierarchical flow control method |
CN116055423A (en) * | 2022-12-26 | 2023-05-02 | 南京中孚信息技术有限公司 | Flow parallel distribution device based on strategy control |
CN116192353A (en) * | 2022-12-16 | 2023-05-30 | 中国科学院声学研究所 | An FPGA-based multiplexer synchronous working system and method |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105763391B (en) * | 2014-12-17 | 2019-06-25 | 中国移动通信集团公司 | A kind of session data stream processing system, method and relevant device |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1905408A (en) * | 2006-08-04 | 2007-01-31 | 华为技术有限公司 | Method and apparatus for monitoring message |
US7263066B1 (en) * | 2001-12-14 | 2007-08-28 | Applied Micro Circuits Corporation | Switch fabric backplane flow management using credit-based flow control |
CN101222431A (en) * | 2008-01-23 | 2008-07-16 | 中兴通讯股份有限公司 | Cable fastener device with strong service quality function and its design method |
-
2011
- 2011-01-10 CN CN201110003670.5A patent/CN102075423B/en not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7263066B1 (en) * | 2001-12-14 | 2007-08-28 | Applied Micro Circuits Corporation | Switch fabric backplane flow management using credit-based flow control |
CN1905408A (en) * | 2006-08-04 | 2007-01-31 | 华为技术有限公司 | Method and apparatus for monitoring message |
CN101222431A (en) * | 2008-01-23 | 2008-07-16 | 中兴通讯股份有限公司 | Cable fastener device with strong service quality function and its design method |
Cited By (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103139093B (en) * | 2013-02-22 | 2016-01-27 | 桂林电子科技大学 | Based on the express network data stream load equalization scheduling method of FPGA |
CN103139093A (en) * | 2013-02-22 | 2013-06-05 | 桂林电子科技大学 | High speed network data flow load balancing scheduling method based on field programmable gate array (FPGA) |
CN104717101B (en) * | 2013-12-13 | 2018-09-14 | 中国电信股份有限公司 | Deep packet inspection method and system |
CN104717101A (en) * | 2013-12-13 | 2015-06-17 | 中国电信股份有限公司 | Deep packet inspection method and system |
CN107147585A (en) * | 2017-03-31 | 2017-09-08 | 北京奇艺世纪科技有限公司 | A kind of flow control methods and device |
CN107147585B (en) * | 2017-03-31 | 2020-02-18 | 北京奇艺世纪科技有限公司 | Flow control method and device |
CN106961445A (en) * | 2017-04-28 | 2017-07-18 | 中国人民解放军信息工程大学 | Message parsing method and its device based on FPGA hardware parallel pipeline |
CN106961445B (en) * | 2017-04-28 | 2019-10-29 | 中国人民解放军信息工程大学 | Packet parsing device based on FPGA hardware parallel pipeline |
CN107995199A (en) * | 2017-12-06 | 2018-05-04 | 锐捷网络股份有限公司 | The port speed constraint method and device of the network equipment |
CN111200561A (en) * | 2019-12-31 | 2020-05-26 | 奇安信科技集团股份有限公司 | Data packet transmission method and apparatus, computer system and readable storage medium |
CN112087395A (en) * | 2020-08-28 | 2020-12-15 | 浪潮云信息技术股份公司 | Service type hierarchical flow control method |
CN112087395B (en) * | 2020-08-28 | 2022-06-24 | 浪潮云信息技术股份公司 | Service type hierarchical flow control method |
CN116192353A (en) * | 2022-12-16 | 2023-05-30 | 中国科学院声学研究所 | An FPGA-based multiplexer synchronous working system and method |
CN116192353B (en) * | 2022-12-16 | 2023-10-13 | 中国科学院声学研究所 | An FPGA-based multiplexer synchronization working system and method |
CN116055423A (en) * | 2022-12-26 | 2023-05-02 | 南京中孚信息技术有限公司 | Flow parallel distribution device based on strategy control |
Also Published As
Publication number | Publication date |
---|---|
CN102075423B (en) | 2013-01-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN102075423B (en) | Hardware multi-level table-based method for controlling output traffic | |
US11711319B2 (en) | Methods and apparatus for flow control associated with a switch fabric | |
US20240348539A1 (en) | Method and system for providing network ingress fairness between applications | |
US10243865B2 (en) | Combined hardware/software forwarding mechanism and method | |
US8467294B2 (en) | Dynamic load balancing for port groups | |
US7916718B2 (en) | Flow and congestion control in switch architectures for multi-hop, memory efficient fabrics | |
CN101616097B (en) | Method and system for managing output port queue of network processor | |
US9065773B2 (en) | Methods and apparatus for virtual channel flow control associated with a switch fabric | |
EP2560333B1 (en) | Methods and apparatus for defining a flow control signal | |
US10778588B1 (en) | Load balancing for multipath groups routed flows by re-associating routes to multipath groups | |
US20120201140A1 (en) | Network system, controller, method, and program | |
CN1638361A (en) | Parallel data link layer controllers in a network switching device | |
US10728156B2 (en) | Scalable, low latency, deep buffered switch architecture | |
CN112242965A (en) | Telemetry event aggregation | |
JP2002044139A (en) | Router and priority control method used for it | |
CN114095448A (en) | Method and equipment for processing congestion flow | |
Avci et al. | Congestion aware priority flow control in data center networks | |
CN107528791A (en) | A kind of jamming control method and equipment | |
US12231342B1 (en) | Queue pacing in a network device | |
Prajapati et al. | Verifying Queue Length Scheme in Wired Communication for Congestion Control | |
CN116686332A (en) | Data exchange control method and device | |
CN117014376A (en) | Congestion flow identification method, device, equipment and computer readable storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C53 | Correction of patent for invention or patent application | ||
CB03 | Change of inventor or designer information |
Inventor after: Tang Yong Inventor after: Chen Shuhui Inventor after: Li Tao Inventor after: Su Jinshu Inventor after: Wang Yongjun Inventor after: Zhao Guohong Inventor after: Xuan Lei Inventor after: Liu Wenhan Inventor after: Lu Huabiao Inventor before: Tang Yong Inventor before: Chen Shuhui Inventor before: Li Tao Inventor before: Su Jinshu Inventor before: Wang Yongjun Inventor before: Zhao Guohong Inventor before: Xuan Lei Inventor before: Lu Huabiao Inventor before: Lu Huabiao |
|
COR | Change of bibliographic data |
Free format text: CORRECT: INVENTOR; FROM: TANG YONG CHEN SHUHUI LI TAO SU JINSHU WANG YONGJUN ZHAO GUOHONG XUAN LEI LU HUABIAO LU HUABIAO TO: TANG YONG CHEN SHUHUI LI TAO SU JINSHU WANG YONGJUN ZHAO GUOHONG XUAN LEI LIU WENHAN LU HUABIAO |
|
C14 | Grant of patent or utility model | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130102 Termination date: 20130110 |