CN103746982A - Automatic generation method and system for HTTP (Hyper Text Transport Protocol) network feature code - Google Patents

Abstract

The invention discloses a method for automatically generating HTTP network feature codes. The method includes: a step of generating packet feature codes, a step of generating URI feature codes, and a step of generating a total set of HTTP network feature codes. The step of generating packet feature codes is for multiple network samples The feature statistics and package content extracted from the question-and-answer package are used to generate coarse-grained cluster sets through secondary clustering, and then to generate fine-grained cluster sets through secondary clustering on the basis of coarse-grained cluster sets. Granular clustering set to generate a set of Q&A packet signatures for network samples

The URI feature code generation step is to perform supplementary extraction of URI path and parameter feature codes for the traffic that is divided into a single category in the network sample, and generate a URI feature code set

Finally, through a question and answer package feature code collection

and URI signature set

Combined to generate a total set of feature codes T _all .

Description

A method and system for automatically generating HTTP network feature codes

technical field

The invention relates to the technology in the field of network security, in particular to a method for generating signatures of unknown HTTP botnets, more specifically, a method for automatically generating HTTP network signatures and a system thereof.

Background technique

In recent years, incidents related to network security have occurred frequently, and network security has risen to become a hot topic at the national strategic level. However, due to the general lack of security awareness among netizens and various loopholes in computer operating systems and application software, more and more computers have quietly become "broilers" in botnets, and they have become targets for others to steal privacy and attack network resources. , Illegal profiteering and other illegal and criminal activities.

Botnet (Botnet) is a kind of "universal computing platform constructed by invading several non-cooperative user terminals in cyberspace, which can be remotely controlled by attackers". Among them, "non-cooperation" means that the user terminal is not aware of the intrusion; "attacker" refers to the controller (Botmaster) who has the power to control the formed botnet; "remote control" means that the attacker can communicate with The control (command and control, abbreviated as C&C) channel controls non-cooperative user terminals one-to-many. A controlled victim user terminal becomes a node of a botnet, which can be called a "zombie host", commonly known as a "broiler". Common botnet command and control protocols mainly include three types: IRC, HTTP, and P2P. Because the HTTP protocol has good penetration and centralized control, more and more botnet controllers use the HTTP protocol as their communication and control protocol. The controller controls a large number of zombie hosts through the botnet, and can obtain powerful distributed computing capabilities and abundant information resource reserves. It is easier for attackers to launch distributed denial of service attacks (DDoS), online identity theft (Online Identity Theft), spam (Spam), click fraud (Click Fraud), Bitcoin mining (BitCoin Mining) and other malicious behaviors. As the most effective general-purpose attack platform in the hands of attackers, botnets have become one of the biggest security threats to the Internet today.

The reason why botnets are such a big threat is mainly due to the following reasons:

Botnet is a new form of attack derived from traditional worms and Trojan horses. Worms have the advantage of taking advantage of security holes to spread quickly but are uncontrollable; Trojan horses have the ability to remotely control victims, but they have the disadvantages of slow infection speed, small management scale and simple control methods. The botnet is a product formed by combining the advantages of the two and making up for the shortcomings of the two, and is more harmful.

Botnet has the characteristics of high controllability and separation of control logic and attack. The "broilers" in the botnet can be manipulated by the controller through the command and control (command and control) channel, and can launch large-scale attacks (DDoS attacks, etc.) on a specific target in a short period of time, with a high degree of controllability . In addition, the bot program on the bot host is responsible for the control logic, and the real attack tasks are dynamically distributed by the controller on demand. This method can divide the complete threat entity into multiple parts, which can not only provide good flexibility for task distribution, but also improve the survivability of botnets.

Security measures often lag behind the emergence of corresponding new botnets. The detection method based on signature is an effective method. However, most of the traditional signature generation techniques are only for worms, and these technologies cannot efficiently and automatically generate high-quality signatures, so they cannot effectively control the botnet at the initial stage of expansion.

At present, there are many detection methods and systems for botnets, but most of these system detections have problems such as high time consumption and difficulty in application deployment, and cannot be widely promoted in a real sense; although traditional intrusion detection systems (IDS) have a wide range of applications, It can be used to effectively discover abnormal network behaviors in a specific network. However, due to the lack of corresponding botnet signatures and corresponding rules, it is impossible to discover potential new botnet hosts in a specific network in time. There are mainly the following problems in the current feature code extraction technology:

Most of the traditional signature generation algorithms are only for worms, and there is no signature generation method for HTTP botnets. Most of the existing signature generation methods are aimed at the extraction of worm signatures. Due to the different characteristics of botnet command and control communication, these traditional signature generation methods are not suitable for HTTP botnet signatures. extraction.

The existing signature generation method has low efficiency and high time consumption. Traditional feature code generation mostly relies on manual judgment and cannot be automated on a large scale. Although a few people have proposed automatic extraction methods for botnet signatures in an attempt to solve this problem, these methods are computationally expensive and cannot be widely applied.

The signatures generated by existing methods are of low quality and poor usability. The traditional signature generation method does not consider the command and control communication characteristics of HTTP botnets, the signature generation method adopted is not targeted, and the generated signature sets are large in number and low in quality.

Contents of the invention

The technical problem to be solved by the present invention is to overcome the problems of long generation time and difficult deployment of existing system feature codes, and propose a method and system for automatically generating HTTP network feature codes.

For reaching above-mentioned purpose, the present invention provides a kind of HTTP network feature code automatic generation method, it is characterized in that, described method comprises:

Packet feature code generation step: for the feature statistics and packet content extracted from the one-question-one-answer packets of multiple network samples, generate a coarse-grained cluster set through secondary clustering, and then based on the coarse-grained cluster set Secondary clustering generates a fine-grained clustering set, and generates a Q&A packet feature code set of the network sample through the fine-grained clustering set

URI feature code generation step: for the traffic that is divided into a single category in the network sample, perform supplementary extraction of URI path and parameter feature codes, and generate a feature code set of the URI

和所述URI的特征码集合

合并生成特征码总集合T_all。The generation step of the total collection of HTTP network signatures: through the collection of the one-question-and-answer packet signatures

and the set of signatures for the URI

Combined to generate a total set of feature codes T _all .

The above-mentioned HTTP network feature code automatic generation method is characterized in that, the packet feature code generation step includes:

Data extraction step: extracting the statistics of the data flow characteristics of the network sample and the content of the question and answer package;

Secondary clustering step: perform secondary clustering according to the statistics of the characteristics of the network samples and the content of the Q&A packet, and generate the fine-grained clustering set on the basis of generating the coarse-grained clustering set;

The step of generating the feature code of the question-and-answer packet: according to the fine-grained clustering set, respectively generate the feature code sets of the request packet and the response packet.

The above-mentioned HTTP network feature code automatic generation method is characterized in that, before the data extraction step, it also includes:

Whitelist filtering step: filtering and removing traffic accessing legitimate websites in the network samples.

The above-mentioned HTTP network feature code automatic generation method is characterized in that, the data extraction step also includes:

Data content extraction step: extracting the content of the one-question-one-answer packet of the HTTP session connection;

Coarse-grained clustering attribute extraction step: taking the network sample as a unit, extract the four-dimensional statistical value of the coarse-grained clustering, including: the total number of HTTP data streams, the number of bytes sent per second, the average size of HTTP data packets, and the HTTP data The total number of packages to get the coarse-grained clustering attributes;

Fine-grained clustering attribute extraction step: taking each HTTP session as a unit, extract the four-dimensional statistical value of the fine-grained clustering, including: the number of session request packets, the number of session response packets, the size of the first request packet, the first Respond to the package size to get fine-grained clustering attributes;

所述五元组的格式为：<样本id，会话id，一问一答包内容，粗粒度聚类属性，细粒度聚类属性>。Summarize the data set step: summarize the content of the question and answer package, the coarse-grained clustering attribute and the fine-grained clustering attribute to obtain a five-tuple data set

The format of the five-tuple is: <sample id, session id, Q&A package content, coarse-grained clustering attribute, fine-grained clustering attribute>.

The above-mentioned HTTP network feature code automatic generation method is characterized in that, the secondary clustering step also includes:

自动对所述粗粒度聚类属性进行聚类，得到粗粒度聚类集C，如果所述粗粒度聚类集C只属于一个所述网络样本，则执行所述URI特征码生成步骤；Coarse-grained clustering step: on the five-tuple dataset

Automatically clustering the coarse-grained clustering attributes to obtain a coarse-grained clustering set C, if the coarse-grained clustering set C only belongs to one of the network samples, then perform the URI signature generation step;

Fine-grained clustering step: based on the coarse-grained clustering set C, for all sessions in each c _i ( _ci ∈ C), automatically install the fine-grained clustering attributes for clustering, and obtain fine-grained Clustering set C′ (C′∈C _i );

Sample coverage judgment step: if there is a fine-grained clustering c _i ′ (ci _′ ∈C′) where all the conversations come from k samples, and the value of k is greater than 1 and less than or equal to the number of network samples, then it is considered The fine-grained clustering is successful; otherwise, the step of generating the URI feature code is performed.

The above-mentioned HTTP network feature code automatic generation method is characterized in that, the question and answer packet feature code generation step also includes:

HTTP feature code set generation step: generate the feature codes of request packets and response packets for all session connections in each fine-grained cluster c _i ′ (ci _′ ∈C’), and automatically calculate the token features sequentially Finally, each fine-grained cluster c _i ′ obtains a signature of a request packet and a signature of a response packet respectively, forming an HTTP signature set W;

Feature code filtering step: filter and screen the HTTP feature code set W, remove the unqualified feature codes, merge the repeated feature codes, and obtain the Q&A packet feature code set

The present invention also provides a system for automatically generating HTTP network feature codes using the method for automatically generating network features, wherein the system includes:

Packet feature code generation module: used for feature statistics and packet content extracted from the one-question-one-answer packets of multiple network samples, generate coarse-grained cluster sets through secondary clustering, and then generate coarse-grained cluster sets in the coarse-grained cluster sets Based on the secondary clustering, a fine-grained clustering set is generated, and a Q&A package feature code set of the network sample is generated through the fine-grained clustering set

URI feature code generation module: for the traffic that is divided into a single category in the network sample, perform supplementary extraction of URI path and parameter feature codes, and generate a feature code set of the URI

和所述URI的特征码集合

合并生成特征码总集合T_all。HTTP network feature code total set generation module: through the question and answer packet feature code set

and the set of signatures for the URI

Combined to generate a total set of feature codes T _all .

The above-mentioned HTTP network signature automatic generation system is characterized in that, the packet signature generation module includes:

Whitelist filtering module: filter and remove traffic that visits legitimate websites;

Data extraction module: extract the statistics of the data flow characteristics of the network sample and the content of the question and answer package;

Secondary clustering module: performing secondary clustering according to the network sample characteristic statistics and the content of the question-and-answer package, and generating the fine-grained clustering set on the basis of generating the coarse-grained clustering set;

A question-and-answer packet signature generation module: according to the fine-grained clustering set, respectively generate signature sets of request packets and response packets.

The above-mentioned HTTP network feature code automatic generation system is characterized in that, before the data extraction module, it also includes:

Whitelist filtering module: filtering and removing the traffic of accessing legitimate websites in the network samples.

The above-mentioned HTTP network feature code automatic generation system is characterized in that, the data extraction module also includes:

Data content extraction module: extract the content of the said Q&A package of HTTP session connection;

Coarse-grained clustering attribute extraction module: taking the network sample as a unit, extract the four-dimensional statistical value of the coarse-grained clustering, including: the total number of HTTP data streams, the number of bytes sent per second, the average size of HTTP data packets, and the HTTP data The total number of packages to get the coarse-grained clustering attribute;

Fine-grained clustering attribute extraction module: taking each HTTP session as a unit, extract the four-dimensional statistical value of the fine-grained clustering, including: the number of session request packets, the number of session response packets, the size of the first request packet, the first Respond to the package size to get fine-grained clustering attributes;

所述五元组的格式为：<样本id，会话id，一问一答包内容，粗粒度聚类属性，细粒度聚类属性>。Summarize data set module: summarize the content of the question and answer package, the coarse-grained clustering attribute and the fine-grained clustering attribute to obtain a five-tuple data set

The format of the five-tuple is: <sample id, session id, Q&A package content, coarse-grained clustering attribute, fine-grained clustering attribute>.

The above-mentioned HTTP network feature code automatic generation system is characterized in that the secondary clustering module also includes:

自动对所述粗粒度聚类属性进行聚类，得到粗粒度聚类集C，如果所述粗粒度聚类集C只属于一个所述网络样本，则通过所述URI特征码生成模块生成所述URI特征码；Coarse-grained clustering module: for the five-tuple data set

automatically clustering the coarse-grained clustering attributes to obtain a coarse-grained clustering set C, if the coarse-grained clustering set C only belongs to one of the network samples, the URI signature generation module generates the URI signature;

Fine-grained clustering module: based on the coarse-grained clustering set C, for all sessions in each c _i ( _ci ∈ C), automatically install the fine-grained clustering attributes for clustering, and obtain fine-grained Clustering set C′ (C′∈C _i );

Sample coverage judgment module: if there is a fine-grained clustering c _i ′ (ci _′ ∈C’) where all the conversations come from k samples, and the value of k is greater than 1 and less than or equal to the number of network samples, then it is considered The fine-grained clustering is successful; otherwise, the URI feature code is generated by the URI feature code generation module.

The above-mentioned HTTP network feature code automatic generation system is characterized in that the question and answer packet feature code generation module also includes:

HTTP feature code set generation module: generate request packet and response packet feature codes for all session connections in each fine-grained cluster c _i ′ (ci _′ ∈ C ′), and automatically calculate token features sequentially Finally, each fine-grained cluster c _i ′ obtains a signature of a request packet and a signature of a response packet respectively, forming an HTTP signature set W;

Feature code filtering module: filter and screen the HTTP feature code set W, remove unqualified feature codes, merge repeated feature codes, and obtain the Q&A package feature code set

Compared with the prior art, the present invention aims at the statistical similarity of HTTP botnet command and control communication data and the principle that one question and one answer contains most of the characteristic information of botnets, and proposes an HTTP protocol based on one question and one answer packet. A method for automatically generating botnet signatures. This method extracts the Q&A packet and related statistical characteristics of the HTTP communication data of the host, performs secondary clustering on the HTTP data through the X-means clustering algorithm, and utilizes the longest common subsequence algorithm and the feature method based on URI Generate signatures.

The present invention has the following beneficial effects:

1. Can automatically extract the communication characteristic code of HTTP botnet;

2. Improve the efficiency of feature code generation and shorten the time and space overhead;

3. The robustness and adaptability of the signature generation system are improved, and the high-quality signatures generated can cooperate with intrusion detection systems such as snort to realize the detection of corresponding botnets in a wide range.

Description of drawings

Fig. 1 is the schematic flow chart of HTTP network characteristic code automatic generation method flow chart of the present invention;

Fig. 2 is the detailed schematic flow chart of HTTP network feature code automatic generation method of the present invention;

FIG. 3 is a schematic structural diagram of the system for automatically generating HTTP network feature codes according to the present invention.

Among them, reference signs:

1 pack signature generation module 2 URI signature generation module

3HTTP network signature total set generation module

11 Whitelist filtering module 12 Data extraction module

13 Quadratic clustering module 14 One question and one answer package feature code generation module

121 Question and answer package extraction module 122 Coarse-grained clustering attribute extraction module

123 Fine-grained clustering attribute extraction module 124 Summary data set module

131 Coarse-grained clustering module 132 Fine-grained clustering module

133 sample coverage judgment module

141 HTTP signature set generation module 142 Feature code filtering module

S1～S3, S11～S14, S121～S124, S131～S133, S141～S142: implementation steps of each embodiment of the present invention

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, but not as a limitation of the present invention.

The purpose of the present invention is to classify numerous HTTP botnet samples, and automatically generate corresponding feature codes for detection. The advantage of the present invention is that the communication feature code of the botnet can be generated without any prior knowledge, and even the feature code of the botnet with encrypted communication content can be generated.

Field of application of the present invention: 1. Propose a kind of efficient method of automatically generating HTTP botnet characteristic code for realizing the detection of large-scale botnet; 2. In the research of botnet, according to its network behavior to different samples Classify and automatically extract signatures.

The invention proposes a method for automatically generating HTTP network characteristic codes, which is based on a question-and-answer package and can accurately and automatically extract HTTP botnet characteristic codes. This method is based on the network behavior analysis of a large number of botnet samples, using the Q&A packet (the first request and the first response HTTP data packet) in the HTTP session connection as the object of feature code extraction, drawing on the longest common subsequence algorithm ( Longest Common Subsequence abbreviated as LCS) to automatically and efficiently generate high-quality HTTP botnet signatures. The present invention designs a set of feature code automatic generation system based on one question one answer packet based on the similarity principle of HTTP botnet command and control communication data.

As shown in Fig. 1 and Fig. 2, the method for automatically generating a network feature code provided by the present invention, the specific steps include:

Packet signature generation step S1: Aiming at the feature statistics and packet content extracted from the Q&A packets of multiple network samples, a coarse-grained cluster set is generated through secondary clustering, and then based on the coarse-grained cluster set, two Sub-clustering generates fine-grained clustering sets, and generates a set of Q&A packet signatures for network samples through fine-grained clustering sets

The feature code generation for Q&A packets, according to a large number of statistics, found that the connection duration of the command and control communication of the botnet is short, and most of the valuable feature content in the communication (zombie host information, requested binary file name , attack commands, etc.) are all concentrated in the Q&A packet (the first request and the first response HTTP packet) of the HTTP session connection. Therefore, the question-and-answer packet of HTTP is used as the signature generation object. The method can greatly reduce the overhead of data packet storage and comparison calculation, and can improve the efficiency of feature code generation.

Compared with the mainstream feature code generation technology (Polygraph, Autograph, etc.), the present invention proposes to calculate the question-and-answer data packets instead of all HTTP data packets for the communication characteristics of HTTP botnets. Compared with traditional methods, this method The generation efficiency of the feature code is improved, and the double overhead of operation time and storage space is reduced.

The present invention adopts high-efficiency secondary clustering. In the present invention, the classic X-means algorithm is used to perform coarse-grained and fine-grained secondary clustering on the statistical characteristics of the sample data stream and the content of the Q&A package of the conversation. . In coarse-grained clustering, the total number of HTTP data streams, the number of bytes sent per second, the average size of HTTP data packets, and the total number of HTTP data packets are selected as the four-dimensional clustering attributes of coarse-grained clustering in units of samples. Aggregate samples with similar network behaviors (assuming they belong to the same class of botnets); in fine-grained clustering, take the HTTP session connection of the sample as a unit, and classify all sessions in each class on the basis of coarse-grained clustering The connection performs fine-grained clustering, and selects the number of session request packets, the number of session response packets, the size of the first request packet, and the size of the first response packet as the four-dimensional clustering attributes of fine-grained clustering. Fine-grained clustering can combine similar aggregate data packets together to generate high-quality feature codes; this secondary clustering method can conveniently and effectively aggregate data packets with similar content together without knowing the communication content, reducing the number of packets between a large number of packets. tedious comparison calculations.

The coarse-grained and fine-grained secondary clustering methods of the present invention can quickly divide data packets with similar statistical characteristics into the same cluster, and improve the speed of feature code generation. This division method does not require prior knowledge and does not depend on specific content, avoiding the time overhead caused by pairwise comparison of a large number of data packets.

URI signature generation step S2: Aiming at traffic that is divided into a single category in the network sample, perform supplementary extraction of URI path and parameter signatures to generate a URI signature set

URI feature code generation step S2 In the process of clustering many sample traffic, it is often encountered that the traffic of one or several samples is divided into a class separately. In this case, a supplementary method is used: Analyze the URI of the start line of the request packet of the sample, and extract the path and request parameters as the feature code of the sample. This improves the robustness and adaptability of the feature code extraction system to a certain extent.

For single sample clustering, sample data that fails in fine-grained clustering, and fails to generate a question-and-answer package feature code will be sent to the URI feature code generation step S2, and the URI path based on the starting line of the HTTP request package (in the first line) A ? sign is the end mark) and parameter (parameter name submitted in the URI) feature code extraction: take a sample as a unit, check all the request packets of the sample, and extract the path of the starting line and the parameter set. For example, if the starting line content is GET/weather/getweather.aspx?t=1377511384901&cityno=HTTP/1.1 data packet, the extracted path is /weather/getweather.aspx, and the parameters are t and cityno. The token feature code is recorded as /weather/getweather.aspx.*t.*cityno. Finally, the URI feature code set of these samples will be obtained, denoted as

The URI path and parameter feature extraction introduced by the invention effectively solves the failure of single-sample clustering in the traditional feature code extraction method, and improves the robustness and adaptability of the system to a certain extent.

和所述URI的特征码集合

合并生成特征码总集合T_all。Step S3 of generating the total collection of HTTP network signatures: through the set of signatures of a question-and-answer packet

and the set of signatures for the URI

Combined to generate a total set of feature codes T _all .

合并得到了最终的特征码集合T_all。同时，在同一粗粒度聚类中，且拥有公共的“代表性细粒度聚类”的样本之间属于同一类僵尸网络。Question and answer package feature code collection Set with URI signature

Combined to obtain the final feature code set T _all . At the same time, in the same coarse-grained cluster, samples that have a common "representative fine-grained cluster" belong to the same type of botnet.

Wherein, the packet signature generation step S1 also includes:

Whitelist filtering step S11: filtering and removing the traffic of accessing legitimate websites;

The HTTP data of the botnet sample first enters the "white list filtering module". In order to resist detection, botnet controllers mix legitimate request data (such as accessing Google, Baidu) in the command and control communication flow with the intention of interfering with detection and signature generation. Therefore, in order not to affect the quality of signature code generation, HTTP traffic to legitimate websites is filtered out according to the third-party authoritative website ranking (such as the top 500 ALEX websites), and the filtered HTTP data is transferred to the "data extraction module" for processing.

Data extraction step S12: extract the statistics of the data flow characteristics of the network sample and the content of the question and answer package;

Secondary clustering step S13: Carry out secondary clustering according to the network sample feature statistics and the content of the Q&A packet, and generate a fine-grained clustering set on the basis of generating a coarse-grained clustering set;

Step S14 of generating the feature code of the question-and-answer packet: according to the fine-grained clustering set, respectively generate the feature code sets of the request packet and the response packet.

Wherein, the data extraction step S12 also includes:

Data content extraction step S121: extract the content of the Q&A package connected by the HTTP session;

Coarse-grained clustering attribute extraction step S122: taking the network sample as a unit, extract the four-dimensional statistical value of coarse-grained clustering, including: the total number of HTTP data streams, the number of bytes sent per second, the average size of HTTP data packets, and the total number of HTTP data packets, Get coarse-grained clustering attributes;

Fine-grained clustering attribute extraction step S123: Taking each HTTP session as a unit, extract the four-dimensional statistical value of fine-grained clustering, including: the number of session request packets, the number of session response packets, the size of the first request packet, and the first response Packet size, get fine-grained clustering attributes;

Summarize the dataset step S124: Summarize the contents of the Q&A package, coarse-grained clustering attributes, and fine-grained clustering attributes to obtain a five-tuple dataset The format of the five-tuple is: <sample id, session id, Q&A package content, coarse-grained clustering attribute, fine-grained clustering attribute>.

其格式为<样本id，会话id，一问一答包内容，粗粒度聚类属性，细粒度聚类属性>：其中”样本id“唯一标示不同的僵尸网络样本（数据来源），该标示并不代表僵尸网络的种类，例如在同一个局域网中A、B两台主机被同一僵尸网络所控制，两者的样本id不同；会话id用于唯一标示样本数据中的某个HTTP的会话连接。提取完毕后将五元数据集

传入二次聚类步骤S13。In the data extraction step S12, data flow feature statistics and data packet content extraction are performed on the HTTP data of each sample, which are mainly divided into three parts: 1. Extract the Q&A packet of the HTTP session connection (the first request and the first content of each response HTTP data packet); second, taking the network sample as a unit, extract the four-dimensional statistical value of coarse-grained clustering, including the total number of HTTP data streams, the number of bytes sent per second, the average size of HTTP data packets, and the total number of HTTP data packets ; 3. Taking the session connection as the unit, extract the four-dimensional statistical value of fine-grained clustering, including the number of session request packets, the number of session response packets, the size of the first request packet, and the size of the first response packet. The three parts can be performed concurrently, resulting in a quintuple dataset

Its format is <sample id, session id, Q&A package content, coarse-grained clustering attribute, fine-grained clustering attribute>: where "sample id" uniquely identifies a different botnet sample (data source), and the identifier is not It does not represent the type of botnet. For example, in the same LAN, two hosts A and B are controlled by the same botnet, and the sample ids of the two are different; the session id is used to uniquely identify a certain HTTP session connection in the sample data. After the extraction is complete, the five-element data set

Into the secondary clustering step S13.

Wherein, the secondary clustering step S13 also includes:

自动对粗粒度聚类属性进行聚类，得到粗粒度聚类集C，如果粗粒度聚类集C只属于一个网络样本，则执行URI特征码生成步骤S2；Coarse-grained clustering step S131: for the five-tuple data set

Automatically cluster the coarse-grained clustering attributes to obtain the coarse-grained clustering set C, if the coarse-grained clustering set C only belongs to one network sample, then execute the URI feature code generation step S2;

Fine-grained clustering step S132: based on the coarse-grained clustering set C, for all sessions in each c _i ( _ci ∈ C), automatically install fine-grained clustering attributes for clustering to obtain a fine-grained clustering set C'(C'∈C _i );

Sample coverage judgment step S133: If there is a fine-grained clustering c _i ′ (ci _′ ∈C’) where all conversations come from k samples, and the value of k is greater than 1 and less than or equal to the number of network samples, it is considered fine-grained If the granularity clustering is successful, otherwise execute the URI feature code generation step S2.

进行粗粒度聚类，聚类算法采用公开的X-means算法，根据四维粗粒度属性值（HTTP数据流总数、每秒发送字节数、HTTP数据包平均大小、HTTP数据包总数）对样本进行聚类，得到粗粒度聚类集C。将只存在单一样本的聚类删除，把其对应的五元数据集

执行URI特征码生成步骤S2。然后在粗粒度的基础上以聚类c_i(c_i∈C)为单位，对每个粗粒度聚类中的所有样本的所有会话连接按照四维细粒度属性值（会话请求包个数、会话响应包个数、首个请求包大小、首个响应包大小）进行聚类，聚类算法依旧为X-means。每一个粗粒度聚类中将会产生新的细粒度聚类集C′（C′∈C_i）。检查C′中每个细粒度聚类的会话连接来源情况，假设c_i′∈C′，如果c_i′中的会话连接来源于至少k个不同的样本（k小于等于该粗粒度聚类c_i中样本个数，大于1，小于等于网络样本个数，具体数值可自由设定），则这样的细粒度聚类c_i′满足要求。对应的样本具有“代表性的聚类”；否则，由于没有涵盖足够多的样本，这样的细粒度聚类不具有代表性。如果某粗粒度类C_i中某个样本（或者多个样本）不存在任何“具有代表性”的细粒度聚类（即没有涵盖足够多的样本数量），对这些样本的细粒度聚类失败，认为没有找到与它们相似的且数量足够多的样本，将这些样本相关的数据集传入”URI特征码生成模块”。将满足要求的细粒度聚类c_i′执行一问一答包特征码生成步骤S14。First, for the dataset

Carry out coarse-grained clustering. The clustering algorithm adopts the public X-means algorithm. According to the four-dimensional coarse-grained attribute values (the total number of HTTP data streams, the number of bytes sent per second, the average size of HTTP data packets, and the total number of HTTP data packets), the samples are analyzed. Clustering to obtain a coarse-grained clustering set C. Delete the clusters that only have a single sample, and delete the corresponding five-element data set

Execute the step S2 of generating the URI feature code. Then, on the basis of coarse-grainedness, clustering c _i ( _ci ∈ C) is used as the unit, and all session connections of all samples in each coarse-grained cluster are classified according to the four-dimensional fine-grained attribute values (number of session request packets, session The number of response packets, the size of the first request packet, and the size of the first response packet) are clustered, and the clustering algorithm is still X-means. Each coarse-grained cluster will generate a new fine-grained cluster set C′ (C′∈C _i ). Check the source of session connections for each fine-grained cluster in C′, assuming c _i ∈ C′, if the session connections in c _i ′ come from at least k different samples (k is less than or equal to the coarse-grained cluster c The number of samples in _i is greater than 1 and less than or equal to the number of network samples, the specific value can be set freely), then such fine-grained clustering c _i ′ meets the requirements. The corresponding samples have a "representative cluster"; otherwise, such fine-grained clusters are not representative because they do not cover enough samples. If there is no "representative" fine-grained cluster for a sample (or multiple samples) in a certain coarse-grained class C _i (that is, it does not cover a sufficient number of samples), the fine-grained clustering of these samples fails , it is considered that there are no samples similar to them and a sufficient number, and the data sets related to these samples are passed into the "URI signature generation module". The fine-grained clustering c _i ′ that meets the requirements is executed in step S14 of generating a question-and-answer package feature code.

Wherein, step S14 of generating the feature code of the question and answer package also includes:

HTTP feature code set generation S141: generate request packet and response packet feature codes for all session connections in each fine-grained cluster c _i ′ (ci _′ ∈C’), and automatically calculate token feature codes in turn, Finally, each fine-grained clustering c _i ′ respectively obtains a signature of a request packet and a signature of a response packet to form an HTTP signature set W;

Generate signatures for all session connections in each fine-grained cluster c _i ′. According to the question-and-answer packet, it is divided into request packet signature generation and response packet signature generation. The longest common subsequence algorithm (LCS) is used as The feature code generation algorithm generates a token feature code (like t ₁ .*t ₂ .*t ₃ .*t ₄ , t _i represents a common character string, .* represents a spacer, indicating that there is a does not match the string). The comparison and calculation process is as follows: Assume that there are four session connections a, b, c, and d. First, the request packets of a and b are calculated by LCS to obtain the token feature code t. Remove all .* from t and convert it to text format. The request packet of c is calculated to obtain the token feature code s, and s is converted into a text format, and finally calculated with the content of the request packet of d to obtain the final request packet feature code w; the same is true for the feature code calculation of the response packet. After calculation, each fine-grained clustering c _i ′ will generate a feature code of a request packet and a feature code of a response packet. These feature codes will be summarized and sorted, and the sample id involved will be marked. Each coarse-grained cluster c _i will get a feature code set W.

Feature code filtering step S142: filter and screen the HTTP feature code set W, remove unqualified feature codes, merge repeated feature codes, and obtain a question-and-answer packet feature code set

在过滤过程中可能存在某样本的特征码因为不符合要求（过短或者均为）而被全部删除，这样的样本被认为是生成一问一答特征码失败，同样被执行URI特征码生成步骤S2。Perform corresponding filtering on the feature code set W of the generated question and answer package. First, delete the public string t _i in the token feature code that is too short (for example, the length is less than 4); Filter the public strings contained in the signature, and filter the common HTTP header fields and some content that often appear in legitimate data packets (such as HTTP/1.1, Cache-Control: no-cache, etc.); finally Deduplicated and merged the duplicate token signatures to obtain the signature collection of the final question-and-answer package botnet

During the filtering process, some sample signatures may be deleted because they do not meet the requirements (too short or both). Such samples are considered to have failed to generate a Q&A signature, and the URI signature generation step will also be performed. S2.

The invention adopts automatic generation of characteristic codes, and the generated characteristic codes are of high quality, and can be combined with intrusion detection systems such as snort to realize extensive detection of corresponding botnets.

The present invention also provides a system for automatically generating HTTP network signatures, which can be deployed separately in a server or host (such as a honeypot host) to obtain all HTTP data generated by botnet samples; or deploy the system in a specified The gateway position of the network, linked with the botnet detection system on the network border, reads the botnet HTTP data stored in the background database of the detection system.

A kind of HTTP network characteristic code automatic generation system, as shown in Figure 3, comprises: package characteristic code generation module 1, URI characteristic code generation module 2 and HTTP network characteristic code total set generation module 3;

Packet feature code generation module 1: used for feature statistics and packet content extracted from the Q&A packets of multiple network samples, generate coarse-grained clustering sets through secondary clustering, and then generate coarse-grained clustering sets in the coarse-grained clustering sets On the basis of secondary clustering, a fine-grained cluster set is generated, and a Q&A package signature set of network samples is generated through the fine-grained cluster set

URI feature code generation module 2: used for supplementary extraction of URI path and parameter feature codes for the traffic that is divided into a single category in the network sample, and generate a URI feature code set

和URI的特征码集合

合并生成特征码总集合T_all。HTTP network feature code total set generation module 3: through a question and answer packet feature code set

and URI signature set

Combined to generate a total set of feature codes T _all .

Among them, the packet signature generation module 1 includes:

Whitelist filtering module 11: filtering and removing the traffic of visiting legitimate websites;

Data extraction module 12: extract the statistics of the data flow characteristics of the network sample and the content of the question and answer package;

Secondary clustering module 13: perform secondary clustering according to network sample feature statistics and Q&A package content, and generate fine-grained clustering sets on the basis of generating coarse-grained clustering sets;

Question-and-answer package feature code generating module 14: according to the fine-grained clustering set, respectively generate feature code sets of request packets and response packets.

Wherein, the data extraction module 12 also includes:

Data content extraction module 121: extract the content of the question-and-answer packet of the HTTP session connection;

Coarse-grained clustering attribute extraction module 122: take the network sample as a unit, extract the four-dimensional statistical value of coarse-grained clustering, including: total number of HTTP data streams, number of bytes sent per second, average size of HTTP data packets and total number of HTTP data packets, Get coarse-grained clustering attributes;

Fine-grained clustering attribute extraction module 123: taking each HTTP session as a unit, extract the four-dimensional statistical value of fine-grained clustering, including: the number of session request packets, the number of session response packets, the size of the first request packet, and the first response Packet size, get fine-grained clustering attributes;

Summarize the data set module 124: summarize the content of the question and answer package, the coarse-grained clustering attribute and the fine-grained clustering attribute to obtain a five-tuple data set The format of the five-tuple is: <sample id, session id, Q&A package content, coarse-grained clustering attribute, fine-grained clustering attribute>.

Wherein, the secondary clustering module 13 also includes:

自动对粗粒度聚类属性进行聚类，得到粗粒度聚类集C，如果粗粒度聚类集C只属于一个网络样本，则通过URI特征码生成模块生成URI特征码；Coarse-grained clustering module 131: For quintuple datasets

Automatically cluster the coarse-grained clustering attributes to obtain a coarse-grained clustering set C, if the coarse-grained clustering set C only belongs to one network sample, then generate a URI signature through the URI signature generation module;

Fine-grained clustering module 132: based on the coarse-grained clustering set C, for all sessions in each c _i ( _ci ∈ C), automatically install fine-grained clustering attributes for clustering, and obtain a fine-grained clustering set C'(C'∈C _i );

Sample coverage judging module 133: if there is a fine-grained clustering c _i ′ (ci _′ ∈C’) where all conversations come from k samples, and the value of k is greater than 1 and less than or equal to the number of network samples, it is considered fine-grained The granularity clustering is successful; otherwise, the URI signature code is generated by the URI signature code generation module.

Wherein, the question and answer package feature code generation module 14 also includes:

HTTP feature code set generation module 141: generate request packet and response packet feature codes for all session connections in each fine-grained cluster c _i ′ (ci _′ ∈C’), and automatically calculate token feature codes in turn , and finally each fine-grained clustering c _i ′ respectively obtains a signature of a request packet and a signature of a response packet to form an HTTP signature set W;

Feature code filtering module 142: filter and screen the HTTP feature code set W, remove unqualified feature codes, merge repeated feature codes, and obtain a question-and-answer packet feature code set

Of course, the present invention can also have other various embodiments, and those skilled in the art can make various corresponding changes and deformations according to the present invention without departing from the spirit and essence of the present invention. All changes and deformations should belong to the protection scope of the appended claims of the present invention.

Claims (12)

1. an HTTP network feature code automatic generation method, is characterized in that, described method comprises: Packet feature code generation step: for the feature statistics and packet content extracted from the one-question-one-answer packets of multiple network samples, generate a coarse-grained cluster set through secondary clustering, and then based on the coarse-grained cluster set Secondary clustering generates a fine-grained clustering set, and generates a Q&A packet feature code set of the network sample through the fine-grained clustering set

URI feature code generation step: for the traffic that is divided into a single category in the network sample, perform supplementary extraction of URI path and parameter feature codes, and generate a feature code set of the URI The generation step of the total collection of HTTP network signatures: through the collection of the one-question-and-answer packet signatures

and the set of signatures for the URI

Combined to generate a total set of feature codes T _all . 2. according to the described HTTP network characteristic code automatic generation method of claim 1, it is characterized in that, described package characteristic code generation step comprises: Data extraction step: extracting the statistics of the data flow characteristics of the network sample and the content of the question and answer package; Secondary clustering step: perform secondary clustering according to the statistics of the characteristics of the network samples and the content of the Q&A packet, and generate the fine-grained clustering set on the basis of generating the coarse-grained clustering set; The step of generating the feature code of the question-and-answer packet: according to the fine-grained clustering set, respectively generate the feature code sets of the request packet and the response packet. 3. according to the described HTTP network feature code automatic generation method of claim 2, it is characterized in that, also comprise before described data extracting step: Whitelist filtering step: filtering and removing traffic accessing legitimate websites in the network samples. 4. according to the described HTTP network feature code automatic generation method of claim 2, it is characterized in that, described data extracting step, also comprises: Data content extraction step: extracting the content of the one-question-one-answer packet of the HTTP session connection; Coarse-grained clustering attribute extraction step: taking the network sample as a unit, extract the four-dimensional statistical value of the coarse-grained clustering, including: the total number of HTTP data streams, the number of bytes sent per second, the average size of HTTP data packets, and the HTTP data The total number of packages to get the coarse-grained clustering attribute; Fine-grained clustering attribute extraction step: taking each HTTP session as a unit, extract the four-dimensional statistical value of the fine-grained clustering, including: the number of session request packets, the number of session response packets, the size of the first request packet, the first Respond to the package size to get fine-grained clustering attributes; Summarize the data set step: summarize the content of the question and answer package, the coarse-grained clustering attribute and the fine-grained clustering attribute to obtain a five-tuple data set The format of the five-tuple is: <sample id, session id, Q&A package content, coarse-grained clustering attribute, fine-grained clustering attribute>. 5. according to the described HTTP network feature code automatic generation method of claim 2, it is characterized in that, described secondary clustering step, also comprises: Coarse-grained clustering step: on the five-tuple dataset

Automatically clustering the coarse-grained clustering attributes to obtain a coarse-grained clustering set C, if the coarse-grained clustering set C only belongs to one of the network samples, then perform the URI signature generation step; Fine-grained clustering step: based on the coarse-grained clustering set C, for all sessions in each c _i ( _ci ∈ C), automatically install the fine-grained clustering attributes for clustering, and obtain fine-grained Clustering set C′ (C′∈C _i ); Sample coverage judgment step: if there is a fine-grained clustering c _i ′ (ci _′ ∈C′) where all the conversations come from k samples, and the value of k is greater than 1 and less than or equal to the number of network samples, then it is considered The fine-grained clustering is successful; otherwise, the step of generating the URI feature code is performed. 6. according to the described HTTP network characteristic code automatic generation method of claim 2, it is characterized in that, described one question one answer packet characteristic code generation step, also comprises: HTTP feature code set generation step: generate the feature codes of request packets and response packets for all session connections in each fine-grained cluster c _i ′ (ci _′ ∈C’), and automatically calculate the token features sequentially Finally, each fine-grained cluster c _i ′ obtains a signature of a request packet and a signature of a response packet respectively, forming an HTTP signature set W; Feature code filtering step: filter and screen the HTTP feature code set W, remove the unqualified feature codes, merge the repeated feature codes, and obtain the Q&A packet feature code set 7. an HTTP network feature code automatic generation system, adopts the network feature automatic generation method as described in any one of claims 1-6, is characterized in that, described system comprises: Packet feature code generation module: used for feature statistics and packet content extracted from the one-question-one-answer packets of multiple network samples, generate coarse-grained cluster sets through secondary clustering, and then generate coarse-grained cluster sets in the coarse-grained cluster sets Based on the secondary clustering, a fine-grained clustering set is generated, and a Q&A package feature code set of the network sample is generated through the fine-grained clustering set

URI feature code generation module: for the traffic that is divided into a single category in the network sample, perform supplementary extraction of URI path and parameter feature codes, and generate a feature code set of the URI

HTTP network feature code total set generation module: through the question and answer packet feature code set

and the set of signatures for the URI

Combined to generate a total set of feature codes T _all . 8. according to the described HTTP network characteristic code automatic generation system of claim 7, it is characterized in that, described package characteristic code generation module comprises: Whitelist filtering module: filter and remove traffic that visits legitimate websites; Data extraction module: extract the statistics of the data flow characteristics of the network sample and the content of the question and answer package; Secondary clustering module: performing secondary clustering according to the network sample characteristic statistics and the content of the question-and-answer package, and generating the fine-grained clustering set on the basis of generating the coarse-grained clustering set; A question-and-answer packet signature generation module: according to the fine-grained clustering set, respectively generate signature sets of request packets and response packets. 9. according to the described HTTP network feature code automatic generation system of claim 8, it is characterized in that, also comprise before described data extraction module: Whitelist filtering module: filtering and removing the traffic of accessing legitimate websites in the network samples. 10. according to the described HTTP network feature code automatic generation system of claim 8, it is characterized in that, described data extraction module, also comprises: Data content extraction module: extract the content of the said Q&A package of HTTP session connection; Coarse-grained clustering attribute extraction module: taking the network sample as a unit, extract the four-dimensional statistical value of the coarse-grained clustering, including: the total number of HTTP data streams, the number of bytes sent per second, the average size of HTTP data packets, and the HTTP data The total number of packages to get the coarse-grained clustering attribute; Fine-grained clustering attribute extraction module: taking each HTTP session as a unit, extract the four-dimensional statistical value of the fine-grained clustering, including: the number of session request packets, the number of session response packets, the size of the first request packet, the first Respond to the package size to get fine-grained clustering attributes; Summarize data set module: summarize the content of the question and answer package, the coarse-grained clustering attribute and the fine-grained clustering attribute to obtain a five-tuple data set

The format of the five-tuple is: <sample id, session id, Q&A package content, coarse-grained clustering attribute, fine-grained clustering attribute>. 11. according to the described HTTP network feature code automatic generation system of claim 8, it is characterized in that, described secondary clustering module, also comprises: Coarse-grained clustering module: for the five-tuple data set

automatically clustering the coarse-grained clustering attributes to obtain a coarse-grained clustering set C, if the coarse-grained clustering set C only belongs to one of the network samples, the URI signature generation module generates the URI signature; Fine-grained clustering module: based on the coarse-grained clustering set C, for all sessions in each c _i ( _ci ∈ C), automatically install the fine-grained clustering attributes for clustering, and obtain fine-grained Clustering set C′ (C′∈C _i ); Sample coverage judgment module: if there is a fine-grained clustering c _i ′ (ci _′ ∈C’) where all the conversations come from k samples, and the value of k is greater than 1 and less than or equal to the number of network samples, then it is considered The fine-grained clustering is successful; otherwise, the step of generating the URI feature code is performed. 12. according to the described HTTP network feature code automatic generation system of claim 8, it is characterized in that, described one question and one answer packet feature code generation module, also comprises: HTTP feature code set generation module: generate request packet and response packet feature codes for all session connections in each fine-grained cluster c _i ′ (ci _′ ∈ C ′), and automatically calculate token features sequentially Finally, each fine-grained cluster c _i ′ obtains a signature of a request packet and a signature of a response packet respectively, forming an HTTP signature set W; Feature code filtering module: filter and screen the HTTP feature code set W, remove unqualified feature codes, merge repeated feature codes, and obtain the Q&A package feature code set

Images