CN114372530A - A method and system for abnormal traffic detection based on deep self-encoding convolutional network - Google Patents

Abstract

The invention discloses an abnormal flow detection method and system based on a deep self-coding convolutional network, wherein the method comprises the following steps: s1, training a plurality of depth autoencoders by using the preprocessed data; s2, inputting the preprocessed data into a plurality of self-encoders to obtain a plurality of different dimension reduction feature vectors; s3, performing feature splicing on the obtained different dimensionality reduction feature vectors and the preprocessed data, and training a convolutional neural network by using the feature splicing to obtain an optimal classification network model; and S4, splicing the preprocessed unknown data and the output of the self-encoder module, inputting the spliced unknown data and the output of the self-encoder module into a trained network model, and classifying the output of the convolutional neural network by using a softmax activation function to obtain a prediction result. The detection system comprises a data preprocessing module, a depth self-encoder module, a convolutional neural network module and a system management module. The invention solves the problems that the traditional abnormal flow detection scheme depends on an expert system and the traditional flow detection model has low accuracy and poor generalization capability.

Description

A method and system for abnormal traffic detection based on deep self-encoding convolutional network

technical field

The invention relates to the technical field of communication networks, in particular to a method and system for detecting abnormal traffic based on a deep self-encoding convolutional network.

Background technique

The current communication network is developing rapidly, and so are the cyber attacks. New vulnerabilities appear every day and are rapidly exploited in zero-day attacks. Signature-based detection cannot detect previously unknown attacks, and abnormal traffic detection techniques can detect deviations from normal communication patterns and are therefore an important tool for improving the security of today's communication networks. Despite the extensive technical and scientific literature on network traffic anomaly detection methods, the important steps of feature selection are often not sufficiently described and addressed in the literature. Anomaly detection in communication networks provides the basis for discovering new attacks, misconfigurations, and network failures.

With the development of the network, malicious behaviors such as network intrusion, service attack, information theft, and virus spread have become more and more common driven by various interests, and the number, type and degree of damage are increasing. Among them, distributed denial of service attacks Resource-consuming attacks represented by Distributed Denial of Service (DDoS) have become a major threat to Internet security due to their simplicity and difficulty in defense. Traditional intrusion detection is mainly deployed on the user side to protect users from attacks, but cannot eliminate the backbone of the network. Malicious traffic in nodes, the existing core nodes generally do not have the ability to identify and control malicious traffic, and can only be allowed to spread, consuming most of the resources of the core nodes, and the speed of network resource update and development cannot keep up with the abuse and misuse of resources. Therefore, the deployment of traffic detection and control mechanisms on core nodes is of great significance to ensure network performance, and can further be used as an information platform for attack source positioning.

The rapid development of the Internet not only brings convenience to people's lives, but also brings new challenges. The network security situation is still severe. Not only the traditional security threats have not been reduced, but new threats are constantly emerging, resulting in a wide variety of network anomalies, which are not conducive to detection and defense. The network scale and network traffic become larger and larger, which brings severe challenges to network traffic storage and detection. Although the special commercial detection instrument based on feature matching can perform online high-speed detection of known attacks, it cannot take into account the detection ability of unknown anomalies and does not have the ability to store traffic. Although the existing machine learning network abnormal traffic detection technology can take into account the detection ability of unknown anomalies, it has the problem of low detection efficiency.

Abnormal network traffic detection is mainly to detect behaviors that deviate from normal data. First, the information source is modeled and analyzed to create a normal system or network datum profile. Anomaly detection systems issue warnings or respond if new data samples deviate from or exceed the contours of the current normal pattern. Since the detection system is customized to describe the normal contour of the system or network according to the normal situation, it is difficult for the attacker to not deviate from the normal contour when attacking from an external attack, so it is easy to be detected by the anomaly detection system; similarly, the anomaly detection system Attacks from the inside can also be detected. In addition, anomaly detection systems have the ability to detect previously unknown attacks. Its main disadvantages are: firstly, only the initial system can be trained to create a normal contour model; secondly, it is also complicated and time-consuming to adjust and maintain the contour model, and creating a wrong contour model may lead to a high false alarm rate. Finally, some carefully constructed malicious attacks can use anomaly detection to train the system to gradually accept malicious behavior, resulting in false negatives.

The main existing abnormal traffic detection methods are:

1. Statistical-based abnormal traffic detection: Statistics-based abnormal traffic detection assumes that the current network environment is in a quasi-steady state. The algorithm collects and organizes a large amount of normal traffic data in the early stage, sets the initial threshold by performing statistical analysis or data transformation on the historical traffic data, and then calculates the current network traffic data, and compares it with the initial threshold to determine whether the current network is abnormal. If a certain statistic of the current network traffic data exceeds the corresponding threshold, it means abnormal traffic has occurred. Common network traffic characteristics include the number of bytes, the number of packets, the flow count, the audit record data, the number of audit events, the interval event, the five yuan Group (protocol, source IP address, destination IP address, destination port, and destination IP address) and resource consumption events, etc.

2. Anomaly detection based on data mining: Anomaly detection based on data mining, using data mining technology to analyze and mine the characteristic information of various types of traffic from massive network traffic, using automatic or semi-automatic modeling algorithms to find out which can reflect the current network conditions. The characteristic parameters of the network, such as correlation, pattern or trend, reveal the potential hidden characteristics of the data from a higher abstraction level, so as to judge the abnormal behavior of the network. Currently commonly used such as generative induction rules, fuzzy logic, genetic algorithm and so on.

3. Anomaly detection based on machine learning: The identification of anomalous traffic is essentially a classification problem, which usually presupposes learning. Abnormal traffic detection based on machine learning is a highly abstract and model expression of previous experience, and is characterized by model building. Different network traffic characteristics, such as the number of bytes, the average packet size, the number of packets, the maximum packet length, the flow duration, the arrival time interval, etc., can be used as modeling objects. Bayesian networks, clustering, support vector machines, Markov models, etc. have all been widely used.

For example, for the traffic monitoring method based on clustering algorithm, the framework is shown in Figure 1, which is divided into four modules as a whole: labeled data auxiliary module, hybrid clustering module, online classification module and system update module. The labeled data auxiliary module is mainly responsible for the information gain and feature weighting of network traffic characteristics; the hybrid clustering module is mainly composed of multiple clustering algorithms, which are trained with the same input but output according to different weights; the system update module is responsible for adding new data , usually some traffic information with new protocols or traffic information selected by the expert system; the online classification module adopts the NCC classifier, and outputs the final traffic detection result based on the clustering algorithm.

However, the existing commonly used traffic monitoring technologies usually need to rely on human experts to select and label traffic features, and continuously add new expert annotation information during the model running, which is highly dependent on the expert system, and this method is not only costly , it cannot effectively deal with new traffic attacks, and the generalization ability for abnormal traffic detection is insufficient, and it cannot maintain high accuracy in multiple different scenarios, and it cannot deal with 0-day vulnerability attacks and some relatively novel attacks.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method and system for detecting abnormal traffic based on a deep self-encoding convolutional network to solve two problems: one is to solve the dependence of the traditional abnormal traffic detection scheme on the expert system; the other is to solve the traditional traffic detection model Problems with low accuracy and poor generalization ability.

In order to achieve the above object, the present invention provides the following technical solutions:

The present invention first provides a method for detecting abnormal traffic based on a deep self-encoding convolutional network, comprising the following steps:

S1. Use the preprocessed data to train multiple deep autoencoders;

S2. Input the preprocessed data into multiple autoencoders to obtain multiple different dimensionality reduction feature vectors;

S3. Perform feature splicing with the obtained different dimensionality reduction feature vectors and the preprocessed data, and use them to train the convolutional neural network to obtain the optimal classification network model;

S4. The preprocessed unknown data and the output of the autoencoder module are spliced and input into the trained network model, and the softmax activation function is used to classify the output of the convolutional neural network to obtain a prediction result.

Further, the preprocessing process of step S1 includes: acquiring the traffic protocol, type and duration, and number of bytes, and converting them into a one-dimensional floating point vector through one-hot encoding.

Further, the deep autoencoder in step S1 is composed of multiple fully connected layers.

Further, the deep autoencoder in step S1 uses the mean square loss function as the loss function, the Adam algorithm as the optimizer, and the tanh method as the activation function, wherein the layer with the least number of dimensions is the required dimension reduction feature information.

Further, the mean square loss function is calculated as follows:

Υ(x,y)=L{L ₁ ,...,L _n }T, Ln=(x _n -y _n ) ²

Among them, L refers to the vector of the input calculation, L _n refers to the different dimensions of the L vector, T refers to the transpose of the vector, n refers to the batch size, and x _i and y _i refer to the parameters of different positions corresponding to the input and output feature vectors. .

Further, the tanh function is calculated as follows:

where tanh is one of the hyperbolic functions, and tanh is the hyperbolic tangent.

Further, in step S3, a two-dimensional convolution layer and a two-dimensional batch normalization method are used, and the relu method is used as the activation function.

Further, step S3 adopts the global pooling technique to map the outputs of different channels into result vectors, and uses the softmax method for normalized representation.

Further, step S3 adopts the cross-entropy loss function as the loss function, and the Adam algorithm as the optimizer.

The present invention also provides an abnormal flow detection system based on a deep self-encoding convolutional network, comprising:

The data preprocessing module is used to perform preliminary processing on the input data, including obtaining the traffic protocol, type and duration, and the number of bytes, and converting it into a one-dimensional floating-point vector through one-hot encoding;

The deep autoencoder module is used to pass the preprocessed traffic information as input to multiple deep autoencoders, train the marked traffic data, and extract the advanced autoencoder from the input traffic information. The abstract features are spliced with the preprocessed data as the input of the convolutional network;

The convolutional neural network module is used to accept the input of the preprocessing module and the deep self-encoder module, and train the network model through the output of the trained deep self-encoder module combined with the labeled traffic data to obtain the optimal classification network. Model;

The system management module is used to manage the configuration of the system.

Compared with the prior art, the beneficial effects of the present invention are:

The abnormal traffic detection method based on the deep self-encoding convolutional network provided by the present invention inputs the traffic information into the deep self-encoder and converts it into abstract features of dimension reduction. This technology can better extract the high-level features of the traffic and mine deeper traffic Containing information; using CNN in the order of "convolution-normalization-activation" architecture as the construction basis, which has higher accuracy than traditional detection models; uses a two-dimensional CNN model based on global pooling technology for classification. This technique can map categorical information to specific output channels, enhancing the interpretability of the model.

The invention solves the problems that the current abnormal flow detection system has a low feature extraction level and the model has insufficient learning ability for the abnormal flow features, and provides the details of the feature extraction and detection algorithm implementation at the same time. Through experiments, the accuracy rate on the kdd99 dataset can reach 94%, which can effectively prevent known attacks and unknown attacks, and the accuracy rate and generality are significantly improved compared with the existing technology.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only described in the present invention. For some of the embodiments, those of ordinary skill in the art can also obtain other drawings according to these drawings.

Figure 1 shows the traffic monitoring framework based on the clustering algorithm.

FIG. 2 is an architecture of an abnormal traffic detection system based on a deep self-encoding convolutional network provided by an embodiment of the present invention.

FIG. 3 is a schematic diagram of a deep autoencoder provided by an embodiment of the present invention.

FIG. 4 is a structure of an autoencoder model provided by an embodiment of the present invention.

FIG. 5 is a structure of a convolution model provided by an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

The abnormal flow detection system based on the deep self-encoding convolutional network provided by the present invention, as shown in Figure 2, includes:

The data preprocessing module is used to perform preliminary processing on the input data, including obtaining the traffic protocol, type and duration, and the number of bytes, and converting it into a one-dimensional floating-point vector through one-hot encoding;

The deep autoencoder module is used to pass the preprocessed traffic information as input to multiple deep autoencoders, train the marked traffic data, and extract the advanced autoencoder from the input traffic information. The abstract features are spliced with the preprocessed data as the input of the convolutional network;

The convolutional neural network module is used to accept the input of the preprocessing module and the deep self-encoder module, and train the network model through the output of the trained deep self-encoder module combined with the labeled traffic data to obtain the optimal classification network. Model;

The system management module is used to manage the configuration of the system.

1. Preprocessing module

This module performs preliminary processing on the input data, including obtaining the traffic protocol, type and duration, number of bytes, etc., and converts it into a one-dimensional floating-point vector through one-hot encoding.

One-hot encoding, also known as one-bit valid encoding, uses an N-bit state register to encode N states, each state has its own register bits, and only one of them is valid at any time. For each feature, if there are n possible values of the feature, then after one-hot encoding, the feature becomes n binary features, and only one of the n binary features is Effective.

2. Deep Autoencoder Module

As shown in Figure 3, this module will use the preprocessed traffic information as input to pass to multiple deep auto-encoders. The deep auto-encoder needs the marked traffic data for training, and the trained deep auto-encoder can be input from the The high-level abstract features are extracted from the traffic information and spliced with the preprocessed data as the input of the convolutional network.

Each deep autoencoder is a deep neural network composed of multiple fully connected layers. Its training goal is to make the final output as close to the input as possible. The mean square loss function (mseloss) is used as the loss function, and the Adam algorithm is used as the optimizer. The tanh method is an activation function, and the layer with the least number of dimensions is the required dimensionality reduction feature information.

The mean squared loss (mseloss) function is calculated as follows:

Υ(x,y)=L{L ₁ ,...,L _n } ^T , L _n =(x _n -y _n ) ²

Among them, L refers to the vector of the input calculation, L _n refers to the different dimensions of the L vector, T refers to the transpose of the vector, n refers to the batch size, and x _i and y _i refer to the parameters of different positions corresponding to the input and output feature vectors. .

The tanh function is calculated as follows:

where tanh is one of the hyperbolic functions, and tanh is the hyperbolic tangent.

The specific structure of the autoencoder model used in the present invention is shown in Figure 4, where FC refers to the fully connected layer, the number refers to the number of nodes in the fully connected layer, and the output of the intermediate fully connected layer with the smallest number of nodes will be spliced with the original traffic characteristics and then input into the volume A neural network module.

3. Convolutional Neural Network Module

This module accepts the input of the preprocessing module and the autoencoder module, and needs to train the network model through the output of the trained deep autoencoder module combined with the labeled traffic data to obtain the optimal classification network model. The trained model will accept the preprocessed data and the autoencoder module output splicing as input, and no manual intervention is required.

Among them, this module adopts a two-dimensional convolution layer (3 × 3 size convolution kernel) and a two-dimensional batch normalization method, uses the relu method as the activation function, and finally adopts the global pooling technology (global-avg-pool) The output of the channel is mapped to the result vector and normalized using the softmax method. The model uses the cross-entropy loss function as the loss function and the Adam algorithm as the optimizer.

The relu function is a linear rectification function, also known as a modified linear unit, and its calculation is as follows:

f(x)=max(0,x),

The relu function can perform gradient descent and backpropagation more efficiently: it avoids the problems of gradient explosion and gradient disappearance, and can simplify the calculation process.

In the early days of the development of convolutional neural networks, the convolutional layer always required one or n fully connected layers after passing through the pooling layer. Its characteristic is that the parameters of the fully connected layer are too many, which makes the model itself very bloated. It brings about the problem that the amount of parameters is too large, the speed of training is reduced, and it is easy to overfit. This model solves the inherent problem of the fully connected layer through the global pooling technology, reduces the amount of parameters, and improves the operation efficiency.

Global pooling means that the size of the pooled sliding window is the same as the size of the entire feature map. In this way, each W×H×C feature map input is transformed into a 1×1×C output. Therefore, it is actually equivalent to the fully connected layer operation where each position weight is 1/(W×H). Among them, W and H are width and height, and C is the number of channels.

The specific pooling method of global pooling in the sliding window can be arbitrary, so it will be subdivided into global average pooling, global maximum pooling, etc.

The convolution model structure adopted by the convolutional neural network module is shown in Figure 5. The convolution kernels are all 3×3 in size and contain four similar parts sequentially linked, each containing convolutional layers, normalization, relu activation functions, and finally a global pooling layer and softmax activation function.

The input of the convolutional neural network comes from the one-dimensional vector spliced by the original traffic feature and the output of the autoencoder module, and the one-dimensional vector is filled in a 20×20 two-dimensional vector from left to right and top to bottom in order. , if the length is not enough, it will be filled with zeros, and the obtained 20×20 two-dimensional vector will be reconstructed.

4. System management module

This module is used to manage the configuration of the system. Users can modify the parameter information of the system, data processing methods, and display effects. Parameter information such as flow sampling frequency, alarm level, alarm style, etc. Data processing methods, for example, whether to adopt a rough (normal/abnormal) dichotomy or a detailed information display (normal/abnormal a/.../abnormal b) for the alarm information, whether the alarm information needs to be notified immediately or fixed time frequency Notification after statistics, etc.

The abnormal traffic detection method based on the deep self-encoding convolutional network provided by the present invention, as shown in FIG. 2, adopts the above-mentioned module and includes the following steps:

S1. Use the preprocessed data to train multiple deep autoencoders;

S2. Input the preprocessed data into multiple autoencoders to obtain multiple different dimensionality reduction feature vectors;

S3. Perform feature splicing with the obtained different dimensionality reduction feature vectors and the preprocessed data, and use them to train the convolutional neural network to obtain the optimal classification network model;

S4. The preprocessed unknown data and the output of the autoencoder module are spliced and input into the trained network model, and the softmax activation function is used to classify the output of the convolutional neural network to obtain a prediction result.

The method and system for detecting abnormal traffic based on a deep self-encoding convolutional network provided by the present invention input the traffic information into the deep self-encoder and convert it into dimensionality-reduced abstract features. This technology can better extract the high-level features of traffic and mine deeper The traffic contains information; the CNN of the "convolution-normalization-activation" architecture sequence is used as the construction basis, which has higher accuracy than the traditional detection model; the two-dimensional CNN based on the global pooling technology is used for classification This technology can map classification information to specific output channels, which enhances the interpretability of the model.

The invention solves the problems that the current abnormal flow detection system has a low feature extraction level and the model has insufficient learning ability for the abnormal flow features, and provides the details of the feature extraction and detection algorithm implementation at the same time. Through experiments, the accuracy rate on the kdd99 data set can reach 94%, which can effectively prevent known attacks and unknown attacks. The accuracy rate and generality are significantly improved compared with the existing technology. See the results in Table 1 for details.

Table 1 The test results of the method of the present invention and the method of the prior art on the kdd99 data set

Model Accuracy recall F1 value K-nearest neighbor algorithm 0.92 0.96 0.94 Adaboost algorithm 0.77 0.84 0.77 random forest 0.88 0.92 0.90 Convolutional Neural Network 0.92 0.97 0.94 method of the invention 0.94 0.98 0.96

Certain exemplary embodiments of the present invention have been described above by way of illustration only, and it is needless to say that those skilled in the art may The described embodiments are modified. Accordingly, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of the claims of the present invention.

Claims (10)

1. An abnormal flow detection method based on a deep self-coding convolutional network is characterized by comprising the following steps:

s1, training a plurality of depth autoencoders by using the preprocessed data;

s2, inputting the preprocessed data into a plurality of self-encoders to obtain a plurality of different dimension reduction feature vectors;

s3, performing feature splicing on the obtained different dimensionality reduction feature vectors and the preprocessed data, and training a convolutional neural network by using the feature splicing to obtain an optimal classification network model;

and S4, splicing the preprocessed unknown data and the output of the self-encoder module, inputting the spliced unknown data and the output of the self-encoder module into a trained network model, and classifying the output of the convolutional neural network by using a softmax activation function to obtain a prediction result.

2. The abnormal traffic detection method based on the deep self-coding convolutional network of claim 1, wherein the preprocessing procedure of step S1 includes: and acquiring a flow protocol, a type, duration and byte number, and converting the flow protocol, the type, the duration and the byte number into a one-dimensional floating-point number vector through one-hot coding.

3. The abnormal traffic detection method based on the deep self-coding convolutional network of claim 1, wherein the deep self-coder of step S1 is composed of a plurality of fully-connected layers.

4. The abnormal traffic detection method based on the depth self-coding convolutional network of claim 1, wherein the depth self-coder of step S1 uses a mean square loss function as a loss function, Adam algorithm as an optimizer, and tanh method as an activation function, wherein the layer with the least dimensionality is the required dimension reduction feature information.

5. The abnormal traffic detection method based on the deep self-coding convolutional network of claim 4, wherein the mean square loss function is calculated as follows:

Υ(x,y)＝L{L₁,…,L_n}T，Ln＝(x_n-y_n)²

where L refers to the vector of the input calculation, L_nRefer to the different dimensions of the L vector, T denotes transpose of the vector, n refers to batch size, x_i、y_iThe parameters refer to different positions corresponding to the input and output characteristic vectors.

6. The abnormal traffic detection method based on the deep self-coding convolutional network of claim 4, wherein the tanh function is calculated as follows:

where tanh is one of the hyperbolic functions and tanh is the hyperbolic tangent.

7. The abnormal traffic detection method based on the deep self-coding convolutional network of claim 1, wherein step S3 adopts two-dimensional convolutional layer and two-dimensional batch normalization method, and adopts relu method as activation function.

8. The abnormal traffic detection method based on the deep self-coding convolutional network of claim 7, wherein step S3 employs a global pooling technique to map the outputs of different channels into a result vector, and uses a softmax method for normalization representation.

9. The abnormal traffic detection method based on the deep self-coding convolutional network of claim 7, wherein step S3 adopts a cross entropy loss function as a loss function, and the Adam algorithm is an optimizer.

10. An abnormal traffic detection system based on a deep self-coding convolutional network, which is characterized by comprising:

the data preprocessing module is used for carrying out primary processing on input data, including acquiring a flow protocol, a type, duration and byte number, and converting the flow protocol, the type, the duration and the byte number into a one-dimensional floating-point number vector through one-hot coding;

the deep self-encoder module is used for transmitting the preprocessed flow information as input to a plurality of deep self-encoders, training the marked flow data, extracting high-level abstract features from the input flow information by the trained self-encoders, splicing the high-level abstract features with the preprocessed data, and using the extracted high-level abstract features as the input of a convolutional network;

the convolutional neural network module is used for receiving the input of the preprocessing module and the deep self-encoder module, and training a network model by combining the output of the trained deep self-encoder module with the marked flow data to obtain an optimal classification network model;

and the system management module is used for managing the configuration of the system.

Images