CN118573487A - Network anomaly detection method and system for detecting zero positive anomaly of isolated forest fusion - Google Patents

Abstract

The present invention discloses a network anomaly detection method and system of isolated forest fusion zero-positive anomaly detection, and the method includes the following steps: 1. data collection and preprocessing; 2. feature extraction and analysis; 3. model integration and training; 4. threat level assessment and adaptive strategy formulation; 5. real-time monitoring and dynamic response; 6. model update and continuous learning; 7. performance evaluation and optimization; the system includes a data collection and preprocessing module, a feature extraction and analysis module, a model integration and training module, a threat level assessment and adaptive strategy formulation module, a real-time monitoring and dynamic response module, a model update and continuous learning module, and a performance evaluation and optimization module. The present invention improves the comprehensiveness, accuracy and efficiency of abnormal network behavior detection, reduces system resource consumption, realizes real-time anomaly detection, and ensures that new threats can be quickly discovered and effective measures can be taken.

Description

Network anomaly detection method and system based on isolation forest fusion zero-positive anomaly detection

Technical Field

The present invention belongs to the technical field of network anomaly detection, and in particular relates to a network anomaly detection method and system for an isolation forest fusion zero-positive anomaly detection.

Background Art

In the field of network security, anomaly detection technology is a key link in protecting information systems from unauthorized access and attacks. With the development of technology and the complexity of the network environment, network attackers use increasingly complex and covert means, such as distributed denial of service (DDoS) attacks and advanced persistent threats (APT). These attacks are not only difficult to detect, but may also cause serious economic losses and service interruptions, bringing unprecedented challenges to information system security. Especially in terms of 0day or other unknown attacks, traditional anomaly detection methods are often unable to effectively identify emerging complex attack methods because they mainly rely on known attack patterns and characteristics. These attacks may lead to data leakage, service interruption or even complete system paralysis, causing huge economic losses and reputation damage. Nowadays, anomaly detection has become one of the hot spots in the field of network security research, and the current research status mainly includes the following aspects:

(1) Rule-based anomaly detection method

The rule-based anomaly detection method is a method to identify and respond to abnormal behaviors in the network through predefined rules. These rules are usually based on known abnormal patterns and network behavior characteristics, such as specific request patterns, traffic anomalies, or unusual user behaviors. Therefore, by performing abnormal rule matching analysis on network traffic, abnormal behaviors can be detected and identified. The rule-based anomaly detection method usually includes the following steps:

① Rule formulation: Analyze known abnormal behaviors and network attack patterns and define a series of detection rules. These rules may include specific traffic patterns, abnormal access frequencies, abnormal request types, etc.

② Traffic monitoring and rule comparison: Continuously monitor network traffic and analyze captured data to check whether there are any abnormal behaviors that comply with preset rules; for example, the rules may involve detecting frequent access from a certain IP address or abnormal data transmission volume.

③ Abnormal behavior identification and processing: Once behavior that meets the rules is detected, the system will mark it as potential abnormal activity and take predetermined response measures; this may include isolating suspicious traffic, generating security warnings, or conducting further analysis and processing.

④ Rule update and maintenance: Regularly review and update detection rules to ensure that they can adapt to emerging abnormal patterns and attack strategies; adjust and optimize rules by learning the latest security threats and network behaviors to improve detection accuracy and reduce false alarm rates.

(2) Anomaly detection methods based on machine learning

The anomaly detection method based on machine learning is a method that uses machine learning algorithms to identify abnormal behaviors. It is applicable to a wide range of application scenarios. This method learns the characteristics of abnormal behaviors from a large amount of network traffic data and builds a classifier to determine whether the data exhibits abnormal behavior. Its core idea is to achieve automatic detection of abnormal behaviors through steps such as feature extraction, feature selection, classifier training, and classifier testing. Specifically, the anomaly detection method based on machine learning usually includes the following key steps:

① Feature extraction: Extract useful features from network traffic data, such as the number of requests, request frequency, request type, request data volume, etc.

②Feature selection: Through the feature selection algorithm, the most discriminative features are screened out.

③Dataset construction: The filtered features and corresponding labels (normal or abnormal) are combined into a dataset for training and testing classifiers.

④Classifier training: Use machine learning algorithms to train the data set to build an effective classifier.

⑤Classifier testing: Use the test dataset to evaluate the performance and accuracy of the classifier.

(3) Anomaly detection method based on deep learning

The anomaly detection method based on deep learning is a method that uses deep learning algorithms such as deep neural networks (DNN) and convolutional neural networks (CNN) to learn the characteristics of abnormal behavior from a large amount of network traffic data and build a model to determine whether the data exhibits abnormal behavior. The anomaly detection method based on deep learning usually includes the following key steps:

① Data preprocessing: preprocess the data, including data cleaning, standardization, sampling, etc.

② Feature extraction: Use deep learning algorithms such as deep neural networks or convolutional neural networks to extract useful features from preprocessed data.

③Model construction: Based on the extracted features, build a deep learning model, such as a multi-layer perceptron (MLP), convolutional neural network (CNN) or recurrent neural network (RNN).

④Model training: Use large-scale labeled data to train the constructed deep learning model.

⑤Model testing: Use test datasets to evaluate the performance and accuracy of deep learning models.

Rule-based anomaly detection methods cannot detect attacks outside the rules and require constant updating of normal behavior models. Machine learning-based anomaly detection methods require a large amount of network traffic data for learning, and the generalization ability of the classifier is limited, which may not be able to effectively respond to new or changing attack patterns. Deep learning-based anomaly detection methods also require a large amount of network traffic data and have high demands on computing resources. In this context, traditional one-size-fits-all anomaly detection methods often lack flexibility and adaptability, and are difficult to effectively respond to rapidly changing threat environments; moreover, a single anomaly detection strategy often leads to excessive consumption of resources and cannot be adaptively adjusted according to the network environment and attack characteristics, which increases the cost and complexity of system operation.

In summary, the existing anomaly detection methods are no longer sufficient to meet current security needs, and there is an urgent need to provide an innovative solution for the increasingly complex network security environment.

Summary of the invention

The technical problem to be solved by the present invention is to provide a network anomaly detection method that integrates isolated forest and zero-positive anomaly detection in order to address the deficiencies in the above-mentioned prior art, which improves the comprehensiveness, accuracy and efficiency of abnormal network behavior detection, reduces system resource consumption, realizes real-time anomaly detection, and ensures that new threats can be quickly discovered and effective measures can be taken.

In order to solve the above technical problems, the technical solution adopted by the present invention is: a network anomaly detection method of isolation forest fusion zero-positive anomaly detection, comprising the following steps:

Step 1: Data collection and preprocessing: Capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data;

Step 2: Feature extraction and analysis: Deeply explore the key features in network traffic data, including abnormal request frequency, time series patterns, and response data volume;

Step 3: Model integration and training: Unsupervised training of the isolation forest model to identify potential anomalies in the data, and the use of zero positive anomaly detection strategy to identify and respond to new threats that do not conform to known patterns, and then combine the isolation forest model with zero positive anomaly detection to obtain an integrated detection model;

Step 4: Threat level assessment and adaptive strategy formulation: Based on the detection results of the integrated detection model, classify the threat level of each detection event, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy;

Step 5: Real-time monitoring and dynamic response: Conduct real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors;

Step 6: Model update and continuous learning: The system updates the isolation forest model and zero-positive anomaly detection strategy using the latest network traffic data according to a predetermined period to maintain the sensitivity and accuracy of the integrated detection model;

Step 7: Performance evaluation and optimization: The system conducts regular comprehensive performance reviews to measure detection accuracy, false alarm rate, and response time, and optimizes and adjusts detection parameters and strategies based on the evaluation results to ensure efficient and stable operation of the entire anomaly detection system.

In the above-mentioned network anomaly detection method of isolation forest fusion zero-positive anomaly detection, the network traffic data in step 1 includes IP address, port information, traffic type and request frequency; the preprocessing operation includes screening, format conversion and purification processing.

In the above-mentioned network anomaly detection method of isolation forest fusion zero-positive anomaly detection, the isolation forest algorithm in step 3 recursively divides the data space by randomly selecting data features and segmentation values until each data point is isolated or the tree reaches its depth limit; the anomaly score of the data point to be evaluated is calculated through the constructed forest to determine whether it is an anomaly point; the anomaly score of the isolation forest model is expressed as:

,

in, is the data point to be evaluated, is the sample size, express The path length in the isolation tree, express The expected path length among all isolated trees, The sample size is The average search path length of the binary sorted tree when is expressed as:

,

in, For the harmonic numbers, and , is Euler's constant;

The closer the anomaly score is to 1, the higher the possibility of being judged as an outlier. When the anomaly score is much smaller than 0.5, it is judged not to be an outlier. When the anomaly scores of sample points are all close to 0.5, it is judged that the sample does not contain obvious outliers.

The above-mentioned network anomaly detection method of isolated forest fusion zero-positive anomaly detection, the zero-positive anomaly detection in step 3 adopts the method based on statistical analysis The scoring method quantifies the degree of deviation of a data point from the normal behavior pattern. The anomaly score calculation formula for zero positive anomaly detection is:

,

in, is the anomaly score for zero positive anomaly detection, is the observed value, is the mean value of the data set, is the standard deviation;

The larger the absolute value of , the more likely the observation is abnormal.

The above-mentioned network anomaly detection method of isolation forest fusion zero positive anomaly detection, in step 3, combines the isolation forest model with zero positive anomaly detection to obtain an integrated detection model expressed as:

,

in, represents the comprehensive score of anomaly detection, Represents the anomaly score of the isolation forest model The weight of Indicates the score of zero positive anomaly detection The weight of =1.

The above-mentioned network anomaly detection method of isolation forest fusion zero-positive anomaly detection, and The value of is:

Step A1: Define a total quantity of Key features of;

Step A2: Check whether each key feature matches a known security threat feature to identify whether it is a known feature or an unknown feature. When a feature exists in the known threat database, it is marked as a known feature and the number of known features is recorded. ; When a feature does not exist in the known threat database, it is marked as an unknown feature and the number of unknown features is recorded as ;

Step A3: Calculate the ratio of known features to , and dynamically adjust The value of ; Calculate the ratio of unknown features to , and dynamically adjust The value of ;

Step A4: For unknown features, once detected, they will be recorded in the known threat database, and the key feature database will be regularly adjusted and updated.

In the above-mentioned network anomaly detection method of the isolation forest fusion zero-positive anomaly detection, the specific process of classifying the threat level of each detection event according to the detection results of the integrated detection model in step 4 is as follows:

Step 401: Comprehensively score the anomaly detection From the minimum value of the anomaly detection comprehensive score To the maximum value of the anomaly detection comprehensive score Divided into 5 areas , ),[ , ),[ , ),[ , ),[ , ];in, , , , All are anomaly detection comprehensive scores The values in are arranged from small to large;

Step 402: Determine The overall score of anomaly detection for each test In which area is it located? The threat level of this detection; specifically:

when lie in[ , ) area, determine The threat level of this detection is normal, indicating no abnormal activities or only safe and expected network behaviors;

when lie in[ , ) area, determine The threat level of this detection is suspicious behavior, which means that the behavior deviates from the normal pattern, but there is not enough evidence to indicate that it is a malicious attack;

when lie in[ , ) area, determine The threat level of this detection is a mild attack, which means that the attack behavior is clear but the impact is limited, such as an attempted intrusion activity;

when lie in[ , ) area, determine The threat level of this detection is medium attack, which means the attack is more serious and may pose a greater threat to system security, such as successful SQL injection;

when lie in[ , ] area, judge the The threat level of this detection is severe attack, which means a high-risk attack that poses a direct and serious threat to system or data security, such as a large-scale DDoS attack.

The above-mentioned network anomaly detection method of isolated forest fusion zero-positive anomaly detection, in step A1, a total number of defined groups is When the key features are detected, two detection modes are defined. The first detection mode is the basic detection mode. The key features include URL features, HTTP methods, request body content, access time, source IP address, The value of is 5; the second detection mode is the enhanced detection mode, and the key features include URL features, HTTP methods, request body content, access time, source IP address, Headers request header content, access frequency, data packet length, attack source address and access port; The value of is 10;

When setting the dual-modal anomaly detection strategy according to the threat level and dynamically adjusting the anomaly detection strategy as described in step 4, lie in[ , ),[ , ) area, adjust the anomaly detection strategy to the basic detection mode; when lie in[ , ),[ , ),[ , ] area, adjust the anomaly detection strategy to enhanced detection mode.

The present invention also discloses a network anomaly detection system for implementing the above-mentioned network anomaly detection method by integrating isolation forest and zero-positive anomaly detection, comprising:

Data collection and preprocessing module: used to capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data;

Feature extraction and analysis module: used to deeply mine the key features in network traffic data, including abnormal request frequency, time series pattern and response data volume, providing data basis for subsequent model training combining isolation forest algorithm and zero positive anomaly detection;

Model integration and training module: used to train the isolation forest model and the zero positive anomaly detection model, and then combine the isolation forest model with the zero positive anomaly detection model to obtain an integrated detection model;

Threat level assessment and adaptive strategy formulation module: used to classify the threat level of each detection event according to the detection results of the integrated detection model, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy;

Real-time monitoring and dynamic response module: used to perform real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors;

Model update and continuous learning module: used to update the isolation forest model and zero-positive anomaly detection strategy according to the scheduled period using the latest network traffic data to maintain the sensitivity and accuracy of the integrated detection model;

Performance evaluation and optimization module: used to conduct regular comprehensive performance reviews of the system, measure key indicators such as detection accuracy, false alarm rate, and response time, and optimize and adjust detection parameters and strategies in a timely manner based on the evaluation results to ensure efficient and stable operation of the entire anomaly detection system.

Compared with the prior art, the present invention has the following advantages:

1. The core advantage of the isolation forest algorithm lies in its unsupervised learning ability and linear time complexity, which enables it to quickly process and identify abnormal behaviors in a large amount of network traffic data, and is particularly suitable for real-time analysis of large-scale data; and by combining the zero-positive anomaly detection technology, the ability to identify emerging threats can be further enhanced, because zero-positive anomaly detection does not rely on historical attack data and can identify attack behaviors without pre-defined features; the present invention combines the isolation forest algorithm with zero-positive anomaly detection to propose a new and efficient network anomaly detection technology, which provides a new perspective for the field of network security, especially in the face of the growing threat of 0day attacks; it can effectively identify 0day attacks, other unknown abnormal behaviors and known anomalies, improve the comprehensiveness and accuracy of network abnormal behavior detection, and improve the efficiency of anomaly detection, which is particularly suitable for dynamic network environments and the identification of unknown attack types.

2. The present invention combines the isolation forest algorithm with zero positive anomaly detection to enhance the system's ability to identify emerging threats without sacrificing detection speed; it can identify unknown or previously unseen threats in a constantly changing network environment in real time, and can improve the accuracy of zero positive anomaly detection in identifying true anomalies in large amounts of network traffic data by combining the unsupervised characteristics of the isolation forest algorithm with zero positive anomaly detection.

3. The present invention proposes a threat level determination mechanism based on data behavior analysis, which realizes a fine-grained assessment of network security threats and provides security analysts with clear threat information, facilitating rapid understanding and response to potential security incidents.

4. The present invention realizes a dynamic adaptive mechanism through the dynamic allocation of the weights of the isolation forest model and the weights of the zero-positive anomaly detection, as well as a method for dynamically adjusting the anomaly detection strategy. The mechanism can adjust the detection parameters and strategies based on the changes in the real-time monitored network conditions and traffic behaviors; in this way, the system can effectively improve the recognition accuracy of emerging threats and abnormal behaviors, and ensure the efficient operation of the security system.

5. The present invention improves the adaptability of network anomaly detection by continuously learning and updating the detection model, so that the system can adapt to changes in network behavior in a timely manner and dynamically adjust security policies according to changes in network behavior, ensuring the system's rapid response and continuous learning to new threats, optimizing the use of system resources, and reducing system resource consumption.

6. The present invention adopts an innovative dual-modal anomaly detection strategy. Under normal circumstances, the system runs in basic detection mode to maintain efficiency and resource economy. Once the threat level is detected to be elevated, the system automatically switches to enhanced detection mode to increase the depth and scope of detection to improve response efficiency. After the threat subsides, the system intelligently switches back to basic detection mode. This adaptive switching mechanism not only improves the ability to respond to network threats, but also effectively balances performance and resource consumption, thereby achieving optimal system performance.

7. The present invention realizes real-time anomaly detection, ensuring that new threats can be quickly discovered and effective measures can be taken.

8. The present invention not only performs well in solving technical problems, but also provides an effective and flexible solution for network security.

The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flowchart of a method for network anomaly detection using isolation forest fusion and zero-positive anomaly detection according to the present invention.

DETAILED DESCRIPTION

Example 1

As shown in FIG1 , the network anomaly detection method of the isolation forest fusion zero-positive anomaly detection of this embodiment includes the following steps:

Step 1: Data collection and preprocessing: Capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data;

In this embodiment, the network traffic data in step one includes key indicators such as IP address, port information, traffic type and request frequency that can characterize the key features of the difference between normal and abnormal behaviors; the preprocessing operations include screening, format conversion and purification processing, aiming to extract effective information that can reflect potential risks, remove redundant and non-critical information, and ensure the quality and efficiency of subsequent analysis.

In specific implementation, the network traffic data may be derived from public resources or real data collected directly from Web server applications.

The core of the Isolation Forest algorithm is to use linear time complexity to "isolate" abnormal data points, assuming that these points are easier to identify because they are different. This method does not rely on historical attack data, making it particularly suitable for detecting unknown or previously unseen threats. In addition, the Isolation Forest algorithm is highly computationally efficient and can quickly process large-scale data sets, which is crucial for real-time network traffic analysis. In addition, the Isolation Forest algorithm is relatively insensitive to parameter settings, which allows the algorithm to run stably in a variety of environments without the need for complex tuning processes.

Step 2: Feature extraction and analysis: Deeply explore the key features in network traffic data, including abnormal request frequency, time series pattern, and response data size. These features help to clearly distinguish the boundary between normal and abnormal behaviors, and provide a data basis for subsequent model training combining isolation forest algorithm and zero positive anomaly detection;

Step 3: Model integration and training: Unsupervised training of the isolation forest model to identify potential anomalies in the data, and the use of zero positive anomaly detection strategy to identify and respond to new threats that do not conform to known patterns, and then combine the isolation forest model with zero positive anomaly detection to obtain an integrated detection model;

By combining the Isolation Forest Algorithm with Zero Positive Anomaly Detection, it is possible to identify unknown or previously unseen threats in a constantly changing network environment in real time. The unsupervised nature of the Isolation Forest Algorithm can be used to improve the accuracy of Zero Positive Anomaly Detection in identifying true anomalies in large amounts of network traffic data, and enhance the system's ability to identify new and unknown attack types.

In this embodiment, the isolation forest algorithm in step 3 recursively divides the data space by randomly selecting data features and segmentation values until each data point is isolated or the tree reaches its depth limit; the anomaly score of the data point to be evaluated is calculated through the constructed forest to determine whether it is an anomaly point; the anomaly score of the isolation forest model is expressed as:

,

in, is the data point to be evaluated, is the sample size, express The path length in the isolation tree, express The expected path length among all isolated trees, The sample size is The average search path length of the binary sorted tree (BST) when , used to sort data points The expected path length is normalized and expressed as:

,

in, For the harmonic numbers, and , is Euler's constant;

When it comes to specific implementation, Approximately equal to 0.57722;

The closer the anomaly score is to 1, the higher the possibility of being judged as an outlier. When the anomaly score is much smaller than 0.5, it is judged not to be an outlier. When the anomaly scores of sample points are all close to 0.5, it is judged that the sample does not contain obvious outliers.

Zero-shot Anomaly Detection is an advanced unsupervised learning method that is specifically used to discover unprecedented anomaly patterns that may be related to security threats from a large amount of unlabeled data. It is particularly important in the field of network security. It is mainly used to deal with zero-day attacks, which have not yet been captured by security defense systems due to their novelty and do not have known characteristic signatures or behavior patterns, making it difficult for traditional signature-based protection methods to effectively identify them.

In practical applications, zero-positive anomaly detection does not rely on previous training of abnormal samples, but builds a model that deeply understands normal behavior patterns through deep statistical analysis of network traffic, system behavior or other related data under normal conditions. The model aims to reveal the inherent structural relationships and typical behavioral characteristics of the data, and once a significant deviation from the normal pattern is encountered, it can be marked as a potential anomaly or attack activity.

The core challenge is how to accurately distinguish true abnormal behaviors from a large amount of network traffic data, which requires the algorithm to be highly robust and accurate. This involves understanding and extracting the effectiveness of key features, fine-tuning model performance, and balancing the delicate relationship between detection sensitivity and false alarm rate. At the same time, given the dynamic evolution of the network environment, the zero-positive anomaly detection system must also have the ability to self-update and continuously learn in order to adapt to new threat situations in a timely manner.

In this embodiment, the zero-positive anomaly detection in step 3 adopts a method based on statistical analysis. The scoring (standard score) method quantifies the degree of deviation of a data point from the normal behavior pattern. The anomaly score calculation formula for zero positive anomaly detection is:

,

in, is the anomaly score for zero positive anomaly detection, is the observed value, is the mean value of the data set, is the standard deviation;

The larger the absolute value of , the more likely the observation is abnormal.

In this embodiment, the isolation forest model is combined with zero positive anomaly detection in step 3 to obtain an integrated detection model represented as:

,

in, represents the comprehensive score of anomaly detection, Represents the anomaly score of the isolation forest model The weight of Indicates the score of zero positive anomaly detection The weight of =1.

For example, The value is 10, when the number of known features is 7, the number of unknown features When the number is 3, dynamic adjustment The value is 70%, which is adjusted dynamically The value is 30%.

The above method realizes the dynamic allocation of the weights of the isolation forest model and the weights of zero-positive anomaly detection, which can ensure the accuracy and interpretability of the anomaly detection results. It can not only detect anomalies, but also give a certain degree of explanation for the type of anomaly. For example, the higher the weight of zero-positive anomaly detection, the more likely it is to be a zero-day vulnerability.

In this embodiment, and The value of is:

Step A1: Define a total quantity of Key features of;

Step A2: Check whether each key feature matches a known security threat feature to identify whether it is a known feature or an unknown feature. When a feature exists in the known threat database, it is marked as a known feature and the number of known features is recorded. ; When a feature does not exist in the known threat database, it is marked as an unknown feature and the number of unknown features is recorded as ;

Step A3: Calculate the ratio of known features to , and dynamically adjust The value of ; Calculate the ratio of unknown features to , and dynamically adjust The value of ;

Step A4: For unknown features, once detected, they will be recorded in the known threat database, and the key feature database will be regularly adjusted and updated.

Step 4: Threat level assessment and adaptive strategy formulation: Based on the detection results of the integrated detection model, classify the threat level of each detection event, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy;

In this embodiment, the specific process of classifying the threat level of each detection event according to the detection results of the integrated detection model in step 4 is as follows:

Step 401: Comprehensively score the anomaly detection From the minimum value of the anomaly detection comprehensive score To the maximum value of the anomaly detection comprehensive score Divided into 5 areas , ),[ , ),[ , ),[ , ),[ , ];in, , , , All are anomaly detection comprehensive scores The values in are arranged from small to large;

When it comes to specific implementation, , , , Average distribution in arrive between;

Step 402: Determine The overall score of anomaly detection for each test In which area is it located? The threat level of this detection; specifically:

when lie in[ , ) area, that is, When The threat level of this detection is normal, which means that the network traffic and behavior patterns conform to the preset normal behavior model, with no abnormal activities or only safe and expected network behaviors;

when lie in[ , ) area, that is, When The threat level of this detection is suspicious behavior, which means that the behavior pattern deviates slightly from the normal model, but does not reach the clear attack characteristics; the behavior deviates from the normal pattern, but there is not enough evidence to indicate that it is a malicious attack;

when lie in[ , ) area, that is, When The threat level of this detection is a mild attack, which means that behaviors with attack characteristics are detected, but the impact on the system or data is small; the attack behavior is clear, but the impact is limited, such as attempted intrusion activities;

when lie in[ , ) area, that is, When The threat level of this detection is moderate attack, which means that the attack behavior is confirmed and the attack has successfully broken through a certain security line, posing an obvious threat to the system; the attack is more serious and may pose a greater threat to system security, such as successful SQL injection;

when lie in[ , ] area, that is, When The threat level of this detection is severe attack, which means large-scale, high-intensity attack behavior that may cause serious system damage or data leakage; high-risk attack poses a direct and serious threat to system or data security, such as large-scale DDoS attack.

Threat level determination is a key part in modern network security systems, especially in complex and dynamic network environments. Accurately identifying and classifying network threat levels is crucial for effectively defending against attacks. The threat level determination method proposed in the present invention monitors and analyzes network activities in real time and divides them into five threat levels: normal, suspicious behavior, mild attack, moderate attack and severe attack. This method can improve the security system's ability to identify various types of attacks, and can also take corresponding defense measures according to different threat levels, thereby improving the security of the entire network.

The above threat level classification method is expressed in a table as follows:

Table 1 Threat level classification standards

,

In practice, the system will analyze and evaluate these activities based on the real-time monitored network activity data. Once suspicious or malicious behavior is detected, it will be classified into the corresponding threat level according to the severity of the behavior, and appropriate response measures will be taken. In this way, the system can respond to various network attacks more flexibly and efficiently, thereby greatly improving the overall effectiveness of network security defense.

In this embodiment, the total number of a group defined in step A1 is When the key features are detected, two detection modes are defined. The first detection mode is the basic detection mode. The key features include URL features, HTTP methods, request body content, access time, source IP address, The value of is 5; the second detection mode is the enhanced detection mode, and the key features include URL features, HTTP methods, request body content, access time, source IP address, Headers request header content, access frequency, data packet length, attack source address and access port; The value of is 10;

When setting the dual-modal anomaly detection strategy according to the threat level and dynamically adjusting the anomaly detection strategy as described in step 4, lie in[ , ),[ , ) area, adjust the anomaly detection strategy to the basic detection mode; when lie in[ , ),[ , ),[ , ] area, adjust the anomaly detection strategy to enhanced detection mode.

The present invention adopts an innovative dual-modal anomaly detection strategy. Under normal circumstances, the system runs in basic detection mode to maintain efficiency and resource economy. Once an increased threat level is detected, the system automatically switches to enhanced detection mode to increase the depth and scope of detection to improve response efficiency. After the threat subsides, the system intelligently switches back to basic detection mode. This adaptive switching mechanism not only improves the response capability to network threats, but also effectively balances performance and resource consumption, thereby achieving optimal system performance.

Step 5: Real-time monitoring and dynamic response: Conduct real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors; this includes generating security alerts, isolating suspicious traffic, etc.

Step 6: Model update and continuous learning: To ensure that the model can effectively respond to the evolving threat situation, the system updates the isolation forest model and zero positive anomaly detection strategy with the latest network traffic data according to a predetermined period to maintain the sensitivity and accuracy of the integrated detection model;

By continuously learning and updating the detection model, the adaptability of network anomaly detection is improved, so that the system can adapt to changes in network behavior in a timely manner, improve the response speed, adaptability and accuracy to new threats, and ensure the efficient operation of the security system;

Step 7: Performance evaluation and optimization: The system conducts regular comprehensive performance reviews to measure key indicators such as detection accuracy, false alarm rate, and response time, and optimizes and adjusts detection parameters and strategies based on the evaluation results to ensure that the entire anomaly detection system operates efficiently and stably.

Example 2

The network anomaly detection system of the isolation forest fusion zero-positive anomaly detection of this embodiment includes:

Data collection and preprocessing module: used to capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data;

Feature extraction and analysis module: used to deeply mine key features in network traffic data, including abnormal request frequency, time series pattern and response data volume. These features help to clearly distinguish the boundary between normal and abnormal behaviors, and provide a data basis for subsequent model training combining isolation forest algorithm and zero positive anomaly detection;

Model integration and training module: used to train the isolation forest model and the zero positive anomaly detection model, and then combine the isolation forest model with the zero positive anomaly detection model to obtain an integrated detection model;

Threat level assessment and adaptive strategy formulation module: used to classify the threat level of each detection event according to the detection results of the integrated detection model, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy;

Real-time monitoring and dynamic response module: used to perform real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors;

Model update and continuous learning module: used to update the isolation forest model and zero-positive anomaly detection strategy using the latest network traffic data according to a predetermined period to maintain the sensitivity and accuracy of the integrated detection model;

Performance evaluation and optimization module: used to conduct regular comprehensive performance reviews of the system, measure key indicators such as detection accuracy, false alarm rate, and response time, and optimize and adjust detection parameters and strategies in a timely manner based on the evaluation results to ensure efficient and stable operation of the entire anomaly detection system.

During the specific implementation, a system information display module and a system control module are also set up. In the system information display module, various important information of the adaptive anomaly detection system for threat level determination will be displayed to the user, mainly including: model parameters, historical versions, abnormal event logs, threat level distribution, security alarm customization, adaptive strategy settings and real-time detection accuracy and other system information; the system control module includes: model parameter update, historical version rollback, abnormal event processing, dynamic threat level adjustment, anomaly detection strategy adjustment and dual-mode anomaly detection model setting and other functions; with these control functions, users can easily use this module in the user space to modify important system data at any time.

In response to rapidly evolving network threats, anomaly detection adaptive technology is crucial to enhancing the response speed and accuracy of network security systems. Faced with increasingly complex network attacks, traditional anomaly detection methods based on fixed rules or preset models are often difficult to adapt to the challenges of new or unknown attacks. Therefore, it is particularly urgent to develop a system that can dynamically adjust the detection strategy according to the real-time changes in the network environment.

In summary, the present invention proposes a network anomaly detection method and system that integrates an isolated forest and zero-positive anomaly detection. First, the isolated forest algorithm is combined with the zero-positive anomaly detection method, and then the potential threat level in the network traffic is dynamically evaluated to automatically adjust the detection strategy and response mechanism. The present invention can automatically adjust the detection accuracy and resource allocation according to the severity of the threat, thereby reducing the resource consumption of the system, and does not require pre-labeled data. It can improve the recognition ability of new attacks by applying it to real-time network traffic. The adaptive anomaly detection technology of the present invention can not only cope with complex attack threats, but also provide more reliable security protection for cloud computing and container technology. It is expected that this technology can significantly reduce the economic losses and service interruptions caused by network attacks, and provide an innovative solution for the current increasingly complex network security environment.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

The present application is described with reference to the flowcharts and/or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the processes and/or boxes in the flowchart and/or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims (9)

1. A network anomaly detection method for detecting zero positive anomaly of an isolated forest fusion is characterized by comprising the following steps of: the method comprises the following steps:

Step one, data collection and pretreatment: capturing network traffic data from a network interface in real time, and preprocessing the network traffic data;

step two, feature extraction and analysis: the key features in the network traffic data are deeply mined, including abnormal request frequency, time sequence mode and response data size;

Step three, model integration and training: unsupervised training of an isolated forest model to identify potential abnormal points in data, identifying and responding to novel threats which do not accord with a known mode by adopting a zero positive abnormal detection strategy, and combining the isolated forest model with zero positive abnormal detection to obtain an integrated detection model;

Step four, threat level evaluation and self-adaptive strategy formulation: threat level classification is carried out on each detection event according to the detection result of the integrated detection model, a bimodal anomaly detection strategy is set according to the threat level, and the anomaly detection strategy is dynamically adjusted;

Step five, real-time monitoring and dynamic response: monitoring in real time in a network environment, detecting abnormality through an integrated detection model, and performing instant response on the detected abnormal behavior;

Step six, model updating and continuous learning: the system updates the isolated forest model and the zero positive anomaly detection strategy by utilizing the recent network traffic data according to a preset period so as to maintain the sensitivity and accuracy of the integrated detection model;

Step seven, performance evaluation and optimization: the system regularly carries out comprehensive performance examination, measures the detection accuracy, false alarm rate and response time, and carries out timely optimization adjustment on detection parameters and strategies based on the evaluation result so as to ensure the high-efficiency stable operation of the whole abnormal detection system.

2. The network anomaly detection method for detecting zero positive anomaly in isolated forest fusion of claim 1, wherein the method comprises the steps of: the network flow data in the first step comprises an IP address, port information, a flow type and a request frequency; the preprocessing operations include screening, format conversion, and purification processes.

3. The network anomaly detection method for detecting zero positive anomaly in isolated forest fusion of claim 1, wherein the method comprises the steps of: in the third step, the isolated forest algorithm recursively divides the data space by randomly selecting data features and segmentation values until each data point is isolated or the tree reaches the depth limit; calculating an abnormal score of the data point to be evaluated through the constructed forest so as to judge whether the data point is an abnormal point or not; the anomaly score of the isolated forest model is expressed as:

，

Wherein, As a data point to be evaluated,For the number of samples to be taken,Representation ofThe path length in the isolated tree is such that,Representation ofThe desired path length in all the orphan trees,For the number of samples ofThe average search path length of the binary ordering tree at this time is expressed as:

，

Wherein, Is the firstNumber of reconciliation, and，Is Euler constant;

When the abnormality score is closer to 1, the likelihood of being judged as an abnormal point is higher; when the abnormality score is far less than 0.5, it is judged as not an abnormal point; when the anomaly scores of the sample points are all close to 0.5, the samples are judged to contain no obvious anomaly points.

4. The network anomaly detection method for detecting zero positive anomaly in isolated forest fusion of claim 3, wherein the method comprises the steps of: the zero positive anomaly detection in the third step adopts statistical analysisThe scoring method quantifies the deviation degree of data points and normal behavior modes, and an abnormal score calculation formula for zero positive abnormal detection is as follows:

，

Wherein, An anomaly score for zero positive anomaly detection,In order to observe the value of the value,As an average value of the data set,Is the standard deviation;

the larger the absolute value of (c) is, the more likely the observed value is to be abnormal.

5. The network anomaly detection method for detecting zero positive anomaly in isolated forest fusion of claim 4, wherein the method comprises the steps of: in the third step, combining the isolated forest model with zero positive anomaly detection to obtain an integrated detection model, wherein the integrated detection model is expressed as:

，

Wherein, Representing the composite score of the anomaly detection,Anomaly scoring representing isolated forest modelsIs used for the weight of the (c),Scoring indicative of zero positive anomaly detectionIs used for the weight of the (c),=1。

6. The network anomaly detection method for detecting zero positive anomaly in isolated forest fusion of claim 5, wherein the method comprises the steps of: And (3) with The value method of (2) is as follows:

Step A1, defining a group of total numbers as Key features of (2);

step A2, checking whether each key feature matches a known security threat feature to identify whether it is a known feature or an unknown feature, when a feature exists in the known threat database, marking the feature as a known feature, recording the number of known features as ; When a feature does not exist in the database of known threats, the feature is marked as an unknown feature, and the number of unknown features is recorded as；

Step A3, calculating the quantity proportion of the known features as followsAnd dynamically adjustThe value of (2) is; Calculating the quantity proportion of unknown features asAnd dynamically adjustThe value of (2) is；

And A4, for the unknown characteristics, after being detected, the unknown characteristics are recorded into a known threat database, and the key characteristic library is adjusted and updated regularly.

7. The network anomaly detection method for detecting zero positive anomaly in isolated forest fusion of claim 6, wherein the method comprises the steps of: in the fourth step, the specific process when threat level classification is performed on each detection event according to the detection result of the integrated detection model is as follows:

step 401, comprehensively scoring anomaly detection Composite score minimum from anomaly detectionTo the maximum value of the anomaly detection comprehensive scoreDivided into 5 regions [,)、[,)、[,)、[,)、[,The ]; wherein,、、、Are all comprehensive scores of anomaly detectionThe values of the two are arranged from small to large;

Step 402, judge the first Anomaly detection composite scoring for secondary detectionIn which region to determineThreat level of secondary detection; the method comprises the following steps:

When (when) Located at [,) Judging the first time when the area is withinThe threat level of the secondary detection is normal, indicating no abnormal activity or only safe, expected network behavior;

When (when) Located at [,) Judging the first time when the area is withinThe threat level of the secondary detection is suspicious behavior, which indicates that the behavior deviates from a normal mode, but no enough evidence indicates a malicious attack;

When (when) Located at [,) Judging the first time when the area is withinThe threat level of the secondary detection is light attack, which means that the attack behavior is definite but the influence is limited;

When (when) Located at [,) Judging the first time when the area is withinThe threat level of the secondary detection is moderate attack, which means that the attack is serious and can cause larger threat to the system security;

When (when) Located at [,Judging the first time when the area is withinThe threat level of the secondary detection is a serious attack, which represents a high-risk attack and forms a direct serious threat to the system or data security.

8. The network anomaly detection method for detecting zero positive anomaly in isolated forest fusion of claim 7, wherein the method comprises the steps of: defining a group of total numbers as in step A1The key features of (1) are defined in two detection modes, the first detection mode is a detection mode based on the first detection mode, the key features comprise URL features, HTTP methods, request body content, access time, source IP address,The value of (2) is 5; the second detection mode is an enhanced detection mode, wherein the key features comprise URL features, HTTP methods, request body contents, access time, source IP addresses, headers request header contents, access frequency, data packet length, attack source addresses and access ports; The value of (2) is 10;

setting a bimodal anomaly detection strategy according to the threat level and dynamically adjusting the anomaly detection strategy in the fourth step, when Located at [,)、[,) When the detection area is within the area, adjusting an abnormal detection strategy as a basic detection mode; when (when)Located at [,)、[,)、[,And adjusting the abnormality detection strategy to the enhanced detection mode when the detection area is within the area.

9. A network anomaly detection system for performing isolated forest fusion zero positive anomaly detection of the network anomaly detection method of claim 1, comprising:

and the data collection and preprocessing module is used for: the network traffic data processing device is used for capturing network traffic data from a network interface in real time and preprocessing the network traffic data;

And the feature extraction and analysis module is used for: the method is used for deeply mining key features in network flow data, including abnormal request frequency, time sequence mode and response data size, and provides a data basis for model training combined with zero positive anomaly detection of a subsequent isolated forest algorithm;

Model integration and training module: the method comprises the steps of training an isolated forest model and a zero positive anomaly detection model, and combining the isolated forest model with the zero positive anomaly detection model to obtain an integrated detection model;

threat level assessment and adaptive policy making module: the method comprises the steps of carrying out threat level classification on each detection event according to the detection result of an integrated detection model, setting a bimodal anomaly detection strategy according to the threat level, and dynamically adjusting the anomaly detection strategy;

And the real-time monitoring and dynamic response module is as follows: the system is used for monitoring in real time in a network environment, detecting abnormality through an integrated detection model and performing instant response on the detected abnormal behavior;

Model updating and continuous learning module: updating the isolated forest model and the zero positive anomaly detection strategy with recent network traffic data at predetermined periods to maintain sensitivity and accuracy of the integrated detection model;

Performance evaluation and optimization module: the system is used for regularly and comprehensively inspecting the performance, measuring the detection accuracy, the false alarm rate and the response time, and timely optimizing and adjusting the detection parameters and strategies based on the evaluation result so as to ensure the high-efficiency and stable operation of the whole abnormal detection system.