CN114124503B - An Intelligent Network Awareness Method for Level-by-Level Concurrent Cache Optimizing Efficiency - Google Patents

Abstract

The invention discloses an intelligent network perception method for level-by-level concurrent cache optimization efficiency, which belongs to the field of network security. High-capacity memory and NIC core, build large-capacity on-chip storage resource SPM, and cache traffic obtained by mirroring. Based on DPDK technology, multiple physical network cards can collect traffic concurrently in sub-regions. Data packets are received and distributed through two-level lock-free concurrency and user-mode protocol stacks; then, through each protocol field analysis and concurrency detection processing, IP protocol data field extraction, filtering, and cache processing are completed, and distributed to different capabilities through three channels The subsystem performs detection processing; finally, it integrates the results of various detections, saves abnormal samples, samples normal samples, and associates rich contextual information. The invention meets the deployment requirements of a high-performance security situational awareness system in a future smart home network environment.

Description

An Intelligent Network Awareness Method for Level-by-Level Concurrent Cache Optimizing Efficiency

technical field

The invention belongs to the technical field of network security, and in particular relates to an intelligent network perception method for level-by-level concurrent cache optimization efficiency.

Background technique

The proliferation and intelligence of IoT devices and the mass adoption of cloud services have driven the rapid rise of new edge computing architectures. The new architecture is around users. By building a large-scale autonomous IoT system, user-oriented characteristic services such as unified management of equipment, data caching, privacy protection and intelligent services are formed, which facilitates one-point management and decision-making for users. Users can use intelligent and high-performance computing type devices to build multiple redundant local aggregated service centers to solve problems such as response time, battery life, intelligent governance of sensor networking, bandwidth saving, and data security and privacy.

However, the traditional intrusion detection and situational awareness system has problems such as high false alarm rate, loopholes and poor application-layer detection capabilities. Different types of data in multiple dimensions such as internal management (existing structured data such as status, audit, and relationship) are associated with extended context information to improve the capabilities of attack detection, anomaly prompts, and associated early warning prompts. And the future-oriented 6G high-performance smart home security gateway can connect mirror ports and home LAN, and collect network traffic data at high speed.

The constructed lightweight intrusion detection and prevention system can be deployed and integrated into the edge devices of the Internet of Things to achieve full-dimensional extraction of features and user portraits. In order to adapt to the characteristics of the massive time series data sources of the Internet of Things, it is also necessary to adopt the big data management technology of the lightweight layered and sharded method to realize the layer-by-level desensitization storage of memory, local hard disk and cloud disk. Iteratively optimizes the appropriate detection models and matching datasets for different time series preservation requirements, forming the ability to automate evolution and adapt to different environments.

With the gradual maturity of edge computing technology, as well as the characteristics of real-time and close-to-user services, it has been able to support the security protection requirements of real-time detection and response of families. The 6G high-speed gateway will become a key network device configuration in the home in the future. Based on the mirror port of the 6G gateway and the transparent connection to the home LAN, it can complete the collection of basic network traffic, asset data, and log recording tasks, and integrate scattered data and redundant multi-source alarms. , in-depth analysis of the content of the message, presenting and predicting the comprehensive security situation of gateway devices, host devices, security devices, mobile devices and IoT devices in the future smart home. It is imminent to study the next-generation intelligent network security situational awareness model with the characteristics of full linkage.

SUMMARY OF THE INVENTION

The centralized acquisition, detection and storage method based on the existing centralized network situational awareness system ignores user data rights and experience, ignores the status quo of environmental limitations, and cannot effectively solve the problem of real-time situational awareness and threat assessment in the smart home environment. The present invention proposes an intelligent network perception method for level-by-level concurrent cache optimization efficiency, which is a parallel implementation of light-weight network traffic collection, detection, analysis and storage based on edge computing technology in an AIoT (Artificial intelligence internet of things) environment. Processing and solving problems such as the efficiency, self-adaptation, interpretability and hierarchy of network security situational awareness in the smart home environment.

The specific steps of the network traffic collection, detection and storage method are as follows:

Step 1. On the SoC-based embedded computing platform, pair a single network card core and a single CPU core to form independent partitions. In each partition, several large-capacity on-chip storage resource SPM (Scratch PadMemory) blocks are constructed through integrated memory. Directly using the memory address for access; thus constitutes a home Internet communication gateway device.

Each SPM block stores 2M data, including the data of the network card and the data processed by the CPU; several SPM blocks build a cache partition to adapt to the packet processing rate requirements of different types of network cards; directly access the storage on each SPM block through the memory address The data, through the starting address and the length of the payload data, perform continuous streaming access between each block without passing through the data bus. It realizes zero-copy direct access to data in user mode and a non-packet loss technology for continuous streaming read from large cache.

Step 2: Connect the home Internet communication gateway device to each Internet access device in the home respectively, each network card collects the traffic of different Internet access devices in parallel, and performs user-mode multi-threaded concurrent processing based on the data plane development kit DPDK (Data Plane DevelopmentKit) technology;

The specific process is:

First, each network card is received by a main distribution process. After sorting the received packets by 5-tuple, a unique concurrent UFID is constructed by hash calculation of the string. The calculation formula is as follows:

UFID=shash(sort(srcip,srcport,dstip,dstport),protocol) (1)

Among them, srcip, srcport, dstip, and dstport are the source ip address, source port, destination ip and destination port respectively; the two-way data packets are guaranteed to be sent and received through sort processing. These four fields are spliced into a string in the same order, and the protocol protocol type field is added after the , use the shash algorithm to calculate the string hash id;

Then, according to the number of concurrent channels C, the strings of the bidirectional data packets of the same flow are hashed and then stored in the corresponding Qi queues, and distributed to the processing sub-processes through the queues.

Step 3: The processed traffic packets are distributed to the three-channel subsystem for detection in parallel according to IP protocol detection and routing configuration;

After each subprocess completes IP protocol data field extraction, filtering and caching for each traffic packet, it is processed by three subsystems, and the records of abnormal network behavior are detected and merged in parallel; the three-channel subsystem includes the application layer abnormal rule detection subsystem, network Layer traffic signature and anomaly model detection subsystem, PCAP and forensic file detection subsystem;

Specific steps are as follows:

Step 301: Distribute the processed traffic packets to the application layer abnormal rule detection subsystem in an IP tunnel mode, extract the metadata of the definition rule and the application layer protocol metadata, and perform rule matching type detection by calculating logical expressions;

Metadata includes statistical fields such as network tcp/ip packet quintuple, each protocol field, the number of upstream bytes and the number of downstream bytes;

Application layer protocol metadata such as http protocol URL, request code, response code, submitted data and response data;

The rule-matching detection detects abnormal behaviors with obvious characteristics of network traffic and protocol fields, and generates real-time alarms.

Step 302: The extracted application layer protocol metadata is forwarded to the machine learning model through the distributed message queue, and the quasi-real-time depth detection is performed by using the network layer traffic characteristics and anomaly model detection subsystem, and the fuzzy user behaviors such as semantics and similarity are found. Tags, such as malicious behavior, abnormal data transmission, information detection and other active tags;

Specifically, for network traffic that carries and triggers malicious behavior at the application layer, multiple analysis models such as feature field scanning, state machine inference, machine learning model classification, abnormal semantics, and abnormal logic flow are established to label network traffic from multiple dimensions. the type of application behavior;

Step 303: At the same time, directly store the IP protocol data fields in the extracted metadata in the time series database, perform stream processing and statistics on various indicators, and forward the statistical indicators to the machine learning model through the distributed message queue, and use the network The layer traffic feature and anomaly model detection subsystem performs long-term network behavior anomaly detection and classification detection, and finds unknown anomalies represented by time-series statistical indicators;

Step 304: Back up the PCAP file through the locally created virtual network card, scan the PCAP file according to the subscription rules, and collect forensic resource files transmitted over the network, and forward it to the PCAP and forensic file detection subsystem through the distributed message queue for virus detection.

At the same time, establish the classified historical archives of data resource files sent/received by users;

Step 4: Iteratively fuse each detection result output by the three-channel subsystem;

The specific iterative fusion process includes:

First, regularly collect the detection results output by the three-channel subsystem, enhance the training data through dimensionality reduction, vectorization, sparseness and reconstruction, train detection and generate a deep learning model, and use reinforcement learning and confrontation generation algorithms to build a sustainable and autonomous model. Learning training environment.

Then, according to the criteria of high accuracy and low time-consuming, the trained deep learning models are sorted and selected, the weights of the long-term historical models are reduced exponentially, and the time-series exponential moving average weighted fusion learning is performed to obtain the fusion detection. result;

The specific integration is carried out according to three themes, including:

1) Integrate the multi-source deep learning model obtained by the network layer, application layer, user and relation layer, and perform integrated learning through Bayesian inference;

2) Build a relationship set and a composite alarm event set through the knowledge graph;

3) Construct typical behavior labels through typical user behavior portraits, divide single-event detection results, and label group event sets.

Finally, the Bayesian network model is modeled based on the historical sample detection accuracy of the model, the joint detection probability of each classification of the new sample is calculated inference, and the deep learning model is continuously updated with the results.

Step 5: Regularly query the fused detection result data set, and process PCAP and forensic documents.

First, define the event storage period of different security levels and divide the storage requirements of data. Then, according to the result samples that need to be stored for a long time, the related files are filtered, abbreviated and indexed to form abbreviated files and sample indexes.

Step 6: Transfer the daily traffic statistics, normal traffic sampling and detection results to the big data platform in batches, store them in categories, and gradually clean up event data of different security levels according to the defined life cycle.

The advantages of the present invention are:

1), an intelligent network perception method that optimizes the efficiency of concurrent caches by level, using the Internet of Things and edge computing technology to refine big data and intelligent detection on the user side, realize the concurrency and caching methods at all levels of network data stream processing, reduce It can reduce energy consumption, improve system performance, and meet the deployment requirements of high-performance security situational awareness systems in the future smart home network environment.

2), an intelligent network perception method that optimizes the efficiency of concurrent caches, and a network security situational awareness model with full linkage characteristics, sinking security services to each edge network, and is closer to the specific user needs of each type of environment;

3), an intelligent network perception method for optimizing the efficiency of concurrent caches layer by layer, collecting and processing the protocol metadata of each layer defined by the interconnection reference model of the open communication system, analyzing the communication protocol and statistical characteristics layer by layer, and establishing the characteristics experimentally Relationship with common attack types; propose a high-performance hardware implementation scheme and software extension mechanism, and implement concurrency and caching methods at all levels of network data stream processing;

4), an intelligent network perception method for level-by-level concurrent cache optimization efficiency, proposes a three-channel subsystem to concurrently extract features of each layer and select a matching model for detection; iterative fusion of detection results through multi-source and multi-modality improves detection. accuracy, interpretability of results, and adaptability to environmental changes.

Description of drawings

Fig. 1 is a kind of flow chart of the intelligent network sensing method of the present invention that optimizes the efficiency of a kind of concurrent cache;

FIG. 2 is a structural diagram of a hierarchically concurrent network traffic collection, detection, and storage system implemented by the present invention.

FIG. 3 is a functional diagram of a physical device based on the 5G edge computing gateway technology of the present invention.

FIG. 4 is a process flow chart of the specific lightweight collection, detection, analysis and storage of the present invention.

FIG. 5 is a flow chart of iterative fusion of multi-source and multi-mode detection results of the present invention.

Detailed ways

The present invention will be described in detail below through the accompanying drawings and embodiments.

The present invention proposes an AIoT network environment for smart homes, a solution for light-weight network traffic collection and network security situation intelligent analysis based on edge computing technology, oriented to the high-performance requirements of the Internet of Things, concurrent division and miniaturization. Data collection, detection and storage solve the problems of detection, fusion classification and response of abnormal network events.

The network traffic collection, detection and storage method is shown in Figure 1, and the specific steps are as follows:

Step 1. On the SoC-based embedded computing platform, pair a single network card core and a single CPU core to form independent partitions. In each partition, several storage resource SPM (Scratch PadMemory) blocks are constructed by integrating high-speed and high-capacity memory. Directly using the memory address for access; thus constitutes a home Internet communication gateway device.

Each NIC core is equipped with a CPU core; SPM organizes data in the form of blocks, each block stores 2M data, and uses 2K to 20K SPM blocks to build a large-scale cache partition to adapt to different types of NIC packet processing speed requirements;

The data processed by the network card and CPU is directly stored in each SPM block. For subsequent data processing, the data stored on each SPM block is directly accessed through the memory address, and the continuous access between the blocks is performed through the starting address and the length of the load data. , without passing over the data bus.

Establish a dedicated addressing access channel for the attributable computing core of SPM, map it into large page memory, and use it as a network card cache and user mode cache. The on-chip network card core only needs to collect and copy data into the SPM area once, and the subsequent processing can be combined with NUMA awareness and large page memory. The page memory mechanism transparently uses SPM memory to achieve zero data copying, reducing latency and power consumption caused by copying operations.

Step 2: Connect the home Internet communication gateway device to each Internet access device in the home respectively, each network card collects the traffic of different Internet access devices in parallel, and performs user-mode multi-thread concurrent processing based on the DPDK technology;

Based on the DPDK technology, multiple physical network cards are implemented in sub-regional and two-level lock-free concurrent traffic collection. The specific process is as follows:

First, each network card is received by a main distribution process. After sorting the received packets by 5-tuple, a unique concurrent UFID is constructed by hash calculation of the string. The calculation formula is as follows:

UFID=shash(sort(srcip,srcport,dstip,dstport),protocol) (1)

Among them, srcip, srcport, dstip, and dstport are the source ip address, source port, destination ip and destination port respectively; the two-way data packets are guaranteed to be sent and received through sort processing. These four fields are spliced into a string in the same order, and the protocol protocol type field is added after the , use the shash algorithm to calculate the string hash id;

Then, according to the number of concurrent channels C, the strings of the bidirectional data packets of the same flow are hashed and then stored in the corresponding Qi queues, and distributed to the processing sub-processes through the queues.

Each process and processing sub-threads are bound to corresponding CPU cores for processing, reducing process and thread scheduling overhead.

The Hash algorithm is completed by comparing the performance with the algorithm that saves the most CPU calculation. Data packet reception is performed through two-level lock-free concurrency and user mode protocol stack, and high-speed traffic above 10G can be decomposed into dozens of parallel processes for processing; fully utilize the capabilities of general-purpose multi-core processors;

The MID obtained after taking the modulo can be used as the subsequent deep packet detection partition, the sensor ID partition of the time series database, the PCAP file stream ID retrieval, and the pre-partition ID of the big data platform, supporting the shunting and concurrent processing of each link partition.

Step 3: The processed traffic packets are distributed to the three-channel subsystem for detection in parallel according to IP protocol detection and routing configuration;

After each sub-process completes the IP protocol data field extraction, filtering and buffering of the traffic packets, it is distributed to subsystems with different capabilities through three channels for parallel detection and integration of abnormal network behavior records; three channels refer to computationally intensive Application layer anomaly rule detection subsystem, memory-consuming network layer traffic characteristics and anomaly model detection subsystem, high-capacity storage type PCAP and forensic file detection subsystem;

Concentrate a batch of data by extracting, filtering, and caching, forward multiple models asynchronously, and detect and fuse records of abnormal network behavior in parallel; the specific steps are as follows:

Step 301: Distribute the processed traffic packets to the application layer anomaly rule detection subsystem in an IP tunnel mode, extract the metadata defining the rule and the application layer protocol metadata in real time, and perform a preliminary rule matching formula based on logic expression calculation. detection;

Metadata includes statistical fields such as network tcp/ip packet quintuple, each protocol field, the number of upstream bytes and the number of downstream bytes;

Application layer protocol metadata such as http protocol URL, request code, response code, submitted data and response data;

Rule-matching detection is designed to detect abnormal behaviors with obvious characteristics of network traffic and protocol fields, such as port scanning, DDoS or brute force cracking.

Step 302, the extracted application layer protocol metadata is forwarded to the machine learning model through the distributed message queue, and the quasi-real-time depth detection is performed by using the network layer traffic characteristics and the abnormal model detection subsystem;

Specifically, for network traffic carried by malicious codes, viruses and Trojan horses, and application layer vulnerability exploits in the application layer and triggering malicious behavior execution, establish feature field scanning, state machine reasoning, machine learning model classification, abnormal semantics and abnormal logic flow and other analysis models to label the application behavior types of network traffic from multiple dimensions;

Step 303: At the same time, perform stream processing on the extracted IP protocol data fields, and store the processed stream characteristics, time, host, service and other related statistical indicators as well as 5-tuples in a time series database; forwarding through distributed message queues To the machine learning model, use the network layer traffic characteristics and anomaly model detection subsystem to perform long-term network behavior anomaly detection and classification detection;

Statistical features are long-period indicators of multiple minutes, hours, days, weeks, months, and years, and the linear combination of the mean, standard deviation, information entropy, and probability density distribution of random processes of statistical feature fields such as the length and frequency of data packets;

Step 304, through the locally created virtual network card, land the PCAP backup file, scan the PCAP file according to the subscription rule and obtain the resource file transmitted by the network, and forward it to the PCAP and the forensic file detection subsystem through the distributed message queue for virus detection;

Use PCAP files to store original network messages, extract various types of files, and distribute them to PCAP and forensic file detection subsystems through distributed message queues; at the same time, establish a classified historical archive of data resource files sent and received by users to store evidence, which is convenient for query, Forensics and source tracing, to discover abnormal leaks of user data resource files.

Step 4: Iterative fusion of multi-source and multi-mode detection results is realized for each detection result output by the three channels;

The specific iterative fusion process is as follows:

Step 401, establishing a self-learning and self-adaptive twin environment, continuously extracting typical data packets and typical statistical features in the environment, iterative training, and enhancing the detection capability of the model;

In the adaptive environment, through the automatic periodic sliding window method, the typical samples of each analysis topic generated in the actual environment are added, and extracted using similarity algorithms such as clustering, compression learning algorithms such as rule and model transfer, and incentive algorithms such as confrontation generation. Typical data packets; and sample and feature extraction through dimensionality reduction, vectorization, sparse and reconstruction methods; assist in the establishment of attack, defense, honeypot nodes, actively produce and label real samples in the actual environment; GAN based on deep learning Adversarial generative network models, training detection and generative deep learning models, multi-stage iterative confrontation, and production of enhanced samples. Divide the training set and the validation set, gradually improve the test set, and build the daily self-learning ability of the model.

Step 402: In the newly constructed twin environment, for data from different sources and features, sort and discover multiple optimal deep learning detection models trained in step 401, and then fuse the detection results of these detection models to perform Bayesian inference. Integrate learning, characterize the composite alarm event set, and perform label division of the group event set on the single event detection result.

According to the Naive Bayes inference formula, the product of the output probability of each model of each category and the posterior probability of the category can be calculated, and the category with the highest probability is taken as the discrimination result;

And further inferred the ensemble learning of Gaussian process probability parameter estimation and typical sample sampling loop iteration. Build a relationship set and characterize a composite alarm event set through the knowledge graph. The typical behavior label is constructed through the user's typical behavior portrait, and the group event collection is divided and labeled for the single-event detection result. Combine multiple alarms through association rules modeled by expert knowledge or association rules automatically counted to form a relationship graph in the form of a knowledge graph. According to this relationship graph, graph structure features such as themes, core events, closeness of relationship, and attack process can be presented to achieve automatic hierarchical fusion results and reduce the details that need attention and analysis.

Assuming that there are n different types of models, ensemble learning to identify the test sample X, it is necessary to build an ensemble model training set of m samples first. As the number of m continues to increase, the accuracy of the ensemble model will continue to improve, but the detection calculation time is fixed. . Let a total of K categories be C _k (k=1,2,...,K), the number of samples of each category in the training set is m _k , and the prior probability of category C _k is:

P(Y=C _k )=(m _k +λ)/(m+Kλ) (2)

λ is a discount factor between 0 and 1,

Define the accuracy of model A in identifying class C _k in the training set as the conditional probability:

For an instance X, run a test for each model n _j , obtain the probability P(n _j (X)|Y=C _k ) that predicts X to be a related model of category C _k , and calculate the prior probability of category C _k , The product of the conditional probabilities of these models that vote for C _k and the class probabilities of the model predicting X, the class corresponding to the maximum posterior probability is taken as the class C of X:

According to the activity characteristics and sequence relationship of each stage of the kill chain, the stage to which each detection result event belongs is deduced. After the complete activity process is established, different group event sets are divided through association analysis and feature iterative clustering.

Step 403 , according to the data of long time period and different time series period range, sort the found optimal detection model set, reduce the number of classification labels by clustering, reduce the weight of the long-term historical model according to the exponential ratio, and perform time series exponential moving average weighting. Integrated learning.

The model trained based on recent data is more in line with the laws of the current environmental characteristics, and continuously improves the system's ability to detect sudden anomalies, as well as the ability to adapt to environmental changes in a timely manner;

Attacks and security problems emerge in an endless stream in cyberspace, and there are strong periodic iterative development characteristics. Relevant models and samples need to keep pace with the times to detect unknown attacks in time and improve the identification accuracy of new attacks. Divide weeks, months, quarters, years, 3 years, etc. according to the time series, and divide the samples to train multiple periodic models mp, which can take into account the preservation and comprehensiveness, and reduce conflicts. At the same time, through the accumulation of different periods of history, iterative training can be used to select the optimal model mpm for the corresponding time period. In the real-time detection process, the following algorithm is used to weight and fuse the probability value results of each model's decision from near to far, and exponentially decay the long-term model to make fast decisions:

In the post-event comprehensive detection process, the Bayesian inference method in step 402 can also be used comprehensively to establish a comprehensive model training sample set covering a wide range, and fuse all the models mp trained according to time series to make more accurate judgments.

The specific integration functions are carried out according to three themes, including:

1) Ensemble learning through Bayesian inference;

Weighted integration according to the time dimension, such as the application of Monte Carlo method and clustering method, dynamic fusion to reduce classification labels, adjust the fitness weight of the model, and gradually adapt to the new environment by self-learning. Joint reasoning by multi-classifier integration, such as normal anomaly classification, can first use statistical models to accurately obtain abnormal samples, then use deep learning models to discover potential unknown anomalies, and finally use whitelist rules to filter abnormal samples to reduce the false positive rate. Finally, the Bayesian inference technology is applied, the Bayesian network model is modeled based on the historical sample detection accuracy of the model, the joint detection probability of each classification of the new sample is calculated by reasoning, and the network model is continuously updated with the results.

2) Build a relationship set and a composite alarm event set through the knowledge graph;

A compound alarm event is a combination of multiple alarms, association rules modeled by expert knowledge, or association rules automatically counted to form a relationship graph in the form of a knowledge graph. According to this relationship graph, graph structure features such as themes, core events, closeness of relationship, and attack process can be presented. Automatically hierarchical fusion results, reducing the need for attention and analysis of the details.

3) Construct typical behavior labels through typical user behavior portraits, divide single-event detection results, and label group event sets.

User portraits are used to analyze user behavior labels, so as to facilitate users to identify their typical daily activities through network data. Statistics such as frequency, amount of data sent and received, and duration can be made to form probabilistic hourly and daily data. Mean, standard deviation, information entropy, Gaussian process and other distribution characteristic indicators. When user behavior changes, these indicators will inevitably fluctuate greatly, and at the same time, the probability, mean, and standard deviation of various activities in time series can be observed.

Step 5: Regularly query the fused detection result data set, and process PCAP and forensic documents.

First, define the event storage period of different security levels and divide the storage requirements of data. Then, according to the result samples that need to be stored for a long time, the relevant files are filtered, abbreviated, and indexed to form abbreviated files and sample indexes; reduce the size of the stored files and speed up the efficiency of historical data retrieval;

Step 6: Transfer the daily traffic statistics, normal traffic sampling and detection results to the big data platform in batches, store them in categories, and gradually clean up event data of different security levels according to the defined life cycle.

Data such as traffic statistical indicators, normal traffic sampling, and test results are transferred to the big data platform in batches every day, and the abbreviated files are uploaded to the big data platform in batches every day.

Data and documents establish associations, extract contextual information and label data, and enrich topic and relationship information. The formed historical data results, combined with the automated training process, can continuously improve the model detection ability. Categorized storage, and gradually clean up event data of different security levels according to the defined life cycle.

Example:

An example of the system structure principle provided in this example is shown in Figure 2. First, a home Internet communication gateway device is built. Through mirroring configuration, all traffic on the Internet egress and traffic on multiple subnet segments within the home can be obtained; then mirror network traffic The acquisition and processing module integrates high-speed and high-capacity memory and network card cores on the SoC-based embedded computing platform, constructs a large-capacity on-chip storage resource SPM, and caches the traffic obtained by mirroring. Based on DPDK technology, multiple physical network cards can collect traffic concurrently in sub-regions. Data packets are received and distributed through two-level lock-free concurrency and user-mode protocol stacks; then, each protocol field analysis and concurrency detection processing module completes IP protocol data field extraction, filtering, and cache processing, and distributes them to different The capability subsystem performs detection processing; finally, the localized sample library and model performance analysis module integrates the results of various detections, saves abnormal samples, samples normal samples, and associates rich contextual information. Through the simulation of attack and defense environment and the generation of deep learning confrontation, the samples are enriched. Train models and evaluate model performance, and continuously improve and integrate excellent models; at the same time, based on the cloud platform's sample and file abbreviated and indexed storage modules, the discovered samples and forensic files need to be stored for a long time, and uploaded to the cloud platform's database and distributed data every day. During storage, it is classified according to the storage duration and automatically cleaned up.

The physical device function diagram based on 5G edge computing gateway technology provided in this example is shown in Figure 3. Through the high-performance 5G communication gateway deployed in the smart home, the Internet traffic and internal home network traffic are mirrored, and multiple groups of agents collect concurrently. The process is distributed to three computing and detection models, forming a multi-layer, multi-modal feature data set, and continuously training and integrating multiple models through the iterative optimization layer of the model to form a joint reasoning capability.

The flow of network traffic collection, detection, analysis, and storage in this embodiment is shown in Figure 4, and the specific steps are as follows:

Step 301: the embedded computing platform uses on-chip storage resources;

On the SoC platform, high-speed and high-capacity memory and network card core are integrated to provide a large-capacity on-chip storage resource SPM. The network traffic is processed by the on-chip network card kernel to control the cache, and the data is copied into the SPM area at a time. The process of reading network packets in user mode can be bound to the CPU of the corresponding area and transparently access the SPM area using the huge page memory mechanism.

Step 302: Two-level concurrent lock-free collection of traffic;

The user mode process is based on DPDK technology, and binds multiple mirrored network cards of different network segments, and collects traffic concurrently in different regions. The unique concurrent UFID is constructed by string hash calculation after sorting the 5-tuple of network packets. According to the number of concurrent channels C, the belonging queue Qi is selected after modulo, and distributed to the processing sub-processes through the queue.

Step 303: three-channel concurrent detection;

After the subprocess completes IP protocol data field extraction, filtering, and cache processing, it is distributed to the following subsystems with different capabilities through three channels for detection processing.

1. The IP tunnel mode distributes traffic to the application layer exception rule detection subsystem. The subsystem extracts the metadata that defines the rules and the metadata of the application layer protocol, performs preliminary rule matching detection, and after obtaining the application layer data, forwards the machine learning model through the distributed message queue for further in-depth detection and user behavior relationship analysis;

2. Network layer traffic characteristics and anomaly model detection subsystem. The extracted IP protocol data fields are directly stored in the time series database in the internal network for stream processing and various indicator statistics. The processed flow characteristics and correlation statistical indicators such as time, host, service, etc., are used for anomaly detection and attack classification detection through the distributed message queue forwarding machine learning model;

3. PCAP and forensic document detection subsystem. Through the locally created virtual network card, the PCAP backup file is landed, and the PCAP file is scanned according to the subscription rules for forensic network transmission of resource files, and virus detection is performed through the distributed message queue forwarding file detection model.

Step 304: Multi-source and multi-mode iterative fusion;

Automatic collection and labeling of new samples, adaptive environment, automatic periodic sliding window approach, append environment generated typical samples for each analysis subject. Assist in the establishment of attack, defense, and honeypot nodes, and actively produce and label real samples in the actual environment. Training detection and generation of two deep learning models, multi-stage iterative confrontation, and production of enhanced samples. Divide the training set and the validation set, and gradually improve the test set.

Sort data from different sources and characteristics to find the optimal detection model. The data in different time series ranges are sorted to find the optimal detection model sets in different time series ranges. In the real-time detection process, ensemble learning is performed using time-series exponential moving weights. Ensemble learning using Bayesian inference during post-hoc comprehensive detection.

Step 305: result data abbreviated and associated index;

The fused detection result dataset is queried regularly, and the process of processing PCAP and forensics files is started. Filter, abbreviate, and index the result samples that need to be stored for a long time to form abbreviated files and sample indexes.

Step 306: dump, multiplex, and classify storage data in batches asynchronously every day;

Traffic statistics indicators, normal traffic sampling, test results and other data are transferred to the big data platform in batches every day. The abbreviated files are uploaded in batches to the big data platform every day. According to the requirements of the preservation period, it is classified and stored, and the event data of different security levels is gradually cleaned up according to the defined life cycle.

This embodiment performs iterative fusion of multi-source and multi-mode detection results, and the process of continuously adapting, updating and enriching the sample set is shown in Figure 5, and the steps are as follows:

Step 401: establish an automatic collection and labeling mechanism for new samples;

A typical sample of each analytical subject produced in the actual environment is collected periodically. Actively produce and label real samples in real environments by establishing attack, defense, and honeypot nodes. Through training detection and generation of two deep learning models, multi-stage iterative confrontation and production of enhanced samples.

Step 402: analyze and extract typical samples;

Deduplication and blurring. Cluster analysis, extracting class center samples matching the labels, and finding outliers. Classification analysis of tree model, PCA principal component analysis, key features are obtained, and similar samples are clustered by key features.

Step 403: Divide the time series periodic data set to extract features;

Create a new time series periodic data set by week, month, quarter, year, and 3 years, and run the feature processing program on the extracted sample data to form a feature data set.

Step 404: Extract 70% of the data as a training set and a validation set.

Step 405: Train multiple machine learning models, and save the model parameters after training.

Step 406: extract 30% typical test data set;

Step 407: For the models obtained in each time sequence period, evaluate through the test data set, and sort and select the models according to the criteria of high accuracy and low time consumption.

Step 408: In order to obtain the matching information of the network protocol, the network layer, the application layer, the user layer and the relationship layer are divided into three layers, and the key elements of the protocol, statistics, and feature samples are extracted respectively.

Step 409: Use the network layer samples to perform training using the previously evaluated optimal model.

Step 410: Using the application layer samples, use the previously evaluated optimal model for training.

Step 411: Use the user and relationship layer samples to train using the previously evaluated optimal model.

Step 412: Multi-mode time series exponential moving weight integration;

In the real-time detection process, the optimal model for each time series range evaluated is fused, weighted from near to far, and the long-term model is exponentially decayed.

Step 413: Multi-source model Bayesian inference integration;

In the post-event comprehensive detection process, the multi-source models obtained in steps 409, 410, and 411 are integrated, and Bayesian inference is used for ensemble learning.

Step 414: Evaluate each optimal sub-model of the above steps, and compare with the multi-mode time series exponential moving weight integration model and the multi-source model Bayesian inference integration model.

Step 415 : At the end of each time sequence stage, after completing the above-mentioned sample set iteration and new model training, extract the model validation environment with improved accuracy for the newly established training data set and test data set. Enter the next round of data extraction, new models, and new and old evaluation and comparison processes.

The invention discloses a method for building a large-capacity on-chip storage resource SPM by integrating a high-speed and high-capacity memory and a network card kernel on an SoC-based embedded computing platform. The on-chip NIC core only needs to copy data into the SPM area once, and use the SPM memory transparently by combining NUMA awareness and huge page memory mechanism; a technology based on DPDK technology to realize concurrent collection of traffic in sub-regions of multiple physical NICs. Data packet reception is performed through two-level lock-free concurrency and user-mode protocol stack, and high-speed traffic above 10G can be decomposed into dozens of parallel processes for processing; a strategy based on IP protocol data field extraction, filtering, and cache processing, adapted to detection classification The resource requirements are distributed to different capability subsystems through three channels for detection processing. Concentrate a batch of data by extracting, filtering, and caching, asynchronously forward multiple models, and detect and fuse abnormal records in parallel; an iterative fusion algorithm for multi-source and multi-mode detection results, adaptive environment, automatic periodic sliding window method, additional environment The typical samples of each analysis topic generated, training detection and generation of two types of deep learning models, multi-stage iterative confrontation, sorting to find the optimal detection model set in different time series ranges, and exponentially reducing the weights of the long-term historical models. The fusion of multi-mode detection results continuously improves the detection capability of the system; a method of abbreviating and storing results, querying the fused detection result data set, and processing PCAP and forensic documents. Define the storage period of events of different security levels, filter, abbreviate, and index the result samples that need to be stored for a long time to form abbreviated files and sample indexes, reduce storage capacity, and improve retrieval efficiency; finally, traffic statistics indicators, normal traffic sampling, Data such as test results are transferred to the big data platform in batches every day, and the abbreviated files are uploaded to the big data platform in batches every day. Categorized storage, and gradually clean up event data of different security levels according to the defined life cycle. The present invention proposes a new high-performance hardware implementation scheme and software expansion mechanism for network traffic collection and analysis scenarios under the background of 6G full connection and high bandwidth, realizes concurrency and caching methods at all levels of network data stream processing, and reduces energy consumption , improve system performance, and meet the deployment requirements of high-efficiency security situational awareness systems in the future smart home network environment.

Claims (6)

1. An intelligent network sensing method for optimizing efficiency of a progressive concurrent cache is characterized by specifically comprising the following steps of: on an embedded computing platform based on SoC, pairing a single network card core and a single CPU core to form independent partitions, respectively constructing a plurality of high-capacity on-chip storage resource SPM (scratch Pad memory) blocks in each partition through an integrated memory, and directly accessing by using memory addresses; thereby constituting a home internet communication gateway apparatus;

then, the home internet communication gateway device is respectively connected with each internet device in the home, each network card parallelly collects the flow of different internet devices, and user-mode multithreading concurrent processing is carried out based on a data Plane Development kit DPDK (data Plane Development kit) technology;

the specific process is as follows:

firstly, each network card is packed by a distribution main process, the packed packets are sequenced according to 5-tuple, and the unique concurrent UFID is constructed by the hash calculation of a character string, wherein the calculation formula is as follows:

UFID＝shash(sort(srcip,srcport,dstip,dstport),protocol)

wherein, srcip, srcport, dstip and dstport are respectively a source ip address, a source port, a destination ip and a destination port; ensuring that a bidirectional data packet is transmitted and received through sort processing, splicing the four fields into character strings in the same sequence, and calculating a hash id of the character strings by using a shock algorithm after adding a protocol type field;

then, according to the number C of the concurrent channels, the character strings of the bidirectional data packets of the same flow are subjected to hash calculation, and then respectively belong to corresponding Qi queues for storage, and are distributed to a processing sub-process through the queues;

the processed flow packets are parallelly distributed to a three-channel subsystem for detection according to IP protocol detection and routing configuration; carrying out iterative fusion on each detection result output by the three-channel subsystem;

the three-channel subsystem comprises an application layer abnormal rule detection subsystem, a network layer flow characteristic and abnormal model detection subsystem, a PCAP (personal computer application protocol) and a evidence obtaining file detection subsystem;

the three-channel subsystem parallel detection method comprises the following specific steps:

step 301, after completing extraction, filtering and caching of IP protocol data fields of each flow packet by each subprocess, distributing the flow packets to an application layer abnormal rule detection subsystem in an IP tunnel mode, extracting metadata defining rules and application layer protocol metadata, and performing rule matching detection through a computational logic expression;

step 302, forwarding the extracted application layer protocol metadata to a machine learning model through a distributed message queue, performing near-real-time deep detection by using a network layer traffic characteristic and abnormal model detection subsystem, and finding out a user behavior label with fuzzy semantics and similarity;

step 303, at the same time, directly storing the IP protocol data fields in the extracted metadata into a time sequence database, performing stream processing and multiple index statistics, forwarding the statistical indexes to a machine learning model through a distributed message queue, performing long-period network behavior anomaly detection and classification detection by using a network layer flow characteristic and anomaly model detection subsystem, and finding unknown anomalies represented by time sequence type statistical indexes;

step 304, backing up the PCAP file through a locally created virtual network card, scanning the PCAP file according to subscription rules and forensics of the resource file transmitted by the network, and forwarding the PCAP file to a PCAP and forensics file detection subsystem through a distributed message queue for virus detection;

finally, inquiring the fused detection result data set regularly, and processing the PCAP and the evidence obtaining file; and (3) unloading daily flow statistical indexes, normal flow sampling and detection results into a big data platform in batches, storing the data in a classified mode, and gradually cleaning event data with different security levels according to a defined life cycle.

2. The intelligent network perception method for progressive concurrent cache optimization performance as claimed in claim 1, wherein said mass on-chip memory resource SPM block stores 2M data including network card data and CPU processed data; a plurality of SPM blocks construct a cache partition, and the requirement of different types of network card packet processing rates is met; the data stored on each SPM block is directly accessed through the memory address, and continuous stream access among the blocks is carried out through the initial address and the load data length without being transmitted through a data bus.

3. The intelligent network sensing method for optimizing performance of progressive concurrent caching as claimed in claim 1, wherein in step 301, the metadata comprises a network tcp/ip packet quintuple, protocol fields, an uplink byte number and a downlink byte number;

the application layer protocol metadata comprises a URL (Uniform resource locator) of an http protocol, a request code, a response code, submitted data and responded data;

and (4) detecting a rule matching type, and alarming in real time in order to find abnormal behaviors with obvious network flow and protocol field characteristics.

4. The intelligent network-aware method for progressive concurrent cache performance optimization according to claim 1, wherein the step 302 specifically comprises: aiming at network traffic carried and triggered by malicious behaviors to be executed in an application layer, a plurality of analysis models of characteristic field scanning, state machine reasoning, machine learning model classification, abnormal semantics and abnormal logic flow are established, and the application behavior types of the network traffic are labeled from a plurality of dimensions.

5. The intelligent network-aware method for progressive concurrent cache optimization performance according to claim 1, wherein the iterative fusion specific process comprises:

firstly, regularly collecting a detection result output by a three-channel subsystem, training, detecting and generating a deep learning model by reducing dimensions, vectorizing, sparsely reconstructing and enhancing training data, and constructing a sustainable self-learning training environment of the model by using an enhanced learning and confrontation generation algorithm;

then, sorting and selecting each trained deep learning model according to the standards of high accuracy and low time consumption, reducing the weight of a long-term historical model according to an index equal ratio, and performing time sequence type index moving average weighted fusion learning to obtain a fused detection result;

specific fusion is performed according to three topics including:

1) integrating a multi-source deep learning model obtained by a network layer, an application layer, a user and a relation layer, and performing integrated learning through Bayesian inference;

2) establishing a relation set and a composite alarm event set through a knowledge graph;

3) constructing a typical behavior label, dividing a single event detection result and labeling a group event set through the typical behavior portrait of the user;

and finally, modeling a Bayesian network model based on the historical sample detection accuracy, calculating the joint detection probability of each classification of the new sample by inference, and continuously updating the deep learning model by using the result.

6. The intelligent network sensing method for optimizing performance of progressive concurrent caching as claimed in claim 1, wherein said regularly querying the fused detection result dataset specifically comprises:

firstly, defining the storage periods of events with different security levels, and dividing the storage requirements of data;

then, according to the result sample which needs to be stored for a long time, the related files are filtered, reduced and indexed to form a reduced file and a sample index.

Images