CN105681303A - Big data driven network security situation monitoring and visualization method - Google Patents

Abstract

The invention relates to a network security situation monitoring and visualization method driven by big data. The method includes: 1) extracting network security basic data of different dimensions; 2) using Storm and Hadoop to store and process network security basic data, wherein Hadoop is used to process historical data, and Strom is used to process real-time data; 3) Hadoop uses The big data processing method extracts key security feature items from historical data, and establishes a database table structure to form a network security feature knowledge base; 4) Strom extracts relevant security feature items from real-time data, and compares them with network security feature The knowledge base performs feature matching to determine the network security situation; 5) dynamically visualizes the network security situation determined by Strom. The invention can effectively monitor the network security situation and comprehensively display the visualization results of the network security situation.

Description

A big data-driven network security situation monitoring and visualization method

technical field

The invention belongs to the field of network technology and information security technology, and in particular relates to a big data-driven network security situation monitoring and visualization method.

Background technique

The development of contemporary information technology promotes the generation, collection, transmission, sharing and analysis of data, making scientific and engineering research increasingly data-intensive. With the increasing network traffic, the types and complexity of attacks are also gradually increasing. The security data provided by various security systems, devices and platforms deployed on the network are widely distributed, cross-organizational, with large differences in formats, massive, and non-trivial. Numerical and other characteristics, the data dimension has been upgraded from a single dimension to multi-dimensional, no matter in terms of storage or computing, it is impossible to use traditional storage integration technology to complete the real-time and accurate judgment of the network security situation.

On the other hand, high-dimensional and massive data increase the difficulty of security personnel's work: (1) The cognitive burden is too heavy. It is difficult for analysts to make detailed reports on hundreds of millions of alarms within a limited time of a day through traditional log analysis methods. (2) The interaction is not enough. When suspicious events are found, the existing analysis methods cannot provide relevant data filtering, event details display and other functions to help analysts make further effective judgments; (3) Lack of With an overall understanding of the network, analysts often see only a single data record, and it is difficult to identify some complex, collaborative, and long-term network abnormal events. (4) Log analysis based on traditional databases is difficult to discover some new attack patterns, and it is impossible to predict or prevent the attack trend in advance.

Contents of the invention

Based on the above problems, the present invention proposes a real-time big data-driven network security situation monitoring and visualization method, which can effectively monitor the network security situation and comprehensively display the visualization results of the network security situation.

The technical scheme that the present invention adopts is as follows:

A big data-driven network security situation monitoring and visualization method, comprising the following steps:

1) Extract basic network security data of different dimensions, including real-time data and historical data;

2) The real-time computing system Storm and the distributed computing system Hadoop are used to store and process the basic data of network security, among which Hadoop is used to process historical data, and Strom is used to process real-time data;

3) The distributed computing system Hadoop uses big data processing methods to extract key security feature items from historical data, and establishes a database table structure to form a network security feature knowledge base;

4) The real-time computing system Strom extracts relevant security feature items from the real-time data, performs feature matching with the network security feature knowledge base, and determines the network security situation according to the matching result;

5) Dynamic and visual display of the network security situation determined by the real-time computing system Strom.

Further, the basic network security data of different dimensions in step 1) include websites, hosts, latitude and longitude, IP addresses, vulnerabilities, security, etc.; the historical data include quarterly website scan assessment reports and quarterly host scan assessment reports.

Further, the process of step 2) for real-time data is: first, real-time data will be sent to Flume, a massive log aggregation system, and data backup will be performed on the HDFS system at the same time; Flume will send the collected real-time data to the distributed message system Kafka for further processing; the data streams processed by Kafka are sent to the real-time computing system Storm one by one, and all real-time business logic is completed in Storm; finally, the processing results are pushed into the Redis storage system in a stack-like form, and the Web front end reads from Redis Extract the results and display them.

Further, step 2) for the processing of historical data is: send the historical data to the preprocessing integration module for simple format processing, then send it to the distributed computing system Hadoop for big data analysis and processing, and then store the statistical simple data To the Mysql database, store unstructured data in the Hbase database, and the web front-end does not need to perform logical processing, and directly reads the data in the database for display.

Further, step 3) the big data processing method includes one or more of the following: clustering and fusion, association analysis, entropy analysis, situation prediction; the security features include: IP source address, IP target address , event name, event category, security level, vulnerability code.

Further, the visual display in step 5) includes network security visualization based on real-time network traffic and network security visualization based on historical reports, and the specific content of the visual display includes a global dynamic attack map, a domestic dynamic attack map, a domestic security situation map, National security vulnerability distribution map, bulletin board and other functions.

The beneficial effects of the present invention are as follows:

In order to realize the powerful underlying analysis capability, the present invention adopts the Hhadoop+Storm distributed architecture; in order to fully and completely reflect the network security situation, it extracts basic network security data in different dimensions such as websites, hosts, latitude and longitude, IP addresses, vulnerabilities, security events, etc. ; In order to obtain real-time and effective security situation evaluation results, a self-learning security anomaly feature library is established; in order to comprehensively display the network security situation results, dynamic visualization technology is adopted.

With the help of the powerful underlying distributed storage and parallel computing capabilities, the present invention intelligently processes various network traffic and security logs from various security devices, obtains the most effective network security situation evaluation criteria at present, and carries out the process under the drive of real-time data. Real-time evaluation and screening, quick visualization of the results, so that security analysts can monitor the current overall security situation of the entire network and pay attention to key risk public opinion in the first place.

Description of drawings

Fig. 1 is a technical architecture diagram of the overall solution of the present invention.

Figure 2 is the overall processing logic diagram of the real-time data network security situation analysis implemented by Storm.

Fig. 3 is a processing flowchart of the ReadBolt module.

Fig. 4 is the processing flowchart of the IPBolt module.

Fig. 5 is a processing flowchart of the RollCountBolt module.

Fig. 6 is a processing flowchart of the FieldRankBolt module.

Fig. 7 is a processing flowchart of the GlobalRankBolt module.

Fig. 8 is a schematic diagram of a massive data visualization solution.

Figure 9 is a schematic diagram of a domestic dynamic attack map.

detailed description

The present invention will be further described below through specific embodiments and accompanying drawings.

The real-time big data-driven network security situation monitoring and visualization method of the present invention has an overall technical architecture as shown in Figure 1, which generally adopts a hadoop+storm distributed architecture to achieve powerful underlying analysis capabilities. The basic data sources of the present invention mainly include: real-time data of security equipment, quarterly website scanning evaluation report (HTML), and quarterly mainframe scanning evaluation report (HTML). Among them, the real-time data of the security equipment can use the IDS (Intrusion Detection Systems) data of the network center.

As shown in Figure 1, the above three types of data are collected from the data sending and receiving server at the hardware level. Among them, real-time data (such as SYSLOG, that is, system logs) will be sent to Flume, and at the same time, data backup will be performed on HDFS. After Flume collects the data according to the customized scheme, it sends it to Kafka for further processing. Flume is a massive log aggregation system that supports the customization of various data senders in the system to collect data. At the same time, it provides the ability to simply process data and write to various data receivers (customizable). Kafka is a distributed messaging system that can handle all action stream data in a consumer-scale website, by processing logs and log aggregation according to throughput requirements. The data stream processed by Kafka will be sent to the real-time computing system Storm one by one. In Storm, all real-time business logic is completed, such as IP address-organization information matching, geographic positioning, security event type classification statistics, high-risk research institute information extraction, etc. . Finally, these processing results are pushed into the Redis storage system in a stack-like form, and at the same time, the web front end extracts the results from Redis for display.

The quarterly website scan assessment report (HTML) and quarterly host scan assessment report (HTML), that is, Net.log and Server.log in Figure 1, will be sent to the preprocessing integration module for simple format processing, and then sent to The distributed computing system Hadoop performs big data analysis and processing. The processing process includes clustering, association, statistics, etc. In order to improve the storage efficiency of the analysis results, the statistical simple data is stored in the Mysql database, and the unstructured data (ie The non-relational data in Figure 1) will be stored in the Hbase database, and the web front-end does not need to perform logical processing, and can directly read the data display in the database.

The detailed design of the technical solution of the present invention is described below.

1. Multi-dimensional network security situation data extraction, integration and storage scheme

At present, the Chinese Academy of Sciences has deployed a large number of probes in various institutes to extract the security logs and network traffic of interconnected network devices, and store them in a unified format in traditional databases or storage devices. This mandatory formatting will lose some key information, and, in order to cater to the traditional relational storage model, it will inevitably bring a great degree of limitations to the data query and analysis mechanism.

Therefore, the present invention aims at retaining the most complete original data by analyzing the characteristics of various types of data, establishes a multi-dimensional model to complete the mapping of dimensions and metrics, and obtains all-round data that can reflect the network security situation. The storage of these data uses the multi-level storage of HDFS+Mysql+Hbase.

After the real-time data and traffic are connected, they are simply processed and classified, and distributed to the message queues of each receiver, waiting for the next step of storage and processing. Each message queue is managed with a topic (session) as a symbol, and the messages published to each topic will be evenly distributed to multiple partitions (regions). When a subscription message is received, the data stream will be published to the real-time computing system Storm In order to ensure the reliability of the system, HDFS storage is performed at the same time.

After the static data/historical data is connected, keywords are extracted and preprocessed, and classified and stored according to the content/data format to form a complete and intelligent basic source database that can reflect the network security situation.

2. Real-time data-driven network security situation decision-making scheme

Real-time data from the underlying layer often has multiple types, such as network traffic, device logs, security reports, etc. The analysis and processing of these data often have a delay of seconds or even minutes, so that the security situation analysis results cannot be displayed in real time, thus Reduced effectiveness of security monitoring. Therefore, in response to real-time display requirements, a solution for historical and real-time linkage analysis is proposed, and the use of linkage data analysis instead of static data processing is proposed, and the experience of historical data is used to assist the security decision-making of current real-time data.

Based on the above scheme, the technical route of combining the real-time computing system Storm and the distributed computing system Hadoop is adopted. Among them, the distributed computing system Hadoop is mainly used for the analysis of historical data (that is, the quarterly website scan evaluation report and the quarterly host scan evaluation report mentioned above), and the real-time computing system Strom is used for processing and pushing real-time data .

The distributed computing system Hadoop integrates multi-dimensional data collected from the bottom layer, performs data preprocessing based on regularly input historical data (that is, the quarterly website scan evaluation report and quarterly host scan evaluation report mentioned above), and uses clustering and Big data processing methods such as fusion, correlation analysis, entropy analysis, and situation prediction can find data correlations from a large number of dynamic and fuzzy information security data, learn network abnormal characteristics, and form a network security feature knowledge base. For example, key security feature items can be extracted from the data, including IP source address, IP destination address, event name, event category, security level, vulnerability code, etc.; after that, a database table structure is established to store these security feature items in multiple An initial security signature database is formed in each table entry; the initial security signature database will be updated regularly. With continuous data access, Hadoop continuously updates iterative analysis results and discovers new threats or makes predictions.

The real-time computing system Strom receives real-time data, extracts key information items according to front-end business requirements, and performs necessary merging and discarding. At the same time, fuzzy or precise matching is performed with the network security feature knowledge base. If the match is successful, the network security situation evaluation result is pushed to the front end in the form of WebSocket for display.

Specifically, after the real-time computing system Strom has processed a piece of real-time data, it extracts relevant security feature items and performs feature matching with the above-mentioned initial security feature library. The matching process is as follows:

1) According to the IP target address, if the IP target address matches the critical host IP in the network security feature knowledge base, then define the security situation of the host as a crisis, submit the result to the upper layer (front-end visualization layer), and report the criticality of the host The number of times plus 1;

2) According to the security level of the piece of data, if it is low, it will be directly filtered; if it is medium, it will match the category of the security event according to the security event rule number; if it is a distributed denial of service attack (DDOS), then Define the security status of the host as a crisis, and submit the result to the upper layer; if it is a detection scan attack, add 1 to the detection scan counter, until the threshold is reached, the corresponding security event type is regarded as high risk, and the result is submitted to the upper layer;

3) The network security feature knowledge base maintains a security event-vulnerability correspondence table. Under normal circumstances, hosts with relevant vulnerabilities encounter attacks corresponding to security events, and there is a high probability that serious consequences will occur. Therefore, when a security event is received , check in the security event-vulnerability correspondence table whether the destination host of the security event has relevant vulnerabilities, if so, define the host as a crisis state, and make a special mark in the critical host IP table.

3. Mass data visualization solution

This solution mainly solves how to display massive high-dimensional data in the form of graphics and images. By realizing image communication between people and data, people can observe hidden patterns in network security data, and can quickly discover patterns and discover potential problems. threat. The massive data visualization scheme realized by the present invention is mainly divided into the following contents:

1) Network security visualization based on real-time network traffic

Because security events such as port scans, worm attacks, and denial of service attacks have obvious one-to-one, one-to-many, or many-to-one characteristics in terms of traffic, such attack events often have obvious abnormalities in traffic, showing that network Traffic can help network security analysts quickly discover network attacks and better prevent and defend against network intrusions. Therefore, the visual display method of point-to-point attack lines is adopted to display information including original IP address, destination IP address, source port, destination port, protocol, time, attack type, etc. At the same time, other methods are assisted while using this technology, such as using color mapping to represent different types of attack events.

2) Network security visualization based on historical reports

In addition to real-time traffic, massive historical reports also need to be visualized in different dimensions according to different security requirements. Including: nationwide distribution of host security situation/website security situation; under different security domains, host security situation distribution/website security situation distribution; distribution relationship between IP address and security risk value; distribution relationship between website and security risk value, etc. .

Specifically, the security data visualization platform of the present invention can realize various visualization contents such as global dynamic attack map, domestic dynamic attack map, domestic security situation map, national security vulnerability distribution map, bulletin board and other functions, as shown in FIG. 8 .

a) Function page 1: domestic dynamic attack map

Real-time attacks across the country show the mutual attacks among domestic institutions, the Chinese Academy of Sciences, and research institutes of the Chinese Academy of Sciences. The attack source 1, attack path 2, and attack target 3 are displayed on the map of China, as shown in Figure 9. At the same time, the four corners of the page (not shown in the figure) respectively display the list of attack sources (Top 10, calculated by the number of attacks launched), the list of attack targets (Top 10, calculated by the number of attacks launched), real-time attack information (including time , attack source, attack source IP, attack target, target IP, attack type, attack port, etc.), attack type leaderboard (Top10, calculated by attack type).

b) Function page 2: Global dynamic attack map

The real-time attack on a global scale shows the attack on the 13th branch of the Chinese Academy of Sciences and its cooperative units by various countries around the world. The attack source, attack path and attack target are displayed on the world map. The page layout is the same as above.

c) Function page 3: domestic security situation map

The page layout is divided into three parts: left, middle and right, with three icons on each side and a map of China in the middle. On the upper left side are the eight research institutes with the highest risk value of the Chinese Academy of Sciences, in the middle is the website vulnerability situation, and the four websites with the highest risk value are displayed in a circle at the bottom; on the upper right side, the security situation of each branch is displayed in a circle, in the middle is the host vulnerability situation, and in the bottom circle The four hosts with the highest risk values are displayed; 12 sub-hospital marks of different colors are displayed on the map of China according to the risk values.

d) Function page 4: National security vulnerability distribution map

The page is divided into left and right parts. The upper left side shows the scanned security vulnerabilities. The numbers are randomly added to the corresponding number of vulnerabilities. The middle cycle shows the ranking of the number of vulnerabilities in the country (Top8), the ranking of Sql injection vulnerabilities (Top8), and cross-site scripting injection. Vulnerability ranking (Top8) and other charts, the lower part shows the security vulnerabilities of more than 100 research institutes of the Chinese Academy of Sciences;

e) Function page 5: bulletin board and other functions

The page is divided into three parts: top, bottom left, and bottom right. There are five function icons on the top. Copyright information is shown below.

4. Design and implementation of real-time data network security situation analysis (Storm part)

In the present invention, the distributed computing system Hadoop can be realized by using the existing technology, so it will not be described in detail. The fourth part mainly explains the concrete realization of the real-time computing system Storm.

In Storm, it is first necessary to design a graph structure for real-time computing, which is called topology. This topology will be submitted to the cluster, and the master node (Masternode) in the cluster will distribute the code and assign tasks to the worker nodes (Workernode) for execution. A topology includes two roles of Spout (data container) and Bolt (processing unit). The Spouts and Bolts in the topology are connected through Streamgroupings (stream grouping strategy). Spout sends messages and is responsible for data flow as Tuple tuples. The Bolt is responsible for converting these data streams. Operations such as calculation and filtering can be completed in the Bolt, and the Bolt itself can randomly send data to other Bolts. The Tuple emitted by the Spout is an immutable array corresponding to a fixed key-value pair.

The present invention develops and designs IPTopology (IP analysis topology), which is formed by connecting modules such as KafkaSpout, ReadBolt, IPBolt, RollCountBolt, FieldRankBolt and GlobalRankBolt through ShuffleGrouping (random grouping), FieldsGrouping (grouping by fields) and GlobalGrouping (global grouping) topology map. IPTopology mainly completes the configuration of KafkaSpout, adds Spouts and Bolts to the topology diagram, and completes the configuration of the operating mode and other tasks. The present invention uses KafkaSpout that comes with Storm as a data source. Kafka is a distributed message processing mechanism.

4.1 Overall processing logic

The overall processing logic of the real-time data network security situation analysis implemented by Storm is shown in Figure 2, including the following steps. The specific implementation of these modules, KafkaSpout, ReadBolt, IPBolt, RollCountBolt, FieldRankBolt, and GlobalRankBolt, will be described later:

1. KafkaSpout continuously reads attack records from external data sources and transmits attack records to ReadBolt.

2. ReadBolt analyzes the attack source and attack purpose in the attack record, and then transmits the attack source, attack target and original attack record to IPBolt.

3. IPBolt is responsible for detailed analysis of attack records, and encapsulates the attack information into the SendData data structure, and then writes it into Redis; at the same time, extracts information such as attack source, attack target, and attack type and sends it to RollCountBolt.

4. RollCountBolt mainly completes the classification and counting of attack sources, attack targets, and attack types, and stores the statistical results in different Map structures (MAP in C++ language), and then transmits all statistical results to FieldRankBolt.

5. Each FieldRankBolt will sort all the statistical results received from different attack sources, different attack targets, and different attack types, and filter their respective topN lists, and transmit these sorted lists to GlobalRankBolt.

6. GlobalRankBolt summarizes the sorted list of different attack sources, different attack targets, and different attack types sent by each FieldRankBolt, re-sorts, obtains the final topN list of different attack sources, different attack targets, and different attack types, and then updates the corresponding in Redis storage space.

4.2 IPTopology module design

The present invention uses KafkaSpout that comes with Storm as a data source. Kafka is a distributed message processing mechanism. A KafkaSpout can only process the content of one topic, so the initialization includes the following information:

1. Broker address (IP+Port) in the Kafka cluster;

2. topic name;

3. The unique identifier Id of the current Spout (hereinafter referred to as $spout_id);

4. The distributed application coordination service (zookeeper) is used to store the current processing offset Offset (hereinafter referred to as $zk_root);

5. The data decoding method in the current topic.

After initialization, spouts and bolts need to be added to the topology. When adding a Spout, you need to use the previously set configuration information kafkaConfig to initialize the Spout. When adding each Bolt, you need to use Streamgroupings (stream grouping strategy) to define what kind of stream each Bolt receives as input. Streamgrouping defines how a Stream should distribute data to each task of each Bolt. In this embodiment, the following three types of grouping are used respectively: random grouping; grouping by field; global grouping. After adding spouts and bolts, perform topology runtime configuration. The configuration content includes runtime status, debugging environment, number of cluster worker processes, etc.

4.3 KafkaSpout module design

Spout is a message producer in a topology. Generally, the data source of Spout comes from an external source, that is, Spout reads data from the external data source and sends a message to the topology: Tuple.

In this embodiment, the KafkaSpout module in the sample that comes with Storm is used as a data source to receive attack records from the outside, and then transmit data to the Bolt.

4.4 ReadBolt module design

The ReadBolt module mainly extracts the attack source IP address and attack target IP address from the message transmitted by KafkaSpout, and transmits the processing result and original record to IPBolt. The key methods to implement the ReadBolt module are:

1. Call the interface to extract the attack source IP address and attack target IP address in the attack record. Send the attack source IP and attack target IP and the original attack record to IPBolt.

2. Define the domain names and meanings included in the content launched by ReadBolt. The launch content of ReadBolt includes 3 domains: "src", "dst" and "line", which correspond to the attack source IP, attack target IP and original attack record respectively.

The processing flow of the ReadBolt module is shown in Figure 3, including: 1) call the interface, extract the attack source IP address and attack target IP address in the attack record; 2) transmit the attack source IP, attack target IP and original attack record to IPBolt .

4.5IPBolt module design

The IPBolt module mainly completes the following parts of work:

1. Analyze the attack record string emitted by ReadBolt, and extract the fields required to encapsulate the SendData data structure. The SendData data structure is used to encapsulate the various fields related to the attack records displayed in the attack graph (such as "global dynamic attack graph", "domestic dynamic attack graph") pages, these fields include: time, attack event name, source IP address , source organization name, destination IP address, destination organization name, security event type.

2. Extract the attack source IP and attack target IP respectively, and access IpService (IP service address library) to obtain the unit and location information of the attack source IP and attack target IP.

3. Write the encapsulated SendData data structure into the SendData.INSTANT_LOG_DATA storage space and SendData.INSTANT_OUR_DATA storage space of Redis to provide basic data for the "global dynamic attack graph" and "domestic dynamic attack graph" respectively.

4. Provide initialization data for the page loading of the "Global Dynamic Attack Graph" and "Domestic Dynamic Attack Graph", and count the topN of global and domestic attack sources, attack targets, and attack types from the early morning to the present day (N=10 in this project) ranking. Therefore, in IPBolt, it is necessary to extract information such as the name of the attack source unit, the name of the attack target unit, and the name of the attack type, and send it to RollCountBolt to provide data for subsequent statistical rankings.

The processing flowchart of the IPBolt module is shown in Figure 4, including the following steps:

1. First parse the attack type number field in the attack record string to obtain the attack type number typeId.

2. According to the typeId, determine whether it belongs to the ignored type of attack. If it is an ignored attack, directly discard this attack record; otherwise, go to step 3.

3. Search _portNameMap to obtain the attack type name typeName corresponding to typeId. If not found, assign the attack type name typeName to "unknown type".

4. Parse the other fields in the attack record string required to construct the SendData data structure.

5. Send the attack source IP and attack target IP obtained from the attack record to IpService respectively, and request the unit and location information of the IP.

6. Encapsulate the Ip data structure and construct srcIp and outIp according to the response information of IpService respectively.

7. Encapsulate srcIp, outIp, typeId, typeName and other fields extracted from the attack record into the SendData data structure to construct data.

8. Write data to SendData.INSTANT_LOG_DATA of redis.

9. If the srcIp of data is from China, write the data to SendData.INSTANT_OUR_DATA of redis.

10. Extract srcIp.Country, srcIp.City, outIp.City, typeName, logData, ourData and send them to RollCountBolt.

4.6RollCountBolt module design

The main function of the RollCountBolt module is to complete the counting of attack sources, attack targets, and attack types in global dynamic attacks and domestic dynamic attacks, and transmit the statistical results to FieldRankBolt for local sorting.

The processing flow of the RollCountBolt module is shown in Figure 5. Each time RollCountBolt receives the attack source, attack target, and attack type (collectively referred to as obj) information in an attack record:

1. RollCountBolt first judges whether the current time is early morning, that is, whether it has entered a new day.

1.1. If it arrives in the early morning, clear the global dynamic attack and the domestic dynamic attack corresponding to the attack source, attack target, and attack type Map structure (collectively referred to as _objCounts), and then perform step 2;

1.2. If it does not arrive in the early morning, go to step 2 directly.

2. Determine whether this piece of information is a domestic dynamic attack.

2.1. If it belongs to domestic dynamic attack, obtain the number of each obj of this information from each _objCounts of global dynamic attack and domestic dynamic attack;

2.2. If it is not a domestic dynamic attack, only obtain the number of each obj related to this record from each _objCounts of the global dynamic attack.

3. Increment the count by 1, and update the corresponding _objCounts with the new value.

4. Send each obj and its corresponding statistics to FieldRankBolt.

4.7 FieldRankBolt module design

The function of the FieldRankBolt module is mainly to complete the sorting of some types of attack sources, attack targets, and attack type statistical information in global dynamic attacks and domestic dynamic attacks, and regularly transmit them to GlobalRankBolt for summary. In one embodiment, the transmission period may be 2s.

When receiving the attack sources, attack targets, attack types (collectively referred to as obj) and their corresponding statistics of global dynamic attacks and domestic dynamic attacks, the processing flow of the FieldRankBolt module is shown in Figure 6, including the following steps:

1. First judge whether it is necessary to update the ranking list of each obj in the domestic dynamic attack based on the content accepted this time.

1.1. If necessary, process the ranking list corresponding to each obj in the global dynamic attack and domestic dynamic attack, including: _logSrcRank, _logDstRank, _logTypeRank, _ourSrcRank, _ourDstRank, _ourTypeRank;

1.2. If not needed, process the ranking list corresponding to each obj in the detachment global dynamic attack, including: _logSrcRank, _logDstRank, _logTypeRank.

2. Determine whether the content of each obj received this time exists in the corresponding ranking list.

2.1. If it exists, directly use the new value to update the corresponding value in the corresponding list.

2.2. If it does not exist, add the received obj and its corresponding value to the item list.

3. Reorder each ranking list in descending order of statistics.

4. Delete sublists that exceed topN in each ranked list.

5. Emit each list to GlobalRankBolt for global ranking.

4.8GlobalRankBolt module design

The function of the GlobalRankBolt module is similar to that of FieldRankBolt. It mainly collects the information emitted by each FieldRankBolt to complete the statistical sorting of attack sources, attack targets, and attack types in all global dynamic attacks and domestic dynamic attacks, and regularly updates the corresponding ones in Redis. storage. In an embodiment, the update period of the GlobalRankBolt may be 2s.

The processing flow of GlobalRankBolt is the same as that of FieldRankBolt, as shown in Figure 7, but the storage space of Redis will be directly updated in the end.

The above embodiments are only used to illustrate the technical solution of the present invention and not to limit it. Those of ordinary skill in the art can modify or equivalently replace the technical solution of the present invention without departing from the spirit and scope of the present invention. The scope of protection should be determined by the claims.

Claims (10)

1. the monitoring of the network safety situation of large data-driven and a method for visualizing, is characterized in that, comprises the steps:

1) the network security basic data of extraction different dimensions, comprises real time data and historical data;

2) adopt real time computation system Storm and distributed computing system Hadoop that network security basic data is stored and locatedReason, wherein Hadoop is for the treatment of historical data, and Strom is for the treatment of real time data;

3) distributed computing system Hadoop utilizes large data processing method from historical data, to extract crucial security feature item,And building database list structure, form network security character knowledge base;

4) real time computation system Strom extracts relevant security feature item from real time data, by itself and described network security spyLevy knowledge base and carry out characteristic matching, and according to matching result decision network security postures;

5) network safety situation of real time computation system Strom being judged carries out dynamic and visual displaying.

2. the method for claim 1, is characterized in that step 1) the network security basic data of described different dimensions comprisesWebsite, main frame, longitude and latitude, IP address, leak, security incident; Described historical data comprises per season web site scan assessmentReport and per season main frame scanning assessment report.

3. the method for claim 1, is characterized in that step 2) for the processing procedure of real time data be: first will be realTime data will be sent to massive logs paradigmatic system Flume, in HDFS system, carry out data backup simultaneously; Flume willThe real time data of collecting is sent to distributed information system Kafka to do further processing; Through Kafka number after treatmentSend into one by one real time computation system Storm according to stream, in Strom, complete all real time business logics; Finally will process knotFruit is pressed into Redis storage system with the form of similar stack, and Web front end extracts result and shows from Redis simultaneously.

4. method as claimed in claim 3, is characterized in that step 2) for the processing procedure of historical data be: by historical numberCarry out after simple format analysis processing according to delivering to pretreatment integrate module, deliver to distributed computing system Hadoop and carry out large dataAnalyzing and processing, then deposits the simple data of statistics the database to Mysql, and non-structured data are stored to HbaseDatabase, Web front end is without carrying out logical process again, and directly the data in reading database are shown.

5. the method for claim 1, is characterized in that step 3) described large data processing method comprises the one in followingOr multiple: cluster and fusion, association analysis, entropy analysis, Tendency Prediction; Described security feature item comprises: IP source address,IP destination address, event title, event category, safe class, leak code name.

6. the method for claim 1, is characterized in that step 3) described network security character knowledge base regular update, withData access endlessly, Hadoop constantly updates iterative analysis result, and finds new threat or predict.

7. the method for claim 1, is characterized in that step 4) described in carry out characteristic matching method be:

A), according to IP destination address, if IP destination address is mated with the critical host ip in network security character knowledge base, determineThis Host Security situation of justice is crisis, is that the visual layer of front end is submitted result to upper strata, and the critical number of times that this main frame is occurredAdd 1;

B), according to the safe class of these data, if low, directly filter; If in, according to security incident rule numbersMate this security incident and belong to which kind of classification; If distributed denial of service attack, defines this Host Security state for dangerMachine, submits result to upper strata; If detection scanning is attacked, detection scanning counter adds 1, until reach threshold value just by phaseThe security incident type of answering is considered as high-risk, submits result to upper strata;

C) the corresponding table of security incident of described network security character knowledge base maintenance-leak, in the time receiving a security incident,Whether the destination host of searching this security incident in the corresponding table of security incident-leak there is relevant vulnerability, if exist, willThis host definition is crisis state, does special mark in critical host ip table simultaneously.

8. the method for claim 1, is characterized in that step 5) described visual presentation comprises:

A) network security of real-time traffic Network Based is visual: adopt the visual display mode of point-to-point attack lines, showFormer IP address, object IP address, source port, destination interface, agreement, time, attack type, simultaneously auxiliary its other partyMethod, comprises and adopts color map to represent dissimilar attack;

B) network security based on historical report is visual: the historical report of magnanimity is carried out to different dimensions according to different demands for securityVisual presentation, comprising: nationwide, the security postures of main frame distributes or web portal security situation distributes; Different safetyUnder territory, the security postures of main frame distributes or web portal security situation distributes; IP address and security risk value distribution relation; Website withSecurity risk value distribution relation.

9. method as claimed in claim 8, is characterized in that step 5) content of described visual presentation comprises: the whole world is dynamicallyAttack graph, domestic dynamic attacks figure, internal security situation map, national security breaches distribution map, announcement board and other functions.

10. the method for claim 1, is characterized in that, described real time computation system Storm comprises as lower module:KafakaSpout, ReadBolt, IPBolt, RollCountBolt, FieldRankBolt and GlobalRankBolt, thisThe disposed of in its entirety process of a little modules is as follows:

A) KafakaSpout constantly reads attack record from external data source, and attack record is transmitted to ReadBolt;

B) ReadBolt analyzes the attack source of attacking in record and attacks object, then by attack source, target of attack and originalAttack record and be transmitted to IPBolt;

C) IPBolt is responsible for carrying out labor to attacking record, and will attack Information encapsulation to SendData data structure, thenWrite Redis; The information such as attack source, target of attack, attack type of simultaneously extracting send to RollCountBolt;

D) RollCountBolt mainly completes the differential counting statistics to attack source, target of attack, attack type, and will add up knotFruit is stored in respectively in different Map structures, then all statisticses is transmitted to FieldRankBolt;

E) the different attack sources that each FieldRankBolt receives oneself, different target of attack, different attack types allStatistics sorts, and screening topN list separately, and these sorted lists are transmitted to GlobalRankBolt;

F) GlobalRankBolt gathers different attack sources, different target of attack and the difference that each FieldRankBolt sends and attacksThe sorted lists that hits type, re-starts sequence, obtains different attack sources, different target of attack and different attack typeWhole topN list, then upgrades memory space corresponding in Redis.