CN111290913A - Fault location visualization system and method based on operation and maintenance data prediction - Google Patents

Abstract

The invention discloses a fault location visualization system and method based on operation and maintenance data prediction. The invention collects the machine and application log information in the network cluster by using the existing log collection framework to generate operation and maintenance big data, and then processes the operation and maintenance big data by using an artificial intelligence method, thereby predicting the network fault in advance and displaying the network fault in a visual mode. The invention has the beneficial effects that: the fault can be identified efficiently and accurately, the network fault can be predicted in advance, the operation and maintenance personnel can be given sufficient time to carry out operation and maintenance work in time, and the work efficiency of the operation and maintenance personnel can be improved effectively.

Description

A fault location visualization system and method based on operation and maintenance data prediction

technical field

The invention relates to a fault location visualization system and method based on operation and maintenance data prediction, and relates to the technical field of computer network and intelligent operation and maintenance.

Background technique

With the maturity of network technology, the coverage of wireless networks has gradually increased, and at the same time, there are more and more smart devices around people, so the number of devices connected to the network has also increased dramatically. On this basis, the quality of the information service provided by the network is a key factor affecting the user experience. For example, if there is an access control authentication system in the network, if there is a failure, the user's work efficiency of the access control system may be greatly reduced. Based on the above two points, the current network puts forward higher requirements for the accuracy and timeliness of fault location, so network fault detection and location has become a key research issue.

Currently, the frameworks for locating and analyzing faults in the network include eSight, ELK, Splunk, and zabbix. The main functions of these frameworks for locating and analyzing faults are similar. They first deploy applications on the terminal to collect log information on the device, and then extract the key information of the device based on the log information, locate and analyze the faults on the device by setting thresholds and performing simple statistical analysis methods, and provide visual information. and information notification mechanism to assist operation and maintenance personnel in operation and maintenance work.

Although there are so many network fault analysis and localization frameworks, these schemes also have many shortcomings and deficiencies in the current environment. First of all, most schemes mainly rely on the threshold method and error log information extraction to detect and locate faults in detecting anomalies. This method is difficult to detect errors in time, and it is easy to have false negatives. In the growing network environment, the more the risk exists. come bigger. Moreover, the thresholds of the framework are generally specified by experts. With the frequent updates of system upgrades, the workload of re-understanding the system setting parameters is too large, and the accuracy is not high; the second problem is that the number of connected devices increases sharply, which makes our logs Information is that the amount of data is too large. Even if it is processed with the current framework, the amount of information covered is too large, and it is difficult to assist the operation and maintenance personnel to understand the current network status in time. Many faults cannot be dealt with in time, and operation and maintenance personnel fall into a passive operation and maintenance state that waits for user feedback before operation and maintenance.

SUMMARY OF THE INVENTION

In view of the deficiencies of the prior art, the purpose of the present invention is to propose a fault location visualization system and method based on operation and maintenance data prediction. The invention efficiently and accurately identifies faults based on artificial intelligence technology, and at the same time can predict network faults in advance, thereby giving operation and maintenance personnel sufficient time to perform operation and maintenance work in a timely manner, and can effectively assist in improving the work efficiency of the operation and maintenance personnel.

The technical solutions of the present invention are specifically introduced as follows.

A fault location visualization system based on operation and maintenance data prediction, which includes a data collection part, an algorithm prediction part and a visual display part; wherein:

Data collection part: collect the machine and application logs of each machine in the cluster, extract the key performance indicators used to monitor the cluster status, and then use the method of time series extraction to construct operation and maintenance big data;

Algorithm prediction part: Use machine learning and neural network methods to learn the prior experience based on the historical operation and maintenance big data that has been collected, so as to generate an artificial intelligence prediction model that meets the requirements of actual production laws; then use the artificial intelligence model. Perform fault prediction on the real-time collected operation and maintenance data;

The visualization part is used to display all log information and fault prediction information.

In the present invention, the data collection part includes a log collection module and a performance index extraction module; wherein:

Log collection module: includes distributed log collection components and centralized log storage components, collects machine and application logs of each machine in the cluster, and completes the work of log redirection and centralized processing;

Performance index extraction module: extracts the index information reflecting machine performance in the log, and uses the method of time series extraction to construct operation and maintenance big data.

In the present invention, in the performance index extraction module, the index information reflecting machine performance is extracted by filtering and keyword extraction methods; the index information of machine performance includes CPU usage information, memory usage information, disk read bandwidth information and network traffic information.

In the present invention, the algorithm prediction part includes an algorithm adaptive module and a prediction fault module; wherein:

Algorithm adaptive module: Based on the historical operation and maintenance big data of the previous day, the model is trained by the statistical feature screening algorithm and the cross-validation screening algorithm, and the artificial intelligence model for predicting the failure of the day is obtained;

Prediction failure module: Use artificial intelligence models to predict failures from real-time collected operation and maintenance data.

In the present invention, the visualization part includes a real-time monitoring module, a historical prediction module, a log information retrieval module and a machine performance curve display module; wherein:

Real-time monitoring module: used to display the real-time performance status of machines in the cluster; performance status includes CPU utilization, memory utilization, disk throughput, network bandwidth, fault prediction information, and machine status obtained by collecting information through thresholds and SNMP methods;

Historical prediction module: used to cache historical failure prediction information of machines in the cluster;

Log information retrieval module: used to provide extracted machine performance log information;

Machine performance curve display module: used to display log information to operation and maintenance personnel in the form of a curve graph.

The present invention also provides a fault location visualization method based on the above system, the specific steps are as follows:

(1) Collect the machine and application logs of each machine in the cluster, uniformly extract the key performance indicators used to monitor the cluster status, and then use the method of time series extraction to construct operation and maintenance big data;

(2) Using machine learning and neural network methods to learn the prior experience based on the historical operation and maintenance big data that has been collected, so as to generate an artificial intelligence prediction model that meets the requirements of actual production laws; The collected operation and maintenance data is used for fault prediction;

(3) Display all log information and fault prediction information.

In the present invention, in step (1), the index information reflecting machine performance is extracted by filtering and keyword extraction methods; the index information of machine performance includes CPU usage information, memory usage information, disk read bandwidth information and network traffic information .

In the present invention, in step (2), the model is trained by statistical feature screening algorithm and cross-validation screening algorithm based on the historical operation and maintenance big data of the previous day to obtain an artificial intelligence model for predicting faults on the current day.

In the present invention, in step (3), it is displayed in the form of a graph and a table.

Compared with the prior art, the beneficial effects of the present invention are:

The present invention can realize accurate prediction and localization of network faults through the method of machine learning and neural network, based on the historical operation and maintenance big data that has been collected to learn the training model obtained by learning the prior experience therein;

The present invention displays conventional log monitoring information and predicted fault information of each machine through a visualized operation and maintenance information platform, so that effective operation and maintenance personnel can intuitively and efficiently complete operation and maintenance tasks.

Description of drawings

Figure 1 is a system architecture diagram.

Figure 2 is a flow chart of system operation.

Figure 3 is a block diagram of the data collection section.

Figure 4 is a structural diagram of the algorithm prediction part.

Figure 5 shows part of the structure diagram visually.

Detailed ways

The technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

The invention first uses a distributed log collection framework to collect machine log information on each device, then extracts key information in the machine log information as the current performance index of the machine, and builds operation and maintenance big data. Afterwards, the processed operation and maintenance big data is used to train the artificial intelligence model. When the latest real-time performance data is input, the model can predict whether these machines will fail in the next period of time, and show the graph as shown in the figure. It can be visualized to the operation and maintenance personnel in the form of a table and a table to help the operation and maintenance personnel identify problems and prepare solutions in advance, thus solving the problem that the complex network operation and maintenance personnel cannot operate and maintain in a timely manner.

1. Overall system architecture

The system architecture is shown in Figure 1. The system is mainly composed of three parts, namely the data collection part (log collection server), the algorithm prediction part (fault prediction server) and the visual display part (visual front end).

The data collection part mainly collects the machine and application logs of each machine in the cluster. The main principle is to deploy lightweight log collection tools to the monitored machines, and use these distributed tools to collect logs on the server. , and uniformly extract the key performance indicators to monitor the cluster status and build operation and maintenance big data.

The algorithm prediction part mainly uses machine learning and neural network methods to learn the prior experience based on the historical operation and maintenance big data that has been collected, so as to generate a prediction model that meets the requirements of actual production laws. In the actual operation prediction process, the real-time performance data is directly provided to the model as input, and the learned rules are used to predict whether there will be a fault in a certain period of time.

The visualization part displays all log information and prediction information. Considering that no matter how detailed the text report is, it is not as popular as the chart. The present invention provides the predicted fault and real-time status information to the operation and maintenance personnel as a reference in combination with the chart. This helps operation and maintenance personnel to efficiently understand and deal with possible faults in network clusters.

The operating principle of the entire system is shown in Figure 2. The first is the log collection lightweight components that exist on each machine. These components are deployed in the cluster in a distributed manner, and all the required system log information and application logs on the machine are collected. The information is transmitted back to a specific storage server through the network, and a log collection and storage tool is deployed on the storage server to accept and save the log information of all machines in the cluster and perform a centralized storage and processing. Keyword extraction and other methods extract the key performance indicator information of each machine from the log, which constitutes the operation and maintenance big data of all our machines; after that, on another server, the neural network algorithm we constructed is based on the collected historical operation and maintenance. Use big data to train and learn models, generate artificial intelligence models that fit our situation, and process the collected real-time operation and maintenance data as the input of the model, so as to predict in real time that the machines in the cluster will be in the next period of time. Whether there will be a fault, complete the fault prediction and location based on operation and maintenance big data; finally, we use the front-end framework to build a visual operation and maintenance data display platform, monitor our cluster status in real time, and carry out equipment with predicted fault information. Early warning, caching predicted failure information for later processing. In this way, operation and maintenance personnel can efficiently understand the cluster status through icons, and provide detailed log information to accurately find faults and design solutions in advance.

2. Data collection part

As shown in Figure 3, the data collection part is mainly composed of a log collection module and a performance index extraction module. This part mainly completes the tasks of collecting monitoring information and extracting key performance indicators in the cluster.

Log collection module: used to complete the task of log redirection and storage. The present invention completes the task of log collection by referring to the existing mature cluster log monitoring and management framework applied to the bottom operation and maintenance work of many large companies. We mainly use the log collection tools and log storage tools in the mainstream framework ELK to complete our log redirection centralized processing, so as to centrally store and process log information in the cluster for unified management.

Performance index extraction module: used to extract the index reflecting machine performance in the log. Our scheme mainly collects the CPU utilization, memory utilization, disk read speed, and network traffic bandwidth of each machine to measure the status of a machine. The key performance indicators we need can be extracted from the system log information of the machine by using keyword extraction and filtering technology, so as to reflect the status of each machine in the cluster, and use the method of time series extraction to construct the operation. dimensional big data.

3. Algorithm prediction part

As shown in Figure 4, the algorithm prediction part adaptively selects suitable artificial intelligence algorithms according to the different types and statistical characteristics of the collected operation and maintenance big data, so as to complete the prediction of different types of data. This part consists of two modules: algorithm adaptive module and prediction fault module.

Algorithm adaptation module: Screen appropriate algorithms to learn from prior experience in O&M big data. Due to the continuous iteration and redesign of the cluster system, the statistical characteristics and distribution of O&M data may also change dramatically. If you insist on using the algorithm model of the previous version for prediction at this time, the accuracy of the prediction will drop significantly, resulting in a large number of false positives. In order to prevent this situation, the present invention adopts statistical feature screening algorithm and cross-validation screening algorithm to train the model; specifically, the model is trained using the historical operation and maintenance data of the previous day every day, and the specific process is to collect the historical operation and maintenance data of the previous day. The data is used as training data, and some statistical indicators are calculated to select the appropriate algorithm. Then, the training data is processed into time series, and divided into training set and test set, the algorithm to be selected is configured with appropriate parameters to learn on the training set, and then the test set is used for cross-validation, and the best training effect is selected. The model is persisted as an artificial intelligence model that predicts failures on the day.

Prediction failure module: Use a persistent model to predict failures from real-time collected operation and maintenance data. The previous module has generated the model used for prediction, and then processes the real-time machine performance data information into operation and maintenance data as input to the model, and the model can output whether the machine may fail in the next period of time. What kind of fault occurs, so as to complete the task of network fault location and prediction.

Fourth, the visual display part

As shown in Figure 5, the visual display part mainly consists of a real-time monitoring module, a historical prediction module, a log information retrieval module and a machine performance curve display module.

Real-time monitoring module: This module is used to display the real-time performance status of the machines in the cluster, mainly showing the real-time CPU utilization, memory utilization, disk throughput, network bandwidth, and whether each machine is in the next period of time. There will be prediction failures, and information will be collected through methods such as thresholds and SNMP to determine whether the machine is normal.

Historical prediction module: This module is mainly used to cache the historical prediction information of each of our machines. After we interviewed professional operation and maintenance personnel, the suggestion from the operation and maintenance personnel is to save the historical record of each prediction. Because in the event of a failure, these prediction information contains important information for the operation and maintenance personnel, which can help to understand the failure in time.

Log information retrieval module: This part mainly provides the machine performance log information we extracted. This information is the main operation and maintenance basis of the current operation and maintenance personnel. The information provided here is mainly for the convenience of the operation and maintenance personnel to perform operation and maintenance in a familiar way. In this way, when the operation and maintenance personnel receive the predicted fault information, they can quickly review the current log information of the machine, so as to more accurately use the log information and their own operation and maintenance knowledge to confirm the exact cause of the fault and prepare for corresponding solutions. Program.

Machine performance curve display module: This module is used to display the log information to the operation and maintenance personnel in the form of a curve graph. In this way, operation and maintenance personnel can intuitively find out whether the performance curve is abnormal and quickly determine the fault. At the same time, we also display the real-time prediction results on the performance curve, providing important auxiliary information for operation and maintenance personnel.

Based on the above system, the present invention proposes a fault location visualization method based on operation and maintenance big data prediction. The specific implementation steps are as follows:

First, users need to install a lightweight log collection tool on machines in each network cluster and keep the component running in real time. By modifying the configuration file, the log collection tool will collect and redirect the system log information on the machine and the specified application performance log information, so that the relevant information of all machines in the cluster can be collected to monitor the performance of the machines in the cluster. . Compared with other tools, this log collection tool is simpler and more convenient, and its overhead is almost negligible. Therefore, using this tool in a cluster will not increase excessive overhead and ensure the performance of the original network. At the same time, all log information is relocated to a server configured with a log storage tool in the cluster for storage and convenient analysis in the next step.

Then, the relevant logs of all machines in the cluster are redirected to the log storage server by the log collection tool deployed in a distributed manner. The log storage server can also use the message queue tool to complete the message caching function, so as to ensure that all log information can be stored. Centrally stored on the server. After that, in order to monitor the performance of the machines in the cluster, you can use the filtering method or the keyword extraction method to extract performance responses such as CPU utilization, memory utilization, disk throughput, network bandwidth traffic, etc. from the system log. An indicator of machine performance. With these metrics, we can monitor the status of the machines in the cluster in real time, and also provide a basis for future prediction work.

In the third step, after collecting the performance index data of all machines, we need to process it as time series data. Our initial settings are 100 seconds as the window size and 1 second as the interval between two time series to construct the time Sequence, the label of each event sequence is whether there is a fault within 1000 seconds after the time window, if there is, the label is 1, if there is no, the label is 0. For example, we now assume that s _n is used to represent the information of the nth second of the machine. We now have the data information of s ₁ , s ₂ , s ₃ ...... s ₁₀₁ , and there is a fault within 1000 seconds, then we construct two times windows, which are the time series containing s ₁ , s ₂ , s ₃ ...... s ₁₀₀ data and the time series containing s ₂ , s ₃ , s ₄ ...... s ₁₀₁ data, the labels of both time series are 1 . In this way, after all data is constructed as time series data storage, the statistical information of all time series windows is counted as new features, thus generating operation and maintenance big data.

In the fourth step, we need to train an algorithmic model to complete the task of failure prediction. At the same time, in order to ensure timeliness and accuracy, we will divide all the operation and maintenance data of the previous day into a training set and a test set to train a suitable model for building a model for predicting the network failure of the next day. Here, the training set and the test set The division ratio of sets is 4:1 by default. After that, in order to complete the algorithm screening of posture factors, each algorithm model with preset parameters is used for training and cross-validation using the obtained training set and test set, and the model with the best cross-validation score is used as the prediction model used in actual production. And persist it for real-time fault prediction and location in the next step.

The fifth step, in the actual production environment, in order to achieve the goal of real-time fault prediction, we process the real-time data of a certain machine and all the data in the first 99 seconds into a time series. Then call our persistently saved prediction model, take the processed data as input, and the output of the algorithm is whether there will be a failure on a machine in the cluster within 1000 seconds after the prediction, thus completing the real-time failure Prediction and positioning work, thus helping operation and maintenance personnel to identify potential failures that may exist in advance.

In the sixth step, when the task of fault prediction is completed and the prediction result is generated, we also need to combine the prediction result with the actual performance information to complete the visual display work to help the operation and maintenance personnel to understand the situation of the machines in the cluster intuitively and simply. First, display the real-time performance status indicators (CPU utilization, memory utilization, etc.) and real-time predicted fault results of each machine in the cluster. These indicators mainly complete the task of real-time monitoring of the machine. If the operation and maintenance personnel find a certain machine With predicted faults, you can click the machine name to view detailed machine information, including all performance log information and performance curve graph information of the machine. The performance curve graph information can help operation and maintenance personnel to view the performance curve intuitively. Find problems and mark the predicted failure information for reference; after confirming the time point at which a specific performance failure may occur, we can review the specific performance log information to understand the failure in advance and provide corresponding solutions. Finally, in order to allow the operation and maintenance personnel to understand the fault more comprehensively, we also keep the historical record of the predicted fault for the operation and maintenance personnel to analyze.

Claims (9)

1. The fault location visualization system based on operation and maintenance data prediction is characterized by comprising a data collection part,

An algorithm prediction part and a visualization display part; wherein:

a data collection part: collecting machine and application logs of each machine in the cluster, uniformly extracting key performance indexes used for monitoring cluster states, and constructing operation and maintenance big data by using a time sequence extraction method;

and an algorithm prediction part: learning prior experience based on collected historical operation and maintenance big data by using a machine learning and neural network method, so as to generate an artificial intelligence prediction model meeting the requirements of an actual production rule; then, carrying out fault prediction on the operation and maintenance data acquired in real time by using an artificial intelligence model;

and a visualization part: and the method is used for displaying all log information and failure prediction information.

2. The fault localization visualization system according to claim 1, wherein the data collection part comprises a log collection module and a performance index extraction module; wherein:

a log collection module: the system comprises a distributed log collection component and a centralized log storage component, wherein the distributed log collection component and the centralized log storage component are used for collecting machine and application logs of all machines in a cluster and finishing the work of log redirection centralized processing;

a performance index extraction module: index information reflecting the performance of the machine in the log is extracted, and operation and maintenance big data are constructed by a time sequence extraction method.

3. The fault location visualization system according to claim 1, wherein in the performance extraction module, index information reflecting machine performance is extracted by filtering and keyword extraction methods; the index information of the machine performance comprises CPU utilization rate information, memory utilization rate information, disk reading bandwidth information and network flow information.

4. The fault localization visualization system of claim 1 wherein the algorithmic prediction portion comprises an algorithmic adaptation module and a predictive fault module; wherein:

an algorithm self-adaptive module: training a model through a statistical feature screening algorithm and a cross validation screening algorithm based on historical operation and maintenance big data of the previous day to obtain an artificial intelligence model for predicting faults on the same day;

a predictive failure module: and carrying out fault prediction on the operation and maintenance data acquired in real time by using an artificial intelligence model.

5. The fault location visualization system according to claim 1, wherein the visualization part comprises a real-time monitoring module, a history prediction module, a log information retrieval module and a machine performance curve display module; wherein:

a real-time monitoring module: the system is used for displaying the real-time performance state of the machines in the cluster; the performance state comprises CPU utilization rate, memory utilization rate, disk throughput rate, network bandwidth, failure prediction information and machine state obtained by collecting information through a threshold value and an SNMP method;

a history prediction module: the system comprises a cache module, a fault prediction module and a fault prediction module, wherein the cache module is used for caching historical fault prediction information of machines in a cluster;

a log information retrieval module: for providing extracted machine performance log information;

the machine performance curve display module: the system is used for displaying the log information to the operation and maintenance personnel in a graph mode.

6. A fault localization visualization method based on the system of claim 1, characterized by comprising the following specific steps:

(1) collecting machine and application logs of each machine in the cluster, uniformly extracting key performance indexes used for monitoring cluster states, and constructing operation and maintenance big data by using a time sequence extraction method;

(2) learning prior experience based on collected historical operation and maintenance big data by using a machine learning and neural network method, so as to generate an artificial intelligence prediction model meeting the requirements of an actual production rule; carrying out fault prediction on the operation and maintenance data acquired in real time by using an artificial intelligence model;

(3) and displaying all log information and fault prediction information.

7. The fault location visualization method according to claim 6, wherein in the step (1), index information reflecting the performance of the machine is extracted by filtering and keyword extraction methods; the index information of the machine performance comprises CPU utilization rate information, memory utilization rate information, disk reading bandwidth information and network flow information.

8. The fault location visualization method according to claim 6, wherein in the step (2), the model is trained through a statistical feature screening algorithm and a cross-validation screening algorithm based on historical operation and maintenance big data of the previous day, so as to obtain an artificial intelligence model of the predicted fault of the current day.

9. The fault localization visualization method according to claim 6, wherein in the step (3), the visualization is performed by graph and table.

Images