CN111858278A - Log analysis method, system and readable storage device based on big data processing - Google Patents

Abstract

The invention provides a log analysis method, a system and a readable storage device based on big data processing, wherein the log analysis method comprises the following steps: s1: the log collection service BLS collects logs from a log source; s2: performing data cold treatment and data heat treatment on the log; s3: performing big data distributed calculation, processing, analysis and reporting through the BMR; s4: the data processed by the BMR is written into a data warehouse or the data in the BOS is combined with the machine learning BML to carry out user behavior prediction analysis operation, and the main source of the user, contents on favorite websites, loyalty of the user and the like can be seen through a log analysis system. By analyzing the user behavior log, the invention can further optimize the layout and the function of the website so as to improve the user experience and the like. And dividing the popularization budget and emphasizing and optimizing the tendency points of the user group and the like through the analysis result.

Description

Log analysis method, system and readable storage device based on big data processing

【Technical field】

The present invention relates to the technical field of big data processing, in particular to a log analysis method, system and readable storage device based on big data processing.

【Background technique】

In the application of the Internet, logs are very important data, because Internet projects often require 7*24 uninterrupted operation, so it is necessary to obtain and analyze the log data related to the operation of the monitoring system. The log analysis system is an analysis-oriented integrated data environment that provides a strategic collection of system data support for the enterprise decision-making process. Through the analysis of the data in the data warehouse, it can help enterprises to improve business processes, control costs, and improve product quality. In the era of big data, log data is a clear record of the operation of each company, organization or institution, and it is also one of the most direct, accessible and wide-ranging data sources for users to use traces of the company's products. The collection and analysis of logs are often processed using big data technology.

Therefore, it is necessary to study a log analysis method, system and readable storage device based on big data processing to deal with the deficiencies of the prior art, so as to solve or alleviate one or more of the above problems.

[Content of the invention]

In view of this, the present invention provides a log analysis method, system and readable storage device based on big data processing. Through log analysis, it is possible to see the main sources of users, what content on the website they like, and the loyalty of the user. The layout and functions are further optimized to improve the user experience.

In one aspect, the present invention provides a log analysis method, which includes the following steps:

S1: log collection service, BLS collects logs from log sources;

S2: Perform data cold processing and data thermal processing on logs respectively;

S3: Big data distributed computing is performed on the processed data through BMR, and analysis is performed;

S4: The data calculated by BMR is written into the data warehouse or the data in the BOS is combined with machine learning BML to perform user behavior prediction and analysis operations;

S5: Apply and display the analysis results, provide alerts to the operation and maintenance personnel for the results of data heat treatment; display the results of cold data processing through BI tools.

The above aspects and any possible implementation manners further provide an implementation manner. The data cold processing in S2 is specifically: writing logs into object storage BOS for storage or writing to HBase clusters, and then accessing Hive or Spark SQL cluster for analytical processing.

The above aspects and any possible implementations further provide an implementation. The data heat treatment in the S2 is specifically: inputting the log into the message service Kafka, as a message queue, and delivering it to the streaming computing BSC, and processing the log data. Perform real-time computing processing, and then write the processed data to Kafka.

According to the above aspect and any possible implementation manner, an implementation manner is further provided, where the BMR is a fully managed Hadoop/Spark cluster.

The above-mentioned aspects and any possible implementation manners further provide an implementation manner. The S1 is specifically: performing a managed log collection service through BLS, and a user needs to configure a source address, a destination address and a collection rule.

The above aspect and any possible implementation manner further provide an implementation manner, and the S3 specifically includes:

The S3 specifically includes:

S31: data cleaning, using a distributed computing framework to clean the data, and storing the cleaned data in the data warehouse or in the computing framework;

S32: Use the Spark, Hive, MapReduce or Flink framework to conduct business statistics on the cleaned data and analyze it according to the big data content.

According to the above aspect and any possible implementation manner, an implementation manner is further provided, wherein the distributed computing framework includes Spark, Hive and MapReduce.

According to the above-mentioned aspect and any possible implementation manner, an implementation manner is further provided. The BI tool display result in S5 includes a pie chart, a bar chart, a map and a line chart.

The above aspects and any possible implementation manner further provide a log analysis device based on big data processing, the device comprising:

A log collection module for collecting logs from log sources;

The processing and analysis module is used for data cleaning, cold processing and heat treatment of logs, and storage and predictive analysis of processing results;

The display module is applied, alerts are made through the stored processing results, and the predictive analysis results are displayed on BI tools.

The above aspect and any possible implementation manner further provide a computer-readable storage medium, where a processing program for log analysis is stored on the computer-readable storage medium, and the processing program for the log analysis method is processed by a processor The steps of implementing the log analysis method described in any one of the above are implemented when executed.

The above aspects and any possible implementation manners further provide an application of a log analysis method based on big data processing, where the log analysis method is used in an electronic bank, a communication operator or an e-commerce operation platform.

Compared with the prior art, the present invention can obtain the following technical effects:

The system of the invention supports offline and real-time processing of cold and hot data; supports the processing of multiple data sources, complex data structures and the construction of complex portraits. Through the log analysis system, you can see the main sources of users, what content they like on the website, and the loyalty of users. By analyzing user behavior logs, the layout and functions of the website can be further optimized to improve user experience. Through the analysis of the results, the promotion budget is divided, and the tendency points of the user groups are optimized.

Of course, any product implementing the present invention does not necessarily need to achieve all the above-mentioned technical effects at the same time.

【Description of drawings】

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a flowchart of a log analysis method provided by an embodiment of the present invention.

【Detailed ways】

In order to better understand the technical solutions of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

It should be understood that the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used in the embodiments of the present invention and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise.

The present invention provides a log analysis method based on big data processing. As shown in FIG. 1 , the log analysis method includes the following steps:

S1: Log collection service BLS collects logs from log sources;

S2: Perform data cold processing and data heat treatment on logs;

S3: Big data distributed computing, processing, analysis and reporting through BMR;

S4: The data processed by BMR is written into the data warehouse or the data in the BOS is combined with machine learning BML to perform user behavior prediction and analysis operations;

S5: Apply and display analysis results, provide alerts to operation and maintenance personnel about the results of data heat treatment; display cold data through BI tools.

The data cold processing in the S2 is specifically: writing the log to the object storage BOS for storage or writing to the HBase cluster, and then accessing the Hive or SparkSQL cluster for analysis and processing.

The data heat treatment in the S2 is specifically: connecting the log to the message service Kafka as a message queue, delivering it to the streaming computing BSC to perform real-time computing processing on the log data, and then writing the processed data to Kafka.

The BMR is a fully managed Hadoop/Spark cluster, and the S1 is specifically: a managed log collection service is performed through BLS, and the user needs to configure the source address, destination address and collection rules.

The S3 specifically includes:

S31: data cleaning, using a distributed computing framework to clean the data, and storing the cleaned data in the data warehouse or in the computing framework;

S32: Use the Spark, Hive, MapReduce or Flink framework to conduct business statistics on the cleaned data and analyze it according to the big data content.

The distributed computing framework includes Spark, Hive and MapReduce. The BI tool display results in the S5 include pie charts, bar charts, maps and line charts.

A log analysis device based on big data processing, the device includes

A log collection module for collecting logs from log sources;

The processing and analysis module is used for data cleaning, cold processing and heat treatment of logs, and storage and predictive analysis of processing results;

The display module is applied, alerts are made through the stored processing results, and the predictive analysis results are displayed on BI tools.

A computer-readable storage medium, on which a processing program for log analysis is stored, and when the processing program of the log analysis method is executed by a processor, the steps of any one of the log analysis methods are implemented .

The present invention aims at users who rely more and more on the communication service provided by the operator in their daily life, so that a large amount of user data is kept in the hands of the operator. Activate the potential value of this part of data based on big data. Operators expect to mine the hidden information value of a specific direction from the retained user log data, such as: user behavior prediction, marketing result monitoring, credit risk prevention and control, real estate site selection, population migration distribution, transportation planning, business public opinion and other information. Aiming at the above needs and pain points of operators, the present invention provides a log-based data mining analysis service.

In the case of ensuring user privacy, the operator log data is used, combined with the operator's DPI system, billing system and other data. Hundreds of first-level dimension classifications and nearly a thousand second-level dimension classifications such as user habits, activity paths, and consumption scenarios are constructed, which are used to construct user portraits, and use specific methods to calibrate different dimensions of user portraits at different time granularities. Gradually Plump user portraits. To ensure the accuracy of providing specific services to specific users.

The working principle of the present invention is as follows:

The log analysis system for big data processing is divided into five modules.

Data collection: Use Flume to collect data and write web logs to HDFS of BLS.

Data cleaning: Using Spark, Hive, MapReduce or some other distributed computing frameworks, the cleaned data is stored in the data warehouse or in Hive and SparkSQL.

Data processing: Statistics and analysis of the corresponding business are performed as needed (using Spark, Hive, MapReduce, Flink and other frameworks).

Storage of data processing results: The results are stored in databases such as RDBMS and NoSQL.

Data visualization: It is displayed through graphical display: pie charts, bar charts, maps, line charts.

The system of the invention supports offline and real-time processing of cold and hot data; supports the processing of multiple data sources, complex data structures and the construction of complex portraits. Through the log analysis system, you can see the main sources of users, what content they like on the website, and the loyalty of users. By analyzing the user behavior log, the present invention can further optimize the layout and function of the website, so as to improve the user's experience and the like. Through the analysis of the results, the promotion budget is divided, and the tendency points of the user groups are optimized.

The log analysis system and method based on big data processing of the present invention simultaneously realizes hot data processing and cold data processing, including three modules: log collection, processing and analysis, and application and display.

In the log collection module, the log collection service BLS collects logs from log sources (such as servers). BLS is a managed log collection service. Users only need to configure simple information such as source address, destination address, and collection rules to achieve highly reliable and highly available log collection.

The collected logs can be connected to the log processing module. On the one hand, for hot data processing scenarios, logs can be connected to the message service Kafka as a message queue, delivered to the streaming computing BSC to perform real-time computing and processing of log data, and then write the processed data to Kafka. On the other hand, for cold data processing scenarios, logs can be first written to the object storage BOS for storage, or directly written to the HBase cluster, and then connected to the Hive and SparkSQL clusters for analysis and processing. BMR is a fully managed Hadoop/Spark cluster that focuses on big data processing, analysis, and reporting with the help of big data distributed computing technology. The data processed by BMR can be written to the data warehouse. At the same time, analysis operations such as user behavior prediction can be performed directly from the data in the BOS combined with the machine learning BML.

In the application and display module, hot data can be processed to provide alerts to operation and maintenance personnel; cold data can be displayed through BI tools.

The log analysis method of the present invention can be used for banks, communication operators and e-commerce operation platforms.

The log analysis method, system, and readable storage device based on big data processing provided by the embodiments of the present application have been described in detail above. The description of the above embodiment is only used to help understand the method of the present application and its core idea; meanwhile, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific embodiment and the scope of application, In conclusion, the content of this specification should not be construed as a limitation on the present application.

As certain terms are used in the specification and claims to refer to particular components. It should be understood by those skilled in the art that hardware manufacturers may refer to the same component by different nouns. The present specification and claims do not use the difference in name as a way to distinguish components, but use the difference in function of the components as a criterion for distinguishing. As mentioned in the entire specification and claims, "comprising" and "including" are open-ended terms, so they should be interpreted as "including/including but not limited to". "Approximately" means that within an acceptable error range, those skilled in the art can solve the technical problem within a certain error range, and basically achieve the technical effect. Subsequent descriptions in the specification are preferred embodiments for implementing the present application. However, the descriptions are for the purpose of illustrating the general principles of the present application and are not intended to limit the scope of the present application. The scope of protection of this application should be determined by the appended claims.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a commodity or system comprising a list of elements includes not only those elements, but also includes not explicitly listed other elements, or elements inherent to the commodity or system. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the article or system that includes the element.

It should be understood that the term "and/or" used in this document is only an association relationship to describe the associated objects, indicating that there may be three kinds of relationships, for example, A and/or B, which may indicate that A exists alone, and A and B exist at the same time. B, there are three cases of B alone. In addition, the character "/" in this document generally indicates that the related objects are an "or" relationship.

The above description shows and describes several preferred embodiments of the present application, but as mentioned above, it should be understood that the present application is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various various other combinations, modifications and environments, and can be modified within the scope of the concept of the application described herein, using the above teachings or skill or knowledge in the relevant field. However, modifications and changes made by those skilled in the art do not depart from the spirit and scope of the present application, and should all fall within the protection scope of the appended claims of the present application.

Claims (10)

1. A log analysis method based on big data processing is characterized by comprising the following steps:

s1: a log collection service, the BLS collecting logs from a log source;

s2: respectively carrying out data cold treatment and data heat treatment on the log;

s3: performing big data distributed calculation on the processed data through the BMR, and analyzing;

s4: writing the data after BMR calculation into a data warehouse or performing user behavior prediction analysis operation by combining data in BOS and machine learning BML;

s5: applying and displaying the analysis result, and providing an alarm for operation and maintenance personnel according to the result of the data heat treatment; and displaying the result of cold data processing through a BI tool.

2. The log analysis method according to claim 1, wherein the data cold processing in S2 is specifically: and writing the log into an object storage BOS for storage or into an HBase cluster, and then accessing a Hive or Spark SQL cluster for analysis and processing.

3. The log analysis method according to claim 1, wherein the data heat treatment in S2 is specifically: and inputting the log into a message service Kafka as a message queue, delivering the log to a streaming computing BSC, performing real-time computing processing on the log data, and writing the processed data into the Kafka.

4. The log analysis method as claimed in claim 1, wherein the S1 is implemented by a hosted log collection service through BLS, and the user needs to configure the source address, the destination address and the collection rule.

5. The log analysis method according to claim 1, wherein the S3 specifically includes:

s31: data cleaning, namely cleaning data by using a distributed computing frame, and storing the cleaned data in a data warehouse or reserving the cleaned data in the computing frame;

s32: and carrying out service statistics on the cleaned data by using Spark, Hive, MapReduce or Flink frames, and analyzing according to the content of the big data.

6. The log analysis method of claim 1, wherein the distributed computing framework comprises Spark, Hive, and MapReduce.

7. The log analysis method as claimed in claim 1, wherein the BI tool presentation result in S5 includes a pie chart, a bar chart, a map and a line chart.

8. A log analysis device based on big data processing, comprising the log analysis method of any of the above claims 1 to 7, wherein the device comprises

The log collection module is used for collecting logs from a log source;

The processing and analyzing module is used for carrying out data cleaning, cold treatment and heat treatment on the log, and storing and predicting and analyzing the processing result;

and the application display module is used for alarming according to the stored processing result and displaying the prediction analysis result in a BI tool.

9. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a processing program of a log analysis method, which when executed by a processor implements the steps of the log analysis method according to any one of claims 1 to 7.

10. Application of a log analysis method based on big data processing, based on the steps of the log analysis method of any one of claims 1 to 7, wherein the log analysis method is used for electronic banking, communication operators or e-commerce operation platforms.

Images