CN103425750A - A cross-platform and cross-application log collection system and its collection management method - Google Patents

Abstract

The present invention relates to the field of information processing technology, and in particular to a cross-platform and cross-application log collection and management method. The present invention includes three main entities: a log collection client, a log server, and a unified management page. The log collection client is deployed on a physical machine where different business systems running on different operating systems are located, and regularly collects important log data and sends it to the log server; the log server and the unified management page are deployed on another physical machine that can be accessed by all log collection clients, providing a unified log management service. The present invention shields the differences in underlying heterogeneous systems, solves the problem that logs between different applications are disorganized and difficult to effectively manage and utilize, and can be used in log collection and management.

Description

A cross-platform and cross-application log collection system and its collection management method

technical field

The invention relates to the technical field of information processing, in particular to a cross-platform and cross-application log collection and management method.

Background technique

The logging module is a basic module that every application and the underlying operating system must have. Log files are generated during system operation, which can record the operating status of the system and various operation behaviors of users. When the system runs slowly or exits abnormally, by viewing the log files, you can quickly locate the problem, fix the defect, and make the system run normally.

At present, the function of logging is very mature both at the operating system level and in ordinary applications. It is made into a common independent module, such as syslog, log4j, etc. In addition to supporting logging, it also supports sending logs to remote servers. , or save directly to the database.

Roughly speaking, logging is divided into the following steps:

1. Call the API interface of the log module to record the log

In the place where logs need to be recorded in the application program, the API interface provided by the log module can be called to realize the log recording. When recording, you can also specify the log storage format, key fields, etc.

2. Persist the recorded log and store it in a designated place

After the log is recorded, it needs to be saved to a certain place, and the storage place is more flexible and changeable. Generally, it is directly saved to a file, or can be saved to a database, or even sent to a Socket server, so that the server's Business process to process this log information.

3. Log view

Logs stored in files can be opened and viewed directly through text programs, and logs stored in databases can be viewed through SQL statements.

Although the log module is very mature, there are still the following disadvantages:

First, to use a certain log technology, you must use the API interface provided by the technology, which is not a problem for newly developed applications; but for original applications, it is necessary to drastically modify the source code of the program, due to the distribution of log-related codes The scope is wide, and it is quite troublesome to modify.

The second is that the customizability is poor when collecting log data. Only a few ways are provided to filter the log data of interest, and even all the logs generated by the system are stored in files, databases or sent to remote servers. . The number of logs is huge, and the useful log data accounts for a small percentage of the total logs. If useful logs cannot be collected effectively, the profitability of the logs will be greatly reduced.

The third is that for log viewing, the existing analysis mechanism is not perfect, and it is often only a simple query to the source of the log, time and other simple information; the log data cannot be used to provide effective charts to fully display the current operating status of the system. More useful information such as operational behavior.

Contents of the invention

One of the technical problems solved by the present invention is to propose a cross-platform and cross-application log collection system aiming at the problems of difficult log collection and management and low efficiency of different applications deployed on different operating systems.

The second technical problem solved by the present invention is to propose a cross-platform and cross-application log collection and management method for the problems of difficulty and low efficiency in log collection and management of different applications deployed on different operating systems.

The technical scheme that the present invention solves one of above-mentioned technical problem is:

Including log collection client, log server and unified management page;

The log collection client collects logs scattered in different operating systems and different applications, and can selectively collect log data according to custom log formats, types or other keywords; it can be applied to Windows or Linux operating systems, mail server or normal web site;

The log server receives the log data sent by the log collection client, and compresses, stores, backs up and restores it; and at the same time, alarms the important logs to notify relevant personnel;

The unified management page provides an interface for interacting with users. In this interface, users can view logs they are interested in, perform system configuration and maintenance on the log collection client and log server, and analyze the operation behavior of end users in different applications, etc. .

The technical scheme that the present invention solves above-mentioned technical problem two is:

Specific steps are as follows,

Step 1, deploy the log collection client on each physical machine scattered in the internal network, collect log data that meets the collection conditions according to the pre-customized log template and collection frequency, and send it to the log server; The logging interface of the application system does not require any modification;

Step 2: The log server listens to the log data sent by each log collection client on the pre-allocated port and stores it in the database; automatically compresses the log data according to the amount of the current database and backs it up to the file; For historical data, decompress the data from the backup file and restore it to the database;

In the third step, the user checks the relevant log information received by the log server through the unified management page, and through the analysis and statistics module, displays the current running status of different applications in intuitive and reliable charts, reflecting different user operation behaviors and trends.

The present invention does not use API to intrude into the original code, but focuses on the log files generated by the original system, analyzes and filters the records in the log files one by one, and then determines whether to collect them. The present invention provides an easy-to-extend log template mechanism; it provides log screening and filtering functions in the form of templates to achieve customization. If the existing templates cannot meet the requirements, only a new template can be developed to quickly meet the current needs . The present invention introduces various web interfaces, report display methods, and integrates multiple analysis and processing methods to carry out in-depth mining and analysis on log data, and display vividly.

Therefore, the present invention quickly and effectively collects the logs generated by different operating systems and different applications, and can analyze and count information more useful than the log data itself, such as system operation status, user operation behavior, etc. Achieved: (1) high maintenance, keeping the original system as much as possible without modification; (2) scalability, meeting the ever-changing log collection requirements; (3) intelligence, mining through ordinary log data analysis More useful information. Improve and enhance the function of the existing log module.

Description of drawings

Below in conjunction with accompanying drawing, the present invention is further described:

Fig. 1 is a system block diagram of the present invention;

Fig. 2 is the flow chart of the implementation of the method of the present invention.

Detailed ways

Through in-depth analysis of the deficiencies of the existing log module, the present invention proposes a cross-platform cross-application log collection system and its collection management method, as shown in Figure 1, the system includes:

Log collection client: collect logs scattered in different operating systems and applications, and selectively collect interesting log data according to custom log formats, types or other keywords. Whether it is Windows or Linux operating system, mail server or ordinary web site, logs can be collected through this client.

Log server: Receive the log data sent by the log collection client, and perform compression, storage, backup and recovery. At the same time, alarms are sent to important logs to notify relevant personnel.

Unified management page: Provide an interface for interacting with users. In this interface, users can view the logs they are interested in, perform system configuration and maintenance on the first two entities, and analyze the operation behavior of end users in different applications.

Processing flow of the present invention is as Fig. 2, mainly comprises following 3 implementation steps:

Step 1, deploy the log collection client on each physical machine scattered in the internal network, collect log data that meets the collection conditions according to the pre-customized log template and collection frequency, and send it to the log server; The logging interface of the application system does not need any modification.

Step 2: The log server listens to the log data sent by each collection client on the pre-allocated port, and stores it in the database; automatically compresses the log data according to the amount of log data in the current database, and backs them up to the file; when you need to view the history When saving data, decompress the data from the backup file and restore it to the database.

In the third step, the user checks the relevant log information received by the log server through the unified management page, and through the analysis and statistics module, displays the current running status of different applications in intuitive and reliable charts, reflecting different user operation behaviors and trends.

Claims (2)

1. A cross-platform cross-application log collection system, characterized in that: comprising a log collection client, a log server and a unified management page; The log collection client collects logs scattered in different operating systems and different applications, and can selectively collect log data according to custom log formats, types or other keywords; it can be applied to Windows or Linux operating systems, mail server or normal web site; The log server receives the log data sent by the log collection client, and compresses, stores, backs up and restores it; and at the same time, alarms the important logs to notify relevant personnel; The unified management page provides an interface for interacting with users. In this interface, users can view logs they are interested in, perform system configuration and maintenance on the log collection client and log server, and analyze the operation behavior of end users in different applications, etc. . 2. A cross-platform and cross-application log collection management method, characterized in that: the specific steps are as follows, Step 1, deploy the log collection client on each physical machine scattered in the internal network, collect log data that meets the collection conditions according to the pre-customized log template and collection frequency, and send it to the log server; The logging interface of the application system does not require any modification; Step 2: The log server listens to the log data sent by each log collection client on the pre-allocated port and stores it in the database; automatically compresses the log data according to the amount of the current database and backs it up to the file; For historical data, decompress the data from the backup file and restore it to the database; In the third step, the user checks the relevant log information received by the log server through the unified management page, and through the analysis and statistics module, displays the current running status of different applications in intuitive and reliable charts, reflecting different user operation behaviors and trends.

Images