CN107635003A - System log management method, device and system - Google Patents

Abstract

The invention discloses a system log management method, which includes: filtering collected log information according to predetermined rules; dividing the filtered log information according to information categories; sorting the category-divided log information according to a predetermined log format Combining; regularly pushing the combined log information as the final log information to the log server, so that the log server can store and visualize the received final log information; this method can make the log information concise and clear enough, convenient Management of log information; the present invention also discloses a system log management device and system, which have the above beneficial effects.

Description

System log management method, device and system

technical field

The invention relates to the field of electronic technology, in particular to a system log management method, device and system.

Background technique

In the system, the log is a very important functional component. The log can record all the behaviors generated by the system and express them according to a certain specification. When an error occurs in the system, the information recorded in the log can be used to troubleshoot the system, optimize the performance of the system, or adjust various behaviors of the system according to the log information. The status of the log in the system is very important.

With the expansion of the scale of the enterprise, the business becomes more diversified and more complex, so the system becomes larger, more and more subsystems are integrated in a system, and the log information generated in a subsystem also becomes more , so the log information in a system becomes very large, and it is particularly important to filter out useful information from the huge log information.

If the log information is classified according to the system type, the log information can be divided into operating system logs, application system logs, security system logs, etc., and the logs of each operating system have their own unique design and specifications, such as Windows system Logs are usually stored according to their customary classification methods such as application, security and system, and various Class UNIX systems like Linux usually use a log system compatible with the Syslog specification; and the operating systems of many hardware devices also have independent log specification. Therefore, different application systems have logs designed according to their own requirements, and their log information formats are different.

In the prior art, when a failure occurs during the operation of the entire system, it is directly connected to the server through tools such as ssh to directly view all system log information, and it is very difficult to find and locate problems in all log information.

Therefore, how to make the log information concise and clear enough to facilitate the management of the log information is a technical problem to be solved by those skilled in the art.

Contents of the invention

The purpose of the present invention is to provide a system log management method, which can make the log information concise and clear enough to facilitate the management of the log information; another purpose of the present invention is to provide a system log management device and system, with The beneficial effects mentioned above.

In order to solve the above technical problems, the present invention provides a system log management method, including:

Filter the collected log information according to predetermined rules;

Divide the filtered log information according to the information category;

The log information after classification is combined according to a predetermined log format; wherein, the predetermined log format includes time, machine name, error category, thread name, log level, and log output location;

The combined log information is regularly pushed to the log server as the final log information, so that the log server can store and visualize the received final log information.

Wherein, the filtering of log information according to predetermined rules includes:

The start and end information of functional modules, user login and exit information, critical system operation information, abnormal information during system operation, and entry and exit information of key methods are screened out from the log information.

Among them, dividing the filtered log information according to information categories includes:

According to the information category in the log information, the log information is divided into system debugging log, critical log of system operation, alarm log and system error log.

Wherein, the filtering of the log information to be collected according to predetermined rules includes:

Collect incremental log information; wherein, the incremental log information includes log information generated after the last push;

The incremental log information is filtered according to predetermined rules.

The present invention provides a system log management device, including:

An information screening module, configured to filter the collected log information according to predetermined rules;

A category classification module, configured to classify the filtered log information according to information categories;

Combination module, used to combine the log information after classification according to a predetermined log format; wherein, the predetermined log format includes time, machine name, error category, thread name, log level, and log output location;

A timing push module, configured to regularly push the combined log information as final log information to a log server, so that the log server can store and visualize the received final log information.

The present invention provides a system log management system, including:

The client is used to filter the collected log information according to predetermined rules; after filtering, divide the log information according to information categories; combine the classified log information according to a predetermined log format; combine the combined log information As the final log information, it is regularly pushed to the log server; wherein, the predetermined log format includes time, machine name, error category, thread name, log level and log output location;

The log server is configured to receive and store the final log information regularly sent by the client, and perform visual processing on the received final log information.

Wherein, the log server is further configured to perform cross-analysis on the received log information of different clients.

Wherein, the log server is further configured to receive a command to update the list, and update the stored list of clients that need to receive log information.

Wherein, the client is also used to filter the collected log information to start and end information of functional modules, user login and logout information, key operating information of the system, abnormal information during system operation, and entry of key methods and exit messages.

Wherein, the client is also used to collect incremental log information; and filter the incremental log information according to predetermined rules; wherein, the incremental log information includes log information generated after the last push .

The management method of the system log provided by the present invention screens the log information according to predetermined rules, divides the log information according to the information category after screening, then combines according to the predetermined log format, and finally pushes the combined log information as the final log information at regular intervals To the log server, the generated log information is uniformly and standardizedly processed and then sent regularly. The log information after uniform and standardized processing only retains useful information, reducing a lot of useless information, making the log information concise enough and useful to the log The uniform format of the information makes the format of different log information received by the log server end the same, and the important information is prominent, so that the log information is concise and clear enough, which can greatly simplify the search for log information during operation, saving manpower and time. The log information is sent regularly to facilitate the timely storage, judgment and processing of the latest log information. Therefore, this method can make the log information concise and clear enough, and facilitate the management of the log information to a large extent.

The present invention also discloses a system log management device and system, which have the above-mentioned beneficial effects, and will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

Fig. 1 is the flowchart of the management method of system log provided by the embodiment of the present invention;

FIG. 2 is a structural block diagram of a system log management device provided by an embodiment of the present invention;

FIG. 3 is a structural block diagram of a system log management system provided by an embodiment of the present invention.

detailed description

The core of the present invention is to provide a system log management method, which can make the log information concise and clear enough to facilitate the management of log information; another core of the present invention is to provide a system log management device and system, with The beneficial effects mentioned above.

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to FIG. 1, which is a flow chart of a method for managing system logs provided by an embodiment of the present invention; the method may include:

Step s100, filtering the collected log information according to predetermined rules.

Preferably, the collected log information may be incremental log information. Of course, collecting incremental log information is only a preferred embodiment, and collecting existing logs can also implement the technical solution of the present invention, which is not limited here. Incremental log information refers to the log information generated since the last push. Only collecting incremental log information can greatly reduce the total amount of log information that needs to be processed, improve work efficiency, and save manpower and time resources for viewing logs .

The collected log information can be filtered according to predetermined rules to filter out useful information in the log information, reduce a large amount of useless information, and make the log information more concise. In this embodiment, the filtered log information is not limited, and users can select according to their own needs and system needs.

Preferably, start and end information of functional modules, user login and exit information, critical system operation information, abnormal information during system operation, and entry and exit information of key methods can be filtered out from the log information.

Generally speaking, a complete system is composed of multiple functional modules, and each module is responsible for different functions. The start and end information of the functional modules can be filtered out from the log information, and the start and end information of the functional modules can be monitored. Know whether the above-mentioned functional modules are loaded normally, and whether they log out normally when the operation is completed normally; you can also filter out the user's login and logout information, and you can know which user logged in and logged out of the system through what IP at what time; you can also filter Key operating information of the system, such as database link information and network communication success information, etc., are used to monitor these key operations; it can also filter out abnormal information during system operation, and can be used when the system runs wrong View the problem through these exception information, where the exception information can include NPE, OOM and other timeout and conversion exception information; you can also filter out the entry and exit of some key methods to monitor whether these methods are running normally, among them , the key method can include the entry and end information of some important business during processing. Of course, other useful information can also be screened out according to certain rules. Wherein, the predetermined rule may be a rule formulated for the log information to be screened out, and this embodiment does not limit the rule.

Step s110, classify the filtered log information according to information categories.

After filtering out the useful log information needed from the collected log information, the log information can be divided according to the information category of the log information, so as to manage different types of log information in a unified way, and make the log information clear. Users can classify the filtered log information according to their own needs, which can be divided into four categories, six categories or eight categories, all of which can realize the technical solution of the present invention, and users can choose by themselves.

Generally speaking, log information can be divided into four categories: DEBUG, INFO, WARN, ERROR.

DEBUG (system debugging information): It can be used to monitor the system operation during the development process, and it is not necessary to output in the actual operating environment.

INFO (key information of system operation): It can be used to monitor the system operation.

WARN (warning information): It can be used to record potential problems in the system and may cause abnormal operation, but no abnormal log information has been generated at this time.

ERROR (system error information): It can be used to record error information in the system and information that needs to be processed in a timely manner.

Of course, the log information can also be divided according to different needs, and here only the above four types of division are used as an example for introduction, other division methods can refer to the introduction of this embodiment, and will not be repeated here.

Step s120, combining the classified log information according to a predetermined log format.

After the log information is divided, the log information can be combined according to the predetermined log format, so that combining different log information with the same predetermined log template can greatly simplify the work of finding the required information from the log information. Among them, the predetermined log format can include time, machine name, error category, thread name, log level, log output location, and information. Of course, it can also include other content, such as error codes, etc. Users can choose the content by themselves. This is not limited. Combine the selected content, for example, you can use "time-[machine name][error category][thread name][log level]-log output location (full class name, which can be accurate to the method name): log information "This form of combination can also be "time-[machine name][thread name][log level][error category]-log output location (full class name, which can be accurate to the method name): log information" like this According to different needs, different log information can be selected to arrange and combine in different orders, and the order is not limited here. The specific log information can be in the following form:

"2013-09-04 10:49:20.296-[server1][systemLog][Thread-initRedis21504][INFO]-com.shanghai.LoginController.initLogInfo: LingMing[User] is login"

Step s130, regularly push the combined log information to the log server as the final log information.

The combined log information can be regarded as the final log information and regularly pushed to the log server, so that the log server can store and visualize the received final log information.

In the present invention, the heartbeat mechanism can be used to regularly send the log information. Each client can send the log information regularly by installing a timing log push script or program to monitor the heartbeat frequency, or it can also push the log information regularly by other means. This is just an example of the heartbeat mechanism. Regularly pushing logs can push the latest logs to the log server. The log server can save and process the latest log information in time to facilitate the management of log information.

When pushing the log information to the log server, a connection can be established with the log server, wherein, preferably, the login connection can be stored for a long time, so as to reduce the resources used by frequently establishing connections. If many clients need to push log information to the log server at the same time, the connection can be released immediately after each client establishes a connection and sends the log, which can avoid errors caused by too many connections to the log server.

Based on the above technical solution, the system log management method provided by the embodiment of the present invention, the method adopts unified and standardized processing of log information, and the processed log information has simplified its content to a large extent, highlighting the key points Information, so that the log information is concise enough, and the useful information in the log is uniformly standardized, so that the different log information received by the log server has the same format, and the important information is prominent, which can greatly simplify the search for log information during operation. It saves manpower and time, and regularly sends the log information to facilitate timely storage, timely judgment and timely processing of the latest log information. Therefore, this method can make the log information concise and clear enough, and facilitates the log information to a large extent. manage.

Please refer to FIG. 2, which is a structural block diagram of a system log management device provided by an embodiment of the present invention; the device may include:

An information screening module 100, configured to filter the collected log information according to predetermined rules;

Wherein, preferably, the collected log information may be incremental log information, and incremental log information includes log information generated after the last push. The screening according to predetermined rules may include screening out the start and end information of functional modules, user login and logout information, critical system operation information, abnormal information during system operation, and entry and exit information of key methods.

A category division module 200, configured to divide the filtered log information according to the information categories;

Wherein, preferably, the logs can be divided into system debugging logs, critical logs of system operation, alarm logs and system error logs.

Combining module 300, configured to combine the classified log information according to a predetermined log format; wherein, the predetermined log format may include time, machine name, error category, thread name, log level, and log output location ;

The timing push module 400 is configured to periodically push the combined log information as final log information to a log server, so that the log server can store and visualize the received final log information.

Please refer to Fig. 3, Fig. 3 is a structural block diagram of the management system of the system log that the embodiment of the present invention provides; The system may include:

The client 500 is configured to filter the collected log information according to predetermined rules; after filtering, divide the log information according to information categories; combine the classified log information according to a predetermined log format; combine the combined log information The information is regularly pushed to the log server as the final log information; wherein, the predetermined log format may include time, machine name, error category, thread name, log level, and log output location;

The log server 600 is configured to receive and store the final log information regularly sent by the client, and perform visual processing on the received final log information.

The log server can store the collected log information to achieve secondary remote backup of the log information. The storage can be stored in the database or directly in the file system, and the storage method is not limited. If the log information is structured, it is preferably stored in the database, so as to facilitate cross-analysis of logs of different application systems and output more valuable log analysis results.

The log server can provide a visual operation page for centralized management of log information, and can present log information of different levels in different pages to present fault logs to users more directly and in a timely manner, and can provide quick search, search, and fast With filtering and positioning functions, users can conduct daily management searches and business analysis. It can be seen that the user can view the operation status of the business system through the visualized page, so as to detect system operation failures in time, efficiently locate the cause of the problem, and save manpower and time costs. Wherein, the visualized operation page can be implemented in the form of web or management software.

Wherein, preferably, the log server is further configured to perform cross-analysis on the received log information of different clients.

The log server can receive log information from one or more clients, and the log server can perform cross-analysis on the received log information from multiple clients to achieve more complex and valuable log analysis. For example, the log server receives log information from 360 anti-virus software and Duba anti-virus software. When the two softwares perform anti-virus operations on the computer respectively, the log information pushed by the 360 anti-virus software shows that the computer network connection failed, specifically because the network maintenance is automatically disconnected. connection, and the log information pushed by Duba antivirus software shows that the network connection user login timed out. Therefore, after cross-analysis of the log information of 360 antivirus software and Duba antivirus software, the two causes of errors can be verified separately. Finally, it was found that the network connection failed due to the user login timeout. After troubleshooting, the network connection was normal. It can be seen that more effective log analysis can be performed by cross-analyzing log information of different clients.

Wherein, preferably, the log server is further configured to receive a command to update the list, and update the stored list of clients that need to receive log information.

The log server can store a list in advance, which can record some relevant information of the client that needs to collect log information, such as the name of the client, the time when the client pushes the log each time, and so on. For a client that is newly added, needs to be deleted, or needs to change information, the log server can only add, delete, or modify the relevant information of the client in the list, so as to achieve faster and more effective horizontal expansion.

Wherein, preferably, the client is also used to filter the collected log information to start and end information of functional modules, user login and logout information, critical operating information of the system, abnormal information during system operation, and critical Method entry and exit information.

Wherein, preferably, the client is also used to collect incremental log information; and filter the incremental log information according to predetermined rules; wherein, the incremental log information includes log information.

Each embodiment in the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

Professionals can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the possible For interchangeability, in the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The system log management method, device and system provided by the present invention have been introduced in detail above. In this paper, specific examples are used to illustrate the principle and implementation of the present invention, and the descriptions of the above embodiments are only used to help understand the method and core idea of the present invention. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (10)

1. A method for managing system logs, comprising: Filter the collected log information according to predetermined rules; Divide the filtered log information according to the information category; The log information after classification is combined according to a predetermined log format; wherein, the predetermined log format includes time, machine name, error category, thread name, log level, and log output location; The combined log information is regularly pushed to the log server as the final log information, so that the log server can store and visualize the received final log information. 2. The log management method according to claim 1, wherein said filtering log information according to predetermined rules comprises: The start and end information of functional modules, user login and exit information, critical system operation information, abnormal information during system operation, and entry and exit information of key methods are screened out from the log information. 3. The log management method according to claim 2, wherein dividing the filtered log information according to information categories includes: According to the information category in the log information, the log information is divided into system debugging log, critical log of system operation, alarm log and system error log. 4. The log management method according to any one of claims 1 to 3, wherein the filtering of the collected log information according to predetermined rules includes: Collect incremental log information; wherein, the incremental log information includes log information generated after the last push; The incremental log information is filtered according to predetermined rules. 5. A system log management device, characterized in that, comprising: An information screening module, configured to filter the collected log information according to predetermined rules; A category classification module, configured to classify the filtered log information according to information categories; Combination module, used to combine the log information after classification according to a predetermined log format; wherein, the predetermined log format includes time, machine name, error category, thread name, log level, and log output location; A timing push module, configured to regularly push the combined log information as final log information to a log server, so that the log server can store and visualize the received final log information. 6. A management system for system logs, comprising: The client is used to filter the collected log information according to predetermined rules; after filtering, divide the log information according to information categories; combine the classified log information according to a predetermined log format; combine the combined log information As the final log information, it is regularly pushed to the log server; wherein, the predetermined log format includes time, machine name, error category, thread name, log level and log output location; The log server is configured to receive and store the final log information regularly sent by the client, and perform visual processing on the received final log information. 7. The log management system according to claim 6, wherein the log server is further configured to perform cross-analysis on the received log information of different clients. 8. The log management system according to claim 7, wherein the log server is further configured to receive a command to update the list, and update the stored list of clients that need to receive log information. 9. The log management system according to claim 8, wherein the client is also used to screen the collected log information into the start and end information of the functional module, the login and exit information of the user, and the criticality of the system. Operational information, abnormal information during system operation, and entry and exit information of key methods. 10. The log management system according to claim 9, wherein the client is also used to collect incremental log information; and filter the incremental log information according to predetermined rules; wherein, the Incremental log information includes log information generated since the last push.