CN112989823B

CN112989823B - Log processing method, device, equipment and storage medium

Info

Publication number: CN112989823B
Application number: CN202110456697.3A
Authority: CN
Inventors: 饶琛琳; 梁玫娟
Original assignee: Beijing Youtejie Information Technology Co ltd
Current assignee: Beijing Youtejie Information Technology Co ltd
Priority date: 2021-04-27
Filing date: 2021-04-27
Publication date: 2021-08-13
Anticipated expiration: 2041-04-27
Also published as: CN112989823A

Abstract

The embodiment of the invention discloses a log processing method, a log processing device, log processing equipment and a log processing storage medium. The method comprises the following steps: acquiring all log information of a first service system and a second service system, and extracting field names and field values of fields in each piece of log information; determining a common field set; for each piece of log information of the first service system, sequentially comparing the field value of each common field in each piece of log information with the field value of each common field in each piece of log information of the second service system, and determining the same number of the field values between each piece of log information of the second service system; and aiming at each piece of log information of the first service system, sequencing each piece of log information of the second service system from high to low according to the same number of field values, and establishing an incidence relation between the log information and the log information at the first position of a sequencing result. The embodiment of the invention can automatically determine the associated log information in different service systems and establish the association relationship.

Description

Log processing method, device, equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of computers, in particular to a log processing method, a log processing device, log processing equipment and a log processing storage medium.

Background

In the current software architecture, a complete service often runs through multiple business systems, and needs to be called in multiple applications to satisfy a user request. The service detail log of each service is obtained by associating the logs related to each service in a plurality of service systems into a complete log. For example, in a financial information system, a complete transaction service relates to an online banking system (ebank system), an enterprise service bus system (esb system), and a financial sharing service system (fss system), and it is necessary to associate logs related to the transaction service in the online banking system, the enterprise service bus system, and the financial sharing service system into a complete log, so as to obtain a transaction detail log of the transaction.

In the related art log system, the log often lacks a global plan. In order to obtain a log related to a certain service in a plurality of service systems, after a certain log related to the service is searched from the service system a, an operation and maintenance person needs to use a related field of the log in the service system a according to experience to search logs related to the service in other service systems, which is time-consuming and labor-consuming, and has a very high requirement on the experience of the operation and maintenance person. In addition, since the logs are often massive data, it is difficult for a manual search to involve all the logs, i.e., to ensure the integrity of the search.

Disclosure of Invention

Embodiments of the present invention provide a log processing method, an apparatus, a device, and a storage medium, which can automatically determine associated log information in different service systems, and quickly establish an association relationship between log information of different service systems.

In a first aspect, an embodiment of the present invention provides a log processing method, including:

acquiring all log information of a first service system and all log information of a second service system, and extracting field names and field values of fields in each piece of log information of the first service system and field names and field values of fields in each piece of log information of the second service system;

determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system; wherein the common field set includes field names of a plurality of common fields;

for each piece of log information of the first service system, sequentially comparing the field value of each common field in each piece of log information with the field value of each common field in each piece of log information of the second service system, and determining the same number of the field values between each piece of log information of the first service system and each piece of log information of the second service system;

and aiming at each piece of log information of the first service system, sequencing each piece of log information of the second service system from high to low according to the same number of the field values, and establishing an incidence relation between each piece of log information and the log information at the first position of a sequencing result.

In a second aspect, an embodiment of the present invention further provides a log processing apparatus, including:

the field extraction module is used for acquiring all log information of a first service system and all log information of a second service system, and extracting the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system;

a set determining module, configured to determine a common field set between the first service system and the second service system according to a field name and a field value of each field in each piece of log information of the first service system and a field name and a field value of each field in each piece of log information of the second service system; wherein the common field set includes field names of a plurality of common fields;

a field comparison module, configured to, for each piece of log information of the first service system, sequentially compare field values of the common fields in each piece of log information with field values of the common fields in each piece of log information of the second service system, and determine that the number of field values between each piece of log information of the first service system and each piece of log information of the second service system is the same;

and the relationship establishing module is used for sequencing each piece of log information of the second service system from high to low according to the same number of the field values aiming at each piece of log information of the first service system, and establishing an incidence relationship between each piece of log information and the log information at the first position of the sequencing result.

In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the log processing method according to the embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the log processing method according to the embodiment of the present invention.

According to the technical scheme of the embodiment of the invention, the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system are extracted; then determining a common field set between the first service system and the second service system; for each piece of log information of the first service system, sequentially comparing the field value of each common field in each piece of log information with the field value of each common field in each piece of log information of the second service system, and determining the same number of the field values between each piece of log information of the first service system and each piece of log information of the second service system; and finally, aiming at each piece of log information of the first service system, sequencing each piece of log information of the second service system from high to low according to the same number of the field values, establishing an association relationship between each piece of log information and the log information at the first position of the sequencing result, automatically extracting field information in the log information of different service systems, and automatically determining the associated log information in different service systems and establishing the association relationship according to the field information in the log information of different service systems, so that the association relationship between the log information of different service systems is quickly established, and operation and maintenance personnel can automatically and quickly acquire the associated log information in different service systems according to the association relationship.

Drawings

Fig. 1 is a flowchart of a log processing method according to an embodiment of the present invention.

Fig. 2 is a flowchart of a log processing method according to a second embodiment of the present invention.

Fig. 3 is a flowchart of a log processing method according to a third embodiment of the present invention.

Fig. 4 is a schematic structural diagram of a log processing apparatus according to a fourth embodiment of the present invention.

Fig. 5 is a schematic structural diagram of a computer device according to a fifth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention.

It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

Example one

Fig. 1 is a flowchart of a log processing method according to an embodiment of the present invention. The embodiment of the invention can be suitable for determining the related log information in different service systems and establishing the incidence relation between the log information of the different service systems. For example in a server. As shown in fig. 1, the method of the embodiment of the present invention specifically includes:

step 101, acquiring all log information of a first service system and all log information of a second service system, and extracting a field name and a field value of each field in each log information of the first service system and a field name and a field value of each field in each log information of the second service system.

Optionally, the first service system and the second service system are two adjacent service systems. The log information of the first business system may be associated with the log information of the second business system.

Optionally, the extracting the field name and the field value of each field in each piece of log information of the first service system, and the field name and the field value of each field in each piece of log information of the second service system, includes: determining separators of fields and field values of each piece of log information of the first service system and the second service system and separators between the fields according to the analysis rules, extracting field names and field values of each field in each piece of log information, and establishing a corresponding relation between the field names and the field values of each field.

Each piece of log information includes a plurality of fields. Each field has a corresponding field name and field value. Illustratively, the field name of the field is "username", and the field value is the real name of the user. The field name of the field is 'identification number', and the field value is a specific identification number. The field name of the field is "amount", and the field value is a specific amount.

Step 102, determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system.

Wherein the common field set includes field names of a plurality of common fields.

Optionally, the common field between the first service system and the second service system is a field included in both the log information of the first service system and the log information of the second service system. The common field set between the first service system and the second service system is a set formed by a plurality of common fields between the first service system and the second service system.

Optionally, the determining, according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system, a common field set between the first service system and the second service system includes: when detecting that a user sets information for a common field fed back by the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system, determining a common field set between the first service system and the second service system according to the common field setting information.

The common field setting information is information for setting a common field set between the first service system and the second service system. The common field setting information may contain field names of a plurality of fields. And the names of a plurality of fields in the common field setting information are the names of a plurality of common fields between the first service system and the second service system specified by a user.

Optionally, the field name and the field value of each field in each piece of log information of the first business system and the field name and the field value of each field in each piece of log information of the second business system are displayed to the user in a form of list items through a visual interface. And feeding back common field setting information by the user according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system. When common field setting information fed back by a user is detected, determining a plurality of field names in the common field setting information as field names of common fields between the first service system and the second service system, and storing the field names into a common field set between the first service system and the second service system. In an initial state, a common field set between the first service system and the second service system is empty.

Optionally, the determining, according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system, a common field set between the first service system and the second service system includes: and determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system through a machine learning algorithm.

Optionally, the machine learning algorithm includes, but is not limited to, a clustering algorithm, a feature extraction algorithm, and the like.

Optionally, the determining, by a machine learning algorithm, a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system includes: clustering all log information of the first service system and all log information of the second service system according to the field name and the field value of each field in each log information through a clustering algorithm to obtain a plurality of categories of log information sets; extracting the same field among the log information sets of all categories, and determining a high-cardinality field in the same field among the log information sets of all categories as a common field between the first service system and the second service system; and storing the field name of the common field between the first service system and the second service system into the common field set between the first service system and the second service system.

The log information set of each category comprises log information with high similarity.

Optionally, extracting the same field between log information sets of each category includes: extracting fields contained in the log information set as the same fields of the log information set aiming at the log information set of each category; and acquiring fields contained in the same fields of the log information sets of all the categories as the same fields among the log information sets of all the categories.

A high radix field is a field of a field value that has more values to take. Optionally, determining a high-radix field in the same field between the log information sets of the respective categories as a common field between the first service system and the second service system includes: and removing fields with single values of field values aiming at the same fields among the log information sets of all the categories, and reserving fields with more values of field values as common fields between the first service system and the second service system. And then storing the field names of the common fields between the first service system and the second service system into the common field set between the first service system and the second service system. In an initial state, a common field set between the first service system and the second service system is empty.

Therefore, a machine learning algorithm is utilized to find a minimum field set jointly covered by the log information of the first business system and the second business system, and the minimum field set is determined as a common field set between the first business system and the second business system.

Optionally, after storing the field name of the common field between the first service system and the second service system in the common field set between the first service system and the second service system, the method further includes: taking the field name and the field value of each common field in each piece of log information of the first service system and the second service system as the characteristic variable of each piece of log information; performing Principal Component Analysis (PCA) calculation according to the characteristic variables of each piece of log information of the first service system and the second service system, and determining the contribution of each common field to the log information principal component; and sequencing the common fields according to the contribution degrees from high to low to obtain the contribution degree sequencing result of each common field in the common field set between the first service system and the second service system.

Optionally, PCA calculation is performed according to the feature variables of each piece of log information of the first service system and the second service system, and the contribution of each common field to the log information principal component is obtained according to a load matrix of the PCA.

Step 103, for each piece of log information of the first service system, sequentially comparing field values of each common field in each piece of log information with field values of each common field in each piece of log information of the second service system, and determining the same number of field values between each piece of log information of the first service system and each piece of log information of the second service system.

Wherein the same number of field values is the number of same field values between each piece of log information.

Optionally, the sequentially comparing, for each piece of log information of the first service system, field values of the common fields in each piece of log information with field values of the common fields in each piece of log information of the second service system, and determining that the number of the field values between each piece of log information of the first service system and each piece of log information of the second service system is the same includes: filtering null value fields and preset non-processing fields in all log information of the first service system and the second service system; acquiring a piece of log information from all the filtered log information of the first service system as current processing log information; sequentially comparing the field value of each common field in the currently processed log information with the field value of each common field in each piece of log information of the second service system, and determining that the currently processed log information is the same as the field value of each piece of log information of the second service system in quantity; and returning to execute the operation of acquiring one piece of log information from all the filtered log information of the first service system as the current processing log information until the processing of all the filtered log information of the first service system is finished. In an initial state, the same number of field values of the currently processed log information and each piece of log information of the second service system is 0.

The null value field is a field whose field value is null. The preset non-processing field is a field specified by the user to be ignored in the field value comparison process. Therefore, before the field value comparison process, the fields irrelevant to the field value comparison process in the log information are filtered, and the field value comparison efficiency is improved.

Optionally, when the common field set between the first service system and the second service system is determined according to the common field setting information, the field values of the common fields in the currently processed log information and the field values of the common fields in each piece of log information of the second service system may be sequentially compared according to a comparison priority order of the common fields in the common field set preset by a user, so as to determine that the number of the field values of the currently processed log information and each piece of log information of the second service system is the same.

Optionally, a field value of each common field in the currently processed log information is obtained. According to the comparison priority sequence of all common fields in a common field set preset by a user, the field value of all the common fields in each piece of log information of a second service system is obtained according to the field name, the field value of all the common fields in the current processing log information is compared with the field value of all the common fields in each piece of log information of the second service system, and the same number of the field values of the current processing log information and each piece of log information of the second service system is determined.

In one specific example, the common field set includes field names of a plurality of common fields: "user name", "identification number" and "amount". The comparison priority sequence of each common field in the common field set preset by the user is as follows: "user name" → "identification number" → "amount". And acquiring field values of fields with field names of 'user name', 'identification number' and 'amount' in the current processing log information. For each piece of log information of the second service system, performing the following operations: and acquiring the field value of a field with the field name of 'user name' in the log information, and comparing whether the field value of the field with the field name of 'user name' in the currently processed log information is the same as the field value of the field with the field name of 'user name' in the log information. In the case where the field value of the field whose field name is "user name" in the currently processed log information is the same as the field value of the field whose field name is "user name" in the log information, the same number of field values between the currently processed log information and the log information is incremented by one. In the case where the field value of the field whose field name is "user name" in the currently processed log information is not the same as the field value of the field whose field name is "user name" in the log information, the same number of field values between the currently processed log information and the log information remains unchanged. And then, acquiring a field value of a field with a field name of 'identity card number' in the log information, and comparing whether the field value of the field with the field name of 'identity card number' in the currently processed log information is the same as the field value of the field with the field name of 'identity card number' in the log information. And under the condition that the field value of the field with the field name of 'identity card number' in the current processing log information is the same as the field value of the field with the field name of 'identity card number' in the log information, adding one to the same number of the field values between the current processing log information and the log information. In the case where the field value of the field whose field name is "identification number" in the currently processed log information is not the same as the field value of the field whose field name is "identification number" in the log information, the same number of field values between the currently processed log information and the log information remains unchanged. And finally, acquiring the field value of the field with the field name of 'amount' in the log information, and comparing whether the field value of the field with the field name of 'amount' in the currently processed log information is the same as the field value of the field with the field name of 'amount' in the log information. In the case where the field value of the field with the field name "amount" in the currently processed log information is the same as the field value of the field with the field name "amount" in the log information, the same number of field values between the currently processed log information and the log information is incremented by one. In the case where the field value of the field with the field name "amount" in the currently processed log information is not the same as the field value of the field with the field name "amount" in the log information, the same number of field values between the currently processed log information and the log information remains unchanged.

Optionally, a field value of each common field in the currently processed log information is obtained. According to the comparison priority sequence of all common fields in a common field set preset by a user, comparing the field value of all common fields in the currently processed log information with the field value of all common fields in each piece of log information of the second service system according to the field values, and determining that the number of the field values of the currently processed log information is the same as that of each piece of log information of the second service system.

In one specific example, the common field set includes field names of a plurality of common fields: "user name", "identification number" and "amount". The comparison priority sequence of each common field in the common field set preset by the user is as follows: "user name" → "identification number" → "amount". And acquiring field values of fields with field names of 'user name', 'identification number' and 'amount' in the current processing log information. For each piece of log information of the second service system: and inquiring whether a field value with a field name of 'user name' in the currently processed log information exists in field values in the log information. In the case where a field value whose field name in the currently processed log information is a "user name" field exists among field values in the log information, the same number of field values between the currently processed log information and the log information is incremented by one. In the case where there is no field value of a field name "user name" field in the currently processed log information among the field values in the log information, the same number of field values between the currently processed log information and the log information remains unchanged. And then inquiring whether a field value with a field name of 'identity card number' in the currently processed log information exists in field values in the log information. And in the case that the field value of the field name of the currently processed log information is the 'identity number' field exists in all the field values in the log information, adding one to the same number of the field values between the currently processed log information and the log information. In the case where there is no field value of the field name "identification number" field in the currently processed log information among the field values in the log information, the same number of field values between the currently processed log information and the log information remains unchanged. And finally, inquiring whether field values of field names of 'money' in the currently processed log information exist in the field values in the log information. In the case where a field value of a field name "amount" in the currently processed log information exists among field values in the log information, the same number of field values between the currently processed log information and the log information is incremented by one. In the case where a field value of a field name "amount" field in the currently processed log information does not exist in the field values in the log information, the same number of field values between the currently processed log information and the log information remains unchanged.

Optionally, when the common field set between the first service system and the second service system is determined by a machine learning algorithm, the sequentially comparing field values of the common fields in the currently processed log information with field values of the common fields in each piece of log information of the second service system, and determining that the currently processed log information is the same as the field values of each piece of log information of the second service system includes: and according to the contribution degree sorting result of each common field, comparing the field value of each common field in the currently processed log information with the field value of each common field in the target log information of the second service system, and determining that the number of the currently processed log information is the same as that of the field values of the target log information of the second service system.

And taking the contribution degree sequencing result of each common field as the comparison priority of each common field, sequentially comparing the field value of each common field in the currently processed log information with the field value of each common field in the target log information of the second service system, and determining that the currently processed log information and the target log information of the second service system have the same number of field values.

Optionally, a field value of each common field in the currently processed log information is obtained. According to the contribution degree sorting result of each common field, the field value of each common field in each piece of log information of a second service system is obtained according to the field name, the field value of each common field in the current processing log information is compared with the field value of each common field in each piece of log information of the second service system, and the same quantity of the field values of the current processing log information and each piece of log information of the second service system is determined.

In one specific example, the common field set includes field names of a plurality of common fields: "user name", "identification number" and "amount". The contribution degree sequencing result of each common field is as follows: "identification number" → "user name" → "amount". And acquiring field values of fields with field names of 'identification number', 'user name' and 'amount' in the current processing log information. For each piece of log information of the second service system, performing the following operations: and acquiring a field value of a field with a field name of 'identity card number' in the log information, and comparing whether the field value of the field with the field name of 'identity card number' in the currently processed log information is the same as the field value of the field with the field name of 'identity card number' in the log information. And under the condition that the field value of the field with the field name of 'identity card number' in the current processing log information is the same as the field value of the field with the field name of 'identity card number' in the log information, adding one to the same number of the field values between the current processing log information and the log information. In the case where the field value of the field whose field name is "identification number" in the currently processed log information is not the same as the field value of the field whose field name is "identification number" in the log information, the same number of field values between the currently processed log information and the log information remains unchanged. And then acquiring the field value of a field with the field name of 'user name' in the log information, and comparing whether the field value of the field with the field name of 'user name' in the currently processed log information is the same as the field value of the field with the field name of 'user name' in the log information. In the case where the field value of the field whose field name is "user name" in the currently processed log information is the same as the field value of the field whose field name is "user name" in the log information, the same number of field values between the currently processed log information and the log information is incremented by one. In the case where the field value of the field whose field name is "user name" in the currently processed log information is not the same as the field value of the field whose field name is "user name" in the log information, the same number of field values between the currently processed log information and the log information remains unchanged. And finally, acquiring the field value of the field with the field name of 'amount' in the log information, and comparing whether the field value of the field with the field name of 'amount' in the currently processed log information is the same as the field value of the field with the field name of 'amount' in the log information. In the case where the field value of the field with the field name "amount" in the currently processed log information is the same as the field value of the field with the field name "amount" in the log information, the same number of field values between the currently processed log information and the log information is incremented by one. In the case where the field value of the field with the field name "amount" in the currently processed log information is not the same as the field value of the field with the field name "amount" in the log information, the same number of field values between the currently processed log information and the log information remains unchanged.

Optionally, a field value of each common field in the currently processed log information is obtained. And according to the contribution degree sorting result of each common field, comparing the field value of each common field in the currently processed log information with the field value of each common field in each piece of log information of the second service system according to the field values, and determining that the number of the field values of the currently processed log information is the same as that of each piece of log information of the second service system.

In one specific example, the common field set includes field names of a plurality of common fields: "user name", "identification number" and "amount". The contribution degree sequencing result of each common field is as follows: "identification number" → "user name" → "amount". And acquiring field values of fields with field names of 'identification number', 'user name' and 'amount' in the current processing log information. For each piece of log information of the second service system: and inquiring whether a field value of a field name, namely an identity card number field, in the currently processed log information exists in field values in the log information. And in the case that the field value of the field name of the currently processed log information is the 'identity number' field exists in all the field values in the log information, adding one to the same number of the field values between the currently processed log information and the log information. In the case where there is no field value of the field name "identification number" field in the currently processed log information among the field values in the log information, the same number of field values between the currently processed log information and the log information remains unchanged. And then inquiring whether a field value with a field name of 'user name' in the currently processed log information exists in field values in the log information. In the case where a field value whose field name in the currently processed log information is a "user name" field exists among field values in the log information, the same number of field values between the currently processed log information and the log information is incremented by one. In the case where there is no field value of a field name "user name" field in the currently processed log information among the field values in the log information, the same number of field values between the currently processed log information and the log information remains unchanged. And finally, inquiring whether field values of field names of 'money' in the currently processed log information exist in the field values in the log information. In the case where a field value of a field name "amount" in the currently processed log information exists among field values in the log information, the same number of field values between the currently processed log information and the log information is incremented by one. In the case where a field value of a field name "amount" field in the currently processed log information does not exist in the field values in the log information, the same number of field values between the currently processed log information and the log information remains unchanged.

And 104, sequencing each piece of log information of the second service system from high to low according to the same number of the field values aiming at each piece of log information of the first service system, and establishing an association relation between each piece of log information and the log information at the first position of a sequencing result.

In general, the more field values identical between two pieces of log information, the higher the number of field values identical, and the greater the relevance. And aiming at each piece of log information of the first service system, sequencing each piece of log information of the second service system from high to low according to the same number of the field values. The log information at the first bit of the sorting result is the log information with the maximum relevance with the log information in the second service system. And establishing an incidence relation between each piece of log information and the log information at the first bit of the sequencing result, namely establishing an incidence relation between each piece of log information and the log information with the maximum incidence with the log information in the second service system.

The embodiment of the invention can automatically determine the associated log information in different service systems and establish the association relationship according to the field information in the log information of different service systems, thereby quickly establishing the association relationship between the log information of different service systems. Therefore, even if other service systems are upgraded or newly added in the later period, the association relationship between the service system and the log information of the original service systems can be quickly established, and time and labor are saved. In addition, after the operation and maintenance personnel search a certain service-related log from the service system a in order to acquire the service-related log from the plurality of service systems, the operation and maintenance personnel can automatically and quickly acquire the log information related to the log in the adjacent service system of the service system a according to the association relationship.

The embodiment of the invention provides a log processing method, which comprises the steps of extracting the field name and the field value of each field in each piece of log information of a first service system and the field name and the field value of each field in each piece of log information of a second service system; then determining a common field set between the first service system and the second service system; for each piece of log information of the first service system, sequentially comparing the field value of each common field in each piece of log information with the field value of each common field in each piece of log information of the second service system, and determining the same number of the field values between each piece of log information of the first service system and each piece of log information of the second service system; and finally, aiming at each piece of log information of the first service system, sequencing each piece of log information of the second service system from high to low according to the same number of the field values, establishing an association relationship between each piece of log information and the log information at the first position of the sequencing result, automatically extracting field information in the log information of different service systems, and automatically determining the associated log information in different service systems and establishing the association relationship according to the field information in the log information of different service systems, so that the association relationship between the log information of different service systems is quickly established, and operation and maintenance personnel can automatically and quickly acquire the associated log information in different service systems according to the association relationship.

Example two

Fig. 2 is a flowchart of a log processing method according to a second embodiment of the present invention. In this embodiment of the present invention, determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system includes: when detecting that a user sets information for a common field fed back by the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system, determining a common field set between the first service system and the second service system according to the common field setting information.

As shown in fig. 2, the method of the embodiment of the present invention specifically includes:

step 201, acquiring all log information of a first service system and all log information of a second service system, and extracting a field name and a field value of each field in each log information of the first service system and a field name and a field value of each field in each log information of the second service system.

Step 202, when detecting that a user sets information for a common field fed back by a field name and a field value of each field in each piece of log information of the first service system and a field name and a field value of each field in each piece of log information of the second service system, determining a common field set between the first service system and the second service system according to the common field setting information.

Step 203, for each piece of log information of the first service system, sequentially comparing field values of each common field in each piece of log information with field values of each common field in each piece of log information of the second service system, and determining the same number of field values between each piece of log information of the first service system and each piece of log information of the second service system.

And 204, sequencing each piece of log information of the second service system from high to low according to the same number of the field values aiming at each piece of log information of the first service system, and establishing an association relation between each piece of log information and the log information at the first position of the sequencing result.

The embodiment of the invention provides a log processing method, which is characterized in that when detecting common field setting information fed back by a user according to the field name and the field value of each field in each piece of log information of a first service system and the field name and the field value of each field in each piece of log information of a second service system, a common field set between the first service system and the second service system is determined according to the common field setting information, and related log information in different service systems can be determined and an association relationship can be established according to the common field set specified by the user.

EXAMPLE III

Fig. 3 is a flowchart of a log processing method according to a third embodiment of the present invention. In this embodiment of the present invention, determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system includes: and determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system through a machine learning algorithm.

As shown in fig. 3, the method of the embodiment of the present invention specifically includes:

step 301, acquiring all log information of a first service system and all log information of a second service system, and extracting a field name and a field value of each field in each log information of the first service system and a field name and a field value of each field in each log information of the second service system.

Step 302, determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system through a machine learning algorithm.

Optionally, determining a high-radix field in the same field between the log information sets of the respective categories as a common field between the first service system and the second service system includes: and removing fields with single values of field values aiming at the same fields among the log information sets of all the categories, and reserving fields with more values of field values as common fields between the first service system and the second service system. And then storing the field names of the common fields between the first service system and the second service system into the common field set between the first service system and the second service system. In an initial state, a common field set between the first service system and the second service system is empty.

Optionally, after storing the field name of the common field between the first service system and the second service system in the common field set between the first service system and the second service system, the method further includes: taking the field name and the field value of each common field in each piece of log information of the first service system and the second service system as the characteristic variable of each piece of log information; performing PCA calculation according to the characteristic variables of each piece of log information of the first service system and the second service system, and determining the contribution degree of each common field to the main component of the log information; and sequencing the common fields according to the contribution degrees from high to low to obtain the contribution degree sequencing result of each common field in the common field set between the first service system and the second service system.

Step 303, for each piece of log information of the first service system, sequentially comparing field values of each common field in each piece of log information with field values of each common field in each piece of log information of the second service system, and determining the same number of field values between each piece of log information of the first service system and each piece of log information of the second service system.

And 304, sequencing each piece of log information of the second service system from high to low according to the same number of the field values aiming at each piece of log information of the first service system, and establishing an association relation between each piece of log information and the log information at the first position of the sequencing result.

The embodiment of the invention provides a log processing method, which is characterized in that a machine learning algorithm is used, a common field set between a first service system and a second service system is determined according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system, the machine learning algorithm can be used for finding the minimum field set jointly covered by the log information of the first service system and the second service system, the minimum field set is determined as the common field set between the first service system and the second service system, and the associated log information in different service systems is determined according to the common field set and the association relationship is established.

Example four

Fig. 4 is a schematic structural diagram of a log processing apparatus according to a fourth embodiment of the present invention. As shown in fig. 4, the apparatus includes: a field extraction module 401, a set determination module 402, a field comparison module 403, and a relationship establishment module 404.

The field extracting module 401 is configured to obtain all log information of a first service system and all log information of a second service system, and extract a field name and a field value of each field in each piece of log information of the first service system and a field name and a field value of each field in each piece of log information of the second service system; a set determining module 402, configured to determine a common field set between the first service system and the second service system according to a field name and a field value of each field in each piece of log information of the first service system, and a field name and a field value of each field in each piece of log information of the second service system; wherein the common field set includes field names of a plurality of common fields; a field comparison module 403, configured to, for each piece of log information of the first service system, sequentially compare field values of the common fields in each piece of log information with field values of the common fields in each piece of log information of the second service system, and determine that the number of field values between each piece of log information of the first service system and each piece of log information of the second service system is the same; a relationship establishing module 404, configured to rank, according to the same number of field values from high to low, each piece of log information of the second service system for each piece of log information of the first service system, and establish an association relationship between each piece of log information and the log information at the first bit of the ranking result.

The embodiment of the invention provides a log processing device, which extracts the field name and the field value of each field in each piece of log information of a first service system and the field name and the field value of each field in each piece of log information of a second service system; then determining a common field set between the first service system and the second service system; for each piece of log information of the first service system, sequentially comparing the field value of each common field in each piece of log information with the field value of each common field in each piece of log information of the second service system, and determining the same number of the field values between each piece of log information of the first service system and each piece of log information of the second service system; and finally, aiming at each piece of log information of the first service system, sequencing each piece of log information of the second service system from high to low according to the same number of the field values, establishing an association relationship between each piece of log information and the log information at the first position of the sequencing result, automatically extracting field information in the log information of different service systems, and automatically determining the associated log information in different service systems and establishing the association relationship according to the field information in the log information of different service systems, so that the association relationship between the log information of different service systems is quickly established, and operation and maintenance personnel can automatically and quickly acquire the associated log information in different service systems according to the association relationship.

In an optional implementation manner of the embodiment of the present invention, optionally, when performing an operation of determining a common field set between the first service system and the second service system according to a field name and a field value of each field in each piece of log information of the first service system and a field name and a field value of each field in each piece of log information of the second service system, the set determining module 402 is specifically configured to: when detecting that a user sets information for a common field fed back by the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system, determining a common field set between the first service system and the second service system according to the common field setting information.

In an optional implementation manner of the embodiment of the present invention, optionally, when performing an operation of determining a common field set between the first service system and the second service system according to a field name and a field value of each field in each piece of log information of the first service system and a field name and a field value of each field in each piece of log information of the second service system, the set determining module 402 is specifically configured to: and determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system through a machine learning algorithm.

In an optional implementation manner of the embodiment of the present invention, optionally, when performing an operation of determining, by a machine learning algorithm, a common field set between the first business system and the second business system according to a field name and a field value of each field in each piece of log information of the first business system and a field name and a field value of each field in each piece of log information of the second business system, the set determining module 402 is specifically configured to: clustering all log information of the first service system and all log information of the second service system according to the field name and the field value of each field in each log information through a clustering algorithm to obtain a plurality of categories of log information sets; extracting the same field among the log information sets of all categories, and determining a high-cardinality field in the same field among the log information sets of all categories as a common field between the first service system and the second service system; and storing the field name of the common field between the first service system and the second service system into the common field set between the first service system and the second service system.

In an optional implementation manner of the embodiment of the present invention, optionally, the set determining module 402 is further configured to: taking the field name and the field value of each common field in each piece of log information of the first service system and the second service system as the characteristic variable of each piece of log information; performing principal component analysis calculation according to the characteristic variables of each piece of log information of the first service system and the second service system, and determining the contribution degree of each common field to the principal component of the log information; and sequencing the common fields according to the contribution degrees from high to low to obtain the contribution degree sequencing result of each common field in the common field set between the first service system and the second service system.

In an optional implementation manner of the embodiment of the present invention, optionally, the field comparing module 403, when performing an operation of sequentially comparing field values of the common fields in each piece of log information of the first service system with field values of the common fields in each piece of log information of the second service system, and determining that the number of the field values between each piece of log information of the first service system and each piece of log information of the second service system is the same, is specifically configured to: filtering null value fields and preset non-processing fields in all log information of the first service system and the second service system; acquiring a piece of log information from all the filtered log information of the first service system as current processing log information; sequentially comparing the field value of each common field in the currently processed log information with the field value of each common field in each piece of log information of the second service system, and determining that the currently processed log information is the same as the field value of each piece of log information of the second service system in quantity; and returning to execute the operation of acquiring one piece of log information from all the filtered log information of the first service system as the current processing log information until the processing of all the filtered log information of the first service system is finished.

In an optional implementation manner of the embodiment of the present invention, optionally, when performing an operation of sequentially comparing field values of each common field in the currently processed log information with field values of each common field in each piece of log information of the second service system, and determining that the currently processed log information and each piece of log information of the second service system have the same number of field values, the field comparing module 403 is specifically configured to: and according to the contribution degree sorting result of each common field, comparing the field value of each common field in the currently processed log information with the field value of each common field in the target log information of the second service system, and determining that the number of the currently processed log information is the same as that of the field values of the target log information of the second service system.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

The log processing device can execute the log processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the log processing method.

EXAMPLE five

Fig. 5 is a schematic structural diagram of a computer device according to a fifth embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 5 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.

As shown in FIG. 5, computer device 12 is in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors 16, a memory 28, and a bus 18 connecting the various business system components (including the memory 28 and the processors 16).

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Computer device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer device 12 and includes both volatile and nonvolatile media, removable and non-removable media.

The memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.

Computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with computer device 12, and/or with any devices (e.g., network card, modem, etc.) that enable computer device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, computer device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with the other modules of computer device 12 via bus 18. It should be appreciated that although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processor 16 executes various functional applications and data processing by running the program stored in the memory 28, thereby implementing the log processing method provided by the embodiment of the present invention: acquiring all log information of a first service system and all log information of a second service system, and extracting field names and field values of fields in each piece of log information of the first service system and field names and field values of fields in each piece of log information of the second service system; determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system; wherein the common field set includes field names of a plurality of common fields; for each piece of log information of the first service system, sequentially comparing the field value of each common field in each piece of log information with the field value of each common field in each piece of log information of the second service system, and determining the same number of the field values between each piece of log information of the first service system and each piece of log information of the second service system; and aiming at each piece of log information of the first service system, sequencing each piece of log information of the second service system from high to low according to the same number of the field values, and establishing an incidence relation between each piece of log information and the log information at the first position of a sequencing result.

EXAMPLE six

A sixth embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the method for processing a log provided in the sixth embodiment of the present invention is implemented: acquiring all log information of a first service system and all log information of a second service system, and extracting field names and field values of fields in each piece of log information of the first service system and field names and field values of fields in each piece of log information of the second service system; determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system; wherein the common field set includes field names of a plurality of common fields; for each piece of log information of the first service system, sequentially comparing the field value of each common field in each piece of log information with the field value of each common field in each piece of log information of the second service system, and determining the same number of the field values between each piece of log information of the first service system and each piece of log information of the second service system; and aiming at each piece of log information of the first service system, sequencing each piece of log information of the second service system from high to low according to the same number of the field values, and establishing an incidence relation between each piece of log information and the log information at the first position of a sequencing result.

Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or computer device. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A log processing method, comprising:

2. The method of claim 1, wherein determining a common set of fields between the first business system and the second business system according to the field name and the field value of each field in each piece of log information of the first business system and the field name and the field value of each field in each piece of log information of the second business system comprises:

when detecting that a user sets information for a common field fed back by the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system, determining a common field set between the first service system and the second service system according to the common field setting information.

3. The method of claim 1, wherein determining a common set of fields between the first business system and the second business system according to the field name and the field value of each field in each piece of log information of the first business system and the field name and the field value of each field in each piece of log information of the second business system comprises:

and determining a common field set between the first service system and the second service system according to the field name and the field value of each field in each piece of log information of the first service system and the field name and the field value of each field in each piece of log information of the second service system through a machine learning algorithm.

4. The method of claim 3, wherein the determining, by a machine learning algorithm, a common set of fields between the first business system and the second business system according to the field name and the field value of each field in each piece of log information of the first business system and the field name and the field value of each field in each piece of log information of the second business system comprises:

clustering all log information of the first service system and all log information of the second service system according to the field name and the field value of each field in each log information through a clustering algorithm to obtain a plurality of categories of log information sets;

extracting the same field among the log information sets of each category, and determining a high-radix field in the same field among the log information sets of each category as a common field between the first service system and the second service system, wherein the high-radix field is a field with a plurality of dereferencing values;

and storing the field name of the common field between the first service system and the second service system into the common field set between the first service system and the second service system.

5. The method of claim 4, further comprising, after storing the field name of the common field between the first business system and the second business system in the common field set between the first business system and the second business system:

taking the field name and the field value of each common field in each piece of log information of the first service system and the second service system as the characteristic variable of each piece of log information;

performing principal component analysis calculation according to the characteristic variables of each piece of log information of the first service system and the second service system, and determining the contribution degree of each common field to the principal component of the log information;

and sequencing the common fields according to the contribution degrees from high to low to obtain the contribution degree sequencing result of each common field in the common field set between the first service system and the second service system.

6. The method of claim 5, wherein the comparing, for each piece of log information of the first service system, field values of the common fields in each piece of log information with field values of the common fields in each piece of log information of the second service system in turn, and determining that the number of field values between each piece of log information of the first service system and each piece of log information of the second service system is the same comprises:

filtering null value fields and preset non-processing fields in all log information of the first service system and the second service system;

acquiring a piece of log information from all the filtered log information of the first service system as current processing log information;

sequentially comparing the field value of each common field in the currently processed log information with the field value of each common field in each piece of log information of the second service system, and determining that the currently processed log information is the same as the field value of each piece of log information of the second service system in quantity;

and returning to execute the operation of acquiring one piece of log information from all the filtered log information of the first service system as the current processing log information until the processing of all the filtered log information of the first service system is finished.

7. The method of claim 6, wherein said sequentially comparing field values of said common fields in said currently processed log information with field values of said common fields in each piece of log information of said second business system, and determining that said currently processed log information is the same number of field values of each piece of log information of said second business system comprises:

and according to the contribution degree sorting result of each common field, comparing the field value of each common field in the currently processed log information with the field value of each common field in the target log information of the second service system, and determining that the number of the currently processed log information is the same as that of the field values of the target log information of the second service system.

8. A log processing apparatus, comprising:

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the log processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the log processing method according to any one of claims 1 to 7.