CN114741368A - Log data statistical method based on artificial intelligence and related equipment - Google Patents

Abstract

The present application proposes an artificial intelligence-based log data statistical method, device, electronic device and storage medium. The artificial intelligence-based log data statistical method includes: receiving a log data statistical request according to a search system, and verifying the statistical request; If the verification is passed, the search system searches the log data obtained from the server according to the statistical request to obtain the first target data set; divides the first target data set according to a preset threshold to obtain the second target data compressing the second target data set according to preset logic to obtain a target index data set; and performing data statistics according to the target index data set to obtain target log data. The present application can use the constructed index value to quickly perform statistics on log data on the basis of saving the storage space of log data, thereby improving the statistical efficiency of large-scale log data.

Description

Artificial intelligence-based log data statistical method and related equipment

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a method, device, electronic device and storage medium for log data statistics based on artificial intelligence.

Background technique

Elasticsearch (ES for short) is a distributed full-text search server based on Lucene's underlying technology. By improving the mechanism of data storage and filtering performance, it can achieve fast query to a certain extent.

The analysis and statistics of logs is an important basis for the work of the log system. Many log systems in the industry store logs in the Elasticsearch cluster. However, when performing statistical analysis on large-scale log data, the Elasticsearch cluster will respond slowly or report errors directly, which greatly reduces the statistical efficiency of large-scale log data.

SUMMARY OF THE INVENTION

In view of the above, it is necessary to propose an artificial intelligence-based log data statistical method and related equipment to solve the technical problem of how to improve the statistical efficiency of large-scale log data, wherein the related equipment includes an artificial intelligence-based log data statistical device , electronic equipment and storage media.

The present application provides a method for statistics of log data based on artificial intelligence, the method comprising:

Receive log data statistics requests according to the search system, and verify the statistics requests;

If the verification is passed, the search system searches the log data obtained from the server according to the statistical request to obtain the first target data set;

dividing the first target data set according to a preset threshold to obtain a second target data set;

compressing the second target data set according to preset logic to obtain a target index data set;

Data statistics are performed according to the target index data set to obtain target log data.

In this way, by classifying and storing the log data, and then compressing the log data according to the preset logic, and then constructing an index value, the log data can be quickly counted by using the constructed index value on the basis of saving the storage space of the log data, thereby improving the performance of the log data. Statistical efficiency of large-scale log data.

In some embodiments, receiving the log data statistics request according to the search system, and verifying the statistics request includes:

Set encoding labels for log data of different data types according to a preset method;

Based on the encoding tag, it is determined whether the data type in the statistics request contains a corresponding encoding tag, so as to determine whether the statistics request is qualified, and if it is qualified, the verification is passed.

In this way, it is possible to judge whether the statistics request is qualified through the set coding label, so as to ensure the accuracy of the user's statistics request and prevent the waste of system resources caused by abnormal statistics requests.

In some embodiments, if the verification is passed, the search system searches the log data obtained from the server according to the statistics request to obtain the first target data set including:

The search system collects corresponding log data based on the data type, time range and value range of the log data, and stores the collected log data in a column format as the first target data set.

In this way, the search system can quickly obtain the corresponding log data from the server according to the statistical request given by the user, so as to provide accurate data support for the subsequent process.

In some embodiments, the dividing the first target data set according to a preset threshold to obtain the second target data set includes:

Judging the data volume of the first target data set according to a preset threshold to obtain a judgment result;

Partitioning the first target data set based on the judgment result to obtain a partitioned data set;

The partition data in the partition data set is divided into batches to obtain the second target data set.

In this way, by further dividing the data in the first target data set, multiple pieces of log data in the partitioned data set can be concurrently counted in the subsequent process, thereby improving the statistical efficiency of log data.

In some embodiments, the partitioning of the first target data set based on the judgment result to obtain the partitioned data set includes:

If the data amount of the first target data set is less than a preset threshold, the first target data set is used as the partition data set;

If the data amount of the first target data set is greater than a preset threshold, the first target data set is divided in units of a preset threshold to obtain the partitioned data set.

In this way, when processing a large batch of log data, partitioning the first target data set can effectively reduce the search range of the log data in the subsequent process, and further improve the statistical efficiency.

In some embodiments, performing batch division of each partition data in the partition data set to obtain the second target data set includes:

Sorting the partition data according to the data volume of each data in the same partition from large to small to obtain a sorted data table;

Calculate the cosine similarity between adjacent data in the sorted data table according to the cosine similarity algorithm;

According to the custom clustering algorithm and the cosine similarity between adjacent data in the sorted data table, each partition data in the partition data set is divided into batches to obtain the second target data set.

In this way, the data with high similarity can be arranged together, so that the corresponding index value can be generated accordingly in the subsequent process, and the corresponding relevant log data can be quickly counted according to the index value, thereby improving the statistical efficiency.

In some embodiments, the compressing the second target data set according to the preset logic to obtain the target index data set includes:

compressing data in the second target data set according to preset logic to obtain a compressed data set;

The data in the compressed data set is transformed according to a compression algorithm to construct the target index data set.

In this way, the corresponding target index data set can be constructed on the basis of compressing the log data, thereby effectively reducing the storage space, so as to realize the rapid statistics of the log data by using the index.

The embodiment of the present application also provides an artificial intelligence-based log data statistics device, the device includes:

a verification unit, configured to receive a log data statistical request according to the search system, and verify the statistical request;

an acquiring unit, configured to search the log data acquired from the server according to the statistical request to acquire the first target data set if the verification is passed;

a dividing unit, configured to divide the first target data set according to a preset threshold to obtain a second target data set;

a compression unit, configured to compress the second target data set according to a preset logic to obtain a target index data set;

A statistics unit, configured to perform data statistics according to the target index data set to obtain target log data.

The embodiment of the present application also provides an electronic device, the electronic device includes:

a memory that stores at least one instruction;

The processor executes the instructions stored in the memory to implement the artificial intelligence-based log data statistics method.

Embodiments of the present application further provide a computer-readable storage medium, where at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the artificial intelligence-based Log data statistics method.

Description of drawings

FIG. 1 is a flowchart of a preferred embodiment of the artificial intelligence-based log data statistics method involved in the present application.

FIG. 2 is a flowchart of a preferred embodiment of dividing the first target data set according to a preset threshold to obtain a second target data set involved in the present application.

FIG. 3 is a functional block diagram of a preferred embodiment of the artificial intelligence-based log data statistics device involved in the present application.

FIG. 4 is a schematic structural diagram of an electronic device according to a preferred embodiment of the artificial intelligence-based log data statistical method involved in the present application.

FIG. 5 is a schematic structural diagram of the global dictionary table and the batch dictionary table involved in the present application.

FIG. 6 is a schematic structural diagram of a B-tree index involved in the present application.

Detailed ways

In order to more clearly understand the purpose, features and advantages of the present application, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features of the embodiments may be combined with each other unless there is conflict. Many specific details are set forth in the following description to facilitate a full understanding of the present application, and the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, features defined as "first", "second" may expressly or implicitly include one or more of said features. In the description of the present application, "plurality" means two or more, unless otherwise expressly and specifically defined.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein in the specification of the application are for the purpose of describing specific embodiments only, and are not intended to limit the application. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The embodiment of the present application provides an artificial intelligence-based log data statistics method, which can be applied to one or more electronic devices, where the electronic device is a kind of automatic numerical calculation and/or information according to pre-set or stored instructions. Processing equipment, its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), programmable gate arrays (Field-Programmable Gate Arrays, FPGAs), digital processors (Digital Signal Processors, DSPs), Embedded devices, etc.

The electronic device can be any electronic product that can interact with customers, such as personal computers, tablet computers, smart phones, personal digital assistants (PDAs), game consoles, Internet Protocol Television, IPTV), smart wearable devices, etc.

Electronic devices may also include network devices and/or client devices. Wherein, the network device includes, but is not limited to, a single network server, a server group formed by multiple network servers, or a cloud formed by a large number of hosts or network servers based on cloud computing (Cloud Computing).

The network where the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN), and the like.

As shown in FIG. 1 , it is a flow chart of a preferred embodiment of the artificial intelligence-based log data statistics method of the present application. According to different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

S10: Receive a log data statistics request according to the search system, and verify the statistics request.

In an alternative embodiment, the search system may use the ClickHouse system, which is a columnar database management system available for online analytical processing (OLAP). Among them, OLAP is the main application of the data warehouse system, which supports complex analysis operations, focuses on decision support, and provides intuitive and easy-to-understand query results.

In this optional embodiment, different from on-line transaction processing (OLTP) scenarios, such as adding a shopping cart, placing an order, and paying in an e-commerce scenario, a large number of insert, update, and delete operations need to be performed on the spot. The analysis (OLAP) scenario is usually to import data in batches, and then perform flexible exploration in any dimension, BI tool insight, and report production. After the data is written at one time, it is necessary to try to mine and analyze the data from various angles until the business value and business trends are found. This is a process that requires repeated trial and error, continuous adjustment, and continuous optimization, in which the number of data reads is much more than the number of writes, which requires the underlying database to be specially designed for this feature.

In this optional embodiment, since ClickHouse is a columnar database, unlike the MySQL database used online and locally, its query speed is very fast, and the amount of stored data is also very large. The query can return query results in seconds, and the use of clickhouse reflects the efficiency of the system. However, clickhouse does not support modifying data, so it is very suitable to store user log information, because log information is incremental data that does not need to be modified.

In an optional embodiment, receiving a log data statistics request according to the search system, and verifying the statistics request includes:

S101, according to a preset method, set coding labels for log data of different data types.

In an optional embodiment, coded labels may be set for log data of different data types according to a preset manner, and the coded labels may be numbers, symbols or letters, which are not required in this solution.

S102: Determine whether the data type in the statistical request contains a corresponding encoded label based on the encoded label, so as to determine whether the statistical request is qualified, and if qualified, the verification is passed.

In this optional embodiment, after coding labels are set for different types of log data, it can be determined based on the coding labels whether the data types in the statistics request contain corresponding coding labels, so as to determine whether the current statistics request is qualified , if it is qualified, the verification is passed, and the search system receives the statistics request; if it is unqualified, the verification fails, and the search system directly rejects the statistics request.

In this way, it is possible to judge whether the statistics request is qualified through the set coding label, so as to ensure the accuracy of the user's statistics request and prevent the waste of system resources caused by abnormal statistics requests.

S11, if the verification is passed, the search system searches the log data obtained from the server according to the statistics request to obtain a first target data set.

In this optional embodiment, the user can specify the data type, corresponding time range and data value range of the log data to be counted through the client of the search system to generate the statistics request and send it to the search system The server side, so as to preliminarily determine the overall scope of the log data to be counted and the corresponding data volume.

In this optional embodiment, after receiving the user's statistical request, the search system can, according to the data type, time range and data range specified in the user request, use ClickHouse's Kafka (open source stream processing platform) to real-time The log data of the terminal is connected to ClickHouse from kafka for columnar storage as the first target data set. In addition, ClickHouse can also store offline log data. This part of the log data stream needs to be accessed offline to ensure that ClickHouse stores the full amount of log data for N days. Usually, the default period of the system is N=15 days.

In this optional embodiment, Kafka is a distributed message system that supports partitioning and multiple copies. Its biggest feature is that it can process large amounts of data in real time, and has high throughput, low latency, and scalability. , durability, reliability, fault tolerance, high concurrency advantages to meet various demand scenarios, such as log collection, user activity tracking, streaming, etc.

In this optional embodiment, the log data may be different types of log data generated by network security devices, such as security detection logs, network traffic logs, protocol audit logs, and third-party device input logs.

In this optional embodiment, the reason for columnar storage of the obtained log data is as follows:

In the row storage mode, the data is stored continuously in rows, the data of all columns is stored in a block, and the columns that do not participate in the calculation must be read out during IO, and the read operation is severely amplified. In the column storage mode, only the columns involved in the calculation need to be read, which greatly reduces the IO cost and speeds up the query.

The data in the same column is of the same type, and the compression effect is significant. Column storage often has a compression ratio as high as ten times or even higher, which saves a lot of storage space and reduces storage costs; a higher compression ratio means a smaller data size, and it takes less time to read the corresponding data from the disk; A high compression ratio also means that the memory of the same size can store more data, and the system cache is better. Therefore, compared with row-based storage, when ClickHouse provides data query services, it is less affected by the scale of data, and has better performance in providing query services with large data volumes, which can improve query efficiency.

In this way, the search system can quickly obtain the corresponding log data from the server according to the statistical request given by the user, so as to provide accurate data support for the subsequent process.

S12: Divide the first target data set according to a preset threshold to obtain a second target data set.

Referring to FIG. 2, in an optional embodiment, dividing the first target data set according to a preset threshold to obtain a second target data set includes:

S121, judging the data amount of the first target data set according to a preset threshold to obtain a judgment result.

In this optional embodiment, the preset threshold may be set to 1T, and the judgment result is obtained by comparing the preset threshold with the data volume of the first target data set. If the amount of data is greater than the preset threshold, the judgment result is partition, and if the data amount of the first target data set is less than the preset threshold, the judgment result is no partition.

S122: Partition the first target data set based on the judgment result to obtain a partitioned data set.

In this optional embodiment, if the data amount of the first target data set is less than a preset threshold, the first target data set is used as the partition data set; if the data of the first target data set is If the amount is greater than the preset threshold, the first target data set is divided in units of the preset threshold to obtain the partitioned data set.

S123: Perform batch division on each partition data in the partition data set to obtain the second target data set.

In this optional embodiment, the process of performing batch division on each partition data in the partition data set is as follows: sorting the partition data according to the data volume of each data in the same partition from large to small to obtain the sorted data table, and calculate the cosine similarity between adjacent data in the sorted data table according to the cosine similarity algorithm, and then according to the custom clustering algorithm and the cosine similarity between adjacent data in the sorted data table The partition data in the partition data set is divided into batches to obtain the second target data set. Among them, the batch is usually used in the batch operation of the database, in order to improve the performance, for example: the batch size is 1000, that is, 1000 pieces of data are processed each time the database interacts.

In this optional embodiment, each partition data in the partition data set is divided into batches according to a custom clustering algorithm and the cosine similarity between adjacent data in the sorted data table to obtain the first The main process of the two-target dataset is:

In the log data in the same partition, take any log data that has not been accessed as the center point in turn, and expand the center point according to the preset cosine similarity threshold, where the expansion step is 1. That is, for a log data, if the cosine similarity between its adjacent log data is greater than the preset cosine similarity threshold, the log data point will be used as the center to start clustering. If the nearby log data points are smaller than the preset cosine similarity threshold Similarity threshold, mark it as a noise log data point first, and the preset cosine similarity threshold can be 0.6;

After the clustering starts, calculate the average value of the cosine similarity between the log data points adjacent to the log data point in the current cluster and all log data points in the current cluster, and calculate whether the average value is greater than the preset cosine similarity. Threshold, if it is greater than, continue to cluster around according to the same step size, and include log data points not less than the preset cosine similarity threshold condition into this cluster;

Repeat the above steps until all log data points have been accessed. At this time, each log data point is marked as belonging to a cluster or noise log data point, and all noise log data points are regarded as a cluster category. The partition data is divided into batches together with other clusters that have been obtained, that is, the data corresponding to each cluster category is regarded as a batch, and all log data after the batch division is completed is regarded as the second target data set.

Exemplarily, there are a total of 100 log data in the current partition. After custom clustering, a total of 5 clusters and 10 noise log data points are obtained. At this time, the 10 noise log data points are classified into the same category. In addition, the obtained 5 clusters have a total of 6 cluster categories, so the current partition is divided into 6 batches, and the second target data set is formed according to the batches of all the partitions after the batch division.

In this way, by further dividing the data in the first target data set, multiple pieces of log data in the partitioned data set can be concurrently counted in the subsequent process, thereby improving the statistical efficiency of log data.

S13, compress the second target data set according to a preset logic to obtain a target index data set.

In an optional embodiment, compressing the second target data set to obtain the target index data set according to preset logic includes:

S131, compress data in the second target data set according to preset logic to obtain a compressed data set.

In this optional embodiment, since the data in the second target data set are all batched data, there may be duplicate log data in each batch, and the strings corresponding to each log data are all A corresponding global ID is stored in the global dictionary table, as shown in FIG. 5 , the global dictionary corresponding to the entire second target data set and the batch dictionary corresponding to each batch.

In this optional embodiment, a batch dictionary table may be created in each batch, and the table stores the global IDs corresponding to all log data in the batch, and each global ID corresponds to a batch ID, Through this second-level dictionary table, a string corresponding to a log data can be mapped to a global ID through the global dictionary table, and then mapped to a batch ID through the batch dictionary table. The string corresponding to the real log data is no longer stored, but the batch id corresponding to the string of the log data is stored, so as to complete the compression of the second target data set, and use the compressed global dictionary table as the The compressed dataset described above. In this way, a column that stores log data strings is transformed into a column that stores 32-bit integer values, and the data space is greatly reduced.

Exemplarily, if you want to query the value actually represented by the second element in the batch dictionary 0 in Figure 5, you need to use the value 2 of this element to query the batch dictionary table to obtain its corresponding global ID as 4, and then Then use 4 to query the global dictionary table to obtain the string corresponding to 4 as "ij", and you can know that the log data corresponding to the second element in batch dictionary 0 is "ij".

S132. Convert the data in the compressed data set according to a compression algorithm to construct the target index data set.

In this optional embodiment, the compression algorithm Bit-VectorEncoding can be used to convert the obtained compressed data set, and the core idea is to convert all the values of the same column attribute in a column into two-tuples (column attribute value, The column attribute value appears in the Bitmap of the position in the column), which can be represented by using a bitmap. Through Bit-VectorEncoding, the entire column can be represented by two simple two-tuples. Using this algorithm, a column can be converted For a plurality of binary groups, the management of the column can be realized by constructing a B-tree (B-Tree) index on these binary groups.

Exemplarily, a column of log data stored in batch 1 is (1000, 2000, 2000, 1000, 1000, 2000, 1000), then after conversion by the compression algorithm Bit-Vector Encoding, the obtained binary is (1000 , 1001101) and (2000, 0110010).

In this optional embodiment, the B-Tree index is the most common index structure, and the index created by the search system by default is the B-Tree index. The B-tree index is based on the binary tree structure, and the B-tree index structure has three basic components: root node, branch node and leaf node. The root node is at the top of the index structure, the leaf node is at the bottom of the index structure, and the middle is the molecular node. The leaf node contains entries that point directly to the data rows in the table, the branch node contains entries that point to other branch nodes or leaf nodes in the index, and the root node is in a B-tree There is only one index in the index, which is actually the branch node at the top of the tree. The organizational structure of the B-tree index is similar to a tree. The main data is concentrated on the leaf nodes. The leaf nodes contain the value of the index column and the physical address corresponding to the record row. ROWID, as shown in Figure 6, can obtain the corresponding data converted by the compression algorithm through the physical address ROWID, and further obtain the corresponding data in the compressed data set. In this scheme, the constructed B-tree index is used as the target index data set.

In this way, the corresponding target index data set can be constructed on the basis of compressing the log data, thereby effectively reducing the storage space, so as to realize the rapid statistics of the log data by using the index.

S14, perform data statistics according to the target index data set to obtain target log data.

In this optional embodiment, the log data that needs to be counted can be quickly queried according to the index value in the obtained target index data set, thereby completing the count.

Exemplarily, as shown in Figure 6, the index value of the log data currently to be counted is 1019, then compare the values of 1019, 1001, and 1013 at the root node, and find that 1019 is behind 1013, thus determining that 1019 is on the right. Child node, then compare 1019 and 1013, 1017, 1021 of the right child node respectively, find that 1019 is between 1017 and 1021 (neutral node), and then find the leaf node 1019 by comparing 1019 and 1017, 1018, 1019 of the neutron node , so as to obtain the actual log data according to the corresponding physical address ROWID.

In this way, when log data statistics are performed, the log data can be quickly matched and searched according to the index in the target index data set, so as to obtain the corresponding log data.

Please refer to FIG. 3 , which is a functional block diagram of a preferred embodiment of the artificial intelligence-based log data statistics device of the present application. The artificial intelligence-based log data statistics device 11 includes a verification unit 110 , an acquisition unit 111 , a division unit 112 , a compression unit 113 , and a statistics unit 114 . The modules/units referred to in this application refer to a series of computer-readable instruction segments that can be executed by the processor 13 and can perform fixed functions, and are stored in the memory 12 . In this embodiment, the functions of each module/unit will be described in detail in subsequent embodiments.

In an optional embodiment, the verification unit 110 is configured to receive a log data statistics request according to the search system, and verify the statistics request.

In an optional embodiment, receiving the log data statistics request according to the search system, and verifying the statistics request includes:

Set encoding labels for log data of different data types according to a preset method;

Based on the encoding tag, it is determined whether the data type in the statistics request contains a corresponding encoding tag, so as to determine whether the statistics request is qualified, and if it is qualified, the verification is passed.

In an alternative embodiment, the search system may use the ClickHouse system, which is a columnar database management system available for online analytical processing (OLAP). Among them, OLAP is the main application of the data warehouse system, which supports complex analysis operations, focuses on decision support, and provides intuitive and easy-to-understand query results.

In this optional embodiment, different from on-line transaction processing (OLTP) scenarios, such as adding a shopping cart, placing an order, and paying in an e-commerce scenario, a large number of insert, update, and delete operations need to be performed on the spot. The analysis (OLAP) scenario is usually to import data in batches, and then perform flexible exploration in any dimension, BI tool insight, and report production. After the data is written at one time, it is necessary to try to mine and analyze the data from various angles until the business value and business trends are found. This is a process that requires repeated trial and error, continuous adjustment, and continuous optimization, in which the number of data reads is much more than the number of writes, which requires the underlying database to be specially designed for this feature.

In this optional embodiment, since ClickHouse is a columnar database, unlike the MySQL database used online and locally, its query speed is very fast, and the amount of stored data is also very large. The query can return query results in seconds, and the use of clickhouse reflects the efficiency of the system. However, clickhouse does not support modifying data, so it is very suitable to store user log information, because log information is incremental data that does not need to be modified.

In an optional embodiment, coded labels may be set for log data of different data types according to a preset manner, and the coded labels may be numbers, symbols or letters, which are not required in this solution.

In this optional embodiment, after coding labels are set for different types of log data, it can be determined based on the coding labels whether the data types in the statistics request contain corresponding coding labels, so as to determine whether the current statistics request is qualified , if it is qualified, the verification is passed, and the search system receives the statistics request; if it is unqualified, the verification fails, and the search system directly rejects the statistics request.

In an optional embodiment, the obtaining unit 111 is configured to, if the verification passes, the search system searches the log data obtained from the server according to the statistical request to obtain the first target data set.

In this optional embodiment, the user can specify the data type, corresponding time range and data value range of the log data to be counted through the client of the search system to generate the statistics request and send it to the search system The server side, so as to preliminarily determine the overall scope of the log data to be counted and the corresponding data volume.

In this optional embodiment, after receiving the user's statistical request, the search system can, according to the data type, time range and data range specified in the user request, use ClickHouse's Kafka (open source stream processing platform) to real-time The log data of the terminal is connected to ClickHouse from kafka for columnar storage as the first target data set. In addition, ClickHouse can also store offline log data. This part of the log data stream needs to be accessed offline to ensure that ClickHouse stores the full amount of log data for N days. Usually, the default period of the system is N=15 days.

In this optional embodiment, Kafka is a distributed message system that supports partitioning and multiple copies. Its biggest feature is that it can process a large amount of data in real time, and has high throughput, low latency, and scalability. , durability, reliability, fault tolerance, high concurrency advantages to meet various demand scenarios, such as log collection, user activity tracking, streaming, etc.

In this optional embodiment, the log data may be different types of log data generated by network security devices, such as security detection logs, network traffic logs, protocol audit logs, and third-party device input logs.

In this optional embodiment, the reason for columnar storage of the obtained log data is as follows:

In the row storage mode, the data is stored continuously in rows, the data of all columns is stored in a block, and the columns that do not participate in the calculation must be read out during IO, and the read operation is severely amplified. In the column storage mode, only the columns involved in the calculation need to be read, which greatly reduces the IO cost and speeds up the query.

The data in the same column is of the same type, and the compression effect is significant. Column storage often has a compression ratio as high as ten times or even higher, which saves a lot of storage space and reduces storage costs; a higher compression ratio means a smaller data size, and it takes less time to read the corresponding data from the disk; A high compression ratio also means that the memory of the same size can store more data, and the system cache is better. Therefore, compared with row-based storage, when ClickHouse provides data query services, it is less affected by the scale of data, and has better performance in providing query services with large data volumes, which can improve query efficiency.

In an optional embodiment, the dividing unit 112 is configured to divide the first target data set according to a preset threshold to obtain a second target data set.

In an optional embodiment, the dividing the first target data set according to a preset threshold to obtain the second target data set includes:

Judging the data volume of the first target data set according to a preset threshold to obtain a judgment result;

Partitioning the first target data set based on the judgment result to obtain a partitioned data set;

The partition data in the partition data set is divided into batches to obtain the second target data set.

In this optional embodiment, the preset threshold may be set to 1T, and the judgment result is obtained by comparing the preset threshold with the data volume of the first target data set. If the amount of data is greater than the preset threshold, the judgment result is partition, and if the data amount of the first target data set is less than the preset threshold, the judgment result is no partition.

In this optional embodiment, if the data amount of the first target data set is less than a preset threshold, the first target data set is used as the partition data set; if the data of the first target data set is If the amount is greater than the preset threshold, the first target data set is divided in units of the preset threshold to obtain the partitioned data set.

In this optional embodiment, the process of performing batch division on each partition data in the partition data set is as follows: sorting the partition data according to the data volume of each data in the same partition from large to small to obtain the sorted data table, and calculate the cosine similarity between adjacent data in the sorted data table according to the cosine similarity algorithm, and then according to the custom clustering algorithm and the cosine similarity between adjacent data in the sorted data table The partition data in the partition data set is divided into batches to obtain the second target data set. Among them, the batch is usually used in the batch operation of the database, in order to improve the performance, for example: the batch size is 1000, that is, 1000 pieces of data are processed each time the database interacts.

In this optional embodiment, each partition data in the partition data set is divided into batches according to a custom clustering algorithm and the cosine similarity between adjacent data in the sorted data table to obtain the first The main process of the two-target dataset is:

In the log data in the same partition, take any log data that has not been accessed as the center point in turn, and expand the center point according to the preset cosine similarity threshold, where the expansion step is 1. That is, for a log data, if the cosine similarity between its adjacent log data is greater than the preset cosine similarity threshold, the log data point will be used as the center to start clustering. If the nearby log data points are smaller than the preset cosine similarity threshold Similarity threshold, mark it as a noise log data point first, and the preset cosine similarity threshold can be 0.6;

After the clustering starts, calculate the average value of the cosine similarity between the log data points adjacent to the log data point in the current cluster and all log data points in the current cluster, and calculate whether the average value is greater than the preset cosine similarity. Threshold, if it is greater than, continue to cluster around according to the same step size, and include log data points not less than the preset cosine similarity threshold condition into this cluster;

Repeat the above steps until all log data points have been accessed. At this time, each log data point is marked as belonging to a cluster or noise log data point, and all noise log data points are regarded as a cluster category. The partition data is divided into batches together with other clusters that have been obtained, that is, the data corresponding to each cluster category is regarded as a batch, and all log data after the batch division is completed is regarded as the second target data set.

Exemplarily, there are a total of 100 log data in the current partition. After custom clustering, a total of 5 clusters and 10 noise log data points are obtained. At this time, the 10 noise log data points are classified into the same category. In addition, the obtained 5 clusters have a total of 6 cluster categories, so the current partition is divided into 6 batches, and the second target data set is formed according to the batches of all the partitions after the batch division.

In an optional embodiment, the compressing unit 113 is configured to compress the second target data set according to preset logic to obtain the target index data set.

In an optional embodiment, the compressing the second target data set according to the preset logic to obtain the target index data set includes:

compressing data in the second target data set according to preset logic to obtain a compressed data set;

The data in the compressed data set is transformed according to a compression algorithm to construct the target index data set.

In this optional embodiment, since the data in the second target data set are all batched data, there may be duplicate log data in each batch, and the strings corresponding to each log data are all A corresponding global ID is stored in the global dictionary table, as shown in FIG. 5 , the global dictionary corresponding to the entire second target data set and the batch dictionary corresponding to each batch.

In this optional embodiment, a batch dictionary table may be created in each batch, and the table stores the global IDs corresponding to all log data in the batch, and each global ID corresponds to a batch ID, Through this second-level dictionary table, a string corresponding to a log data can be mapped to a global ID through the global dictionary table, and then mapped to a batch ID through the batch dictionary table. The string corresponding to the real log data is no longer stored, but the batch id corresponding to the string of the log data is stored, so as to complete the compression of the second target data set, and use the compressed global dictionary table as the The compressed dataset described above. In this way, a column that stores log data strings is transformed into a column that stores 32-bit integer values, and the data space is greatly reduced.

Exemplarily, if you want to query the value actually represented by the second element in the batch dictionary 0 in Figure 5, you need to use the value 2 of this element to query the batch dictionary table to obtain its corresponding global ID as 4, and then Then use 4 to query the global dictionary table to obtain the string corresponding to 4 as "ij", and you can know that the log data corresponding to the second element in batch dictionary 0 is "ij".

In this optional embodiment, the compression algorithm Bit-VectorEncoding can be used to convert the obtained compressed data set, and the core idea is to convert all the values of the same column attribute in a column into two-tuples (column attribute value, The column attribute value appears in the Bitmap of the position in the column), which can be represented by using a bitmap. Through Bit-VectorEncoding, the entire column can be represented by two simple two-tuples. Using this algorithm, a column can be converted For a plurality of binary groups, the management of the column can be realized by constructing a B-tree (B-Tree) index on these binary groups.

Exemplarily, a column of log data stored in batch 1 is (1000, 2000, 2000, 1000, 1000, 2000, 1000), then after conversion by the compression algorithm Bit-Vector Encoding, the obtained binary is (1000 , 1001101) and (2000, 0110010).

In this optional embodiment, the B-Tree index is the most common index structure, and the index created by the search system by default is the B-Tree index. The B-tree index is based on the binary tree structure, and the B-tree index structure has three basic components: root node, branch node and leaf node. The root node is at the top of the index structure, the leaf node is at the bottom of the index structure, and the middle is the molecular node. The leaf node contains entries that point directly to the data rows in the table, the branch node contains entries that point to other branch nodes or leaf nodes in the index, and the root node is in a B-tree There is only one index in the index, which is actually the branch node at the top of the tree. The organizational structure of the B-tree index is similar to a tree. The main data is concentrated on the leaf nodes. The leaf nodes contain the value of the index column and the physical address corresponding to the record row. ROWID, as shown in Figure 6, can obtain the corresponding data converted by the compression algorithm through the physical address ROWID, and further obtain the corresponding data in the compressed data set. In this scheme, the constructed B-tree index is used as the target index data set.

In an optional embodiment, the statistics unit 114 is configured to perform data statistics according to the target index data set to obtain target log data.

In this optional embodiment, the log data that needs to be counted can be quickly queried according to the index value in the obtained target index data set, thereby completing the count.

Exemplarily, as shown in Figure 6, the index value of the log data currently to be counted is 1019, then compare the values of 1019, 1001, and 1013 at the root node, and find that 1019 is behind 1013, thus determining that 1019 is on the right. Child node, then compare 1019 and 1013, 1017, 1021 of the right child node respectively, find that 1019 is between 1017 and 1021 (neutral node), and then find the leaf node 1019 by comparing 1019 and 1017, 1018, 1019 of the neutron node , so as to obtain the actual log data according to the corresponding physical address ROWID.

It can be seen from the above technical solutions that the present application can classify and store log data, and then compress the log data according to a preset logic to build an index value, so that the built index can be used on the basis of saving the storage space of the log data. The value of the log data can be quickly counted to improve the statistical efficiency of large-scale log data.

Please refer to FIG. 4 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 1 includes a memory 12 and a processor 13 . The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement the artificial intelligence-based log data statistics method described in any of the foregoing embodiments.

In an optional embodiment, the electronic device 1 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13, such as a log data statistics program based on artificial intelligence.

FIG. 4 only shows the electronic device 1 having the memory 12 and the processor 13. Those skilled in the art can understand that the structure shown in FIG. 4 does not constitute a limitation on the electronic device 1, and may include less Or more components, or a combination of certain components, or a different arrangement of components.

1, the memory 12 in the electronic device 1 stores a plurality of computer-readable instructions to implement an artificial intelligence-based log data statistics method, and the processor 13 can execute the plurality of instructions to implement:

Receive log data statistics requests according to the search system, and verify the statistics requests;

If the verification is passed, the search system searches the log data obtained from the server according to the statistical request to obtain the first target data set;

dividing the first target data set according to a preset threshold to obtain a second target data set;

compressing the second target data set according to preset logic to obtain a target index data set;

Data statistics are performed according to the target index data set to obtain target log data.

Specifically, for the specific implementation method of the above-mentioned instruction by the processor 13, reference may be made to the description of the relevant steps in the embodiment corresponding to FIG. 1 , which is not repeated here.

Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1, and does not constitute a limitation on the electronic device 1. The electronic device 1 may be a bus-type structure or a star-shaped structure, and the electronic device 1 may also include More or less other hardware or software than shown, or different component arrangements, for example, the electronic device 1 may also include input and output devices, network access devices, and the like.

It should be noted that the electronic device 1 is only an example, and other existing or future electronic products that can be adapted to the present application should also be included within the protection scope of the present application, and are incorporated herein by reference.

The memory 12 includes at least one type of readable storage medium, and the readable storage medium may be non-volatile or volatile. The readable storage medium includes flash memory, removable hard disk, multimedia card, card-type memory (eg: SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, and the like. The memory 12 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1 . In other embodiments, the memory 12 may also be an external storage device of the electronic device 1, such as a pluggable mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) equipped on the electronic device 1 ) card, flash memory card (Flash Card) and so on. The memory 12 can not only be used to store application software installed in the electronic device 1 and various types of data, such as the code of a log data statistics program based on artificial intelligence, etc., but also can be used to temporarily store data that has been output or will be output.

The processor 13 may be composed of integrated circuits in some embodiments, for example, may be composed of a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same function or different functions, including one or more central processing units. CPU (Central Processing Unit, CPU), microprocessor, digital processing chip, graphics processor and combination of various control chips, etc. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect the various components of the entire electronic device 1, by running or executing the programs or modules stored in the memory 12 (for example, executing manual-based operations). Intelligent log data statistics program, etc.), and call the data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

The processor 13 executes the operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application program to implement the steps in each of the above-mentioned embodiments of the artificial intelligence-based log data statistics method, for example, the steps shown in FIG. 1 to FIG. 2 .

Exemplarily, the computer program may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 12 and executed by the processor 13 to complete the present invention. Application. The one or more modules/units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 1 . For example, the computer program may be divided into a verification unit 110 , an acquisition unit 111 , a division unit 112 , a compression unit 113 , and a statistics unit 114 .

The above-mentioned integrated units implemented in the form of software functional modules may be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, and include several instructions to enable a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor (processor) to execute the various embodiments of the present application. Part of artificial intelligence-based log data statistical methods.

If the modules/units integrated in the electronic device 1 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, this application can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing relevant hardware devices through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented.

Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory) , random access memory and other memories.

Further, the computer-readable storage medium may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required by at least one function, and the like; Use the created data, etc.

The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

The bus may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (extended industry standard architecture, EISA for short) bus, or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one arrow is shown in FIG. 4, but it does not mean that there is only one bus or one type of bus. The bus is arranged to enable connection communication between the memory 12 and at least one processor 13 and the like.

Embodiments of the present application further provide a computer-readable storage medium (not shown), where computer-readable instructions are stored in the computer-readable storage medium, and the computer-readable instructions are executed by a processor in an electronic device to implement any of the foregoing implementations The artificial intelligence-based log data statistics method described in the example.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. A plurality of units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than limitations. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present application.

Claims (10)

1. An artificial intelligence based log data statistical method, characterized in that the method comprises:

receiving a log data statistical request according to a search system, and verifying the statistical request;

if the verification is passed, the search system searches the log data acquired from the server according to the statistical request to acquire a first target data set;

dividing the first target data set according to a preset threshold value to obtain a second target data set;

compressing the second target data set according to preset logic to obtain a target index data set;

and carrying out data statistics according to the target index data set to obtain target log data.

2. The artificial intelligence based log data statistical method of claim 1, wherein receiving a log data statistical request pursuant to a search system and verifying the statistical request comprises:

setting encoding labels for log data of different data types according to a preset mode;

and judging whether the data type in the statistical request contains a corresponding encoding label or not based on the encoding label so as to determine whether the statistical request is qualified or not, and if so, passing the verification.

3. The artificial intelligence based log data statistical method of claim 1, wherein if the verification is passed, the search system searches the log data obtained from the server according to the statistical request to obtain the first target data set comprises:

the search system collects corresponding log data based on the data type, the time range and the value range of the log data, and stores the collected log data in a column mode to serve as the first target data set.

4. The artificial intelligence based log data statistics method of claim 1 wherein said partitioning the first target data set to obtain a second target data set in accordance with a preset threshold comprises:

judging the data volume of the first target data set according to a preset threshold value to obtain a judgment result;

partitioning the first target data set based on the judgment result to obtain a partitioned data set;

and carrying out batch division on each partition data in the partition data set to obtain the second target data set.

5. The artificial intelligence based log data statistics method of claim 4 wherein said partitioning the first target dataset based on the determination to obtain a partitioned dataset comprises:

if the data volume of the first target data set is smaller than a preset threshold value, taking the first target data set as the partition data set;

if the data volume of the first target data set is larger than a preset threshold, dividing the first target data set by taking the preset threshold as a unit to obtain the partitioned data set.

6. The artificial intelligence based log data statistics method of claim 5 wherein the batch partitioning of each of the partitioned data sets to obtain the second target data set comprises:

sorting the data of the partitions from large to small according to the data quantity of the data in the same partition to obtain a sorting data table;

calculating cosine similarity between every two adjacent data in the sorting data table according to a cosine similarity algorithm;

and dividing each partition data in the partition data set in batches according to a user-defined clustering algorithm and cosine similarity between each adjacent data in the sorting data table to obtain the second target data set.

7. The artificial intelligence based log data statistics method of claim 1 wherein the compressing the second target data set according to a predetermined logic to obtain a target index data set comprises:

compressing the data in the second target data set according to a preset logic to obtain a compressed data set;

and converting the data in the compressed data set according to a compression algorithm to construct the target index data set.

8. An artificial intelligence based log data statistics apparatus, characterized in that the apparatus comprises:

the verification unit is used for receiving a log data statistical request according to a search system and verifying the statistical request;

the acquisition unit is used for searching the log data acquired from the server by the search system according to the statistical request to acquire a first target data set if the verification is passed;

the dividing unit is used for dividing the first target data set according to a preset threshold value to obtain a second target data set;

the compression unit is used for compressing the second target data set according to preset logic to obtain a target index data set;

and the statistical unit is used for carrying out data statistics according to the target index data set so as to obtain target log data.

9. An electronic device, characterized in that the electronic device comprises:

a memory storing computer readable instructions; and

a processor executing computer readable instructions stored in the memory to implement the artificial intelligence based log data statistics method of any of claims 1 to 7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, which when executed by a processor implement the artificial intelligence based log data statistics method of any of claims 1-7.

Images