CN114595129A - Configurable multi-dimensional data monitoring method and device and storage medium - Google Patents

Abstract

The invention discloses a configurable multi-dimensional data monitoring method, a configurable multi-dimensional data monitoring device and a storage medium, wherein the method comprises the following steps: configuring a data field to obtain a configuration field; the Flink filters the first service data in the log server according to the configuration field to obtain second service data; storing the second service data into a database; configuring the monitoring index to obtain a configuration index; and querying the database according to the configuration index to obtain third service data, and performing index display on the third service data. According to the embodiment of the invention, the data of the log server is recorded into the system, and the X-axis, the Y-axis and the function configuration are supported, so that the SQL writing of the log server is avoided, and the configuration threshold is effectively reduced; and a multi-field multi-condition legend is adopted, so that multi-dimensional index configuration is realized, and service buried points can be flexibly assembled into the same index graph.

Description

A configurable multi-dimensional data monitoring method, device and storage medium

technical field

The present invention relates to the field of computer technology, and in particular, to a configurable multi-dimensional data monitoring method, device and storage medium.

Background technique

Large-scale APPs often carry many services. Taking shopping APPs as an example, there are six services such as community, transaction, and live broadcast. With the iteration of APP versions, the business is also developing rapidly. Real-time access to the APP's business monitoring can monitor the operation of the current business. Quality and subsequent business decisions have important benefits.

Business monitoring is often used to monitor the use of a function in a business. The traditional method is to report each function, and then configure the data source and display charts. The typical solution in this regard is ElasticSearch+Grafana. Using ElasticSearch as the storage and search service, and Grafana as the indicator display system, this solution is well recognized by major companies, but it also has technical barriers in actual use. The infrastructure requires regular maintenance by the team, and indicators also require professional R&D personnel. If you are not familiar with the embedded code, you cannot configure it. This limitation prevents the product manager from intervening in the indicator configuration.

An important user of business tracking is the product manager. A general-purpose, configurable, multi-dimensional indicator system for product managers can reduce the complexity of tracking configuration and improve configuration efficiency, and at the same time control the cost of tracking.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a configurable multi-dimensional data monitoring method, device and storage medium, aiming to solve the problems of complex configuration and low configuration efficiency of the index monitoring system in the prior art.

In order to achieve the above object, the present invention provides a configurable multi-dimensional data monitoring method, the method includes the following steps:

Configure the data field to get the configuration field;

Flink filters the first business data in the log server according to the configuration field to obtain second business data; stores the second business data in the database;

Configure monitoring indicators to get configuration indicators;

The database is queried according to the configuration index to obtain third service data, and an index display is performed on the third service data.

Optionally, obtaining the configuration field from the configuration data field includes the following steps:

Set business ID and/or functional ID and/or base fields and/or custom fields;

establishing an association relationship between the service identifier and the database, and establishing an association relationship between N (N is greater than or equal to 1) the service identifier and one of the databases;

Establish an association relationship between the function identifier and the database table, and establish an association relationship between N (N is greater than or equal to 1) of the function identifier and one of the database tables;

An association relationship between the basic field and/or the custom field and the field of the database table is established.

Optionally, the method further includes:

Calculate the number of records corresponding to the function identifier;

Judging the first proportion of the number of records in the total records of the database table corresponding to the function identifier and/or the second proportion within a specified time range;

The first proportion is greater than the first threshold and/or the second proportion is greater than the second threshold, the second business data is sampled to obtain fourth business data, and then the fourth business data is stored in in the database.

Optionally, obtaining the configuration index from the configuration monitoring index includes the following steps:

The monitoring indicators include at least one of the following: chart type, function indicators, X-axis indicators, Y-axis indicators, legend indicators, query conditions, and filter conditions;

The function index includes N (N is greater than or equal to 1) the function identifiers;

The X-axis indicator includes time units and/or N (N greater than or equal to 1) basic fields and/or N (N greater than or equal to 1) custom fields;

The Y-axis indicator includes N (N greater than or equal to 1) basic fields and/or N (N greater than or equal to 1) custom fields;

The filter conditions are used to limit the data scale and filter useless data;

The query condition is used for the query of the legend indicator;

The legend index performs data grouping according to the function index, and obtains a plurality of index curves.

Optionally, the method further includes the following steps:

The X-axis indicator further includes a first function, and the first function includes at least one of the following: a mean value function and an accumulation function;

The Y-axis indicator supports numerical and proportional types. When the Y-axis is proportional, configure the numerator and denominator; the numerator is the basic field and/or a custom field, and the denominator is the basic field. fields and/or custom fields;

The Y-axis indicator further includes a second function, and the second function includes at least one of the following: a mean value function and an accumulation function.

Optionally, the method further includes the following steps:

The third service data is cached, and the cache duration is configurable.

Optionally, the legend indicators perform data grouping according to the function indicators, including the following steps:

configuring the grouping conditions corresponding to the basic field and/or the custom field;

The data of the basic field and/or the custom field is grouped according to the grouping condition.

In addition, in order to achieve the above object, the present invention also provides a configurable multi-dimensional data monitoring device, the device comprising:

The field configuration unit is used to configure the data field to obtain the configuration field;

a data filtering unit, used by Flink to filter the first business data in the log server according to the configuration field to obtain second business data; and store the second business data in the database;

The indicator configuration unit is used to configure monitoring indicators and obtain configuration indicators;

An indicator query unit, configured to query the database according to the configuration indicator to obtain third business data, and display indicators for the third business data.

In addition, in order to achieve the above object, the present invention also provides an electronic device, the electronic device includes: a memory, a processor, and a configurable multi-dimensional data monitoring device stored on the memory and running on the processor A program, the configurable multi-dimensional data monitoring program configured to implement the steps of the configurable multi-dimensional data monitoring method as described above.

In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the configurable multi-dimensional data monitoring method as described above is implemented. step.

Through the embodiment of the present invention, the data of the log server is recorded in the system, and the configuration of the X axis, the Y axis and the function configuration is supported, which avoids the SQL writing of the log server, and effectively reduces the configuration threshold; In this way, multi-dimensional indicator configuration can be realized, and business buried points can be flexibly assembled into the same indicator graph.

Description of drawings

FIG. 1 is a schematic flowchart of a configurable multi-dimensional data monitoring method provided by the present invention.

FIG. 2 is a structural diagram of a configurable multi-dimensional data monitoring system provided by the present invention.

FIG. 3 is a schematic flowchart of a configuration data storage rule provided by the present invention.

FIG. 4 is a system structure diagram of service data storage provided by the present invention.

FIG. 5 is a schematic flowchart of monitoring record sampling provided by the present invention.

FIG. 6 is a schematic flowchart of a monitoring data packet provided by the present invention.

FIG. 7 is a structural block diagram of an embodiment of a configurable multi-dimensional data monitoring apparatus provided by the present invention.

FIG. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer and more comprehensible, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention, but not for limiting the present invention.

In the following description, suffixes such as 'module', 'component' or 'unit' used to represent elements are used only to facilitate the description of the present invention and have no specific meaning per se. Thus, "module", "component" or "unit" may be used interchangeably.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence.

In one embodiment, as shown in FIG. 1 , the present invention provides a configurable multi-dimensional data monitoring method, the method comprising:

Step 101: Configure the data field to obtain the configuration field.

Large-scale APPs often carry many services. Taking shopping APPs as an example, there are six services such as community, transaction, and live broadcast. With the iteration of APP versions, the business is also developing rapidly. Real-time access to the APP's business monitoring can monitor the operation of the current business. Quality and subsequent business decisions have important benefits.

Business monitoring is often used to monitor the use of a function in a business. The traditional method is to report each function, and then configure the data source and display charts. The typical solution in this regard is ElasticSearch+Grafana. Using ElasticSearch as the storage and search service, and Grafana as the indicator display system, this solution is well recognized by major companies, but it also has technical barriers in actual use. The infrastructure requires regular maintenance by the team, and indicators also require professional R&D personnel. If you are not familiar with the embedded code, you cannot configure it. This limitation prevents the product manager from intervening in the indicator configuration.

An important user of business tracking is the product manager. A general-purpose, configurable, multi-dimensional indicator system for product managers can reduce the complexity of tracking configuration and improve configuration efficiency, and at the same time control the cost of tracking.

The configurable multi-dimensional data monitoring system is shown in Figure 2. Including configuration layer, storage layer, and indicator panel. The configuration layer supports the APP to flexibly define the buried point fields. After the log records are generated by the SDK in the APP, they are reported to Alibaba Cloud SLS for caching, and then the data records are consumed from the Alibaba Cloud SLS with Flink and entered into the ClickHouse table corresponding to the records. data storage.

When the Flink cluster processes the data in the Alibaba Cloud SLS server, it needs to be processed according to the fields configured in the configuration layer. Therefore, the fields that need to be saved in the journal record need to be configured first. To configure the fields to be saved, see the process shown in Figure 3.

Step 201 , set a service identifier and/or a function identifier and/or a basic field and/or a custom field.

APP data indicators, from a business perspective, are divided into multiple business lines, such as community, transaction, live broadcast and other business lines. The data indicators of each business line need to be divided and identified by business identifiers in log records. The business identifier of each business line is unique, and the data of those business lines needs to be configured and saved during multi-dimensional data monitoring. For example, the business line includes: community, transaction, live broadcast, and the business identifier is shown in the following table:

Business Line business identity Community Community_id trade Trade_id live streaming Broadcast_id

During multi-dimensional data monitoring, if you only need to monitor transaction-related indicators, you only need to configure the business ID of the transaction business line into the multi-dimensional data monitoring system.

APP data indicators, from the dimension of use, APP data indicators are mainly divided into two categories, one is statistical data indicators, such as PV, UV, etc., only to calculate the user has used a certain function, such as clicking a certain button Or have entered a certain page, the content of this kind of buried point is simple, just need to generate a record every time the user uses it, which is mainly used to display the number of times the user has used a certain function and the number of users; one is Quality buried points. This type of buried points mainly records the performance and abnormal data during the user's use. The data content is more complex and needs to include abnormal stacks, abnormal versions, models, and networks. Different environments have different effects on the quality of the APP. Influence, the fields of such buried points are relatively flexible, and different business scenarios are different, and it is necessary to provide sufficient support for quality iteration.

After collecting user's operation records through the SDK, the APP will generate log files and report them to the Alibaba Cloud SLS server. Each log record has a unique identifier to distinguish the log records; at the same time, each log record contains corresponding basic fields and custom fields. Basic fields include: UUID, userID, app version number, device type, os type and other fields; custom fields can be customized according to each type of diary record, such as custom fields that record the log records of users browsing products include: log Types, product detail pages, favorites, purchases. When analyzing log files, the system can set corresponding basic fields and custom fields according to business requirements, and this technical solution does not impose specific restrictions.

Operation and maintenance personnel can dynamically modify business IDs, function IDs, basic fields, and custom fields in the fields to be saved as needed. If you need to analyze new buried point data (that is, new log records), you need to set the corresponding identifiers according to the new log records; you can also reset the basic fields and autologous fields for the log records that have been analyzed (that is, configured identifiers). Define fields. If you need to analyze a user's operation behavior, you can add corresponding fields. The added fields require corresponding records in the buried point data.

The underlying fields are shown in the following table:

When the above fields do not meet the business needs, the business adds custom fields to record more content: for example:

path: the path of the http network log, only exists in the log of the http service type, the type is String

url: URL of http web log, type is String

tab: the page tag of the log of AB Test, the type is String

duration: the duration field of the time-consuming log, the type is long

detail: the content field of the exception log

Stack: stack field of exception log

Step 202: Establish an association relationship between the service identifiers and the database, and establish an association relationship between N (N greater than or equal to 1) the service identifiers and one of the databases.

Diary records related to each line of business are kept in a separate database, such as database A in the ClickHouse database system. It is also possible to save the related data of multiple business lines in one database, for example, the data of the two business lines of transaction and live broadcast are all stored in database A. Configure the relationship between the line-of-business data that needs to be monitored and the stored database, as shown in the following table:

Business Line business identity database Community Community_id database B trade Trade_id Database A live streaming Broadcast_id Database A

Step 203: Establish an association relationship between the function identifiers and a database table, and establish an association relationship between N (N greater than or equal to 1) of the function identifiers and one of the database tables.

The present invention uses a structured data table to generate a general-purpose multi-dimensional table, and performs association configuration between the buried point objects and the storage table, thereby supporting the processing of field differences of different business buried points, and associating N buried point objects into one Structure table, one-to-one correspondence between the buried point fields and the fields of the structure table, thus completing the structural transformation of the buried point definition.

The log data usually does not change, and ClickHouse is used as the storage database to maintain a good query speed in the case of supporting large-scale storage.

The only mark of the buried point is called Section. The relationship between the buried point Section and the ClickHouse table is n:1. The relationship between the buried point and the ClickHouse table is managed by manual configuration, and the n buried points associated with the ClickHouse table will be entered. Library to this ClickHouse table, and then perform SQL query on a single table, and then provide three-dimensional (horizontal axis, vertical axis, polyline name) query results to the front end, thereby supporting multi-dimensional data indicators.

For example, if the log records corresponding to the three buried points A, B, and C are stored in the Table_ID_001 table of ClickHouse data, then the identifiers of the log records corresponding to the three buried points and the table names are associated and configured, such as the configuration as shown in the following table shown:

journal entry name Feature ID table name Buried point A Commodity Transaction Journal A Table_ID_001 Buried point B Commodity Transaction Journal B Table_ID_001 Buried point C Commodity Transaction Log C Table_ID_001

Step 204: Establish an association relationship between the basic field and/or the custom field and the field of the database table.

The present invention needs to manually create a ClickHouse table, the table contains all the basic fields, one-to-one correspondence with the fields in the buried point record, and also leaves 10 String fields, 5 int fields, and 5 floating point fields in the ClickHouse table. And three time fields, these fields are simply named str1, str2, str3...int1, int2, int3 these business-independent names, and then the ClickHouse table is entered into the indicator system of the present invention, for business fields, section fields and basic Fields are automatically mapped by default. For custom fields, they are configured using the system configuration scheme.

After associating N buried point objects into a structure table, one-to-one correspondence is made between the buried point fields and the fields of the structure table, thus completing the structural transformation of the buried point definition. For example, buried points A and B contain basic fields and multiple custom fields, and then map these fields to the database table fields one by one. Business fields, section fields, and basic fields correspond one-to-one with ClickHouse's database table fields, and custom fields are configured with reserved fields in the database table, as shown in the following table:

The fields of the buried log records need to correspond one-to-one with the corresponding database table fields, and then flink needs to store the field data in the log records into the corresponding field relationships of the database table one by one according to the corresponding relationship.

After configuring the data fields that need to be saved in the log records of buried points, the association between the data fields to be saved and the database (that is, the configuration field rules) needs to be delivered to the Flink cluster. The data fields that need to be saved in the present invention are configured through the WEB management terminal, and are saved in a database table, such as a section configuration table, after configuration, and then the Flink cluster periodically obtains data storage rules from the section configuration table.

After the Flink cluster obtains the relationship between the data fields to be saved and the database from the section configuration table, it is saved locally. If there is a local relationship between the corresponding data field to be saved and the database, such as the relationship between the data field to be saved and the database of the log record of buried point A, use the latest data field and database to be saved. The association relationship updates the locally saved data fields that need to be saved and the database association relationship.

Step 102: Flink filters the first service data in the log server according to the configuration field to obtain second service data; and stores the second service data in a database.

The present invention adopts Alibaba Cloud SLS as the buried log record data cache of the APP. The storage cost of SLS is not high, it can support high concurrency, and can make up for the problem of ClickHouse concurrency limitation, and the data analysis and graph configuration of SLS need to create an index, The cost of indexing is relatively high, and the present invention adopts the lower cost ClickHouse for storage.

Flink connects to the corresponding Logstore for data consumption, obtains the Section configuration (that is, the relationship between the data fields to be saved and the database) from the Section configuration table every 5 seconds, and filters according to the business line ID and Section field to reduce invalid data. The fields and associated tables configured by Section, dynamically generate warehousing statements, and splicing the data into the warehousing. The structure of the data storage system is shown in Figure 4.

Each table that stores data in SLS is called LogStore. Flink tasks use Source to consume data from SLS to obtain data, use Sink to store data, and enter data into ClickHouse. After the source obtains the log record, according to the business identifier and function identifier of the log record, it searches the association between the data fields that need to be saved and the database saved locally to find out whether the corresponding business identifier and function identifier are configured. If the corresponding identifier is not configured , the log record is discarded; if the corresponding identifier is configured, the log record data is obtained for subsequent processing. If the business identifier corresponding to buried point A is a transaction business line, and the function identifier is commodity transaction log A, these identifiers have been configured in the data field to be saved and the relationship between the database, then Source obtains the log record data corresponding to buried point A, such as The acquired log record data is log record_buried point A.

According to the business ID (such as Trade_id) and function ID (such as commodity transaction log A) of the obtained log records, obtain the database corresponding to the business ID, the basic fields and custom fields corresponding to the function ID, such as obtaining the function ID of the commodity transaction log A. field, and then obtain the data content of the field corresponding to the ID from the obtained log record.

The log record data may contain other data than the fields corresponding to the identifiers. Flink will only obtain the data of the fields corresponding to the function identifiers, and Flink will not obtain other data.

Flink obtains the corresponding ClickHouse database table according to the ID of the log record. For example, the database table corresponding to commodity transaction log A is Table_ID_001. Then, according to the field of the log record corresponding to the identifier of the log record, the corresponding database table field in the database table Table_ID_001 is obtained.

Flink constructs an SQL statement based on the data content of the log record field, the database table corresponding to the function identifier, and the corresponding database table field. Then execute the SQL statement to save the data content corresponding to the field of the log record to the corresponding database table, such as the table Table_ID_001.

When saving log records to the database, to prevent too many log records from being stored in the database, it is necessary to judge according to the number of records corresponding to each function identifier in the current database table. If the number of records exceeds a custom, the log records need to be sampled. For the specific process, see the process shown in Figure 5:

Step S301: Calculate the number of records corresponding to the function identifier.

The ClickHouse database table uses the function field (section field) of the log record as the index of the table, and will calculate the number of records occupied by each section in real time. For example, if the function identifier is the number of records of the commodity transaction log A, the number of records obtained is 1500.

Step S302: Determine the first proportion of the number of records in the total records of the database table corresponding to the function identifier and/or the second proportion within a specified time range.

Determine the proportion of the number of records corresponding to a function identifier in the entire database table. For example, the total number of records in the database table Table_ID_001 corresponding to the commodity transaction log A is 15,000, then the records of the function identifier and the total records of the database table Table_ID_001 The proportion of the number of bars is 10%.

It is also possible to determine the ratio of the number of records of the functional identifier in a certain period of time to the number of records of the database table in this period of time. The total number of records in Table_ID_001 is 5000, and the ratio of the records of this function identifier to the records of database table Table_ID_001 within the specified time range is 4%.

Step S303, the first proportion is greater than the first threshold and/or the second proportion is greater than the second threshold, the second service data is sampled to obtain fourth service data, and then the fourth service Data is stored in the database.

The judgment function identifies whether the ratio of the records of the commodity transaction log A to the total number of records of the database table Table_ID_001 is greater than a threshold, such as 5%. If it is greater than this threshold, it means that there are many log records corresponding to the function identifier, and the log records corresponding to the function identifier need to be sampled.

It can also be judged whether the ratio of the records of the functionally identified commodity transaction log A to the number of records of the database table Table_ID_001 within a certain time range (eg 24 hours) is greater than a threshold, such as 5%. If it is less than the threshold, it is not necessary to sample the diary record corresponding to the function identifier.

The log record sampling can be sampled by the timestamp field. For example, if timestamp modulo 1 equals 0, all data is taken; if timestamp modulo 2=0, 50% of the data is taken, and the modulo is configurable. The sampling may also be performed in other manners, which are not limited in this technical solution.

Step 103: Configure monitoring indicators to obtain configuration indicators.

After the configuration of the fields required for logging is completed, the technical solution needs to configure the monitoring indicators to obtain the configuration indicators. Ability to configure chart types, functional indicators, X-axis indicators, Y-axis indicators, legend indicators, query conditions, filter conditions, etc.

Chart types include line charts, bar charts, etc.; each configuration indicator includes corresponding functional indicators, and one configuration indicator can contain multiple functional indicators. Function indicators and function identifiers are associated. When configuring a function indicator, you can select a function identifier through a drop-down list, or you can obtain a function identifier through a query. For example, put the fluency index under the live wifi condition and the fluency index under the 4G condition in the same index table and display them with different curves.

The X-axis indicator is generally a time unit, or it can be a custom field. For example, in a bar chart, the X-axis can represent various functions. The custom field of the X-axis is selected from the base field and custom field corresponding to the function indicator corresponding to the function indicator. When a custom field is selected for the X-axis, a function wrapping the field can be set for each selected custom field, such as a function for accumulating the field and a function for the mean value. After configuration, it is added to the data merge in the way of group by. If multiple custom fields are configured, you can also configure the sorting method, such as sorting from small to large.

The Y-axis indicator supports numerical and proportional types, and the proportional type supports the configuration of numerator and denominator. When Y is a numeric type, the Y-axis indicator is selected from the basic field and the custom field corresponding to the function identifier corresponding to the function indicator; a function wrapping the field is set for each selected custom field. A function of accumulation, a function of mean.

When the Y-axis indicator is proportional, it is necessary to configure corresponding fields for the numerator, and select from the basic fields and custom fields corresponding to the function identifier corresponding to the function indicator; configure the corresponding field for the denominator, from the function indicator corresponding to the function indicator. to choose between the base field and the custom field. For the fields configured in the numerator and denominator, a function that wraps the field can be set, such as a function for accumulating the field and a function for the mean value.

The legend indicator is to group the data corresponding to the section (function identifier) to obtain multiple indicator curves. The grouping can be configured separately under what conditions a field becomes a group, and the present invention can configure various conditions for multiple fields. , so as to achieve multi-dimensional indicators. For the grouping process, see the process shown in Figure 6:

Step 401: Configure grouping conditions corresponding to the basic field and/or the custom field.

Set the grouping conditions for the fields of the Y-axis indicator configuration, such as setting the grouping conditions for the APP version number, group those with version numbers less than 2.0 into one group, those with version numbers from 2.1 to 3.0 into one group, and those with version numbers above 3.1 into one group.

Step 402: Group the data of the basic field and/or the custom field according to the grouping condition.

According to the grouping condition, the data of the corresponding field is grouped. For example, the fluency indicator in the case of live wifi is grouped according to the APP version number, and the data belonging to the corresponding APP version number is added to the corresponding group. After grouping, the corresponding groups are represented by different charts. For example, the fluency indicator in the case of live wifi is divided into 3 groups according to the APP version number, and 3 broken lines are used to represent the data of the corresponding group.

The query conditions will be brought into the indicator chart, and the filter conditions are used to limit the data size and filter out useless data. The above configuration can cover most of the indicator configuration scenarios, without writing SQL, and click to complete the configuration.

The query condition can be the data of a certain APP version, such as the data of the APP version number 1.0. The filter condition can be filtered by a certain device type, such as the data reported by the iphone8 device.

Step 104: Query the database according to the configuration index to obtain third service data, and display the index of the third service data.

After the monitoring indicators are configured, the indicator monitoring system queries the corresponding data from the ClickHouse database according to the configuration indicators, such as querying the Y-axis indicator data and querying the X-axis indicator data. Then query whether the corresponding function is configured in the field in the configuration indicator. If the corresponding function is configured, the corresponding function is called to process the field. If the accumulation function is configured, the data of the field is accumulated.

If the Y-axis indicator is proportional, obtain the indicator data corresponding to the numerator, and then call the corresponding function for processing; obtain the indicator data corresponding to the denominator, then call the corresponding function for processing, and then divide the numerator by the denominator to get the Y-axis index.

If the Y-axis indicator is configured with the fluency index in the case of live wifi and the fluency index in 4G, then obtain the fluency index in the log record of the current moment in the case of live wifi and the fluency index in 4G, and then By averaging these indicators, the fluency index in the case of live wifi and the fluency index in the case of 4G are obtained at that moment. Then display the Y-axis indicator according to the chart type (such as a line chart).

When displaying monitoring indicators, an intermediate cache can be added. For example, the cache is updated every 3 minutes, and all default global queries are cached, thereby reducing the frequency of directly querying the ClickHouse database and maintaining system performance. The cache duration can be configured, such as 5 minutes.

Through the embodiment of the present invention, the data of the log server is recorded in the system, and the configuration of the X axis, the Y axis and the function configuration is supported, which avoids the SQL writing of the log server, and effectively reduces the configuration threshold; In this way, multi-dimensional indicator configuration can be realized, and business buried points can be flexibly assembled into the same indicator graph.

In addition, an embodiment of the present invention also provides a configurable multi-dimensional data monitoring device. Referring to FIG. 7 , the configurable multi-dimensional data monitoring device includes:

The field configuration unit 10 is used to configure the data field to obtain the configuration field;

The data filtering unit 20 is used for Flink to filter the first service data in the log server according to the configuration field to obtain the second service data; and store the second service data in the database;

an indicator configuration unit 30, configured to configure monitoring indicators to obtain configuration indicators;

The indicator query unit 40 is configured to query the database according to the configuration indicator to obtain third business data, and display indicators for the third business data.

Through the embodiment of the present invention, the data of the log server is recorded in the system, and the configuration of the X axis, the Y axis and the function configuration is supported, which avoids the SQL writing of the log server, and effectively reduces the configuration threshold; In this way, multi-dimensional indicator configuration can be realized, and business buried points can be flexibly assembled into the same indicator graph.

It should be noted that, each unit in the above-mentioned apparatus may be used to implement each step in the above-mentioned method, and at the same time achieve corresponding technical effects, which will not be repeated in this embodiment.

Referring to FIG. 8 , FIG. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

As shown in FIG. 8 , the electronic device may include: a processor 1001 , such as a CPU, a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include standard wired interfaces and wireless interfaces (eg, WI-FI, 4G, and 5G interfaces). The memory 1005 may be high-speed RAM memory, or may be non-volatile memory, such as disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Those skilled in the art can understand that the structure shown in FIG. 8 does not constitute a limitation to the electronic device, and may include more or less components than the one shown, or combine some components, or arrange different components.

As shown in FIG. 8 , the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a configurable multi-dimensional data monitoring program.

In the electronic device shown in FIG. 8 , the network interface 1004 is mainly used for data communication with an external network; the user interface 1003 is mainly used to receive input instructions from users; Multidimensional Data Monitor, and does the following:

Configure the data field to get the configuration field;

Flink filters the first business data in the log server according to the configuration field to obtain second business data; stores the second business data in the database;

Configure monitoring indicators to get configuration indicators;

The database is queried according to the configuration index to obtain third service data, and an index display is performed on the third service data.

Optionally, obtaining the configuration field from the configuration data field includes the following steps:

Set business ID and/or functional ID and/or base fields and/or custom fields;

establishing an association relationship between the service identifier and the database, and establishing an association relationship between N (N is greater than or equal to 1) the service identifier and one of the databases;

Establish an association relationship between the function identifier and the database table, and establish an association relationship between N (N is greater than or equal to 1) of the function identifier and one of the database tables;

An association relationship between the basic field and/or the custom field and the field of the database table is established.

Optionally, the method further includes:

Calculate the number of records corresponding to the functional identifier;

Judging the first proportion of the number of records in the total records of the database table corresponding to the function identifier and/or the second proportion within a specified time range;

The first proportion is greater than the first threshold and/or the second proportion is greater than the second threshold, the second business data is sampled to obtain fourth business data, and then the fourth business data is stored in in the database.

Optionally, obtaining the configuration index from the configuration monitoring index includes the following steps:

The monitoring indicators include at least one of the following: chart type, function indicators, X-axis indicators, Y-axis indicators, legend indicators, query conditions, and filter conditions;

The function index includes N (N is greater than or equal to 1) the function identifiers;

The X-axis indicator includes time units and/or N (N greater than or equal to 1) basic fields and/or N (N greater than or equal to 1) custom fields;

The Y-axis indicator includes N (N greater than or equal to 1) basic fields and/or N (N greater than or equal to 1) custom fields;

The filter conditions are used to limit the data scale and filter useless data;

The query condition is used for the query of the legend indicator;

The legend index performs data grouping according to the function index, and obtains a plurality of index curves.

Optionally, the method further includes the following steps:

The X-axis indicator further includes a first function, and the first function includes at least one of the following: a mean value function and an accumulation function;

The Y-axis indicator supports numerical and proportional types. When the Y-axis is proportional, configure the numerator and denominator; the numerator is the basic field and/or a custom field, and the denominator is the basic field. fields and/or custom fields;

The Y-axis indicator further includes a second function, and the second function includes at least one of the following: a mean value function and an accumulation function.

Optionally, the method also includes the following steps:

The third service data is cached, and the cache duration is configurable.

Optionally, the legend indicators perform data grouping according to the function indicators, including the following steps:

configuring the grouping conditions corresponding to the basic field and/or the custom field;

The data of the basic field and/or the custom field is grouped according to the grouping condition.

Through the embodiment of the present invention, the data of the log server is recorded in the system, and the configuration of the X axis, the Y axis and the function configuration is supported, which avoids the SQL writing of the log server, and effectively reduces the configuration threshold; In this way, multi-dimensional indicator configuration can be realized, and business buried points can be flexibly assembled into the same indicator graph.

In addition, an embodiment of the present invention also provides a computer-readable storage medium, where a configurable multi-dimensional data monitoring program is stored on the computer-readable storage medium, and the configurable multi-dimensional data monitoring program is executed by a processor to achieve the following operations:

Configure the data field to get the configuration field;

Flink filters the first business data in the log server according to the configuration field to obtain second business data; stores the second business data in the database;

Configure monitoring indicators to get configuration indicators;

The database is queried according to the configuration index to obtain third service data, and an index display is performed on the third service data.

Optionally, obtaining the configuration field from the configuration data field includes the following steps:

Set business ID and/or functional ID and/or base fields and/or custom fields;

establishing an association relationship between the service identifier and the database, and establishing an association relationship between N (N is greater than or equal to 1) the service identifier and one of the databases;

Establish an association relationship between the function identifier and the database table, and establish an association relationship between N (N is greater than or equal to 1) of the function identifier and one of the database tables;

An association relationship between the basic field and/or the custom field and the field of the database table is established.

Optionally, the method further includes:

Calculate the number of records corresponding to the functional identifier;

Judging the first proportion of the number of records in the total records of the database table corresponding to the function identifier and/or the second proportion within a specified time range;

The first proportion is greater than the first threshold and/or the second proportion is greater than the second threshold, the second business data is sampled to obtain fourth business data, and then the fourth business data is stored in in the database.

Optionally, obtaining the configuration index from the configuration monitoring index includes the following steps:

The monitoring indicators include at least one of the following: chart type, function indicators, X-axis indicators, Y-axis indicators, legend indicators, query conditions, and filter conditions;

The function index includes N (N is greater than or equal to 1) the function identifiers;

The X-axis indicator includes time units and/or N (N greater than or equal to 1) basic fields and/or N (N greater than or equal to 1) custom fields;

The Y-axis indicator includes N (N greater than or equal to 1) basic fields and/or N (N greater than or equal to 1) custom fields;

The filter conditions are used to limit the data scale and filter useless data;

The query condition is used for the query of the legend indicator;

The legend index performs data grouping according to the function index, and obtains a plurality of index curves.

Optionally, the method also includes the following steps:

The X-axis indicator further includes a first function, and the first function includes at least one of the following: a mean value function and an accumulation function;

The Y-axis indicator supports numerical and proportional types. When the Y-axis is proportional, configure the numerator and denominator; the numerator is the basic field and/or a custom field, and the denominator is the basic field. fields and/or custom fields;

The Y-axis indicator further includes a second function, and the second function includes at least one of the following: a mean value function and an accumulation function.

Optionally, the method also includes the following steps:

The third service data is cached, and the cache duration is configurable.

Optionally, the legend indicators perform data grouping according to the function indicators, including the following steps:

configuring the grouping conditions corresponding to the basic field and/or the custom field;

The data of the basic field and/or the custom field is grouped according to the grouping condition.

Through the embodiment of the present invention, the data of the log server is recorded in the system, and the configuration of the X axis, the Y axis and the function configuration is supported, which avoids the SQL writing of the log server, and effectively reduces the configuration threshold; In this way, multi-dimensional indicator configuration can be realized, and business buried points can be flexibly assembled into the same indicator graph.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or system. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system that includes the element.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on such understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM) as described above. , magnetic disk, optical disk), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, a controller, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields , are similarly included in the scope of patent protection of the present invention.

Claims (10)

1. A configurable, multi-dimensional data monitoring method, comprising the steps of:

configuring a data field to obtain a configuration field;

the Flink filters the first service data in the log server according to the configuration field to obtain second service data; storing the second service data into a database;

configuring the monitoring index to obtain a configuration index;

and querying the database according to the configuration index to obtain third service data, and performing index display on the third service data.

2. The method of claim 1, wherein the configuration data field is derived from a configuration field, comprising the steps of:

setting a service identifier and/or a function identifier and/or a basic field and/or a custom field;

establishing an incidence relation between the service identification and the database, and establishing an incidence relation between N (N is more than or equal to 1) service identifications and 1 database;

establishing an association relation between the function identifiers and a database table, and establishing an association relation between N (N is more than or equal to 1) function identifiers and 1 database table;

and establishing an incidence relation between the basic field and/or the custom field and the field of the database table.

3. The method of claim 2, further comprising:

calculating the number of records corresponding to the function identification;

judging a first proportion of the number of the records in the total records of the database table corresponding to the function identifier and/or a second proportion in a specified time range;

and sampling the second service data to obtain fourth service data, and storing the fourth service data into a database, wherein the first ratio is greater than a first threshold value and/or the second ratio is greater than a second threshold value.

4. The method of claim 2, wherein configuring the monitoring metrics to obtain the configuration metrics comprises:

the monitoring index comprises at least one of the following: chart type, function index, X-axis index, Y-axis index, legend index, query condition, and filtering condition;

the function index comprises N (N is more than or equal to 1) function identifiers;

the X-axis index comprises a time unit and/or N (N is more than or equal to 1) basic fields and/or N (N is more than or equal to 1) custom fields;

the Y-axis index comprises N (N is more than or equal to 1) basic fields and/or N (N is more than or equal to 1) custom fields;

the filtering condition is used for limiting the data size and filtering useless data;

the query condition is used for querying the legend index;

and the legend indexes carry out data grouping according to the function indexes to obtain a plurality of index curves.

5. The method according to claim 4, characterized in that it further comprises the steps of:

the X-axis index further includes a first function including at least one of: a mean function, an accumulation function;

the Y-axis index supports a numerical type and a proportional type, and when the Y-axis index is the proportional type, a numerator and a denominator are configured; the numerator is the basic field and/or the custom field, and the denominator is the basic field and/or the custom field;

the Y-axis index further includes a second function including at least one of: mean function, accumulation function.

6. The method according to claim 1, characterized in that the method further comprises the steps of:

and caching the third service data, wherein the caching time length can be configured.

7. The method of claim 4, wherein the legend indicator performs data grouping according to the function indicators, comprising the steps of:

configuring grouping conditions corresponding to the basic fields and/or the custom fields;

and grouping the data of the basic field and/or the custom field according to the grouping condition.

8. A configurable, multi-dimensional data monitoring apparatus, the apparatus comprising:

the field configuration unit is used for configuring the data field to obtain a configuration field;

the data filtering unit is used for the Flink to filter the first service data in the log server according to the configuration field to obtain second service data; storing the second service data into a database;

the index configuration unit is used for configuring the monitoring index to obtain a configuration index;

and the index query unit is used for querying the database according to the configuration index to obtain third service data and performing index display on the third service data.

9. An electronic device, characterized in that the electronic device comprises: a memory, a processor and a configurable multi-dimensional data monitoring program stored on the memory and executable on the processor, the configurable multi-dimensional data monitoring program being configured to implement the steps of the configurable multi-dimensional data monitoring method according to any of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the configurable multi-dimensional data monitoring method according to any one of claims 1 to 7.

Images