CN115221157A - Data processing method and apparatus, computer readable storage medium and electronic device - Google Patents

Abstract

The disclosure provides a data processing method and device, a computer readable storage medium and an electronic device. The method comprises the following steps: obtaining dimension fields in historical query sentences of a target table and query co-occurrence frequency among different dimension fields; determining initial-order edge weights among different nodes according to query co-occurrence frequencies among different dimension fields by taking the dimension fields as nodes to form a dimension co-occurrence graph; determining a first core node in the dimension co-occurrence graph according to the degree of each node in the dimension co-occurrence graph; obtaining a first connection subgraph after a first core node in the dimension co-occurrence graph is deleted; obtaining a split tree of the dimension co-occurrence graph according to the dimension co-occurrence graph, a first core node and a first connection subgraph of the dimension co-occurrence graph; determining an aggregation group of the dimension fields according to the target leaf nodes of the split tree; and constructing a data cube facing the target table according to the aggregation group of the dimension fields. The constructed data cube can be stored in a database for improving query efficiency.

Description

Data processing method and apparatus, computer readable storage medium and electronic device

technical field

The present disclosure relates to the field of computer technology, and in particular, to a data processing method and apparatus, a computer-readable storage medium, and an electronic device.

Background technique

With the development of mobile Internet, Internet of Things and other technologies, the accumulated data has exploded, and the era of big data has arrived. The collection of massive data is only the first step of big data technology. How to make data generate value is the ultimate goal of big data field. The emergence of Hadoop (Hedup, a distributed system infrastructure) has solved the problem of data storage, but there has been no satisfactory solution for how to query massive data in real time. In most cases, the query needs to respond to the user's operation in real time. In the related technology, the query engine often has a response time of several minutes or even tens of minutes, which obviously cannot meet the demand.

In some data platform products, data analysis or report business, it is usually necessary to choose an OLAP (Online Analytical Process) data query engine. As an open source OLAP engine built on the Hadoop big data platform, Apache Kylin uses a multi-dimensional cube (Cube) , also known as data cube) pre-computing technology, using the method of changing space for time, compared with the OLAP query engine of the traditional MPP (Massive Parallel Processing, massively parallel processing) architecture, the query speed is improved to sub-second level, And high concurrency capability, which greatly improves the efficiency of data analysis. The emergence of Apache Kylin not only solves the problem of fast query of massive data, but also avoids a series of troubles caused by manual development and maintenance of advance calculation programs.

When Apache Kylin is used as the OLAP query engine as the technical solution, it is faced with the calculation problem of complex cubes: the pre-computing of cubes with super-large dimension combinations generally uses Apache spark as the construction engine, but the computing tasks of large cubes will increase the computing resources. A lot of pressure will not only consume a lot of computing resources, but also have a super high construction time, which is not friendly to some businesses with high data timeliness, and super-large spark computing tasks may fail due to resource problems. , which in turn leads to more waste of resources.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide a data processing method and apparatus, a computer-readable storage medium, and an electronic device, which can solve the technical problems in the above-mentioned related technologies that Cube construction takes a long time and consumes more computing resources.

An embodiment of the present disclosure provides a data processing method, the method includes: obtaining a dimension field in a historical query statement of a target table and a query co-occurrence frequency between different dimension fields; The query co-occurrence frequency between different nodes determines the initial edge weights between different nodes to form a dimension co-occurrence graph; according to the degree of each node in the dimension co-occurrence graph, the first core node in the dimension co-occurrence graph is determined; the deleted dimension co-occurrence graph is obtained. Now the first connected subgraph after the first core node in the figure; according to the dimension co-occurrence graph and its first core node and the first connected sub-graph, a split tree of the dimension co-occurrence graph is obtained; according to the target leaf node of the split tree Determines the aggregation group of the dimension field; builds the data cube for the target table based on the aggregation group of the dimension field.

In some exemplary embodiments of the present disclosure, determining pairs of associated dimension groups that satisfy the merging condition in each associated dimension group according to the correlation between the associated dimension groups and the combined expansion rate, including: if the correlation between the associated dimension groups is The correlation is greater than the first correlation threshold; or if the correlation between the associated dimension groups is greater than the second correlation threshold and the combined expansion rate is less than the first expansion rate threshold; or if the correlation between the associated dimension groups is greater than the third correlation degree threshold and the combined expansion rate is less than the second expansion rate threshold, then it is determined that the corresponding associated dimension group is an associated dimension group pair that satisfies the combination condition; wherein, the first correlation threshold is greater than the second correlation threshold, and the second correlation threshold is greater than The third correlation threshold, the first expansion rate threshold is greater than the second expansion rate threshold.

An embodiment of the present disclosure provides a data processing apparatus, the apparatus includes: a co-occurrence frequency obtaining unit, configured to obtain a dimension field in a historical query statement of a target table and a query co-occurrence frequency between different dimension fields; dimension co-occurrence frequency The graph construction unit is used to use the dimension field as a node, and determine the primary edge weights between different nodes according to the query co-occurrence frequency between different dimension fields to form a dimension co-occurrence graph; the core node determination unit is used to co-occur according to dimensions The degree of each node in the graph is to determine the first core node in the dimension co-occurrence graph; the connected subgraph obtaining unit is used to obtain the first connected subgraph after deleting the first core node in the dimension co-occurrence graph; the graph is split The tree obtaining unit is used to obtain the split tree of the dimensional co-occurrence graph according to the dimensional co-occurrence graph and its first core node and the first connected subgraph; the field aggregation group determination unit is used to determine the dimension according to the target leaf node of the split tree The aggregation group of the field; the data cube building unit is used to construct the data cube oriented to the target table based on the aggregation group of the dimension field.

Embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the data processing method described in the foregoing embodiments is implemented.

An embodiment of the present disclosure provides an electronic device, including: at least one processor; and a storage device configured to store at least one program, and when the at least one program is executed by the at least one processor, the at least one processor implements the above embodiments data processing methods in .

In the technical solutions provided by some embodiments of the present disclosure, on the one hand, the historical query statement of the target table is used for analysis, and the dimension co-occurrence is constructed according to the dimension fields in the historical query statement and the query co-occurrence frequency between different dimension fields. and determine the first core node in the dimension co-occurrence graph according to the degree of each node in the dimension co-occurrence graph, and then obtain the first connected subgraph after deleting the first core node in the dimension co-occurrence graph, according to The dimensional co-occurrence graph and its first core node and the first connected subgraph are split on the dimensional co-occurrence graph to obtain a split tree of the dimensional co-occurrence graph, which can be determined according to the target leaf node of the split tree The aggregation group of the dimension field in the above historical query statement, when the aggregation group of the dimension field is used to construct the data cube (Cube) oriented to the target table, the large cube is divided into aggregation groups, and each aggregation group can be It is used to build various small cubes, that is, by splitting the dimensional co-occurrence graph, the data scale is compressed, the computing resources consumed in the cube construction process are reduced, the construction time of the cube can be shortened, and the construction resources and construction time can be realized. balance between. At the same time, since the dimension co-occurrence graph in the embodiment of the present disclosure is constructed according to the dimension fields in the historical query statements of the target table and the query co-occurrence frequency between different dimension fields, the dimension co-occurrence graph is split to achieve this. Splitting the construction task of a large cube into multiple small cube construction tasks can also ensure that each small cube constructed can meet the query requirements in the actual business, and will not increase the amount of repeated calculations. On the other hand, by analyzing historical query statements, the automation of cube construction is realized, the difficulty of designing cubes is simplified, and the design of cubes is optimized.

Description of drawings

FIG. 1 schematically shows a flowchart of a data processing method according to an embodiment of the present disclosure.

FIG. 2 schematically shows a schematic diagram of a dimensional co-occurrence graph according to an embodiment of the present disclosure.

FIG. 3 schematically shows a schematic diagram of a connected subgraph after deleting the node with the largest degree in the dimensional co-occurrence graph in FIG. 2 .

FIG. 4 schematically shows a schematic diagram of a connected subgraph after deleting the node with the largest degree in the connected subgraph in FIG. 3 .

FIG. 5 schematically shows a flow chart of dimensional co-occurrence graph splitting according to an embodiment of the present disclosure.

FIG. 6 schematically shows a schematic structural diagram of a split tree of a dimensional co-occurrence graph according to an embodiment of the present disclosure.

FIG. 7 schematically shows a schematic diagram of performing a pruning operation on a split tree according to an embodiment of the present disclosure.

FIG. 8 schematically shows a flowchart of a data processing method according to another embodiment of the present disclosure.

FIG. 9 schematically shows a schematic flowchart of a data processing method according to yet another embodiment of the present disclosure.

FIG. 10 schematically shows a schematic diagram of a system architecture to which the data processing method provided by the embodiment of the present disclosure is applied.

FIG. 11 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

FIG. 12 shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated descriptions will be omitted.

The features, structures, or characteristics described in this disclosure may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

The accompanying drawings are merely schematic illustrations of the present disclosure, and the same reference numerals in the drawings denote the same or similar parts, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the figures do not necessarily necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in at least one hardware module or integrated circuit, or in different networks and/or processor devices and/or microcontroller devices.

The flowcharts shown in the figures are only exemplary illustrations, and do not necessarily include all contents and steps, nor do they have to be performed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may be changed according to the actual situation.

In this specification, the terms "a", "an", "the", "the" and "at least one" are used to indicate the presence of at least one element/component/etc.; the terms "comprising", "including" and "having" " is used to indicate an open-ended inclusive meaning and to mean that there may be additional elements/components/etc. in addition to the listed elements/components/etc; the terms "first", "second" and "Third" etc. are used only as markers, not as a limit on the number of their objects.

In related technologies, users are allowed to design the aggregation group of cubes by themselves. When faced with very complex business, users need to spend a lot of time to understand the business and then design the cube. Usually, they are faced with the task of determining which dimension combinations can be classified into an aggregation. Group; in the aggregation group, which dimension combinations can form a joint dimension; how to design the RowKey (row key) to make the query faster. The dimension combination of the Cube designed by the user is very large, and the user designed it by himself. Cube does not use historical query data, so that the designed Cube cannot meet the actual query requirements well.

In addition, to design a good Cube, there are high requirements for developers, not only to understand data, but also to be familiar with business, and to be proficient in Kylin and Spark principles, and the design process is not achieved overnight, but through continuous adjustment. final result. Therefore, automating Cube design is a necessary task.

Based on the technical problems existing in the above-mentioned related technologies, an embodiment of the present disclosure proposes a data processing method for at least partially solving the above-mentioned problems. The methods provided by the embodiments of the present disclosure may be executed by any electronic device, such as a server, or a terminal, or the interaction between a server and a terminal, which is not limited in the present disclosure.

The server mentioned in the embodiments of the present disclosure may be an independent server, a server cluster or a distributed system composed of multiple servers, and may also provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network Services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.

The terminal may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected through wired or wireless communication, which is not limited in this disclosure.

The exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

FIG. 1 schematically shows a flowchart of a data processing method according to an embodiment of the present disclosure. As shown in FIG. 1 , the method provided by the embodiment of the present disclosure may include the following steps.

In step S110, the dimension field in the historical query statement of the target table and the query co-occurrence frequency between different dimension fields are obtained.

In the embodiment of the present disclosure, the target table refers to the table of the cube to be constructed. The target table can be from the same data source. In the embodiment of the present disclosure, the data source may be Hive (a data warehouse tool based on Hadoop) or Kafka (a distributed publish-subscribe messaging system), etc., which are not limited in the present disclosure. In the following embodiments, all Taking Hive as an example for illustration, the same data source refers to the same table on Hive or the combination of the same Fact Table and Dimension Table. When the target table comes from different data sources, it builds multiple cubes facing multiple data sources.

The fact table refers to a table that stores fact records, such as system logs, sales records, sensor values, and so on. A dimension table or dimension table, also known as a lookup table, is a table corresponding to a fact table; it stores the attribute values of the dimension and can be associated with the fact table; it is equivalent to repeating the fact table frequently The attributes are extracted and standardized with a table for management. Common dimension tables include: date table (stores attributes such as week, month, quarter, etc. corresponding to the date), region table (including attributes such as country, province/state, city, etc.), etc.

In the embodiment of the present disclosure, the historical query statement of the target table may refer to the SQL (Structured Query Language, Structured Query Language) statement used to query the same data source on Hive in the history of the user, and these historical query SQL statements can be stored in the in the query log.

In this embodiment of the present disclosure, a dimension is an angle from which data is observed, and is generally a set of discrete values, for example, each independent date in the time dimension, each independent commodity in the commodity dimension, and each individual commodity in the device dimension stand-alone equipment, etc. The measure is the aggregated statistical value, that is, the result of the aggregation calculation (such as accumulation, mean, maximum, minimum, etc.), which is generally a continuous value, such as sales, average sales price, and the total number of items sold Wait.

By analyzing the fields involved in the historical query statement, it can be classified to determine whether it is a dimension field, and the query co-occurrence frequency of different dimension fields can be determined. The query co-occurrence frequency refers to the number of times that different dimension fields appear in the same historical query statement in all historical query statements of the target table at the same time.

In the embodiment of the present disclosure, an AST (Abstract Syntax Tree) can be obtained by converting the SQL statements in the historical query statements with the help of a SQL Parser, and then all historical queries can be extracted by traversing the edges of the AST The fields of the physical table (referring to a table in a certain data source) corresponding to each SQL statement involved in the statement, and the field type is marked for each field. For example, the marked field type can include aggregate fields (group), filtering Field (filter) and indicator fields, etc.

For example, for the following SQL statement: SELECT user_id,user_name,sum(value)as valueFROM user_data WHERE partition_time=20200101GROUP BY user_id,user_name, the following results can be obtained:

[(user_data, user_id, ("group")),

(user_data, user_name, ("group")),

(user_data, partition_time, ("filter")),

(user_data, value, ("sum"))]

That is, assuming that the table name of the target table of the historical query is user_data (user data), in the above SQL statement, the fields user_id (user ID), user_name (user name), partition_time (partition time) and value (value) are extracted, The field type of the fields user_id and user_name is an aggregate field, the field type of the field partition_time is a filter field, and the field type of the field value is an indicator field. The value field is an indicator field and will not enter the query network. Fields such as user_id, user_name, and partition_time are determined as dimension fields.

In step S120, the dimension field is used as a node, and the primary edge weight between different nodes is determined according to the query co-occurrence frequency between different dimension fields, so as to form a dimension co-occurrence graph.

In the embodiment of the present disclosure, a dimension co-occurrence graph can be constructed by using the dimension fields obtained in the above step S110 and the query co-occurrence frequencies of different dimension fields. The initial edge weight represents the query co-occurrence frequency between the two dimension fields corresponding to the two nodes connecting the edge.

For example, taking the target table user_data of the above historical query as an example, according to the three dimension fields of user_id, user_name, and partition_time determined above, the following information such as node, edge and primary edge weight can be obtained, thereby forming the target table user_data. Dimensional co-occurrence graph:

[(user_name,user_id,1),

(partition_time, user_id, 1),

(partition_time,user_name,1)]

Among them, (user_name, user_id, 1) indicates that user_name and user_id are used as two nodes, and there is an edge between these two nodes, and since the two dimension fields of user_name and user_id appear together in the above SQL statement, therefore, At this time, the initial edge weight of this edge is 1. (partition_time, user_id, 1) and (partition_time, user_name, 1) express similar meanings.

After analyzing all historical query statements, the initial edge weights on the same edge are accumulated, that is, each edge is determined according to the query co-occurrence frequency between the two dimension fields corresponding to the two nodes connected by each edge. Then we can get an undirected weighted dimension co-occurrence graph G containing all dimension fields in all historical query sentences.

In step S130, a first core node in the dimensional co-occurrence graph is determined according to the degrees of each node in the dimensional co-occurrence graph.

In the embodiment of the present disclosure, the degree of each node in the dimension co-occurrence graph refers to the number of edges connected to each node. For example, if user_id is connected to partition_time and user_name respectively, the degree of the node corresponding to the dimension field of user_id is 2.

In the embodiment of the present disclosure, the node with the highest degree in the dimensional co-occurrence graph G may be determined as the first core node of the dimensional co-occurrence graph G.

In step S140, a first connected subgraph after deleting the first core node in the dimension co-occurrence graph is obtained.

In the embodiment of the present disclosure, after obtaining the first core node with the largest degree in the dimensional co-occurrence graph G, the first core node can be deleted from the dimensional co-occurrence graph, corresponding to the deletion of the first core node and other nodes. and then judge whether there are still nodes in the dimensional co-occurrence graph after deleting the first core node. If there are nodes, then further judge whether the remaining two nodes are still connected. If there are at least two nodes in the remaining nodes that are not connected, it means that there are two or more first connected subgraphs in the dimensional co-occurrence graph after the first core node is deleted. There is connectivity between any two nodes in each first connected subgraph.

Among them, in an undirected graph, if there is a path connecting from node i to node j, then node i and node j are said to be connected, and both i and j are positive integers greater than or equal to 1. If any two nodes in the graph are connected, then the graph is considered connected, otherwise, it is considered disconnected.

In step S150, a split tree of the dimensional co-occurrence graph is obtained according to the dimensional co-occurrence graph and its first core node and first connected subgraph.

In an exemplary embodiment, obtaining a split tree of the dimensional co-occurrence graph according to the dimensional co-occurrence graph and its first core node and first connected subgraph may include: combining nodes in the dimensional co-occurrence graph to form a split tree The root node; according to the degree of each node in the first connected subgraph, determine the second core node in the first connected subgraph; if the second core node in the first connected subgraph is deleted, the first connected subgraph will not be If there is a node, the first core node and the second core node are combined as the primary leaf node of the split tree.

In an exemplary embodiment, obtaining a split tree of the dimensional co-occurrence graph according to the dimensional co-occurrence graph and its first core node and the first connected subgraph, may further include: if the second core in the first connected subgraph is deleted After the node, there are nodes in the first connected subgraph, and the nodes after deleting the second core node are connected, then the second connected subgraph after deleting the second core node in the first connected subgraph is obtained; if If the current number of split layers of the split tree is less than or equal to the threshold of the number of split layers, the third core node in the second connected subgraph is determined according to the degree of each node in the second connected subgraph; After the third core node, if there is no node in the second connected subgraph, the first core node, the second core node and the third core node are combined as the primary leaf node of the split tree. For a specific manner of splitting the dimensional co-occurrence graph to obtain a split tree, reference may be made to FIG. 2 to FIG. 7 below.

In step S160, an aggregation group of dimension fields is determined according to the target leaf node of the split tree.

In an exemplary embodiment, determining the aggregation group of dimension fields according to the target leaf node of the split tree may include: obtaining the split income of each branch node in the split tree; according to the size of the split income of each branch node in the split tree and the leaf node Threshold, prune the split tree; take the initial leaf node retained after the split tree pruning operation as the target leaf node, and the number of target leaf nodes is less than or equal to the leaf node threshold; take the target leaf node as the aggregation of dimension fields Group.

In the embodiment of the present disclosure, the leaf node threshold may be set according to actual requirements, which is not limited in the present disclosure.

In an exemplary embodiment, each branch node in the split tree may include a first branch node, and the first branch node may include each first child node of the first parent node. Wherein, obtaining the split income of each branch node in the split tree may include: obtaining the dimension combination size of each dimension field in the first parent node; obtaining the query co-occurrence frequency between each dimension field in the first parent node; obtaining the root node The query co-occurrence frequency between each dimension field in The query co-occurrence frequency of each first child node is obtained, and the query construction cost of the first parent node is obtained; the dimension combination size of each dimension field in each first child node is obtained; the query co-occurrence frequency between each dimension field in each first child node is obtained; The dimension combination size of each dimension field in each first child node, the query co-occurrence frequency between each dimension field in each first child node, and the query co-occurrence frequency between each dimension field in the root node, obtain each first child node The query construction cost of the node; according to the query construction cost of the first parent node and the query construction cost of each first child node, the split income of the first branch node is obtained. For the specific manner of calculating the split income and performing the pruning operation on the split tree according to the split income, reference may be made to the embodiment in FIG. 7 .

In step S170, a data cube oriented to the target table is constructed according to the aggregation group of dimension fields.

In the embodiment of the present disclosure, a cube is a multi-dimensional cube, also called a data cube, which is a multi-dimensional space constructed according to dimensions, and contains basic data to be analyzed, and aggregated data operations are performed on the cube. Among them, for each combination of dimensions, the measures are aggregated, and then the result of the operation is saved as a materialized view, called Cuboid. A Cuboid that combines all dimensions as a whole is called a Cube.

The data processing method provided by the embodiment of the present disclosure, on the one hand, analyzes the historical query statement of the target table, constructs a dimension co-occurrence graph according to the dimension fields in the historical query statement and the query co-occurrence frequency between different dimension fields, and according to The degree of each node in the dimensional co-occurrence graph is used to determine the first core node in the dimensional co-occurrence graph, and then the first connected subgraph after deleting the first core node in the dimensional co-occurrence graph is obtained. According to the dimensional co-occurrence graph The graph and its first core node and the first connected subgraph are split on the dimension co-occurrence graph to obtain a split tree of the dimension co-occurrence graph, so that the above historical query statement can be determined according to the target leaf node of the split tree When using the aggregation group of the dimension field to build a data cube (Cube) oriented to the target table, the large cube is divided into aggregation groups, and each aggregation group can be used to construct various aggregation groups. Small cubes, that is, by splitting the dimensional co-occurrence graph, compresses the data scale, reduces the computing resources consumed in the cube construction process, and can shorten the construction time of the cube, achieving a balance between construction resources and construction time. . At the same time, since the dimension co-occurrence graph in the embodiment of the present disclosure is constructed according to the dimension fields in the historical query statements of the target table and the query co-occurrence frequency between different dimension fields, the dimension co-occurrence graph is split to achieve this. Splitting the construction task of a large cube into multiple small cube construction tasks can also ensure that each small cube constructed can meet the query requirements in the actual business, and will not increase the amount of repeated calculations. On the other hand, by analyzing historical query statements, the automation of cube construction is realized, the difficulty of designing cubes is simplified, and the design of cubes is optimized.

FIG. 2 schematically shows a schematic diagram of a dimensional co-occurrence graph according to an embodiment of the present disclosure.

As shown in Figure 2, it is assumed that the historical query statement includes dimension fields city_name (city name), city_id (city ID), user_name, user_id, partition_time, Y (year, year), M (month, month) and D (day, Day), and assume that partition_time has an edge connected to city_name, city_id, user_name, user_id, Y, M, and D, respectively, and the primary edge weights of each edge are 56, 106, 70, 100, 300, 60, and 80; The primary edge weight of the edge connected by the two nodes user_name and user_id is 61, the primary edge weight of the edge connected by the city_name and city_id nodes is 200, and the primary edge weight of the edge connected by the city_id and Y nodes is 77 , the primary edge weight of the edge connected by the two nodes Y and M is 53, the primary edge weight of the edge connected by the two nodes D and M is 96, and the primary edge weight of the edge connected by city_id and the two nodes of Y is 89, the initial edge weight of the edge connecting the two nodes Y and D is 21.

In the embodiment of the present disclosure, after the dimensional co-occurrence graph G is obtained, the following graph decomposition algorithm can be used to divide the dimensional co-occurrence graph to obtain a split tree:

The first step is to set the core node list cores and the child node list children, and initialize the core node list to be empty, which means: cores=[], and initialize the child node list to include all dimension fields in the co-occurrence graph G of this dimension.

The second step is to calculate the degrees of all nodes in the dimensional co-occurrence graph G, and arrange them in reverse order according to the size of the degree, and determine the node with the largest degree as the first core node.

The third step is to delete the first core node with the largest degree from the dimension co-occurrence graph G, and put the first core node into the core node list cores.

The fourth step is to judge whether there are still nodes in the dimensional co-occurrence graph G after the first core node with the largest deletion degree. If there are no nodes, the splitting of the dimensional co-occurrence graph is completed; otherwise, if there are still nodes, judge Whether any two remaining nodes after deleting the first core node are connected; if any two remaining nodes after deleting the first core node are connected, then return to the above second step, after deleting the first core node The remaining nodes in the dimensional co-occurrence graph G of the dimensional co-occurrence graph G are processed similarly to the above-mentioned steps 2 to 4, that is, continue to calculate the degree of all the remaining nodes in the dimensional co-occurrence graph G after the deletion of the first core node, and Arrange in reverse order according to the size of the degree, and determine the node with the largest degree as another first core node; then delete the other first core node with the largest degree from the dimensional co-occurrence graph G after deleting the first core node, and Put the other first core node into the core node list cores, and then judge whether there are still nodes in the dimension co-occurrence graph G behind the other first core node with the largest deletion degree, if there is no node, complete the The splitting of the dimensional co-occurrence graph; on the contrary, if there are still nodes, it is judged whether any two remaining nodes after deleting the other first core node are connected; if any two remaining nodes after deleting the other first core node are connected If the two nodes are connected, then go back to the above second step again, and process all the remaining nodes in the dimensional co-occurrence graph G after the deletion of the other first core node similar to the above second step to fourth step. , ... until a first core node is deleted, there is no node in the dimensional co-occurrence graph, or there are at least two nodes between the remaining nodes in the dimensional co-occurrence graph after a first core node is deleted. connected.

Step 5: If at least two nodes are non-connected in the remaining nodes after deleting the first core node, then add all the first connected subgraphs G' in the dimension co-occurrence graph after deleting the first core node. to the child node list children.

After that, split each first connected subgraph G' in the child node list children. The splitting method can refer to the above-mentioned dimensional co-occurrence graph G, that is, splitting in a tree-like manner, and finally a splitting tree can be obtained. .

Take the splitting of any one of the first connected subgraphs G' as an example:

1), set the core node list cores and the child node list children, and initialize the core node list cores to include the first core node in the above-mentioned dimension co-occurrence graph, and initialize the child node list to include all dimensions in the first connected subgraph G' field.

2), calculate the degree of all nodes in the first connected subgraph G', and arrange in reverse order according to the size of the degree, and determine the node with the largest degree as the second core node.

3), delete the second core node with the largest degree from the first connected subgraph G', and put the second core node into the core node list cores.

4), judge whether there is a node in the first connected subgraph G' behind the second core node with the largest deletion degree, if there is no node, then complete the splitting of the first connected subgraph G'; otherwise, if there is still node, then determine whether any two remaining nodes after deleting the second core node are connected; if there is connectivity between any two remaining nodes after deleting the second core node, then return to the above step 2), to delete the first node. All the remaining nodes in the first connected subgraph G' after the two core nodes are processed similar to the above steps 2) to 4), that is, continue to calculate the remaining nodes in the first connected subgraph G' after the second core node is deleted. The degrees of all nodes are arranged in reverse order according to the size of the degree, and the node with the largest degree is determined as another second core node; then delete the largest degree from the first connected subgraph G' after deleting the second core node. the other second core node, put the other second core node into the core node list cores, and then judge whether the first connected subgraph G' behind the other second core node with the largest deletion degree is in the first connected subgraph G' There are still nodes. If there are no nodes, the splitting of the first connected subgraph G' is completed; otherwise, if there are nodes, it is determined whether any two remaining nodes are connected after deleting the other second core node. If the connection between any two remaining nodes after deleting this other second core node is connected, then return to above-mentioned step 2) again, to delete the first connected subgraph G' after this other second core node All the remaining nodes are processed similar to the above steps 2) to 4), ... until after deleting a certain second core node, there is no node in the first connected subgraph G', or delete a certain second core node. Between the remaining nodes in the first connected subgraph G' after the core node, at least two nodes are not connected.

5), if there are at least two non-connected nodes in the remaining nodes after deleting the second core node, then all the second connected subgraphs in the first connected subgraph G' after deleting the second core node will be deleted. Add to the child node list children.

After that, each second connected subgraph in the child node list children is split again, and the splitting method can refer to the above-mentioned splitting of the dimensional co-occurrence graph G and the first connected subgraph.

Take the splitting of any of the second connected subgraphs as an example:

(1), set the core node list cores and the child node list children, and initialize the core node list cores to include the first core node in the above-mentioned dimensional co-occurrence graph and the second core node corresponding to the first connected subgraph, and initialize the child The node list includes all dimension fields in the second connected subgraph.

(2), calculate the degree of all nodes in the second connected subgraph, and arrange them in reverse order according to the size of the degree, and determine the node with the largest degree as the third core node.

(3), delete the third core node with the largest degree from the second connected subgraph, and put the third core node into the core node list cores.

(4), judging whether there are still nodes in the second connected subgraph after the third core node with the largest deletion degree, if there are no nodes, the splitting of the second connected subgraph is completed; otherwise, if there are still nodes, then Determine whether any two remaining nodes after deleting the third core node are connected; if any two nodes remaining after deleting the third core node are connected, then return to the above step (2), to delete the third core node. All the remaining nodes in the second connected subgraph after the node are processed similar to the above steps (2) to (4), that is, continue to calculate all the remaining nodes in the second connected subgraph after deleting the third core node. degree, and arrange them in reverse order according to the size of the degree, and determine the node with the largest degree as another third core node; then delete the other third core node with the largest degree from the second connected subgraph after deleting the third core node. core node, and put the other third core node into the core node list cores, and then judge whether there is a node in the second connected subgraph behind the other third core node with the largest deletion degree, if there is no node , then the splitting of the second connected subgraph is completed; on the contrary, if there are nodes, it is judged whether any two remaining nodes after deleting the other third core node are connected; if the other third core node is deleted After the connection between any two remaining nodes is connected, then return to the above step (2) again, and perform similar steps (2) on all the remaining nodes in the second connected subgraph after the deletion of the other third core node. ) to the processing of step (4), ... until after deleting a certain third core node, there is no node in the second connected subgraph, or after deleting a certain third core node, the second connected subgraph remains There are at least two nodes that are not connected between the nodes.

(5), if at least two nodes are disconnected between the remaining nodes after deleting the third core node, then add all third connected subgraphs in the second connected subgraph after deleting the third core node to the to the child node list children.

After that, split each third connected subgraph in the child node list children... .

In the above splitting process, it can be judged whether the current split level of the split tree is greater than the split level threshold, and if the current split level is greater than the split level threshold, the splitting is stopped; otherwise, if the current split level is less than or equal to the split level The number of layers threshold, and the children of the currently split graph (such as the above-mentioned dimensional co-occurrence graph or the first connected subgraph, the second connected subgraph, and the third connected subgraph) is not empty, then continue to the current split graph. Split all connected subgraphs in children, for example, split each first connected subgraph of the dimension co-occurrence graph, split each second connected subgraph of the first connected subgraph... to get each connected subgraph The split result of the subgraph.

In this embodiment of the present disclosure, the current number of split layers refers to the current depth of the split tree. Among them, the threshold of the number of split layers can be determined according to the specific business. For example, the number of dimension fields and the number of required cubes can be comprehensively considered. Here, it is assumed that the threshold of the number of classification layers is set to 6.

The above splitting process will be illustrated below with reference to FIG. 3 to FIG. 6 .

According to Figure 2 above, partition_time is the node with the largest degree in the dimensional co-occurrence graph shown in Figure 2, and it connects 7 edges. After deleting the first core node partition_time in Figure 2, the two nodes shown in Figure 3 are obtained. A first connected subgraph, that is, the dimensional co-occurrence graph shown in Figure 2 is split into: the first connected subgraph 301 composed of two nodes user_name and user_id and an edge connected to them, city_name, city_id, Y, M and D A first connected subgraph 302 composed of four nodes and 6 edges formed by the connection between the four nodes.

For the connected subgraph 302 in Figure 3 above, it can be further determined that the node with the largest degree is Y, then after deleting the second core node Y in the connected first subgraph 302, two second core nodes as shown in Figure 4 are obtained. Connected subgraph: a second connected subgraph 401 composed of two nodes city_name and city_id and an edge connected to them, and a second connected subgraph 402 composed of two nodes M and D and an edge connected to each other.

As shown in FIG. 5 , taking the above-mentioned FIGS. 2 to 4 as examples, the splitting process may include the following steps.

In step S11, the initialized core node list is empty, that is, cores=[], and the child node list children=[partition_time, city_id, city_name, user_id, user_name, Y, M, D].

In step S12, the first core node is added to the core node list, then cores=[partition_time].

At the same time, add the two first connected subgraphs to the child node list, then children=[user_id, user_name], [city_id, city_name, Y, M, D].

In step S13, it is determined that the second core node of the first connected subgraph 301 is user_id, and the second core node user_id is added to the core node list, then cores=[partition_time, user_id], and the child node list children=[user_name].

In step S14, it is determined that another second core node of the first connected subgraph 301 is user_name, and another second core node user_name is added to the core node list, then cores=[partition_time, user_id, user_name], and the child node list children=[].

In step S15, it is determined that the second core node of the first connected subgraph 302 is Y, and the second core node Y is added to the core node list, then cores=[partition_time, Y], and the two nodes of the first connected subgraph 302 are obtained. There are two second connected subgraphs 401 and 402, and these two second connected subgraphs 402 are added to the child node list, then children=([city_id,tity_name],[M,D]).

In step S16, it is determined that the third core node of the second connected subgraph 401 is city_id, and the third core node city_id is added to the core node list, then cores=[partition_time, Y, city_id], and the child node list children=[city_name ].

In step S17, it is determined that another third core node of the second connected subgraph 401 is city_name, and the other third core node city_name is added to the core node list, then cores=[partition_time, Y, city_id, city_name], and List of child nodes children=[].

In step S18, it is determined that the third core node of the second connected subgraph 402 is M, and the third core node M is added to the core node list, then cores=[partition_time, Y, M], and the child node list children=[D ].

In step S19, it is determined that another third core node of the second connected subgraph 402 is D, and the other third core node D is added to the core node list, then cores=[partition_time, Y, M, D], and List of child nodes children=[].

Through the splitting process shown in Figure 5, the splitting tree shown in Figure 6 can be obtained.

The node list nodes of the root node 600 of the split tree includes [partition_time, city_id, city_name, user_id, user_name, Y, M, D]. The node list nodes of the two branch nodes included in the first split layer 601 are respectively [partition_time, user_id, user_name], [partition_time, Y, city_id, tity_name, M, D], namely [partition_time, user_id, user_name] and [partition_time, Y, city_id, city_name, M, D] are all child nodes of the root node 600 .

The branch node [partition_time, Y, city_id, tity_name, M, D] in the first split level 601 is used as the parent node of the second split level 602 , and the node list of the two branch nodes included in the second split level 602 Respectively [partition_time, user_id, user_name], [partition_time, Y, city_id, tity_name, M, D], namely [partition_time, user_id, user_name] and [partition_time, Y, city_id, tity_name, M, D] are the parent nodes [ child nodes of partition_time, Y, city_id, city_name, M, D].

In the related art, there is the problem of data expansion: that is, in the related art, the dimension combination is pre-calculated, and the formula for calculating the dimension combination is 2^N (N is the number of dimensions, and N is a positive integer greater than or equal to 1), such as For a model containing (A, B, C, D), where D is the indicator field, and A, B, and C are the dimension fields, then all the corresponding dimensions are combined as follows: [(), (A), (B),(C),(A,B),(A,C),(B,C),(A,B,C)], if the dimension of the data model is very large, such as 60 or more In this case, dimension explosion is unacceptable. Although dimension pruning is provided in the related art to reduce the number of dimension combinations, for more complex businesses such as advertising, the single data model There are still more than 1,000 combinations that cannot be tailored for the construction tasks of . In the case of a particularly large number of dimensional combinations, it will inevitably lead to a rapid expansion of the resulting data scale. Therefore, how to prune more effectively is an important issue in Cube design.

In the related art, in order to reduce the complexity of cubes and reduce the number of cubes, although some cube pruning algorithms are provided, there is a problem with these cube pruning algorithms. They are based on that all query requests are evenly distributed in each This assumption is used to calculate the rate of return on the combination of dimensions, which is impossible in reality. In actual business, users' queries usually focus on some dimension combinations, and other dimension combinations rarely or even no query falls on top, if some dimension combinations are pre-computed, but not queried; or some dimension combinations Frequently queried, but not pre-computed, these gaps affect Cube's resource utilization and query performance.

In addition, some pruning methods in the related art are not suitable for cube pruning with too many dimensions, or other pruning methods have stability problems, that is, the dimension combinations calculated each time are completely different.

After obtaining the split tree shown in Fig. 6, its leaf nodes are used as primary leaf nodes. Some branch nodes in Fig. 6 may not need to be subdivided, but are split. The split income after each split is pruned to obtain the target leaf node of the split tree.

FIG. 7 schematically shows a schematic diagram of performing a pruning operation on a split tree according to an embodiment of the present disclosure.

As shown in FIG. 7 , it is assumed that the first parent node 701 in the split tree includes three dimension fields a, b, and c, and the first branch node includes two first child nodes 702 of the first parent node 701 , that is, a first The child node 702 includes two dimension fields a and b, and another first child node 702 has two dimension fields a and c. It can be understood that any parent node in the split tree can be selected as the first parent node, and each parent node is a child node relative to the previous split level.

It is assumed that the two dimension fields a and c in the first child node 702 are used as the second parent node 703, and further includes two second child nodes 704, namely the second child node 704 including the a dimension field and the second child node including the c dimension field. Child node 704.

In the embodiment of the present disclosure, the following formula (1) can be used to calculate the split income Split_income1 of the first branch node:

Alternatively, the split income Split_income1 of the first branch node can also be calculated according to the following formula (2):

In the above formulas (1) and (2), D(a,b,c) represents the dimension combination size of the dimension combination formed by the combination of the three dimension fields (a,b,c) in the first parent node, specifically The calculation can be obtained by counting the number of records combined by the three dimension fields in the original data of the target table, for example, obtained by unique values; Q(a,b,c) means (a,b,c) in the first parent node ) query co-occurrence frequency between these three dimension fields; Q(all) represents the query co-occurrence frequency between all dimension fields in the root node; D(a,b,c)*(0.5*Q(a,b) ,c)/Q(all)+0.5) considers the query cost and construction cost of the first parent node, which is called the query construction cost of the first parent node; D(a,b) represents the ( a,b) The dimension combination size of the dimension combination formed by the combination of these two dimension fields; Q(a,b) represents the query co-occurrence frequency between the two dimension fields (a,b) in the first child node; (D(a,b)*(0.5*Q(a,b)/Q(all)+0.5), which is called the query construction cost of the first child node; D(a,c) represents the cost of the first child node The dimension combination size of the dimension combination formed by the combination of these two dimension fields (a, c); Q(a, c) represents the query co-occurrence between the two dimension fields (a, c) in the first child node Frequency; D(a,c)*(0.5*Q(a,c)/Q(all)+0.5), which is called the query construction cost of the first child node.

In the embodiment of the present disclosure, the following formula (3) can be used to calculate the split income Split_income2 of the second branch node:

Alternatively, the split income Split_income2 of the second branch node can also be calculated according to the following formula (4):

In the above formulas (3) and (4), D(a,c) represents the dimension combination size of the dimension combination formed by the combination of the two dimension fields (a,c) in the second parent node; Q(a,c ) represents the query co-occurrence frequency between the two dimension fields (a,c) in the second parent node; D(a,c)*(0.5*Q(a,c)/Q(all)+0.5) is called It is the query construction cost of the second parent node; D(a) represents the dimension combination size of (a) this dimension field in the second child node; Q(a) represents the (a) dimension field in the second child node. The co-occurrence frequency of queries between the In (c) the dimension combination size of this dimension field; Q(c) represents the query co-occurrence frequency between (c) this dimension field in the second child node; D(c)*(0.5*Q(c)/ Q(all)+0.5), which is called the query construction cost of the second child node.

However, in the embodiment of the present disclosure, the calculation method of the split income of each branch node is not limited to the example given by the above formula.

Using the above calculation method, the split income of all branch nodes of the split tree can be calculated, and then the low-income splits can be pruned according to the size of the split income. For example, it is assumed that the split income of the first branch node is smaller than that of the second branch node To split the income, delete the first child nodes (a, b) and (a, c) until the number of primary leaf nodes of the split tree meets the leaf node threshold, and take each remaining primary leaf node as the target leaf node, and each A target leaf node is represented as an aggregate group of Cubes, and all dimension fields on each target leaf node can be used as the dimensions of a single Cube, so that multiple Cubes of the target table can be constructed through the above splitting process, that is, the The purpose of splitting a larger cube into multiple smaller cubes.

In this embodiment of the present disclosure, the leaf node threshold may be set according to actual needs, which is not limited in the present disclosure. In addition, the specific value of the low profit is not limited, and it can be determined according to the set leaf node threshold. For example, assuming that 4 aggregation groups need to be set currently, the leaf node threshold is set to 4, then in the above pruning process, the first 4 primary leaf nodes with the highest splitting benefits of branch nodes can be reserved as the target leaf nodes.

It should be noted that, in the above-mentioned pruning process, it is not required that the number of target leaf nodes finally obtained is equal to the leaf node threshold, as long as the number of target leaf nodes is less than or equal to the leaf node threshold.

In the data processing method provided by the embodiment of the present disclosure, a dimensional co-occurrence graph is constructed by analyzing the historical query statements of the target table, and by splitting the dimensional co-occurrence graph, an aggregation group of the split tree is obtained, and the aggregation group is constructed according to a plurality of aggregation groups. Multiple smaller cubes, thereby improving the computing resources and construction time during the cube construction process, and reducing the number of cubes. In addition, through the above pruning operation, the number of aggregation groups is further reduced, thereby further reducing the consumption of computing resources, further reducing the construction time, and at the same time making the constructed cubes meet actual business needs. At the same time, it realizes the automatic determination of the aggregation group, and does not require the user to design manually, thereby reducing the design complexity of the cube, and in the case of a particularly large number of dimensions, the design difficulty of the cube is reduced.

FIG. 8 schematically shows a flowchart of a data processing method according to another embodiment of the present disclosure. As shown in FIG. 8 , compared with other above-mentioned embodiments, the method provided in the embodiment of FIG. 8 further includes the following steps.

In step S810, the target edge weight of each edge of the dimensional co-occurrence graph is obtained according to the primary edge weight of each edge of the dimensional co-occurrence graph.

In an exemplary embodiment, the dimensional co-occurrence graph may include a first edge and first and second nodes of the first edge. Wherein, obtaining the target edge weight of each edge of the dimensional co-occurrence graph according to the initial edge weight of each edge of the dimensional co-occurrence graph may include: obtaining the first query frequency of the dimension field corresponding to the first node in the historical query statement ; obtain the second query frequency of the dimension field corresponding to the second node in the historical query statement; determine the target query frequency according to the first query frequency and the second query frequency; according to the initial edge weight of the first side and the target query frequency, Get the target edge weight for the first edge.

In the embodiment of the present disclosure, some dimension fields with relatively high correlation degree can be combined together to form a joint dimension by analyzing the query co-occurrence frequency of dimension fields in historical query sentences. Specifically, the following steps may be used to analyze the degree of association between dimension fields: recalculate the edge weights on the dimension co-occurrence graph G to obtain the target edge weights.

For example, for the first edge (a, b, w) in the dimensional co-occurrence graph, where a and b are the first and second nodes of the first edge, respectively, and w is the initial edge weight of the first edge , the target edge weight w' of the first edge can be calculated by the following formula:

w′=w/max(count(a),count(b)) (5)

In the above formula, count(a) represents the first query frequency of the first node a in the historical query statement, that is, the total number of occurrences; count(b) represents the second query frequency of the second node b in the historical query statement, the total number of occurrences.

In step S820, the edges whose weights of the target edges are lower than the weight threshold are removed to obtain each subgraph of the dimension co-occurrence graph.

Then, the edge with the target edge weight less than the weight threshold in the dimensional co-occurrence graph is removed from the dimensional co-occurrence graph, and after removing the edge with the target edge weight lower than the weight threshold, the dimensional co-occurrence graph may be split into multiple sub-graphs picture.

In the embodiment of the present disclosure, the weight threshold may be set according to experience, and the value range is (0-1), for example, it is assumed to be set to 0.6 here.

In step S830, the largest clique in each subgraph is obtained, and each associated dimension group is formed according to the dimension field in each largest clique.

After that, the maximum clique algorithm is used to find the largest clique in the graph on each subgraph after removing the edge whose target edge weight is lower than the weight threshold, and each largest clique is used as an associated dimension group.

Among them, given each subgraph, if there is a graph in the subgraph and there is an edge between any two nodes in the graph, then the graph is called the complete subgraph of the subgraph, and the complete subgraph of the subgraph is the subgraph The clique of the subgraph refers to the largest complete subgraph of the subgraph. The present disclosure does not limit which maximal clique algorithm is adopted.

In step S840, the degree of correlation and the combined expansion rate between each associated dimension group are obtained.

In an exemplary embodiment, each associated dimension group may include a first associated dimension group and a second associated dimension group. Wherein, obtaining the correlation between each associated dimension group may include: obtaining the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; obtaining the dimension combination size of the first associated dimension group and the second associated dimension group. The product result of the dimension combination size of the associated dimension group; merge the first associated dimension group and the second associated dimension group to obtain the merged associated dimension group; obtain the dimension combination size of the merged associated dimension group; according to the combined dimension combination size of the merged associated dimension group and The correlation between the first associated dimension group and the second associated dimension group is obtained by multiplying the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group.

In an exemplary embodiment, each associated dimension group may include a first associated dimension group and a second associated dimension group. Wherein, obtaining the combined expansion rate between each associated dimension group may include: obtaining the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; obtaining the dimension combination size of the first associated dimension group and the first associated dimension group. The summation result of the dimension combination size of the two associated dimension groups; merge the first associated dimension group and the second associated dimension group to obtain the merged associated dimension group; obtain the dimension combination size of the merged associated dimension group; according to the dimension combination of the merged associated dimension group The size and the summation result of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group obtains the combined expansion rate between the first associated dimension group and the second associated dimension group.

In step S850, each associated dimension group is merged according to the correlation between the respective associated dimension groups and the combined expansion rate to obtain a joint dimension (Joint Dimension) of the dimension co-occurrence graph, where the number of joint dimensions is less than or equal to the joint dimension threshold.

In the embodiment of the present disclosure, some dimension fields are combined together as a joint dimension, and the joint dimension is pre-calculated, thereby reducing the number of dimension combinations, the complexity of cubes, and the number of cubes.

For example, if the dimension fields a, b, and c are defined as joint dimensions, only Cuboid a b C will be constructed, and Cuboid ab, b c, a, and so on will not be generated.

In an exemplary embodiment, combining each associated dimension group to obtain a joint dimension of the dimension co-occurrence graph according to the correlation degree and the combined expansion rate between the respective associated dimension groups may include: according to the correlation between the respective associated dimension groups Relevance and merger expansion rate, determine the associated dimension group pairs that meet the merger conditions in each associated dimension group; obtain the target associated dimension group pair with the highest correlation in the associated dimension group pair; merge the associated dimension group in the target associated dimension group pair, Obtain the target merged associated dimension group, and delete the associated dimension group in the target associated dimension group pair; if the number of the target merged associated dimension group and the unmerged associated dimension group is less than or equal to the union dimension threshold, the target merged associated dimension group and The unmerged associated dimension groups are respectively used as the joint dimension of the dimension co-occurrence graph.

In an exemplary embodiment, determining the pair of associated dimension groups in each associated dimension group that satisfies the merging condition according to the degree of correlation and the combined expansion rate of each associated dimension group may include: if the degree of correlation between the associated dimension groups is greater than the first correlation threshold; or if the correlation between the associated dimension groups is greater than the second correlation threshold and the combined expansion rate is less than the first expansion rate threshold; or if the correlation between the associated dimension groups is greater than the third correlation threshold and If the combined expansion rate is less than the second expansion rate threshold, it is determined that the corresponding associated dimension group is an associated dimension group pair that satisfies the combination condition; wherein the first correlation threshold is greater than the second correlation threshold, and the second correlation threshold is greater than the third correlation threshold The first expansion rate threshold is greater than the second expansion rate threshold.

Specifically, after each associated dimension group is obtained, continue to further aggregate the dimension fields. For example, it is assumed that the following combination groups of associated dimension groups are obtained through the above method, and it is assumed that the goal is to compress the number of associated dimension groups in the combination to A given joint dimension threshold N (N is a positive integer greater than or equal to 1, which can be set according to actual needs, here it is assumed to be 4):

Groups=[

[user_id,user_name],

[partition_time],

[city_id,city_name],

[year],

[month],

[day]]

That is, it is assumed that six associated dimension groups of [user_id, user_name], [partition_time], [city_id, city_name], [year], [month], [day] are obtained. Then it is calculated by the following steps:

1.1: Traverse all associated dimension groups in groups, and obtain the dimension combination size of each associated dimension group by counting the number of unique values of the dimension combination of each associated dimension group in the source data of the target table.

1.2: Calculate the correlation between all associated dimension groups in groups.

For example, assuming that the first associated dimension group is [user_id, user_name] and the second associated dimension group is [partition_time], [user_id, user_name, partition_time] represents the merged associated dimension obtained by merging the first associated dimension group and the second associated dimension group group, the correlation between the first associated dimension group and the second associated dimension group can be calculated by the following formula: Rel:

In the above formula, D(user_id, user_name) represents the dimension combination size of the first associated dimension group; D(partition_time)) represents the dimension combination size of the second associated dimension group; (D(user_id, user_name)*D(partition_time)) Represents the product result of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; D(user_id, user_name, partition_time) represents the dimension combination size of the merged associated dimension group.

Calculate the combined expansion rate of the dimension combination size of the merged associated dimension group relative to the dimension combination size of the unmerged first associated dimension group and the dimension combination size of the second associated dimension group. For example, the combined expansion rate Exp can be calculated according to the following formula:

In the above formula, (D(user_id, user_name)+D(partition_time)) represents the summation result of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group.

1.3: Determine whether any two associated dimension groups meet the merging conditions through the calculation of the correlation between the above-mentioned various associated dimension groups and the combined expansion rate. For example, if any one of the following conditions is satisfied, it is determined as the corresponding two associations The dimension group meets the merge condition:

1) The correlation is greater than 0.85, that is, it is assumed that the first correlation threshold is 0.85, but the present disclosure is not limited thereto, and the first correlation threshold is a real number greater than 0 and less than 1.

2) The correlation is greater than 0.75 and the combined expansion ratio is less than 10, that is, it is assumed that the second correlation threshold is 0.75 and the first expansion ratio threshold is 10, but the present disclosure is not limited to this, and the second correlation threshold is greater than 0 and A real number less than 1, the first expansion rate threshold is a real number greater than or equal to 1.

3) The correlation is greater than 0.5 and the combined expansion ratio is less than 5, that is, it is assumed that the third correlation threshold is 0.5 and the second expansion ratio threshold is 5, but the present disclosure is not limited to this, and the third correlation threshold is greater than 0 and A real number less than 1, the second dilation rate threshold is a real number greater than or equal to 1.

1.4: If there are two associated dimension groups that satisfy the merging condition, the two associated dimension groups that satisfy the merging condition are regarded as a pair of associated dimension groups. Then, sort in reverse order according to the correlation degree of each associated dimension group pair, determine the associated dimension group pair with the highest degree of correlation as the target associated dimension group pair, merge the two associated dimension groups in the target associated dimension group pair, and merge Put the new associated dimension group into groups, and delete the two associated dimension groups in groups before merging. If there are no two associated dimension groups that satisfy the merge condition, the merge is terminated.

1.5: If the number of associated dimension groups in groups is less than or equal to the given joint dimension threshold, terminate the merge, otherwise repeat steps 1.1 to 1.5 above.

In an exemplary embodiment, constructing a data cube oriented to a target table according to an aggregation group of dimension fields may include: obtaining the row key sequence of each field in the historical query statement, where the fields include dimension fields; The joint dimension of the co-occurrence graph and the aggregated group of dimension fields are used to construct a data cube for the target table.

In an exemplary embodiment, obtaining the row key sequence of each field in the historical query statement may include: obtaining weight parameters of different field types in the historical query statement; obtaining each field and its field type in the historical query statement; The weight parameter corresponding to the field type and the query frequency of each field in the historical query statement, obtain the field weight value of each field; according to the field weight value of each field, determine the row key sequence of each field.

Specifically, each field obtained by analyzing the above historical query statement is used to calculate the field weight value of each field.

For example, suppose that for the aggregation field (group) in the dimension field, a weight parameter is set, for example, it is assumed to be 4; for the filter field (filter) in the dimension field, a weight parameter is set, for example, it is assumed to be 2. In other embodiments, a weight parameter may also be set for the indicator field, such as the above-mentioned sum, for example, it is assumed to be 1. However, the present disclosure is not limited to this, and can be set according to actual needs. Considering the frequency of fields of different field types being used in actual queries, the more frequently used field types can be set to have larger weight parameters, such as setting aggregation. The weight parameter of the field is greater than the weight parameter of the filter field, the weight parameter of the filter field is greater than the weight parameter of the indicator field, etc. At the same time, the weight parameters of each field type will not differ greatly. In this way, the field weight values of each field in the above historical query statement can be obtained as follows:

(user_data, user_id, 4),

(user_data, user_name, 4),

(user_data, partition_time, 2),

(user_data, value, 1)

For each field obtained in all historical query statements, multiply the weight parameter of the field type to which the field belongs and the query frequency of the field in all historical query statements to obtain the field weight value of the field. For example, for the field type of Assuming that the field user_id of the aggregated field appears 50 times in all historical query statements, the field weight of the field user_id is 200.

After obtaining the field weight value of each field in all historical query statements, sort each field in reverse order according to the size of the field weight value. The order of the fields obtained in the reverse order can be directly used for the design of the RowKey order (row key order).

The data processing method provided by the embodiment of the present disclosure extracts the fields involved by analyzing the historical query statements of the target table, and marks the field type of each field, thereby obtaining the dimension field and the query co-occurrence frequency between different dimension fields, Take the dimension field as a node, connect the two nodes corresponding to the two dimension fields with the query co-occurrence frequency to form an edge, and use the query co-occurrence frequency between the two dimension fields corresponding to the two nodes connected by each edge as the corresponding The initial edge weight of the edge, thus automatically constructing the dimension co-occurrence graph of the target table. On the one hand, by splitting the dimension co-occurrence graph, and by calculating the split income of each branch node, the pruning operation can be performed. Obtain the target leaf nodes that are less than or equal to the set leaf node threshold in the co-occurrence graph of the dimension, and then combine the dimension fields contained in each target leaf node into an aggregation group. A single aggregation group can be used to construct a relatively small Cube, so that a large cube construction task can be split into multiple smaller cube construction tasks, which saves computing resources and reduces the construction time. The distribution of each query request in the After removing the edges of , multiple subgraphs of the co-occurrence graph of this dimension can be obtained, and then the maximum cliques of each subgraph are obtained respectively, and all dimension fields in each maximum clique are combined to form an associated dimension group, and then according to different associated dimensions The correlation between the groups and the combined expansion rate are used to combine the two associated dimension groups that meet the combined conditions and have the highest correlation to form a joint dimension less than or equal to the joint dimension threshold, that is, to realize the highest correlation. It is further combined into a joint dimension, which reduces the size of the constructed cube and reduces the occupied storage resources, so that the final constructed cube meets the requirements of the relevant expansion rate. In addition, weight parameters of different field types can also be obtained, the field weight value of each field is determined according to the query frequency of each field and the weight parameter of the field type to which it belongs, and the RowKey order is designed according to the size of the field weight value, so that the designed RowKey order satisfies actual query requirements. After the above process, several core parameters for designing Cubes are available, including the division of aggregation groups, the design of joint dimensions, and the design of RowKey sequences, so that Cubes can be automatically constructed.

FIG. 9 schematically shows a schematic flowchart of a data processing method according to yet another embodiment of the present disclosure. As shown in FIG. 9 , the method provided by the embodiment of the present disclosure may include the following steps.

In step S910, the historical query statement of the target table is obtained, the historical query statement is analyzed, and the query co-occurrence frequency between dimension fields and different dimension fields is obtained.

As shown in Figure 9, it is assumed that the system inputs the records of historical query SQL statements on the same data source (same table or the same combination of fact table and dimension table) on Hive by the user. After analyzing the historical query statements, we obtain The query co-occurrence frequency between dimension fields and different dimension fields can also be obtained. The weight parameters of fields and different field types can also be obtained. The query frequency of each field in historical query statements, and the dimension fields are included in these fields.

In step S920, a dimension co-occurrence graph is constructed according to the dimension field and the query co-occurrence frequency between different dimension fields.

Then, each dimension field is used as each node, and then the edge is constructed according to the query co-occurrence frequency and the initial edge weight of the edge is determined, and a weighted and undirected dimension co-occurrence graph is constructed.

In step S930, the dimension co-occurrence graph is decomposed to obtain a split tree.

The dimensional co-occurrence graph is decomposed in the manner described in the above embodiment to obtain a split tree including primary leaf nodes.

In step S940, a pruning operation is performed on the split tree to determine an aggregation group.

The obtained split tree is pruned in the manner described in the above embodiment, some primary leaf nodes are reduced according to the split income, target leaf nodes are obtained, and all dimension fields in each target leaf node are combined into a single aggregation group.

In step S950, the joint dimension in the dimension co-occurrence graph is found.

By recalculating the primary edge weights of the edges of the dimensional co-occurrence graph in the above embodiment, the target edge weight is determined, the dimensional co-occurrence graph is divided into multiple subgraphs according to the target edge weight, and one or more subgraphs in each subgraph are determined respectively. Multiple maximal cliques, according to which a joint dimension with a high degree of correlation and meeting the requirements of the expansion rate is found.

In step S960, the field weight value of the field is calculated to determine the row key sequence.

The field weight value of each field can also be calculated and obtained according to the weight parameter of the field type to which each field belongs and the query frequency of each field, and the row key sequence of each field can be determined according to the size of the field weight value.

In step S970, the design and generation of the data cube is performed according to the aggregation group, the joint dimension and the row key sequence.

According to the aggregation group, union dimension and row key order obtained in the above steps, a complete Cube design can be obtained.

In the data processing method provided by the embodiment of the present disclosure, by analyzing the historical query statements of the target table, extracting all the fields involved in the historical query statements, using the query co-occurrence frequency between the dimension fields to construct a dimension co-occurrence graph, and then using the maximum cluster Algorithm to find all the largest cliques of all subgraphs in the dimension co-occurrence graph, and combine all nodes in each largest clique into a joint dimension, which further improves the correlation between dimension fields; it can also be based on weighted undirected query The graph division algorithm of co-occurrence graph (the above graph splitting process) divides the query co-occurrence graph to obtain a split tree, and obtains the final division result through pruning, namely the target leaf node, which can be directly used in the design of Cube , which can realize the automation of Cube design and reduce the design difficulty of Cube construction in large-scale dimensions. In addition, through the method of graph division, large cubes can be divided into aggregate groups or multiple small cubes, and the subtasks corresponding to each small cube can meet the needs without increasing the amount of repeated calculations. By splitting, the computing resources occupied in the Cube construction process are reduced, and the construction time is reduced, so as to achieve a balance between construction resources and construction time. In some complex businesses, there are generally very complex data models, and there are usually many data dimensions. Using the solutions provided by the embodiments of the present disclosure, a good cube design can be given based on business requirements, so that the designed cubes not only It can meet the query requirements, and can ensure the query efficiency and construction efficiency, and at the same time ensure that the data expansion rate is within an appropriate range.

In the test, for a data with 61 dimension fields and a daily data scale of about 300 million, when the solution provided by the embodiment of the present disclosure is not used, the cube is obtained by manually designing the cube and then using the optimization tool provided by the related technology. The average construction time is about 120 minutes (minutes), and the resulting data size after daily pre-calculation is about 600G. After using the solution provided by the embodiment of the present disclosure to optimize the split, the original cube is split into three cubes, and the average construction time is It is compressed to about 50 minutes, and the storage space occupied by the overall result data is about 200G. In several other medium-scale Cube optimization tasks, it also shows good results. For example, as shown in Table 1 below:

Table 1

It can be seen from the above test results that the embodiment of the present disclosure analyzes the business through historical query statements, and automatically generates the Cube design, which simplifies the user's design difficulty and optimizes the Cube design. Through graph splitting, large cubes can be split, which can reduce the construction time of cubes and compress the data size.

The methods provided by the embodiments of the present disclosure can be applied to the optimization work of OLAP data query, data platform products, etc., which can reduce the difficulty of designing OLAP cubes for users, and can automatically design cubes through analysis and optimization, which can reduce the time and resources of pre-computing tasks Occupancy can also be used in some cache design applications, and can also be used in the scenario of reverse analysis of business through query and splitting of business data. Figure 10 takes the Apache Kylin solution that generates cubes based on MapReduce precomputing and provides low-latency queries as an example for illustration.

FIG. 10 schematically shows a schematic diagram of a system architecture to which the data processing method provided by the embodiment of the present disclosure is applied. First look at the offline build part. As can be seen from Figure 10, the left side is the data source 1010, the default data source can be Apache Hive (or Kafka, RDBMS (Relational Database Management System, relational database management system), etc.), which saves the user data to be analyzed . According to the definition of the metadata (Metadata) 1024, the data cube build engine (Cube Build Engine) 1025 extracts data from the data source 1010, and builds a cube. The construction technology can be MapReduce. The constructed OLAP data cube (OLAP Cube) 1030 is stored in the storage engine on the right, and the default storage is Apache HBase.

After the offline construction is completed, the user can send SQL from the upper query system (eg, third-party APP 1040, SQL-Based tool 1050) for query analysis. Provides RESTful API (Representational StateTransfer Application Programming Interface), JDBC (Java Database Connectivity, Java Database Connectivity)/ODBC (Open Database Connectivity, Open Database Connectivity) and other interfaces for users to call. No matter which interface is used to enter, the SQL will eventually come to the REST server (REST Server) 1021, and then be transferred to the Query Engine (Query Engine) 1022 for processing. The query engine (QueryEngine) 1022 parses the SQL, generates a logical execution plan based on the relational table, and then translates it into a physical execution plan based on the cube, and queries the pre-computed generated cube based on the routing (Routing) 1023 and generates a result. The entire process does not access the original data source. If the user-submitted query statement is not predefined, an error is returned.

In the embodiment of FIG. 10, an abstraction layer is extracted for the three modules of the data source, the data cube construction engine and the cube storage, and these three modules can be extended and replaced arbitrarily. For example, Spark can be used instead of MapReduce as the data cube construction engine of Cube, and Cassandra can be used instead of HBase as the storage of data after Cube calculation.

Apache Kylin uses precomputing technology to solve the problem of limited data set size and can better support the query of massive data sets. Also benefiting from the precomputing technology, Kylin's query speed is very fast, because complex joins, aggregations and other operations have been completed during the construction of Cube. Apache Kylin can also be scaled horizontally using cluster deployment.

The method provided by the embodiments of the present disclosure may adopt the database technology in the cloud technology.

Among them, cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing and sharing of data.

Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on cloud computing business models. Cloud computing technology will become an important support. Background services of technical network systems require a lot of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the back-end system for logical processing. Data of different levels will be processed separately, and all kinds of industry data need to be strong. The system backing support can only be achieved through cloud computing.

Database, in short, can be regarded as an electronic filing cabinet—a place where electronic files are stored, and users can perform operations such as adding, querying, updating, and deleting data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of applications.

Database Management System (DBMS for short) is a computer software system designed for database management. It generally has basic functions such as storage, interception, security, and backup. The database management system can be classified according to the database model it supports, such as relational, XML (Extensible Markup Language, Extensible Markup Language); or according to the type of computer it supports, such as server clusters, mobile phones; Or classify according to the query language used, such as SQL, XQuery; or classify according to the focus of performance impulse, such as the largest scale, the highest running speed; or other classification methods. Regardless of the classification method used, some DBMSs are capable of cross-classification, for example, simultaneously supporting multiple query languages.

FIG. 11 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure. As shown in FIG. 11 , the data processing apparatus 1100 provided by this embodiment of the present disclosure may include a co-occurrence frequency obtaining unit 1110 , a dimensional co-occurrence graph constructing unit 1120 , a core node determining unit 1130 , a connected subgraph obtaining unit 1140 , and a graph splitting tree obtaining unit 1140 . unit 1150 , field aggregation group determination unit 1160 , and data cube construction unit 1170 .

In the embodiment of the present disclosure, the co-occurrence frequency obtaining unit 1110 may be configured to obtain the dimension field in the historical query statement of the target table and the query co-occurrence frequency between different dimension fields. The dimension co-occurrence graph construction unit 1120 may be configured to use the dimension field as a node, and determine the primary edge weight between different nodes according to the query co-occurrence frequency between different dimension fields to form a dimension co-occurrence graph. The core node determining unit 1130 may be configured to determine the first core node in the dimensional co-occurrence graph according to the degree of each node in the dimensional co-occurrence graph. The connected subgraph obtaining unit 1140 may be configured to obtain the first connected subgraph after deleting the first core node in the dimensional co-occurrence graph. The graph splitting tree obtaining unit 1150 may be configured to obtain a splitting tree of the dimensional co-occurrence graph according to the dimensional co-occurrence graph and its first core node and first connected subgraph. The field aggregation group determination unit 1160 may be configured to determine the aggregation group of dimension fields according to the target leaf node of the split tree. The data cube building unit 1170 may be configured to build a target table-oriented data cube according to the aggregation group of dimension fields.

The data processing device provided by the embodiment of the present disclosure, on the one hand, analyzes the historical query statement of the target table, constructs a dimension co-occurrence graph according to the dimension fields in the historical query statement and the query co-occurrence frequency between different dimension fields, and according to The degree of each node in the dimensional co-occurrence graph is used to determine the first core node in the dimensional co-occurrence graph, and then the first connected subgraph after deleting the first core node in the dimensional co-occurrence graph is obtained. According to the dimensional co-occurrence graph The graph and its first core node and the first connected subgraph are split on the dimension co-occurrence graph to obtain a split tree of the dimension co-occurrence graph, so that the above historical query statement can be determined according to the target leaf node of the split tree When using the aggregation group of the dimension field to build a data cube (Cube) oriented to the target table, the large cube is divided into aggregation groups, and each aggregation group can be used to construct various aggregation groups. Small cubes, that is, by splitting the dimensional co-occurrence graph, compresses the data scale, reduces the computing resources consumed in the cube construction process, and can shorten the construction time of the cube, achieving a balance between construction resources and construction time. . At the same time, since the dimension co-occurrence graph in the embodiment of the present disclosure is constructed according to the dimension fields in the historical query statements of the target table and the query co-occurrence frequency between different dimension fields, the dimension co-occurrence graph is split to achieve this. Splitting the construction task of a large cube into multiple small cube construction tasks can also ensure that each small cube constructed can meet the query requirements in the actual business, and will not increase the amount of repeated calculations. On the other hand, by analyzing historical query statements, the automation of cube construction is realized, the difficulty of designing cubes is simplified, and the design of cubes is optimized.

In an exemplary embodiment, the graph splitting tree obtaining unit 1150 may include: a root node forming unit, which may be used to combine nodes in the dimensional co-occurrence graph to form the root node of the splitting tree; a second core node determining unit, which may be used for According to the degree of each node in the first connected subgraph, the second core node in the first connected subgraph is determined; the first preliminary leaf node obtains the unit, which can be used to delete the second core node in the first connected subgraph After that, if there is no node in the first connected subgraph, the first core node and the second core node are combined as the primary leaf node of the split tree.

In an exemplary embodiment, the graph-splitting tree obtaining unit 1150 may further include: a second connected subgraph obtaining unit, which may be configured to, if after deleting the second core node in the first connected subgraph, exist in the first connected subgraph nodes, and the nodes after deleting the second core node are connected, then the second connected subgraph after deleting the second core node in the first connected subgraph is obtained; the third core node determination unit can be used if If the current number of split layers of the split tree is less than or equal to the threshold of the number of split layers, the third core node in the second connected subgraph is determined according to the degree of each node in the second connected subgraph; the second primary leaf node obtains the unit, It can be used to combine the first core node, the second core node and the third core node as the initial stage of the split tree if there is no node in the second connected subgraph after deleting the third core node in the second connected subgraph. leaf node.

In an exemplary embodiment, the field aggregation group determination unit 1160 may include: a division benefit obtaining unit, which may be used to obtain the division revenue of each branch node in the split tree; The size of the split income and the threshold of the leaf node, the pruning operation is performed on the split tree; the target leaf node determination unit can be used to use the primary leaf node retained after the split tree pruning operation as the target leaf node, and the target leaf node The number is less than or equal to the leaf node threshold; the aggregation group determines the unit, which can be used for the aggregation group that uses the target leaf node as the dimension field.

In an exemplary embodiment, each branch node in the split tree may include a first branch node, and the first branch node may include each first child node of the first parent node. Wherein, the unit for obtaining the split income may include: a unit for obtaining the combined size of the dimensions of the first parent node, which can be used to obtain the combined size of dimensions of each dimension field in the first parent node; a unit for obtaining the co-occurrence frequency by querying the first parent node, which can be used for Obtain the query co-occurrence frequency between each dimension field in the first parent node; the root node query co-occurrence frequency obtaining unit can be used to obtain the query co-occurrence frequency between each dimension field in the root node; the first parent node query construction A cost acquisition unit, which can be used to combine the dimensions of each dimension field in the first parent node, the query co-occurrence frequency between each dimension field in the first parent node, and the query co-occurrence frequency between each dimension field in the root node. , to obtain the query construction cost of the first parent node; the unit for obtaining the dimension combination size of the first child node can be used to obtain the dimension combination size of each dimension field in each first child node; the first child node query co-occurrence frequency to obtain the unit, It can be used to obtain the query co-occurrence frequency between each dimension field in each first sub-node; the first sub-node query construction cost obtaining unit can be used to obtain the unit according to the dimension combination size of each dimension field in each first sub-node, each The query co-occurrence frequency between each dimension field in the first child node and the query co-occurrence frequency between each dimension field in the root node are used to obtain the query construction cost of each first child node; the first branch node splits the revenue to obtain the unit, It can be used to obtain the split benefit of the first branch node according to the query construction cost of the first parent node and the query construction cost of each first child node.

In an exemplary embodiment, the data processing apparatus 1100 may further include: a target edge weight obtaining unit, which may be configured to obtain the target edge weight of each edge of the dimensional co-occurrence graph according to the primary edge weight of each edge of the dimensional co-occurrence graph ; Co-occurrence graph sub-graph obtaining unit, which can be used to remove the edges whose target edge weight is lower than the weight threshold to obtain each sub-graph of the dimension co-occurrence graph; Associated dimension group obtaining unit can be used to obtain each sub-graph. The largest clique, which forms each associated dimension group according to the dimension fields in each largest clique; the correlation degree expansion rate obtaining unit can be used to obtain the correlation degree and combined expansion rate between each associated dimension group; the joint dimension obtaining unit can be used for According to the correlation between each associated dimension group and the combined expansion rate, each associated dimension group is merged to obtain the joint dimension of the dimension co-occurrence graph, and the number of joint dimensions is less than or equal to the joint dimension threshold.

In an exemplary embodiment, the dimensional co-occurrence graph may include a first edge and first and second nodes of the first edge. The target edge weight obtaining unit may include: a first query frequency obtaining unit, which can be used to obtain the first query frequency of the dimension field corresponding to the first node in the historical query statement; a second query frequency obtaining unit, which can be used to obtain The second query frequency of the dimension field corresponding to the second node in the historical query statement; the target query frequency determination unit can be used to determine the target query frequency according to the first query frequency and the second query frequency; the first target edge weight obtaining unit , which can be used to obtain the target edge weight of the first edge based on the initial edge weight of the first edge and the target query frequency.

In an exemplary embodiment, each associated dimension group may include a first associated dimension group and a second associated dimension group. Wherein, the correlation degree expansion rate obtaining unit may include: an associated dimension group dimension combination size obtaining unit, which can be used to obtain the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; the dimension combination size product unit, can be used to obtain the product result of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; the unit for merging associated dimension groups can be used to merge the first associated dimension group and the second associated dimension group, Obtain the merged associated dimension group; merge the associated dimension group dimension combination size obtaining unit, which can be used to obtain the dimension combination size of the merged associated dimension group; the associated dimension group correlation degree obtaining unit can be used to merge the associated dimension group according to the dimension combination size and The correlation between the first associated dimension group and the second associated dimension group is obtained by multiplying the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group.

In an exemplary embodiment, each associated dimension group may include a first associated dimension group and a second associated dimension group. Wherein, the correlation expansion rate obtaining unit may include: an associated dimension group dimension combination size obtaining unit, which may be used to obtain the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; a dimension combination size summation unit , which can be used to obtain the summation result of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; the unit for merging associated dimension groups can be used to merge the first associated dimension group and the second associated dimension group, to obtain the merged associated dimension group; the unit to obtain the size of the combined associated dimension group dimension combination, which can be used to obtain the size of the combined associated dimension group; The combined size and the summation result of the combined dimension size of the first associated dimension group and the dimension combined size of the second associated dimension group are used to obtain the combined expansion rate between the first associated dimension group and the second associated dimension group.

In an exemplary embodiment, the joint dimension obtaining unit may include: an association dimension group pair determination unit, which may be configured to determine, according to the correlation between the respective association dimension groups and the merge expansion rate, the associations in the respective association dimension groups that satisfy the merge condition Dimension group pair; target associated dimension group pair acquisition unit, which can be used to obtain the target associated dimension group pair with the highest correlation in the associated dimension group pair; Associated dimension group, obtains the target merged associated dimension group, and deletes the associated dimension group in the target associated dimension group pair; the joint dimension determination unit can be used if the number of the target merged associated dimension group and the unmerged associated dimension group is less than or equal to If the joint dimension threshold is set, the target merged associated dimension group and the unmerged associated dimension group are respectively used as the joint dimension of the dimension co-occurrence graph.

In an exemplary embodiment, the associated dimension group pair determination unit may include: a merging condition determination unit, which may be configured to: if the correlation between the associated dimension groups is greater than the first correlation threshold; or if the correlation between the associated dimension groups is greater than the second correlation threshold and the combined expansion rate is less than the first expansion rate threshold; or if the correlation between the associated dimension groups is greater than the third correlation threshold and the combined expansion rate is less than the second expansion rate threshold, then determine the corresponding associated dimension A group is a pair of associated dimension groups that meet the merging conditions; wherein the first correlation threshold is greater than the second correlation threshold, the second correlation threshold is greater than the third correlation threshold, and the first expansion rate threshold is greater than the second expansion rate threshold.

In an exemplary embodiment, the data cube construction unit 1170 may include: a row key sequence determination unit, which may be used to obtain the row key sequence of each field in the historical query statement, the fields including dimension fields; a data cube generation unit, which may be used to The row key order of each field, the joint dimension of the dimension co-occurrence graph, and the aggregation group of dimension fields are used to construct a data cube for the target table.

In an exemplary embodiment, the row key sequence determination unit may include: a field type weight parameter obtaining unit, which can be used to obtain weight parameters of different field types in the historical query statement; each field and its field type; the field weight value obtaining unit can be used to obtain the field weight value of each field according to the weight parameter corresponding to the field type of each field and the query frequency of each field in the historical query statement; the row key sequence The design unit can be used to determine the row key sequence of each field according to the field weight value of each field.

For other contents of the data processing apparatus in the embodiments of the present disclosure, reference may be made to the above-mentioned embodiments.

It should be noted that although several units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, in accordance with embodiments of the present disclosure, the features and functions of two or more units described above may be embodied in a single unit. Conversely, the features and functions of one unit described above may be further subdivided to be embodied by multiple units.

Referring to FIG. 12 below, it shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic device shown in FIG. 12 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

12 , an electronic device provided by an embodiment of the present disclosure may include: a processor 1201 , a communication interface 1202 , a memory 1203 , and a communication bus 1204 .

The processor 1201 , the communication interface 1202 and the memory 1203 communicate with each other through the communication bus 1204 .

Optionally, the communication interface 1202 may be an interface of a communication module, such as an interface of a GSM (Global System for Mobile communications, global system for mobile communications) module. The processor 1201 is used to execute programs. The memory 1203 is used to store programs. A program may include a computer program including computer operating instructions. Wherein, the program may include: the program of the game client.

The processor 1201 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present disclosure.

The memory 1203 may include a high-speed RAM (random access memory, random access memory) memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

The program can be specifically used to: obtain the dimension field in the historical query statement of the target table and the query co-occurrence frequency between different dimension fields; take the dimension field as a node, and determine the difference according to the query co-occurrence frequency between different dimension fields The first-order edge weights between nodes form a dimensional co-occurrence graph; according to the degrees of each node in the dimensional co-occurrence graph, determine the first core node in the dimensional co-occurrence graph; obtain and delete the first core node in the dimensional co-occurrence graph After the first connected subgraph; according to the dimension co-occurrence graph and its first core node and the first connected sub-graph, the split tree of the dimension co-occurrence graph is obtained; according to the target leaf node of the split tree, the aggregation group of the dimension field is determined; Aggregate groups of dimension fields build a data cube for the target table.

According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods provided in various optional implementations of the above-described embodiments.

It should be understood that any number of elements in the drawings of the present disclosure is for illustration rather than limitation, and any designation is for distinction only and does not have any limiting meaning.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (15)

1. a data processing method, is characterized in that, comprises: Obtain the dimension field in the historical query statement of the target table and the query co-occurrence frequency between different dimension fields; The dimension field is used as a node, and the primary edge weight between different nodes is determined according to the query co-occurrence frequency between different dimension fields, so as to form a dimension co-occurrence graph; determining the first core node in the dimension co-occurrence graph according to the degree of each node in the dimension co-occurrence graph; obtaining the first connected subgraph after deleting the first core node in the dimension co-occurrence graph; obtaining a split tree of the dimensional co-occurrence graph according to the dimensional co-occurrence graph, its first core node and the first connected subgraph; Determine the aggregation group of the dimension field according to the target leaf node of the split tree; A data cube oriented to the target table is constructed according to the aggregation group of the dimension fields. 2. The method according to claim 1, wherein, according to the dimensional co-occurrence graph and its first core node and the first connected subgraph, obtaining a split tree of the dimensional co-occurrence graph, comprising: combining the nodes in the dimension co-occurrence graph to form the root node of the split tree; determining the second core node in the first connected subgraph according to the degree of each node in the first connected subgraph; If there is no node in the first connected subgraph after deleting the second core node in the first connected subgraph, combine the first core node and the second core node as the split The primary leaf node of the tree. 3. The method according to claim 2, wherein, according to the dimensional co-occurrence graph and its first core node and the first connected subgraph, obtaining a split tree of the dimensional co-occurrence graph, further comprising: : If after deleting the second core node in the first connected subgraph, there are nodes in the first connected subgraph, and the nodes after deleting the second core node are connected, then the deletion is obtained. the second connected subgraph after the second core node in the first connected subgraph; If the current number of split layers of the split tree is less than or equal to the threshold of the number of split layers, determining the third core node in the second connected subgraph according to the degree of each node in the second connected subgraph; If there is no node in the second connected subgraph after deleting the third core node in the second connected subgraph, then combine the first core node, the second core node and the first core node. The three core nodes serve as the primary leaf nodes of the split tree. 4. The method according to claim 2 or 3, wherein determining the aggregation group of the dimension field according to the target leaf node of the split tree, comprising: Obtain the split income of each branch node in the split tree; According to the size of the split income of each branch node in the split tree and the leaf node threshold, the pruning operation is performed on the split tree; The primary leaf node retained after the split tree pruning operation is used as the target leaf node, and the number of the target leaf node is less than or equal to the leaf node threshold; The target leaf node is used as the aggregation group of the dimension field. 5. The method according to claim 4, wherein each branch node in the split tree comprises a first branch node, and the first branch node comprises each first child node of the first parent node; wherein, obtaining The split income of each branch node in the split tree includes: obtaining the combined dimension size of each dimension field in the first parent node; obtaining the query co-occurrence frequency between each dimension field in the first parent node; obtaining the query co-occurrence frequency between each dimension field in the root node; According to the dimension combination size of each dimension field in the first parent node, the query co-occurrence frequency between each dimension field in the first parent node and the query co-occurrence frequency between each dimension field in the root node, obtaining the query construction cost of the first parent node; Obtain the dimension combination size of each dimension field in each first child node; Obtain the query co-occurrence frequency between each dimension field in each first child node; According to the dimension combination size of each dimension field in each first subnode, the query co-occurrence frequency between each dimension field in each first subnode, and the query co-occurrence frequency between each dimension field in the root node, obtain each The query construction cost of the first child node; The split benefit of the first branch node is obtained according to the query construction cost of the first parent node and the query construction cost of each first child node. 6. The method of claim 1, further comprising: Obtain the target edge weights of each edge of the dimensional co-occurrence graph according to the primary edge weights of each edge of the dimensional co-occurrence graph; Remove the edge whose weight of the target edge is lower than the weight threshold to obtain each subgraph of the dimension co-occurrence graph; Obtain the largest clique in each subgraph, and form each associated dimension group according to the dimension field in each largest clique; Obtain the correlation and combined expansion rate between each associated dimension group; According to the correlation degree and the combined expansion rate of each associated dimension group, each associated dimension group is merged to obtain a joint dimension of the dimension co-occurrence graph, and the number of the joint dimension is less than or equal to the joint dimension threshold. 7. The method according to claim 6, wherein the dimensional co-occurrence graph comprises a first edge and a first node and a second node of the first edge; wherein, according to the dimensional co-occurrence graph The initial edge weight of each edge, to obtain the target edge weight of each edge of the dimension co-occurrence graph, including: obtaining the first query frequency of the dimension field corresponding to the first node in the historical query statement; obtaining the second query frequency of the dimension field corresponding to the second node in the historical query statement; determining a target query frequency according to the first query frequency and the second query frequency; The target edge weight of the first edge is obtained according to the initial edge weight of the first edge and the target query frequency. 8. The method according to claim 6, wherein each associated dimension group comprises a first associated dimension group and a second associated dimension group; wherein, obtaining the correlation between each associated dimension group comprises: obtaining the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; obtaining a product result of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; Merging the first associated dimension group and the second associated dimension group to obtain a merged associated dimension group; obtaining the dimension combination size of the merged associated dimension group; The first associated dimension group and the The correlation between the second associated dimension groups. 9. The method according to claim 6, wherein each associated dimension group comprises a first associated dimension group and a second associated dimension group; wherein, obtaining the combined expansion rate between each associated dimension group comprises: obtaining the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; obtaining a summation result of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group; Merging the first associated dimension group and the second associated dimension group to obtain a merged associated dimension group; obtaining the dimension combination size of the merged associated dimension group; According to the dimension combination size of the merged associated dimension group and the summation result of the dimension combination size of the first associated dimension group and the dimension combination size of the second associated dimension group, the first associated dimension group and the all associated dimension groups are obtained. The combined inflation rate between the second associated dimension groups. 10 . The method according to claim 6 , wherein, according to the correlation and the combined expansion rate between the respective associated dimension groups, the respective associated dimension groups are merged to obtain the joint dimension of the dimension co-occurrence graph, comprising: 11 . : According to the correlation between each associated dimension group and the combined expansion rate, determine the associated dimension group pair in each associated dimension group that satisfies the merge condition; obtaining the target associated dimension group pair with the highest degree of correlation in the associated dimension group pair; Merging the associated dimension groups in the target associated dimension group pair, obtaining a target merged associated dimension group, and deleting the associated dimension group in the target associated dimension group pair; If the number of the target merged associated dimension group and the unmerged associated dimension group is less than or equal to the combined dimension threshold, the target merged associated dimension group and the unmerged associated dimension group are respectively used as the dimension co-occurrence map the joint dimension of . 11. The method according to claim 6, wherein constructing a data cube oriented to the target table according to the aggregation group of the dimension field, comprising: Obtain the row key sequence of each field in the historical query statement, where the field includes the dimension field; A data cube oriented to the target table is constructed according to the row key sequence of each field, the joint dimension of the dimension co-occurrence graph, and the aggregation group of the dimension fields. 12. The method according to claim 11, wherein obtaining the row key sequence of each field in the historical query statement comprises: Obtain the weight parameters of different field types in the historical query statement; Obtain each field and its field type in the historical query statement; Obtain the field weight value of each field according to the weight parameter corresponding to the field type of each field and the query frequency of each field in the historical query statement; Determine the row key order of each field according to the field weight value of each field. 13. A data processing device, comprising: The co-occurrence frequency obtaining unit is used to obtain the dimension field in the historical query statement of the target table and the query co-occurrence frequency between different dimension fields; A dimension co-occurrence graph construction unit, configured to use the dimension field as a node, and determine the primary edge weight between different nodes according to the query co-occurrence frequency between different dimension fields to form a dimension co-occurrence graph; a core node determination unit, configured to determine the first core node in the dimensional co-occurrence graph according to the degree of each node in the dimensional co-occurrence graph; a connected subgraph obtaining unit for obtaining the first connected subgraph after deleting the first core node in the dimension co-occurrence graph; A graph splitting tree obtaining unit, configured to obtain a splitting tree of the dimensional co-occurrence graph according to the dimensional co-occurrence graph and its first core node and the first connected subgraph; a field aggregation group determination unit, configured to determine the aggregation group of the dimension field according to the target leaf node of the split tree; A data cube construction unit, configured to construct a data cube oriented to the target table according to the aggregation group of the dimension fields. 14. A computer-readable storage medium, characterized in that a computer program is stored thereon, and when the program is executed by a processor, the data processing method according to any one of claims 1 to 12 is implemented. 15. An electronic device, comprising: at least one processor; A storage device configured to store at least one program that, when executed by the at least one processor, causes the at least one processor to implement the data processing method according to any one of claims 1 to 12.

Images