CN115168361A - Label management method and device - Google Patents

Abstract

The invention relates to the technical field of label management, and provides a label management method and a device, wherein the label management method comprises the following steps: creating a main body object, and correspondingly creating a label storage table in a ClickHouse warehouse by a back end; acquiring an external data source, mapping a database table structure and data to be marked into a CLickHouse through a FLinkX, and associating the database table structure and the data with a main object; creating a static label under the subject object; creating a dynamic label under the main object through SQL or a dragging mode; and issuing the dynamic label and the static label under the subject object into an API interface. According to the label management method and device provided by the exemplary embodiment of the invention, the production speed of label data can be increased; the ready speed of the label data is accelerated; reducing the average response time length of the query request; the method supports the quasi-real-time updating of the tag data; the label expression and the query SQL are friendly to users, and the maintainability of the system is improved; marking and inquiring are processed in the ClickHouse, and hardware resources can be saved by half.

Description

A label management method and device

technical field

The invention relates to the technical field of label management, in particular to a label management method and device.

Background technique

Label is a semantic expression of platform business data, which can be either the basic attribute of the object or a certain characteristic attribute of the description object obtained after calculation and analysis of the original data. Tags are widely used in user portraits, product portraits and other application scenarios.

The existing tag management platform defines the calculation process of tags on the interface in a visual way, and uses big data spark, hive, hbase and other technologies to calculate the tag data of compound business requirements in pb-level data, and further group users and users. Labels provide the data base. The label management platform is a label life cycle management platform. The overall structure is divided into three layers: label management layer, label library, and label service layer. The life cycle of tags is divided into creation, storage and query. The label management layer is supported by the label engine, and provides visual label rule definition modes such as drag-and-drop, circle selection, and lightweight script for demand analysts. Management and measurement statistics functions to realize the full life cycle management of labels. The tag library is the carrier of tags, realizes the precipitation of value data, and provides resource support for external services of tags. The label service layer includes a series of terminal application-oriented services such as label API service and dynamic marking service to realize the rapid service of value data.

At present, the field is mainly based on data synchronization engine (DataX) + big data storage (Hive) + search engine (ElasticSearch) to realize data synchronization, data storage, and label search in the label creation process. In practical applications, there are many steps in the label creation process. , takes a long time, and has the following defects:

1. The data synchronization efficiency is low. Creating tags relies on the Datax platform to synchronize the original data to the tag storage engine library. In the scenario of large data volume, the synchronization speed is slow; It increases the complexity of the system and reduces the efficiency of label data usage. The number of labels under the main body determines the number of columns in the static label table. When the number of columns in the static label table is too large, data synchronization becomes a bottleneck.

2. The real-time data is poor. Before creating a label, you need to preview some label data according to the creation rules. The label data is provided to the external platform by creating an API service, which requires that the query response time should not be too slow. Labels are stored in the hive library. Taking user labels as an example, the storage table structure is as follows: create a portrait table with userid as the primary key, and other fields in the table as the feature fields of the portrait, perform the in operation on the circled people and the portrait table, and then In the group by operation, when adding or deleting feature fields, the table structure of the portrait table needs to be modified; when the number of people to be circled is relatively large, the group by operation of a large record set is involved, and the execution performance of the hive statement is poor and the execution delay is high. , cannot quickly obtain query results, and is not suitable for real-time scenarios. On the other hand, the generated tags are retrieved by storing the tags in the large-width table of ElasticSearch. The structure of the large-width table is: when inserting data into the large-width table, it is necessary to wait for the business data to be ready before running the association. Table operations and then insert the associated results into ElasticSearch. It is often encountered that the task of a business party is delayed, so that the associated task inserted into ElasticSearch cannot be executed, and the operator cannot use the latest portrait data in time.

3. Data redundancy, data rules occupy a row of storage space for the value of each tag corresponding to each object, resulting in a large number of redundancy of data items such as tag name and tag value, and tag marking will be in the static tag table and dynamic tag table. Store multiple pieces of redundant data.

4. The label semantics are monotonous, and labels can only be created based on the original static data. It does not have the function of creating new labels based on the aggregation operation of existing labels. The range of features that a single label can describe is small. The marking process completely depends on the SQL statement of the database table. A subject object can only select the same database under one data source, and cannot be used for associated query marking of different database tables from different sources.

Therefore, how to construct tags of various dimensions and semantics and implement a tag management platform that can be quickly queried in real time is an urgent problem to be solved.

Therefore, how to provide an efficient and widely applicable label management method has become an urgent technical problem to be solved.

SUMMARY OF THE INVENTION

In view of this, the present invention mainly solves:

On the one hand, the present invention provides a label management method, comprising:

Step S1: Create a main object, and create a tag storage table in the ClickHouse warehouse correspondingly in the back end;

Step S2: Obtain an external data source, map the database table structure and data to be marked to CLickHouse through FLinkX, and associate with the main object;

Step S3: create a static label under the main object;

Step S4: create a dynamic label through SQL or drag and drop mode under the main object;

Step S5: Publish the dynamic tags and static tags under the main object as API interfaces. Further, in step S1 of the tag management method of the present invention, the tag storage table includes a static tag storage table and a dynamic tag storage table, wherein the static tag storage table is used to store the data corresponding to the basic attributes of the object, and the dynamic tag storage table is used to store The tags that can describe a batch of objects are calculated and aggregated by static tags.

Further, step S2 of the label management method of the present invention includes:

Step S21: acquiring and saving the external data source, and selecting a data table from the external data source as the label data source table;

Step S22: Obtain the field information in the tag data source table and map it to the ClickHouse field type;

Step S23: build a table statement according to the field set, build a target table and execute the table building statement in the target table;

Step S24: Assemble the tag data source table and target table into FlinkX task execution parameters and submit them to ClickHouse for data synchronization.

Further, step S3 of the label management method of the present invention includes:

Step S31: select one or more columns of the added data source table as the label data source;

Step S32: adding a static label column to the static label storage table;

Step S33: assemble the marking SQL statement;

Step S34: copy the structure of the static label storage table to create a new table;

Step S35: execute the marking SQL statement, write the label data into the new table, and delete the copied static label storage table;

Step S36: Modify the name of the new table to the name of the copied static label storage table.

Further, step S4 of the tag management method of the present invention includes: performing a set operation on static tags in combination with SQL operators to create dynamic tags.

Further, step S4 of the label management method of the present invention includes:

Step S41: establish a dynamic label creation rule;

Step S42: the front end is assembled into an expression in json format, and the back end converts the json into an object and verifies whether the expression conforms to the dynamic label creation rule;

Step S43: perform a set operation on the static label in combination with the SQL operator to create a dynamic label; parse the expression, construct the marking SQL of the dynamic label and execute the marking SQL of the dynamic label;

Step S44: Check the execution state of the marking SQL of the dynamic label, and produce the distribution of the value range for the dynamic label.

Further, step S4 of the label management method of the present invention further includes: creating a new dynamic label by dragging the dynamic label and the static label and combining the judgment condition and the filtering condition.

Further, step S5 of the label management method of the present invention includes: according to the dynamic label or static label under the main object as an input parameter or an output parameter, dragging to generate an API interface for a third party to call.

Further, step S5 of the label management method of the present invention further includes: automatically bringing out the static label storage table and dynamic label storage table and fields corresponding to the main object by dragging and dropping the main object to the SQL editing page, and writing SQL to generate an API interface, For third-party platforms to call.

In another aspect, the present invention provides a label management device, comprising:

The tag storage table creation module is used to create the main object, and the back end creates a tag storage table in the ClickHouse warehouse correspondingly, wherein the static tag storage table is used to store the data corresponding to the basic attributes of the object, and the dynamic tag storage table is used to store the static tags Labels that can describe a batch of objects obtained by calculation and aggregation;

The data synchronization module is used to obtain and save the external data source, select a data table from the external data source as the label data source table; obtain the field information in the label data source table and map it to the ClickHouse field type; build the table according to the field group statement, create the target table and execute the table creation statement in the target table; assemble the tag data source table and target table into FlinkX task execution parameters and submit them to ClickHouse for data synchronization;

The static label creation module is used to select one or more columns of the added data source table as the label data source; add a static label column to the static label storage table; assemble the marking SQL statement; copy the structure of the static label storage table to create a new table ;Execute the marking SQL statement, write the label data into the new table, delete the copied static label storage table; change the name of the new table table to the name of the copied static label storage table;

The dynamic label creation module is used to establish dynamic label creation rules; the front end is assembled into an expression in json format, and the back end converts the json into an object and verifies whether the expression conforms to the dynamic label creation rules; combined with SQL operators, the static label is processed. Set operation, create dynamic tags; parse expressions, construct dynamic tag marking SQL and execute dynamic tag marking SQL; check the execution status of dynamic tag marking SQL, and produce value range distribution for dynamic tags; also used for Create new dynamic tags by dragging dynamic tags and static tags and combining judgment conditions and filter conditions;

The label publishing module is used to generate an API interface by dragging and dropping according to the dynamic label or static label under the main object as an input or output parameter for third-party calls; it is also used to automatically bring out the main object by dragging the main object to the SQL editing page For the corresponding static label storage table and dynamic label storage table and fields, write SQL to generate API interfaces for third-party platforms to call.

The label management method and device of the present invention have the following beneficial effects:

1) Speed up the production of label data by constructing label data in parallel;

2) Speed up the readiness of label data by concurrently importing HDFS files into ClickHouse;

3) Reduce the average response time of query requests;

4) Support quasi-real-time update of tag data;

5) Tag expressions and query SQL are friendly to users and improve the maintainability of the system;

6) Marking and query are processed in ClickHouse, which can save half of hardware resources.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a flowchart of a label management method according to an exemplary first embodiment of the present invention.

FIG. 2 is a flowchart of a label management method according to an exemplary second embodiment of the present invention.

FIG. 3 is a flowchart of a label management method according to an exemplary third embodiment of the present invention.

FIG. 4 is a flowchart of a label management method according to an exemplary fourth embodiment of the present invention.

FIG. 5 is a structural diagram of a label management apparatus according to an exemplary eighth embodiment of the present invention.

Detailed ways

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict; and, based on the embodiments in the present disclosure, those of ordinary skill in the art can obtain the results obtained without creative work. All other embodiments fall within the protection scope of the present disclosure.

It is noted that various aspects of embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is illustrative only. Based on this disclosure, those skilled in the art should appreciate that an aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method may be practiced using any number of the aspects set forth herein. Additionally, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.

FIG. 1 is a flowchart of a label management method according to an exemplary first embodiment of the present invention. As shown in FIG. 1 , the method of this embodiment includes:

Step S1: Create a main object, and create a tag storage table in the ClickHouse warehouse correspondingly in the back end;

Step S2: Obtain an external data source, map the database table structure and data to be marked to CLickHouse through FLinkX, and associate with the main object;

Step S3: create a static label under the main object;

Step S4: create a dynamic label through SQL or drag and drop mode under the main object;

Step S5: Publish the dynamic tags and static tags under the main object as API interfaces.

In step S1 of the method in this embodiment, the tag storage table includes a static tag storage table and a dynamic tag storage table, wherein the static tag storage table is used to store the data corresponding to the basic attributes of the object, and the dynamic tag storage table is used to store the data calculated by the static tag. , Aggregated tags that describe a batch of objects.

FIG. 2 is a flowchart of a label management method according to an exemplary second embodiment of the present invention. This embodiment is a preferred embodiment of the method shown in FIG. 1 . As shown in FIG. 2 , step S2 of the method of this embodiment is: include:

Step S21: acquiring and saving the external data source, and selecting a data table from the external data source as the label data source table;

Step S22: Obtain the field information in the tag data source table and map it to the ClickHouse field type;

Step S23: build a table statement according to the field set, build a target table and execute the table building statement in the target table;

Step S24: Assemble the tag data source table and target table into FlinkX task execution parameters and submit them to ClickHouse for data synchronization.

FIG. 3 is a flowchart of a method for label management according to an exemplary third embodiment of the present invention. This embodiment is a preferred embodiment of the method shown in FIG. 1 . As shown in FIG. 3 , step S3 of the method of this embodiment is: include:

Step S31: select one or more columns of the added data source table as the label data source;

Step S32: adding a static label column to the static label storage table;

Step S33: assemble the marking SQL statement;

Step S34: copy the structure of the static label storage table to create a new table;

Step S35: execute the marking SQL statement, write the label data into the new table, and delete the copied static label storage table;

Step S36: Modify the name of the new table to the name of the copied static label storage table.

FIG. 4 is a flowchart of a label management method according to an exemplary fourth embodiment of the present invention. This embodiment is a preferred embodiment of the method shown in FIG. 1 . As shown in FIG. 4 , step S4 of the method of this embodiment is: Including: combining SQL operators to perform set operations on static tags to create dynamic tags.

Step S4 of the method in this embodiment includes:

Step S41: establish a dynamic label creation rule;

Step S42: the front end is assembled into an expression in json format, and the back end converts the json into an object and verifies whether the expression conforms to the dynamic label creation rule;

Step S43: perform a set operation on the static label in combination with the SQL operator to create a dynamic label; parse the expression, construct the marking SQL of the dynamic label and execute the marking SQL of the dynamic label;

Step S44: Check the execution state of the marking SQL of the dynamic label, and produce the distribution of the value range for the dynamic label.

An exemplary fifth embodiment of the present invention provides a label management method. This embodiment is a preferred embodiment of the method shown in FIG. 1 . Step S4 of the method in this embodiment further includes: dragging a dynamic label and a static label and combining Judgment conditions and filter conditions to create new dynamic tags. In the label management method of this embodiment, a label can be dragged and dropped through a visual editing interface, and a new label can be assembled by graphically setting and, or, and equal connection conditions and judging conditions such as size and comparison.

An exemplary sixth embodiment of the present invention provides a method for managing tags. This embodiment is a preferred embodiment of the method shown in FIG. 1 . Step S5 of the method in this embodiment includes: taking a dynamic tag or a static tag under a subject object as a Input or output parameters, drag and drop to generate API interfaces for third-party calls.

An exemplary seventh embodiment of the present invention provides a tag management method. This embodiment is a preferred embodiment of the method shown in FIG. 1 . Step S5 of the method in this embodiment includes: dragging a subject object to a SQL editing page, automatically Bring out the static label storage table and dynamic label storage table and fields corresponding to the main object, and write the SQL generation API interface for the third-party platform to call.

FIG. 5 is a structural diagram of a label management apparatus according to an exemplary eighth embodiment of the present invention. As shown in FIG. 5 , the label management apparatus of this embodiment includes:

The tag storage table creation module is used to create the main object, and the back end creates a tag storage table in the ClickHouse warehouse correspondingly, wherein the static tag storage table is used to store the data corresponding to the basic attributes of the object, and the dynamic tag storage table is used to store the static tags Labels that can describe a batch of objects obtained by calculation and aggregation;

The data synchronization module is used to obtain and save the external data source, select a data table from the external data source as the label data source table; obtain the field information in the label data source table and map it to the ClickHouse field type; build the table according to the field group statement, create the target table and execute the table creation statement in the target table; assemble the tag data source table and target table into FlinkX task execution parameters and submit them to ClickHouse for data synchronization;

The static label creation module is used to select one or more columns of the added data source table as the label data source; add a static label column to the static label storage table; assemble the marking SQL statement; copy the structure of the static label storage table to create a new table ;Execute the marking SQL statement, write the label data into the new table, delete the copied static label storage table; change the name of the new table table to the name of the copied static label storage table;

The dynamic label creation module is used to establish dynamic label creation rules; the front end is assembled into an expression in json format, and the back end converts the json into an object and verifies whether the expression conforms to the dynamic label creation rules; combined with SQL operators, the static label is processed. Set operation, create dynamic tags; parse expressions, construct dynamic tag marking SQL and execute dynamic tag marking SQL; check the execution status of dynamic tag marking SQL, and produce value range distribution for dynamic tags; also used for Create new dynamic tags by dragging dynamic tags and static tags and combining judgment conditions and filter conditions;

The label publishing module is used to generate an API interface by dragging and dropping according to the dynamic label or static label under the main object as an input or output parameter for third-party calls; it is also used to automatically bring out the main object by dragging the main object to the SQL editing page For the corresponding static label storage table and dynamic label storage table and fields, write SQL to generate API interfaces for third-party platforms to call.

In practical application, the label management device of this embodiment has the following characteristics:

1) By constructing label data in parallel, the production speed of label data is accelerated; the shortage of data synchronization needs to be queued when it is considered that it exceeds a certain number.

2) By concurrently importing HDFS files into ClickHouse, and directly marking with hdfs directly through the ClickHouse table engine to speed up the readiness of label data.

3) Through single-table and multi-table association queries, the average response time of query requests is less than 2 seconds, and the complex query is less than 5 seconds.

4) Support the quasi-real-time update of label data; mark and query directly in ClickHouse, and generate api results in real time; avoid the shortage of the existing technology, which can only be queried through the api interface on the second day after the marking data is first synchronized to the es.

5) Tag expressions and query SQL are friendly to users and improve the maintainability of the system; through columnar storage, logic calculations based on bitmap intersection, union and difference set processing scripts are concise and efficient. It avoids the shortage of complex row-column conversion in the back-end query SQL of the prior art.

6) Marking and query are processed in ClickHouse, which can save half of hardware resources compared to the method of storing data in hive first and publishing API after synchronizing es.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical scope disclosed by the present invention can easily think of changes or substitutions. All should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. A label management method, characterized in that the label management method comprises:

step S1: creating a main body object, and correspondingly creating a label storage table in a ClickHouse warehouse by a back end;

step S2: acquiring an external data source, mapping a database table structure and data to be marked into a CLickHouse through FLinkX, and associating the database table structure and the data with a main object;

and step S3: creating a static label under the subject object;

and step S4: creating a dynamic label under the main object through SQL or a dragging mode;

step S5: and publishing the dynamic label and the static label under the subject object as an API interface.

2. The tag management method according to claim 1, wherein in step S1, the tag storage table includes a static tag storage table and a dynamic tag storage table, wherein the static tag storage table is used for storing data corresponding to basic attributes of the objects, and the dynamic tag storage table is used for storing tags that are obtained by static tag calculation and aggregation and can describe a batch of objects.

3. The tag management method according to claim 1, wherein step S2 comprises:

step S21: acquiring and storing an external data source, and selecting a data sheet from the external data source as a tag data source sheet;

step S22: acquiring field information in a tag data source table and mapping the field information into a click House field type;

step S23: building a table building statement according to the fields, building a target table and executing the table building statement in the target table;

step S24: and assembling the tag data source table and the target table into a FlinkX task execution parameter and submitting the FlinkX task execution parameter to a ClickHouse for data synchronization.

4. The tag management method according to claim 1, wherein step S3 comprises:

step S31: selecting one or more columns of the added data source table as a label data source;

step S32: adding a static label column in a static label storage table;

step S33: assembling marking SQL sentences;

step S34: copying the structure of a static tag storage table, and creating a new table;

step S35: executing a marking SQL sentence, writing the tag data into a new table, and deleting the copied static tag storage table;

step S36: the new table name is modified to the name of the static tag storage table being copied.

5. The tag management method according to claim 1, wherein step S4 comprises: and performing set operation on the static tags by combining SQL operators to create dynamic tags.

6. The tag management method according to claim 5, wherein step S4 comprises:

step S41: establishing a dynamic label establishing rule;

step S42: the front end is assembled into an expression in a json format, the back end converts the json into an object and checks whether the expression meets the dynamic label creation rule or not;

step S43: performing set operation on the static tags by combining SQL operators to create dynamic tags; analyzing the expression, constructing marking SQL of the dynamic label and executing the marking SQL of the dynamic label;

step S44: and checking the execution state of the marking SQL of the dynamic label to produce the value range distribution condition of the dynamic label.

7. The tag management method according to claim 1, wherein step S4 further comprises: and creating a new dynamic label by dragging the dynamic label and the static label and combining the judgment condition and the filtering condition.

8. The tag management method according to claim 1, wherein step S5 comprises: and dragging and generating an API interface for a third party to call according to the dynamic label or the static label under the main object as the input parameter or the output parameter.

9. The tag management method according to claim 1, wherein step S5 further comprises: and automatically taking out the static label storage table and the dynamic label storage table and the fields corresponding to the subject object by dragging the subject object to the SQL editing page, and writing the SQL to generate an API interface for a third-party platform to call.

10. A tag management apparatus, characterized in that the tag management apparatus comprises:

the system comprises a tag storage table creating module, a tag storage module and a tag matching module, wherein the tag storage table creating module is used for creating a main object, and a tag storage table is correspondingly created in a ClickHouse warehouse by a back end, a static tag storage table is used for storing data corresponding to basic attributes of the object, and a dynamic tag storage table is used for storing tags which can describe a batch of objects and are obtained through static tag calculation and aggregation;

the data synchronization module is used for acquiring and storing external data sources and selecting one data sheet from the external data sources as a tag data source sheet; acquiring field information in a tag data source table and mapping the field information into a ClickHouse field type; building a table building statement according to the fields, building a target table and executing the table building statement in the target table; assembling a tag data source table and a target table into a FlinkX task execution parameter and submitting the FlinkX task execution parameter to a ClickHouse for data synchronization;

the static label creating module is used for selecting one or more columns of the added data source table as label data sources; adding a static label column in a static label storage table; assembling marking SQL sentences; copying the structure of a static tag storage table, and creating a new table; executing a marking SQL statement, writing the tag data into a new table, and deleting the copied static tag storage table; modifying the name of the new table into the name of the copied static label storage table;

the dynamic label creating module is used for creating a dynamic label creating rule; the front end is assembled into an expression in a json format, and the back end converts json into an object and checks whether the expression meets the dynamic label creation rule or not; performing set operation on the static tags by combining SQL operators to create dynamic tags; analyzing the expression, constructing marking SQL of the dynamic label and executing the marking SQL of the dynamic label; checking the execution state of the marking SQL of the dynamic label to obtain the distribution condition of the production value range of the dynamic label; the dynamic label creating module is also used for creating a new dynamic label by dragging the dynamic label and the static label and combining the judging condition and the filtering condition;

the tag issuing module is used for dragging and generating an API interface for a third party to call according to the dynamic tag or the static tag under the main object as the input parameter or the output parameter; and the system is also used for automatically bringing out the static label storage table and the dynamic label storage table and the fields corresponding to the subject object by dragging the subject object to the SQL editing page, and writing the SQL to generate an API interface for a third party platform to call.

Images