CN109117440B - Method, system and computer-readable storage medium for acquiring metadata information - Google Patents

Abstract

The embodiment of the invention discloses a metadata information acquisition method, which generates data to be identified according to an acquired data source, wherein the data to be identified comprises the following steps: a plurality of row data and a plurality of column data; scanning and matching the data to be identified according to a pre-stored rule to obtain metadata information of the data source, wherein the metadata information comprises at least one of the following information: data structure information, data metric information, and data relationship information. The embodiment of the invention discloses a metadata information acquisition system and a computer readable storage medium.

Description

Method, system and computer-readable storage medium for acquiring metadata information

technical field

The present invention relates to the field of big data, and in particular, to a method, system and computer-readable storage medium for obtaining metadata information.

Background technique

In big data scenarios, more and more data is being shared instead of traditional data warehouses within enterprises. Metadata information can be strictly defined for the internal data warehouse of the enterprise, but in the big data scenario, more and more data comes from the outside, and the data shared by everyone is open on the platform. These data lack unified description and standard in structure and definition.

The data format, data structure and data location of the data in the big data platform scenario have the following characteristics. Data format: The data is stored in a file format, in a certain basic format, such as each row represents a row of data, and a certain function is used between fields. Separated by spacers; data structure: various types, including structured, unstructured, semi-structured and other formats; data location: a large amount of data exists in a large number of heterogeneous, different systems, and even different regions; a large number of data integration operations : It is mainly the integration of cross-domain data, and there are many data interoperability. However, there is currently no way to automatically identify the metadata of the data.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the embodiments of the present invention provide a metadata information acquisition method, system, and computer-readable storage medium, which can automatically acquire information such as data field types, metrics, data relationships, etc. time and efficiency.

The technical scheme of the present invention is realized as follows:

An embodiment of the present invention provides a method for obtaining metadata information, and the method includes:

Generate the data to be identified according to the acquired data source, the data to be identified includes: a plurality of row data and a plurality of column data;

The data to be identified is scanned and matched according to pre-stored rules to obtain metadata information of the data source, where the metadata information includes at least one of the following information: data structure information, data metric information and data relationship information.

Further, the pre-stored rules include: multiple matching rules; the method further includes:

When scanning and matching the data to be identified according to the pre-stored rules, a matching log is recorded in the matching rule log, where the matching log includes: the number of matches of each matching rule in the multiple matching rules, the multiple matching rules The matching success rate of each matching rule in the matching rules;

The matching sequence of the multiple matching rules is optimized according to the matching log.

Further, the first row is any row of data in the data to be identified, and the first row of the data to be identified is parsed to obtain the data structure information of the data source, including:

Starting from the first row and the first column in the data to be identified, scan and match the characters in each column of the first row in the data to be identified with the matching rules pre-stored in the knowledge base, and obtain the The data structure information of the data to be identified, the data structure information of the data to be identified includes: the field type of each column of the data to be identified.

Further, scanning and matching the characters in the first row and the first column of the data to be identified with the matching rules pre-stored in the knowledge base to obtain the field type of the first column of the data to be identified, including:

Scanning and matching the first character with the first matching rule pre-stored in the knowledge base, where the first character is the first character in the first row and first column in the data to be identified;

When the first character is the same as the first pre-stored character in the first matching rule, it is determined that the classification of the first column of the data to be recognized is the first classification, according to all the characters in the first row and the first column The numerical range of the first column determines the field type of the first column, and the first classification includes: digital classification or timestamp classification;

When the first character is different from the first pre-stored character in the first matching rule, scan and match the first character with the second matching rule pre-stored in the knowledge base;

When the first character is the same as the second pre-stored character in the second matching rule, it is determined that the classification of the first column of the data to be recognized is the second classification, according to all the characters in the first row and the first column The format and character length of the first column determine the field type of the first column, and the second classification includes: character classification or date classification;

When the first character is different from the second pre-stored character in the second matching rule, scan and match the first character with the third matching rule pre-stored in the knowledge base;

When the first character is the same as the third pre-stored character in the third character matching rule, it is determined that the classification of the first column of the data to be recognized is a complex classification, according to all the characters in the first row and the first column The combination of symbols and characters in determines the field type of the first column.

Further, after scanning and matching the characters in the first row and the first column of the data to be identified with the matching rules pre-stored in the knowledge base to obtain the field type of the first column of the data to be identified, the method further includes:

If the characters in the second row and the first column in the data to be recognized are scanned and matched with the matching rules pre-stored in the knowledge base, the field type in the first column obtained by scanning and matching is the same as the characters in the first row and the first column in the data to be recognized. It is different from the field type of the first column obtained by scanning and matching the pre-stored matching rules in the knowledge base, and the field type of the first column after the matching order includes a larger range than the field type of the first column before the matching order. When the range of the first column in the matching order is changed to the field type in the first column in the matching order, the second row in the data to be identified includes: the same as the data in the data to be identified The first row of different rows of data.

Further, scan and match the to-be-identified data according to pre-stored rules to obtain data measurement information and data relationship information of the data source, including:

Perform full table scan matching on the data to be identified according to pre-stored rules to obtain data measurement information of the data source, determine the correlation of every two columns in the data to be identified according to the pre-stored rules, and determine the data Data relationship information for the source.

Further, the method also includes:

If the metadata information of the data source cannot be determined by scanning and matching the data to be identified according to the pre-stored rules, a type suggestion is provided, so that the metadata information of the data source can be manually judged, and the result of the manual judgment can be determined according to the metadata information of the data source. Refine the pre-stored rules.

Further, after obtaining the metadata information of the data source, the method further includes:

The metadata information is stored, so that external services are performed according to the metadata information.

An embodiment of the present invention further provides a system for obtaining metadata information, the system includes: a transceiver, a processor, and a memory, wherein,

the transceiver, for obtaining a data source;

The processor is configured to generate data to be identified according to the acquired data source, the data to be identified includes: a plurality of row data and a plurality of column data; and is further configured to scan the to-be-identified data according to pre-stored rules matching, to obtain metadata information of the data source, where the metadata information includes at least one of the following information: data structure information, data metric information, and data relationship information;

The memory is used for pre-storing the rules for scanning and matching the data to be identified.

Further, the pre-stored rules include: multiple matching rules;

The memory is further configured to record a matching log in a matching rule log when scanning and matching the data to be identified according to a pre-stored rule, where the matching log includes: each matching rule in the multiple matching rules The number of matches, the matching success rate of each matching rule in the multiple matching rules;

The processor is further configured to optimize the matching sequence of the multiple matching rules according to the matching log.

Further, the first row is any row of data in the data to be identified, and the processor is specifically configured to start from the first row and first column of the data to be identified, and sequentially The characters in each column of the row are scanned and matched with the matching rules pre-stored in the knowledge base, and the data structure information of the data to be identified is obtained according to the matching result, and the data structure information of the data to be identified includes: The field type of a column.

Further, the processor is specifically used for:

Scanning and matching the first character with the first matching rule pre-stored in the knowledge base, where the first character is the first character in the first row and first column in the data to be identified;

When the first character is the same as the first pre-stored character in the first matching rule, it is determined that the classification of the first column of the data to be recognized is the first classification, according to all the characters in the first row and the first column The numerical range of the first column determines the field type of the first column, and the first classification includes: digital classification or timestamp classification;

When the first character is different from the first pre-stored character in the first matching rule, scan and match the first character with the second matching rule pre-stored in the knowledge base;

When the first character is the same as the second pre-stored character in the second matching rule, it is determined that the classification of the first column of the data to be recognized is the second classification, according to all the characters in the first row and the first column The format and character length of the first column determine the field type of the first column, and the second classification includes: character classification or date classification;

When the first character is different from the second pre-stored character in the second matching rule, scan and match the first character with the third matching rule pre-stored in the knowledge base;

When the first character is the same as the third pre-stored character in the third character matching rule, it is determined that the classification of the first column of the data to be recognized is a complex classification, according to all the characters in the first row and the first column The combination of symbols and characters in determines the field type of the first column.

Further, the processor is further configured to match the field type of the first column obtained by scanning and matching the characters in the second row and the first column of the data to be identified with the matching rules pre-stored in the knowledge base and the to-be-identified data. The characters in the first row and the first column of the identification data are scanned and matched with the matching rules pre-stored in the knowledge base to obtain a field type that is different in the first column, and the field type of the first column after the matching order includes a larger range than the matching order. When the field type of the previous first column includes the range, modify the field type of the first column in the matching order to the field type of the first column in the matching order. The row includes: row data different from the first row in the data to be identified.

Further, the processor is specifically configured to perform full table scan matching on the data to be identified according to pre-stored rules, obtain data measurement information of the data source, and be further configured to determine the to-be-identified data according to the pre-stored rules. Identify the correlation of every two columns in the data, and determine the data relationship information of the data source.

Further, the processor is also configured to provide a type suggestion if the metadata information of the data source cannot be determined by scanning and matching the data to be identified according to the pre-stored rules, so that the data source can be manually judged. metadata information, and improve the pre-stored rules according to the results of human judgment.

Further, the memory is configured to store the metadata information, so that external services are performed according to the metadata information.

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the foregoing method for obtaining metadata information is implemented.

An embodiment of the present invention further provides a system for obtaining metadata information, the system includes a processor and a memory; wherein,

the memory for storing a computer program executable on the processor;

The processor is configured to execute the steps of the above method for acquiring metadata information when running the computer program.

The method, system, and computer-readable storage medium for obtaining metadata information provided by the embodiments of the present invention can, in view of the characteristics of big data, face data sources from different systems and unknown structures, and scan and match pre-stored rules to obtain, for example, data field types. , metrics, data relationships and other information, saving time and improving efficiency.

Description of drawings

1 is a schematic flowchart of a method for obtaining metadata information according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a setting position of a system for obtaining metadata information provided by an embodiment of the present invention;

3 is an example diagram of identification types included in metadata information acquisition provided by an embodiment of the present invention;

FIG. 4 is an example diagram of parsing and matching of to-be-identified data provided by an embodiment of the present invention;

FIG. 5 is an example diagram of a field parsing process provided by an embodiment of the present invention;

FIG. 6 is an example diagram 1 of character matching provided by an embodiment of the present invention;

FIG. 7 is an example FIG. 2 of character matching provided by an embodiment of the present invention;

FIG. 8 is an example FIG. 3 of character matching provided by an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of a system for obtaining metadata information provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

An embodiment of the present invention provides a method for obtaining metadata information. As shown in FIG. 1 , the method may include:

Step 101: Generate data to be identified according to the acquired data source.

Wherein, the data to be identified includes: a plurality of row data and a plurality of column data.

The execution body of the method for acquiring metadata information provided by the embodiment of the present invention may be a system for acquiring metadata information, and the system for acquiring metadata information may be set in the data acquisition module as shown in FIG. 2 , that is, in the process of data acquisition and loading. , to achieve the acquisition of metadata information, so that it can be deployed in the data collection process without adding new data processing steps, thereby saving time and improving efficiency.

The method provided by this embodiment of the present invention can acquire metadata information of a data source in the process of collecting data. Here, the acquired data source may be an external data source or an internal data source, and the internal data source refers to the data that has been read. For the data in the metadata collection system, the external data source refers to the data that has not been read into the metadata collection system. For the external data source, when obtaining metadata information, the data source needs to be read into the metadata information acquisition system first. middle. The metadata information acquisition method can be applied to the acquisition of metadata information of a data source other than a database.

Here, the acquired data source can be a data source with unknown structure from different systems. In the process of reading the data source, the metadata information acquisition system converts the read data source into data to be identified arranged in row and column data. It can also be understood that the data to be identified exists in the form of a table. The table consists of multiple rows and multiple columns, that is, the data to be identified includes multiple row data and multiple column data.

Step 102: Scan and match the to-be-identified data according to a pre-stored rule to obtain metadata information of the data source.

The metadata information includes at least one of the following information: data structure information, data measurement information, and data relationship information.

Here, the metadata information (Metadata) is data describing other data (data about other data), or structured data (structured data) for providing information about a certain resource. Metadata is data that describes objects such as information resources or data, and its purpose is to: identify resources; evaluate resources; track changes in resources during use; achieve simple and efficient management of large amounts of networked data; Find, integrate and organize and efficiently manage usage resources. Metadata is the basis and premise of information sharing and exchange. It records the structure and size of all data, when it was created, when it died, when it was used by those people, the relationship between data tables, the relationship between data flow, and so on. Metadata mainly includes data structure, data flow, data measurement, data model, data access log, data service description, etc., among which, the data structure is used to describe the basic information of the data.

Based on different applications of metadata, metadata can be divided into: data structure, data flow, data deployment, quality measurement, measurement logical relationship, ETL (Extract-Transform-Load) process, data set snapshot, star schema metadata, report semantics Layers, data access logs, quality audit logs, data loading logs, etc. Among them, data structure refers to the name, relationship, field, constraint, etc. of the dataset; data deployment refers to the physical location of the dataset; data flow refers to the process dependency (non-reference dependency) between datasets, including the dataset to Rules for another dataset; quality metrics, which refer to computable metrics on datasets; metric logic relationships, which refer to the logical operation relationships between dataset metrics; ETL processes, which refer to the order in which the processes run, parallel and serial, used to Describe the process of extracting, transforming, and loading data from the source to the destination; data set snapshots refer to the distribution of data in all data sets at a point in time; star schema elements Data refers to fact tables, dimensions, attributes, levels, etc.; report semantic layer: refers to the rules of report indicators, the correspondence between physical names of filter conditions and business names; data access log refers to which data is accessed by whom and when; quality audit log, That is, when and what metrics are audited, and the results; data loading logs, that is, which data is loaded by whom and when.

Specifically, the metadata information acquisition system can acquire metadata information of unknown data sources through a full table scan method and/or a partial scan method, including data field names, data types, attributable users, creation time, modification time, table relationships, constraints , quality measurement.

Among them, the full table scan method scans all the data in the data to be identified, and this method can obtain more metadata information, such as numerical data metrics, including data maximum and minimum values, mean, variance, etc. The partial scanning method only needs to judge the top row (for example, 500 rows) of the data to be identified to obtain the result. For example, the type of the first 500 rows of the data to be identified is judged, but the data measurement cannot be obtained.

Exemplarily, as shown in Figure 3, the metadata information acquisition system for big data may mainly include: data field identification, that is, identifying data field types; data metric identification, that is, identifying various data metrics, such as mean value. , maximum and minimum values, etc., for numerical fields; data relationship identification, that is, to identify the relationship between data tables, among which table relationships include: one-to-one, one-to-many, many-to-many; When a new data source is used, it will give suggestions on the fields that may be associated with the table and other tables; data constraint identification, that is, to realize the identification of data constraints, the identified constraints include: non-null, unique constraint unique, primary key constraint, foreign key constraint, Verification constraints, etc.; knowledge base, which stores the rules and knowledge required for data relationship identification, field identification, and associated field suggestions.

In a possible implementation manner, the first line of the data to be identified is parsed to obtain the data structure information of the data source, including:

Starting from the first row and the first column in the data to be identified, scan and match the characters in each column of the first row in the data to be identified with the matching rules pre-stored in the knowledge base, and obtain the The data structure information of the data to be identified, the data structure information of the data to be identified includes: the field type of each column of the data to be identified.

In a possible implementation manner, the characters in the first row and first column of the data to be identified are scanned and matched with matching rules pre-stored in the knowledge base to obtain the field type of the first column of the data to be identified, including:

Scanning and matching the first character with the first matching rule pre-stored in the knowledge base, where the first character is the first character in the first row and first column in the data to be identified;

When the first character is the same as the first pre-stored character in the first matching rule, it is determined that the classification of the first column of the data to be recognized is the first classification, according to all the characters in the first row and the first column The numerical range of the first column determines the field type of the first column, and the first classification includes: digital classification or timestamp classification;

When the first character is different from the first pre-stored character in the first matching rule, scan and match the first character with the second matching rule pre-stored in the knowledge base;

When the first character is the same as the second pre-stored character in the second matching rule, it is determined that the classification of the first column of the data to be recognized is the second classification, according to all the characters in the first row and the first column The format and character length of the first column determine the field type of the first column, and the second classification includes: character classification or date classification;

When the first character is different from the second pre-stored character in the second matching rule, scan and match the first character with the third matching rule pre-stored in the knowledge base;

When the first character is the same as the third pre-stored character in the third character matching rule, it is determined that the classification of the first column of the data to be recognized is a complex classification, according to all the characters in the first row and the first column The combination of symbols and characters in determines the field type of the first column.

Specifically, the data field can automatically identify the type of the number, that is, the identification of the field type can first analyze each number, and then record all the character types of the column through a temporary storage calculation space. The specific process is shown in Figure 4. Among them, the character type temporary storage 1, the character type temporary storage 2, and the character type temporary storage n respectively include arrays.

Since there are rules between numeric types, for example, in 10 rows, 9 rows are int, but one row is Float, then the entire column should be judged as Float type; so the coverage relationship of numeric types is as follows:

TINYINT---SMALLINT---INT-----BIGINT-----FLOAT-----DOUBLE------DEICIMAL

Among them, the timestamp type is a fixed number of digits, for example, 11 digits or 13 digits. If all columns of an integer type INT type have fixed digits, they can be considered as other types: characters require record length, and the length is fixed to CHAR , non-fixed-bit String and VARCHAR, such as the types shown in Table 1. As long as there are different digits, it can be recorded as String and VARCHAR, and it is VARCHAR if it does not exceed 65534. Among them, for special types, the result can be directly matched.

Table 1

Specifically, in the process of character matching, the coverage relationship can be used for judgment, that is, the temporary storage of field types is refreshed after each line is recognized. Wherein, the coverage relationship is that, once a later-judged type appears, the rule type to be judged later shall prevail. For example, a certain field was previously judged to be of smallint type, and once int appears, the entire field is recorded as int type.

Exemplarily, taking tinyint, smallint, int, and bigint as examples, the judgments of these four types are in order. Once a field appears BIGINT once, the entire field needs to be defined as BIGINT. Similarly, once double appears After the type, the entire field cannot fall back to types such as int.

Exemplarily, how to identify a specific field is described. First distinguish the classification of the fields. The first character can identify the time type, number type, complex type, etc. The matching process is shown in Figure 5. Take the first character in the field of any row and any column in the data to be identified, and judge Whether the first character is a negative sign or a number, when the first character is a negative sign - or a number, the field of any row or column is a numeric type or a timestamp type, when the first character is not a negative sign - or numbers, determine whether the first character is a double quotation mark "or single quotation mark', if so, determine whether the field of any row or column is a character type or a date type, if not, determine whether the first character is a Brackets (, if it is, it is determined that the field of any row and any column is a complex type, if not, it is classified as other, and judged manually.

Exemplarily, the number/timestamp type matching process, as shown in Figure 6, judges whether the currently matched character has a decimal point, and if there is a decimal point, further judges whether the character is a Float type or a Double according to the value range to which the character belongs. If there is no decimal point, then judge whether the character is TINYINT or one of SMALLINT, INT or BIGINT according to the value range that the character belongs to.

Exemplarily, the character type/date type matching process, as shown in Figure 7, determines whether the currently matched character conforms to the date format YYYY-MM-DD, and if it conforms to the date, determine that the character type is a date, and if it does not conform to the date , get the length of the character, and determine the type of the character by the length of the character.

Exemplary, complex type matching process: As shown in Figure 8, it is judged whether the currently matched characters and commas are of the same type. If so, it is determined that the character is of type ARRAY. If not, it is judged whether the currently matched characters are separated by commas. It is ABAB type. If it is, it is determined that the character is of MAP type. If it is not, it is determined whether the current matching character is of various types. If it is, it is determined that the character is of UNION type. If not, it is determined whether the current matching character is not. It is nested with multiple brackets (), if it is, it is determined that the character is of STRUCT type, if not, it is classified as other, and it is judged manually.

Optionally, after scanning and matching the characters in the first row and the first column of the data to be identified with the matching rules pre-stored in the knowledge base to obtain the field type of the first column of the data to be identified, the method further includes:

If the characters in the second row and the first column in the data to be recognized are scanned and matched with the matching rules pre-stored in the knowledge base, the field type in the first column obtained by scanning and matching is the same as the characters in the first row and the first column in the data to be recognized. It is different from the pre-stored matching rules in the knowledge base to scan and match to obtain the field type of the first column, and the field type of the first column after the matching order includes a larger range than the field type of the first column before the matching order. In the range, modify the field type of the first column in the matching order to the field type of the first column in the matching order, wherein the second row in the data to be identified includes: A row of different row data.

Specifically, since the data types of each column of the data to be identified are the same, after the data of the first row of each column is judged, the values of the subsequent rows are used to correct the previous judgment. For example, a certain field was previously judged to be of smallint type, and once int appears, the entire field is recorded as int type.

In a possible implementation manner, scanning and matching the data to be identified according to pre-stored rules to obtain data measurement information and data relationship information of the data source may include: The data is scanned and matched in a full table to obtain the data metric information of the data source, and the correlation of every two columns in the to-be-identified data is determined according to the pre-stored rule, and the data relationship information of the data source is determined.

Specifically, for the data measurement information, a full table scan can be selected to obtain the data measurement information, and the data measurement information may include the maximum and minimum values, mean, variance, summation, and the like of a certain field. Specifically, it can be implemented through the metadata acquisition in FIG. 2 .

Specifically, the metadata information acquisition system can automatically identify two columns of data that are very similar according to pre-stored rules, indicating that the two columns can be correlated and expand the value. The rules for identifying data correlation in the pre-stored rules may be a Pearson Correlation Coefficient method, a text similarity algorithm, or other rules for calculating correlations, which are not limited in this embodiment of the present invention .

Exemplarily, the Pearson correlation coefficient method can be used for numbers, and the Pearson correlation coefficient can measure the correlation of two columns of data.

The formula for calculating the Pearson correlation coefficient is:

Simple classifications of Pearson correlation include: very strong correlation, strong correlation, moderate correlation, weak correlation, very weak correlation, or no correlation. Very strong correlation when r is between 0.8-1.0, strong correlation when r is between 0.6-0.8, moderate correlation when r is between 0.4-0.6, and moderate correlation when r is between 0.2-0.4 It is a weak correlation, and when r is between 0.0 and 0.2, it is a very weak correlation or no correlation.

For highly correlated columns, they can be recommended as associated columns, that is, associated field recommendations, indicating that these are two columns of very similar data that can be associated.

Exemplarily, for character type data, a text similarity algorithm can be used, and then the relationship between two columns can be obtained through vector calculation. A brief description is as follows:

The first step is to count all the words and word frequencies in column A and column B to generate a vector;

The second step is to calculate the cosine distance between column A and column B. The closer the value is to 1, the more similar the two columns are;

For example, column A: Zhejiang, Shanghai, Zhejiang, Shandong, Jiangsu

Column B: Beijing, Shanghai, Zhejiang, Zhejiang, Shandong

The generated vector is: A={Zhejiang 2, Shanghai 1, Shandong 1, Jiangsu 1, Beijing, 0}

B={Zhejiang 2, Shanghai 1, Shandong 1, Jiangsu 0, Beijing 1}

Calculate the cosine similarity between {2, 1, 1, 1, 0} and {2, 1, 1, 0, 1} to get the similarity of these two columns.

Further, the above-mentioned method for obtaining metadata information may further include: if the metadata information of the data source cannot be determined by scanning and matching the data to be identified according to pre-stored rules, providing a type suggestion, so that the manual judgment of the metadata information of the data source, and improve the pre-stored rules according to the results of human judgment.

Specifically, the metadata information acquisition system can provide metadata information according to pre-stored rules for different data sources. However, if the result cannot be confirmed according to the pre-stored rules, it can provide possible types of suggestions, which can be reviewed manually. Confirmation, after manual review and confirmation, can automatically learn to improve and optimize the rule knowledge base.

Exemplarily, for the types that cannot be matched, as shown in Figures 5, 6, and 8, all data types that match other types are judged manually. After manual judgment, the accumulation can be used as the input for perfect rules. The metadata information acquisition system improves the rule knowledge base.

Further, the pre-stored rules include: a plurality of matching rules; the above-mentioned metadata information acquisition method further includes: when scanning and matching the data to be identified according to the pre-stored rules, recording the matching in a matching rule log log, the matching log includes: the matching times of each matching rule in the multiple matching rules, the matching success rate of each matching rule in the multiple matching rules; The matching order of the rules is optimized.

Specifically, in the process of automatically matching data types, the sequence of matching rules can be continuously optimized, which is related to the specific data situation. The matching log can be recorded in the matching rule log, and the rules can be matched according to the number of matches and the success rate. order of optimization.

Exemplarily, as shown in FIG. 5 , the matching rules are: Step 1, firstly match whether the number/time stamp; Step 2, then match whether the character type/date type; Step 3, finally match the complex type. In the process of predicting a certain column, if the previous data are all character types, you can skip the matching process of whether step 1 is a negative sign - or a number, and directly enter the matching process of step 2, that is, step 1, step 2 , The sequence of matching in step 3 is not limited to that shown in Figure 5, it can be selected according to the actual matching process, it can be skip step 1 and go directly to step 2, or skip step 1 and step 2 and go directly to step 3 .

Similarly, after entering a specific matching process, as shown in Figures 6, 7, and 8, the sequence of matching rules can also be further optimized. As shown in FIG. 8 , taking the complex type matching process as an example, if the previous data is judged to be MAP, the first step of ARRAY judgment can be skipped, so that the matching sequence performance can be optimized.

Further, after the metadata information of the data source is obtained, the method for obtaining the metadata information further includes: storing the metadata information, so that external services are performed according to the metadata information.

Specifically, after obtaining the metadata information of the unknown data source through the above method, the metadata information can be stored to provide external services, that is, the metadata can be queried and used in subsequent business use. The storage format may be structured information or a file or other manners, which are not limited in this embodiment of the present invention.

The metadata acquisition method provided by the embodiment of the present invention is aimed at the problem of numerous big data data sources and different data structures, and provides a method for automatically acquiring metadata during the data loading process; it can be based on sampling and full scanning. way, obtain metadata, and more metadata information, such as constraints, quality metrics, etc.; assisting manual judgment and rule matching log analysis, can achieve the improvement of the knowledge base and the optimization of performance.

The metadata acquisition method provided by the embodiment of the present invention can automatically provide metadata information, especially the data type, for unknown data through data scanning, form metadata, and perform automatic learning and optimization. The prior art mainly focuses on some data operations in traditional databases or extracts metadata according to definitions, and does not involve the field of automatically obtaining metadata according to data conditions. In a mixed big data scenario, how to identify massive unknown data is the implementation of the present invention. The problem solved by the metadata information acquisition method provided by the example.

An embodiment of the present invention provides a system 90 for obtaining metadata information. As shown in FIG. 9 , the system includes: a processor 901, a memory 902, and a transceiver 903, wherein:

The transceiver 903 is used to obtain a data source;

The processor 901 is configured to generate data to be identified according to the acquired data source, and the data to be identified includes: multiple row data and multiple column data; and is further configured to perform a process on the to-be-identified data according to pre-stored rules. Scanning and matching to obtain metadata information of the data source, where the metadata information includes at least one of the following information: data structure information, data metric information, and data relationship information;

The memory 902 is configured to pre-store the rules for scanning and matching the data to be identified.

Further, the pre-stored rules include: multiple matching rules;

The memory 902 is further configured to record a matching log in a matching rule log when scanning and matching the data to be identified according to a pre-stored rule, where the matching log includes: each matching of the multiple matching rules The number of matching rules, and the matching success rate of each matching rule in the multiple matching rules;

The processor 901 is further configured to optimize the matching sequence of the multiple matching rules according to the matching log.

Further, the first row is any row of data in the data to be identified, and the processor 901 is specifically configured to start from the first row and first column of the data to be identified, and sequentially convert the first row of the data to be identified. The characters in each column of a row are scanned and matched with the matching rules pre-stored in the knowledge base, and the data structure information of the data to be recognized is obtained according to the matching result. The data structure information of the data to be recognized includes: the data to be recognized Field type for each column.

Further, the processor 901 is specifically used for:

Scanning and matching the first character with the first matching rule pre-stored in the knowledge base, where the first character is the first character in the first row and first column in the data to be identified;

When the first character is the same as the first pre-stored character in the first matching rule, it is determined that the classification of the first column of the data to be recognized is the first classification, according to all the characters in the first row and the first column The numerical range of the first column determines the field type of the first column, and the first classification includes: digital classification or timestamp classification;

When the first character is different from the first pre-stored character in the first matching rule, scan and match the first character with the second matching rule pre-stored in the knowledge base;

When the first character is the same as the second pre-stored character in the second matching rule, it is determined that the classification of the first column of the data to be recognized is the second classification, according to all the characters in the first row and the first column The format and character length of the first column determine the field type of the first column, and the second classification includes: character classification or date classification;

When the first character is different from the second pre-stored character in the second matching rule, scan and match the first character with the third matching rule pre-stored in the knowledge base;

When the first character is the same as the third pre-stored character in the third character matching rule, it is determined that the classification of the first column of the data to be recognized is a complex classification, according to all the characters in the first row and the first column The combination of symbols and characters in determines the field type of the first column.

Further, the processor 901 is further configured to scan and match the characters in the second row and the first column of the to-be-recognized data with the matching rules pre-stored in the knowledge base to obtain the field type of the first column and the The characters in the first row and the first column of the data to be identified are scanned and matched with the matching rules pre-stored in the knowledge base to obtain a different field type in the first column, and the field type in the first column after the matching sequence includes a range greater than the matching range. When the field type of the first column in the previous order includes the range, the field type of the first column in the matching order is modified to the field type of the first column in the matching order. The two rows include: row data different from the first row in the data to be identified.

Further, the processor 901 is specifically configured to perform full table scan matching on the data to be identified according to pre-stored rules, obtain data measurement information of the data source, and be further configured to determine the data according to the pre-stored rules. For the correlation of every two columns in the data to be identified, the data relationship information of the data source is determined.

Further, the processor 901 is further configured to provide type suggestions if the metadata information of the data source cannot be determined by scanning and matching the data to be identified according to pre-stored rules, so that the data can be judged manually. Source metadata information, and improve the pre-stored rules according to the results of human judgment.

Further, the memory 902 is used to store the metadata information, and the transceiver 903 is used to output the metadata information stored in the memory 902, so that external services can be performed according to the metadata information output by the transceiver 903 .

Specifically, for an understanding of the metadata information obtaining system provided by the embodiment of the present invention, reference may be made to the description of the above-mentioned embodiment of the method for obtaining metadata information, which is not repeated in this embodiment of the present invention.

The metadata information acquisition system provided by the embodiment of the present invention can, according to the characteristics of big data, face data sources from different systems and unknown structures, and can automatically acquire data fields such as data field types, metrics, and data relationships in the process of data to be identified. Save time and improve efficiency.

In an exemplary embodiment, an embodiment of the present invention further provides a computer-readable storage medium, such as a memory 902 including a computer program, and the computer program can be executed by the processor 901 in the metadata information acquisition system 90 to complete the foregoing the steps of the method. The computer-readable storage medium can be memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM; it can also be various devices including one or any combination of the above-mentioned memories, such as Mobile phones, computers, tablet devices, personal digital assistants, etc.

A computer-readable storage medium, storing a computer program in the computer-readable storage medium, and when the computer program is run by a processor, executes:

Generate the data to be identified according to the acquired data source, the data to be identified includes: a plurality of row data and a plurality of column data;

The data to be identified is scanned and matched according to pre-stored rules to obtain metadata information of the data source, where the metadata information includes at least one of the following information: data structure information, data metric information and data relationship information.

The pre-stored rules include: a plurality of matching rules; when the computer program is run by the processor, it also executes: when the data to be identified is scanned and matched according to the pre-stored rules, record in the matching rule log. a matching log, the matching log includes: the number of matches of each of the multiple matching rules, and the matching success rate of each of the multiple matching rules;

The matching sequence of the multiple matching rules is optimized according to the matching log.

The first row is any row of data in the data to be identified. When the computer program is run by the processor, it also executes:

Starting from the first row and the first column in the data to be identified, scan and match the characters in each column of the first row in the data to be identified with the matching rules pre-stored in the knowledge base, and obtain the The data structure information of the data to be identified, the data structure information of the data to be identified includes: the field type of each column of the data to be identified.

When the computer program is run by the processor, it also executes: scanning and matching the first character with the first matching rule pre-stored in the knowledge base, where the first character is the first row and the first column in the data to be identified the first character of ;

When the first character is the same as the first pre-stored character in the first matching rule, it is determined that the classification of the first column of the data to be recognized is the first classification, according to all the characters in the first row and the first column The numerical range of the first column determines the field type of the first column, and the first classification includes: digital classification or timestamp classification;

When the first character is different from the first pre-stored character in the first matching rule, scan and match the first character with the second matching rule pre-stored in the knowledge base;

When the first character is the same as the second pre-stored character in the second matching rule, it is determined that the classification of the first column of the data to be recognized is the second classification, according to all the characters in the first row and the first column The format and character length of the first column determine the field type of the first column, and the second classification includes: character classification or date classification;

When the first character is different from the second pre-stored character in the second matching rule, scan and match the first character with the third matching rule pre-stored in the knowledge base;

When the first character is the same as the third pre-stored character in the third character matching rule, it is determined that the classification of the first column of the data to be recognized is a complex classification, according to all the characters in the first row and the first column The combination of symbols and characters in determines the field type of the first column.

When the computer program is run by the processor, it also executes: if the character in the second row and the first column in the data to be recognized is scanned and matched with the matching rules pre-stored in the knowledge base, the field type of the first column obtained by scanning and matching is the same as the selected character. The characters in the first row and the first column of the data to be identified are different from the field type of the first column obtained by scanning and matching the pre-stored matching rules in the knowledge base, and the field type of the first column after the matching order includes the range. When it is greater than the range included in the field type of the first column in the matching order, modify the field type of the first column in the matching order to the field type of the first column in the matching order, wherein the data to be identified The second row in the above includes: row data different from the first row in the to-be-identified data.

When the computer program is run by the processor, it also executes: scanning and matching the data to be identified according to pre-stored rules to obtain data measurement information and data relationship information of the data source, including:

Perform full table scan matching on the data to be identified according to pre-stored rules to obtain data measurement information of the data source, determine the correlation of every two columns in the data to be identified according to the pre-stored rules, and determine the data Data relationship information for the source.

When the computer program is run by the processor, it also executes: if the metadata information of the data source cannot be determined by scanning and matching the to-be-identified data according to the pre-stored rules, provide a type suggestion so as to make the manual judgment of the data source. metadata information of the data source, and improve the pre-stored rules according to the results of manual judgment.

When the computer program is run by the processor, it further executes: storing the metadata information, so that external services are performed according to the metadata information.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention.

Claims (16)

1. A method for obtaining metadata information, wherein the method comprises: Generate the data to be identified according to the acquired data source, the data to be identified includes: a plurality of row data and a plurality of column data; Scan and match the data to be identified according to pre-stored rules to obtain metadata information of the data source, where the metadata information includes at least one of the following information: data structure information, data metric information, and data relationship information; Wherein, scanning and matching the to-be-identified data according to pre-stored rules to obtain data measurement information and data relationship information of the data source, including: Perform full table scan matching on the data to be identified according to pre-stored rules to obtain data measurement information of the data source, determine the correlation of every two columns in the data to be identified according to the pre-stored rules, and determine the data Data relationship information for the source. 2. The method according to claim 1, wherein the pre-stored rules include: a plurality of matching rules; the method further comprises: When scanning and matching the data to be identified according to the pre-stored rules, a matching log is recorded in the matching rule log, where the matching log includes: the number of matches of each matching rule in the multiple matching rules, the multiple matching rules The matching success rate of each matching rule in the matching rules; The matching sequence of the multiple matching rules is optimized according to the matching log. 3. The method according to claim 1 or 2, wherein the first row is any row of data in the data to be identified, and the first row of the data to be identified is parsed to obtain the data of the data source. Data structure information, including: Starting from the first row and the first column in the data to be identified, scan and match the characters in each column of the first row in the data to be identified with the matching rules pre-stored in the knowledge base, and obtain the The data structure information of the data to be identified, the data structure information of the data to be identified includes: the field type of each column of the data to be identified. 4. The method according to claim 3, wherein the characters in the first row and the first column of the data to be identified are scanned and matched with the matching rules pre-stored in the knowledge base to obtain the first row of the data to be identified. The field type of the column, including: Scanning and matching the first character with the first matching rule pre-stored in the knowledge base, where the first character is the first character in the first row and first column in the data to be identified; When the first character is the same as the first pre-stored character in the first matching rule, it is determined that the classification of the first column of the data to be recognized is the first classification, according to all the characters in the first row and the first column The numerical range of the first column determines the field type of the first column, and the first classification includes: digital classification or timestamp classification; When the first character is different from the first pre-stored character in the first matching rule, scan and match the first character with the second matching rule pre-stored in the knowledge base; When the first character is the same as the second pre-stored character in the second matching rule, it is determined that the classification of the first column of the data to be recognized is the second classification, according to all the characters in the first row and the first column The format and character length of the first column determine the field type of the first column, and the second classification includes: character classification or date classification; When the first character is different from the second pre-stored character in the second matching rule, scan and match the first character with the third matching rule pre-stored in the knowledge base; When the first character is the same as the third pre-stored character in the third matching rule, it is determined that the classification of the first column of the to-be-recognized data is a complex classification. The combination of symbols and characters determines the field type of the first column. 5 . The method according to claim 4 , wherein the characters in the first row and first column of the data to be identified are scanned and matched with matching rules pre-stored in the knowledge base to obtain the first row of the data to be identified. 6 . After the field type of a column, it also includes: If the characters in the second row and the first column in the data to be recognized are scanned and matched with the matching rules pre-stored in the knowledge base, the field type in the first column obtained by scanning and matching is the same as the characters in the first row and the first column in the data to be recognized. It is different from the field type of the first column obtained by scanning and matching the pre-stored matching rules in the knowledge base, and the field type of the first column after the matching order includes a larger range than the field type of the first column before the matching order. When the range of the first column in the matching order is changed to the field type in the first column in the matching order, the second row in the data to be identified includes: the same as the data in the data to be identified The first row is different row data. 6. The method according to claim 1 or 2, wherein the method further comprises: If the metadata information of the data source cannot be determined by scanning and matching the data to be identified according to the pre-stored rules, a type suggestion is provided, so that the metadata information of the data source can be manually judged, and the result of the manual judgment can be determined according to the metadata information of the data source. Refine the pre-stored rules. 7. The method according to claim 1 or 2, wherein after obtaining the metadata information of the data source, the method further comprises: The metadata information is stored, so that external services are performed according to the metadata information. 8. A system for obtaining metadata information, wherein the system comprises: a transceiver, a processor and a memory, wherein, the transceiver, for obtaining a data source; The processor is configured to generate data to be identified according to the acquired data source, the data to be identified includes: a plurality of row data and a plurality of column data; and is further configured to scan the to-be-identified data according to pre-stored rules matching, to obtain metadata information of the data source, where the metadata information includes at least one of the following information: data structure information, data metric information, and data relationship information; The processor is specifically configured to perform full table scan matching on the data to be identified according to pre-stored rules, and obtain data measurement information of the data source; determine every two columns in the data to be identified according to the pre-stored rules Relevance, determine the data relationship information of the data source; The memory is used for pre-storing the rules for scanning and matching the data to be identified. 9. The system according to claim 8, wherein the pre-stored rules include: a plurality of matching rules; The memory is further configured to record a matching log in a matching rule log when scanning and matching the data to be identified according to a pre-stored rule, where the matching log includes: each matching rule in the multiple matching rules The number of matches, the matching success rate of each matching rule in the multiple matching rules; The processor is further configured to optimize the matching sequence of the multiple matching rules according to the matching log. 10. The system according to claim 8 or 9, wherein the first row is any row of data in the to-be-identified data, and the processor is specifically configured to extract the first row of the to-be-identified data from the first row Starting from one column, the characters in each column of the first row in the data to be recognized are scanned and matched with the matching rules pre-stored in the knowledge base in turn, and the data structure information of the data to be recognized is obtained according to the matching result. The data structure information of the identification data includes: the field type of each column of the data to be identified. 11. The system according to claim 10, wherein the processor is specifically configured to: Scanning and matching the first character with the first matching rule pre-stored in the knowledge base, where the first character is the first character in the first row and first column in the data to be identified; When the first character is the same as the first pre-stored character in the first matching rule, it is determined that the classification of the first column of the data to be recognized is the first classification, according to all the characters in the first row and the first column The numerical range of the first column determines the field type of the first column, and the first classification includes: digital classification or timestamp classification; When the first character is different from the first pre-stored character in the first matching rule, scan and match the first character with the second matching rule pre-stored in the knowledge base; When the first character is the same as the second pre-stored character in the second matching rule, it is determined that the classification of the first column of the data to be recognized is the second classification, according to all the characters in the first row and the first column The format and character length of the first column determine the field type of the first column, and the second classification includes: character classification or date classification; When the first character is different from the second pre-stored character in the second matching rule, scan and match the first character with the third matching rule pre-stored in the knowledge base; When the first character is the same as the third pre-stored character in the third matching rule, it is determined that the classification of the first column of the to-be-recognized data is a complex classification. The combination of symbols and characters determines the field type of the first column. 12 . The system according to claim 11 , wherein the processor is further configured to perform scan matching if the characters in the second row and the first column in the data to be recognized are matched with the matching rules pre-stored in the knowledge base. 13 . The obtained field type of the first column is different from the character in the first row and the first column of the data to be identified and the matching rules pre-stored in the knowledge base are scanned and matched to obtain the field type of the first column, and the matching order is later. When the range of the field type of the first column is larger than that of the field type of the first column in the matching order, change the field type of the first column of the matching order to the field of the first column of the matching order. type, wherein the second row in the data to be identified includes: row data different from the first row in the data to be identified. 13. The system according to claim 8 or 9, wherein the processor is further configured to scan and match the to-be-identified data according to a pre-stored rule and cannot determine the metadata information of the data source , providing type suggestions, so that the metadata information of the data source is judged manually, and the pre-stored rules are perfected according to the result of the manual judgment. 14. The system according to claim 8 or 9, wherein the memory is configured to store the metadata information, so that external services are performed according to the metadata information. 15. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the metadata information according to any one of claims 1 to 7 is realized get method. 16. A system for obtaining metadata information, wherein the system comprises a processor and a memory; wherein, the memory for storing a computer program executable on the processor; The processor is configured to execute the steps of the method for obtaining metadata information according to any one of claims 1 to 7 when running the computer program.

Images