CN114880387A - Data integration script generation method and device, storage medium and electronic equipment - Google Patents

Abstract

The application discloses a data integration script generation method and device, a storage medium and electronic equipment, which can be applied to the financial field or other fields. The method comprises the following steps: when a task script of a data integration task needs to be generated, data acquisition information is determined; according to the data acquisition information, determining a target metadata identifier and target metadata information corresponding to the target metadata identifier in a preset data lake metadata information base, and according to the target metadata information, determining a database type corresponding to the target metadata identifier; determining a script generation strategy corresponding to the database type; determining a data acquisition mode corresponding to the data integration task; and if the data acquisition mode is a full acquisition mode, generating a full data integration script according to the script generation strategy and the data acquisition information to obtain a task script of the data integration task. By the method, the automatic generation of the data integration script can be realized, the script does not need to be manually compiled, and the working efficiency can be improved.

Description

Data integration script generation method and device, storage medium and electronic device

technical field

The present invention relates to the technical field of data processing, and in particular, to a method and device for generating a data integration script, a storage medium and an electronic device.

Background technique

In the informatization construction of enterprise organizations, data lake is one of the commonly used data storage architectures. The data lake obtains raw data from multiple data sources of the enterprise, and all data is stored in its original form, including structured data (such as relational database data), semi-structured data (such as CSV, XML, JSON, etc.), non- Structured data (e.g. emails, documents, PDFs) and binary data (e.g. images, audio, video) to form a centralized data store for all forms of data for subsequent reporting, visual analysis, real-time analysis or Business scenarios such as machine learning provide data support.

In the process of data analysis based on the data lake, it is necessary to perform data integration on the data in the data lake, that is, extract, transform and load the data into the specified data storage, that is, carry out ETL (Extract-Transform-Load) processing.

Data integration tasks are usually implemented based on data integration scripts, ie ETL scripts. At present, ETL scripts are mainly written manually by technicians using database knowledge, data table structure and SQL technology according to their own analysis of business data. However, the way of manually writing ETL scripts requires a lot of human resources, the processing process is time-consuming, and the efficiency is low. Secondly, in the process of manually writing ETL scripts, it is necessary to manually check the data sources, which is prone to omissions or errors, which has a negative impact on the data integration work.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present invention provide a method for generating a data integration script, so as to solve the problems that manually writing an ETL script is time-consuming, low in efficiency, and prone to errors.

The embodiment of the present invention also provides a data integration script generation device, so as to ensure the actual realization and application of the above method.

To achieve the above purpose, the embodiments of the present invention provide the following technical solutions:

A data integration script generation method, comprising:

When a task script of the data integration task needs to be generated, determine the data collection information corresponding to the data integration task;

According to the data collection information, in the preset data lake metadata information database, determine the target metadata identifier corresponding to the data integration task and the target metadata information corresponding to the target metadata identifier, the data lake metadata The data information base contains a plurality of preset metadata identifiers and metadata information corresponding to each of the preset metadata identifiers;

According to the target metadata information, determine the database type corresponding to the target metadata identifier;

determining the script generation strategy corresponding to the database type;

determining the data collection mode corresponding to the data integration task;

If the data collection mode is the full data collection mode, generate a full data integration script corresponding to the target metadata identifier according to the script generation strategy and the data collection information, and use the full data integration script as the Task scripts for data integration tasks.

In the above method, optionally, the determining of the data collection information corresponding to the data integration task includes:

Obtain the business subject, the department, the product, the metadata identifier and the collection mode entered by the user, and use the business subject, the department, the product, the metadata identifier and the collection mode as the Data collection information corresponding to the data integration task.

In the above method, optionally, determining the target metadata identifier corresponding to the data integration task in a preset data lake metadata information base according to the data collection information, including:

The data collection information is respectively matched with the metadata information corresponding to each of the preset metadata identifiers, and the preset metadata identifier corresponding to the metadata information matched with the data collection information is determined as The target metadata identifier.

The above method, optionally, further includes:

If the data collection mode is not the full collection mode, determining the start and end conditions corresponding to the data integration task;

According to the target metadata information and the starting and ending conditions, condition checking is performed on the data integration task;

If the data integration task passes the condition check, an incremental data integration script corresponding to the target metadata identifier is generated according to the script generation strategy, the data collection information, the start and end conditions, and the target metadata information , and use the incremental data integration script as the task script of the data integration task.

In the above method, optionally, the condition checking on the data integration task according to the target metadata information and the start and end conditions includes:

determining the incremental information corresponding to the target metadata information, where the incremental information includes an incremental variable identifier and an incremental variable data structure;

Judging whether the start and end conditions match the incremental information;

If the start and end conditions match the incremental information, determine that the data integration task passes the condition check;

If the start and end conditions do not match the incremental information, it is determined that the data integration task fails the condition check.

The above method, optionally, further includes:

If the data integration task fails the condition check, an error message is prompted, and the generation process of the task script of the data integration task ends.

A data integration script generation device, comprising:

a first determining unit, configured to determine data collection information corresponding to the data integration task when a task script of the data integration task needs to be generated;

The second determining unit is configured to, according to the data collection information, determine the target metadata identifier corresponding to the data integration task and the target metadata corresponding to the target metadata identifier in the preset data lake metadata information database information, the data lake metadata information base contains multiple preset metadata identifiers and metadata information corresponding to each of the preset metadata identifiers;

a third determining unit, configured to determine the database type corresponding to the target metadata identifier according to the target metadata information;

a fourth determining unit, configured to determine a script generation strategy corresponding to the database type;

a fifth determination unit, configured to determine a data collection mode corresponding to the data integration task;

The first generation unit is configured to generate a full data integration script corresponding to the target metadata identifier according to the script generation strategy and the data collection information if the data collection mode is the full collection mode, and integrate the The full data integration script is used as the task script of the data integration task.

The above-mentioned device, optionally, also includes:

a sixth determination unit, configured to determine the start and end conditions corresponding to the data integration task if the data collection mode is not the full collection mode;

a verification unit, configured to perform condition verification on the data integration task according to the target metadata information and the start and end conditions;

A second generating unit, configured to generate the target metadata identifier according to the script generation strategy, the data collection information, the start and end conditions, and the target metadata information if the data integration task passes the condition check corresponding incremental data integration script, and use the incremental data integration script as the task script of the data integration task.

A storage medium comprising stored instructions, wherein when the instructions are executed, a device where the storage medium is located is controlled to execute the above-mentioned method for generating a data integration script.

An electronic device includes a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to execute, by one or more processors, the data integration script generation method as described above.

A method for generating a data integration script provided based on the above embodiments of the present invention includes: when a task script of a data integration task needs to be generated, determining data collection information corresponding to the data integration task; according to the data collection information, in a preset data lake In the metadata information database, the target metadata identifier and target metadata information corresponding to the data integration task are determined, and the data lake metadata information database contains a plurality of preset metadata identifiers corresponding to each of the preset metadata identifiers. According to the target metadata information, determine the database type corresponding to the target metadata identifier; determine the script generation strategy corresponding to the database type; determine the data collection mode corresponding to the data integration task; if the data collection mode is the full collection mode , a full data integration script is generated according to the script generation strategy and data collection information, and the full data integration script is used as the task script of the data integration task. By applying the method provided by the embodiment of the present invention, the task script of the data integration task can be generated through an automated processing process, and the automatic generation of the data integration script can be realized without manual script writing, which can save a lot of human resources and shorten the processing time. Improve work efficiency. Secondly, it can avoid human mistakes caused by manual scripting, and avoid the adverse effects of human factors on data integration work.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

1 is a method flowchart of a method for generating a data integration script provided by an embodiment of the present invention;

2 is a schematic interface diagram of a submission interface of a data integration task provided by an embodiment of the present invention;

3 is an exemplary diagram of a data integration script generation process provided by an embodiment of the present invention;

4 is a schematic structural diagram of an apparatus for generating a data integration script according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In this application, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also no Other elements expressly listed, or which are also inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

An embodiment of the present invention provides a method for generating a data integration script. The method can be applied to a data integration platform, and the execution body of the method can be a server of the data integration platform. The method flowchart of the method is shown in FIG. 1 , including:

S101: when a task script of a data integration task needs to be generated, determine data collection information corresponding to the data integration task;

In the method provided by the embodiment of the present invention, the user can submit the relevant information of the data integration task through the front end, so as to send the trigger instruction of the task script that needs to generate the data integration task to the server. When the server receives the trigger instruction, it can obtain the relevant information input by the user to determine the data collection information corresponding to the data integration task that needs to generate the task script currently. The data collection information may include the identification of the metadata to be collected, the department to which it belongs, the product to which it belongs, and so on.

S102: According to the data collection information, in a preset data lake metadata information database, determine a target metadata identifier corresponding to the data integration task and target metadata information corresponding to the target metadata identifier, and the data The lake metadata information base contains multiple preset metadata identifiers and metadata information corresponding to each of the preset metadata identifiers;

In the method provided by the embodiment of the present invention, corresponding metadata identifiers and metadata information corresponding to each metadata identifier may be preset according to each metadata item of the data lake, so as to construct a metadata information database of the data lake. The data lake metadata information base contains multiple preset metadata identifiers and metadata information corresponding to each preset metadata identifier, and the metadata information corresponding to each preset metadata identifier is related to each metadata item in the data lake. correspond. The metadata information corresponding to the preset metadata identifier may include information such as the preset metadata identifier, metadata name, database type, collection mode, incremental variable identifier, and incremental variable data structure.

In the method provided by the embodiment of the present invention, the target metadata identifier can be determined in the data lake metadata information base according to the data collection information of the data integration task, and the target metadata identifier corresponding to the data lake metadata information base can be determined. The metadata information is determined as the target metadata information.

S103: According to the target metadata information, determine the database type corresponding to the target metadata identifier;

In the method provided by the embodiment of the present invention, the database type corresponding to the target metadata identifier can be obtained from the target metadata information, that is, the type of the database storing the metadata corresponding to the target metadata identifier. The database type can be Hive, HBase, Hdfs, or Nosql, or other database types.

S104: Determine a script generation strategy corresponding to the database type;

In the method provided by the embodiment of the present invention, a script generation strategy corresponding to each type of database can be set in advance according to the data integration mode of each type of database. Specifically, a script template can be set, and the template parameter can be replaced to generate a script, or Preset script statements, import relevant parameters to generate scripts, etc.

In the method provided by the embodiment of the present invention, the corresponding database type may be identified by target metadata, and among each preset script generation strategy, the script generation strategy corresponding to the database type is obtained by matching.

S105: Determine the data collection mode corresponding to the data integration task;

In the method provided by the embodiment of the present invention, the user can set the collection mode of the data integration task through the front end, the information can be included in the data collection information, the collection mode set by the user can be obtained from the data collection information, and the collection mode is determined as The data collection mode corresponding to the data integration task. Specifically, the data collection mode may be a full collection mode or an incremental collection mode.

S106: If the data collection mode is the full collection mode, generate a full data integration script corresponding to the target metadata identifier according to the script generation strategy and the data collection information, and use the full data integration script as The task script for the data integration task.

In the method provided by the embodiment of the present invention, if the data collection mode is the full data collection mode, a full data integration script, that is, an ETL script, is generated according to the script generation strategy and data collection information. Use the full data integration script as the task script of the data integration task. For example, according to the script template of the full collection in the script generation strategy, parameters can be replaced based on the metadata identification, business subject, department, product and other data in the data collection information, and based on the script template that has completed the parameter replacement, the corresponding full Data integration scripts.

In the method provided by the embodiment of the present invention, the generated task script can be displayed to the user, the user can check the task script, and subsequently the task script of the data integration task can be executed by invoking the data integration engine, that is, the ETL execution engine can be invoked to complete the ETL execution.

Based on the method provided by the embodiment of the present invention, when the task script of the data integration task needs to be generated, the data collection information corresponding to the data integration task is determined; according to the data collection information, in the preset data lake metadata information database, the data integration task is determined The target metadata identifier corresponding to the task and the target metadata information corresponding to the target metadata identifier, according to the target metadata information, determine the database type corresponding to the target metadata identifier; determine the script generation strategy corresponding to the database type; determine the corresponding data integration task. Data collection mode; if the data collection mode is the full collection mode, the full data integration script corresponding to the target metadata identifier is generated according to the script generation strategy and data collection information, and the full data integration script is used as the task script of the data integration task. By applying the method provided by the embodiment of the present invention, a script generation strategy can be determined based on the data collection information of a data integration task and a preset data lake metadata information base, and then a task script can be generated. The task script of the data integration task can be generated through the automated processing process, and the automatic generation of the data integration script can be realized without manual script writing, which can save a lot of human resources, shorten the processing time, and improve the work efficiency. Secondly, it can avoid human mistakes caused by manual scripting, and avoid the adverse effects of human factors on data integration work.

On the basis of the method shown in FIG. 1 , in the method provided by the embodiment of the present invention, the process of determining the data collection information corresponding to the data integration task mentioned in step S101 includes:

Obtain the business subject, the department, the product, the metadata identifier and the collection mode entered by the user, and use the business subject, the department, the product, the metadata identifier and the collection mode as the Data collection information corresponding to the data integration task.

In the method provided by the embodiment of the present invention, the user can input the relevant information of the metadata to be collected at the front end, including the business subject, the affiliated department, the affiliated product, the metadata identification and the collection mode, and the information input by the user can be regarded as the data integration task. Corresponding data collection information.

It should be noted that the specific content of the data collection information mentioned in the method provided by the embodiment of the present invention is only to better illustrate a specific embodiment provided by the method provided by the present invention. The collected information may also contain other data content.

On the basis of the method shown in FIG. 1, in the method provided by the embodiment of the present invention, as mentioned in step S102, according to the data collection information, in the preset data lake metadata information database, determine the data integration The process of identifying the target metadata corresponding to the task, including:

The data collection information is respectively matched with the metadata information corresponding to each of the preset metadata identifiers, and the preset metadata identifier corresponding to the metadata information matched with the data collection information is determined as The target metadata identifier.

In the method provided by the embodiment of the present invention, each metadata information in the metadata information database of the data lake can be matched with the data collection information respectively. Specifically, the metadata identifier included in the metadata information can be matched with the metadata in the data collection information The metadata identifiers are compared, and if the two are the same, it is considered that the metadata information matches the data collection information. The preset metadata identifier corresponding to the metadata information matching the data collection information is used as the target metadata identifier. In a specific application scenario, the preset metadata identifiers in the metadata information database of the data lake are different from each other, so usually only one metadata information matches the data collection information.

On the basis of the method shown in FIG. 1, the method provided by the embodiment of the present invention further includes:

If the data collection mode is not the full collection mode, determining the start and end conditions corresponding to the data integration task;

In the method provided by the embodiment of the present invention, if the data collection mode corresponding to the data integration task is not the full collection mode, that is, the data collection mode is the incremental collection mode. When the user selects the incremental acquisition mode, the start and end conditions of the data acquisition can be input through the front end, and the start and end conditions may include a start condition and an end condition, or only a start condition without an end condition. The start and end conditions refer to conditions that characterize the data range, such as data generated after the first preset time point and before the second preset time point, and so on.

According to the target metadata information and the starting and ending conditions, condition checking is performed on the data integration task;

In the method provided by the embodiment of the present invention, the condition check of the data integration task can be performed based on the target metadata information and the start and end conditions, that is, it is determined whether the start and end conditions corresponding to the data integration task are feasible according to the target metadata information.

If the data integration task passes the condition check, an incremental data integration script corresponding to the target metadata identifier is generated according to the script generation strategy, the data collection information, the start and end conditions, and the target metadata information , and use the incremental data integration script as the task script of the data integration task.

In the method provided by the embodiment of the present invention, if the data integration task passes the condition verification, an incremental data integration script is generated based on the script generation strategy, data collection information, start and end conditions, and target metadata information, and the incremental data integration script is generated. Scripts are task scripts for data integration tasks. Specifically, the script template for incremental collection in the script generation strategy can be generated based on the metadata identification, business subject, department, product and other data in the data collection information, start and end conditions, and incremental variables contained in the target metadata information. Identify data such as identifiers, perform parameter replacement, and generate incremental data integration scripts based on the script template that completes the parameter replacement.

On the basis of the method provided by the above embodiment, in the method provided by the embodiment of the present invention, the process of performing condition verification on the data integration task according to the target metadata information and the start and end conditions includes:

determining the incremental information corresponding to the target metadata information, where the incremental information includes an incremental variable identifier and an incremental variable data structure;

In the method provided by the embodiment of the present invention, the target metadata information includes an incremental variable identifier and an incremental variable data structure corresponding to the target metadata identifier, and the incremental variable identifier and the incremental variable data structure can be determined as incremental information.

Judging whether the start and end conditions match the incremental information;

In the method provided by this embodiment of the present invention, it can be determined whether the start and end conditions match the incremental information through the increment variables involved in the start and end conditions. For example, the increment variable involved in the start and end conditions is time, while the increment information The incremental variable identifier in the incremental variable and the incremental variable represented by the incremental variable data structure are also time, then it is determined that the start and end conditions match the incremental information. If the increment variable is different, it is considered that the two do not match.

If the start and end conditions match the incremental information, determine that the data integration task passes the condition check;

If the start and end conditions do not match the incremental information, it is determined that the data integration task fails the condition check.

In the method provided by the embodiment of the present invention, if the start and end conditions match the incremental information, it is determined that the data integration task has passed the condition check; otherwise, the condition check has not been passed.

On the basis of the methods provided in the foregoing embodiments, the methods provided in the embodiments of the present invention further include:

If the data integration task fails the condition check, an error message is prompted, and the generation process of the task script of the data integration task ends.

In the method provided by the embodiment of the present invention, if the data integration task fails the condition check, an error message is reported, and the generation process of the task script is ended.

In order to better illustrate the method provided by the embodiment of the present invention, the embodiment of the present invention provides another method for generating a data integration script in combination with an actual application scenario.

The method provided by the embodiment of the present invention is applied to a data analysis scenario based on a data lake architecture in a banking institution. A data lake architecture is a way of storing data in a natural format in a system or repository. A data lake is a large warehouse that stores various raw data of an enterprise, which can be accessed, processed, analyzed and transmitted. The three levels of the data lake are divided into underlying storage such as databases, metadata management, and SQL engines across different data sources. Whereas traditional data warehouses store data in relational tables, data lakes use a flat structure. Each data element is assigned a unique identifier and is marked with a set of metadata tags.

The application process of the method provided by the embodiment of the present invention includes:

Sort out all enterprise-level metadata in the data lake and its related business topics, departments, products, collection modes, incremental variable identifiers, and incremental variable data structures and other ETL-related information, as the subsequent ETL parameters (that is, metadata). data information) to prepare for import, and the sorting content can be recorded according to the structure shown in the following table:

Table 1

Among them, the order of index items from high-level to low-level is: business topic to which they belong > department to which they belong > product to which they belong > metadata.

Create a task number to prepare for the subsequent import of metadata parameter information and new ETL parameters. The task number data structure can be as follows:

Table 2

For the first-time imported base-laying data, the task number required for the first base-laying needs to be established to prepare for the subsequent base-laying import.

Select the established task number and import the metadata ETL parameters (that is, the metadata information corresponding to the metadata identifier). For the import of the metadata ETL parameters, the system assigns a unique identifier to each data element during import, and uses a set of metadata The content of the metadata tag (18 bytes in total) is as follows:

table 3

Store the metadata ETL parameter table, where the structure of the ETL parameter table can be as follows:

Table 4

The database type may be Hive, HBase, Hdfs, Nosql, etc., but is not limited to these types.

After the ETL parameter table is constructed, when data analysis needs to obtain and load data, the user can select the metadata, collection mode, incremental condition or segment interval that requires ETL according to the number fetching rules provided by the data analyst. Choose to automatically match the parameters in the ETL parameter table, and obtain the database type of the corresponding metadata. If the acquisition mode is incremental, obtain the incremental variable identifier, and automatically generate the ETL script for this data analysis.

ETL parameters support adding, deleting, modifying, checking and maintaining. When ETL parameters are added in the future, the system will add the task number, metadata name, business subject, department, product, collection mode and input selected by the operator. Quantitative variable identification, incremental variable data structure and other information, the parameter validity check is carried out. After the check is passed, the ETL parameters of the metadata are established. The query of ETL parameters supports metadata-level ETL parameter query and panorama-level list query. Metadata The level ETL parameter query displays all ETL parameters related to the metadata, and the panorama level list query displays all metadata ETL parameters.

Since the ETL work is closely related to the task, the related data ETL work is only provided after the data analysis task is started. The processing mechanism is mainly to increase the authorization of the data lake data ETL.

The method provided by the embodiment of the present invention can be applied to a data integration platform. The data integration platform is an instantiation of the method shown in FIG. 1 , and the platform may include the following modules:

Online module, online module mainly includes the following sub-modules:

Task maintenance sub-module: The task maintenance sub-module is mainly responsible for the maintenance function of task information, mainly the addition, deletion, modification and checking of ETL parameters in the ETL parameter table Online maintenance transactions, the main information includes metadata tags, metadata names, collection modes, incremental variables Information about identification, incremental variable data structures, task numbers, etc. is maintained online for transactions. The ETL required for data analysis in the data lake is mainly driven by the task of data analysis. Without the task of data analysis, no one or organization can arbitrarily change the ETL parameters that have been published, including the first laying of the groundwork. Bottom import.

The maintenance of batch task information mainly includes four functions: adding, modifying, deleting, and checking. Each operation must be authorized by the supervisor before it can take effect.

New sub-module for ETL parameters of metadata: This sub-module mainly completes the addition of ETL parameters of metadata. There are two ways to add ETL parameters of metadata: single addition and batch import. The main fields of the new online transaction interface for a single transaction include: metadata name, collection mode, business subject, department, product, database type, incremental variable identifier, incremental variable data structure, and task number, among which the task number For the pre-established task label, select from the drop-down menu. Batch import supports the function of loading one or more ETL parameters through the import table after filling in the EXCEL import table according to the template in Table 1. When adding ETL parameters, the system will check the validity of the ETL parameters according to the metadata name entered by the operator, the business subject of the metadata, and the product it belongs to. The ETL parameter record.

Metadata label generation sub-module: When the user submits a new transaction after the user's input or import is completed, the system generates the entered or imported metadata label according to the relevant information encoding of the metadata ETL parameters submitted by the user, and treats it as a piece of information Stored in the database in subsequent storage modules.

Query sub-module: The query of ETL parameters supports single-record-level query and panorama-level list query. The single-record-level query displays all ETL parameters related to the record, and the panorama-level list query displays the ETL parameters of all records.

Delete submodule: Contrary to the add operation, the delete operation will remove the ETL parameter of the corresponding record from the ETL parameter table.

Modify sub-module: This sub-module supports selecting a list to modify in the ETL parameter list query. The system will regenerate the metadata tag information based on the modified information and update it to the relevant fields in the modified record.

Storage module: mainly includes ETL parameter storage.

ETL generation module: It mainly provides a visual online transaction screen, allowing users to select the ETL generation conditions required for data analysis according to their needs, and automatically generate the required ETL scripts after submission.

In the method provided by the embodiment of the present invention, the user interface for the user to submit the relevant information of the data integration task may be as shown in FIG. 2, including the selection of the task number, the business topic, the department, the product, the metadata (identification), the collection mode, Input of start condition and end condition, among which, start condition and end condition can be filled in manually, and other data items can be selected from drop-down menu.

As shown in FIG. 3 , the data integration script generation process provided by the embodiment of the present invention specifically includes:

S201: The user submits the task;

In the method provided by the embodiment of the present invention, the front-end interface shown in FIG. 2 can be initialized, and the user can input the relevant information of the ETL through the front-end interface, including selecting the data analysis task number, the business topic to which the data belongs, the department to which it belongs, and the product to which it belongs. , select the metadata (identification) and select the collection mode, and submit the task of generating the ETL script by clicking the submit control.

S202: Determine whether the collection mode selected by the user is the full collection mode;

S203: If the collection mode is not the full collection mode, determine whether the collection mode is the interval collection mode;

In the method provided by the embodiment of the present invention, the incremental acquisition mode can be further subdivided into an interval acquisition mode and a non-interval acquisition mode. In the interval acquisition mode, the user needs to input a start condition and an end condition. , the user only needs to enter the starting conditions.

S204: If the collection mode is the interval collection mode, obtain the start condition and the end condition input by the user to determine the start and end conditions;

S205: If the collection mode is a non-interval collection mode, obtain the starting condition input by the user, and the default ending condition is also the starting condition, so as to determine the starting and ending conditions;

S206: Read in the ETL parameters corresponding to the metadata selected by the user;

S207: Obtain the data structure corresponding to the database type, the incremental variable identifier and the incremental variable identifier from the ETL parameters;

S208: Check whether the data structure of the incremental variable (that is, the data structure corresponding to the incremental variable identifier) matches the start and end conditions;

S209: According to the inspection result, determine whether the inspection has passed;

S210: If the check is passed, an incremental ETL script is generated according to the information input by the user, the database type, the incremental variable identifier and the starting and ending conditions;

S211: Show the ETL script to the user so that the user can review it;

S212: If the check fails, report an error and exit;

S213: If in the judgment process of step S202, it is judged that the collection mode is the full collection mode, then read the ETL parameter corresponding to the metadata selected by the user;

S214: Obtain the database type from the ETL parameter;

S215: Generate a full ETL script according to the information input by the user and the database type, and go to step S211.

Based on the method provided by the embodiments of the present invention, ETL scripts can be quickly generated on the basis of the data lake architecture, the repetitive work in the ETL operation infrastructure can be eliminated, and the data sets in the data lake can be discovered and used for query and analysis. Greatly shortens the time for ETL and data cataloging in analysis projects, making ETL generation more automatic and smarter.

Corresponding to the method for generating a data integration script shown in FIG. 1 , an embodiment of the present invention further provides a device for generating a data integration script, which is used for the specific implementation of the method shown in FIG. 1 , and its structural diagram is shown in the figure. 4, including:

a first determining unit 301, configured to determine data collection information corresponding to the data integration task when a task script of the data integration task needs to be generated;

The second determining unit 302 is configured to determine, according to the data collection information, in a preset data lake metadata information base, a target metadata identifier corresponding to the data integration task and a target metadata corresponding to the target metadata identifier data information, the data lake metadata information base contains a plurality of preset metadata identifiers and metadata information corresponding to each of the preset metadata identifiers;

A third determining unit 303, configured to determine the database type corresponding to the target metadata identifier according to the target metadata information;

a fourth determining unit 304, configured to determine a script generation strategy corresponding to the database type;

a fifth determining unit 305, configured to determine a data collection mode corresponding to the data integration task;

The first generating unit 306 is configured to generate a full data integration script corresponding to the target metadata identifier according to the script generation strategy and the data collection information, if the data collection mode is the full data collection mode, and integrate all the data. The full data integration script is used as the task script of the data integration task.

Based on the device provided by the embodiment of the present invention, when a task script of a data integration task needs to be generated, the data collection information corresponding to the data integration task is determined; according to the data collection information, in the preset data lake metadata information base, the data integration task is determined The target metadata identifier corresponding to the task and the target metadata information corresponding to the target metadata identifier, according to the target metadata information, determine the database type corresponding to the target metadata identifier; determine the script generation strategy corresponding to the database type; determine the corresponding data integration task. Data collection mode; if the data collection mode is the full collection mode, the full data integration script corresponding to the target metadata identifier is generated according to the script generation strategy and data collection information, and the full data integration script is used as the task script of the data integration task. By applying the device provided by the embodiment of the present invention, a script generation strategy can be determined based on the data collection information of the data integration task and the preset data lake metadata information base, and then a task script can be generated. The task script of the data integration task can be generated through the automated processing process, and the automatic generation of the data integration script can be realized without manual script writing, which can save a lot of human resources, shorten the processing time, and improve the work efficiency. Secondly, it can avoid human mistakes caused by manual scripting, and avoid the adverse effects of human factors on data integration work.

On the basis of the device provided by the foregoing embodiment, the device provided by the embodiment of the present invention further includes:

a sixth determination unit, configured to determine the start and end conditions corresponding to the data integration task if the data collection mode is not the full collection mode;

a verification unit, configured to perform condition verification on the data integration task according to the target metadata information and the start and end conditions;

A second generating unit, configured to generate the target metadata identifier according to the script generation strategy, the data collection information, the start and end conditions, and the target metadata information if the data integration task passes the condition check corresponding incremental data integration script, and use the incremental data integration script as the task script of the data integration task.

On the basis of the device provided by the above embodiment, in the device provided by the embodiment of the present invention, the first determining unit 301 includes:

The obtaining subunit is used to obtain the business subject, the department to which the user belongs, the product to which it belongs, the metadata identifier and the collection mode entered by the user, and the The collection mode is used as the data collection information corresponding to the data integration task.

On the basis of the apparatus provided by the above embodiment, in the apparatus provided by the embodiment of the present invention, the second determining unit 302 includes:

A matching subunit, configured to respectively match the data collection information with the metadata information corresponding to each of the preset metadata identifiers, and match the preset metadata information corresponding to the metadata information matched with the data collection information The metadata identifier, which is determined as the target metadata identifier.

On the basis of the device provided by the above embodiment, in the device provided by the embodiment of the present invention, the verification unit includes:

a first determination subunit, configured to determine incremental information corresponding to the target metadata information, where the incremental information includes an incremental variable identifier and an incremental variable data structure;

A judging subunit for judging whether the start and end conditions match the incremental information;

a second determination subunit, configured to determine that the data integration task passes the condition check if the start and end conditions match the incremental information; if the start and end conditions do not match the incremental information, determine that the The data integration task described above failed the condition check.

On the basis of the device provided by the foregoing embodiment, the device provided by the embodiment of the present invention further includes:

An error reporting subunit, configured to report an error if the data integration task fails the condition check, and end the generation process of the task script of the data integration task.

An embodiment of the present invention further provides a storage medium, where the storage medium includes stored instructions, wherein when the instructions are executed, a device where the storage medium is located is controlled to execute the above-mentioned method for generating a data integration script.

An embodiment of the present invention also provides an electronic device, the schematic structural diagram of which is shown in FIG. 5 , and specifically includes a memory 401 and one or more instructions 402 , wherein one or more instructions 402 are stored in the memory 401 and processed through the memory 401 . The one or more instructions 402 are configured to be executed by the one or more processors 403 to:

When a task script of the data integration task needs to be generated, determine the data collection information corresponding to the data integration task;

According to the data collection information, in the preset data lake metadata information database, determine the target metadata identifier corresponding to the data integration task and the target metadata information corresponding to the target metadata identifier, the data lake metadata The data information base contains a plurality of preset metadata identifiers and metadata information corresponding to each of the preset metadata identifiers;

According to the target metadata information, determine the database type corresponding to the target metadata identifier;

determining the script generation strategy corresponding to the database type;

determining the data collection mode corresponding to the data integration task;

If the data collection mode is the full data collection mode, generate a full data integration script corresponding to the target metadata identifier according to the script generation strategy and the data collection information, and use the full data integration script as the Task scripts for data integration tasks.

It should be noted that the data integration script generation method and device, storage medium and electronic device provided by the present invention can be used in the financial field or other fields, for example, can be used in data analysis application scenarios in the financial field. The other field is any field other than the financial field, for example, the field of communication services. The above is only an example, and does not limit the application fields of the data integration script generation method and device, storage medium and electronic device provided by the present invention.

Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for related parts. The systems and system embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, It can be located in one place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the possibilities of hardware and software. Interchangeability, the above description has generally described the components and steps of each example in terms of functionality. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. a data integration script generation method, is characterized in that, comprises: When a task script of the data integration task needs to be generated, determine the data collection information corresponding to the data integration task; According to the data collection information, in the preset data lake metadata information database, determine the target metadata identifier corresponding to the data integration task and the target metadata information corresponding to the target metadata identifier, the data lake metadata The data information base contains a plurality of preset metadata identifiers and metadata information corresponding to each of the preset metadata identifiers; According to the target metadata information, determine the database type corresponding to the target metadata identifier; determining the script generation strategy corresponding to the database type; determining the data collection mode corresponding to the data integration task; If the data collection mode is the full data collection mode, generate a full data integration script corresponding to the target metadata identifier according to the script generation strategy and the data collection information, and use the full data integration script as the Task scripts for data integration tasks. 2. The method according to claim 1, wherein the determining the data collection information corresponding to the data integration task comprises: Obtain the business subject, the department, the product, the metadata identifier and the collection mode entered by the user, and use the business subject, the department, the product, the metadata identifier and the collection mode as the Data collection information corresponding to the data integration task. 3 . The method according to claim 1 , wherein determining the target metadata identifier corresponding to the data integration task in a preset data lake metadata information base according to the data collection information, comprising: 4 . : The data collection information is respectively matched with the metadata information corresponding to each of the preset metadata identifiers, and the preset metadata identifier corresponding to the metadata information matched with the data collection information is determined as The target metadata identifier. 4. The method of claim 1, further comprising: If the data collection mode is not the full collection mode, determining the start and end conditions corresponding to the data integration task; According to the target metadata information and the starting and ending conditions, condition checking is performed on the data integration task; If the data integration task passes the condition check, an incremental data integration script corresponding to the target metadata identifier is generated according to the script generation strategy, the data collection information, the start and end conditions, and the target metadata information , and use the incremental data integration script as the task script of the data integration task. 5 . The method according to claim 4 , wherein the condition checking on the data integration task according to the target metadata information and the start and end conditions comprises: 5 . determining the incremental information corresponding to the target metadata information, where the incremental information includes an incremental variable identifier and an incremental variable data structure; Judging whether the start and end conditions match the incremental information; If the start and end conditions match the incremental information, determine that the data integration task passes the condition check; If the start and end conditions do not match the incremental information, it is determined that the data integration task fails the condition check. 6. The method of claim 4, further comprising: If the data integration task fails the condition check, an error message is prompted, and the generation process of the task script of the data integration task ends. 7. A data integration script generation device, characterized in that, comprising: a first determining unit, configured to determine data collection information corresponding to the data integration task when a task script of the data integration task needs to be generated; The second determining unit is configured to, according to the data collection information, determine the target metadata identifier corresponding to the data integration task and the target metadata corresponding to the target metadata identifier in the preset data lake metadata information database information, the data lake metadata information base contains multiple preset metadata identifiers and metadata information corresponding to each of the preset metadata identifiers; a third determining unit, configured to determine the database type corresponding to the target metadata identifier according to the target metadata information; a fourth determining unit, configured to determine a script generation strategy corresponding to the database type; a fifth determination unit, configured to determine a data collection mode corresponding to the data integration task; The first generation unit is configured to generate a full data integration script corresponding to the target metadata identifier according to the script generation strategy and the data collection information if the data collection mode is the full collection mode, and integrate the The full data integration script is used as the task script of the data integration task. 8. The apparatus of claim 7, further comprising: a sixth determination unit, configured to determine the start and end conditions corresponding to the data integration task if the data collection mode is not the full collection mode; a verification unit, configured to perform condition verification on the data integration task according to the target metadata information and the start and end conditions; a second generating unit, configured to generate the target metadata identifier according to the script generation strategy, the data collection information, the start and end conditions and the target metadata information if the data integration task passes the condition check corresponding incremental data integration script, and use the incremental data integration script as the task script of the data integration task. 9 . A storage medium, characterized in that the storage medium comprises stored instructions, wherein when the instructions are executed, a device where the storage medium is located is controlled to execute the data according to any one of claims 1 to 6 Integration script generation method. 10. An electronic device, comprising a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors as claimed in claims 1- 6. The method for generating a data integration script according to any one of the items.

Images