CN116028192A - A multi-source heterogeneous data acquisition method, device and storage medium - Google Patents

Abstract

The application discloses a multi-source heterogeneous data acquisition method, a multi-source heterogeneous data acquisition device and a storage medium, and aims to realize universal multi-source heterogeneous data acquisition through a distributed system and improve data processing efficiency. The method comprises the following steps: determining the type of a data source and configuring data source information of the data source; configuring a task scheduler, wherein the task scheduler is used for executing tasks at regular time, periodically executing the tasks, determining service nodes and determining an execution strategy; creating a data acquisition task, wherein the data acquisition task comprises a data source, a data target source and a data acquisition strategy; executing the data acquisition task according to the data acquisition strategy through the configured task scheduler; and outputting a data acquisition result.

Description

A multi-source heterogeneous data acquisition method, device and storage medium

technical field

The present application relates to the field of data processing, in particular to a multi-source heterogeneous data acquisition method, device and storage medium.

Background technique

The 21st century is an era of big data, data is everywhere and exists in all aspects of life. Whether it is for data analysis or product needs, we need to extract interesting and valuable content from certain websites. However, because collecting data is a repetitive task, and manpower is exhausted, reptiles came into being and developed rapidly.

As the underlying structure of data application and digitalization, data collection and access plays an irreplaceable role in data governance, collection and unification of data resources in various projects.

From the perspective of big data, it is necessary to carry out in-depth research on data collection methodology and integrate multiple data sources. It is necessary to consider the association and fusion processing of multi-source heterogeneous data sources to consolidate the infrastructure.

A reasonable data collection method is a key step to make data available. The diversity of data sources has become a basic feature in the big data environment. However, the traditional data collection source is single, and the amount of data stored, managed and analyzed is relatively small. Relational databases and parallel data warehouses are used. Therefore, in the context of diverse information sources and massive data volumes, the challenges faced by data acquisition technology have become particularly prominent. Data collection methods can be divided into offline collection, real-time collection, and Internet collection according to data types.

Contents of the invention

In order to solve the above technical problems, the present application provides a multi-source heterogeneous data acquisition method, device and storage medium.

The first aspect of the present application provides a multi-source heterogeneous data collection method, the method comprising:

Determine the type of data source, and configure the data source information of the data source;

Configuring a task scheduler, the task scheduler is used to execute tasks regularly, execute tasks periodically, determine service nodes, and determine execution strategies;

Create a data collection task, the data collection task includes a data source, a data target source, and a data collection strategy;

Execute the data collection task according to the data collection strategy through the configured task scheduler;

Output data collection results.

Optionally, executing the data collection task according to the data collection strategy through the configured task scheduler includes:

The data source information includes a data source table, and the data source table lists the fields of the data to be collected;

Select the data source and download all the data in the data source;

Determining the target source table to be collected in all the data;

Constructing a mapping relationship between the data source table and the target source table according to the correlation between the fields in the data source table and the fields in the target source table;

Data collection is performed according to the mapping relationship.

Optionally, when the data source is a website, the data source information configuring the data source includes:

Obtain pre-configured acquisition script information;

If the script is a custom script, obtain the custom script file;

If the script is a java script, after obtaining the script file, configure the package name, class name, and method name of the script.

Optionally, when the data source is a relational database, the data source information configuring the data source includes:

Configure the ip address, port, user name and password information of the data source;

Described relational database comprises: MySQL, Oracle, SQLServer, PostgreSQL, Hive, HDFS, MongoDB, Gbase, Kingbase.

Optionally, the configuration task scheduler includes:

Configure blocking processing strategies, configure subtasks, configure task retry strategies, configure task execution trigger strategies, and execute alarm strategies.

Optionally, it also includes: during the process of collecting data, generating task logs and statistical reports;

A detailed error log during the execution of the task is recorded in the task log;

The statistical report includes: data collection information and node information, the data collection information includes: total amount, total capacity, new incremental information of the day and 30-day trend statistics; the node information includes: CPU usage rate, memory usage rate , the total amount of memory, the remaining amount of memory; task information: number of successes, number of failures, number of execution suspensions, and a weekly statistical chart.

The second aspect of the present application provides a multi-source heterogeneous data acquisition device, including:

A determining unit, configured to determine the type of the data source, and configure the data source information of the data source;

The configuration unit is used to configure a task scheduler, and the task scheduler is used to execute tasks periodically, execute tasks periodically, determine service nodes, and determine execution strategies;

A creation unit is used to create a data collection task, and the data collection task includes a data source, a data target source and a data collection strategy;

The collection unit is configured to execute the data collection task according to the data collection strategy through the configured task scheduler;

The output unit is used to output data collection results.

Optionally, the collection unit is specifically used for:

The data source information includes a data source table, and the data source table lists the fields of the data to be collected;

Select the data source and download all the data in the data source;

Determining the target source table to be collected in all the data;

Constructing a mapping relationship between the data source table and the target source table according to the correlation between the fields in the data source table and the fields in the target source table;

Data collection is performed according to the mapping relationship.

The third aspect of the present application provides a multi-source heterogeneous data acquisition device, the device includes:

Processor, memory, I/O unit and bus;

The processor is connected to the memory, the input and output unit and the bus;

The memory stores a program, and the processor invokes the program to execute the first aspect and any optional method in the first aspect.

The fourth aspect of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a program, and when the program is executed on a computer, the first aspect and any optional one of the first aspect are executed. the method.

As can be seen from the above technical solutions, the present application has the following advantages:

This application provides a multi-source heterogeneous data collection method. Before data collection, through the configuration of task scheduler and data source, the applicability and ease of use of the method can be improved, and the dependence on user expertise can be reduced. At the same time, it can be extended horizontally according to the needs to realize distributed data collection. After the configuration is completed, it can be applied to the automatic collection of various types of data through the task scheduler, which integrates various types of data such as web page data, offline data, and real-time data. The collection function makes it not affected by the heterogeneity of data sources and data storage systems when crawling data from massive multi-data sources, which greatly improves the efficiency of data collection.

Description of drawings

In order to illustrate the technical solutions in this application more clearly, the accompanying drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the application. Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort.

Fig. 1 is a schematic flow chart of an embodiment of the multi-source heterogeneous data acquisition method provided in the present application;

FIG. 2 is a schematic structural diagram of an embodiment of a multi-source heterogeneous data acquisition device provided in the present application;

Fig. 3 is a schematic structural diagram of another embodiment of the multi-source heterogeneous data collection device provided in this application.

Detailed ways

As the underlying structure of data application and digitalization, data collection and access plays an irreplaceable role in data governance, collection and unification of data resources in various projects.

A reasonable data collection method is a key step to make data available. The diversity of data sources has become a basic feature in the big data environment. However, the traditional data collection source is single, and the amount of data stored, managed and analyzed is relatively small. Relational databases and parallel data warehouses are used. Therefore, in the context of diverse information sources and massive data volumes, the challenges faced by data acquisition technology have become particularly prominent. Data collection methods can be divided into offline collection, real-time collection, and Internet collection according to data types.

For several existing data collection methods:

(1) Offline collection

The core of the offline data collection method is the ETL tool. In the context of the data warehouse, the ETL data collection process includes data extraction (Extract), transformation (Transform) and loading (Load). Common mainstream ETL tools include Apache Sqoop, Kettle, Talend, etc.

Apache Sqoop (SQL-to-Hadoop) is mainly used for data transmission between HDFS/Hive and relational databases (Mysql, Oracle, SQL server, etc.), and is the bridge between Hodoop and SQL. Sqoop can import data from a relational database into HDFS/Hive/Hbase, and can also import data from HDFS into a relational database. Using Mapreduce as the underlying engine, it can synchronize data in parallel with high efficiency. Kettle is an open source ETL tool written in Java launched by Pentaho. It can run on Windows, Linux, and Unix. It has efficient and stable data extraction, powerful functions, API, supports secondary development, and is easy to integrate. Kettle has two script files, transformation and job. Transformation completes the basic conversion of data, and job completes the control of the entire workflow. Talend is an ETL tool that supports AWS Redshift and S3. It has a graphical interface and can use Hadoop, Spark, SparkStream and NoSQL databases in a drag-and-drop manner without code.

ETL usually adopts a graphical configuration method, which is simple and flexible, so that users do not need to care too much about various internal details of the database, but focus on functions and simplify user operations; support all common data sources, such as Oracle, Sqlserver, DB2, Mysql, Access, etc. also provide support for various flat data sources, such as txt, excel, csv, xml, etc.; strong compatibility, support various hardware and software platforms, software platforms such as operating systems, support windows, linux and Domestic operating system; supports various hardware platforms at the same time, such as x86, Godson, etc.; has more powerful functions, rich data processing components, stronger versatility, and easy reuse of components; provides flexible customization rules and better control Data quality; provide powerful management functions, such as rights management, log management. However, most ETL tools do not have real-time data perception of the database, and cannot meet the scenarios of real-time processing of big data. Tools for real-time database synchronization, such as Ali's Canal, can realize real-time data synchronization by reading the mysqlbin log.

(2) Real-time collection

Common collection tools for log files include Flume, Logstash, FileBeat, etc. Real-time collection is mainly used in business scenarios that consider stream processing, for example, to record various operational activities performed by data sources, such as traffic management for network monitoring, financial User access behavior recorded by the application's stock accounting and web server. In the stream processing scenario, data acquisition will become a consumer of Kafka, intercept the continuous stream of upstream data, and then perform corresponding processing according to the business scenario (such as deduplication, denoising, intermediate calculation, etc.), and then write to the corresponding in the data storage. This process is similar to traditional ETL, but it is a stream processing method rather than a scheduled batch processing job.

(3) Internet data collection

The means of Internet data collection is web crawler technology, which uses Internet search engine technology to achieve targeted, industry-specific, and precise data capture, and classifies data according to certain rules and screening standards, and forms a database file. process. With the development of Internet technology and the growth of massive information, the acquisition and sorting of information has become an increasing demand. At this stage, there are many tools for Internet data acquisition in China. The mainstream tools on the market mainly include locomotives and octopus Fish, Jisooke, Archer Cloud Crawler and Madman Collector, etc. Crawlers obtain interesting data on the target website by writing a piece of code, reducing human workload. The main process of the crawler: obtain the URL of an initial webpage, obtain all the information on the initial URL, grab the content we are interested in again from the obtained information, parse and match the content, and finally save it to the database. The crawler can be divided into three modules as a whole: network request module, crawling process control module, and content analysis and extraction module.

For the network request module, the essence is to send and receive http(s) requests. HTTP request information consists of three parts: request line (line), request header (headers), and request body (body). The request line includes the request method and the request address URL. The common request methods (methods) are: GET, POST methods, and also include OPTIONS, DELETE, PUT and other methods. The request header generally includes information such as Accept, Referer, Accept-language, User-Agent, and Host. The request body contains key-value pairs of parameter names and values.

The crawling process control module is mainly to set the crawling rate, whether an agent is needed, and how to crawl. If the crawling rate is too fast, the IP will be blocked, or the server of the other party will crash, so it is necessary to limit the crawling rate. Secondly, after crawling the same website multiple times, the account IP may be blocked. At this time, we need to join a proxy to solve this problem. For different websites, due to the different front-end writing methods, the crawling methods may also be different, such as: WebClient, WebDriver, etc.

The content analysis and extraction module mainly obtains the part of the content we are interested in after obtaining the entire content of the corresponding URL address. A common method is to obtain the xpath path by setting the corresponding node node and attributes, and then obtain the information of the corresponding node through the xpath path. Most of the web page information formats are Html, Json, and JavaScript, which are often different from the format of our storage information. Therefore, after obtaining the corresponding information, we need to analyze and convert the format, and then store it in the database.

Today's crawler functions on the market are generally divided into three parts: data collection (downloading related web pages), data processing (analyzing the data of related web pages) and data storage (saving the content to be crawled). Advanced crawlers use distributed technology and concurrent programming for related data crawling and analysis. In the past, the website pages crawled by crawlers were mainly html documents themselves, so most of the content crawled by crawlers was contained in html. However, with the development of front-end technology, the probability of dynamic web pages is increasing. These dynamic websites use ajax technology. Compared with traditional html documents, the information of many web pages is dynamically generated through javascript.

The diversity of web pages makes common crawlers unable to apply to web pages of different formats. When crawling websites of different formats, back-end programmers need to modify the original code, which greatly reduces work efficiency and code maintainability. .

Based on this, the present application improves and optimizes the existing technology, and provides a multi-source heterogeneous data collection method, which aims to realize general multi-source heterogeneous data collection through a distributed system and improve data processing efficiency.

It should be noted that the multi-source heterogeneous data collection method provided by this application can be applied to terminals, systems, or servers. For example, terminals can be smart phones or computers, tablet computers, smart TVs, smart Wrist watches and portable computer terminals may also be fixed terminals such as desktop computers. For convenience of description, in this application, a terminal is used as an execution subject for illustration.

Embodiments of the application are described in detail below:

Please refer to Figure 1, Figure 1 is a schematic flow chart of an embodiment of the multi-source heterogeneous data acquisition method provided by the present application, the method includes:

101. Determine the type of the data source, and configure the data source information of the data source;

The method provided by this application can automatically collect different types of data sources. At the beginning, it is necessary to determine the type of data source and configure the data source information of the data source in the system. The configuration process can be guided by visual prompt information. In order to reduce the difficulty of operation for operators, this application is constructed using Framework + plugin architecture. Applicable data sources can be relational or non-relational databases, structured, unstructured or semi-structured databases, relational or non-relational data sources mainly include: MySQL, Oracle, SQLServer, PostgreSQL, Hive , HDFS, MongoDB, Gbase, Kingbase, etc. For relational databases, the main configuration data source information includes ip address, port, user name, and password information. Semi-structured data sources include txt, csv, excel, access. For semi-structured data sources, the main configuration data source information includes communication protocol type (ftp/sftp), communication protocol information (ip, port, user, Password, file path) and other information, as well as data file information (such as start line, start column, file content delimiter); custom script collection needs to upload script files, java script additionally needs to configure package name, class name, method Name, Js script and python script need to define the startup method in the script; real-time data data sources mainly include: kafka, this type of data source needs to be configured with service address, group ID of consumption topic, key and value decoder, etc.

102. Configuring a task scheduler, the task scheduler is used to execute tasks regularly, execute tasks periodically, determine service nodes, and determine execution strategies;

In the process of actual data collection, the collection task is executed through the task scheduler. Before the collection, it is necessary to configure the task scheduler for the actual data source. Configuring the task scheduler can specifically determine the service node, determine the execution strategy, Determining the execution period and time, etc. may also include configuring blocking processing strategies, configuring subtasks, configuring task retry strategies, configuring task execution trigger strategies, and executing alarm strategies. For example, configuring the blocking processing strategy can include: single-machine serial - put into the task execution queue, execute in sequence according to time, discard subsequent scheduling - put the task into the queue, this task will be discarded and marked as failed, overwrite the previous scheduling - Put the task execution queue, the previous task will be terminated and run this time.

Configuration subtasks can include:

If task construction needs to trigger the next task when the execution of this task is completed and executed successfully, the tasks are interdependent, task A -> task B -> task C, that is to say, tasks are scheduled serially. Then you can run another task as a subtask of this task. Therefore, the strategy for configuring subtasks may include triggering execution conditions of the subtasks, for example, the triggering execution conditions are that the execution of the current task is completed, or a specified time is reached.

Configuring task retry policies can include:

Customize the number of retries for task failures. When the task fails, it will actively retry according to the preset number of failed retries; among them, the shard task supports shard-grained failure retries, and other retry strategies can be used in practice , is not limited here.

Configuring trigger policies can specifically include:

Configure the trigger mode. In addition to "Cron mode" and "task-dependent mode" to trigger task execution, you can also trigger task mode based on events.

103. Create a data collection task, where the data collection task includes a data source, a data target source, and a data collection strategy;

104. Execute the data collection task according to the data collection strategy through the configured task scheduler;

105. Output the data collection result.

After the data source and the task scheduler are respectively configured through steps 101 and 102, a specific data collection task is created, which includes specific data sources, data target sources and data collection strategies.

Wherein, through the configured task scheduler, the specific manner of executing the data collection task according to the data collection strategy is as follows:

The data source information includes a data source table, and the data source table lists the fields of the data to be collected;

Select the data source and download all the data in the data source;

Determining the target source table to be collected in all the data;

Constructing a mapping relationship between the data source table and the target source table according to the correlation between the fields in the data source table and the fields in the target source table;

Data collection is performed according to the mapping relationship.

In this application, the data source information includes a data source table, which lists the fields of the data to be collected, downloads all the data in the data source during collection, and determines the target of data collection from it The source table, and then construct a mapping relationship between the two tables according to the correlation of the fields between the two tables, and the mapping relationship is used for subsequent data collection.

Through the method provided by this application, the problem of inability to collect data due to inconsistency or incompleteness of the field names can be solved, and when collecting for multiple data sources, it has the flexibility of collection, such as the data source form ( The fields in A) are: name, age, address information, and the target source table (B) field information is: user_name, user_age, addr. Since the field names may be consistent or inconsistent, it is necessary to bind A.name through the mapping relationship Collect the data in B.user_name, complete the task collection, and finally output the data collection results. For another example, when many-to-many data collection is performed, the mapping relationship can be adjusted according to the correlation between the fields, and this correlation can be expressed as the order of the fields, or others.

The following is an example to illustrate:

Select the data source that has been built, select the library name -> table name and query conditions;

Select the target data source that has been built, select the library name -> table name -> collection mode (clear the table and then write/append write) -> write mode (insert, update);

Configure the field mapping method, such as using subscripts for field relationship mapping.

Configure task information, configure task scheduling, such as selecting a task scheduler, configuration preview, and task generation. By selecting rules, generate a configuration preview and display it, which is convenient for operators to view and confirm, and finally successfully create tasks.

The process of creating a data collection task is described below with two examples:

Example 1, collect relational data sources, from Mysql to Mysql:

a. Create data source - data source

Enter the data source name, Mysql address, port, user name, password, and database name information.

b. Create data source-data target source, the storage location of the data to be collected

Enter the data source name, Mysql address, port, user name, password, and database name information.

c. Create an execution strategy

Select the node where the task needs to be executed -> configure the scheduling cycle (Corn expression), -> task execution time (you can choose a fixed time, or you can choose to follow the actual time of the task) -> select the task execution strategy (fix the first , the last one, failover, busy transfer, etc.) -> Configure subtasks (tasks to be executed after the current task is executed, can be empty)

d. Create tasks

After creating the task, select the data source, the system will download all the data sources, select the data source in step a, the system will automatically list all the tables under the library, and then determine the data table to be collected, the system will list according to the table Select all fields under the selected table;

Select the data target, the system will download all the data target sources, select the data source in step b, the system will automatically list all the tables under the database, select the data table to be collected, the system will list the selected table according to the table all fields of the

After downloading the data, the system performs relational mapping on the fields in the source and target data tables, namely the data source table and the target source table. For example, the fields in the data source table (A) are: name, age, address information, and the field information in the target source table (B) is: user_name, user_age, addr. Since the field names may be consistent or inconsistent, they need to be bound through the mapping relationship , collect the data in A.name to B.user_name, complete the task collection, and finally output the data collection results.

Example 2, custom script collection sample, from webpage to Mysql:

A. Create data source - data source, input data source name, java project package name, class name, method name, upload script compiled executable file (*.jar).

B. Create data source-data target source, the storage location of the data to be collected

Enter the data source name, Mysql address, port, user name, password, and database name information.

C. Create an execution strategy,

Select the node where the task needs to be executed -> configure the scheduling cycle (Corn expression), -> task execution time (you can choose a fixed time, or you can choose to follow the actual time of the task) -> select the task execution strategy (fix the first , the last one, failover, busy transfer, etc.) -> Configure subtasks (tasks to be executed after the current task is executed, can be empty).

d. Create tasks

Select the data source, the system will list all the fields under the selected table according to the table (the script is written according to the rules); select the target data source, the system will download all the target data sources, select the data source in step B, the system will Automatically list all the tables under the library, select the data table to be collected, and the system will list all the fields under the selected table according to the table. Relational mapping between fields in the table and the target source table. For example, the fields in the data source table (A) are: name, age, address information, and the field information in the table (B) in the data target is: user_name, user_age, addr. Since the field names may be consistent or inconsistent, you need to pass the mapping relationship Binding, collect the data in A.name to B.user_name.

This application provides a multi-source heterogeneous data collection method. Before data collection, through the configuration of task scheduler and data source, the applicability and ease of use of the method can be improved, and the dependence on user expertise can be reduced. At the same time, it can be extended horizontally according to the needs to realize distributed data collection. After the configuration is completed, it can be applied to the automatic collection of various types of data through the task scheduler, which integrates various types of data such as web page data, offline data, and real-time data. The collection function makes it not affected by the heterogeneity of data sources and data storage systems when crawling data from massive multi-data sources, which greatly improves the efficiency of data collection.

The method provided by the foregoing embodiments has the following advantages:

Ease of use, through configurable guided task configuration, makes the method easier to use, reduces the threshold of operation, and improves the universality of customers.

Scalability, which can be extended horizontally according to needs, so that the method is suitable for various application scenarios and can realize distributed data collection;

Universality, this method integrates various types of data collection functions such as webpage data, offline data, and real-time data;

Data heterogeneity, when crawling official document data from a large number of multiple data sources, the heterogeneity of data sources and data storage systems is shielded, and the scheduler system is used to complete automatic periodic collection.

In another optional embodiment, in the process of data collection, task logs and statistical reports can also be generated. The task log records detailed error logs during the execution of the task. The entire acquisition system is running due to the need Performing list crawling and detail crawling may cause task failures due to network or other reasons. In this case, effective task result feedback is required to ensure that failures are found in a large number of tasks and modified in a timely manner. The task log module mainly records detailed error logs during task execution, which is helpful for operation and maintenance personnel to troubleshoot the cause. At the same time, task logs can be deleted within a specified date to reduce storage pressure. The statistical report can include: data collection information and node information, the data collection information includes: total amount, total capacity, new incremental information of the day, and 30-day trend statistics; the node information includes: CPU usage, memory usage, The total amount of memory, the remaining amount of memory; task information: number of successes, number of failures, number of execution suspensions, and a weekly statistical chart.

The above embodiments describe the method in this application, and the device and storage medium involved in this application are described below.

Referring to Figure 2, this application provides a multi-source heterogeneous data acquisition device, including:

A determining unit 201, configured to determine the type of the data source, and configure the data source information of the data source;

The configuration unit 202 is configured to configure a task scheduler, and the task scheduler is used for timing execution of tasks, periodic execution of tasks, determination of service nodes, and determination of execution strategies;

The creating unit 203 is configured to create a data collection task, where the data collection task includes a data source, a data target source, and a data collection strategy;

The collection unit 204 is configured to execute the data collection task according to the data collection strategy through the configured task scheduler;

An output unit 205, configured to output data collection results;

The acquisition unit 204 is specifically used for:

The data source information includes a data source table, and the data source table lists the fields of the data to be collected;

Select the data source and download all the data in the data source;

Determining the target source table to be collected in all the data;

Constructing a mapping relationship between the data source table and the target source table according to the correlation between the fields in the data source table and the fields in the target source table;

Data collection is performed according to the mapping relationship.

Optionally, when the data source is a website, the configuration unit 202 is specifically used for:

Obtain pre-configured acquisition script information;

If the script is a custom script, obtain the custom script file;

If the script is a java script, after obtaining the script file, configure the package name, class name, and method name of the script.

Optionally, when the data source is a relational database, the determining unit 201 is specifically configured to:

Configure the ip address, port, user name and password information of the data source;

Described relational database comprises: MySQL, Oracle, SQLServer, PostgreSQL, Hive, HDFS, MongoDB, Gbase, Kingbase.

Optionally, the configuration unit 202 is specifically used for:

Configure blocking processing strategies, configure subtasks, configure task retry strategies, configure task execution trigger strategies, and execute alarm strategies.

Optionally, the device further includes a generating unit 206, specifically configured to:

In the process of collecting data, generate task logs and statistical reports;

A detailed error log during the execution of the task is recorded in the task log;

The statistical report includes: data collection information and node information, the data collection information includes: total amount, total capacity, new incremental information of the day and 30-day trend statistics; the node information includes: CPU usage rate, memory usage rate , the total amount of memory, the remaining amount of memory; task information: number of successes, number of failures, number of execution suspensions, and a weekly statistical chart.

The device provided in this embodiment corresponds to the method provided in the foregoing embodiments, and has corresponding specific technical features, so it has the same technical effect as the method embodiment, and will not be repeated here.

Referring to Figure 3, the present application also provides a multi-source heterogeneous data acquisition device, including:

Processor 301, memory 302, input and output unit 303, bus 304;

The processor 301 is connected to the memory 302, the input and output unit 303 and the bus 304;

The memory 302 stores a program, and the processor 301 invokes the program to execute any one of the multi-source heterogeneous data collection methods above.

The present application also relates to a computer-readable storage medium, on which a program is stored, which is characterized in that, when the program is run on a computer, the computer is made to execute any one of the multi-source heterogeneous data acquisition methods above.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for enabling a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk, and other media that can store program codes.

Claims (8)

1. A multi-source heterogeneous data collection method, the method comprising:

determining the type of a data source and configuring data source information of the data source;

configuring a task scheduler, wherein the task scheduler is used for executing tasks at regular time, periodically executing the tasks, determining service nodes and determining an execution strategy;

creating a data acquisition task, wherein the data acquisition task comprises a data source, a data target source and a data acquisition strategy;

executing the data acquisition task according to the data acquisition strategy through the configured task scheduler;

outputting a data acquisition result;

executing the data acquisition task according to the data acquisition strategy through the configured task scheduler comprises the following steps:

the data source information comprises a data source table, wherein the data source table enumerates fields of data to be acquired;

selecting the data source and downloading all data in the data source;

determining a target source form to be acquired in all the data;

constructing a mapping relation between the data source table and the target source table according to the relevance between the fields in the data source table and the fields in the target source table;

and collecting data according to the mapping relation.

2. The multi-source heterogeneous data collection method according to claim 1, wherein when the data source is a website, the configuring the data source information of the data source includes:

acquiring preconfigured acquisition script information;

if the script is a custom script, acquiring a custom script file;

if the script is a java script, after the script file is acquired, the package name, class name and method name of the script are configured.

3. The method for multi-source heterogeneous data collection according to claim 1, wherein when the data source is a relational database, the configuring the data source information of the data source includes:

configuring the ip address, port, user name and password information of the data source;

the relational database comprises: mySQL, oracle, SQLServer, postgreSQL, hive, HDFS, mongoDB, gbase, kingbase.

4. The multi-source heterogeneous data collection method of claim 1, wherein the configuration task scheduler comprises:

configuring a blocking processing strategy, configuring a subtask, configuring a task retry strategy, configuring a task execution triggering strategy and executing an alarm strategy.

5. The multi-source heterogeneous data collection method of claim 1, further comprising: in the process of collecting data, generating a task log and a statistical report;

the task log records detailed error log of the task in the executing process;

the statistical report includes: data acquisition information and node information, the data acquisition information includes: total amount, total capacity, new increment information of the day and trend statistics of 30 days; the node information includes: CPU utilization rate, memory occupancy rate, total memory and residual memory; task information: success number, failure number, execution suspension number, and one week statistics.

6. A multi-source heterogeneous data acquisition device, comprising:

a determining unit, configured to determine a type of a data source and configure data source information of the data source;

the configuration unit is used for configuring a task scheduler, wherein the task scheduler is used for executing tasks at regular time, periodically executing the tasks, determining service nodes and determining an execution strategy;

the system comprises a creation unit, a data acquisition unit and a data processing unit, wherein the creation unit is used for creating a data acquisition task, and the data acquisition task comprises a data source, a data target source and a data acquisition strategy;

the acquisition unit is used for executing the data acquisition task according to the data acquisition strategy through the configured task scheduler;

the output unit is used for outputting a data acquisition result;

the acquisition unit is specifically used for:

the data source information comprises a data source table, wherein the data source table enumerates fields of data to be acquired;

selecting the data source and downloading all data in the data source;

determining a target source form to be acquired in all the data;

constructing a mapping relation between the data source table and the target source table according to the relevance between the fields in the data source table and the fields in the target source table;

and collecting data according to the mapping relation.

7. A multi-source heterogeneous data collection device, the device comprising:

a processor, a memory, an input-output unit, and a bus;

the processor is connected with the memory, the input/output unit and the bus;

the memory holds a program which the processor invokes to perform the method of any one of claims 1 to 5.

8. A computer readable storage medium having a program stored thereon, which when executed on a computer performs the method of any of claims 1 to 5.

Images