CN101216839A - Method and device for network data collection - Google Patents

Abstract

The present invention provides a device and method for concentrating network data. The device includes a scheduler, a work engine, and a data integrator. The scheduler is used to analyze the terminal's data source acquisition request, and generate data access-related work instructions and data integration-related The data instructions are issued to the work engine and the data integrator respectively, and the returned data result set is packaged as a data source and returned to the end user; the work engine is used to create and start the corresponding task to obtain data according to the work order; the data The integrator is used to integrate the obtained data into a data result set in a unified format according to the data instruction, and return it to the scheduler. The device and method of the invention can make users not concerned about how to integrate data, reduce the workload of developers, and at the same time, the whole device can be easily integrated into other systems, and has strong flexibility and scalability.

Description

Method and device for network data collection

technical field

The invention relates to data processing technology in computer software, more specifically, a method and equipment for realizing data collection in different formats on the network.

Background technique

With the development of computer software, there are a large number of different software systems, and the data output and storage of each software system are different, and are often stored in different formats. How to effectively integrate these data so that they can be accessed as a whole It is a problem often encountered in the application system to use it in the application system. Some commercial systems provide some data integration frameworks, but users are required to write software under this framework, which limits the flexibility of the system, has high performance overhead, and supports limited data formats. In practical applications, when the data is needed, it is often necessary to process the data in advance, such as fetching them into the database and then using them. This method leads to a heavy workload for developers, who need to read the data first, and then It is very inconvenient to insert into the database. In actual implementation, due to the variety of ways to operate data and the different storage formats, it is not an easy task to put these data into the database and read the data. How to use these data stored in different locations on the network as a data source, conveniently access to the application system, and reduce the workload of developers is a relatively difficult problem to solve.

Contents of the invention

The technical problem to be solved by the present invention is to provide a method and equipment for concentrating network data, so as to facilitate concentrating data in different formats on the network.

In order to solve the above-mentioned problems, the present invention provides a kind of network data centralization method, and this method comprises the following steps:

(a) Receive and analyze the data source acquisition request sent by the terminal, and generate work instructions related to data access and data instructions related to data integration;

(b) According to the work order, create and start the corresponding task to obtain the data;

(c) Integrate the acquired data into a unified data result set according to the data instruction;

(d) Encapsulate the data result set as a data source and return it to the end user.

Further, in step (a), the data source acquisition request or the configuration file corresponding to the data source acquisition request includes data acquisition methods, data connection and filtering rules at different locations on the network, and finally generated dataset formats.

Further, in step (b), the work orders are classified according to the instruction sequence network addresses and file formats in the work orders, and a task is created for a classification.

Further, in step (c), the acquired data is sequentially filtered, connected and formatted and integrated according to the data instruction, so as to obtain the same data result set.

In order to solve the above technical problems, the present invention also provides a network data concentration device, which includes a scheduler, a work engine and a data integrator, wherein,

The scheduler is used to analyze the data source acquisition request of the terminal, generate data access-related work instructions and data integration-related data instructions, and send them to the work engine and data integrator respectively, and encapsulate the returned data result set as a data source return to end users;

The work engine is used to create and start corresponding tasks to obtain data according to work instructions;

The data integrator is used to integrate the obtained data into a data result set in a unified format according to the data instruction, and return it to the scheduler.

Further, the device also includes a memory for storing several pre-set configuration files, and the content of the configuration files includes data acquisition methods, data connection and filtering rules at different locations on the network, and finally generated data set formats.

Further, the scheduler includes a configuration file acquisition module, a configuration file parsing module and a data source encapsulation module, wherein,

The configuration file acquisition module is configured to receive a terminal data source acquisition request, and find a corresponding configuration file according to the information contained in the request;

The configuration file parsing module analyzes the content of the obtained configuration file, obtains relevant information about data access and data integration, and then converts them into work instructions that can be recognized and processed by the work engine and data instructions that can be recognized and processed by the data integrator, and Send these two parts to the job engine and data integrator respectively;

The data set encapsulation module is used to encapsulate the obtained data result set, convert the data set into a data source, return it to the terminal in the form of a data source, and provide an interface method for accessing data records.

Further, the work engine includes an instruction receiving module, an instruction analysis module and a data acquisition module, wherein:

The instruction receiving module is used to receive work instructions, cache the instructions until all instructions are received, and pass these instructions to the instruction analysis module;

The instruction analysis module is used to classify and process the received instructions;

The data acquisition module searches for stored instructions according to the information after classification, obtains data access methods, and creates data acquisition tasks according to the obtained instructions. One classification corresponds to one task, and the corresponding resources and operation steps for obtaining data are set. Tasks Start the task when ready.

Further, the basis for classifying the instruction analysis module is the instruction sequence network address and file format.

Further, the data integrator includes an instruction receiving module, an instruction classification module, a data set receiving module and a data set processing module, wherein:

The instruction receiving module is used to receive the data instructions of the scheduler and store them;

The instruction classification module is used to classify the received instructions according to the functions of the data instructions, and convert them into corresponding processing rules, including data filtering rules, data connection rules and data set formats;

The data set receiving module is used to receive all data result sets after receiving the notification from the working engine;

The data set processing module is used to filter, connect and format the data sets. Specifically, take out the data sets in turn, find the corresponding data filtering rules, and filter the data. The filtered results form a temporary data set. When After all the data sets are filtered, according to the connection rules, the filtered temporary data sets are connected until a complete data set is formed. According to the defined data structure, the format of the data set is converted. After completion, the The dataset is passed to the scheduler.

Compared with the prior art, the device and method of the present invention can make users not care about how to integrate data, reduce the workload of developers, and at the same time, the whole device can be easily integrated into other systems, with strong flexibility and scalability sex.

Description of drawings

Fig. 1 is a schematic diagram of the overall framework structure of the network data concentration equipment of the present invention.

Fig. 2 is a schematic structural diagram of a scheduler of a network data concentration device according to the present invention.

Fig. 3 is a schematic diagram of the working engine structure of the network data concentration device of the present invention.

Fig. 4 is a flow chart of the data integrator of the network data concentration device of the present invention.

Fig. 5 is a schematic flow chart of the network data centralization method of the present invention.

Detailed ways

As shown in Figure 1, the network data concentration device of the present invention can centralize network data according to the end user's request. The network data concentration device includes a scheduler 1, a work engine 2, a data integrator 3 and a memory 4, wherein:

The scheduler 1 is used to analyze the data source acquisition request of the terminal, generate work instructions and data instructions and send them to the work engine and data integrator respectively, and encapsulate the returned data result set as a data source and return it to the end user;

Work engine 2 is used to create and start corresponding tasks to obtain data according to work instructions;

The data integrator 3 is used to integrate the obtained data into a data result set in a unified format according to the data instruction, and return it to the scheduler.

The memory 4 is used to store several pre-set configuration files, which are saved in the form of XML, and the content includes data acquisition methods, data connection and filtering rules at different locations on the network, and the format of the final generated data set.

When the network data concentration device of the present invention receives a data source acquisition request from a terminal, the scheduler can match the corresponding configuration file in the memory according to the request, thereby parsing out work instructions and data instructions.

Presetting several configuration files in the memory is the best implementation mode of the method and device of the present invention, which can make the operation of the terminal user simple and easy, and of course the device can also be simplified. The data source acquisition request carries corresponding information, and the scheduler parses out the work instructions and data instructions for sending to the work engine and data integrator according to the corresponding information. At this point, the information carried in the data source acquisition request is equivalent to the configuration file, including the way to read and access data, the way to connect to data, and the data format of the data result set.

Storer can be used as the maintenance tool of network data concentration equipment of the present invention, supports newly-increased data type, can be by expanding the access data of this type and data reading class in XML file when adding new data type (that is how to put in the file Guide to data reading and processing, because the data reading method of each file format is different), so that the scheduler converts the operation of obtaining data into corresponding work instructions to the work engine, and the work engine can obtain the data through the instruction the corresponding data set.

The network data concentration device will be described in detail below in conjunction with the accompanying drawings.

Scheduler 1 is mainly responsible for finding the corresponding configuration file according to the data source acquisition request of the terminal, parsing and processing the configuration file, and obtaining the work instruction of the work engine and the data instruction of the data integrator. The work instruction refers to a series of data obtained from different locations on the network sequence of instructions. Data instructions refer to a series of instruction sequences including data connection, filtering rules and the format of the final generated dataset. A data set is obtained through the data processing of the working engine and the operation of the data integrator, and the data set is packaged as a data source and passed to the terminal.

As shown in Figure 2, the scheduler 1 mainly includes a configuration file acquisition module, a configuration file parsing module and a data source encapsulation module, wherein:

The configuration file acquisition module is used to receive the data source acquisition request of the terminal, and find the corresponding configuration file according to the information contained in the request;

The configuration file parsing module analyzes the content of the obtained configuration file, obtains relevant information about data access and data integration, and then converts them into work instructions that can be recognized and processed by the work engine and data instructions that can be recognized and processed by the data integrator, and Send these two parts to the job engine and data integrator respectively;

The work instruction refers to the location of the data source on the network and related methods of accessing the data source. The location on the network can be represented by IP-related information, and the information of the data source corresponds to data sources in different formats in different locations on the network. Specifically, a work instruction is an instruction sequence that can be executed on the basis of the corresponding software. If it is FTP, then the work instruction refers to "open an IP address", "switch to the directory where the data file is located", "obtain the file", etc. sequence of instructions.

If the data to be obtained is an EXCEL file placed on FTP, the parsed work instruction includes the URL information of the file FTP, the user name and password required for access, and the access file data reading class, which is used to obtain The data in the file (the data in EXCEL can be obtained by calling the corresponding API, and the data in EXCEL can be obtained through the POI package in the JAVA environment). If it is a database, the parsed work order defines the database type, database name, Resources such as tables or views in the library, connected user names and passwords, etc.

Data instructions refer to the rules for integrating a series of different data sets obtained into one data set, which are divided into data filtering rules and connection rules, and data set formats, which are sent to the data integrator in the form of data instructions. Filtering rules are used to filter unqualified data, and data connection rules are used to assemble these data into a complete data set; the data set format is used to specify the data structure of the formed data set.

The data set encapsulation module is used to encapsulate the obtained data result set, convert the data set into a data source, return it to the terminal in the form of a data source, and provide an interface method for accessing data records.

Work engine 2, according to the work instructions sent by the scheduler, starts corresponding tasks to obtain relevant data from the network, and passes the obtained data to the data integrator for processing. As shown in Figure 3, the working engine mainly includes an instruction receiving module, an instruction analysis module and a data acquisition module, among which:

The instruction receiving module is used to receive work instructions, cache the instructions until all instructions are received, and then pass these instructions to the instruction analysis module;

The instruction analysis module is used to classify and process the received instructions;

Classification is based on the command sequence network address and file format, for example, two EXCEL files on the same FTP address will be classified into one class.

The data acquisition module searches for stored instructions according to the information after classification, obtains data access methods, and creates data acquisition tasks according to the obtained instructions. One classification corresponds to one task, and the corresponding resources and operation steps for obtaining data are set. Tasks When ready, modify the status information of the task to the execution status, and start the task.

When the task of collecting the obtained data result set is completed, an execution completion notification will be sent. After receiving this notification, it will judge whether all the data acquisition tasks have been completed. If not, continue to wait. When all All the data collection tasks have been completed, notify the data integrator to fetch the data, and wait for the data integrator to take away all the obtained data result sets.

The data integrator 3 integrates the acquired data sets according to the relevant rules passed by the scheduler, that is, gathers multiple data into one data set, and then returns the data set to the scheduler. As shown in Figure 4, the data integrator mainly includes an instruction receiving module, an instruction classification module, a data set receiving module and a data set processing module, wherein:

The instruction receiving module is used to receive the data instructions of the scheduler and store them;

The instruction classification module is used to classify the received instructions according to the functions of the data instructions, and convert them into corresponding processing rules, including data filtering rules, data connection rules and data set formats;

The data set receiving module is used to receive all data result sets after receiving the notification from the working engine;

The data set processing module is used to filter, connect and format the data sets. Specifically, take out the data sets in turn, find the corresponding data filtering rules, and filter the data. The filtered results form a temporary data set. When After all the data sets are filtered, according to the connection rules, the filtered temporary data sets are connected until a complete data set is formed. According to the defined data structure, the format of the data set is converted. After completion, the The dataset is passed to the scheduler.

The method for data collection of the present invention comprises the following steps:

Step A: Receive and analyze the data source acquisition request sent by the terminal, and generate work instructions related to data access and data instructions related to data integration;

Step B: According to the work order, create and start the corresponding task to obtain the data;

Step C: Integrate the acquired data into a unified data result set according to the data instruction;

Step D: Encapsulate the data result set as a data source and return it to the end user.

Specifically, the network data concentration method of the present invention can be implemented by using the data concentration device of the present invention. The data concentration method of the present invention will be described in detail below in conjunction with the accompanying drawings:

As shown in Figure 5, the overall flow of the data concentration method of the present invention is as follows:

Step 1: Provide an interface for obtaining an extended data source, and this interface monitors the data source acquisition request sent by the terminal.

Step 2: Find the corresponding configuration file through the requested information, and read the file into the memory.

Step 3: call the parsing module to parse the content of the file, and obtain relevant information about data access and data integration.

Step 4: According to the definition of the data file, convert the related information of data access and data integration into work instructions and data instructions, where the instructions can be recognized and processed by the delivery module;

Step 5: Send the work instruction to the work engine, and send the data instruction to the data integrator;

Step 6: The work engine receives the work order, caches the received order until all the orders are received, and the work engine classifies the order until the classification process is completed;

Step 7: Receive the notification that the classification processing is completed, create a task for each classification, set the information of the data obtained in the task, start the task after the task is created, modify the status information of the task to the execution status, and start the task;

Each task is responsible for acquiring data in one location and in the same format, returning the data set after the acquisition is completed, and notifying the working engine to complete the execution;

Step 8: Listen for the notification that the task has been processed, and when receiving a notification that the task has been processed, change the status of the task from execution to completion;

Step 9: Check whether all tasks have been executed, if executed, go to step 10, otherwise go to step 8.

Create and start the corresponding data acquisition tasks in sequence. When all the data acquisition tasks are started, the working engine enters the waiting state;

Step 10: Send the data acquisition completion message to the data integrator, and the data integrator receives the message and obtains the data set from the working engine;

Step 11: Filter, connect and format the data according to the received data instruction in step 4;

Specifically, the data set processing module takes out the data sets in turn, searches for the corresponding data filtering rules, and performs filtering processing on the data. The filtered results form a temporary data set. When all the data sets are filtered, then according to the connection rules , connect the filtered temporary data sets until a complete data set is formed, convert the data set according to the defined data structure, and pass the data set to the scheduler after completion.

Step 12: After data integration is complete, send a message to the scheduler and wait for the scheduler to take the final data set away;

Step 13: The scheduler monitors whether a data set arrives, and retrieves the data set from the data integrator after receiving a message that a data set has been formed;

Step 14: The scheduler encapsulates the retrieved data set into the form of a data source, and sends the handle to operate the data source to the terminal.

Step 15: The terminal obtains the data source operation handle and obtains data.

Compared with the prior art, the present invention introduces the modules of scheduler, work engine and data integrator, and these modules cooperate with each other to complete the function of data concentration on the network. The configuration file can access the data in the newly added format according to the extension method of the data file, and at the same time, the configuration file defines the format of the returned data, so that the result data format can be customized. The scheduler can obtain data sources in different formats, and it can be used directly without pre-processing the data. These data can exist in different locations in the network, and the file formats can also be different, directly through related protocols, such as FTP or Relevant resources accessed by the database can obtain these data. In the data integrator, the data collected by the work engine is synthesized into corresponding data sets through the rules set by the data instructions. During this process, the data can be connected and filtered. Data can also be processed, and the data can be mapped and converted, with certain data processing and processing functions.

In addition, when the system needs to process data types in newly added formats, it only needs to expand the definition of the data file and add the processing logic of the scheduler parsing class to access other types of data, thereby improving the flexibility of the system.

The device and method of the invention can make users not concerned about how to integrate data, reduce the workload of developers, and at the same time, the whole device can be easily integrated into other systems, and has strong flexibility and scalability.

Claims (10)

1. A method for concentrating network data, characterized in that the method comprises the following steps: (a) Receive and analyze the data source acquisition request sent by the terminal, and generate work instructions related to data access and data instructions related to data integration; (b) According to the work order, create and start the corresponding task to obtain the data; (c) Integrate the acquired data into a unified data result set according to the data instruction; (d) Encapsulate the data result set as a data source and return it to the end user. 2. The method according to claim 1, wherein in step (a), the data source acquisition request or the configuration file corresponding to the data source acquisition request includes data acquisition methods and data connections and data connections in different locations on the network Filtering rules and the format of the final generated dataset. 3. The method according to claim 2, characterized in that: in step (b), classify the work order according to the network address and file format of the order sequence in the work order, and create a task for a classification. 4. The method according to any one of claims 1 to 3, characterized in that: in step (c), the obtained data is sequentially filtered, connected and formatted and integrated according to the data instruction, so as to obtain the same data result set. 5. A network data concentration device, characterized in that: the device includes a scheduler, a work engine and a data integrator, wherein, The scheduler is used to analyze the data source acquisition request of the terminal, generate data access-related work instructions and data integration-related data instructions, and send them to the work engine and data integrator respectively, and encapsulate the returned data result set as a data source return to end users; The work engine is used to create and start corresponding tasks to obtain data according to work instructions; The data integrator is used to integrate the obtained data into a data result set in a unified format according to the data instruction, and return it to the scheduler. 6. The device according to claim 5, characterized in that: the device also includes a memory for storing a number of preset configuration files, and the content of the configuration files includes data acquisition methods and data connections in different locations on the network And filtering rules and the format of the final generated dataset. 7. The device according to claim 6, wherein the scheduler includes a configuration file acquisition module, a configuration file parsing module and a data source encapsulation module, wherein, The configuration file acquisition module is configured to receive a terminal data source acquisition request, and find a corresponding configuration file according to the information contained in the request; The configuration file parsing module analyzes the content of the obtained configuration file, obtains relevant information about data access and data integration, and then converts them into work instructions that can be recognized and processed by the work engine and data instructions that can be recognized and processed by the data integrator, and Send these two parts to the job engine and data integrator respectively; The data set encapsulation module is used to encapsulate the obtained data result set, convert the data set into a data source, return it to the terminal in the form of a data source, and provide an interface method for accessing data records. 8. The device according to any one of claims 5 to 7, wherein the working engine includes an instruction receiving module, an instruction analysis module and a data acquisition module, wherein: The instruction receiving module is used to receive work instructions, cache the instructions until all instructions are received, and pass these instructions to the instruction analysis module; The instruction analysis module is used to classify and process the received instructions; The data acquisition module searches for stored instructions according to the information after classification, obtains data access methods, and creates data acquisition tasks according to the obtained instructions. One classification corresponds to one task, and the corresponding resources and operation steps for obtaining data are set. Tasks Start the task when ready. 9. The device according to claim 8, characterized in that: the basis for classifying the instruction analysis module is the instruction sequence network address and file format. 10. The device according to any one of claims 5 to 7, wherein the data integrator includes an instruction receiving module, an instruction classification module, a data set receiving module and a data set processing module, wherein: The instruction receiving module is used to receive the data instructions of the scheduler and store them; The instruction classification module is used to classify the received instructions according to the functions of the data instructions, and convert them into corresponding processing rules, including data filtering rules, data connection rules and data set formats; The data set receiving module is used to receive all data result sets after receiving the notification from the working engine; The data set processing module is used to filter, connect and format the data sets. Specifically, take out the data sets in turn, find the corresponding data filtering rules, and filter the data. The filtered results form a temporary data set. When After all the data sets are filtered, according to the connection rules, the filtered temporary data sets are connected until a complete data set is formed. According to the defined data structure, the format of the data set is converted. After completion, the The dataset is passed to the scheduler.

Images