CN115525364A - Method, device and system for processing web data based on button and electronic equipment - Google Patents

Abstract

The application provides a method, a device and a system for processing web data based on a button and electronic equipment, wherein a middle layer system and a button application are constructed in a server, so that when a user terminal sends a processing request based on the operation of a web browser, the middle layer system is started, and a data processing task and a configuration scheduling rule are obtained. The communication protocol between the foreground and the background of the keylet application is simulated through the intermediate layer system, so that the interface of the application is called to access the function of the application, corresponding data processing is executed in the intermediate layer system based on the data processing task and the configuration scheduling rule, and a processing result is obtained. In the scheme, only the middle layer system and the button application are constructed in the server, the middle layer system is used for accessing the button application function, and the user terminal can use all functions of the button only by accessing the middle layer system based on the operation of the web browser, so that the performance resource of the terminal is saved.

Description

Kettle-based web data processing method, device, system and electronic equipment

technical field

The present application relates to the technical field of computer software, in particular, to a kettle-based web data processing method, device, system and electronic equipment.

Background technique

Kettle is a desktop software and an open source ETL tool. It can build data pipelines by dragging and dropping components, connecting lines, and configuring. It can be connected to traditional databases, files, big data platforms, interfaces, streaming data, etc. data source. As an end-to-end data integration platform, Kettle can extract, load, clean, transform, and blend data from various data sources.

However, the current application of kettle is to download, install and use it on the terminal device. The installation package of kettle itself is relatively large, and the configuration requirements for the terminal device are relatively high, and the software interface and interactive logic are not satisfactory. Learning to use The cost is high, and the window response speed is slow, which brings a lot of inconvenience to users.

Contents of the invention

The purpose of this application includes, for example, to provide a kettle-based web data processing method, device, system and electronic equipment, which can save terminal performance resources by building an intermediate system and a kettle application in a server.

The embodiment of the application can be realized like this:

In a first aspect, the present application provides a kettle-based web data processing method applied to a server, the server communicates with a plurality of user terminals, the server includes an intermediate system and a kettle application, and the method includes:

When obtaining a processing request for any user terminal to access the intermediate system, start the intermediate system, wherein the processing request is issued based on the operation of the web browser of the user terminal;

Obtaining data processing tasks and configuring scheduling rules based on the processing request;

Simulate the communication protocol between the foreground and the background of the kettle application by the middle layer system, to call the interface of the kettle application;

The function of the kettle application is accessed based on the interface of the kettle application called, so as to perform corresponding data processing in the middle layer system based on the data processing task and configuration scheduling rules, and obtain a processing result.

In an optional implementation manner, the step of starting the middle-level system when obtaining a processing request for any user terminal to access the middle-level system includes:

When obtaining a processing request for any user terminal to access the middle layer system, obtain a user account corresponding to the user terminal, and determine user rights according to the user account;

After starting the middle layer system, feed back corresponding data resources to the user terminal based on the user authority, so that the user terminal displays the data resources in the web browser.

In an optional implementation manner, the step of obtaining data processing tasks and configuring scheduling rules based on the processing request includes:

Obtain a job and an executable task in the data processing task based on the processing request, wherein the job includes at least one executable task, and the executable task is obtained by operating the component based on the web browser of the user terminal of;

Obtain trigger information of the job and trigger information of the at least one executable task in the configuration scheduling rule.

In an optional implementation manner, the step of obtaining jobs and executable tasks in data processing tasks based on the processing request includes:

Obtaining the job created in the data processing task and the executable task created under the job based on the processing request;

obtaining the components corresponding to the executable task, and obtaining the source database, source data table, target database and target data table of the executable task;

A query statement for the source data table and the target data table is obtained to determine matching fields of the source data table and the target data table.

In an optional embodiment, the query statement is a query statement for a single field of the source data table and the target data table, or the query statement is for multiple fields of the source data table and the target data table query statement.

In an optional implementation manner, the step of performing corresponding data processing in the middle layer system based on the data processing task and configuration scheduling rules includes:

In the middle layer system, if the configuration scheduling rule indicates that the trigger information of each of the executable tasks is consistent with the trigger information of the job, then based on the trigger information of the job, each task under the job is synchronously triggered data processing of said executable tasks;

If the configuration scheduling rule indicates that the trigger information of each of the executable tasks is inconsistent, the data processing of each of the executable tasks under the job is sequentially triggered according to the trigger information of each of the executable tasks.

In an optional embodiment, the method also includes:

Analyzing the processing result to obtain an analysis result, the analysis result includes the number of successfully processed items, the number of failed processed items, and the data processing speed;

Accessing the short message platform through the middle layer system, so as to send the analysis result in the form of short message through the short message platform.

In a second aspect, the present application provides a kettle-based web data processing device, which is applied to a server, the server communicates with a plurality of user terminals, the server includes an intermediate system and a kettle application, and the device includes:

A starting module, configured to start the middle-level system when obtaining a processing request from any user terminal to access the middle-level system, wherein the processing request is issued based on the operation of the web browser of the user terminal of;

An obtaining module, configured to obtain data processing tasks and configure scheduling rules based on the processing request;

Invoking a module, for simulating the communication protocol between the foreground and the background of the kettle application by the middle layer system, so as to call the interface of the kettle application;

A processing module, configured to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and configuration scheduling rules, and obtain a processing result .

In a third aspect, the present application provides an electronic device, including one or more storage media and one or more processors in communication with the storage media, where one or more storage media store machine-executable instructions executable by the processors, When the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the foregoing embodiments.

In a fourth aspect, the present application provides a kettle-based web data processing system, the system includes a server and a plurality of user terminals communicating with the server, and the server includes an intermediate system and a kettle application;

The user terminal is configured to obtain a processing request for accessing the middle layer system based on the operation of the web browser, and send the processing request to the server;

The server is configured to start the middle-level system when obtaining a processing request from any user terminal to access the middle-level system, obtain data processing tasks and configure scheduling rules based on the processing request;

The server is also used to simulate the communication protocol between the foreground and the background of the kettle application through the middle layer system, so as to call the interface of the kettle application;

The server is further configured to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing based on the data processing task and configuration scheduling rules in the middle layer system, and obtain process result.

The beneficial effects of the embodiments of the present application include, for example:

The present application provides a kettle-based web data processing method, device, system and electronic equipment. By constructing an intermediate layer system and a kettle application in the server, when the user terminal sends a processing request based on the operation of the web browser, the intermediate layer is started system, and obtain data processing tasks and configure scheduling rules based on processing requests. The server simulates the communication protocol between the foreground and background of the kettle application through the middle-tier system, thereby calling the interface of the kettle application to access the functions of the kettle application, so as to execute corresponding data based on data processing tasks and configuration scheduling rules in the middle-tier system processing to get the processing result. In this solution, it is only necessary to build the middle-tier system and kettle application in the server, and use the middle-tier system to access the kettle application function. The user terminal only needs to access the middle-tier system based on the web browser operation, and can use all the functions of Kettle. The performance resource of the terminal is saved.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, so It should be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings based on these drawings without creative work.

Fig. 1 is the application scenario schematic diagram of the kettel-based web data processing method that the embodiment of the present application provides;

Fig. 2 is the flowchart of the web data processing method based on kettel provided by the embodiment of the present application;

Fig. 3 is the flowchart of the substep that step S101 comprises in Fig. 2;

Fig. 4 is the flowchart of the substep that step S102 comprises in Fig. 2;

Fig. 5 is the flowchart of the substep that step S1021 comprises in Fig. 4;

Fig. 6 is a flow chart of the analysis method in the kettel-based web data processing method provided by the embodiment of the present application;

FIG. 7 is a structural block diagram of an electronic device provided by an embodiment of the present application;

FIG. 8 is a block diagram of functional modules of a kettel-based web data processing device provided by an embodiment of the present application.

Icons: 110-storage medium; 120-processor; 130-kettel-based web data processing device; 131-starting module; 132-obtaining module; 133-calling module; 134-processing module; 140-communication interface.

detailed description

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. The components of the embodiments of the application generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations.

Accordingly, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

It should be noted that, in the case of no conflict, the features in the embodiments of the present application may be combined with each other.

Please refer to FIG. 1 , which is a schematic diagram of an application scenario of a kettel-based web data processing method provided by an embodiment of the present application. The application scenario includes a kettel-based web data processing system, and the processing system includes a server and multiple users communicating with the server. terminal.

In this embodiment, the server may be a single server, or a server cluster composed of multiple servers. The server includes middle-tier system and kettel application. The user terminal may be a terminal device used on the user side, such as a personal computer, a tablet computer and other devices. The middle layer system in the server can be accessed through the web browser on the user terminal, and then the kettel application can be accessed to utilize various functions of the kettel application.

Please refer to FIG. 2 , which is a flow chart of the kettel-based web data processing method provided by the embodiment of the present application. The method steps defined by the process related to the kettel-based web data processing method can be implemented by the above-mentioned server. The specific process described in FIG. 2 will be described in detail below.

S101. Start the middle-tier system when obtaining a processing request from any user terminal to access the middle-tier system, where the processing request is sent based on an operation of a web browser of the user terminal.

S102. Obtain a data processing task and configure a scheduling rule based on the processing request.

S103, simulating the communication protocol between the foreground and the background of the kettle application through the middle layer system, so as to call the interface of the kettle application.

S104. Access the function of the kettle application based on the called interface of the kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and configuration scheduling rules, and obtain a processing result.

With the development of the times, the importance of data has gradually become prominent. Data usually comes from various business systems, which are very scattered, and there is no unified data standard, so it is difficult to mine the value. To give full play to the value of data, it is necessary to unify data standards and aggregate data scattered in various business systems.

Kettle is a commonly used desktop application for data processing. In the prior art, when using related functions of the kettel application, the user needs to download and install the kettel application on the user terminal. The large installation package of the Kettel application and the high configuration requirements for the terminal have high requirements for the terminal performance, and the processing efficiency is not good.

In this embodiment, the middle-tier system and the kettel application are configured in the server. When the user needs to use the relevant functions of the kettel application, he can log in through the web browser on the user terminal, thereby sending a processing request to the server to access the middle-tier system .

When the server receives the processing request sent by the user terminal, it will start the middle layer system, so that the user terminal can access the middle layer system. After the user terminal accesses the middle-tier system, it can synchronously implement the relevant processing procedures of the middle-tier system on the web page of the web browser.

In this embodiment, on the server side, after the middle layer system is started, it can simulate the communication protocol between the foreground and the background of the kettel application. Among them, the foreground of the kettel application can be understood as the front-end page that presents the data processing process, and the background of the kettel application can be understood as the back-end for actually executing instructions and processing data.

The middle layer system can call the interface of the kettel application by simulating the communication protocol between the foreground and the background of the kettel application. The functions of the kettel application can be accessed through the interface of the kettel application. That is, the middle-tier system can have related functions of the kettel application.

In addition, the processing request initiated by the user terminal includes a data processing task and a configuration scheduling rule, wherein the data processing task may be used to indicate what kind of data processing is to be performed, and the configuration scheduling rule may be used to indicate how to specifically execute the data processing process.

Based on this, on the server side, corresponding data processing can be performed in the middle layer system according to the data processing tasks and configuration scheduling rules, and the processing results can be obtained.

In the kettel-based web data processing method provided in this embodiment, by constructing the middle-tier system and the kettel application in the server, the middle-tier system is used to access the kettel application function, and the user terminal only needs to access the middle-tier system based on a web browser operation , you can use all the functions of the kettel application, which saves the performance resources of the terminal.

In existing kettel applications, there is basically no concept of roles and permissions, which leads to the problem of information insecurity in some important data resources, and is not conducive to the isolation of functions and data.

Based on this, please refer to FIG. 3. In this embodiment, in the step of obtaining the processing request of the user terminal in the above step S101 and starting the middle layer system, the following sub-steps may be included:

S1011. When obtaining a processing request of any user terminal to access the middle layer system, obtain a user account corresponding to the user terminal, and determine user rights according to the user account.

S1012. After starting the middle layer system, feed back corresponding data resources to the user terminal based on the user authority, so that the user terminal displays the data resources in the web browser.

In this embodiment, on the side of the user terminal, the user can use the user account to log in on the web browser, and then access the middle layer system. After connecting to the middle layer system, the relevant interfaces such as data information and data configuration information can be displayed on the web browser of the user terminal, and the relevant interface can also be used by the user for data selection and configuration, and then the relevant configuration results will be sent to the server for processing.

Different user accounts may correspond to different user roles, and different user roles may have different user permissions. Different user permissions have processing permissions for different data resources, including but not limited to viewing, modifying, configuring, and deleting.

Therefore, after obtaining the processing request initiated by the user terminal, it may first be verified whether the user account is legal, for example, whether it is a legally registered user or not. If the user account is a legitimate account, the user authority corresponding to the user account can be determined, and then it is known which data resources the user account has processing authority to.

After starting the middle layer system, the server can feed back to the user terminal the data resource with processing authority based on the corresponding user authority of the user account, so that the data resource can be displayed in the web browser of the user terminal. So that the user can perform processing such as configuration on the data resource on the interface related to the web browser of the user terminal.

In this embodiment, by assigning different roles to different user accounts, different user permissions are set up in combination with business requirements, so as to isolate functions and data permissions. Through permission classification, each role can work together to improve work efficiency and protect data access rights.

On the side of the user terminal, the user can generate the above-mentioned processing request based on the selection and configuration of the authorized data resource, and then send it to the server. After receiving the processing request, the server can obtain data processing tasks and configure scheduling rules based on the processing request. Referring to Figure 4, this step specifically may include the following sub-steps:

S1021. Obtain a job and an executable task in the data processing task based on the processing request, where the job includes at least one executable task, and the executable task is to operate a component based on the web browser of the user terminal what you get.

S1022. Obtain trigger information of the job and trigger information of the at least one executable task in the configuration scheduling rule.

In this embodiment, data processing tasks are divided into jobs (jobs) and executable tasks (tasks). Among them, a job is a collection of executable tasks, and the job can be divided into different business fields or business modules, which has business significance. The executable task is the smallest execution unit.

Executable tasks can be created under the job, and the executable tasks are obtained based on the component configuration in the web browser, which has the same functions as the components provided by the kettel application. An executable task can be composed by, for example, dragging and dropping.

When jobs and executable tasks are configured, scheduling rules can be configured for jobs and executable tasks, for example, configurations including job trigger information, trigger information of each executable task, and the like. It is used for subsequent timing triggering of jobs and executable tasks, where the priority of the configuration scheduling rule of the executable task is higher than that of the job, that is, during execution, the configuration scheduling rule of the executable task is allow.

In this embodiment, jobs are divided and configured by business domain, and one or more executable tasks are constructed under each job, and then scheduling rules are configured for jobs and executable tasks. It can support job execution in different business fields, and refine the smallest execution unit under the job.

In this embodiment, in the process of unified data management, it usually involves migration to a specified database or data table. Therefore, the constructed jobs and executable tasks often involve the processing of data in databases and data tables. Specifically, please refer to FIG. 5. In this embodiment, in the above step of obtaining the jobs and executable tasks in the data processing task, the following sub-steps may be included:

S10211. Obtain the job created in the data processing task and the executable task created under the job based on the processing request.

S10212. Obtain the component corresponding to the executable task, and obtain the source database, source data table, target database, and target data table of the executable task.

S10213. Obtain query statements for the source data table and the target data table, so as to determine matching fields of the source data table and the target data table.

In this embodiment, the data processing request initiated by the user terminal side includes a newly created job and an executable task constructed under the newly created job. Executable tasks.

When constructing an executable task, for example, by dragging a table input component, selecting a source database, selecting a source data table, and generating a query statement, such as an SQL statement. Then drag the table output component, and select the target database and target data table. Based on the generated query statement, match the fields of the source data table and the target data table, and then generate and save the executable task.

When the processing request is received on the server side, the job in the processing request and the specific information under the executable tasks can be restored in the above manner.

In this embodiment, through the above-mentioned management of databases and data tables, and the construction of corresponding jobs and executable tasks, the data migration of corresponding fields can be successfully implemented in the subsequent implementation of data migration by executing tasks.

In this embodiment, the query statement generated above may be a query statement for a single field of the source data table and the target data table, or may be a joint query statement for multiple fields of the source data table and the target data table.

In this way, the migration of single-field data between data tables, or the data migration of multi-field linkage between data tables can be realized.

In this embodiment, if the server needs to perform corresponding data processing in the middle layer system, in addition to the above data processing tasks, it also needs to be based on related configuration scheduling rules. It can be known from the above that the configuration scheduling rules include job scheduling rules and scheduling rules for each executable task. The scheduling rules of the job and the scheduling rules of the executable tasks may or may not be consistent. Specifically, data processing may be performed based on configuration scheduling rules in the following manner.

In the middle layer system, if the configuration scheduling rules indicate that the trigger information of each executable task is consistent with the trigger information of the job, then the data processing of each executable task under the job is triggered synchronously based on the trigger information of the job.

In this embodiment, when multiple executable tasks are constructed under a single job, the trigger information of each executable task is consistent with the trigger information of the job, in order to avoid performing tasks based on the trigger information of each executable task Trigger, the concurrent execution of executable tasks can be performed directly based on the trigger information of the job. In this way, processing complexity can be reduced on the basis of ensuring execution accuracy.

In addition, in the middle layer system, if the configuration scheduling rules indicate that the trigger information of each executable task is inconsistent, the data processing of each executable task under the job is triggered sequentially according to the trigger information of each executable task.

It can be known from the above that the priority of executable tasks is higher than that of jobs, and if the trigger information of each executable task is inconsistent, it must not all be consistent with the trigger information of jobs. Therefore, in this case, it is necessary to trigger the execution data processing based on the trigger information of each executable task, so as to ensure the trigger accuracy of task execution.

In this embodiment, when the data processing process is executed based on the above information in the middle layer system, if an error occurs during data processing, the scheduling status of jobs and executable tasks will be automatically set to disabled. At this point, you can view the task execution log, and choose to continue the execution after the error problem is resolved.

In this embodiment, after the data processing task is performed, the processing result can also be analyzed, please refer to FIG. 6, specifically, the following steps can also be included:

S105, analyzing the processing result to obtain an analysis result, the analysis result including the number of successfully processed items, the number of failed processed items, and the data processing speed.

S106, accessing a short message platform through the middle layer system, so as to send the analysis result in a short message through the short message platform.

In this embodiment, during and after data processing execution, an execution log can be generated, and based on the execution log, indicators such as how many pieces of data are successfully acquired, how many pieces of data fail to be acquired, and execution speed during data processing can be obtained.

In addition, in this embodiment, the summarization module can also be used to count the executed data from multiple dimensions, such as the number of data sources, the number of successfully executed data pieces, the number of failed executed data pieces, and the like.

In addition to viewing the execution log, it is also possible to download the execution log, access the SMS platform through the middle layer system, and send the above analysis results in the downloaded execution log to the user's terminal device by SMS.

In the kettel-based web data processing method provided in this embodiment, by configuring the middle-tier system and the kettel application in the server, the user terminal can access the middle-tier system, and the middle-tier system simulates the communication protocol between the foreground and the background of the kettel application Call the interface of the kettel application, and then access the functions of the kettel application. The user only needs to log in to the middle layer system through a web browser based on the user terminal, and can configure data sources, jobs, and executable tasks in the kettel application, and can also view the execution status of executable tasks and view execution error messages Etc., can control the execution of the process anytime, anywhere.

Furthermore, in this embodiment, the concept of rights management is entered, and different roles can be assigned to different users to correspond to different user rights, so as to realize function isolation and execution data isolation.

In addition, it has perfect scheduling rules, and can configure scheduling rules for jobs and executable tasks to realize the division of business domains and the division of fine-grained executable units under business domains.

In addition, the execution log can be analyzed from multiple dimensions, and the execution status can be counted from multiple dimensions.

The processing method provided in this embodiment has a better experience in terms of interface usability, task scheduling, and statistical analysis, and can control various links of data processing more freely and conveniently.

As can be seen from the above, the embodiment of the present application also provides a kettel-based web data processing system, the processing system includes a server and a plurality of user terminals, specifically, the user terminals are used to obtain an access based on the operation of the web browser. The processing request entered into the middle layer system, and the processing request is sent to the server.

The server is configured to start the middle-tier system when obtaining a processing request from any user terminal to access the middle-tier system, obtain data processing tasks and configure scheduling rules based on the processing request.

The server is further configured to simulate the communication protocol between the foreground and the background of the kettle application through the middle layer system, so as to call the interface of the kettle application.

The server is further configured to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing based on the data processing task and configuration scheduling rules in the middle layer system, and obtain process result.

The kettel-based web data processing system provided in this embodiment, by constructing the middle layer system and the kettle application in the server, utilizes the middle layer system to access the kettle application function, the user terminal only needs to access the middle layer system based on the web browser operation, You can use all the functions of kettel, saving the performance resources of the terminal.

For the description of the processing flow of each device in the system and the interaction flow between each device, reference may be made to the relevant descriptions in the foregoing method embodiments, which will not be described in detail here.

Please refer to FIG. 7 , which is a schematic diagram of exemplary components of an electronic device provided in an embodiment of the present application. The electronic device may be the above-mentioned server. The electronic device may include a storage medium 110 , a processor 120 , a kettel-based web data processing device 130 and a communication interface 140 . In this embodiment, both the storage medium 110 and the processor 120 are located in the electronic device and are set separately. However, it should be understood that the storage medium 110 may also be independent from the electronic device, and may be accessed by the processor 120 through the bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, may be a cache and/or a general-purpose register.

The kettel-based web data processing device 130 can be understood as the above-mentioned electronic device, or the processor 120 of the electronic device, and can also be understood as realizing the above-mentioned kettel-based web data under the control of the electronic device independently of the above-mentioned electronic device or the processor 120 A software function module of the data processing method.

As shown in FIG. 8 , the aforementioned kettel-based web data processing apparatus 130 may include a starting module 131 , an obtaining module 132 , a calling module 133 and a processing module 134 . The functions of each functional module of the kettel-based web data processing device 130 will be described in detail below.

The starting module 131 is configured to start the middle-level system when obtaining a processing request from any user terminal to access the middle-level system, wherein the processing request is based on the operation of the web browser of the user terminal issued;

It can be understood that the startup module 131 can be used to execute the above step S101, and for the detailed implementation of the startup module 131, please refer to the content related to the above step S101.

Obtaining module 132, configured to obtain data processing tasks and configure scheduling rules based on the processing request;

It can be understood that the obtaining module 132 can be used to execute the above step S102, and for the detailed implementation of the obtaining module 132, reference can be made to the content related to the above step S102.

Calling module 133, for simulating the communication protocol between the foreground and the background of the kettle application by the middle layer system, to call the interface of the kettle application;

It can be understood that the calling module 133 can be used to execute the above step S103, and for the detailed implementation of the calling module 133, please refer to the content related to the above step S103.

The processing module 134 is used to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and configuration scheduling rules, and obtain the processed result.

It can be understood that the processing module 134 can be used to execute the above step S104, and for the detailed implementation of the processing module 134, reference can be made to the content related to the above step S104.

In a possible implementation manner, the above startup module 131 may be used to:

When obtaining a processing request for any user terminal to access the middle layer system, obtain a user account corresponding to the user terminal, and determine user rights according to the user account;

After starting the middle layer system, feed back corresponding data resources to the user terminal based on the user authority, so that the user terminal displays the data resources in the web browser.

In a possible implementation manner, the above obtaining module 132 may be used to:

Obtain a job and an executable task in the data processing task based on the processing request, wherein the job includes at least one executable task, and the executable task is obtained by operating the component based on the web browser of the user terminal of;

Obtain trigger information of the job and trigger information of the at least one executable task in the configuration scheduling rule.

In a possible implementation manner, the above obtaining module 132 may be used to:

Obtaining the job created in the data processing task and the executable task created under the job based on the processing request;

obtaining the components corresponding to the executable task, and obtaining the source database, source data table, target database and target data table of the executable task;

A query statement for the source data table and the target data table is obtained to determine matching fields of the source data table and the target data table.

In a possible implementation manner, the query statement is a query statement for a single field of the source data table and the target data table, or the query statement is a query statement for multiple fields of the source data table and the target data table Combined query statement for fields.

In a possible implementation manner, the above-mentioned processing module 134 may be used to:

In the middle layer system, if the configuration scheduling rule indicates that the trigger information of each of the executable tasks is consistent with the trigger information of the job, then based on the trigger information of the job, each task under the job is synchronously triggered data processing of said executable tasks;

If the configuration scheduling rule indicates that the trigger information of each of the executable tasks is inconsistent, the data processing of each of the executable tasks under the job is sequentially triggered according to the trigger information of each of the executable tasks.

In a possible implementation manner, the kettel-based web data processing device 130 also includes an analysis module, which can be used for:

Analyzing the processing result to obtain an analysis result, the analysis result includes the number of successfully processed items, the number of failed processed items, and the data processing speed;

Accessing the short message platform through the middle layer system, so as to send the analysis result in the form of short message through the short message platform.

For the description of the processing flow of each module in the device and the interaction flow between the modules, reference may be made to the relevant description in the above method embodiment, and details will not be described here.

Further, the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the kettel-based web data processing method provided in the above-mentioned embodiments is implemented.

Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk, a hard disk, etc., and when the computer program on the computer-readable storage medium is run, the above kettel-based web data processing method can be executed. As for the processes involved when the executable instructions in the computer-readable storage medium are executed, reference may be made to relevant descriptions in the foregoing method embodiments, and no further details are given here.

To sum up, the kettle-based web data processing method, device, system, and electronic equipment provided by the embodiments of the present application construct an intermediate system and a kettle application in the server, so that the user terminal can send out processing based on the operation of the web browser. When requested, start the middle layer system, and obtain data processing tasks and configure scheduling rules based on processing requests. The server simulates the communication protocol between the foreground and background of the kettle application through the middle-tier system, thereby calling the interface of the kettle application to access the functions of the kettle application, so as to execute corresponding data based on data processing tasks and configuration scheduling rules in the middle-tier system processing to get the processing result. In this solution, it is only necessary to build the middle-tier system and kettle application in the server, and use the middle-tier system to access the kettle application function. The user terminal only needs to access the middle-tier system based on the web browser operation, and can use all the functions of Kettle. The performance resource of the terminal is saved.

The above is only a specific embodiment of the application, but the scope of protection of the application is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. All should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (10)

1. A web data processing method based on a button is characterized in that the method is applied to a server, the server is communicated with a plurality of user terminals, the server comprises a middle layer system and a button application, and the method comprises the following steps:

when a processing request of any user terminal for accessing the middle layer system is obtained, starting the middle layer system, wherein the processing request is sent out based on the operation of a web browser of the user terminal;

obtaining a data processing task and a configuration scheduling rule based on the processing request;

simulating a communication protocol between a foreground and a background of the button application through the middle layer system to call an interface of the button application;

and accessing the function of the key application based on the called interface of the key application so as to execute corresponding data processing based on the data processing task and the configuration scheduling rule in the middle layer system to obtain a processing result.

2. The method of claim 1, wherein the step of starting the middle layer system when obtaining a processing request for any user terminal to access the middle layer system comprises:

when a processing request of any user terminal accessing the middle layer system is obtained, a user account corresponding to the user terminal is obtained, and user permission is determined according to the user account;

and after the middle layer system is started, feeding back corresponding data resources to the user terminal based on the user authority so that the user terminal displays the data resources in the web browser.

3. The method according to claim 1, wherein the step of obtaining the data processing task and configuring the scheduling rule based on the processing request comprises:

obtaining a job and an executable task in a data processing task based on the processing request, wherein the job comprises at least one executable task, and the executable task is obtained by operating a component based on a web browser of the user terminal;

and acquiring the triggering information of the job and the triggering information of the at least one executable task in the configuration scheduling rule.

4. The method according to claim 3, wherein the step of obtaining the job and the executable task in the data processing task based on the processing request comprises:

obtaining a job created in a data processing task and an executable task created under the job based on the processing request;

obtaining a component corresponding to the executable task, and obtaining a source database, a source data table, a target database and a target data table of the executable task;

obtaining a query statement for the source data table and the target data table to determine matching fields of the source data table and the target data table.

5. A method for Kettle-based web data processing according to claim 4, wherein the query statement is a query statement for a single field of the source data table and the target data table, or the query statement is a query statement for a plurality of fields of the source data table and the target data table.

6. The method according to claim 3, wherein the step of executing corresponding data processing based on the data processing task and the configuration scheduling rule in the middle layer system comprises:

in the intermediate layer system, if the configuration scheduling rule indicates that the triggering information of each executable task is consistent and is consistent with the triggering information of the job, synchronously triggering the data processing of each executable task under the job based on the triggering information of the job;

and if the configuration scheduling rule indicates that the trigger information of each executable task is inconsistent, sequentially triggering the data processing of each executable task under the operation according to the trigger information of each executable task.

7. A method for keytle-based web data processing according to claim 1, further comprising:

analyzing and processing the processing result to obtain an analysis result, wherein the analysis result comprises the number of successful processing pieces, the number of failed processing pieces and the data processing speed;

and accessing a short message platform through the intermediate layer system so as to send the analysis result in a short message mode through the short message platform.

8. A keyword-based web data processing apparatus, which is applied to a server, the server communicating with a plurality of user terminals, the server including a middle layer system and a keyword application, the apparatus comprising:

the system comprises a starting module, a processing module and a control module, wherein the starting module is used for starting the middle layer system when a processing request of any user terminal for accessing the middle layer system is obtained, and the processing request is sent out based on the operation of a web browser of the user terminal;

an obtaining module, configured to obtain a data processing task and a configuration scheduling rule based on the processing request;

the calling module is used for simulating a communication protocol between a foreground and a background of the keytte application through the middle layer system so as to call an interface of the keytte application;

and the processing module is used for accessing the function of the button application based on the called interface of the button application so as to execute corresponding data processing based on the data processing task and the configuration scheduling rule in the middle layer system and obtain a processing result.

9. An electronic device comprising one or more storage media and one or more processors in communication with the storage media, the one or more storage media storing processor-executable machine-executable instructions that, when executed by the electronic device, are executed by the processor to perform the method steps of any of claims 1-7.

10. The system is characterized by comprising a server and a plurality of user terminals which are communicated with the server, wherein the server comprises a middle layer system and a keyword application;

the user terminal is used for obtaining a processing request for accessing the middle layer system based on the operation of a web browser and sending the processing request to the server;

the server is used for starting the middle layer system when a processing request of any user terminal accessing the middle layer system is obtained, and obtaining a data processing task and configuring a scheduling rule based on the processing request;

the server is also used for simulating a communication protocol between a foreground and a background of the keytte application through the middle layer system so as to call an interface of the keytte application;

the server is further configured to access a function of the keylet application based on the called keylet application interface, so as to execute corresponding data processing based on the data processing task and the configuration scheduling rule in the middle-layer system, and obtain a processing result.

Images