CN108985367A - Computing engines selection method and more computing engines platforms based on this method - Google Patents

Abstract

The invention provides a calculation engine selection method and a multi-calculation engine platform based on the method. The method includes: inputting task feature data corresponding to the task to be calculated into a task execution time prediction model of each of the multiple calculation engines, and obtaining a task execution time prediction result of the task to be calculated on each calculation engine, wherein , the task execution time prediction model is obtained through training based on a training sample set, the training sample set includes a plurality of task feature data and corresponding task execution time; according to the task execution time prediction results from the plurality of computing engines Select the computing engine that executes the task to be calculated. The method of the invention can automatically select a computing engine with high efficiency, and reduces the task execution time.

Description

Computing engine selection method and multi-computing engine platform based on the method

technical field

The invention relates to the field of information technology, in particular to a calculation engine selection method and a multi-calculation engine platform based on the method.

Background technique

With the development of a large number of new equipment in the sea, air, sky, deep sea and other directions of the country, equipment testing has become more and more important. For example, during the development of the J-10 fighter jet, tens of thousands of wind tunnel tests were carried out, and millions of pieces of aerodynamic data were obtained. The processing and analysis of these data became an important basis for the successful development of the J-10. Equipment test includes two processes of "test" and "evaluation", which is a way to obtain data, and then analyze, process and compare various data to help make decisions. At present, the test data processing method that still mainly relies on expert experience and computer-aided processing can no longer meet the needs of current test data processing. Moreover, due to the need to process different data volumes in test data processing, structured and unstructured processing Mixed, real-time and offline processing, etc., using a single engine can no longer meet the needs of various test processing. To solve this problem, there are currently three solutions: first, manually manage multiple engines, deploy the computing engines separately, and manually manage the computing engines and perform computing tasks. This method requires a lot of manpower and is inefficient. At the same time, if the system does not maintain Full load will cause a huge waste of resources; the second way is to use a "super" engine that supports various computing needs, specifically to deploy an engine that supports all processing methods, and use this engine to meet all experimental data processing demand, but the current maturity of this method is not high, and it will take time before large-scale use; the third method is a compromise between the first two, using a computing platform that supports multiple computing engines, this method can be applied on the one hand At present, various mature computing engine technologies, on the other hand, use automated methods to manage computing engines and computing tasks, which can improve resource utilization and task execution efficiency. In short, for the above three methods, manual management of multiple engines is inefficient, and "super" engines are difficult to meet urgent needs for a while. A computing platform with multiple computing engines is currently the solution to balance efficiency and feasibility.

However, to apply a multi-computing engine platform, it is necessary to solve the problems of multi-computing engine compatibility, unified management of computing tasks, and future engine expansion. Therefore, it is necessary to be able to automatically select a task execution engine to improve platform efficiency. None of the existing platforms that support multiple computing engines can solve the above problems. For example, Twitter SummingBird uses the Lambda architecture to integrate the distributed batch processing engine (Hadoop) and the distributed stream computing engine (Storm). It can integrate the results of batch processing and stream computing when executing requests, but it does not have a convenient engine management mechanism. At the same time, it does not provide engine operating environment isolation; Apache Ambari is based on Web implementation, supports the supply, management and monitoring of the Apache Hadoop ecosystem, and provides custom interfaces to support adding various stand-alone or distributed engines, but it does not provide unified computing task management , can only guarantee the compatibility of a specific engine, and at the same time need to manually select a computing engine to perform computing tasks; Google Kubernetes is implemented based on Docker, which can run computing engines in the form of containers, and can run stand-alone engines and distributed engines according to requirements, providing container deployment. Functions such as scheduling and node cluster expansion, but it does not have a task management mechanism, and it also requires manual selection of computing engines.

Therefore, it is necessary to improve the prior art to provide a multi-computing engine platform and a method for automatically selecting a computing engine for the multi-computing engine platform.

Contents of the invention

The purpose of the present invention is to overcome the above-mentioned defects in the prior art, and provide a computing engine selection method and a multi-computing engine platform based on the method.

According to a first aspect of the present invention, a calculation engine selection method is provided. The method includes the following steps:

Step 1: Input the task characteristic data corresponding to the task to be calculated into the task execution time prediction model of each of the multiple calculation engines, and obtain the task execution time prediction result of the task to be calculated on each calculation engine, wherein, The task execution time prediction model is obtained through training based on a training sample set, and the training sample set includes a plurality of task feature data and corresponding task execution time;

Step 2: Select a computing engine that executes the task to be calculated from the plurality of computing engines according to the task execution time prediction result.

In one embodiment, the task feature data includes at least one of algorithm type, algorithm parameter, data type, data amount, and data storage location.

In one embodiment, a training sample set of a computing engine is constructed through the following steps:

Step 31: collecting multiple pieces of task description data for describing task information;

Step 32: use the calculation engine to execute the task corresponding to each task description data, and obtain the task execution time corresponding to each task description data;

Step 33: Extract features that affect task execution time from each piece of task description data to form task feature data, and combine the obtained task execution time to construct a training sample set for the calculation engine.

In one embodiment, a task execution time prediction model of a computing engine is obtained by performing the following steps:

Step 41: Based on the training sample set of the calculation engine, with the task feature data as the independent variable and the task execution time as the dependent variable, establish a linear regression model, expressed as:

y _i =β ₀ +β ₁ x _i1 +...+β _p x _ip , i=1,2,...,n

Among them, x _i1 to x _ip represent the task features contained in the training sample set of the computing engine, i represents the serial number of the number of sample data contained in the training sample set of the computing engine, and n is the sample data of the training sample set of the computing engine The number of bars, β ₀ is the bias value to be optimized, β ₁ to β _p is the weight value to be optimized;

Step 42: using the least square method to solve the optimal weight value and bias value of the linear regression model;

Step 43: express the linear regression model according to the obtained optimized weight value and bias value, and obtain a task execution time prediction model of the calculation engine.

In one embodiment, step 2 includes the following sub-steps:

Step 51: Select the computing engine with the shortest predicted execution time; or

Step 52: When the remaining resources of the computing engine with the shortest predicted execution time cannot support the task to be calculated, preferentially select the computing engine with the shortest predicted time according to the prediction result of the task execution time.

According to a second aspect of the present invention, a multi-computing engine platform is provided. The platform includes:

Computing task management module: used to manage the processing flow of computing tasks and generate computing task information;

Engine management module: used to select a computing engine according to the computing task information from the computing task management module according to the computing engine selection method of the present invention;

Task execution module: used to execute computing tasks and output task execution time.

In one embodiment, the multi-computing engine selection platform of the present invention also includes:

Container management module: used to call the task execution module to execute computing tasks;

User interaction module: used to receive user operation instructions and information;

Debugging task management module: used to execute user debugging tasks and output debugging information.

In one embodiment, when the computing engine included in the platform is changed, the engine management module activates the task execution time prediction model of the new computing engine, and simultaneously sets the task execution time prediction model of the replaced computing engine to non- active state.

Compared with the prior art, the present invention has the advantages that: the provided calculation engine selection method can utilize machine learning methods to build task execution time prediction models for multiple calculation engines, and automatically select based on the prediction results of the constructed models combined with resource conditions The most efficient computing engine can significantly reduce task execution time and improve the efficiency of test data processing; the provided multi-calculation engine platform based on the calculation engine selection method provides a task management mechanism and can support calculation engine changes, which improves flexibility and at the same time Future engine extensions are well supported.

Description of drawings

The following drawings only illustrate and explain the present invention schematically, and are not intended to limit the scope of the present invention, wherein:

FIG. 1 shows a flowchart of a calculation engine selection method according to an embodiment of the present invention;

Fig. 2 shows a schematic framework diagram of a multi-computing engine platform according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solution, design method and advantages of the present invention clearer, the present invention will be further described in detail through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

According to an embodiment of the present invention, a computing engine selection method for a multi-computing engine platform is provided. In short, the method includes: collecting task execution data of multiple computing engines, constructing a training sample set; using the constructed The training sample set trains the task execution time prediction model through machine learning; use the trained task execution time prediction model to predict the task execution time of each computing engine, and then select the appropriate computing engine. Specifically, referring to FIG. 1, the computing engine selection method of the present invention includes the following steps:

Step S110, collecting task execution data of multiple computing engines to construct a training sample set.

In this step, the running time data of multiple computing engines facing computing tasks under different conditions is collected, and various tasks are included as much as possible, so as to construct a comprehensive training sample set.

According to one embodiment of the present invention, the process of constructing the training sample set includes the following sub-steps:

Step S111, preparing data to be tested

Collect the algorithms to be tested and prepare appropriate data to be tested for each algorithm. For example, the amount of data to be tested can be determined according to the performance of the training platform and the amount of data that each algorithm usually needs to process. Another example is to obtain an upper limit for the total amount of data to be tested according to the actual situation.

Step S112, preparing task description data

The task description data is used to describe the information of the executed task, for example, the algorithm for task execution, the relevant parameters of the algorithm, and so on. A calculation engine can perform specific tasks according to the task description data.

In one embodiment, the task description data is defined as a six-tuple Task_Info, including <Task_ID, Algorithm_ID, Algorithm_Args, Data_Type, Data_Size, Data_Path>, wherein Task_ID represents the task sequence number, and the content is an integer; Algorithm_ID represents the algorithm sequence number, and the content is Integer, according to the algorithm number, it can correspond to a specific algorithm. For example, the algorithm includes FP-Growth algorithm, K-Means algorithm, PageRank algorithm and Pearson correlation coefficient algorithm, etc.; Algorithm_Args indicates the algorithm parameters, and the content is a Json encoded string, starting with K -Means algorithm as an example, the algorithm parameters can include the number of fractional clusters k, the initialization method initMode, the maximum number of iterations maxItr, etc.; Data_Type indicates the data type, and the content is a discrete value of a digital label; Data_Size indicates the amount of data, the content is an integer, and the unit is byte ;Data_Path indicates the data storage location, the content has digital label discrete values, and the value can be local file system or distributed file system.

It should be noted that the defined task description data may include any other content that affects task execution time, for example, in addition to the above-mentioned algorithm sequence number and algorithm correlation coefficient, it may also include task execution priority and the like. In addition, for different algorithms, the content contained in the algorithm parameters is also different.

Step S113, execute the computing task according to the task description data, and obtain the task execution time data of multiple computing engines

For each computing engine, use the prepared description data Task_Info of multiple computing tasks to execute computing tasks, collect the execution time Run_Time of each computing task, and combine the task description data and execution time into a tuple <Task_Info, Run_Time> to obtain Task execution data.

Step 114, perform data cleaning on task execution data

The purpose of data cleaning is to remove potentially erroneous or incomplete data. For example, all task execution data can be statistically analyzed to obtain the standard deviation of task execution time. If a task execution time exceeds 3 times the standard deviation of the average task execution time, it will be marked as abnormal data and eliminated. In addition, the data with missing attribute columns is also eliminated. The missing attribute columns include missing task execution time or missing task description information, and the removal operation is also performed.

Step 115, feature engineering

The purpose of feature engineering is to select features that have a significant impact on the execution time of the task from the task description data to construct a training sample set. The training sample set of the present invention includes task feature data and corresponding task execution time, and the task feature data includes multiple task features that affect task execution time.

According to an embodiment of the present invention, the Task_ID in the task description data Task_Info has no influence on the execution time of the computing task, so this feature can be eliminated, and the algorithm sequence number (ie algorithm type), algorithm parameters, and data storage location in the task description description data The execution time of tasks such as , data type, and data volume are all affected, so they are reserved as task feature data in the training sample set.

When constructing the final training sample set, for the discretized task features, if serial number encoding is used, the order of the serial number will affect the unordered discrete quantities, resulting in additional information input. Therefore, according to an embodiment of the present invention, one-hot encoding (one-hot encoding) may be used to encode discrete data. For example, for a task description data, the discrete feature includes the data storage location, and the values of the discrete feature include "local file system" and "distributed file system". After using one-hot encoding, the discrete feature becomes two independent features: "data storage location - local" and "data storage location - distributed". When the feature value of the original data storage location is "local file system", the feature value of "data storage location-local" is 1, and the value of "data storage location-distributed" is 0; when the feature value of the original data storage location is For the "distributed file system", the value of the "data storage location - local" feature is 0, and the value of the "data storage location - distributed" feature is 1. Similarly, one-hot encoding can also be used for the algorithm serial number Algorithm_ID, that is, when four algorithms are included, the discrete feature becomes four independent features.

For clarity, Table 1 below illustrates an example of the constructed training sample set.

Table 1: Examples of training sample sets

Table 1 shows the training sample set of computing engine 1. It should be noted that the task characteristic data in it can include algorithm type, algorithm parameter, data storage location, data type, At least one of the data volumes or also other task characteristics can be added. In addition, when one-hot encoding is used to encode the discrete features of the task feature data, the discrete features will become multiple independent features, but the present invention is not limited to one-hot encoding to encode the discrete features.

Step S120, using the constructed training sample set to train a task execution time prediction model.

In this step, a machine learning method is used for training with a training sample set to obtain a task execution time prediction model corresponding to each computing engine.

For example, machine learning models such as linear regression, gradient boosted regression tree (GBRT), and XGBoost can be used for training.

In one embodiment, a linear regression model is used for training, the task feature data in the training sample set is used as an independent variable, and the task execution time is used as a dependent variable to establish a linear regression model. For example, a linear regression model can be expressed as:

Y=e+a ₁ X ₁ +a ₂ X ₂ +a ₃ X ₃ +a ₄ X ₄ +a ₅ X ₅ +a ₆ X ₆ +a ₇ X ₇ (1)

Among them, X ₁ represents the amount of data, X ₂ and X ₃ represent the characteristics of the data storage location using one-hot encoding, X ₄ to X ₇ represent the characteristics of the algorithm serial number Algorithm_ID using one-hot encoding, Y represents the task execution time, a ₁ to a ₇ represent the weight value to be optimized, and e represents the bias value to be optimized.

Generally, the p-element linear regression model that can be established can be expressed as:

y _i =β ₀ +β ₁ x _i1 +...+β _p x _ip , i=1, 2,..., n (2)

Among them, p is the number of features contained in the task feature data of the training sample set, i corresponds to the number of the number of data pieces contained in the training sample set, n is the number of data pieces in the training sample set, β ₀ is the bias to be optimized, β ₁ to β _p are the weight values to be optimized, and x _i1 to x _ip correspond to the task features in the training sample set.

During training, the optimal weight value and bias value can be obtained using the least squares method. The goal of the least squares method is to minimize the sum of squares of errors, namely:

Then, take partial derivatives for each parameter:

Get the normal system of equations:

Written in matrix form as:

X'Xβ=X'Y (6)

So as to get the solution of the parameters (including weight value and bias value):

In this step, the optimized weight value and bias value can be obtained through training. The model represented by these optimized weights and bias is the task execution time prediction model. In this way, the task execution time of each computing engine can be obtained predictive model.

Step S130, using the trained task execution time prediction model to predict the task execution time of each computing engine, and then select a suitable computing engine.

When a new computing task needs to be executed, the computing task characteristic data is first generated according to the computing task attributes, and then input into the task execution prediction model of each computing engine in turn to obtain the task execution time prediction results of each computing engine, and finally Combined with system resource conditions and computing task requirements, select the most appropriate computing engine to perform computing tasks. According to an embodiment of the present invention, the following sub-steps are included:

Step 131, generating computing task feature data

The process of generating task characteristic data to be calculated is similar to the process of generating task characteristic data when constructing a training sample set. For example, one-hot can also be used to encode the discrete features, and finally verify whether the converted computing task characteristic data conforms to the task execution Input to the temporal forecasting model.

Step 132, obtain task execution time prediction results

Input the characteristic data of the computing task to be predicted into the task execution time prediction model Model _i of a computing engine, and obtain the execution time prediction result P_Time _i of the computing task on the i-th engine, and then obtain the execution time prediction of all engines result.

Step 132, selecting a computing engine that will execute the computing task according to the prediction result.

According to the time prediction results of each computing engine combined with resource usage, an appropriate engine is selected to perform computing tasks.

For example, select the engine with the shortest time among all the execution time prediction results, and judge whether its remaining resources can support the operation of computing tasks, and if not, select the engine with the second shortest time among all the execution time prediction results, and judge its resources. Until the calculation task requirements are met, select this calculation engine to run the calculation task.

According to an embodiment of the present invention, a multi-computing engine platform is provided, the platform includes the calculation engine selection method provided by the present invention, and can be applied to experimental data processing. 2, the multi-computing engine platform of this embodiment includes a user interaction module 210, a debugging task execution module 220, a computing task management module 230, an engine management module 240, a container management module 250, a task execution module 260, and an algorithm management module 270 and algorithm library 271, task information management module 280 and task information library 281.

The user interaction module 210 is used for information interaction with users. Specifically, Flask (which is a Web microframework written in Python) can be used as the back-end part of the user interaction module 210, and bootstrap and jquery are used to build Web pages, and the background can provide a RESTful software architecture style interface for secondary development. For example, the specific process during actual application includes: using the Web interface to collect user operation instructions and input information, etc.; ) calls the routing interface at the back end of the user interaction module 210; the back end responds to the routing interface request, completes a specific function and returns a processing result; the Web interface accepts the processing result and responds to user operations.

The debugging task management module 220 is used to implement the function of pushing back-end algorithm output and debugging information to the Web interface in real time. For example, WebSocket can be used as the front-end and back-end communication protocol, and the implementation process includes: the user writes the algorithm and algorithm debugging data in the web interface; the user submits the debugging task; the back-end executes the user debugging task, and returns the execution result and debugging information; The user confirms the programming of the algorithm, and returns to the first step when debugging is still required; the user confirms that the algorithm is normal and adds the algorithm to the algorithm library 271 .

The computing task management module 230 is used to manage the processing flow of computing tasks. For example, the directed acyclic graph can be used to arrange the processing flow of test data, and the directed acyclic graph can be visualized on the web interface so that users can view and confirm the process information. Specifically, the steps of the calculation task management module 230 include: the user adds a calculation task according to the test data flow; confirms whether to modify the added calculation task according to the arrangement visualization result; after confirming that it is correct, the engine management module 240 can be requested to obtain the task execution engine information; The container management module 250 performs computing tasks.

The engine management module 240 is used for selecting a computing engine and managing the state of the computing engine, etc. The computing engine can be packaged through the open source application container engine Docker and turned into individual containers for resource management. The functions of the engine management module 240 include: calling the calculation engine selection algorithm of the present invention according to each task information, and selecting the optimal calculation engine; judging whether the selected engine is in an active state, and activating the engine if it is in an inactive state; The information of the selected task execution computing engine is returned to the computing task management module 230 .

The container management module 250 is used to manage container functions. For example, docker-python can be used to manage containers. Each container consists of a set of specific applications and necessary dependent libraries. The functions of the container management module 250 include providing functions such as adding containers, starting containers, stopping containers, and querying containers, and invoking the task execution module 260 to complete computing task execution.

The task execution module 260 uses container technology to package computing tasks uniformly, and provides functions of algorithm calling, operation monitoring and data collection. The functions of the task execution module 260 include: checking the integrity of the task data; judging whether the algorithm needs a third-party library, and installing the third-party library if necessary; judging whether the input data form of the algorithm is consistent with the algorithm requirements, if inconsistent, convert the input data form according to the algorithm requirements; Algorithm and statistics of algorithm execution time; collect algorithm execution results and update computing task information, etc.

The algorithm management module 270 and the algorithm library 271 are used to manage the algorithms implemented by the multi-computing engine platform, for example, use the databases MongoDB and HDFS based on distributed file storage to provide distributed algorithm library support, and provide algorithm addition, deletion and query functions.

Task information management module 280 and task information library 281 are used to manage task information, for example, use relational database management system MySQL to provide storage and management of task information, and provide functions such as adding, deleting, and querying task information.

It should be noted that, since the method of data processing flow changes rapidly when carrying out actual test data, in order to save the resources of the multi-computing engine platform and improve the calculation efficiency, the computing engine used by the platform should be changed according to the task of test data processing. When the computing engine is changed, it is necessary to dynamically adjust the state of the task execution time prediction model obtained according to the present invention according to the engine change, so as to realize the self-adaptation to the engine change. For example, when the changed computing engine is a newly added computing engine, the engine is trained using the training sample set to obtain a task execution time prediction model of the computing engine. For another example, if the engine that has been trained is replaced, the task execution time prediction model of the new computing engine is activated, and the task execution time prediction model of the replaced engine is set to an inactive state.

In summary, the multi-computing engine platform provided by the present invention is convenient and easy to use, can meet the requirements of test data processing, support online arrangement of multiple computing tasks, and can automatically select the most efficient computing engine based on the computing engine selection method of the present invention, Supports online algorithm debugging, automatic encapsulation and switching of computing engines, etc. Specifically, the beneficial effects of the multi-computing engine platform of the present invention are mainly reflected in: using container technology to encapsulate computing engines and computing tasks, fast start and stop speed, low resource consumption, and achieving the purpose of environmental isolation and resource limitation between engines and tasks; Utilizes the general computing task call process to unify the differences between different tasks and facilitate unified management of computing tasks; supports online editing and testing of algorithms, and can try to view algorithm debugging results, which can greatly improve algorithm editing efficiency; oriented to test data processing It provides visual arrangement of computing tasks, which can reduce user programming errors and improve the flexibility of test data processing; the multi-computing engine selection algorithm based on task execution time prediction can use machine learning methods to automatically discover the operating rules of computing engines and target specific calculations. The calculation engine with the highest efficiency is selected according to the task combined with the resource situation, thereby significantly reducing the execution time of the calculation task and improving the efficiency of test data processing; supporting the change of the calculation engine can improve the flexibility of the system and provide good support for future engine expansion.

It should be noted that although the steps are described above in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be performed concurrently, or even change the order, as long as it can be realized The required functions are sufficient.

The present invention can be a system, method and/or computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present invention.

A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may include, for example, but is not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above.

Having described various embodiments of the present invention, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or technical improvement in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

Claims (10)

1. a kind of computing engines selection method, comprising the following steps:

Step 1: the corresponding task characteristic of task to be calculated is input to each of multiple computing engines computing engines Task execution time prediction model, obtain task to be calculated on each computing engines task execution time prediction knot Fruit, wherein the task execution time prediction model is to be obtained based on training sample set by training, the training sample set packet Include a plurality of task characteristic and corresponding task execution time；

Step 2: being selected to execute task to be calculated from the multiple computing engines according to the task execution time prediction result Computing engines.

2. according to the method described in claim 1, wherein, the task characteristic includes algorithm types, algorithm parameter, data At least one of in type, data volume and data storage position.

3. according to the method described in claim 1, wherein, the training sample set of a computing engines is constructed by following steps:

Step 31: a plurality of task description data for being used to describe mission bit stream are collected,；

Step 32: executing the corresponding task of each task description data using the computing engines, obtain each task description The corresponding task execution time of data；

Step 33: the feature composition task characteristic for influencing task execution time is extracted from each task description data, The training sample set of the computing engines is constructed in conjunction with task execution time obtained.

4. according to the method described in claim 1, wherein, the task execution of a computing engines is obtained by executing following steps Time prediction model:

Step 41: the training sample set based on the computing engines is with task execution time using task characteristic as independent variable Dependent variable establishes linear regression model (LRM), indicates are as follows:

y_i=β₀+β₁x_i1+…+β_px_ip, i=1,2 ..., n

Wherein, x_i1To x_ipIndicate the task feature that the training sample set of the computing engines includes, i indicates the training of the computing engines The number for the sample data item number for including in sample set, n are the sample data item number of the training sample set of the computing engines, β₀For Bias to be optimized, β₁To β_pFor weighted value to be optimized；

Step 42: the optimization weighted value and bias of the linear regression model (LRM) are solved using least square method；

Step 43: the linear regression model (LRM) being indicated according to the optimization weighted value and bias of acquisition, obtains the computing engines Task execution time prediction model.

5. method according to any one of claims 1 to 4, wherein step 2 includes following sub-step:

Step 51: selection prediction executes time shortest computing engines；Or

Step 52: when the prediction execute time shortest computing engines surplus resources cannot support task to be calculated the case where Under, according to the task execution time prediction result successively preferential short computing engines of selection predicted time.

6. a kind of more computing engines platforms characterized by comprising

Calculating task management module: for managing the process flow of calculating task and generating calculating task information；

Engine management module: for according to from the calculating task management module calculating task information according to claim 1 to 5 described in any item method choice computing engines；

Task execution module: for executing calculating task and exporting task execution time.

7. more computing engines platforms according to claim 6, which is characterized in that further include:

Container Management module: for calling the task execution module to execute calculating task；

User interactive module: for receiving user operation instruction and information；

Debugging task management module: for executing user's debugging task and exporting Debugging message.

8. platform according to claim 7, which is characterized in that described when the computing engines change that the platform includes Engine management module activates the task execution time prediction model of new computing engines, while appointing the computing engines replaced Business running time prediction model is set as unactivated state.

9. a kind of computer readable storage medium, is stored thereon with computer program, wherein real when the program is executed by processor Now according to claim 1 to any one of 5 the method the step of.

10. a kind of computer equipment, including memory and processor, be stored on the memory to transport on a processor Capable computer program, which is characterized in that the processor realizes any one of claims 1 to 5 institute when executing described program The step of method stated.