CN116775233A - A task scheduling method and device - Google Patents

Abstract

This application provides a task scheduling method, device, electronic equipment and computer program product, which relates to the field of information technology. The method includes: obtaining the resource usage of the host; predicting the available host resources at the future target time; obtaining the tasks to be scheduled, based on The host resource weight set builds a prediction model, and uses the prediction model to obtain the required host resource ratio of the task to be scheduled; calculates the task weight of the task to be scheduled based on the host resource weight set; based on the required host resource ratio and The task weights are calculated respectively to calculate the first remaining resources after resource allocation of currently available host resources and the second remaining resources after resource allocation of available host resources in the future; when it is judged that the second remaining resources are larger, according to each waiting The time limit for scheduling tasks requires task scheduling. This application can ensure the balanced use of host resources while ensuring the consistency of task scheduling results, effectively improving resource utilization.

Description

A task scheduling method and device

Technical field

This application relates to the field of information technology, and specifically to a task scheduling method, device, electronic equipment and computer program product.

Background technique

As service providers modernize their networks, their workloads and services are moving from the core network (located within the data center) to the edge of the network. In the distributed task scheduling of NFV, consistency in task scheduling needs to be ensured. Common Technical solutions include relative balancing solutions based on the number of hosts and tasks, and comprehensive balancing solutions based on the number of resources and tasks. However, most of these solutions are implemented based on distributed scheduling solutions of management nodes and scheduling agent nodes. The more popular ones are Kubernetes. , this type of solution requires application adaptation and transformation. Another example is some solutions that provide task scheduling, which allocate the hash values of execution units and tasks to a consistent hash ring, and find the n corresponding executions closest to the hash value position of the task on the consistent hash ring. The hash value of the unit realizes flexible matching between execution units and tasks. The same task can be assigned to different execution units to meet the requirement of executing a task multiple times. However, this task scheduling scheme does not consider the host of each host node. resources, it will cause the problem of imbalance of host resources after the task is started.

In summary, the existing technology has the following shortcomings: the scheduling task algorithm is complex, using specific hash mixed operations to randomly specify the scheduling task host node or formulating a closed scheduling strategy based on open source products (such as zookeeper, Kubernetes), and the scheduling task scheduling results cannot be predicted. , thus unable to ensure balanced use of host resources.

Contents of the invention

Embodiments of the present application provide a task scheduling method, device, electronic equipment and computer program product to solve the above technical problems and ensure the consistency of task scheduling results and the balanced use of host resources.

In a first aspect, embodiments of the present application provide a task scheduling method, including:

Obtain the resource usage reported by each host in the distributed system in real time; wherein the resource usage includes the available host resources at the current moment;

Predict available host resources at the future target time based on the resource usage and current task running information;

Obtain all to-be-scheduled tasks in the time period between the current time and the future target time, build a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain all the tasks to be scheduled. The proportion of host resources required;

Calculate the task weight of the task to be scheduled according to the host resource weight set;

According to the proportion of required host resources and the task weight, the first remaining resources after resource allocation based on the available host resources at the current time are calculated, and the first remaining resources are calculated based on the available host resources at the future target time. The second remaining resource after resource allocation;

When it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirement of each task to be scheduled.

In one embodiment, the prediction of available host resources at a future target time based on the resource usage and current task running information includes:

Screen out tasks that end before the future target time according to the current task running information, and record the resource occupancy of the tasks;

Prediction is made based on the resource occupancy situation and the resource usage situation to obtain the available host resources at the future target time.

In one embodiment, the method of obtaining the host resource weight set includes:

Obtain the observation values of historical tasks within a preset historical time range, as well as the resource usage of each host corresponding to the historical task, and construct a mapping matrix between the resource usage of each host and the historical task;

Based on the mapping matrix, use the preset exponential smoothing algorithm to calculate and obtain the predicted value of each host resource at the current moment;

The host resource weight set is obtained by performing a weighted average calculation based on the predicted value of each host resource and the preset weight value of each host resource.

In one embodiment, calculating the task weight of the to-be-scheduled task based on the host resource weight set includes:

Using a preset secondary algorithm to calculate each host resource weight in the host resource weight set to obtain the secondary weight of each host;

Accumulate the secondary weights of each host to obtain the maximum weight of the cluster;

The task weight of the task to be scheduled is calculated by combining the maximum weight of the cluster and the random result obtained by the host.

In one embodiment, when it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirement of each task to be scheduled, which specifically includes:

When it is determined that the first priority resource among the second remaining resources is greater than the first priority resource among the first remaining resources, task scheduling is performed according to the time limit requirement of each to-be-scheduled task.

In one embodiment, before obtaining the host resource weight, the method further includes:

The weight value of each host resource is adaptively adjusted according to the available host resources at the future target time.

In one embodiment, the resource type of the host includes one or more of MEM resources, CPU resources, memory resources, IO resources, file handle resources and file descriptor resources.

In a second aspect, embodiments of the present application provide a task scheduling device, including:

The data acquisition module is used to obtain the resource usage reported by each host in the distributed system in real time; wherein the resource usage includes the available host resources at the current moment;

A resource prediction module, used to predict available host resources at a future target time based on the resource usage and current task running information;

The demand prediction module is used to obtain all to-be-scheduled tasks in the time period between the current time and the future target time, build a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain each Describe the proportion of host resources required for tasks to be scheduled;

A weight calculation module, configured to calculate the task weight of the to-be-scheduled task according to the host resource weight set;

A resource calculation module, configured to calculate the first remaining resources after resource allocation based on the available host resources at the current moment and the future target based on the required host resource proportion and the task weight respectively. The second remaining resources after resource allocation based on the available host resources at the moment;

A task scheduling module, configured to perform task scheduling according to the time limit requirement of each task to be scheduled when it is determined that the second remaining resource is greater than the first remaining resource.

In a third aspect, embodiments of the present application provide an electronic device, including a processor and a memory storing a computer program. When the processor executes the program, the steps of the task scheduling method described in the first aspect are implemented.

In a fourth aspect, embodiments of the present application provide a computer program product, including a computer program that implements the steps of the task scheduling method described in the first aspect when executed by a processor.

The task scheduling methods, devices, electronic equipment and computer program products provided by the embodiments of the present application predict the available resources at the future target time based on the resources and task running information reported by the host in real time, and at the same time predict the required resources for the tasks to be scheduled, and then Combine the calculated task weights to allocate resources and predict the remaining resources in the future, and then schedule the tasks based on the remaining resources and the time limit requirements of the tasks to be scheduled, thus ensuring the consistency of the task scheduling results while ensuring the availability of host resources. Balanced use effectively improves resource utilization.

Description of drawings

In order to explain the technical solutions in this application or the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are of the present invention. For some embodiments of the application, those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is one of the flow diagrams of the task scheduling method provided by the embodiment of the present application;

Figure 2 is the second schematic flowchart of the task scheduling method provided by the embodiment of the present application;

Figure 3 is a schematic structural diagram of a task scheduling device provided by an embodiment of the present application;

Figure 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are part of this application. Examples, not all examples. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

Figure 1 is a schematic flow chart of the task scheduling method. Referring to Figure 1, an embodiment of the present application provides a task scheduling method, which may include the steps:

S1. Obtain the resource usage reported by each host in the distributed system in real time; wherein the resource usage includes the available host resources at the current moment. The resource type of the host includes one or more of MEM resources, CPU resources, memory resources, IO resources, file handle resources and file descriptor resources.

It is understandable that each host in the distributed system can regularly report resource usage at minute intervals, including host IP, CPU usage, memory usage, IO usage, file handle usage and other information. These data can Stored in the database, one piece of data is collected every minute.

S2. Predict available host resources at a future target time based on the resource usage and current task running information.

It is understandable that based on the resource usage obtained in real time and combined with the running information of the running tasks, the available host resources at the future target time can be predicted. For example, each task includes the start time, current resource occupancy ratio, running cycle and Estimated end time and other information.

S3. Obtain all to-be-scheduled tasks in the time period between the current time and the future target time, build a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain each of the to-be-scheduled tasks. The proportion of required host resources.

It should be noted that for all to-be-scheduled tasks in the future period, the proportion of resources required for each to-be-scheduled task can be calculated using a prediction model constructed based on the pre-obtained set of host resource rights.

S4. Calculate the task weight of the task to be scheduled according to the host resource weight set.

It should be noted that according to the host resource weight set, the preset algorithm model can be used to further obtain the task weight of the scheduled task. Since each host uses the same computing model, each task can be obtained on each host. Same task weight.

S5. According to the proportion of required host resources and the task weight, respectively calculate the first remaining resources after resource allocation based on the available host resources at the current time, and the available hosts based on the future target time. The second remaining resource after resource allocation.

It should be noted that based on the available host resources at the current moment, the host resources are allocated to multiple to-be-scheduled tasks according to the required host resource proportions and task weights, and the first remaining resources can be calculated. Similarly, based on future goals The available host resources at the moment can be calculated to obtain the second remaining resources.

S6. When it is determined that the second remaining resource is greater than the first remaining resource, perform task scheduling according to the time limit requirement of each task to be scheduled.

Compare the size of the first remaining resource and the second remaining resource. If the second remaining resource is larger, determine the urgency of the task to be scheduled. If one or some of the tasks to be scheduled can withstand the corresponding delay, the future-based task can be used. The allocation method of available host resources at the target time is used to allocate resources to multiple tasks to be scheduled, and schedule the tasks according to the resource allocation.

The task scheduling method provided by the embodiment of the present application predicts the available resources at the future target time based on the resources reported by the host in real time and the task running information, and at the same time predicts the resources required for the tasks to be scheduled, and then combines the calculated task weights to calculate the resources. Allocate and predict the remaining resources in the future, and then perform task scheduling based on the remaining resources and the time limit requirements of the tasks to be scheduled. This can ensure the consistency of task scheduling results while ensuring the balanced use of host resources, effectively improving resource utilization. Rate.

In one embodiment, step S2 may specifically include:

Screen out tasks that end before the future target time according to the current task running information, and record the resource occupancy of the tasks;

Prediction is made based on the resource occupancy situation and the resource usage situation to obtain the available host resources at the future target time.

It should be noted that, based on the running information of currently running tasks, tasks that end before the future target time can be identified and filtered out, and based on the resource occupancy ratio of these tasks, the available host resources at the future target time can be calculated.

The task scheduling method provided by the embodiments of this application can further predict the scheduling results at the future target time by predicting the available host resources at the future target time, and then perform resource allocation and task scheduling based on the scheduling results, effectively improving resource utilization.

In one embodiment, the acquisition method of the host resource weight set may include:

Obtain the observation values of historical tasks within a preset historical time range, as well as the resource usage of each host corresponding to the historical task, and construct a mapping matrix between the resource usage of each host and the historical task;

Based on the mapping matrix, use the preset exponential smoothing algorithm to calculate and obtain the predicted value of each host resource at the current moment;

The host resource weight set is obtained by performing a weighted average calculation based on the predicted value of each host resource and the preset weight value of each host resource.

It should be noted that when obtaining the host resource weight set, first construct a mapping matrix based on the observed values of historical tasks and resource usage, then use the exponential smoothing algorithm to predict the resource prediction value at the current moment, and then use the weighted average algorithm to calculate Obtain the resource weights of all hosts in the host set to form a host resource weight set.

In one embodiment, step S4 may specifically include:

Using a preset secondary algorithm to calculate each host resource weight in the host resource weight set to obtain the secondary weight of each host;

Accumulate the secondary weights of each host to obtain the maximum weight of the cluster;

The task weight of the task to be scheduled is calculated by combining the maximum weight of the cluster and the random result obtained by the host.

It should be noted that in order to calculate the weights of each host resource in proportion according to the percentage system so that the values of the host resource weight set become more concentrated, a secondary algorithm can be used to calculate the values of the host resource weight set to obtain each value. The quadratic weight of the host; the maximum weight of the cluster is obtained by accumulating each quadratic weight, and then the random result calculation is performed based on the random seed of the host. Combining the maximum weight of the cluster and the same random result, we can get the weight of each host for the same The same task weight of the tasks to be scheduled.

The task scheduling method provided by the embodiment of the present application uses a secondary algorithm to calculate the host resource weight to obtain the secondary weight, ensuring the concentration of the weight set and preventing the host resource weight from being too large or too small. The algorithm is abnormal and effectively improves the reliability of resource prediction.

In one embodiment, when it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirement of each task to be scheduled, which specifically includes:

When it is determined that the first priority resource among the second remaining resources is greater than the first priority resource among the first remaining resources, task scheduling is performed according to the time limit requirement of each to-be-scheduled task.

It should be noted that when comparing the sizes of the second remaining resources and the first remaining resources, the resources can be compared hierarchically. For example, resources necessary for executing tasks such as CPU and MEN are set as the first priority. For IO usage, etc. Resources that affect task execution speed are set to the second priority; when comparing sizes, first compare the size of the first priority resource. If the size of the first priority resource is equal, compare the size of the second priority resource. The first priority resources and the second priority resources can also be weighted and then compared in size. For example, for the first remaining resources, the first priority resources and the second priority resources can be compared in a ratio of 2:1. The weighted sum is used as the comparison value of the first remaining resource, and similarly the comparison value of the second remaining resource is obtained, and then the comparison value of the first remaining resource and the comparison value of the second remaining resource are compared.

The task scheduling method provided by the embodiment of the present application compares the sizes of the first remaining resources and the second remaining resources by dividing the first priority and the second priority resources, so as to obtain a more accurate comparison that is more suitable for the actual scenario. As a result, the reliability and accuracy of resource prediction and task scheduling are effectively improved, and resource utilization is further improved.

In one embodiment, before obtaining the host resource weight, the method further includes:

The weight value of each host resource is adaptively adjusted according to the available host resources at the future target time.

In the embodiment of the present application, before calculating the host resource weight, the host resource weight value used to calculate the host resource weight is first adapted and adjusted. The adjustment is based on the predicted available host resources at the future target time. For example, if Currently, CPU and MEM resources are abundant, but IO resources are few. When predicting the resource weights of each host, the weights of IO resources can be appropriately increased so that the resource weights obtained from the predicted values can satisfy the requirement of less IO calls. Resource requirements.

The task scheduling method provided by the embodiment of the present application effectively improves the prediction accuracy of subsequent host resource weights, avoids task execution failures caused by the predicted value of a certain scheduling factor being lower than the actual usage value, and thereby effectively improves resource prediction and task performance. The reliability and accuracy of scheduling further improves resource utilization.

Please refer to Figure 2. Based on the above solution, in order to better understand the task scheduling method provided by the embodiment of the present application, the following is a detailed description:

The embodiments of this application mainly provide a task scheduling method based on the resource priority consistency algorithm, which is used to solve the problem of how to ensure the consistency of task scheduling results on the basis of non-communication between host nodes in a distributed system, and how to ensure sufficient The problem of utilizing host resources applies to scenarios where a single or multiple tasks are started on one or several host nodes.

The embodiment of this application mainly includes the following steps:

Step A101: Calculate the resource weight of each host based on the host resources;

Step A102: Predict the resources that each host may release in the future based on the scheduling task resource factors;

Step A103: Predict the host resources required for tasks to be scheduled in the future based on the exponential smoothing algorithm;

Step A104: Calculate the algorithm's ratio-reduced resource weight according to the basic resource weight calculation format;

Step A105: Calculate the maximum resource weight of the algorithm;

Step A106: Calculate the scheduling task weight of the current task based on the maximum weight of the scheduling task and the name of the scheduling task;

Step A107: Schedule tasks based on the available host resources in a future period, the host resources required by each task to be scheduled, and the time limit requirements of each task to be scheduled.

Based on the above solution, the specific implementation process is listed below for detailed description:

Step S101: Regularly report resources.

Scheduling units can be deployed on each distributed system host to regularly report the host's IP, CPU usage, memory usage, IO usage, file handle usage and other information according to minutes (these data can be stored in the database, one piece of data per minute ).

Step S102: Predict the host resources that may be released in the future t1 period, and obtain the available host resources at the current time + t1 time point. Forecasting methods may specifically include:

Step S1021: Obtain the host resource usage S _t at the current time t and the running result information of the currently executing task through the fusion scheduling module. The running result information includes keywords, resource occupation ratio after running, running cycle, etc. (stored in the database, There is one scheduling task for each, and the proportion of running resources can be configured such as MEM, CPU, IO, etc.). This embodiment obtains information such as the start time, current resource occupancy ratio, and expected end time of the running scheduler according to the running keyword.

Step S1022: Filter out the tasks that are currently being executed and will end in the future t1 time, and record the current resource occupancy ratio of these tasks as C _i,j , where i represents the resource type, including CPU usage and memory usage. rate, IO usage, file handle usage, etc., j represents the task number.

Step S1023, using the formula Predict the resource utilization rate of each host at time t+t1, where S _i,t is the utilization rate of the i-th host resource at the current time t, and S _i,t+t1 is the utilization rate of the i-th host resource at time t+t1. .

Take the MEM resource election as an example: the current host MEM usage has reached 75%, and the current resource occupancy ratio required by the scheduling module is 15%. For example, after the current scheduling module is started, the host MEM will reach 90%, exceeding the safety setting of 85 %, but according to the algorithm prediction, in the next 5 minutes, after the original scheduling task ends, the host will release 20% of the MEM usage, then we determine that the current host can schedule this module to ensure that the operating system does not force killing based on security protection. A process with high MEM usage requires that when the resource usage of the current host + the resource usage after scheduling is greater than 90%, the current host cannot schedule this task.

Step S103: Obtain all tasks to be scheduled in the future t1 time period, and calculate the proportion of host resources required by the tasks.

After completing the resource reporting, perform task scheduling, pass the task name and task number to be scheduled to the calculation module, predict the proportion information of various host resources required for the task to be scheduled based on the task name and task number, and obtain the scheduling factors of various host resources. (Stored in the database, the number of factors should not exceed 9. In the embodiment of this application, the configuration is CPU, MEM, IO, and FD. The specific number of factors can be added and deleted through the database configuration. Each piece of data Record a resource factor and configure it according to the importance of the resource factor. The larger the value, the greater the identification proportion, and the value of the resource factor can be adjusted in real time according to the distribution of currently available resources).

Predict the proportion of various host resources required for tasks to be scheduled based on the task name and task number, including:

Step S1031: Obtain historical task information, and build a model based on the neural network algorithm to predict the usage proportion of each resource based on the historical task information.

Among them, the neural network algorithm can be an exponential smoothing algorithm, and the specific method of building a prediction model is:

Obtain multiple observation values such as task name, task number, and host resource information within a certain historical period; obtain the usage of each host resource corresponding to each task, and calculate the actual usage rate of each resource; construct observation values and resource usage rate the mapping matrix;

Use machine learning (the embodiment of this application uses exponential smoothing algorithm to achieve intelligent prediction) to predict the resource weight A of all hosts in the host set; calculate based on 10 pieces of resource information reported in the past 10 minutes, the mathematical calculation formula is S _t =a *y _t +(1-a)*S _t-1 , where a is the first-order exponential smoothing value of the prediction algorithm, which can be set to 0.5 in this embodiment, y _t is the reported resource usage rate at time t, and S _t-1 is the previous The predicted value at a moment, the predicted value of the resource at the current moment is sequentially predicted as CPU _t , MEM _t , IO _t , FD _t , and the weighted average algorithm is used to obtain the host resource weight through the formula:

A＝(CPU _t *CPU _sa2 +MEM _t *MEM _sa2 +IO _t *IO _sa2 +FD _t *FD _sa2 )/(CPU _sa2 +MEM _sa2 +IO _sa2 +FD _sa2 ), calculate the resource weight of each host A1～Am, and then construct a prediction model. Among them, CPU _sa2 , MEM _sa2 , IO _sa2 and FD _sa2 are the weight values of each resource. The weight values can be adjusted in real time according to the usage of various resources at time t1 in the future predicted in step S102. For example, if the current CPU and MEM resources are abundant, but IO resources are few. At this time, when predicting the resource weight of each host, the weight of IO resources can be appropriately increased so that the resource weight obtained by the predicted value can meet the demand of calling fewer IO resources. Require.

Step S1032: Predict and obtain the proportion of each resource required for the task to be scheduled based on the above prediction model.

Step S104: Format cluster host weight B.

For the resource weight value A of each host obtained in the previous step, the secondary weight value B1~Bm of each host is obtained according to the second-level algorithm. The value algorithm is calculated according to the normalization algorithm. The mathematical calculation formula is: Among them, Ai is the corresponding value of A1~Am, and Ai is the current normalized element;

The secondary weight is added here in order to proportion the original A1 ~ Am data according to the percentage system and ensure that the weight B is between 0 and 100. If the secondary weight algorithm is not added, the first weight will be The value in A will become more discrete. For example, the weight of A is 20, 80, or 100. Use the quadratic weight algorithm to change the value of B to 10, 40, or 50 to prevent the weight A from being too large or too small. abnormal.

Step S105: Obtain the maximum weight C of the cluster.

The values from B1 to Bm in the previous steps are accumulated to obtain the cluster maximum weight C.

Step S106: Obtain task weight D.

First, obtain the random seed of the host. The random seed uses the timestamp of the last minute of the scheduling time, task name and task number for random operation. The ASCII code value of the timestamp, task name and task number is converted into the corresponding number. The bkdrhash algorithm can be used. The task name and task number are passed in from the fusion scheduling module. Since the random seeds of each host are the same, the random results obtained by each host are the same. After obtaining the random result D1, the random result is obtained according to the algorithm D=C*D1/(double)(RAND_MAX) Task weight D, where D1=rand(atoi(time_stamp)+to_number(application name+task number)), the to_number algorithm can use the bkdrhash algorithm, and atoi() is a function that converts a string into an integer. So far, each task has obtained the same computing task weight D on each host.

Step S107: Task screening.

The specific steps to perform task screening include:

S1071: Based on the host resources available at the current moment, allocate host resources to the multiple to-be-scheduled tasks, and calculate the number of remaining resources after task allocation S1;

S1072: Based on the predicted available host resources at time t1, allocate host resources to the multiple to-be-scheduled tasks, and calculate the number of remaining resources after task allocation S2. At this time, only how to optimize the resource configuration of multiple tasks is considered, and the execution order of each task is not considered;

S1073: Compare S1 and S2. If S2>S1, determine the urgency of the tasks to be scheduled in the queue. If one or some tasks that need to be adjusted in execution order can withstand the delay of t1, use the prediction-based t1 Allocate resources to multiple tasks to be scheduled based on available host resources at all times.

When comparing S1 and S2, you can classify resources. For example, resources necessary for task execution such as CPU and MEN are set as the first priority, and resources that affect task execution speed such as IO usage are set as the second priority. First compare the sizes of various types of resources with the first priority, and then compare the sizes of various types of resources with the second priority.

S1074: Schedule tasks for which resources have been allocated.

It should be noted that the embodiments of this application provide a consensus algorithm for distributed hosts, and also take into account the optimized use of host resources in distributed mode. The algorithm can be expanded according to actual needs, and the algorithm can be evolved on this basis. It has excellent practicability, and the scheduling algorithm used at the same time is simple and lightweight. It can be simulated directly according to the example, the scheduling results can be predicted, and the trend of scheduling results can be reproduced according to specific scenarios, and it has strong scalability. The calculation algorithms of each resource weight A and secondary weight B involved can be adjusted for rationality and adaptability, such as adjusting the CPU weight, increasing the SOCKET connection number weight, etc.

The task scheduling method provided by the embodiment of the present application predicts the host resources that may be released in the future, uses an exponential smoothing algorithm to predict the host resources required for tasks to be scheduled in the future, and obtains the results based on the prediction. The distribution of host resources adjusts the weight of each resource in the algorithm in real time, ensuring that during resource prediction, sufficient margin is given to resources that are focused on or currently have few remaining resources, and avoids tasks caused by small deviations in prediction results. Impossible to perform.

Compared with the prior art, the embodiments of the present application have the following beneficial effects:

1. The embodiments of this application not only provide a consensus algorithm for distributed hosts, but also take into account the optimized use of host resources in distributed mode. The algorithm can be expanded according to actual needs, and the algorithm can be evolved on this basis. Practicality Excellent.

2. The scheduling algorithm used in the embodiments of this application is simple and lightweight, can be simulated directly according to examples, can predict scheduling results, and can reproduce the trend of scheduling results according to specific scenarios, has strong scalability, and involves the weight of each resource. Both the A and B calculation algorithms can be adjusted for rationality and adaptability, such as adjusting the CPU weight, increasing the weight of the SOCKET connection number, etc.

3. The embodiment of this application can solve the problem of "difficulty in changing the scheduling algorithm and the inability to add other scheduling factors". The scheduling factor configuration can be added and deleted in the database to ensure that the scheduling factors and the number of reported resources are consistent. When making exponential predictions, dynamic calculations can be performed based on configured parameters.

The task scheduling device provided by the embodiment of the present application is described below. The task scheduling device described below and the task scheduling method described above can be referred to correspondingly.

Referring to Figure 3, an embodiment of the present application provides a task scheduling device, including:

The data acquisition module 1 is used to obtain the resource usage reported by each host in the distributed system in real time; wherein the resource usage includes the available host resources at the current moment;

Resource prediction module 2, used to predict available host resources at a future target time based on the resource usage and current task running information;

Demand prediction module 3 is used to obtain all to-be-scheduled tasks in the time period between the current time and the future target time, build a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain each The proportion of host resources required for the tasks to be scheduled;

Weight calculation module 4, configured to calculate the task weight of the to-be-scheduled task according to the host resource weight set;

Resource calculation module 5, configured to calculate the first remaining resources after resource allocation based on the available host resources at the current moment and the future resources based on the required host resource proportion and the task weight respectively. The second remaining resources after resource allocation of the available host resources at the target time;

The task scheduling module 6 is configured to perform task scheduling according to the time limit requirement of each task to be scheduled when it is determined that the second remaining resource is greater than the first remaining resource.

In one embodiment, the resource prediction module 2 is specifically used to:

Screen out tasks that end before the future target time according to the current task running information, and record the resource occupancy of the tasks;

Prediction is made based on the resource occupancy situation and the resource usage situation to obtain the available host resources at the future target time.

In one embodiment, the method of obtaining the host resource weight set includes:

Obtain the observation values of historical tasks within a preset historical time range, as well as the resource usage of each host corresponding to the historical task, and construct a mapping matrix between the resource usage of each host and the historical task;

Based on the mapping matrix, use the preset exponential smoothing algorithm to calculate and obtain the predicted value of each host resource at the current moment;

The host resource weight set is obtained by performing a weighted average calculation based on the predicted value of each host resource and the preset weight value of each host resource.

In one embodiment, the weight calculation module 4 is specifically used to:

Using a preset secondary algorithm to calculate each host resource weight in the host resource weight set to obtain the secondary weight of each host;

Accumulate the secondary weights of each host to obtain the maximum weight of the cluster;

The task weight of the task to be scheduled is calculated by combining the maximum weight of the cluster and the random result obtained by the host.

In one embodiment, the task scheduling module 6 is specifically used to:

When it is determined that the first priority resource among the second remaining resources is greater than the first priority resource among the first remaining resources, task scheduling is performed according to the time limit requirement of each to-be-scheduled task.

In one embodiment, it also includes a weight adjustment module for:

Before obtaining the host resource weight value, the host resource weight value is adaptively adjusted according to the available host resources at the future target time.

In one embodiment, the resource type of the host includes one or more of MEM resources, CPU resources, memory resources, IO resources, file handle resources and file descriptor resources.

It can be understood that the above device embodiments correspond to the method embodiments of the present application. The task scheduling device provided by the embodiments of the present application can implement the task scheduling method provided by any method embodiment of the present application.

Figure 4 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 4, the electronic device may include: a processor (processor) 410, a communication interface (Communication Interface) 420, a memory (memory) 430 and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 complete communication with each other through the communication bus 440. The processor 410 can call the computer program in the memory 430 to perform the steps of the task scheduling method, for example, including:

S1. Obtain the resource usage reported by each host in the distributed system in real time; wherein the resource usage includes the available host resources at the current moment;

S2. Predict the available host resources at the future target time based on the resource usage and current task running information;

S3. Obtain all to-be-scheduled tasks in the time period between the current time and the future target time, build a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain each of the to-be-scheduled tasks. The proportion of required host resources;

S4. Calculate the task weight of the task to be scheduled according to the host resource weight set;

S5. According to the proportion of required host resources and the task weight, respectively calculate the first remaining resources after resource allocation based on the available host resources at the current time, and the available hosts based on the future target time. The second remaining resource after resource allocation;

S6. When it is determined that the second remaining resource is greater than the first remaining resource, perform task scheduling according to the time limit requirement of each task to be scheduled.

In addition, the above-mentioned logical instructions in the memory 430 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code. .

On the other hand, embodiments of the present application also provide a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. The computer program is executed by a processor. When , the computer can perform the steps of the task scheduling method provided by the above embodiments, including, for example:

S1. Obtain the resource usage reported by each host in the distributed system in real time; wherein the resource usage includes the available host resources at the current moment;

S2. Predict the available host resources at the future target time based on the resource usage and current task running information;

S3. Obtain all to-be-scheduled tasks in the time period between the current time and the future target time, build a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain each of the to-be-scheduled tasks. The proportion of required host resources;

S4. Calculate the task weight of the task to be scheduled according to the host resource weight set;

S5. According to the proportion of required host resources and the task weight, respectively calculate the first remaining resources after resource allocation based on the available host resources at the current time, and the available hosts based on the future target time. The second remaining resource after resource allocation;

S6. When it is determined that the second remaining resource is greater than the first remaining resource, perform task scheduling according to the time limit requirement of each task to be scheduled.

On the other hand, embodiments of the present application also provide a processor-readable storage medium. The processor-readable storage medium stores a computer program. The computer program is used to cause the processor to execute the methods provided in the above embodiments. Steps, for example include:

S1. Obtain the resource usage reported by each host in the distributed system in real time; wherein the resource usage includes the available host resources at the current moment;

S2. Predict the available host resources at the future target time based on the resource usage and current task running information;

S3. Obtain all to-be-scheduled tasks in the time period between the current time and the future target time, build a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain each of the to-be-scheduled tasks. The proportion of required host resources;

S4. Calculate the task weight of the task to be scheduled according to the host resource weight set;

S5. According to the proportion of required host resources and the task weight, respectively calculate the first remaining resources after resource allocation based on the available host resources at the current time, and the available hosts based on the future target time. The second remaining resource after resource allocation;

S6. When it is determined that the second remaining resource is greater than the first remaining resource, perform task scheduling according to the time limit requirement of each task to be scheduled.

The processor-readable storage medium may be any available media or data storage device that the processor can access, including but not limited to magnetic storage (such as floppy disks, hard disks, tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor memories (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid state drive (SSD)), etc.

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (10)

1. A method for task scheduling, comprising:

acquiring the resource use condition reported by each host in the distributed system in real time; wherein, the resource use condition comprises available host resources at the current moment;

predicting available host resources at a future target moment according to the resource use condition and the current task running information;

acquiring all tasks to be scheduled in a time period between a current time and the future target time, constructing a prediction model based on a host resource weight set acquired in advance, and acquiring the required host resource occupation ratio condition of each task to be scheduled by utilizing the prediction model;

Calculating task weights of the tasks to be scheduled according to the host resource weight sets;

according to the required host resource duty ratio situation and the task weight, respectively calculating a first residual resource after resource allocation based on the available host resources at the current moment and a second residual resource after resource allocation based on the available host resources at the future target moment;

and when the second residual resources are judged to be larger than the first residual resources, task scheduling is carried out according to the time limit requirement of each task to be scheduled.

2. The task scheduling method according to claim 1, wherein predicting available host resources at a future target time according to the resource usage and current task operation information includes:

screening out tasks ending before the future target moment according to the current task running information, and recording the resource occupation condition of the tasks;

and predicting according to the resource occupation condition and the resource use condition to obtain the available host resource at the future target moment.

3. The task scheduling method according to claim 1, wherein the obtaining manner of the host resource weight set includes:

Obtaining an observed value of a historical task in a preset historical time range and the utilization rate of each host resource corresponding to the historical task, and constructing a mapping matrix of each host resource utilization rate and the historical task;

calculating by using a preset exponential smoothing algorithm based on the mapping matrix to obtain each host resource predicted value at the current moment;

and carrying out weighted average calculation based on the predicted values of the host resources and preset weight values of the host resources to obtain the host resource weight set.

4. The task scheduling method according to claim 1, wherein the calculating the task weight of the task to be scheduled according to the set of host resource weights includes:

respectively calculating each host resource weight in the host resource weight set by using a preset secondary algorithm to obtain a secondary weight of each host;

accumulating the secondary weights of the hosts to obtain a cluster maximum weight;

and calculating by combining the maximum weight of the cluster and a random result obtained by the host to obtain the task weight of the task to be scheduled.

5. The task scheduling method according to claim 1, wherein when the second remaining resources are determined to be greater than the first remaining resources, performing task scheduling according to a time limit requirement of each task to be scheduled, specifically includes:

And when judging that the first priority resource in the second residual resources is larger than the first priority resource in the first residual resources, carrying out task scheduling according to the time limit requirement of each task to be scheduled.

6. A task scheduling method according to claim 3, further comprising, before obtaining the host resource weight:

and adaptively adjusting the weight value of each host resource according to the available host resource at the future target moment.

7. The task scheduling method of any one of claims 1 to 6, wherein the resource types of the host include one or more of MEM resources, CPU resources, memory resources, IO resources, file handle resources, and file descriptor resources.

8. A task scheduling device, comprising:

the data acquisition module is used for acquiring the resource use condition reported by each host in the distributed system in real time; wherein, the resource use condition comprises available host resources at the current moment;

the resource prediction module is used for predicting available host resources at a future target moment according to the resource use condition and the current task running information;

The demand prediction module is used for acquiring all tasks to be scheduled in a time period between the current moment and the future target moment, constructing a prediction model based on a host resource weight set acquired in advance, and acquiring the required host resource occupation ratio condition of each task to be scheduled by utilizing the prediction model;

the weight calculation module is used for calculating the task weight of the task to be scheduled according to the host resource weight set;

the resource calculation module is used for respectively calculating a first residual resource after resource allocation based on the available host resources at the current moment and a second residual resource after resource allocation based on the available host resources at the future target moment according to the required host resource duty ratio situation and the task weight value;

and the task scheduling module is used for performing task scheduling according to the time limit requirement of each task to be scheduled when the second residual resource is judged to be larger than the first residual resource.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that the processor implements the steps of the task scheduling method of any one of claims 1 to 7 when executing the computer program.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the task scheduling method of any one of claims 1 to 7.