CN107797863B - Fine-grained resource matching method in cloud computing platform - Google Patents

Abstract

The invention discloses a fine-grained resource matching method in a cloud computing platform, which belongs to the field of computers, and is characterized in that task execution time and various resource demand quantities are presumed based on the same task execution logic in cloud computing load, task execution stages are divided according to task resource demands, the resource demands are matched in stages, the problems of resource fragmentation, excessive distribution and the like are avoided, computing resources are abstracted into two types of compressible and incompressible in the resource matching process, and resource distribution is compressed if necessary, so that the resource utilization rate is further improved. The invention improves the matching granularity in the aspects of time allocation and resource quantity and has lower average scheduling response time. The method can be applied to the aspects of resource management, job scheduling and the like of each cloud computing platform, avoids the problems of resource fragmentation, excessive allocation and the like, improves the utilization efficiency of computing resources in the platform, and finally improves the overall throughput rate of the cloud computing platform.

Description

A fine-grained resource matching method in cloud computing platform

technical field

The invention belongs to the computer field, relates to a resource management and job scheduling method in a cloud computing platform, and in particular relates to a fine-grained resource matching method in a cloud computing platform.

Background technique

Cloud computing is an Internet-based computing method, through which users can obtain computing resources and computing power on demand. The infrastructure of the cloud computing platform is generally composed of many computer nodes interconnected through a high-performance network, which organizes many nodes into a single image with high performance, high availability, and scalability, and provides it to users.

The resource management of the cloud computing platform is usually based on one resource (usually memory) or a bundle of two fixed amounts of computing resources is defined as a slot, and the slot is used as the unit of resource allocation. Due to the diverse demands on resources in cloud computing, the resource allocation method based on fixed units is prone to resource fragmentation, resulting in resource waste, or poor resource sharing due to excessive allocation. Poor sharing of some resources (such as CPUs) can lead to competition for resources between tasks, and execution time increases dramatically. In a data center, over 53% of lagging tasks are caused by high resource utilization caused by poor sharing, and 4%-6% of out-of-order tasks affect 37%-49% of jobs, resulting in significant job completion times extend. And some resources (such as memory) over-allocation will directly lead to task failure or server crash.

Existing cloud computing resource management platforms such as Yarn, Fuxi, Borg, and cloud computing scheduling algorithms such as Apollo, Omega, Tetris, DRF, and Carbyne allocate a fixed amount of resources based on job resource application to avoid resource fragmentation and over-allocation. Since the amount of resource requests for a job is usually designated by humans, there is a big difference between resource requests and actual usage. In addition, the resource usage of a task fluctuates greatly and does not always maintain a peak value. When a job uses the maximum resource usage of a task as the request amount, there is still a difference between the resource request amount and the actual usage amount. Therefore, when the resource management platform or scheduler allocates resources according to the amount of resource requests, resource fragments still exist. The allocation method based on resource application is difficult to achieve high resource utilization rate, which limits the resource utilization rate of the cluster to a certain extent.

Therefore, how to avoid resource fragmentation and over-allocation in cloud computing resource management and job scheduling has become an important issue in cloud computing platform research.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a fine-grained resource matching method in a cloud computing platform, which improves the matching granularity in terms of allocation time and resource amount, and has a lower average response time for scheduling, which can effectively improve the computing resources in the platform. improve the utilization efficiency of cloud computing platform and improve the overall throughput rate of cloud computing platform.

The present invention is achieved through the following technical solutions:

The invention discloses a fine-grained resource matching method in a cloud computing platform, comprising the following steps:

Step 1: The roles of the servers in the cloud computing platform are divided into two types: computing servers and management servers. The computing server is responsible for the execution of specific loads and regularly reports the resource status to the management server; the management server is responsible for the management of the entire cloud computing platform, including Allocate computing tasks to computing servers;

Step 2: The management server receives the information regularly reported by each computing server, and infers various resource requirements and durations of a certain task based on similar tasks and resource compression ratios;

Wherein, similar tasks refer to tasks in the load that have the same execution logic and the same amount of input data as the certain task;

Step 3: The management server analyzes the inferred CPU and memory space resource requirements of each progress of the task, and divides the task into multiple execution stages;

Step 4: The management server selects tasks from the to-be-scheduled set, matches task resource requirements and server available computing resources in stages, and compresses and matches resource requirements as needed;

Step 5: If the resource requirements are matched successfully, the management server assumes that the resources have been allocated, and checks whether all tasks on the computing server will be affected and cannot satisfy the constraints, and if all the task constraints are satisfied, the computing resources are allocated.

Preferably, after step 5, the following operations are further included: the management server checks whether there are enough resources left on the computing server to enter the next round of matching, and if the remaining resources meet the conditions, enter the next round of matching, that is, repeat steps 4 and 5.

Preferably, in step 1, the computing server is responsible for the execution of specific loads, and regularly reports the resource status to the management server. Specifically, the computing server periodically collects the resource usage of tasks in the running of the server, calculates the available resource information of the server, and reports it to the management server. management server;

Among them, the available resource r of a certain resource on the computing server is calculated according to formula (1):

In the formula, ri is the resource amount of the _{ith sampling, t i is} the duration of the ith sampling, T is the total calculation time of the sampling, and n is the number of samplings.

Preferably, in step 2, various resource requirements and durations of a certain task are estimated according to similar tasks and resource compression ratios, and the types of resource requirements include CPU, memory space, disk space, disk bandwidth and network bandwidth;

Among them, the disk and network bandwidth resource requirements of the task are divided into three categories according to the relative position of the task and the data. The first category: the task and the data are on the same server; Three categories: other;

The specific operations are:

The resource requirements and duration required by a task in a certain progress are calculated according to formula (2):

In the formula, α _n is the nth estimation result, β _n is the nth similar task resource information, _rc is the resource compression rate of the task information, Thr is the maximum compression rate limit coefficient, and _e is the natural base.

Preferably, the specific operations of step 3 are:

The management server traverses the CPU and memory space resource requirements of the task in each progress, and records the maximum and minimum values of these two resources respectively;

When the difference between the maximum and minimum CPU or memory space resource requirements is greater than the division threshold, and the progress of the traversal reaches the division length, the progress of the traversal is divided into an execution stage of the task;

The resource requirements of CPU, memory space and disk space in the task execution phase are the maximum values of the resource requirements of each progress in the execution phase, and the requirements of disk bandwidth and network bandwidth resources are the average values of the resource requirements of each progress in the execution phase.

Further preferably, referring to Fig. 1, the method for dividing the task execution stage specifically includes the following steps:

1) P, P _s and P _e represent task progress, stage start progress and stage end progress respectively, C _max , C _min , M _max and M _min respectively represent the maximum and minimum values of CPU and memory space resource requirements in the stage; Among them, P, P _s , and _Pe are initialized to 0; C _min and M _min are 100, and C _max and M _max are initialized to 0;

2) The task progress P and the stage end progress P _e are increased by 1, if the task progress P reaches 100%, then divide P _s to 100% as a new stage and end, otherwise continue to step 3);

3) If the current progress CPU demand C _p is greater than C _max , update the value of C _max to C _p , and go to step 5);

4) If the current progress CPU requirement C _p is less than C _min , update the value of C _min to C _p ;

5) If the current progress memory requirement M _p is greater than M _max , update the value of M _max to M _p , and enter step 7);

6) If the current progress memory requirement M _p is less than M _min , update the value of M _min to M _p ;

7) If the difference between C _max and C _min is greater than the product of the total amount of CPU resources C and the threshold Th _c or the difference between M _max and M _min is greater than the product of the total memory M and the threshold Th _m , then go to step 8, otherwise Go to step 2);

8) If the difference between _Pe and P _s is greater than the threshold Th _p , divide P _s to _Pe into new stages, update P _s to _Pe , re-initialize C _min , M _min to 100, C _max , M _max is 0; otherwise, go back to step 2).

Preferably, the specific operations of step 4 are:

First, the management server checks whether the available computing resources on the computing server meet the matching requirements;

Second, the management server sorts the to-be-scheduled sets according to resource allocation fairness and data locality policies;

Finally, the resource management server takes out the task, and obtains the guess information of the task from the guess result;

Wherein, if the acquisition of the presumed information fails, the computing resources are matched according to the application amount of the task resources; if the acquisition of the presumed information is successful, the resource requirements of the tasks are matched in sequence in stages.

Preferably, the management server abstracts the computing resources into compressible resources and incompressible resources according to resource characteristics; wherein the computing resources include CPU resources, memory resources, disk resources and network resources;

In the process of task execution, if a certain resource allocated by the task is less than the task's demand for the resource, and the task can be completed normally by extending the execution time, the resource is a compressible resource, otherwise it is an incompressible resource;

The resource compression ratio rc on the computing server is calculated according to formula ( ₃ ):

_{rc is the resource compression rate, R r is} the resource demand, and R _u is the resource allocation;

If the resource is an incompressible resource, its resource compression ratio _rc is always 0.

Preferably, in the matching process, if the demand is greater than the available resources, the matching fails;

For an incompressible resource, the computing server does not process the available resource amount of the resource;

For compressible resources, the management server calculates the maximum compression ratio of various resources in each stage of the server according to the computing server resources and load conditions, and calculates the maximum compression ratio _rmax of a certain compressible resource in each stage of the computing server according to formula (4). :

In the formula, n tasks have been matched on the server, the current matching task is the n+1th task, the total resource demand of the n+1 task is greater than the total amount of resources, and the workload completed by the ith task in the stage is w _i , Δp is the performance change caused by resource compression;

In the compressible resource matching, the actual available resource amount of the resource of the computing server is calculated according to formula (5),

a _i =n _i +r _i ×N _i (5);

In the formula, ri is the maximum compressibility ratio of the current matching stage of the resource on the server, _Ni is the total amount of the resource on the server _{, and ni is} the available resource amount of the resource obtained by collection;

If the requirements of all resources in all execution stages of the task are not matched and failed, the matching is successful.

Compared with the prior art, the present invention has the following beneficial technical effects:

The fine-grained resource matching method in the cloud computing platform disclosed by the invention abstracts the computing resources into two types: compressible and incompressible according to resource characteristics; infers the resource requirements and Duration: Divide the task into multiple execution stages according to the task resource requirements, and match the resource requirements and available resources respectively by stages and resource types; in the matching process, the resource compression method is used to slightly extend the task completion time in exchange for Overall resource utilization and load performance improvement. This method improves the matching granularity in terms of allocation time and resource amount, and has a lower average scheduling response time, so it can be applied to resource management and job scheduling of various cloud computing platforms to avoid problems such as resource fragmentation and over-allocation. Improve the utilization efficiency of computing resources in the platform, and ultimately improve the overall throughput rate of the cloud computing platform.

Description of drawings

Fig. 1 is a flow chart of a task execution phase partitioning algorithm.

Figure 2 is an architecture diagram for implementing a fine-grained resource matching method in the Yarn platform.

Detailed ways

The present invention will be further described in detail below in conjunction with specific embodiments, which are to explain rather than limit the present invention.

The fine-grained resource matching method in a cloud computing platform disclosed by the invention predicts task resource requirements and duration according to similar tasks; abstracts resources into two types: compressible and incompressible according to resource characteristics; divides tasks into multiple execution stages and Match the requirements of various types of resources in each execution stage in turn. The method can improve resource utilization and performance by improving resource allocation time granularity and resource amount granularity when matching resources in cloud computing platform resource management and job scheduling, avoiding resource fragmentation and over-allocation in resource management.

The fine-grained resource matching method in the disclosed cloud computing platform of the present invention includes the following steps:

Step 1: The roles of servers in the cloud computing platform are divided into two types: computing servers and management servers: the computing server is responsible for the execution of specific loads and regularly reports the resource status to the management server; the management server is responsible for various management tasks of the entire cloud platform, including Allocate computing tasks to computing servers.

The computing server periodically collects the resource usage of tasks in the running of the server, calculates the available resource information of the server, and reports it to the management server.

The amount of resources available for a resource on a computing server is calculated by the following formula:

Among them, ri is the resource amount of the _{ith sampling, t i is} the duration of the ith sampling, T is the total calculation time of the sampling, and n is the number of sampling times.

Step 2: The management server receives the information periodically reported by each computing server, and infers various resource requirements and durations of tasks according to similar tasks and resource compression ratios. Here, similar tasks refer to tasks in the workload that have the same execution logic and the same amount of input data as the task. The disk and network bandwidth resource requirements of tasks are divided into three categories according to the relative positions of tasks and data: tasks and data are in the same server, in the same rack, and other three categories.

A computing resource requirement and time required by a task in a certain progress are calculated according to the following formula:

Among them, α _n is the nth estimation result, β _n is the nth similar task resource information, _rc is the resource compression rate of the task information, Thr is the maximum compression rate limit coefficient, and _e is the natural base.

Step 3: The management server analyzes the CPU and memory space resource requirements of each progress of the task, and divides the task into multiple execution stages.

The management server traverses the CPU and memory space resource requirements of the task in each progress, and records the maximum and minimum values of these two resources respectively. When the difference between the maximum value and the minimum value of CPU or memory space resource requirements is greater than a certain threshold, and the progress of the traversal reaches a certain length, the progress of the traversal is divided into one execution stage of the task. In the task execution phase, the resource requirements such as CPU, memory space, and disk space are the maximum values of the resource requirements of each progress in the phase. The resource requirements such as memory bandwidth, disk bandwidth, and network bandwidth are the average values of the resource requirements of each progress in the phase. .

Step 4: When the available computing resources on the computing server reach a certain amount, the management server selects tasks from the to-be-scheduled set, and matches the task resource requirements and the available computing resources of the server in stages.

The management server first checks whether the available resources on the computing server meet the matching requirements. Second, the management server sorts the to-be-scheduled sets according to policies such as resource allocation fairness and data locality. Again, the resource management server takes out the task and tries to obtain the task's speculation information from the speculation result. If the presumed information acquisition fails, the computing resources are matched according to the requested amount of task resources. If the speculative information is obtained successfully, the resource requirements of the tasks are matched in sequence in stages.

The management server abstracts computing resources such as CPU, memory, disk, and network into two categories: compressible and incompressible, according to resource characteristics. During task execution, if a certain resource allocated by the task is less than the task's demand for the resource, and the task can be completed normally by extending the execution time, the resource is a compressible resource, otherwise it is an incompressible resource.

The resource compression ratio on the calculation server is calculated according to the following formula:

Among them, _{rc is the resource compression ratio, R r is} the resource demand, and R _u is the resource allocation. If the resource is incompressible, its compression ratio _rc is always 0.

During the matching process, if the demand is greater than the available resources, the matching fails. For incompressible resources, the calculation server does not process the available resources of the resource; for compressible resources, the management server calculates the maximum compression ratio of various resources of the server at each stage according to the calculation server resources and load conditions.

The maximum compression ratio of a compressible resource of a computing server in a stage is calculated by the following formula:

Among them, n tasks have been matched on the server, the current matching task is the n+1th task, the total resource demand of the n+1 task is greater than the total amount of resources, and the workload completed by the ith task in the stage is w _i , △ p is the performance change caused by resource compression.

In the compressible resource matching, the actual available resource amount of the resource on the computing server is calculated according to the formula a _i =n _i +r _i ×N _i ; where _ri is the maximum compressibility ratio of the current matching stage of the resource on the server , _{Ni is the total amount of this kind of resource on the server, and ni is} the available resource amount of this kind of resource obtained by collection. If the requirements of all resources in all execution phases of the task are not matched and failed, the matching is successful.

Specifically, the matching method performs resource matching according to the following steps:

step1), check the amount of available resources of the server, if the amount of available resources is less than the scheduling threshold, end the scheduling;

step 2), sort the set of tasks to be scheduled according to strategies such as resource allocation fairness and data locality.

step3), select a task from the set of tasks to be scheduled, and obtain the resource requirements and duration of each execution stage of the task; if the acquisition is successful, go to step5), otherwise match according to the application amount, and go to step7) after passing; if not, go to step5) );

step4), take the task resource application amount as the demand to match resources, and enter step7) if the match is successful;

step5), s is the task execution stage, n is the corresponding available resource stage of the server, compare the resource demand si of the i-th resource in this execution stage with the resource availability a _i of the available resource stage in turn, if the resource in the _i -th is unavailable If the resource is compressed, the value of a _i is the corresponding resource value _ni in the available resource phase; if the resource is a compressible resource, the value of a _i is calculated according to the formula a _i =n _i +r _i ×N _i , where _ri is the server The maximum compressibility ratio in the corresponding available resource phase of the resource on the server, and _Ni is the total amount of the resource on the server. If the resource requirement is greater than the available resource, the match fails, and go to step3), otherwise go to step6);

step6), if s is the last execution stage of the task, the match is successful and go to step 7), otherwise step5) is repeated to match the next stage.

step7), check whether the matching affects the execution of other tasks on the server, if the check is passed, the matching is completed, otherwise go back to step3).

Step 5: If the resource matching is successful, the management server assumes that the resource has been allocated, and checks that all tasks on the computing server will be affected and cannot satisfy the constraints. If all task constraints can be satisfied, compute resources are allocated.

Step 6: The management server checks whether there are enough resources left on the computing server to enter the next round of matching, and if the remaining resources meet the conditions, enter the next round of matching.

According to the above method, the fine-grained matching of the computing resources of the cloud computing platform can be realized.

A specific application example of adopting the inventive method is provided below:

Referring to FIG. 2 , this example is combined with the open source cloud computing platform Yarn. It should be noted that the method of the present invention is not only applicable to the open source cloud computing platform Yarn, but also applicable to other application platforms that meet the conditions.

Step 1: When the Application Master registers with the Resource Manager, it provides the MD5 value of the application code and parameters and the input data volume information. When the Application Master applies to the Resource Manager for computing resources required by a task, the Application Master and the task type information are used to identify the task.

Step 2: Node Manager obtains the running task information and remaining resource information on the node by analyzing the information in the Linux Proc folder.

The amount of available resources of a certain resource of Node Manager is calculated according to the following formula:

Among them, ri is the resource amount of the _{ith sampling, t i is} the duration of the ith sampling, T is the total time of sampling, and n is the number of sampling times.

The Node Manager reports the collected resource information and the available resources through analysis and calculation to the Resource Manager by heartbeat.

Step 3: The Resource Manager receives the request and report of the Application Master and the Node Manager, and sends the application registration and resource application information of the Application Master to the Scheduler for processing, and the resource information reported by the NodeManager to the Estimator for processing.

Step 4: The Estimator identifies similar tasks according to the application registration information and resource request information of the Application Master. Similar tasks refer to tasks in the workload that have the same execution logic and the same amount of input data as the task.

Estimator processes Node Manager report information and infers the resource requirements and duration of tasks. The disk and network bandwidth resource requirements of tasks are divided into three categories according to the relative positions of tasks and data: tasks and data are in the same server, in the same rack, and other three categories.

A computing resource requirement and time required by a task in a certain progress are calculated according to the following formula:

Among them, α _n is the nth estimation result, β _n is the nth similar task resource information, _rc is the resource compression rate of the similar task information, Thr is the maximum compression rate limit coefficient, and _e is the natural base.

Estimator analyzes the task CPU and memory space requirement information, and divides the task into multiple execution stages. Estimator traverses the CPU and memory space resource requirements of tasks in each progress, and records the maximum and minimum values of these two resources respectively. When the difference between the maximum and minimum CPU or memory space resource requirements is greater than the division threshold, and the traversal progress reaches the division length, the traversal progress is divided into one execution stage of the task. Here, the division threshold is 20% of the total resources by default, and the division length is 5% of the total length of the task progress by default. The division threshold and division length should be adjusted according to the specific platform and load. In the task execution phase, the resource requirements such as CPU, memory space, and disk space are the maximum values of the resource requirements in the phase, and the resource requirements such as memory bandwidth, disk bandwidth, and network bandwidth are the average values of the resource requirements in the phase.

Step 5: The Scheduler matches the resource request of the Application Master with the available computing resources on the Node Manager.

The Scheduler adds the resource application of the Application Master to the to-be-scheduled set.

The Scheduler checks whether the available computing resources on the Node Manager meet the matching requirements.

Scheduler sorts the to-be-scheduled set according to policies such as resource allocation fairness and data locality

The Scheduler takes out the tasks to be scheduled from the sorted set to be scheduled, and obtains the estimated task resource requirements and duration information from the Estimator. If the information acquisition fails, it will be matched according to the resource application amount. If the information is obtained successfully, all resource requirements of all execution phases of the task are matched in turn.

In matching, computing resources such as CPU, memory, disk, and network are abstracted into two categories, compressible and incompressible, according to resource characteristics. During task execution, if a certain resource allocated by the task is less than the task's demand for the resource, and the task can be completed normally by extending the execution time, the resource is a compressible resource, otherwise it is an incompressible resource.

The degree of Node Manager resource compression is calculated by the following formula:

Among them, _{rc is the resource compression ratio, R r is} the resource demand, and R _u is the resource allocation. If the resource is incompressible, its compression ratio _rc is always 0.

The maximum compression ratio of a compressible resource of Node Manager in a certain stage is calculated according to the following formula:

Among them, there are n matched tasks on the Node Manager, the current matching task is the n+1th task, the total resource demand of the n+1 task is greater than the total amount of resources, and the workload of the i-th task completed in the stage is w _i , Δp is the performance change caused by resource compression.

During the matching process, if the demand for a certain resource of the task is greater than the corresponding available resource on the Node Manager, the matching fails. The amount of available resources of incompressible resources on Node Manager is the collection amount; the amount of available resources of compressible resources is calculated according to the formula a _i =n _i +r _i ×N _i , where ri _i is the amount of resources on the server in this matching stage The maximum compressible ratio, _{Ni is the total amount of this kind of resource on the server, and ni is} the available resource amount of this kind of resource obtained by collection. If the requirements of all resources in all execution stages of the task are not matched and failed, the matching is successful.

Step 6: After the matching is successful, the Scheduler will check whether the constraints of all executing tasks on the Node Manager can be satisfied if the matching decision takes effect. If the check passes, the Scheduler allocates computing resources to the Application Master that made the resource request.

After allocating resources, the Scheduler will check the remaining resources of the server where the Node Manager is located and decide whether to enter the next round of scheduling.

Step 7: The Application Master that obtains the resource allocation communicates with the Node Manager where the computing resource is located, and starts the corresponding task.

The actual test results show that the resource matching results obtained by this method can avoid resource fragmentation and over-allocation, improve the resource utilization efficiency of the cloud computing platform, and ultimately improve the overall throughput rate of the cloud computing platform.

It can be seen from the above embodiments that the present invention can be used in cloud computing platform resource management and job scheduling. The invention infers task resource requirements and duration based on similar tasks, divides tasks into multiple execution stages based on resource requirements, matches task resource requirements and server computing resources respectively by stage and resource characteristics, and extends a single task within an acceptable range if necessary In exchange for higher resource utilization and the number of tasks in parallel, the overall performance is ultimately improved.

The present invention can not only be used for resource management and job scheduling in cloud computing platforms, but also can be used for reference and improvement in cluster-based resource management platforms.

Claims (8)

1. a fine-grained resource matching method in a cloud computing platform, is characterized in that, comprises the following steps: Step 1: The roles of the servers in the cloud computing platform are divided into two types: computing servers and management servers. The computing server is responsible for the execution of specific loads and regularly reports the resource status to the management server; the management server is responsible for the management of the entire cloud computing platform, including Allocate computing tasks to computing servers; Step 2: The management server receives the information regularly reported by each computing server, and infers various resource requirements and durations of a certain task based on similar tasks and resource compression ratios; Wherein, similar tasks refer to tasks in the load that have the same execution logic and the same amount of input data as the certain task; Step 3: The management server analyzes the inferred CPU and memory space resource requirements of each progress of the task, and divides the task into multiple execution stages; Step 4: The management server selects tasks from the to-be-scheduled set, matches task resource requirements and server available computing resources in stages, and compresses and matches resource requirements as needed. The specific operations are as follows: First, the management server checks whether the available computing resources on the computing server meet the matching requirements; Second, the management server sorts the to-be-scheduled sets according to resource allocation fairness and data locality policies; Finally, the resource management server takes out the task, and obtains the guess information of the task from the guess result; Among them, if the acquisition of the presumed information fails, the computing resources are matched according to the application amount of the task resource; if the acquisition of the presumed information is successful, the resource requirements of the task are matched in sequence in stages; Step 5: If the resource requirements are matched successfully, the management server assumes that the resources have been allocated, and checks whether all tasks on the computing server will be affected and cannot satisfy the constraints, and if all the task constraints are satisfied, the computing resources are allocated. 2. The fine-grained resource matching method in the cloud computing platform according to claim 1, is characterized in that, after step 5, also comprises the following operations: the management server checks whether sufficient resources are left on this computing server to enter the next round of matching, if the remaining If the resource meets the conditions, the next round of matching is entered, that is, steps 4 and 5 are repeated. 3. The fine-grained resource matching method in the cloud computing platform according to claim 1, is characterized in that, in step 1, the computing server is responsible for the execution of the specific load, and regularly reports the resource status to the management server, specifically refers to: the computing server Regularly collect the resource usage of tasks in the running of the server, calculate the available resource information of the server and report it to the management server; Among them, the available resource r of a certain resource on the computing server is calculated according to formula (1):

In the formula, ri is the resource amount of the _{ith sampling, t i is} the duration of the ith sampling, T is the total calculation time of the sampling, and n is the number of samplings. 4. The fine-grained resource matching method in cloud computing platform according to claim 1, is characterized in that, in step 2, according to similar task and resource compression ratio, infer various resource requirements and duration of a certain task, resource requirement type Including CPU, memory space, disk space, disk bandwidth and network bandwidth; Among them, the disk and network bandwidth resource requirements of the task are divided into three categories according to the relative position of the task and the data. The first category: the task and the data are on the same server; Three categories: other; The specific operations are: The resource requirements and duration required by a task in a certain progress are calculated according to formula (2):

In the formula, α _n is the nth estimation result, β _n is the nth similar task resource information, _rc is the resource compression rate of the task information, Thr is the maximum compression rate limit coefficient, and _e is the natural base. 5. The fine-grained resource matching method in the cloud computing platform according to claim 1, is characterized in that, the specific operation of step 3 is: the management server traverses the CPU and memory space resource requirements of each progress of the task, and records these two respectively. The maximum and minimum value of each resource; When the difference between the maximum and minimum CPU or memory space resource requirements is greater than the division threshold, and the progress of the traversal reaches the division length, the progress of the traversal is divided into an execution stage of the task; The resource requirements of CPU, memory space and disk space in the task execution phase are the maximum values of the resource requirements of each progress in the execution phase, and the requirements of disk bandwidth and network bandwidth resources are the average values of the resource requirements of each progress in the execution phase. 6. The fine-grained resource matching method in the cloud computing platform according to claim 5, wherein the division method of the task execution stage specifically comprises the following steps: 1) P, P _s and P _e represent task progress, stage start progress and stage end progress respectively, C _max , C _min , M _max and M _min respectively represent the maximum and minimum values of CPU and memory space resource requirements in the stage; Among them, P, P _s , and _Pe are initialized to 0; C _min and M _min are 100, and C _max and M _max are initialized to 0; 2) The task progress P and the stage end progress P _e are increased by 1, if the task progress P reaches 100%, then divide P _s to 100% as a new stage and end, otherwise continue to step 3); 3) If the current progress CPU demand C _p is greater than C _max , update the value of C _max to C _p , and go to step 5); 4) If the current progress CPU requirement C _p is less than C _min , update the value of C _min to C _p ; 5) If the current progress memory requirement M _p is greater than M _max , update the value of M _max to M _p , and enter step 7); 6) If the current progress memory requirement M _p is less than M _min , update the value of M _min to M _p ; 7) If the difference between C _max and C _min is greater than the product of the total amount of CPU resources C and the threshold Th _c or the difference between M _max and M _min is greater than the product of the total memory M and the threshold Th _m , then go to step 8, otherwise Go to step 2); 8) If the difference between _Pe and P _s is greater than the threshold Th _p , divide P _s to _Pe into new stages, update P _s to _Pe , re-initialize C _min , M _min to 100, C _max , M _max is 0; otherwise, go back to step 2). 7. The fine-grained resource matching method in a cloud computing platform according to claim 1, wherein the management server abstracts the computing resources into compressible resources and incompressible resources according to resource characteristics; wherein, the computing resources include CPU resources, memory resources, disk resources and network resources; In the process of task execution, if a certain resource allocated by the task is less than the task's demand for the resource, and the task can be completed normally by extending the execution time, the resource is a compressible resource, otherwise it is an incompressible resource; The resource compression ratio rc on the computing server is calculated according to formula ( ₃ ):

_{rc is the resource compression rate, R r is} the resource demand, and R _u is the resource allocation; If the resource is an incompressible resource, its resource compression ratio _rc is always 0. 8. The fine-grained resource matching method in a cloud computing platform according to claim 7, wherein in the matching process, if the demand is greater than the available resource, the matching fails; For incompressible resources, the computing server does not process the available resources of the resource; For compressible resources, the management server calculates the maximum compression ratio of various resources in each stage of the server according to the computing server resources and load conditions, and calculates the maximum compression ratio _rmax of a certain compressible resource in each stage of the computing server according to formula (4). :

In the formula, n tasks have been matched on the server, the current matching task is the n+1th task, the total resource demand of the n+1 task is greater than the total amount of resources, and the workload completed by the ith task in the stage is w _i , Δp is the performance change caused by resource compression; In the compressible resource matching, the actual available resource amount of the resource of the computing server is calculated according to formula (5), a _i =n _i +r _i ×N _i (5); In the formula, ri is the maximum compressibility ratio of the current matching stage of the resource on the server, _Ni is the total amount of the resource on the server _{, and ni is} the available resource amount of the resource obtained by collection; If the requirements of all resources in all execution stages of the task are not matched and failed, the matching is successful.

Images