CN112363821A - Computing resource scheduling method and device and computer equipment - Google Patents
Computing resource scheduling method and device and computer equipment Download PDFInfo
- Publication number
- CN112363821A CN112363821A CN202110032737.1A CN202110032737A CN112363821A CN 112363821 A CN112363821 A CN 112363821A CN 202110032737 A CN202110032737 A CN 202110032737A CN 112363821 A CN112363821 A CN 112363821A
- Authority
- CN
- China
- Prior art keywords
- computing
- task
- computing task
- waiting
- idle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the invention provides a computing resource scheduling method, a computing resource scheduling device and computer equipment, and relates to the field of computers. The computing resource scheduling method comprises the following steps: if the computing task is in a waiting queue, splitting the computing task into a plurality of subtasks based on the minimum scheduling granularity in the super computer; and respectively processing each subtask by using the idle computing resource with the minimum scheduling granularity. By splitting the computing task into a plurality of subtasks, because the amount of computing resources required by each subtask is reduced relative to that of a large-example task, the phenomenon of starvation caused by long-term waiting of resources by a single large-example is effectively avoided; meanwhile, the 'fragment resources' which cannot provide support due to the large arithmetic waiting can be fully utilized. Therefore, the utilization rate of computing resources is greatly improved, and the computing time is shortened.
Description
Technical Field
The invention relates to the field of computers, in particular to a computing resource scheduling method, a computing resource scheduling device and computer equipment.
Background
At present, a mainstream calculation-centered scheduling algorithm in a super computer scheduling scheme often has a large amount of data transmission, and particularly, when a data-intensive calculation task is processed, the performance overhead is high. Moreover, because the size of the computing resources required by each task is different, the task complexity is different, and the computing time is different, under the condition that part of the computing resources are idle to a certain extent according to the traditional super computing queuing waiting mechanism, in order to meet the resource requirements of the large-computation-quantity task, the computing resources in operation need to be continuously waited to be released, so that the total amount of the idle computing resources can meet the large-computation-quantity task, the large task is in a waiting state for a long time, and the current idle computing resources cannot be fully utilized.
Therefore, the existing method for scheduling the computing resources of the supercomputer has the problems that a large task cannot be scheduled for a long time and idle computing resources cannot meet the scheduling condition of the large task and are idle for a long time.
Disclosure of Invention
In order to solve the technical problem, the invention provides a method, a device and a computer device for scheduling computing resources, wherein the specific scheme is as follows:
in a first aspect, an embodiment of the present disclosure provides a method for scheduling computing resources, where the method includes:
if the computing task is in a waiting queue, splitting the computing task into a plurality of subtasks based on the minimum scheduling granularity in the super computer;
and respectively processing each subtask by using the idle computing resource with the minimum scheduling granularity.
According to a specific embodiment of the present disclosure, before the step of splitting the computation task into a plurality of sub-tasks based on a minimum scheduling granularity in the supercomputer if the computation task is in a waiting queue, the method further includes:
when the computing task is received, judging whether the amount of computing resources required by the computing task is larger than the total amount of the idle computing resources;
if the amount of the computing resources required by the computing task is larger than the total amount of the idle computing resources, adding the computing task into the waiting queue according to historical data of the computing task, wherein the historical data of the computing task comprises estimated processing time consumption and waiting time length of the computing task, or the priority of a source user of the computing task;
and if the amount of the computing resources required by the computing task is less than the total amount of the idle computing resources, directly calling the idle computing resources to process the computing task.
According to a specific embodiment of the present disclosure, the step of splitting the computing task into a plurality of subtasks further includes:
dividing all subtasks into priority subtasks and waiting subtasks;
and calling the idle computing resources to execute the priority subtasks, and adding the waiting subtasks into the waiting queue.
According to a specific embodiment of the present disclosure, the step of adding the computing task to the wait queue further includes:
acquiring the estimated processing time consumption and the waiting time of the computing task;
calculating the response ratio of the calculation task according to the estimated processing time and the waiting time;
and adding the computing tasks to corresponding positions in the waiting queue according to the response ratios of the computing tasks, wherein all the computing tasks in the waiting queue are sorted according to the response ratios, and the computing task with the smallest response ratio is positioned at the head of the waiting queue.
According to a specific embodiment of the present disclosure, the step of adding the computing task to the wait queue further includes:
acquiring first-class historical use data of a source user corresponding to the computing task, wherein the first-class historical use data comprises the priority of the user;
and adding the computing tasks to corresponding positions in the waiting queue according to the first type of historical use data, wherein all the computing tasks in the waiting queue are sorted according to the priority of the user, and the computing task with the highest priority of the user is positioned at the head of the waiting queue.
According to a specific embodiment of the present disclosure, the step of processing each sub-task by using the idle computing resource with the minimum scheduling granularity includes:
acquiring second type historical use data of a source user corresponding to the computing task, wherein the second type historical use data comprises a historical task use resource type of the source user;
and calling the idle computing resources matched with the historical task use resource types of the source users to execute the subtasks.
According to a specific embodiment of the present disclosure, the step of obtaining the estimated processing time consumption of the computing task further includes:
dividing the completed historical computing tasks into a plurality of communities, wherein each community corresponds to a reference processing time;
determining a target community corresponding to the current computing task according to similarity clustering, wherein the similarity between the historical computing task in the target community and the current computing task is highest;
and determining the estimated processing time consumption of the current computing task according to the reference processing time consumption of the target community.
In a second aspect, an embodiment of the present disclosure further provides an apparatus for scheduling computing resources, where the apparatus includes:
the splitting module is used for splitting the computing task into a plurality of subtasks based on the minimum scheduling granularity in the super computer if the computing task is in the waiting queue;
the calling module is used for processing each subtask by using the idle computing resource with the minimum scheduling granularity;
the computing resource scheduling apparatus further includes:
the judging module is used for judging whether the amount of the computing resources required by the computing task is larger than the total amount of the idle computing resources or not when the computing task is received;
the adding module is used for adding the computing task into the waiting queue according to historical data of the computing task if the computing resource amount required by the computing task is larger than the total amount of the idle computing resources, wherein the historical data of the computing task comprises estimated processing time consumption and waiting time length of the computing task or the priority of a source user of the computing task;
and the processing module is used for directly calling the idle computing resources to process the computing task if the computing resource amount required by the computing task is less than the total amount of the idle computing resources.
In a third aspect, an embodiment of the present disclosure further provides a computer device, where the computer device includes a display terminal, a memory and a processor, where the memory is connected to the processor, the memory is used to store a computer program, and the processor runs the computer program to enable the computer device to execute the method for scheduling computing resources in the first aspect.
In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores a computer program, where the computer program, when running on a processor, executes the method for scheduling computing resources in the first aspect.
According to the computing resource scheduling method, the computing resource scheduling device and the computer equipment, when the computing task is in the waiting queue, the current big computing example computing task is divided into a plurality of subtasks based on the total amount of idle computing resources, and then the idle computing resources are called to execute the plurality of subtasks.
Because the amount of computing resources required by each subtask is far less than the total amount of the current idle computing resources, the idle computing resources can be directly called to execute a plurality of subtasks by splitting the large-scale computation task into the subtasks. On one hand, the calculation resource amount required by each large-arithmetic-example calculation task is effectively reduced, and a single large-arithmetic-example is prevented from being in a waiting queue for a long time. On the other hand, because the large-computation-example computing task is in a waiting processing state, the 'fragment resource' which cannot provide computing support for other computing tasks can be fully utilized, the utilization rate of the computing resource is greatly improved, and the computing time is shortened.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a schematic flowchart of a computing resource scheduling method according to this embodiment;
fig. 2 is an illustration of a computing resource utilization situation related to the computing resource scheduling method provided in this embodiment;
fig. 3 is a schematic diagram of the amount of computing resources required by a split computing task according to the computing resource scheduling method provided in this embodiment;
fig. 4 is a schematic diagram of a computing resource utilization condition after a priority sub-task is executed according to the computing resource scheduling method provided in this embodiment;
fig. 5 is a schematic diagram of an amount of computing resources required for waiting for a subtask after a priority subtask is executed according to the method for scheduling computing resources provided in this embodiment;
fig. 6 is a schematic diagram of a queuing process of the computing resource scheduling method according to this embodiment;
fig. 7 is a block diagram of a computing resource scheduling apparatus according to this embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. indicate an orientation or a positional relationship based on that shown in the drawings or that the product of the present invention is used as it is, this is only for convenience of description and simplification of the description, and it does not indicate or imply that the device or the element referred to must have a specific orientation, be constructed in a specific orientation, and be operated, and thus should not be construed as limiting the present invention.
Furthermore, the appearances of the terms "first," "second," and the like, if any, are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
It should be noted that the features of the embodiments of the present invention may be combined with each other without conflict.
Referring to fig. 1, a flowchart of a computing resource scheduling method is provided in this embodiment. As shown in fig. 1, the method for scheduling computing resources mainly includes the following steps:
s101, if the computing task is in a waiting queue, the computing task is divided into a plurality of subtasks based on the minimum scheduling granularity in the super computer.
In the current mainstream scheduling method of the supercomputer, the parallel granularity of the supercomputer mostly takes service as the minimum processing granularity, that is, independent parallel computation can be realized among single tasks submitted by different users. When large examples of super computing platforms come, the problems of "resource fragmentation" and "starvation" easily occur.
When the amount of computing resources required for the large-scale computation task is greater than the total amount of idle computing resources, the large-scale computation task is placed in a wait queue. In order to avoid that the large-computation-example computing task cannot be processed for a long time, the large-computation-example computing task is divided into a plurality of sub-tasks based on the minimum scheduling granularity in the super computer, and the total amount of computing resources required by part of the sub-tasks is smaller than the total amount of the currently idle computing resources. The subtasks are used as the minimum processing granularity of the super computing center to carry out parallel processing, so that the current idle computing resources of the super computing platform are fully utilized, and the hunger phenomenon of the large-example computing tasks in a high-concurrency scene is effectively avoided.
And S102, processing each subtask by using the idle computing resource with the minimum scheduling granularity.
Because the amount of computing resources required by each subtask is reduced relative to the amount of computing resources required by a large-scale computation task, by calling the idle computing resources with the minimum scheduling granularity on the super computing platform to process a plurality of subtasks, the 'fragment resources' which cannot provide processing support for the large-scale computation task before can be fully utilized, the utilization rate of the computing resources is greatly improved, and the computing time of the tasks is reduced.
The computing resource scheduling method provided by the embodiment of the invention further refines the parallel scheduling granularity of the large-scale computation task in the waiting queue, divides the single large-scale computation task into sub-tasks with equal data size according to the data fragmentation method, controls the parallel dimension in the single computation task, effectively utilizes the idle computing resource, and greatly reduces the time length of the large-scale computation task in the waiting queue.
On the basis of the foregoing embodiment, according to an implementation manner in an embodiment of the present disclosure, in step S101, if the computing task is in the waiting queue, before splitting the computing task into a plurality of sub-tasks based on the minimum scheduling granularity in the supercomputer, the method may further include:
when a computing task is received, judging whether the amount of computing resources required by the computing task is larger than the total amount of idle computing resources;
if the amount of the computing resources required by the computing task is larger than the total amount of the idle computing resources, adding the computing task into the waiting queue according to historical data of the computing task, wherein the historical data of the computing task comprises estimated processing time consumption and waiting time length of the computing task, or the priority of a source user of the computing task;
and if the amount of the computing resources required by the computing task is less than the total amount of the idle computing resources, directly calling the idle computing resources to process the computing task.
When the super computing platform receives the computing task, whether the amount of computing resources required by the computing task is larger than the amount of idle computing resources is judged. And if the total amount of the idle computing resources can not meet the requirements of the computing task, putting the computing task into a waiting queue. Until the total amount of idle computing resources is greater than the total amount of computing resources required by the current computing task, the computing tasks in the waiting queue have an opportunity to be processed by means of a scheduling algorithm. If the total amount of the computing resources required by the current computing task is less than the total amount of the idle computing resources, the computing task is not put into a waiting queue any more, and the idle computing resources can be directly called to process the computing task.
Specifically, the estimated processing time of the calculation task and the waiting time of the calculation task can be obtained in advance; alternatively, when a user has a new computing task coming, the system prioritizes analyzing the user's historical data. At any moment when a task arrives, the system updates each task response ratio (task response ratio = task estimated running time/time that the task has been waited) according to the task estimated time state, the residual resource condition and the calculation resource and estimated time required by the current task in the waiting queue, inserts the current task into the waiting queue, and reorders each task from small to large according to the response ratio. On the basis of the foregoing embodiment, according to an implementation manner in the embodiment of the present disclosure, specifically, the step of splitting the computing task into a plurality of subtasks further includes:
dividing all subtasks into priority subtasks and waiting subtasks, calling idle computing resources to execute the priority subtasks, and adding the waiting subtasks into a waiting queue.
The big arithmetic example computing task is divided into a plurality of subtasks with equal data size, the required computing resource amount of each subtask is also equal, and furthermore, the divided subtasks are divided into a priority subtask and a waiting subtask.
Optionally, the sub-tasks are sorted according to the deadline by using the time length of the sub-tasks from the deadline, and the sub-tasks with higher priority are sorted according to the shorter the task deadline.
And the waiting subtasks are put into a waiting queue, and the waiting subtasks are executed by calling the idle computing resources until the next time when the total amount of the idle computing resources is more than the total amount of the computing resources required by the waiting subtasks. The method for scheduling the computing resources takes the subtasks as the minimum scheduling granularity can greatly avoid the phenomenon that the large-scale computation tasks are continuously positioned in the waiting queue, and improve the utilization rate of the computing resources.
Fig. 2 is a schematic diagram of the current utilization of the computing resources at time t, and fig. 3 is a schematic diagram of the amount of computing resources required by the split computing task. In a specific operation, if a large-example computing task request comes at time t, the amount of computing resources required for completely executing the computing task is 12, and assuming that the amount of computing resources is quantized in units of blocks, the amount of idle computing resources under the super computing platform is 11 at this time. According to the traditional task-centric scheduling strategy, the algorithm is placed in a waiting queue and can be scheduled only when the platform idle computing resources are not less than 12, so that the computing task is divided into 12 subtasks.
Corresponding to the scenario shown in fig. 2, as shown in fig. 4 and 5, fig. 4 is a schematic diagram of the utilization of the computing resources after the priority subtask is executed, and fig. 5 is a schematic diagram of the amount of computing resources required for waiting for the subtask after the priority subtask is executed. The scheduling scheme provided by the invention can divide a large task into 12 independent subtasks, 11 priority subtasks are processed by the current idle computing resources at the time t, so that all the current computing resources are reasonably utilized, and only one waiting subtask in the divided computing tasks is added into the waiting queue to wait for the next idle computing resource meeting the computing requirement to perform computing processing.
In addition, the super computing platform is preferably a heterogeneous grid structure, and comprises a plurality of super computing centers. By calling computing resources in different regions to execute a plurality of subtasks, the resource utilization rate of the supercomputer in a high concurrency scene is improved to a great extent.
By refining the parallel granularity, the computing task is divided into subtasks with uniform data size of the needed computing resource, the data size of the computing task is unified, the idle resource under a super computing platform is fully utilized, and the starvation phenomenon of the large task under a high concurrency scene is effectively avoided.
According to an implementation manner in the embodiment of the present disclosure, the step of adding the computation task to the wait queue further includes:
acquiring estimated processing time consumption and waiting time of a calculation task;
calculating the response ratio of the calculation task according to the estimated processing time and the waiting time;
and adding the computing tasks to corresponding positions in the waiting queue according to the response ratios of the computing tasks, wherein all the computing tasks in the waiting queue are sorted according to the response ratios, and the computing task with the smallest response ratio is positioned at the head of the waiting queue.
As shown in fig. 6, in the case of multiple computing tasks, the tasks meeting the scheduling conditions are directly pushed to the computing center to complete specific logic computation, while those tasks that need to be queued to wait are pushed to the waiting queue, and when there is free resource release under the platform, the tasks in the queue head are automatically pulled to complete computation.
The queuing position of the computing task in the waiting queue can be determined according to the response ratio of the computing task, wherein the response ratio refers to the corresponding indication value of the computing task. Specifically, the estimated processing time consumption of the calculation task and the waiting time of the calculation task can be obtained in advance, and the value obtained by comparing the estimated processing time consumption and the waiting time of the calculation task is used as the response ratio of the calculation task through the interface. The smaller the response ratio, the faster the computing task will be responded to, and the larger the response ratio, the later the computing task will be relatively responded to. Thus, when queuing according to the response ratio, the calculation task with the minimum response ratio is positioned at the head of the waiting queue.
When a new calculation task arrives at a certain user, the system can give priority to analysis of historical data of the user, firstly, the number of CPUs (central processing units), the number of GPUs (graphic processing units), the number of memories and the number of License-order task quantities required by the task are abstracted into a characteristic vector W, the characteristic vector is added into an existing scattered point set S to perform local clustering adjustment, the type of the task and the corresponding processing time are judged, and the time obtained in the step is called estimated time. Then, according to the user's historical behavior (mainly, the score on the previous calculation task), the calculation resources (resource allocation) whose historical score is 3 or more are recommended for the user. The super computing platform will sort the tasks in the scheduling sequence based on the estimated time.
At any moment when a task arrives, the system updates the response ratio of each task ∈ (task response ratio = task estimated running time/time that the task has been waited) according to the estimated time state, the residual resource condition and the computing resource required by the current task in the waiting queue and the estimated time, inserts the current task into the waiting queue, and reorders each task from small to large according to the response ratio. For example, if there are 3 waiting tasks and their response ratio information in the current waiting queue as follows, a (3/4), B (5/3), and C (10/1), and if the response ratio information of the incoming task is D (1/-) at the current time, the scheduling order queued according to the response ratio in the task queue after the next time point is a (3/5), D (1, 1), B (5/4), and C (10/2). In the case of resource idleness, the system will pull the specific logic of the waiting task from the queue head directly.
In addition, the priority of the calculation task determined only according to the estimated processing time consumption can be high, all the calculation tasks in the waiting queue are ordered according to the processing time consumption, the calculation task with the shortest estimated processing time consumption is located at the head of the waiting queue and is preferentially processed by idle calculation resources. The calculation tasks are subjected to priority ranking according to the difference of estimated processing time consumption, so that the average waiting time can be reduced, and the satisfaction degree of a user is improved.
According to another implementation manner in the embodiment of the present disclosure, the step of adding the computation task to the wait queue further includes:
acquiring first-class historical use data of a source user corresponding to the computing task, wherein the first-class historical use data comprises the priority of the user;
and adding the computing tasks to corresponding positions in the waiting queue according to the first type of historical use data, wherein all the computing tasks in the waiting queue are sorted according to the priority of the user, and the computing task with the highest priority of the user is positioned at the head of the waiting queue.
In order to avoid the phenomena of congestion of the computing tasks in the waiting queue and contention for idle computing resources, and simultaneously, to improve the satisfaction degree of the user, based on the priority of the user submitting the computing task, the system may obtain the first type of historical use data of the source user corresponding to the computing task, for example, the priority of the user submitting the computing task. Generally, the platform personnel can grade the customers according to the importance of the customers, for the vip customers with high importance, the priority of the submitted computing tasks is also high, and the system can process the computing tasks with high priority in priority, so that the computing tasks with the highest user priority are positioned at the head of the waiting queue.
According to an implementation manner in the embodiment of the present disclosure, the step of processing each sub-task by using the idle computing resource with the minimum scheduling granularity includes:
acquiring second type historical use data of a source user corresponding to the computing task, wherein the second type historical use data comprises a historical task use resource type of the source user;
and calling the idle computing resources matched with the historical task use resource types of the source users to execute the subtasks.
After each computing task is completed, the user scores the satisfaction degree of processing the computing task, for example, the scoring range can be set to 1-5, and the super computer platform system stores the scoring data of the user on the task in a back-end database, for example, by
Is stored in a form of a liquid crystal or a liquid crystal,
and the comprehensive evaluation of the user u on the computing quality and the satisfaction degree of the task t is shown.
When the platform receives a new computing task, the system firstly analyzes the second type of historical use data of the user and preferentially matches the computing resources suitable for the user according to the second type of historical use data of the user. For example, a user is matched with a high historical scoring computing resource, taking into account the user's scoring of past computing tasks. By calling the matched idle computing resources to execute the subtasks, the higher satisfaction degree of the user is effectively ensured, and the problem that the computing tasks are not matched with the idle computing resources is avoided.
According to an implementation manner in the embodiment of the present disclosure, the step of obtaining the estimated processing time consumption of the calculation task further includes:
dividing the completed historical computing tasks into a plurality of communities, wherein each community corresponds to a reference processing time;
determining a target community corresponding to the current computing task according to the similarity clustering, wherein the similarity between the historical computing task and the current computing task in the target community is the highest;
and determining the estimated processing time consumption of the current computing task according to the reference processing time consumption of the target community.
The platform records the calculation force used by the completed calculation task, the size of the task amount, the calculation duration and other data, wherein the calculation force represents the calculation resources participating in the scheduling, the calculation resources comprise user-specified or system-allocated CPU number, GPU number, memory number, License number and other key information of resource scheduling, and the calculation tasks using the approximately same calculation force can be regarded as the same magnitude.
Feature extraction is carried out on computing power data corresponding to computing tasks, the computing power data are abstracted into feature vectors, meanwhile, the computed tasks are supposed to form a scattered point set S, self-adaptive clustering is carried out on the scattered point set S according to the feature vectors of the computing tasks, the computing tasks with similar magnitude are divided into communities, and the threshold value of the Euler distance of the feature vectors in each community is not larger than that of the feature vectors in each community
. And each community corresponds to a rough consumed time for processing the calculation task of the magnitude, which is called reference processing time consumption.
When a new computing task submitted by a user is received, the system abstracts computing power data such as the number of CPUs (central processing units), the number of GPUs (graphic processing units), the number of memories, License and the like required by the new computing task into a feature vector W, adds the feature vector into an existing scattered point set S, performs clustering processing, finally judges the reference processing time consumption corresponding to a community where the computing task is located, and takes the reference processing time consumption as the estimated time consumption of the new computing task.
The method has the advantages that the acceptance degree of the user to the historical calculation task is analyzed according to the historical use data of the user, the estimated time consumption of the current calculation task is estimated, calculation resources with higher satisfaction can be more accurately distributed to the user, and the average waiting time is reduced.
The computing resource scheduling method provided by the embodiment of the invention adopts a self-adaptive clustering method to estimate the time length of the computing task after the execution from the historical use data of the user, calls the computing resource according with the user satisfaction according to the user rating record, refines the parallel scheduling granularity of the super computing platform, and divides the large-scale computing task into a plurality of subtasks, thereby effectively and fully utilizing the fragment resource, improving the utilization rate of the computing resource, greatly reducing the waiting time of the large-scale computing task in a waiting queue and being beneficial to enhancing the user satisfaction.
Corresponding to the above method embodiments, referring to fig. 7, a block diagram of a computing resource scheduling apparatus according to an embodiment of the present disclosure is provided. As shown in fig. 7, the computing resource scheduling apparatus 700 mainly includes:
a splitting module 701, configured to split a computation task into multiple sub-tasks based on a minimum scheduling granularity in a supercomputer if the computation task is in a waiting queue;
a calling module 702, configured to process each subtask by using an idle computing resource with a minimum scheduling granularity;
the computing resource scheduling apparatus further includes:
the judging module is used for judging whether the amount of the computing resources required by the computing task is larger than the total amount of the idle computing resources or not when the computing task is received;
the adding module is used for adding the computing task into the waiting queue according to historical data of the computing task if the computing resource amount required by the computing task is larger than the total amount of the idle computing resources, wherein the historical data of the computing task comprises estimated processing time consumption and waiting time length of the computing task or the priority of a source user of the computing task;
and the processing module is used for directly calling the idle computing resources to process the computing task if the computing resource amount required by the computing task is less than the total amount of the idle computing resources.
In addition, an embodiment of the present disclosure further provides a computer device, which includes a display terminal, a memory and a processor, where the memory is connected to the processor, the memory is used to store a computer program, and the processor runs the computer program to make the computer device execute the computing resource scheduling method in any of the foregoing embodiments.
Furthermore, the embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and when the computer program runs on a processor, the computer program performs the method for scheduling computing resources according to any one of the above embodiments.
The specific implementation processes of the computing resource scheduling apparatus, the computer device, and the computer-readable storage medium provided in the embodiments of the present invention may refer to the specific implementation processes of the computing resource scheduling method, which are not described in detail herein.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative and, for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, each functional module or unit in each embodiment of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention or a part of the technical solution that contributes to the prior art in essence can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention.
Claims (9)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110032737.1A CN112363821A (en) | 2021-01-12 | 2021-01-12 | Computing resource scheduling method and device and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110032737.1A CN112363821A (en) | 2021-01-12 | 2021-01-12 | Computing resource scheduling method and device and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112363821A true CN112363821A (en) | 2021-02-12 |
Family
ID=74534790
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110032737.1A Pending CN112363821A (en) | 2021-01-12 | 2021-01-12 | Computing resource scheduling method and device and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112363821A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113419833A (en) * | 2021-06-24 | 2021-09-21 | 中国信息通信研究院 | Method and device for quantum cloud computing platform task scheduling and quantum cloud computing platform task scheduling server |
| CN113535400A (en) * | 2021-07-19 | 2021-10-22 | 闻泰通讯股份有限公司 | Parallel computing resource allocation method and device, storage medium, and terminal device |
| CN113590030A (en) * | 2021-06-30 | 2021-11-02 | 济南浪潮数据技术有限公司 | Queue scheduling method, system, equipment and medium |
| CN113656166A (en) * | 2021-09-02 | 2021-11-16 | 上海联影医疗科技股份有限公司 | Task processing system and computing resource allocation method thereof |
| CN113687951A (en) * | 2021-09-01 | 2021-11-23 | 睿云奇智(重庆)科技有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN113961328A (en) * | 2021-10-26 | 2022-01-21 | 深圳大学 | Task processing method and device, storage medium and electronic equipment |
| CN114168302A (en) * | 2021-12-28 | 2022-03-11 | 中国建设银行股份有限公司 | Task scheduling method, device, device and storage medium |
| CN114327811A (en) * | 2021-11-19 | 2022-04-12 | 浪潮电子信息产业股份有限公司 | Task scheduling method, device and equipment and readable storage medium |
| CN114780253A (en) * | 2022-06-15 | 2022-07-22 | 广东省信息工程有限公司 | Big data intelligent analysis system |
| CN114840380A (en) * | 2022-04-26 | 2022-08-02 | 北京快乐茄信息技术有限公司 | Alarm method, alarm device, electronic equipment and storage medium |
| US20220291959A1 (en) * | 2021-06-03 | 2022-09-15 | Research Institute Of Physical And Chemical Engineering Of Nuclear Industry | Activity scheduling method, system, terminal and storage medium based on high response ratio |
| CN117095804A (en) * | 2023-10-19 | 2023-11-21 | 江苏健康无忧网络科技有限公司 | A method for scheduling idle periods of medical examinations |
| CN119988037A (en) * | 2025-04-14 | 2025-05-13 | 成都秦川物联网科技股份有限公司 | Computing and scheduling system, method and medium based on industrial Internet of Things data center |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080040630A1 (en) * | 2005-09-29 | 2008-02-14 | Nortel Networks Limited | Time-Value Curves to Provide Dynamic QoS for Time Sensitive File Transfers |
| CN103336719A (en) * | 2013-06-04 | 2013-10-02 | 江苏科技大学 | Distribution rendering system and method in P2P mode |
| CN105955873A (en) * | 2016-04-27 | 2016-09-21 | 乐视控股（北京）有限公司 | Task processing method and apparatus |
| US20190340026A1 (en) * | 2018-05-02 | 2019-11-07 | Servicenow, Inc. | Periodic task execution in an automated context |
| CN110874271A (en) * | 2019-11-20 | 2020-03-10 | 山东省国土测绘院 | Method and system for rapidly calculating mass building pattern spot characteristics |
-
2021
- 2021-01-12 CN CN202110032737.1A patent/CN112363821A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080040630A1 (en) * | 2005-09-29 | 2008-02-14 | Nortel Networks Limited | Time-Value Curves to Provide Dynamic QoS for Time Sensitive File Transfers |
| CN103336719A (en) * | 2013-06-04 | 2013-10-02 | 江苏科技大学 | Distribution rendering system and method in P2P mode |
| CN105955873A (en) * | 2016-04-27 | 2016-09-21 | 乐视控股（北京）有限公司 | Task processing method and apparatus |
| US20190340026A1 (en) * | 2018-05-02 | 2019-11-07 | Servicenow, Inc. | Periodic task execution in an automated context |
| CN110874271A (en) * | 2019-11-20 | 2020-03-10 | 山东省国土测绘院 | Method and system for rapidly calculating mass building pattern spot characteristics |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220291959A1 (en) * | 2021-06-03 | 2022-09-15 | Research Institute Of Physical And Chemical Engineering Of Nuclear Industry | Activity scheduling method, system, terminal and storage medium based on high response ratio |
| CN113419833B (en) * | 2021-06-24 | 2023-12-29 | 中国信息通信研究院 | Method and device for task scheduling of quantum cloud computing platform and task scheduling server of quantum cloud computing platform |
| CN113419833A (en) * | 2021-06-24 | 2021-09-21 | 中国信息通信研究院 | Method and device for quantum cloud computing platform task scheduling and quantum cloud computing platform task scheduling server |
| CN113590030A (en) * | 2021-06-30 | 2021-11-02 | 济南浪潮数据技术有限公司 | Queue scheduling method, system, equipment and medium |
| CN113590030B (en) * | 2021-06-30 | 2023-12-26 | 济南浪潮数据技术有限公司 | Queue scheduling method, system, equipment and medium |
| WO2023000443A1 (en) * | 2021-07-19 | 2023-01-26 | 闻泰通讯股份有限公司 | Parallel computing resource allocation method and apparatus, storage medium, and terminal device |
| CN113535400A (en) * | 2021-07-19 | 2021-10-22 | 闻泰通讯股份有限公司 | Parallel computing resource allocation method and device, storage medium, and terminal device |
| CN113687951A (en) * | 2021-09-01 | 2021-11-23 | 睿云奇智(重庆)科技有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN113687951B (en) * | 2021-09-01 | 2024-09-20 | 睿云奇智(重庆)科技有限公司 | Task processing method, device, electronic equipment and computer readable storage medium |
| CN113656166A (en) * | 2021-09-02 | 2021-11-16 | 上海联影医疗科技股份有限公司 | Task processing system and computing resource allocation method thereof |
| CN113961328A (en) * | 2021-10-26 | 2022-01-21 | 深圳大学 | Task processing method and device, storage medium and electronic equipment |
| CN114327811A (en) * | 2021-11-19 | 2022-04-12 | 浪潮电子信息产业股份有限公司 | Task scheduling method, device and equipment and readable storage medium |
| CN114168302A (en) * | 2021-12-28 | 2022-03-11 | 中国建设银行股份有限公司 | Task scheduling method, device, device and storage medium |
| CN114840380A (en) * | 2022-04-26 | 2022-08-02 | 北京快乐茄信息技术有限公司 | Alarm method, alarm device, electronic equipment and storage medium |
| CN114780253B (en) * | 2022-06-15 | 2022-08-30 | 广东省信息工程有限公司 | Big data intelligent analysis system |
| CN114780253A (en) * | 2022-06-15 | 2022-07-22 | 广东省信息工程有限公司 | Big data intelligent analysis system |
| CN117095804B (en) * | 2023-10-19 | 2023-12-29 | 江苏健康无忧网络科技有限公司 | Medical examination idle period scheduling method |
| CN117095804A (en) * | 2023-10-19 | 2023-11-21 | 江苏健康无忧网络科技有限公司 | A method for scheduling idle periods of medical examinations |
| CN119988037A (en) * | 2025-04-14 | 2025-05-13 | 成都秦川物联网科技股份有限公司 | Computing and scheduling system, method and medium based on industrial Internet of Things data center |
| CN119988037B (en) * | 2025-04-14 | 2025-06-20 | 成都秦川物联网科技股份有限公司 | Computing scheduling system, method and medium based on industrial Internet of things data center |
| US12498978B2 (en) | 2025-04-14 | 2025-12-16 | Chengdu Qinchuan Iot Technology Co., Ltd. | Methods, systems, and storage media for computation scheduling based on IIoT data centers |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112363821A (en) | Computing resource scheduling method and device and computer equipment | |
| CN112416585B (en) | GPU resource management and intelligent scheduling methods for deep learning | |
| CN110990154B (en) | A big data application optimization method, device and storage medium | |
| CN111506434B (en) | Task processing method and device and computer readable storage medium | |
| CN109491775B (en) | Task processing and scheduling method used in edge computing environment | |
| CN113886030A (en) | Resource scheduling method, electronic device and storage medium | |
| CN112148454A (en) | Edge computing method supporting serial and parallel and electronic equipment | |
| CN112035229A (en) | Calculation graph processing method and device and storage medium | |
| CN114896070B (en) | GPU resource allocation method for deep learning task | |
| CN106897199B (en) | Batch job execution time prediction method based on big data processing framework | |
| CN113608845B (en) | Data processing method and device | |
| CN119356888B (en) | Intelligent management method for computer hardware resources | |
| CN114637584A (en) | Method, device, equipment and medium for adjusting parallelism of data processing task | |
| CN114756375A (en) | GPU resource scheduling method, system, electronic equipment and storage medium | |
| CN111930485B (en) | Job scheduling method based on performance expression | |
| CN120011093A (en) | Multi-task processing method and related device for accelerator | |
| CN113094175A (en) | Load balancing method and device | |
| CN113127173A (en) | Heterogeneous sensing cluster scheduling method and device | |
| CN116405543A (en) | A method, device, equipment and medium for dynamic scheduling of microservices | |
| CN112114967B (en) | GPU resource reservation method based on service priority | |
| CN116719628A (en) | A concurrent task preemptive scheduling method, system and medium | |
| CN117806801A (en) | Cluster scheduling method and system for federal learning tasks | |
| CN118819748A (en) | A task scheduling method, scheduling management system and multi-core processor | |
| CN113032098B (en) | Virtual machine scheduling method, device, equipment and readable storage medium | |
| CN115168028A (en) | Task processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210212 |
|
| RJ01 | Rejection of invention patent application after publication |