CN118069363A - Cluster task resource estimation method and device and electronic equipment - Google Patents
Cluster task resource estimation method and device and electronic equipment Download PDFInfo
- Publication number
- CN118069363A CN118069363A CN202410243663.XA CN202410243663A CN118069363A CN 118069363 A CN118069363 A CN 118069363A CN 202410243663 A CN202410243663 A CN 202410243663A CN 118069363 A CN118069363 A CN 118069363A
- Authority
- CN
- China
- Prior art keywords
- category
- resource
- task
- task request
- historical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/30—Circuit design
- G06F30/32—Circuit design at the digital level
- G06F30/33—Design verification, e.g. functional simulation or model checking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2111/00—Details relating to CAD techniques
- G06F2111/02—CAD in a network environment, e.g. collaborative CAD or distributed simulation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Geometry (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides a method and a device for estimating task resources of a cluster and electronic equipment, wherein when the electronic equipment obtains a target task request submitted by a user, the type of the target task request is determined according to the association relation between the target task request and at least one type of historical task requests in the cluster, and the resource demand information of the target task request is estimated based on a resource bias curve fitted by a plurality of historical task requests corresponding to the type according to the similarity of the resource utilization of the tasks in the same type, so that the accuracy of an estimation result is ensured, the capacity of dynamically adjusting the allocation of the cluster resources is improved, meanwhile, the estimation process saves the evaluation process of staff on the application resources, the application flow of the staff on the resources in the cluster is simplified, and the efficiency of the task application is improved.
Description
Technical Field
The application relates to a cluster task resource estimation method and device and electronic equipment, but is not limited to the method and device.
Background
In the IC design and verification process, technicians need to submit corresponding use cases to a computer cluster for operation, the computer cluster schedules cluster resources, and corresponding nodes are allocated to operate the use cases. In the process of submitting a task, memory and CPU resources of a computer cluster are required to be occupied when the task is submitted and operated, and when the application evaluation of a technician to the self-calculation task is inaccurate, the situation that the applied memory resources deviate from the memory resources used when the task is actually operated, so that the cluster can provide insufficient resources or waste resources exists.
Disclosure of Invention
The application provides a method and a device for estimating cluster task resources and electronic equipment, which are used for solving the technical problem of poor estimation results of the cluster task resources.
In a first aspect, an embodiment of the present application provides a method for estimating a task resource of a cluster, where the method is applied to an electronic device, and the method includes:
Obtaining a target task request submitted by a user terminal;
determining the category of the target task request according to the association relation of the target task request and the historical task request of at least one category in the cluster;
and estimating the resource demand information of the target task request based on the resource bias curves fitted by the plurality of historical task requests corresponding to the categories.
In the technical scheme, when the electronic equipment obtains a target task request submitted by a user, the category of the target task request is determined according to the association relation between the target task request and at least one category of historical task requests in the cluster, the resource demand information of the target task request is estimated based on a plurality of resource bias curves fitted by the historical task requests corresponding to the category according to the similarity of the resource utilization of the tasks in the same category, so that the accuracy of an estimation result is ensured, the capability of dynamic adjustment of cluster resource allocation is improved, meanwhile, the estimation process saves the evaluation process of staff on application resources, simplifies the application flow of the staff on the resources in the cluster, and improves the efficiency of task application.
Optionally, determining the category of the target task request according to the association relation between the target task request and the historical task request of at least one category in the cluster includes:
calculating the Levenstein ratio of the target task request and historical task requests of various types in the cluster;
and determining the category of the target task request based on the Levenstein ratio and the Levenstein threshold value corresponding to each category.
Optionally, determining the category of the target task request based on the levenstein ratio and the levenstein threshold corresponding to each category includes:
Determining a category of a maximum value of the levenstein ratio that is greater than the levenstein threshold as a category of the target task request when at least one levenstein ratio is greater than or equal to a corresponding levenstein threshold;
Or alternatively
And when each Levenstein ratio is smaller than a corresponding Levenstein threshold value, determining the first category which is not recorded in the cluster as the category of the target task request.
Optionally, before calculating the levenstein ratio of the target task request and the historical task requests of various types in the cluster, the method further includes:
acquiring task configuration information from the target task request;
determining a category to which the target task request belongs based on the task configuration information and at least one fitting factor;
each of the categories includes at least one category.
In the technical scheme, before the Levenstein ratio is calculated, the category to which the target task request belongs is determined based on at least one fitting factor so as to roughly classify the target task request, the situation that the Levenstein of different task requests submitted by different queues aiming at the same project is larger to cause misjudgment of category division is prevented, and the accuracy of category determination and subsequent resource demand information estimation is improved.
Optionally, before estimating the resource demand information of the target task request based on the resource bias curves fitted by the plurality of historical task requests corresponding to the category, the method further includes:
determining at least one category based on the association of a plurality of historical task requests sequentially obtained in the historical period;
And determining the resource application information of the first obtained historical task request as a standard resource bias curve of each category aiming at a plurality of historical task requests corresponding to each category, and correcting the standard resource bias curve by utilizing the resource application information of the remaining obtained historical task requests to obtain the resource bias curve.
Optionally, determining at least one category based on the association relationship of the plurality of historical task requests sequentially obtained in the historical period includes:
Determining a second category as the category of the first acquired historical task request in the historical period;
Calculating the Levenstein ratio of the historical task request of the category to be determined and the historical task request corresponding to each determined category based on the task acquisition sequence in the historical period;
when at least one levenstein ratio is greater than or equal to a corresponding levenstein threshold, determining a category corresponding to the maximum value of the at least one levenstein ratio as the category of the historical task request of the category to be determined;
And when the Levenstein ratios corresponding to the respective categories are smaller than the corresponding Levenstein thresholds, determining the third category as the category of the historical task request of the category to be determined.
In the technical scheme, the tasks obtained by the clusters are classified, the resource bias curve fitted by at least one group of similar tasks is determined based on the Levens ratio calculation among the tasks, the accuracy of the cluster fitting curve is ensured, when a user submits the tasks, the bias curve corresponding to the task is determined based on the class of the task, so that the resource demand information required by the task is determined, the application resource is not required to be evaluated by a worker, the application flow of the worker on the resources in the clusters is simplified, the accuracy of the clusters on the initial memory allocation of the tasks can be improved, the failure of the cluster on the balance strategy of the resources due to the overlarge application resource deviation is prevented, and the dynamic adjustment capability of the cluster resource allocation is improved.
Optionally, the resource bias curve includes a maximum resource bias curve, a resource fitting bias curve and/or a minimum resource bias curve;
Estimating the resource demand information of the target task request based on a resource bias curve fitted by a plurality of historical task requests corresponding to the category, including:
acquiring resource bias curves corresponding to each type from a database;
acquiring a target resource bias curve from at least one resource bias curve fitted by a plurality of historical task requests corresponding to the category according to a preset curve selection standard;
And determining the maximum value of the target resource bias curve as the resource demand information of the target task request.
Optionally, after estimating the resource demand information of the target task request based on the resource bias curves fitted by the plurality of historical task requests corresponding to the category, the method further includes:
After the target task request is operated, correcting a resource bias curve of a category corresponding to the target task request based on the resource application information of the target task request;
Uploading the corrected resource bias curve to a database.
In a second aspect, the present application provides a cluster task resource estimation device, including:
the acquisition module is used for acquiring a target task request submitted by the user terminal;
the processing module is used for determining the category of the target task request according to the association relation of the target task request and the historical task request of at least one category in the cluster;
The processing module is also used for estimating the resource demand information of the target task request based on a resource bias curve fitted by a plurality of historical task requests corresponding to the category.
In a third aspect, the present application provides an electronic device comprising: a processor and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor, when executing the computer-executable instructions, is configured to implement the method of any one of the first aspects.
According to the cluster task resource estimation method, the cluster task resource estimation device and the electronic equipment, when the electronic equipment obtains the target task request submitted by the user, the category of the target task request is determined according to the association relation between the target task request and the historical task requests of at least one category in the cluster, the resource demand information of the target task request is estimated based on the resource bias curve fitted by the plurality of historical task requests corresponding to the category according to the similarity of the resource utilization of the tasks of the same category, so that the accuracy of an estimation result is ensured, the capacity of dynamically adjusting the cluster resource allocation is improved, meanwhile, the estimation process saves the evaluation process of staff on the application resource, the application flow of the staff on the resources in the cluster is simplified, and the efficiency of task application is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
FIG. 1 is an application scenario diagram of a cluster task resource estimation method according to some embodiments of the present application;
FIG. 2 is a flowchart illustrating a method for estimating resources of a cluster task according to some embodiments of the present application;
FIG. 3 is a flowchart illustrating a method for estimating resources of a cluster task according to further embodiments of the present application;
FIG. 4 is a flowchart of a method for fitting a resource bias curve according to other embodiments of the present application;
FIG. 5 is a flowchart illustrating a method for estimating resources of a cluster task according to further embodiments of the present application;
FIG. 6 is a flowchart illustrating a method for estimating resources of a cluster task according to further embodiments of the present application;
FIG. 7 is a schematic structural diagram of a task resource estimation device according to some embodiments of the present application;
Fig. 8 is a schematic structural diagram of an electronic device according to some embodiments of the present application.
Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terminology used in the embodiments of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this embodiment of the application, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a product or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such product or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a commodity or system comprising such elements.
Fig. 1 is an application scenario diagram of a cluster task resource estimation method according to some embodiments of the present application, where, as shown in fig. 1, the scenario includes a cluster 100 and a user terminal 106, and the user terminal 106 is communicatively connected to the cluster 100.
The cluster 100 includes a plurality of node devices including a management device 107 and a plurality of managed node devices, the management device 107 being communicatively coupled to each of the managed node devices.
In some embodiments, the plurality of managed node devices are divided into a plurality of queues based on device configuration information of each managed node device. The device configuration information comprises CPU configuration information and memory configuration information of the managed node device, the CPU configuration information comprises CPU quantity, and the memory configuration information comprises physical memory quantity.
In the application scenario shown in fig. 1, a plurality of managed node devices are divided into two queues: a first queue 109 and a second queue 111, the first queue 109 including a plurality of first managed node devices 110, and the second queue 111 including a plurality of second managed node devices 112. The first managed node device 110 may be a device with a larger number of CPUs, and the second managed node device 112 may be a device with a larger number of physical memories.
The cluster 100 is provided with a cluster management system, and the management device 107 allocates and schedules resources of each node based on the cluster management system. In the process of IC design and verification, a technician needs to submit a corresponding application case and a task to the cluster 100 through the user terminal 106 to run, more specifically, the user terminal 106 submits the task to the management device 107, and the management device 107 distributes the task to a proper managed node device according to the task information, the configuration information and the running condition of each managed node device.
When the management device 107 performs task allocation, the application memory set when the user terminal 106 submits the task is considered, and the task is submitted to the managed node device with the actual free memory larger than the application memory, where in the related art, after the management device 107 allocates the task to the corresponding managed node device, the actual memory usage condition of the task is not regulated.
When the application evaluation of the self-calculation task by the technician is inaccurate, there is a deviation between the applied memory resource and the memory resource used when the task actually runs, and there is a situation that the applied memory submitted by the user terminal 106 is inconsistent with the running memory actually applied by the task, so that the resource utilization rate or the running process of the managed node device is affected.
More specifically, when the application memory submitted by the user terminal 106 is smaller than the running memory of the task actually applied, the running of the node device will be affected, and the machine will be blocked; when the application memory submitted by the user terminal 106 is larger than the actual demand of the task, the cluster resource is wasted.
Therefore, how to improve accuracy of task resource estimation results when users submit tasks to a cluster becomes an important point of research.
In order to solve the problems, the application provides a cluster task resource estimation method, a cluster task resource estimation device and electronic equipment. The technical conception of the application is as follows: when the electronic equipment obtains a target task request submitted by a user, the category of the target task request is determined according to the association relation between the target task request and at least one category of historical task requests in the cluster, the resource demand information of the target task request is estimated based on a plurality of resource bias curves fitted by the historical task requests corresponding to the category according to the similarity of the resource utilization of the tasks in the same category, so that the accuracy of an estimation result is ensured, the capability of dynamic adjustment of cluster resource allocation is improved, meanwhile, the estimation process saves the evaluation process of the application resources by staff, simplifies the application flow of the staff on the resources in the cluster, and improves the efficiency of task application.
The technical scheme of the application is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
In the present application, the management node device may be used as an execution subject to execute the method according to the following embodiment. The controller communicatively connected to the management node device may be used as an execution body, and the controller may be a managed node device, or may be another electronic device that is communicatively connected to the management node device outside the cluster, as shown by the electronic device 101 in fig. 1.
In particular, the execution body may be a hardware device that executes the method, or a software application that implements the embodiments described below in the hardware device, or a computer-readable storage medium on which the software application that implements the embodiments described below is installed, or code of the software application that implements the embodiments described below.
Fig. 2 is a flow chart of a cluster task resource estimation method according to some embodiments of the present application, as shown in fig. 2, where the method includes:
s101, obtaining a target task request submitted by a user terminal.
In some embodiments, the target task request includes task attribute information including a task ID, a task name, a task affiliated user ID, an affiliated queue, and the like.
S102, determining the category of the target task request according to the association relation between the target task request and the historical task request of at least one category in the cluster.
And running and completing a plurality of historical task requests in the cluster, and dividing the plurality of historical task requests into at least one category according to task attribute information of each historical task request.
In some embodiments, the historical task request of the determined category may be determined as the historical task request corresponding to the category.
The electronic device determines the category of the history task request with highest relevance as the category of the target task request when at least one history task request with definite relevance exists based on the association relation between the task attribute information of the target task request and the history task requests corresponding to each class, and enters step S103 to estimate the resource demand information of the target task request.
When there is no history task request with clear relevance, determining the category of the target task request as a new category, and determining the resource demand information of the target task request as the resource demand information provided by the user in the task attribute information.
In some embodiments, the resource requirement information includes run memory requirement information.
S103, estimating resource demand information of the target task request based on a resource bias curve fitted by the plurality of historical task requests corresponding to the category.
The same category corresponds to a plurality of historical task requests, each task request corresponds to a plurality of data, and the data is the use amount of the running memory in the running process of the historical task requests. The curve of the composition of the running memory usage is a bias curve.
The resource bias curve fitted by the plurality of historical task requests corresponding to the same category is a resource bias curve fitted by a plurality of running memory usage amounts corresponding to a plurality of running time points in the task running process of the plurality of historical task requests.
For each historical task request, in the task operation period, taking the task starting point as a reference origin, and the subsequent points are operation points relative to the reference origin.
In some embodiments, the electronic device determines a maximum value of the resource bias curve as resource requirement information of the target task request, so that the cluster performs task allocation and dynamic resource regulation based on the resource requirement information.
In the technical scheme, when the electronic equipment obtains a target task request submitted by a user, the category of the target task request is determined according to the association relation between the target task request and at least one category of historical task requests in the cluster, the resource demand information of the target task request is estimated based on a plurality of resource bias curves fitted by the historical task requests corresponding to the category according to the similarity of the resource utilization of the tasks in the same category, so that the accuracy of an estimation result is ensured, the capability of dynamic adjustment of cluster resource allocation is improved, meanwhile, the estimation process saves the evaluation process of staff on application resources, simplifies the application flow of the staff on the resources in the cluster, and improves the efficiency of task application.
Fig. 3 is a flow chart of a cluster task resource estimation method according to another embodiment of the present application, where, as shown in fig. 3, the method includes:
s101, obtaining a target task request submitted by a user terminal.
S104, obtaining task configuration information from the target task request.
In some embodiments, the task configuration information includes, but is not limited to, a task item name, a task queue name.
S105, determining the door class to which the target task request belongs based on the task configuration information and at least one fitting factor.
The fitting factor is a reference factor for roughly classifying the obtained task request. In some embodiments, the fit factors include, but are not limited to, task item names, task queue names.
At least one fitting factor is obtained from the set of fitting factors when the task is coarsely classified.
And comparing the task configuration information of the target task request with the at least one selected fitting factor, and determining the category determined by the at least one fitting factor by the task after the task configuration information and each selected fitting factor are successfully matched.
Each category includes at least one category. Each class is a more detailed division of the class.
S1021, calculating the Levenstein ratio of the target task request and the historical task requests of various types in the cluster.
In some embodiments, the Levenstein ratio is calculated for the request statements of the target task request and the request statements of the historical task requests of various classes in the cluster.
S1022, determining the category of the target task request based on the Levenstein ratio and the Levenstein threshold corresponding to each category.
Determining a category corresponding to a maximum value of the at least one levenstein ratio as a category of the target task request when the at least one levenstein ratio is greater than or equal to a corresponding levenstein threshold;
Or alternatively
When each lycenstant ratio is less than a corresponding lycenstant threshold, determining a first category not recorded within the cluster as a category of the target task request.
In some embodiments, the levenstein threshold is a fixed value, and the levenstein thresholds corresponding to each class are the same.
In other embodiments, each class sets a corresponding levenstein threshold value:
When classifying the classes of each task according to the acquisition sequence of the task request, if the task is the task acquired by the cluster system for the first time, the Levenstein threshold is a default threshold set by the cluster system;
if the task is a task that is not acquired for the first time by the cluster system, the levenstein threshold may be updated based on the levenstein ratios corresponding to the plurality of historical tasks determined to be of the class and the initially set levenstein threshold.
For example: based on a preset time interval, updating the average value of at least one Levenstein ratio corresponding to the category obtained by calculation in the current Levenstein threshold and the previous time interval into the Levenstein threshold.
S103, estimating resource demand information of the target task request based on a resource bias curve fitted by the plurality of historical task requests corresponding to the category.
The fitting process of the resource bias curve for the multiple historical task request fits is explained below.
Fig. 4 is a flow chart of a resource bias curve fitting method according to another embodiment of the present application, as shown in fig. 4, where the method includes:
S1001, determining at least one category based on the association relation of a plurality of historical task requests sequentially obtained in the historical period.
More specifically, during the history period, determining the second category as the category of the first obtained history task request;
calculating the Levenstein ratio of the historical task request of the category to be determined and the historical task request corresponding to each determined category based on the task acquisition sequence in the historical period;
When the at least one levenstein ratio is greater than or equal to the corresponding levenstein threshold, determining a category corresponding to the maximum value of the at least one levenstein ratio as the category of the historical task request of the category to be determined;
And when the Levenstein ratios corresponding to the respective categories are smaller than the corresponding Levenstein thresholds, determining the third category as the category of the historical task request of the category to be determined.
More specifically, when the cluster obtains the first task, an operating resource bias curve is generated based on the operating memory change data of the first task. And classifying a plurality of tasks and a first task after the cluster is obtained based on at least one fitting factor, respectively carrying out Lavensitan ratio calculation on the plurality of tasks classified into the same class and the first task, determining a task corresponding to a Lavensitan Wen Sitan ratio which is larger than or equal to a preset Lavensitan ratio as a task similar to the first task, generating a bias curve based on the running memory data of the first task, and correcting the bias curve based on the running memory data of the similar task obtained later.
A task that is classified into different categories based on at least one fit or that is in the same category but corresponds to a Levenostane ratio of less Yu Laiwen Stant threshold than the first task is determined to be a task that is dissimilar to the first task, and a new resource bias curve is generated based on the task.
And when the Lei Wen Sitan ratio of the task obtained by the cluster to the task corresponding to each running memory bias curve is larger than the corresponding Levenstein threshold, selecting the task associated with the maximum value as a similar task.
S1002, determining the resource application information of the first obtained historical task request as a standard resource bias curve of each category aiming at a plurality of historical task requests corresponding to each category, and correcting the standard resource bias curve by utilizing the resource application information of the remaining obtained historical task requests to obtain the resource bias curve.
That is, curve fitting is performed for data of a plurality of tasks of the same class: in the task operation period, determining a curve formed by the maximum values of a plurality of data corresponding to each operation time point as a maximum resource bias curve; the running time point is a time point determined by taking the task running time as an abscissa and the task starting running time as an initial coordinate, and the current time is relative to the time of the task starting running time.
And in the task operation period, determining a curve composed of the minimum values of a plurality of data corresponding to each operation time point as a minimum resource bias curve.
And in the task operation period, determining a curve composed of the mean value or the median value of a plurality of data corresponding to each operation time point as a resource fitting bias curve.
And aiming at the current task, carrying out Lavensitan ratio calculation on the task and the historical task corresponding to each curve, selecting a target resource bias curve according to a preset curve selection standard from a plurality of curves (maximum bias curve, minimum bias curve and fitting bias curve) corresponding to the maximum value of the Lavensitan ratio, and taking the maximum value of the fitting curve as resource demand information of a target task request.
The method for dynamically adjusting cluster resources provided by the application is specifically explained below through an embodiment shown in fig. 5. As shown in fig. 5, includes:
s201, loading cluster system configuration parameters from the data.
Basic configuration information is loaded from the database and stored in a system cache, so that excessive pressure of the database caused by repeated inquiry is avoided. Its basic configuration information includes, but is not limited to, "lycenstant ratio", "exclude user name", "exclude queue", "exclude task type", "source data sampling period", "data fitting factor", etc.
S202, monitoring the completed tasks in the cluster.
S203, obtaining a fitted resource bias curve from the database.
The fitted resource bias curve is a curve fitted with the resource application data of the completed task.
S204, correcting the corresponding resource bias curve by utilizing the resource application information of the completed task.
After the currently running task is completed, based on the category of the task, historical running data and a resource bias curve related to the category are determined, and the running data of the current task is utilized to correct the resource bias curve.
For example: and aiming at each target time point, taking the average value of the memory quantity on the resource bias curve corresponding to the category and the memory quantity corresponding to the current task at the same time point as the updated data of the resource bias curve aiming at the time point so as to update the resource bias curve.
The fitting process of the resource bias curve is shown in fig. 6, and includes:
S301, loading system configuration information.
S302, data sampling.
According to the system query interface of the cluster management software system and the configuration parameters of the 'source data sampling period' in the system (the parameters are determined according to specific items, such as 30 minutes, 1 hour, etc.), the task information of all users running is periodically obtained from the cluster scheduling system, and the specific information includes but is not limited to: task ID, task execution queue, task execution host, task execution time, current occupied memory of task, maximum occupied memory of internal task, average memory of task, task name, user ID of task, etc. After the source data is acquired, preprocessing and filtering are required according to screening conditions such as 'user name elimination', 'queue elimination', 'task type elimination', and the like.
After the tasks are operated, classifying the tasks according to time sequences in a history period where the tasks are located, and counting operation data corresponding to the classes.
S303, directly warehousing the first task, determining the category of the task as a second category, and determining a corresponding resource bias curve based on the resource application information of the task.
I.e. fit the memory data applied by the task operation to a bias curve.
S304, obtaining tasks, roughly classifying the tasks based on the items and the queues of the tasks, and determining the belonging categories.
S305, respectively calculating the Levenstein ratio of the task and the first task of each category in the gate category.
S306, judging whether the Levin Stant ratio of at least one category is larger than or equal to the corresponding Lychnophor threshold.
If yes, go to step S307; otherwise, step S309 is entered.
S307, the category of the task is determined as the category of the maximum value of the levenstein ratio.
S308, recording resource application information of the task, and correcting a resource bias curve corresponding to the category to which the resource application information belongs by using the resource application information after the task is finished.
For example, at the first time point of the resource bias curve, the data corresponding to the original curve is 1, the data corresponding to the current task is 3, and the data on the resource bias curve can be modified to be the average value of the two data: 2.
After the current step is completed, step S304 is performed to acquire a next task, and the resource bias curve is updated based on the running information of the task.
S309, determining the category of the task as a third category.
The first category, the second category and the third category are used for distinguishing, and no limitation exists.
Fig. 7 is a schematic structural diagram of a device for estimating task resources of a cluster according to an embodiment of the present application, and as shown in fig. 7, the device 400 for estimating task resources of a cluster includes an obtaining module 401 and a processing module 402.
The obtaining module 401 is configured to obtain a target task request submitted by a user terminal.
The processing module 402 is configured to determine a category of the target task request according to an association relationship between the target task request and at least one category of historical task requests in the cluster.
The processing module 402 is further configured to estimate resource demand information of the target task request based on a resource bias curve fitted by the plurality of historical task requests corresponding to the category.
In one possible embodiment, the processing module 402 is specifically configured to:
Calculating the Levenstein ratio of the target task request and the historical task requests of various types in the cluster;
and determining the category of the target task request based on the Levenstein ratio and the Levenstein threshold value corresponding to each category.
In one possible embodiment, the processing module 402 is specifically configured to:
determining a category of a maximum value of the levenstein ratio greater than the levenstein threshold as a category of the target task request when the at least one levenstein ratio is greater than or equal to the corresponding levenstein threshold;
Or alternatively
When each lycenstant ratio is less than a corresponding lycenstant threshold, determining a first category not recorded within the cluster as a category of the target task request.
In a possible embodiment, the processing module 402 is further configured to:
obtaining task configuration information from a target task request;
Determining a category to which the target task request belongs based on the task configuration information and at least one fitting factor;
each category includes at least one category.
In a possible embodiment, the processing module 402 is further configured to:
determining at least one category based on the association of a plurality of historical task requests sequentially obtained in the historical period;
And determining the resource application information of the first obtained historical task request as a standard resource bias curve of each category aiming at a plurality of historical task requests corresponding to each category, and correcting the standard resource bias curve by utilizing the resource application information of the remaining obtained historical task requests to obtain the resource bias curve.
In one possible embodiment, the processing module 402 is specifically configured to:
Determining the second category as the category of the first acquired historical task request during the historical period;
calculating the Levenstein ratio of the historical task request of the category to be determined and the historical task request corresponding to each determined category based on the task acquisition sequence in the historical period;
When the at least one levenstein ratio is greater than or equal to the corresponding levenstein threshold, determining a category corresponding to the maximum value of the at least one levenstein ratio as the category of the historical task request of the category to be determined;
And when the Levenstein ratios corresponding to the respective categories are smaller than the corresponding Levenstein thresholds, determining the third category as the category of the historical task request of the category to be determined.
In one possible embodiment, the processing module 402 is specifically configured to:
acquiring resource bias curves corresponding to each type from a database;
Acquiring a target resource bias curve from at least one resource bias curve fitted by a plurality of historical task requests corresponding to the category according to a preset curve selection standard;
and determining the maximum value of the target resource bias curve as the resource demand information of the target task request.
In one possible embodiment, the processing module 402 is specifically configured to:
after the target task request is completed, correcting the resource bias curve of the category corresponding to the target task request based on the resource application information of the target task request;
Uploading the corrected resource bias curve to a database.
Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
In this structural schematic diagram, the electronic device 500 includes a memory 501 and a processor 502.
Memory 501 is used to store computer instructions executable by the processor.
The Memory 501 may include a high-speed random access Memory (Random Access Memory, RAM), and may further include a Non-Volatile Memory (NVM), such as at least one magnetic disk Memory, and may also be a U-disk, a removable hard disk, a read-only Memory, a magnetic disk, or an optical disk.
The processor 502, when executing computer instructions, implements the steps of the clustered task resource estimation method of the above embodiment. Reference may be made in particular to the relevant description of the embodiments of the method described above.
The Processor 502 may be a central processing unit (Central Processing Unit, CPU), or may be another general purpose Processor, a digital signal Processor (DIGITAL SIGNAL Processor, DSP), an Application Specific Integrated Circuit (ASIC), or the like.
Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.
Alternatively, the memory 501 may be separate or integrated with the processor 502.
When the memory 501 is provided separately, the electronic device 500 further comprises a bus for connecting the memory 501 and the processor 502. The bus may be an industry standard architecture (Industry Standard Architecture, ISA) bus, an external device interconnect (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others.
The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, the buses in the drawings of the present application are not limited to only one bus or to one type of bus.
The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores computer instructions, and when the processor executes the computer instructions, the steps in the cluster task resource estimation method in the embodiment are realized.
The embodiment of the application also provides a computer program product, which comprises computer instructions, wherein the computer instructions, when being executed by a processor, realize each step in the cluster task resource estimation method in the embodiment.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (10)
1. A method for estimating cluster task resources, wherein the method is applied to an electronic device, and the method comprises:
Obtaining a target task request submitted by a user terminal;
determining the category of the target task request according to the association relation of the target task request and the historical task request of at least one category in the cluster;
and estimating the resource demand information of the target task request based on the resource bias curves fitted by the plurality of historical task requests corresponding to the categories.
2. The method of claim 1, wherein determining the category of the target task request based on the association of the target task request with at least one category of historical task requests in the cluster comprises:
calculating the Levenstein ratio of the target task request and historical task requests of various types in the cluster;
and determining the category of the target task request based on the Levenstein ratio and the Levenstein threshold value corresponding to each category.
3. The method of claim 2, wherein determining the category of the target task request based on the levenstein ratio and levenstein threshold corresponding to each category comprises:
Determining a category of a maximum value of the levenstein ratio that is greater than the levenstein threshold as a category of the target task request when at least one levenstein ratio is greater than or equal to a corresponding levenstein threshold;
Or alternatively
And when each Levenstein ratio is smaller than a corresponding Levenstein threshold value, determining the first category which is not recorded in the cluster as the category of the target task request.
4. The method of claim 2, further comprising, prior to calculating the levenstein ratio for the target task request and the historical task requests of each category in the cluster:
acquiring task configuration information from the target task request;
determining a category to which the target task request belongs based on the task configuration information and at least one fitting factor;
each of the categories includes at least one category.
5. The method of claim 1, further comprising, prior to estimating the resource demand information for the target task request based on a resource bias curve fitted to a plurality of historical task requests corresponding to the category:
determining at least one category based on the association of a plurality of historical task requests sequentially obtained in the historical period;
And determining the resource application information of the first obtained historical task request as a standard resource bias curve of each category aiming at a plurality of historical task requests corresponding to each category, and correcting the standard resource bias curve by utilizing the resource application information of the remaining obtained historical task requests to obtain the resource bias curve.
6. The method of claim 5, wherein determining at least one category based on the association of a plurality of historical task requests obtained sequentially over the historical period of time comprises:
Determining a second category as the category of the first acquired historical task request in the historical period;
Calculating the Levenstein ratio of the historical task request of the category to be determined and the historical task request corresponding to each determined category based on the task acquisition sequence in the historical period;
when at least one levenstein ratio is greater than or equal to a corresponding levenstein threshold, determining a category corresponding to the maximum value of the at least one levenstein ratio as the category of the historical task request of the category to be determined;
And when the Levenstein ratios corresponding to the respective categories are smaller than the corresponding Levenstein thresholds, determining the third category as the category of the historical task request of the category to be determined.
7. The method according to any one of claims 1-6, wherein the resource bias curve comprises a maximum resource bias curve, a resource fit bias curve, and/or a minimum resource bias curve;
Estimating the resource demand information of the target task request based on a resource bias curve fitted by a plurality of historical task requests corresponding to the category, including:
acquiring resource bias curves corresponding to each type from a database;
acquiring a target resource bias curve from at least one resource bias curve fitted by a plurality of historical task requests corresponding to the category according to a preset curve selection standard;
And determining the maximum value of the target resource bias curve as the resource demand information of the target task request.
8. The method of claim 7, wherein after estimating the resource demand information for the target task request based on the resource bias curves fitted to the plurality of historical task requests corresponding to the category, the method further comprises:
After the target task request is operated, correcting a resource bias curve of a category corresponding to the target task request based on the resource application information of the target task request;
Uploading the corrected resource bias curve to a database.
9. A cluster task resource estimation device, comprising:
the acquisition module is used for acquiring a target task request submitted by the user terminal;
the processing module is used for determining the category of the target task request according to the association relation of the target task request and the historical task request of at least one category in the cluster;
The processing module is also used for estimating the resource demand information of the target task request based on a resource bias curve fitted by a plurality of historical task requests corresponding to the category.
10. An electronic device, comprising: a processor and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
The processor, when executing the computer-executable instructions, is configured to implement the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410243663.XA CN118069363A (en) | 2024-03-04 | 2024-03-04 | Cluster task resource estimation method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410243663.XA CN118069363A (en) | 2024-03-04 | 2024-03-04 | Cluster task resource estimation method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118069363A true CN118069363A (en) | 2024-05-24 |
Family
ID=91103580
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410243663.XA Pending CN118069363A (en) | 2024-03-04 | 2024-03-04 | Cluster task resource estimation method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118069363A (en) |
-
2024
- 2024-03-04 CN CN202410243663.XA patent/CN118069363A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111381970B (en) | Cluster task resource allocation method and device, computer device and storage medium | |
| US8375228B2 (en) | Multiple-node system power utilization management | |
| US10712945B2 (en) | Deduplication processing method, and storage device | |
| CN107025137B (en) | Resource query method and device | |
| CN108205469B (en) | MapReduce-based resource allocation method and server | |
| CN112068934B (en) | Control system and method for shrinking container cloud service instance | |
| WO2024120205A1 (en) | Method and apparatus for optimizing application performance, electronic device, and storage medium | |
| CN110659137A (en) | Processing resource allocation method and system for offline tasks | |
| CN114143327B (en) | Cluster resource quota allocation method and device and electronic equipment | |
| CN116627356B (en) | Distribution control method and system for large-capacity storage data | |
| CN110362275B (en) | Method for improving service quality of solid state disk and solid state disk | |
| CN106020984B (en) | Method and device for creating process in electronic equipment | |
| CN110389817B (en) | Scheduling method, device and computer readable medium of multi-cloud system | |
| CN113626162A (en) | Data center task hybrid deployment method and system based on dynamic resource sharing | |
| WO2019024475A1 (en) | Breakpoint list distribution method and apparatus, and server and medium | |
| CN109284193B (en) | Distributed data processing method based on multithreading and server | |
| CN118069363A (en) | Cluster task resource estimation method and device and electronic equipment | |
| CN112619129B (en) | Cloud game operation method, server and storage medium | |
| CN112181498B (en) | Concurrency control method, device and equipment | |
| CN117112236B (en) | Jurisdictional server configuration method and system based on data inrush current and volatility prediction | |
| CN110069340B (en) | Thread number evaluation method and device | |
| CN115600111B (en) | Resource prediction model training method, cloud resource prediction method and device | |
| CN114942830A (en) | Container scheduling method, container scheduling device, storage medium, and electronic apparatus | |
| US11544171B2 (en) | Method for monitoring the free space of a memory stack | |
| CN118132259A (en) | Cluster resource dynamic adjustment method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |