CN111338802A - Method, system, equipment and medium for optimizing performance of big data cluster - Google Patents
Method, system, equipment and medium for optimizing performance of big data cluster Download PDFInfo
- Publication number
- CN111338802A CN111338802A CN202010132949.2A CN202010132949A CN111338802A CN 111338802 A CN111338802 A CN 111338802A CN 202010132949 A CN202010132949 A CN 202010132949A CN 111338802 A CN111338802 A CN 111338802A
- Authority
- CN
- China
- Prior art keywords
- cpu
- reserved
- threshold value
- resource
- judging whether
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 135
- 230000008569 process Effects 0.000 claims abstract description 92
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 8
- 230000006870 function Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F9/526—Mutual exclusion algorithms
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method, a system, equipment and a storage medium for optimizing the performance of a big data cluster, wherein the method comprises the following steps: creating a plurality of CPU virtual cores based on the number of the CPU physical cores, and locking the CPU virtual cores with a preset number as reserved CPUs; judging whether the CPU core number resources occupied by the current process number exceed a threshold value; responding to the CPU core number resources occupied by the current process number exceeding a threshold value, and judging whether processes densely scheduled by the CPU exist or not; and responding to the process of the CPU intensive scheduling, unlocking the reserved CPU and executing the process of the CPU intensive scheduling by using the reserved CPU. The method, the system, the equipment and the medium for optimizing the performance of the big data cluster, which are provided by the invention, execute the process of the CPU intensive scheduling by setting the virtual core as the reserved CPU, effectively prevent the problem that the process of the CPU intensive scheduling in the big data cluster cannot be executed due to insufficient CPU core number, and greatly improve the performance of the big data cluster.
Description
Technical Field
The present invention relates to the field of big data clusters, and more particularly, to a method, a system, a computer device, and a readable medium for optimizing performance of a big data cluster.
Background
Along with the rapid development of servers, many domestic CPUs and similar products are widely accepted and used by people, and at present, the development direction of the domestic CPUs is to give up the hyper-threading technology and use multiple physical cores to be aggregated into one CPU to improve the overall performance of the CPUs. Because the structure difference between the domestic CPU and the current mainstream CPU is large, although the comprehensive hardware performance of the domestic CPU is equivalent to the performance of some mainstream CPUs, the expected performance value cannot be reached when a server matched with the domestic CPU runs a task in the aspect of large data.
On the current big data platform built by using a multi-physical-core server cluster, the resource scheduler yar n of the big data platform can continuously use the virtual core technology for the physical core, because domestic CPUs use a large number of physical cores and have a single core capability inferior to that of mainstream intel gold series and other CPUs, a plurality of processes calling less CPUs often occur to make a process densely scheduled by a CPU unable to be executed, because the number of the display cores is large, a plurality of processes calling the CPU is not queued to be executed quickly in a queue mode, but executed concurrently, when the processes are scheduled by the CPU intensively and concurrently, the single-core capability of calling the CPU core number is insufficient, and the core number is not much because of being concurrent with other processes, which causes that the task GC (Allocation Failure) time is too long, the execution efficiency is too slow, and the process can not be released for a long time to occupy resources to aggravate the performance problem, thereby greatly influencing the cluster performance.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, a system, a computer device, and a computer readable storage medium for optimizing performance of a big data cluster, where a virtual core is set as a reserved CPU to execute a process of CPU intensive scheduling, so as to effectively prevent the problem that the process of CPU intensive scheduling in the big data cluster cannot be executed due to insufficient number of CPU cores, and greatly improve performance of the big data cluster.
Based on the above object, an aspect of the embodiments of the present invention provides a method for optimizing performance of a big data cluster, including the following steps: creating a plurality of CPU virtual cores based on the number of CPU physical cores, and locking a preset number of the CPU virtual cores as reserved CPUs; judging whether the CPU core number resources occupied by the current process number exceed a threshold value; responding to the CPU core number resources occupied by the current process number exceeding a threshold value, and judging whether processes densely scheduled by the CPU exist or not; and responding to the process of the CPU intensive scheduling, unlocking the reserved CPU and executing the process of the CPU intensive scheduling by using the reserved CPU.
In some embodiments, the determining whether there is a CPU-intensive scheduling process comprises: and sequentially judging whether dense response identification exists in each process.
In some embodiments, the determining whether there is a CPU-intensive scheduling process comprises: and monitoring a resource container corresponding to the process, and judging whether the task number of the memory release time in the resource container, which is greater than the preset time, exceeds a second threshold value.
In some embodiments, said locking a predetermined number of said CPU virtual cores therein as reserved CPUs comprises: and sending a signal that the reserved CPU resource is occupied to a resource manager.
In another aspect of the embodiments of the present invention, a system for optimizing performance of a big data cluster is further provided, including: the system comprises a reservation module, a storage module and a locking module, wherein the reservation module is configured to create a plurality of CPU virtual cores based on the number of CPU physical cores and lock a predetermined number of the CPU virtual cores as reserved CPUs; the first judgment module is configured to judge whether the CPU core number resource occupied by the current process number exceeds a threshold value; the second judgment module is configured to respond to the situation that the number of CPU core resources occupied by the current process number exceeds a threshold value, and judge whether processes densely scheduled by the CPU exist or not; and the execution module is configured to respond to the process with the CPU intensive scheduling, unlock the reserved CPU and execute the process with the CPU intensive scheduling by using the reserved CPU.
In some embodiments, the second determining module is further configured to: and sequentially judging whether dense response identification exists in each process.
In some embodiments, the second determining module is further configured to: and monitoring a resource container corresponding to the process, and judging whether the task number of the memory release time in the resource container, which is greater than the preset time, exceeds a second threshold value.
In some embodiments, the reservation module is further configured to: and sending a signal that the reserved CPU resource is occupied to a resource manager.
In another aspect of the embodiments of the present invention, there is also provided a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method as above.
In a further aspect of the embodiments of the present invention, a computer-readable storage medium is also provided, in which a computer program for implementing the above method steps is stored when the computer program is executed by a processor.
The invention has the following beneficial technical effects: the virtual core is set as the reserved CPU to execute the process of the CPU intensive scheduling, so that the problem that the process of the CPU intensive scheduling in the big data cluster cannot be executed due to insufficient CPU cores is effectively solved, and the performance of the big data cluster is greatly improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic diagram of an embodiment of a method for optimizing big data cluster performance according to the present invention;
fig. 2 is a schematic structural diagram of a computer device according to an embodiment of the method for optimizing performance of a big data cluster provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
Based on the above object, a first aspect of the embodiments of the present invention provides an embodiment of a method for optimizing performance of a big data cluster. Fig. 1 is a schematic diagram illustrating an embodiment of a method for optimizing large data cluster performance according to the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
s1, creating a plurality of CPU virtual cores based on the number of the CPU physical cores, and locking the CPU virtual cores with a preset number as reserved CPUs;
s2, judging whether the CPU core number resources occupied by the current process number exceed a threshold value;
s3, responding to the CPU core number resources occupied by the current process number exceeding a threshold value, and judging whether processes densely scheduled by the CPU exist or not; and
and S4, responding to the process of the CPU intensive scheduling, unlocking the reserved CPU and executing the process of the CPU intensive scheduling by using the reserved CPU.
A plurality of CPU virtual cores are created based on the number of CPU physical cores, and a predetermined number of CPU virtual cores are locked as reserved CPUs. The CPU virtual cores are arranged in the big data cluster, for example, the number of the CPU virtual cores can be set to be 2 times of the number of the CPU physical cores, and the CPU physical cores can be managed and scheduled by managing the CPU virtual cores. A predetermined number of CPU virtual cores may be locked as reserved CPUs, and the predetermined number may be set according to specific situations, for example, 12% of the virtual cores may be set as reserved CPUs.
In some embodiments, said locking a predetermined number of said CPU virtual cores as reserved CPUs comprises: and sending a signal that the reserved CPU resource is occupied to a resource manager. For example, the virtual core selected to the reserved CPU sends a signal that the resource is occupied to the ResourceManager through the nodemanager (node manager) of each node, and thus, these reserved CPUs indicate the occupied status.
And judging whether the CPU core number resources occupied by the current process number exceed a threshold value. And judging whether processes densely scheduled by the CPU exist or not in response to the CPU core number resources occupied by the current process number exceeding a threshold value. Monitoring can be set in a resource manager of a big data platform, and processes and corresponding container requests initiated from each node manager of a cluster to an applicationmaster are monitored. The available resource report can be obtained through a resource manager, and when the number of processes occupies more than 70% of the CPU resources in the resource container, the cluster is judged to be in a multi-process concurrence state and a CPU use frequency state. At this time, whether a process of intensive CPU scheduling exists is detected.
In some embodiments, the determining whether there is a CPU-intensive scheduling process comprises: and sequentially judging whether dense response identification exists in each process. When a resource request is sent to the Applicationmaster, an identifier can be set for the process, namely the process is identified as a process with intensive CPU scheduling, and when the ResourceManager and the Applicationmaster schedule resources, the reserved CPU is released and packed into a container on the basis of the number of CPU cores allocated by the resource scheduler so as to support the calculation power of the process.
In some embodiments, the determining whether there is a CPU-intensive scheduling process comprises: and monitoring a resource container corresponding to the process, and judging whether the task number of the memory release time in the resource container, which is greater than the preset time, exceeds a second threshold value. Monitoring the container under the concurrent process task, and setting the normal memory release time of the container as m. If the number of tasks with the memory release time exceeding m under the container is more than 30%, the process can be determined to be a process intensively scheduled by the CPU.
And responding to the process with the CPU intensive scheduling, unlocking the reserved CPU and executing the process with the CPU intensive scheduling by using the reserved CPU. The reserved cpu release can be packed as container to join this process.
It should be particularly noted that, the steps in the embodiments of the method for optimizing big data cluster performance described above may be mutually intersected, replaced, added, and deleted, and therefore, these methods for optimizing big data cluster performance by reasonable permutation and combination transformation also should belong to the scope of the present invention, and should not limit the scope of the present invention to the embodiments.
In view of the above object, a second aspect of the embodiments of the present invention provides a system for optimizing performance of a big data cluster, including: the system comprises a reservation module, a storage module and a locking module, wherein the reservation module is configured to create a plurality of CPU virtual cores based on the number of CPU physical cores and lock a predetermined number of the CPU virtual cores as reserved CPUs; the first judgment module is configured to judge whether the CPU core number resource occupied by the current process number exceeds a threshold value; the second judgment module is configured to respond to the situation that the number of CPU core resources occupied by the current process number exceeds a threshold value, and judge whether processes densely scheduled by the CPU exist or not; and the execution module is configured to respond to the process with the CPU intensive scheduling, unlock the reserved CPU and execute the process with the CPU intensive scheduling by using the reserved CPU.
In some embodiments, the second determining module is further configured to: and sequentially judging whether dense response identification exists in each process.
In some embodiments, the second determining module is further configured to: and monitoring a resource container corresponding to the process, and judging whether the task number of the memory release time in the resource container, which is greater than the preset time, exceeds a second threshold value.
In some embodiments, the reservation module is further configured to: and sending a signal that the reserved CPU resource is occupied to a resource manager.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executable by the processor to perform the steps of: s1, creating a plurality of CPU virtual cores based on the number of the CPU physical cores, and locking the CPU virtual cores with a preset number as reserved CPUs; s2, judging whether the CPU core number resources occupied by the current process number exceed a threshold value; s3, responding to the CPU core number resources occupied by the current process number exceeding a threshold value, and judging whether processes densely scheduled by the CPU exist or not; and S4, responding to the process of the CPU intensive scheduling, unlocking the reserved CPU and executing the process of the CPU intensive scheduling by using the reserved CPU.
In some embodiments, the determining whether there is a CPU-intensive scheduling process comprises: and sequentially judging whether dense response identification exists in each process.
In some embodiments, the determining whether there is a CPU-intensive scheduling process comprises: and monitoring a resource container corresponding to the process, and judging whether the task number of the memory release time in the resource container, which is greater than the preset time, exceeds a second threshold value.
In some embodiments, said locking a predetermined number of said CPU virtual cores therein as reserved CPUs comprises: and sending a signal that the reserved CPU resource is occupied to a resource manager.
Fig. 2 is a schematic structural diagram of a computer device according to an embodiment of the method for optimizing performance of a big data cluster provided in the present invention.
Taking the apparatus shown in fig. 2 as an example, the apparatus includes a processor 301 and a memory 302, and may further include: an input device 303 and an output device 304.
The processor 301, the memory 302, the input device 303 and the output device 304 may be connected by a bus or other means, and fig. 2 illustrates the connection by a bus as an example.
The memory 302 is used as a non-volatile computer-readable storage medium and can be used for storing non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the method for optimizing big data cluster performance in the embodiments of the present application. The processor 301 executes various functional applications of the server and data processing by running the non-volatile software programs, instructions and modules stored in the memory 302, i.e. implementing the method for optimizing the performance of a large data cluster of the above-described method embodiments.
The memory 302 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of a method of optimizing the performance of the big data cluster, and the like. Further, the memory 302 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 302 optionally includes memory located remotely from processor 301, which may be connected to a local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 303 may receive information such as a user name and a password that are input. The output means 304 may comprise a display device such as a display screen.
Program instructions/modules corresponding to one or more methods of optimizing big data cluster performance are stored in the memory 302 and, when executed by the processor 301, perform the method of optimizing big data cluster performance in any of the above-described method embodiments.
Any embodiment of a computer device implementing the method for optimizing large data cluster performance described above may achieve the same or similar effects as any of the preceding method embodiments corresponding thereto.
The invention also provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the method as above.
Finally, it should be noted that, as one of ordinary skill in the art can appreciate that all or part of the processes of the methods of the above embodiments can be implemented by a computer program to instruct related hardware, and the program of the method for optimizing the performance of a big data cluster can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods as described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Furthermore, the methods disclosed according to embodiments of the present invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. Which when executed by a processor performs the above-described functions defined in the methods disclosed in embodiments of the invention.
Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.
Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM is available in a variety of forms such as synchronous RAM (DRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.
The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A method for optimizing big data cluster performance is characterized by comprising the following steps:
creating a plurality of CPU virtual cores based on the number of CPU physical cores, and locking a preset number of the CPU virtual cores as reserved CPUs;
judging whether the CPU core number resources occupied by the current process number exceed a threshold value;
responding to the CPU core number resources occupied by the current process number exceeding a threshold value, and judging whether processes densely scheduled by the CPU exist or not; and
and responding to the process of the CPU intensive scheduling, unlocking the reserved CPU and executing the process of the CPU intensive scheduling by using the reserved CPU.
2. The method of claim 1, wherein the determining whether there is a process that is densely scheduled by the CPU comprises:
and sequentially judging whether dense response identification exists in each process.
3. The method of claim 1, wherein the determining whether there is a process that is densely scheduled by the CPU comprises:
and monitoring a resource container corresponding to the process, and judging whether the task number of the memory release time in the resource container, which is greater than the preset time, exceeds a second threshold value.
4. The method of claim 1, wherein locking a predetermined number of the CPU virtual cores as reserved CPUs comprises:
and sending a signal that the reserved CPU resource is occupied to a resource manager.
5. A system for optimizing large data cluster performance, comprising:
the system comprises a reservation module, a storage module and a locking module, wherein the reservation module is configured to create a plurality of CPU virtual cores based on the number of CPU physical cores and lock a predetermined number of the CPU virtual cores as reserved CPUs;
the first judgment module is configured to judge whether the CPU core number resource occupied by the current process number exceeds a threshold value;
the second judgment module is configured to respond to the situation that the number of CPU core resources occupied by the current process number exceeds a threshold value, and judge whether processes densely scheduled by the CPU exist or not; and
and the execution module is configured to respond to the process with the CPU intensive scheduling, unlock the reserved CPU and execute the process with the CPU intensive scheduling by using the reserved CPU.
6. The system of claim 5, wherein the second determining module is further configured to:
and sequentially judging whether dense response identification exists in each process.
7. The system of claim 5, wherein the second determining module is further configured to:
and monitoring a resource container corresponding to the process, and judging whether the task number of the memory release time in the resource container, which is greater than the preset time, exceeds a second threshold value.
8. The system of claim 5, wherein the reservation module is further configured to:
and sending a signal that the reserved CPU resource is occupied to a resource manager.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 4.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010132949.2A CN111338802B (en) | 2020-02-29 | 2020-02-29 | Method, system, equipment and medium for optimizing performance of big data cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010132949.2A CN111338802B (en) | 2020-02-29 | 2020-02-29 | Method, system, equipment and medium for optimizing performance of big data cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111338802A true CN111338802A (en) | 2020-06-26 |
| CN111338802B CN111338802B (en) | 2022-08-09 |
Family
ID=71185887
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010132949.2A Active CN111338802B (en) | 2020-02-29 | 2020-02-29 | Method, system, equipment and medium for optimizing performance of big data cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111338802B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112134752A (en) * | 2020-09-10 | 2020-12-25 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for monitoring switch based on BMC |
| CN112231066A (en) * | 2020-10-29 | 2021-01-15 | 北京思特奇信息技术股份有限公司 | JVM (Java virtual machine) memory use-based optimization processing method and system |
| CN113448749A (en) * | 2021-06-04 | 2021-09-28 | 山东英信计算机技术有限公司 | Method, system, device and medium for optimizing execution of expected timing task |
| CN113608834A (en) * | 2021-07-29 | 2021-11-05 | 济南浪潮数据技术有限公司 | Resource scheduling method, device and equipment based on super-fusion and readable medium |
| WO2024120205A1 (en) * | 2022-12-05 | 2024-06-13 | 中国科学院深圳先进技术研究院 | Method and apparatus for optimizing application performance, electronic device, and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105550040A (en) * | 2015-12-29 | 2016-05-04 | 四川中电启明星信息技术有限公司 | KVM platform based virtual machine CPU resource reservation algorithm |
| US20160179560A1 (en) * | 2014-12-22 | 2016-06-23 | Mrittika Ganguli | CPU Overprovisioning and Cloud Compute Workload Scheduling Mechanism |
| CN110442423A (en) * | 2019-07-09 | 2019-11-12 | 苏州浪潮智能科技有限公司 | A kind of method and apparatus for realizing that virtual machine reserves CPU using control group |
-
2020
- 2020-02-29 CN CN202010132949.2A patent/CN111338802B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160179560A1 (en) * | 2014-12-22 | 2016-06-23 | Mrittika Ganguli | CPU Overprovisioning and Cloud Compute Workload Scheduling Mechanism |
| CN105550040A (en) * | 2015-12-29 | 2016-05-04 | 四川中电启明星信息技术有限公司 | KVM platform based virtual machine CPU resource reservation algorithm |
| CN110442423A (en) * | 2019-07-09 | 2019-11-12 | 苏州浪潮智能科技有限公司 | A kind of method and apparatus for realizing that virtual machine reserves CPU using control group |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112134752A (en) * | 2020-09-10 | 2020-12-25 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for monitoring switch based on BMC |
| CN112134752B (en) * | 2020-09-10 | 2022-05-13 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for monitoring switch based on BMC |
| US11706086B1 (en) | 2020-09-10 | 2023-07-18 | Inspur Suzhou Intelligent Technology Co., Ltd. | Method and system for monitoring switch on basis of BMC, and device and medium |
| CN112231066A (en) * | 2020-10-29 | 2021-01-15 | 北京思特奇信息技术股份有限公司 | JVM (Java virtual machine) memory use-based optimization processing method and system |
| CN112231066B (en) * | 2020-10-29 | 2024-02-13 | 北京思特奇信息技术股份有限公司 | Optimization processing method and system based on JVM memory use |
| CN113448749A (en) * | 2021-06-04 | 2021-09-28 | 山东英信计算机技术有限公司 | Method, system, device and medium for optimizing execution of expected timing task |
| CN113448749B (en) * | 2021-06-04 | 2023-03-24 | 山东英信计算机技术有限公司 | Method, system, device and medium for optimizing execution of expected timing task |
| CN113608834A (en) * | 2021-07-29 | 2021-11-05 | 济南浪潮数据技术有限公司 | Resource scheduling method, device and equipment based on super-fusion and readable medium |
| WO2024120205A1 (en) * | 2022-12-05 | 2024-06-13 | 中国科学院深圳先进技术研究院 | Method and apparatus for optimizing application performance, electronic device, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111338802B (en) | 2022-08-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111338802B (en) | Method, system, equipment and medium for optimizing performance of big data cluster | |
| CN111756811B (en) | Method, system, device and medium for actively pushing distributed system | |
| JP6490913B2 (en) | Task execution by idle resources of grid computing system | |
| US10630566B1 (en) | Tightly-coupled external cluster monitoring | |
| CN108429631A (en) | Method and device for instantiating network services | |
| CN111367693B (en) | Method, system, device and medium for scheduling plug-in tasks based on message queue | |
| US10609123B2 (en) | Hybrid quorum policies for durable consensus in distributed systems | |
| CN113031870A (en) | Dynamic management method, device, storage medium and equipment for cluster system | |
| CN113608834A (en) | Resource scheduling method, device and equipment based on super-fusion and readable medium | |
| CN112433845A (en) | HBase service management method, device, equipment and readable medium | |
| CN106095483A (en) | The Automation arranging method of service and device | |
| CN106936931A (en) | The implementation method of distributed lock, relevant device and system | |
| CN111159195A (en) | Data storage control method and equipment in block chain system | |
| US9832281B2 (en) | Method, device and system for carrying out telecommunication capability group sending | |
| CN111930493A (en) | NodeManager state management method, device and computing device in cluster | |
| CN115617497B (en) | Thread processing method, scheduling component, monitoring component, server and storage medium | |
| CN112416594A (en) | Micro-service distribution method, electronic equipment and computer storage medium | |
| CN111104198A (en) | Method, equipment and medium for improving operation efficiency of scanning system plug-in | |
| Hassan et al. | Efficient resource scheduling for big data processing in cloud platform | |
| CN110780993A (en) | Kubernetes-based resource scheduling optimization method, equipment and medium | |
| CN118113485B (en) | Task execution method and device, storage medium and electronic equipment | |
| CN114629960A (en) | Resource scheduling method, device, system, device, medium, and program product | |
| CN109257256A (en) | Apparatus monitoring method, device, computer equipment and storage medium | |
| CN114281476A (en) | Kubernetes cloud native cluster resource deletion protection method, device, equipment and storage medium | |
| CN113986523A (en) | Method, system, equipment and medium for optimizing resource allocation of Flink system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |