CN116467066A - Method and device for allocating number bin resources, electronic equipment and storage medium - Google Patents
Method and device for allocating number bin resources, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116467066A CN116467066A CN202310247038.8A CN202310247038A CN116467066A CN 116467066 A CN116467066 A CN 116467066A CN 202310247038 A CN202310247038 A CN 202310247038A CN 116467066 A CN116467066 A CN 116467066A
- Authority
- CN
- China
- Prior art keywords
- task
- processor
- processing
- structured query
- bin
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5022—Mechanisms to release resources
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, a device, electronic equipment and a storage medium for allocating a plurality of bins of resources, wherein the method for allocating the plurality of bins of resources comprises the following steps: responding to a task processing request sent by a target object, and acquiring a structured query language in the task processing request and the target processing quantity of a task processor; comparing the target processing number of the task processor with the number of the structured query language to obtain a comparison result; and allocating task processors in the multi-bin task system according to the comparison result to perform structured query language processing. By the scheme, the resource utilization rate and the task processing efficiency of the multi-bin task system can be improved.
Description
Technical Field
The present disclosure relates to the field of data warehouse technologies, and in particular, to a method and apparatus for allocating a plurality of resources, an electronic device, and a storage medium.
Background
With the modern development of society, various data become indispensable demands in development, and big data technology becomes a hot spot technology.
Currently, big data technology is very mature, for example, a data warehouse tool Hive developed by Apache company based on Hadoop is commonly used, and massive data in a data warehouse can be inquired and analyzed through Hive.
In order to realize the concurrent processing of the multi-bin tasks, more resources are required to be allocated before the multi-bin tasks are submitted, but the resources required after the multi-bin tasks are submitted are very few, so that the resource waste is caused in the multi-bin task processing process.
Disclosure of Invention
The application provides at least a method, a device, equipment and a computer readable storage medium for allocating a plurality of bin resources.
The first aspect of the present application provides a method for allocating a plurality of bins, where the method is applied to a task manager in a plurality of bins task system, and the plurality of bins task system further includes a task processor, and the method includes: responding to a task processing request sent by a target object, and acquiring a structured query language in the task processing request and the target processing quantity of the task processor; comparing the target processing number of the task processor with the number of the structured query language to obtain a comparison result; and allocating task processors in the multi-bin task system to perform structured query language processing according to the comparison result.
In an embodiment, the comparing result includes that the target processing number of the task processors is less than or equal to the number of the structured query languages, and the step of allocating the task processors in the multi-bin task system to perform the structured query language processing according to the comparing result includes: if the target processing number of the task processors is smaller than or equal to the number of the structured query language, requesting a new task processor from the task system with a plurality of bins; and forwarding the task processing request to the newly-added task processor for structured query language processing.
In an embodiment, the comparing result includes that the target processing number of the task processors is less than or equal to the number of the structured query languages, and the step of allocating the task processors in the multi-bin task system to perform the structured query language processing according to the comparing result includes: if the target processing number of the task processor is smaller than or equal to the number of the structured query languages and the number of the structured query languages is smaller than the preset processing number of the task processor, the task processor waits for the task processor to process the existing number of bins of tasks, and then forwards the task processing request to the task processor for structured query language processing.
In an embodiment, the comparing result includes that the target processing number of the task processors is less than or equal to the number of the structured query languages, and the step of allocating the task processors in the multi-bin task system to perform the structured query language processing according to the comparing result includes: if the target processing number of the task processor is smaller than or equal to the number of the structured query languages and the number of the structured query languages is larger than or equal to the preset processing number of the task processor, requesting new resources to the multi-bin task system, wherein the new resources are used for creating new task processors; and after the task processor in the multi-bin task system finishes the existing multi-bin task processing, forwarding the task processing request to the task processor in the multi-bin task system and the newly-added task processor to perform structural query language processing.
In an embodiment, the comparing result includes that the target processing number of the task processors is greater than the number of the structured query languages, and the step of allocating the task processors in the multi-bin task system to perform the structured query language processing according to the comparing result includes: and if the target processing number of the task processor is greater than the number of the structured query languages, forwarding the task processing request to the task processor so that the task processor performs structured query language processing.
In an embodiment, the method further comprises: detecting a target state of a task processor in the bin-counting task system, wherein the target state comprises an idle state; destroying task processors in the idle state of the task processors in the bin-counting task system; and applying for recycling the resource data corresponding to the task processor in the idle state to the bin-counting task system.
In one embodiment, the step of performing target state detection on the task processor in the multi-bin task system includes: searching task processors with target processing quantity equal to preset processing quantity; idle timing is carried out on the task processors with the target processing quantity equal to the preset processing quantity, so that idle time length is obtained; and if the idle time length is greater than a preset idle threshold value, judging that the task processor with the idle time length greater than the idle threshold value is in the idle state.
In an embodiment, the step of obtaining the structured query language in the task processing request and the target processing number of the task processor includes: polling each task processor according to the number of the structured query language to obtain the number of the task bins being processed in each task processor; and calculating the difference between the preset processing quantity of each task processor and the number of the processing bins of the task processor to obtain the target processing quantity.
In an embodiment, the step of obtaining the structured query language in the task processing request and the target processing number of the task processor includes: screening each task processor according to the resource occupancy rate of each task processor to obtain task processors with the resource occupancy rate smaller than a preset resource occupancy threshold; and taking the target processing quantity of the task processor with the resource occupancy rate smaller than a preset resource occupancy threshold value as the target processing quantity of the task processor.
The second aspect of the present application provides a several-bin resource allocation device, including: the acquisition module is used for responding to a task processing request sent by a target object and acquiring a structured query language in the task processing request and the target processing quantity of the task processor; the comparison module is used for comparing the target processing number of the task processor with the number of the structured query language to obtain a comparison result; and the allocation module is used for allocating the task processor in the multi-bin task system to perform structured query language processing according to the comparison result.
The third aspect of the present application provides an electronic device, including a memory and a processor, where the processor is configured to execute program instructions stored in the memory, so as to implement the method for allocating a plurality of bins of resources.
A fourth aspect of the present application provides a computer readable storage medium having program instructions stored thereon, which when executed by a processor implement the above-described method for allocating resources in bins.
According to the scheme, whether the task processor can process the task request is judged according to the received task processing request, and the task processor in the multi-bin task system is allocated according to the judging result to perform structural query language processing, so that task management and task processing in the multi-bin task system can be separated, the purpose of allocating resources of the multi-bin task system is achieved by allocating the task processor, and the resource utilization rate and the task processing efficiency of the multi-bin task system are improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the technical aspects of the application.
FIG. 1 is a schematic view of an application environment of an exemplary embodiment of a method for allocating a plurality of resources in a plurality of bins of the present application;
FIG. 2 is a flow diagram of an exemplary embodiment of a method of allocating a plurality of bins of the present application;
FIG. 3 is a schematic flow chart of a task allocation processor in the method for allocating a plurality of resources in the present application;
FIG. 4 is a flow diagram of an exemplary embodiment of a task processor being deployed based on different comparison results in a method of deploying a multi-bin resource of the present application;
FIG. 5 is a flow diagram of another exemplary embodiment of a provisioning task processor in a multi-bin resource provisioning method of the present application;
FIG. 6 is a block diagram of a several-bin resource allocation device according to an exemplary embodiment of the present application;
FIG. 7 is a schematic diagram of an embodiment of an electronic device of the present application;
FIG. 8 is a schematic diagram of an embodiment of a computer-readable storage medium of the present application.
Detailed Description
The following describes the embodiments of the present application in detail with reference to the drawings.
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, interfaces, techniques, etc., in order to provide a thorough understanding of the present application.
The term "and/or" is herein merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship. Further, "a plurality" herein means two or more than two. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
For ease of understanding, some of the techniques referred to herein are briefly described as including the main execution flow of Hive tasks in the prior art:
the JDBC client is connected to Hiveserver2 (HS 2) and submits SQL to HS 2; after the HS2 receives the SQL sent by the client, processing modes such as analysis, compiling and the like are performed on the SQL, and the SQL is converted into a logic execution plan; selecting an execution engine according to the configuration information of the client, currently supporting the engine such as MR, spark, tez, translating the execution plan into a physical plan corresponding to the execution engine, and optimizing the physical plan; and packaging the physical plan and the context information and submitting the packaged physical plan and the context information to an execution engine for execution. Taking Spark as an example, this step starts a Spark task commit sub-process for the HS2 process. After the Spark task is started, the HS2 monitors the execution of the Spark task, and the client is notified after the execution is completed.
For Hive data warehouse, the current container degree of Hive is low, and the resource utilization is not flexible enough. Based on the task submission flow of the prior Hive, the Hive only divides the hiveServer2 and the Metastore into two services for deployment when in containerized deployment. The hiveServer2 occupies larger resources when deployed, and the task processing processes such as analysis, compiling, optimization and submission of the Hive task are all carried out in the hiveServer2 service, but the task processing processes are very short in terms of the processing consumption of Spark tasks, and the resources consumed in the process of monitoring the Spark tasks after the completion and submission of the Hive task are very few. Therefore, in the whole Hive task, the hiveServer2 service only needs larger resources before the task is submitted, and the resources needed in other time are very few, but the proportion of the total time of the task before the task is submitted is very few, so that very serious resource waste is caused.
Therefore, the method for allocating the resources of the multiple bins improves the resource utilization rate and the task processing efficiency of the task system of the multiple bins.
It should be noted that, the method for allocating the resources in the plurality of bins is applied to a task manager in a task system in the plurality of bins, and the task system in the plurality of bins further comprises a task processor, and the task manager and the task processor are software modules which are respectively and independently deployed in the task system in the plurality of bins by utilizing a containerization technology. Taking a Hive data warehouse based on a Kubernetes system as an example, referring to fig. 1, fig. 1 is a schematic view of an application environment of a multi-bin task system of the present application, including a Hive system architecture, and a multi-bin resource allocation method of the present application is exemplarily described below around the application environment shown in fig. 1.
Referring to fig. 2, fig. 2 is a flow chart of an exemplary embodiment of a method for allocating a plurality of resources in a bin according to the present application. Specifically, the method may include the steps of:
step S210, in response to the task processing request sent by the target object, the structured query language in the task processing request and the target processing number of the task processor are obtained.
The target object may be a device such as a notebook computer, a tablet computer, a personal computer, and a mobile phone, which are used as clients, and is not limited herein.
Task processing requests refer primarily to structured query language (Structured Query Language, SQL) requests, i.e., SQL requests, which include structured query language.
The target processing number of the task processor refers to the number of tasks that the task processor can currently process, i.e., the remaining processing capacity of the task processor.
After receiving the SQL request sent by the target object, the SQL in the SQL request is added into a queue to be processed, the SQL in the queue to be processed is circularly consumed, and the target processing quantity of each task processor is polled for each SQL in the queue to be processed, so that the quantity of the structured query language and the target processing quantity of the task processors are obtained.
Before responding to a task processing request sent by a target object, the task system with a plurality of bins needs to be successfully started, and the starting process of the task system with a plurality of bins comprises: a request management module (clientrequest manager) in the boot task manager for managing client requests, a service management module (hs 2CliServiceMan ager) in the boot task manager for managing Hive services, and a task processor (hs 2 client) for processing Hive tasks. After the task system of the several bins is successfully started, the Hive service is in an available state, and then the task processing request sent by the corresponding target object can be received.
Specifically, the start-up procedure of the Hive service system is as follows: starting a clientrequest manager module; starting an hs2CliServiceManager module, reading the size of an hs2Cliser device resource pre-configured by a user, and if the resource is not configured, using default configuration; an initial hs2Cliser device service is started.
It should be further noted that, after the task manager establishes a connection with the clientrequest manager module in the client, the task manager records the connection, and after receiving a request sent by the client, queries whether there is an already-bound application manager (ApplicationMaster, AM) between the task manager and the client and whether the configuration of the client changes according to the connection. If the AM binding with the client is inquired and the configuration of the client is not changed, the AM information and the client request are sent to the hs2CliServiceManager. If there is binding AM between the client and the client but the configuration of the client changes, the binding AM is logged off through the queried information of the AM, and the client request is only sent to hs2CliServiceManager, and the AM information is null. If there is no binding AM with the client, the direct hs2CliServiceManager sends the client request, and the AM information is null. Wait for hs2CliServiceManager to return the result. This flow is to ensure that an AM that has been created between the task manager and the client can be reused. For the connection established between the task manager and the same client, after the first task is completed and the connection is not disconnected, and the resources requested again by the client are unchanged, the resources which have been applied before can be reused, unnecessary task resource application is reduced, and efficiency is improved.
Step S220, comparing the target processing number of the task processor and the number of the structured query language to obtain a comparison result.
Illustratively, by comparing the number relationship between the target processing number of the current task processor and the number of SQL in the queue to be processed, it is determined whether the task processor has sufficient computational resources to process the SQL in the queue.
And step S230, allocating task processors in the multi-bin task system to perform structured query language processing according to the comparison result.
Among other things, the process of deploying task processors in a binned task system includes, but is not limited to: calling a task processor, adding a task processor and destroying the task processor.
Structured query language processing, also known as SQL processing, includes, but is not limited to: and analyzing, compiling, optimizing and submitting SQL.
It should be noted that, the task processor needs to occupy the resources in the task system with a plurality of bins to perform the structured query language processing, so that the task processor is allocated to allocate the resources of the task system with a plurality of bins.
Specifically, before starting the task processor, a resource needs to be applied to Kubernetes to start the pod of hs2Cliservice, where the pod is the minimum management unit of Kubernetes, and may include one or more containers, enter a waiting state if there is no resource, perform waiting timing to obtain a waiting duration, and if the waiting duration is greater than a preset waiting threshold, suspend the starting of the task processor. If there are resources, the Kubernetes allocates resources to start the task processor hs2Cliser device.
It can be seen that the task processor can process the task request according to the received task processing request, and the task processor in the task system with a plurality of bins is allocated according to the judging result to perform structured query language processing, so that task management and task processing in the task system with a plurality of bins can be separated, the task processor is allocated to fulfill the aim of allocating resources of the task system with a plurality of bins, and the resource utilization rate and the task processing efficiency of the task system with a plurality of bins are improved.
Based on the above embodiments, the embodiment of the present application adopts a flowchart as shown in fig. 3 to describe in detail how to allocate task processors according to the comparison result if the comparison result is that the target processing number of the task processors is less than or equal to the number of the structured query languages, please refer to fig. 3, and fig. 3 is a flowchart of an exemplary embodiment of step 230 in the method for allocating multiple resources shown in fig. 2. Specifically, the method of the embodiment comprises the following steps:
step 310, if the target processing number of task processors is less than or equal to the number of structured query languages, requesting a new task processor from the multi-bin task system.
Step 320, forwarding the task processing request to the newly added task processor for structured query language processing.
The newly added task processor refers to a newly added task processor which is started by applying resources to a kubernetes system in the multi-bin task system by a task manager in addition to the currently existing task processor. After the newly added task processor is successfully started, the task manager registers the newly added task processor so that the task manager and the newly added task processor can interact.
It can be appreciated that, under the condition that the comparison result is that the target processing number of the task processor is less than or equal to the number of the structured query languages, the task processor does not have enough processing capability to process the structured query languages to be processed currently, so that additional resources can be applied to the task system with a plurality of bins to obtain a new task processor.
It should be noted that, under the condition that the target processing number of the task processor is equal to the number of the structured query languages, the task processor may be considered to be in a full-load running state, and the task processor does not have enough fault-tolerant space, which is dangerous for the whole system, so that in a normal case, the task processor is not allowed to run fully as much as possible.
Further, the task manager forwards the task processing request to the newly added task processor, so that the newly added task processor and the task processor establish a remote procedure call (Remote Procedure Call, RPC) connection, and perform structured query language processing according to the forwarded task processing request. The structured query language processing, namely SQL processing, mainly comprises tasks of compiling, submitting and the like of SQL. The task processing request includes context information required for SQL compilation and submission of tasks, such as information for a task management module (JobManager), and jar packages (Java Archive File).
It should be further noted that, the target processing number of the task processor is determined by the number of tasks that the task processor can currently concurrently compile (for convenience of description, hereinafter referred to as residual compiling capability) and the number of tasks that can also be concurrently submitted (for convenience of description, hereinafter referred to as residual submitting capability). The target processing number takes the minimum value of the residual compiling capacity and the residual submitting capacity, and the mathematical expression of the target processing number is as follows:
target processing number = Min (remaining compiling capability, remaining submitting capability)
If the target processing number is greater than 0, the task processor is not in a full-load running state, and a certain amount of SQL can be received and processed.
Further, if the spark task needs to be executed, whether the context information has the bound AM information or not is judged, and if so, the context information is directly submitted to the AM. Otherwise, starting the job sub, returning the AM information to the JobManager after the job sub is executed successfully, monitoring the spark task by the JobManager, and binding the AM with the client. If the spark task does not need to be executed, the task state and the result are returned to the JobManager, and then returned to the client.
Based on the above embodiments, the embodiments of the present application describe how to allocate the processes of the task processors according to the comparison result on the condition that the comparison result is that the target processing number of the task processors is less than or equal to the number of the structured query languages. Specifically, the method of the embodiment comprises the following steps:
If the target processing number of the task processor is smaller than or equal to the number of the structured query languages and the number of the structured query languages is smaller than the preset processing number of the task processor, the task processor waits for the task processor to process the existing number of bins of tasks, and then forwards a task processing request to the task processor for structured query language processing.
The preset processing quantity refers to the maximum processing capacity of the task processor for SQL processing.
It can be understood that, for the condition that the target processing number of the task processor is less than or equal to the number of the structured query languages, and the number of the structured query languages is less than the preset processing number of the task processor, that is, the condition that the task processor can process the structured query languages to be processed after completing the processing of the task of the currently existing bins is explained. Therefore, the task manager waits for the task processor to finish the existing task processing of the plurality of bins, and forwards the task processing request to the task processor for structured query language processing.
Preferably, since the target processing number of the task processor varies according to the completion of the existing multi-bin task, the task processor can be supervised in real time or in a certain frequency, and if the target processing number of the task processor is monitored to be greater than the number of the structured query languages to be processed, the task processing request can be forwarded to the task processor instead of waiting for the task processor to complete the existing multi-bin task.
Based on the above embodiments, the embodiments of the present application describe how to allocate the processes of the task processors according to the comparison result on the condition that the comparison result is that the target processing number of the task processors is less than or equal to the number of the structured query languages. Specifically, the method of the embodiment comprises the following steps:
if the target processing number of the task processor is smaller than or equal to the number of the structured query languages and the number of the structured query languages is larger than or equal to the preset processing number of the task processor, requesting a new resource to the multi-bin task system, wherein the new resource is used for creating the new task processor; after the task processor in the multi-bin task system finishes the processing of the existing multi-bin task, the task processing request is forwarded to the task processor and the newly-added task processor in the multi-bin task system for structural query language processing.
Illustratively, the newly added resource is obtained by applying an hs2CliServiceManager in the task manager to a kubernetes system in the multi-bin task system.
Specifically, for the condition that the target processing number of the task processor is less than or equal to the number of the structured query languages, and the number of the structured query languages is greater than or equal to the preset processing number of the task processor, it is stated that after the task processor finishes the existing several bins of task processing, there is still insufficient capability to process the structured query language to be processed currently, so that an additional task processor, i.e. a new task processor needs to be created. After the previous task processor in the task system finishes the existing task processing of the multiple bins, the task processing request to be processed is forwarded to the previous task processor and an additional new task processor to process the structured query language, and the task processor can only have enough processing capacity to process the structured query language to be processed currently.
Based on the above embodiments, the embodiments of the present application describe how to allocate the processes of the task processors according to the comparison result on the condition that the comparison result is that the target processing number of the task processors is greater than the number of the structured query languages. Specifically, the method of the embodiment comprises the following steps:
if the target processing number of the task processor is greater than the number of the structured query languages, forwarding the task processing request to the task processor so that the task processor performs structured query language processing.
It will be appreciated that for conditions where the target number of processing by the task processor is greater than the number of structured query languages, it is stated that the task processor has sufficient processing power to process the structured query languages, and therefore the task processing request may be forwarded directly to the task processor to cause the task processor to process the structured query languages.
For ease of understanding the various comparison results, further reference may be made to fig. 4, where fig. 4 is a schematic flow chart of the task processor configured under the various comparison results, and it may be understood in conjunction with fig. 4 and the above embodiments.
It should be noted that, the steps of the flow shown in the flow diagram are not exhaustive or limiting, but only illustrate some of the steps in the method for allocating the resources in several bins.
Based on the above embodiments, the present application further describes how to allocate the task processor according to the comparison result on the condition that the comparison result is that the target processing number of the task processor is greater than the number of the structured query languages by using the flow chart shown in fig. 5. Specifically, the method of the embodiment comprises the following steps:
step 510, performing target state detection on task processors in the log bin task system, wherein the target state comprises an idle state.
It should be noted that, the target state detection of the task processor is to obtain the running state of the task processor, and the idle state refers to that the task processor does not process a plurality of bins of tasks, which may be represented as that the target processing number of the task processor is equal to the preset processing number.
Step 520, destroying task processors in idle state in the task processor in the bin-counting task system.
Destruction refers to a method of processing computer resources such as process destruction.
When detecting that the task processor in the idle state exists in the multi-bin task system, the task processor in the idle state can be destroyed in order to avoid the waste of resources.
Preferably, in order to ensure the data processing efficiency of the multi-bin task system, when the number of task processors in an idle state is larger than the preset idle number, the task processors exceeding the preset idle number can be destroyed, so that the task processors in the idle state in the multi-bin task system are ensured to be used as spare, resources can be saved, and sudden increase of the multi-bin task can be prevented.
And 530, applying for recycling the resource data corresponding to the task processor in the idle state from the multi-bin task system.
The resource data includes address space, memory, etc. occupied by the task processor.
Specifically, after destroying the task processor in the idle state, the resource data corresponding to the destroyed task processor is also released, so that an application is initiated to the multi-bin task system, and the multi-bin task system is informed to recover the resource data, so that the resource is reused.
It should be noted that, when each task processor is started, the task manager correspondingly registers, and when the task processor is destroyed, the destroyed task processor is also correspondingly destroyed, and removed from the registration list.
It should be noted that, after the process of the task processor is started and registered, if the task processor does not have SQL processing, the task manager receives idle heartbeat information sent by the task processor according to the heartbeat detection mechanism, for example, the idle heartbeat information is sent once in 60 seconds, so as to avoid the task processor being recovered due to idle; if the task processor has SQL processing, the task manager receives SQL processing heartbeat information sent by the task processor, wherein the SQL processing heartbeat information comprises the current processed SQL quantity, the to-be-processed SQL quantity, the use state of the memory outside the heap and the like, and the SQL processing heartbeat information can be set to be sent once in 10 seconds.
On the basis of the above embodiments, the present embodiment exemplarily describes a target state detection process of a task manager. Specifically, the method of the embodiment comprises the following steps:
searching task processors with target processing quantity equal to preset processing quantity; idle timing is carried out on task processors with the target processing quantity equal to the preset processing quantity, so that idle time length is obtained; if the idle time length is greater than the preset idle threshold, judging that the task processor with the idle time length greater than the idle threshold is in an idle state.
Specifically, the task processors with the target processing number equal to the preset processing number can be regarded as task processors without processing the tasks with a plurality of bins, namely, task processors in an idle state, and when the task processors are in the idle state for too long, the task processors can be destroyed and corresponding resources can be recovered in order to avoid wasting resources.
On the basis of the above-described embodiments, the present application exemplarily describes the acquisition steps of the target processing number. Specifically, the method of the embodiment comprises the following steps:
polling each task processor according to the number of the structured query language to obtain the number of the task bins being processed in each task processor; and calculating the difference between the preset processing quantity of each task processor and the number of the processing bins of the task processor to obtain the target processing quantity.
It should be noted that, for each to-be-processed SQL task manager, whether each task processor can process the SQL is polled, if not, whether the current number of to-be-processed SQL is greater than the target number of to-be-processed SQL of the task processor is judged, and then the allocation mode of the task processor is determined.
On the basis of the above-described embodiments, the present application exemplarily describes the acquisition steps of the target processing number. Specifically, the method of the embodiment comprises the following steps:
screening each task processor according to the resource occupancy rate of each task processor to obtain task processors with the resource occupancy rate smaller than a preset resource occupancy threshold; and taking the target processing quantity of the task processor with the resource occupancy rate smaller than the preset resource occupancy threshold value as the target processing quantity of the task processor.
It should be noted that, the determination of whether the task processor can continue to process the SQL may be performed not only according to the target processing number and the SQL number of the task manager in the above embodiment, but also according to the resources occupied by the task processor.
The resource occupancy rate includes a memory occupancy rate, and the memory usage condition corresponding to each task processor is determined according to the memory occupancy rate, if the occupancy rate in or out of the heap memory is greater than a preset memory occupancy rate, in order to avoid that after SQL compiling, garbage collection (Garbage Collection, GC) is frequently triggered in the heap due to insufficient memory, so that compiling is too slow or resources outside the heap are insufficient to submit tasks, which results in overlong or even failure of task waiting time, the task processors are screened, and the task processors with the resource occupancy rate smaller than a preset resource occupancy threshold are used as available task processors, and the corresponding target processing quantity is used as target processing quantity.
It should be further noted that, the execution body of the several-bin resource allocation method of the present application may be a several-bin resource allocation device applied to a task manager of a several-bin task system, for example, the several-bin resource allocation method may be executed by a terminal device or a server or other processing devices, where the terminal device may be a User Equipment (UE), a computer, a mobile device, a User terminal, a cellular phone, a cordless phone, a personal digital processing (Personal Digital Assistant, PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like. In some possible implementations, the method of allocating resources in bins may be implemented by a processor invoking computer readable instructions stored in a memory.
FIG. 6 is a block diagram of a bin resource allocation device, which may be used in a task manager of a bin task system, as shown in an exemplary embodiment of the present application. As shown in fig. 6, the exemplary several-bin resource allocation apparatus 600 includes: an acquisition module 610, a comparison module 620, and a deployment module 630. Specifically:
an obtaining module 610, configured to obtain, in response to a task processing request sent by a target object, a structured query language in the task processing request and a target processing number of a task processor.
And the comparison module 620 is configured to compare the target processing number of the task processor and the number of the structured query language to obtain a comparison result.
And the allocating module 630 is used for allocating the task processor in the task system with the number bin according to the comparison result to perform the structured query language processing.
In the exemplary task processing device, whether the task processor can process the task request is judged according to the received task processing request, and the task processor in the task system with a plurality of bins is allocated according to the judging result to perform structured query language processing, so that task management and task processing in the task system with a plurality of bins can be separated, the task processor is allocated to fulfill the aim of allocating the resources of the task system with a plurality of bins, and the resource utilization rate and the task processing efficiency of the task system with a plurality of bins are improved.
The functions of each module may be described in the embodiments of the method for allocating several bins of resources, which are not described herein.
Referring to fig. 7, fig. 7 is a schematic structural diagram of an embodiment of an electronic device of the present application. The electronic device 700 comprises a memory 701 and a processor 702, the processor 702 being configured to execute program instructions stored in the memory 701 to implement the steps of any of the above-described embodiments of the method for allocating resources in bins. In one particular implementation scenario, electronic device 700 may include, but is not limited to: the microcomputer and the server, and the electronic device 700 may also include a mobile device such as a notebook computer and a tablet computer, which is not limited herein.
Specifically, the processor 702 is configured to control itself and the memory 701 to implement the steps in any of the several-bin resource allocation method embodiments described above. The processor 702 may also be referred to as a CPU (Central Processing Unit ). The processor 702 may be an integrated circuit chip with signal processing capabilities. The processor 702 may also be a general purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a Field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 702 may be commonly implemented by an integrated circuit chip.
According to the scheme, whether the task processor can process the task request is judged according to the received task processing request, and the task processor in the multi-bin task system is allocated according to the judging result to perform structural query language processing, so that task management and task processing in the multi-bin task system can be separated, the purpose of allocating resources of the multi-bin task system is achieved by allocating the task processor, and the resource utilization rate and the task processing efficiency of the multi-bin task system are improved.
Referring to fig. 8, fig. 8 is a schematic structural diagram of an embodiment of a computer readable storage medium of the present application. The computer readable storage medium 800 stores program instructions 801 that can be executed by a processor, where the program instructions 801 are configured to implement the steps in any of the above-described embodiments of a method for allocating resources in bins.
According to the scheme, whether the task processor can process the task request is judged according to the received task processing request, and the task processor in the multi-bin task system is allocated according to the judging result to perform structural query language processing, so that task management and task processing in the multi-bin task system can be separated, the purpose of allocating resources of the multi-bin task system is achieved by allocating the task processor, and the resource utilization rate and the task processing efficiency of the multi-bin task system are improved.
In some embodiments, functions or modules included in an apparatus provided by the embodiments of the present disclosure may be used to perform a method described in the foregoing method embodiments, and specific implementations thereof may refer to descriptions of the foregoing method embodiments, which are not repeated herein for brevity.
The foregoing description of various embodiments is intended to highlight differences between the various embodiments, which may be the same or similar to each other by reference, and is not repeated herein for the sake of brevity.
In the several embodiments provided in the present application, it should be understood that the disclosed methods and apparatus may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules or units is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical, or other forms.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units. The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all or part of the technical solution contributing to the prior art or in the form of a software product stored in a storage medium, including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Claims (12)
1. A method for allocating a plurality of bins of resources, wherein the method is applied to a task manager in a plurality of bins of task systems, the plurality of bins of task systems further comprise task processors, and the method comprises:
responding to a task processing request sent by a target object, and acquiring a structured query language in the task processing request and the target processing quantity of the task processor;
comparing the target processing number of the task processor with the number of the structured query language to obtain a comparison result;
and allocating task processors in the multi-bin task system to perform structured query language processing according to the comparison result.
2. The method according to claim 1, wherein the comparison result includes that the target processing number of the task processors is less than or equal to the number of structured query languages, and the step of allocating task processors in the multi-bin task system to perform structured query language processing according to the comparison result includes:
if the target processing number of the task processors is smaller than or equal to the number of the structured query language, requesting a new task processor from the task system with a plurality of bins;
And forwarding the task processing request to the newly-added task processor for structured query language processing.
3. The method according to claim 1, wherein the comparison result includes that the target processing number of the task processors is less than or equal to the number of structured query languages, and the step of allocating task processors in the multi-bin task system to perform structured query language processing according to the comparison result includes:
if the target processing number of the task processor is smaller than or equal to the number of the structured query languages and the number of the structured query languages is smaller than the preset processing number of the task processor, the task processor waits for the task processor to process the existing number of bins of tasks, and then forwards the task processing request to the task processor for structured query language processing.
4. The method according to claim 1, wherein the comparison result includes that the target processing number of the task processors is less than or equal to the number of structured query languages, and the step of allocating task processors in the multi-bin task system to perform structured query language processing according to the comparison result includes:
If the target processing number of the task processor is smaller than or equal to the number of the structured query languages and the number of the structured query languages is larger than or equal to the preset processing number of the task processor, requesting new resources to the multi-bin task system, wherein the new resources are used for creating new task processors;
and after the task processor in the multi-bin task system finishes the existing multi-bin task processing, forwarding the task processing request to the task processor in the multi-bin task system and the newly-added task processor to perform structural query language processing.
5. The method of claim 1, wherein the comparison result includes a target processing number of the task processors being greater than the number of structured query languages, and wherein the step of deploying the task processors in the multi-bin task system for structured query language processing based on the comparison result comprises:
and if the target processing number of the task processor is greater than the number of the structured query languages, forwarding the task processing request to the task processor so that the task processor performs structured query language processing.
6. The method of claim 5, wherein the method further comprises:
detecting a target state of a task processor in the bin-counting task system, wherein the target state comprises an idle state;
destroying task processors in the idle state of the task processors in the bin-counting task system;
and applying for recycling the resource data corresponding to the task processor in the idle state to the bin-counting task system.
7. The method of claim 6, wherein the step of performing target state detection on a task processor in the binned task system comprises:
searching task processors with target processing quantity equal to preset processing quantity;
idle timing is carried out on the task processors with the target processing quantity equal to the preset processing quantity, so that idle time length is obtained;
and if the idle time length is greater than a preset idle threshold value, judging that the task processor with the idle time length greater than the idle threshold value is in the idle state.
8. The method of claim 1, wherein the step of obtaining the structured query language in the task processing request and the target processing number of the task processor comprises:
Polling each task processor according to the number of the structured query language to obtain the number of the task bins being processed in each task processor;
and calculating the difference between the preset processing quantity of each task processor and the number of the processing bins of the task processor to obtain the target processing quantity.
9. The method of claim 1, wherein the step of obtaining the structured query language in the task processing request and the target processing number of the task processor comprises:
screening each task processor according to the resource occupancy rate of each task processor to obtain task processors with the resource occupancy rate smaller than a preset resource occupancy threshold;
and taking the target processing quantity of the task processor with the resource occupancy rate smaller than a preset resource occupancy threshold value as the target processing quantity of the task processor.
10. A multi-bin resource allocation device, wherein the device is applied to a task manager in a multi-bin task system, the multi-bin task system further comprises a task processor, and the device comprises:
the acquisition module is used for responding to a task processing request sent by a target object and acquiring a structured query language in the task processing request and the target processing quantity of the task processor;
The comparison module is used for comparing the target processing number of the task processor with the number of the structured query language to obtain a comparison result;
and the allocation module is used for allocating the task processor in the multi-bin task system to perform structured query language processing according to the comparison result.
11. An electronic device comprising a memory and a processor for executing program instructions stored in the memory to implement the method of any one of claims 1 to 9.
12. A computer readable storage medium having stored thereon program instructions, which when executed by a processor, implement the method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310247038.8A CN116467066A (en) | 2023-03-09 | 2023-03-09 | Method and device for allocating number bin resources, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310247038.8A CN116467066A (en) | 2023-03-09 | 2023-03-09 | Method and device for allocating number bin resources, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116467066A true CN116467066A (en) | 2023-07-21 |
Family
ID=87181468
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310247038.8A Pending CN116467066A (en) | 2023-03-09 | 2023-03-09 | Method and device for allocating number bin resources, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116467066A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117009327A (en) * | 2023-09-27 | 2023-11-07 | 腾讯科技（深圳）有限公司 | Data processing method and device, computer equipment and medium |
| CN119130089A (en) * | 2024-11-13 | 2024-12-13 | 苏州仕泰得智能科技有限公司 | Warehouse system scheduling control method, device, equipment and storage medium |
-
2023
- 2023-03-09 CN CN202310247038.8A patent/CN116467066A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117009327A (en) * | 2023-09-27 | 2023-11-07 | 腾讯科技（深圳）有限公司 | Data processing method and device, computer equipment and medium |
| CN117009327B (en) * | 2023-09-27 | 2024-01-05 | 腾讯科技（深圳）有限公司 | Data processing method and device, computer equipment and medium |
| CN119130089A (en) * | 2024-11-13 | 2024-12-13 | 苏州仕泰得智能科技有限公司 | Warehouse system scheduling control method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11467890B2 (en) | Processing event messages for user requests to execute program code | |
| CA2962633C (en) | Programmatic event detection and message generation for requests to execute program code | |
| US20200004571A1 (en) | Operating system customization in an on-demand network code execution system | |
| US8583756B2 (en) | Dynamic configuration and self-tuning of inter-nodal communication resources in a database management system | |
| CN107832100B (en) | APK plug-in loading method and terminal thereof | |
| CN116467066A (en) | Method and device for allocating number bin resources, electronic equipment and storage medium | |
| CN111290838B (en) | Application access request processing method and device based on container cluster | |
| CN108762931A (en) | Method for scheduling task, server based on distributed scheduling system and storage medium | |
| CN109614167B (en) | Method and system for managing plug-ins | |
| US11144359B1 (en) | Managing sandbox reuse in an on-demand code execution system | |
| US11228658B1 (en) | Pre-caching data for use upon execution of program code | |
| CN110659104B (en) | Service monitoring method and related equipment | |
| WO2019228346A1 (en) | Method and apparatus for executing task by scheduling device | |
| CN116795647A (en) | A database heterogeneous resource management and scheduling method, device, equipment and medium | |
| CN113032153A (en) | Dynamic capacity expansion method, system, device and storage medium for container service resources | |
| CN113157411A (en) | Reliable configurable task system and device based on Celery | |
| CN112286688A (en) | Memory management and use method, device, equipment and medium | |
| CN110928888A (en) | Data feedback method, device, server and storage medium | |
| CN114567507A (en) | Current limiting method, system, device and storage medium | |
| CN112559132B (en) | Safe static detection method and device for containerized deployment application | |
| CN113905091A (en) | Method and device for processing access request | |
| US10846156B2 (en) | Methods, devices and computer program products for managing software function | |
| CN114040378B (en) | Application orchestration methods, devices, computer equipment and storage media | |
| CN113703976B (en) | A FPGA resource allocation method, device, equipment and readable storage medium | |
| US20100199281A1 (en) | Managing the Processing of Processing Requests in a Data Processing System Comprising a Plurality of Processing Environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |