CN105049268A - Distributed computing resource allocation system and task processing method - Google Patents

Abstract

The present invention provides a distributed computing resource allocation system and task processing method. The system includes: a central server is used to receive video image tasks sent by users, and according to the type of video image tasks, and the current tasks of multiple node servers According to the quantity and load conditions, the video image task is distributed to the corresponding node server through the central end of the computing unit; the node server is used to process the video image task through the corresponding computing unit node end of the video image task. The distributed computing resource allocation system and task processing method provided by the present invention can use multiple servers of a server cluster to process different types of video image tasks, thereby improving the utilization rate of the server cluster.

Description

Distributed computing resource allocation system and task processing method

technical field

The invention relates to the field of multimedia playing, in particular to a distributed computing resource allocation system and task processing method.

Background technique

At present, in daily life, video-based applications are more and more widely used, and there are more and more ways to process videos, such as image conversion of videos, rotation of pictures in videos, comparison of real-time images in videos, etc. For video Different processing methods, in order to speed up the processing speed of videos or images, different computing modes will be used to process videos, for example: when converting and processing videos, due to the large amount of processed data, parallel computing methods are generally used (such as MapReduce) divide the video into blocks and then process it; and for the comparison of real-time images in the video, since the image content in the video image needs to be determined in a short time, the comparison of massive images is performed using the in-memory computing method. Therefore, different server clusters will be deployed to achieve different computing methods.

In the existing video processing process, when the image of the video needs to be converted, the video file will be distributed to a server cluster with parallel computing mode for processing. When it is necessary to compare the real-time images in the video, The video image will be distributed to the server cluster with in-memory computing method for processing.

In the existing method of processing video in units of server clusters, one server cluster can only implement one video processing method. When there are many video processing methods, a large number of server clusters must be deployed to meet the requirements of video processing. Therefore, the construction cost of the server cluster is greatly increased, and the utilization rate of each server cluster is different, and the server resources in the server cluster with low utilization rate cannot be used by other video processing methods, resulting in Waste of resources.

Contents of the invention

The purpose of the present invention is to provide a distributed computing resource allocation system and a task processing method, which can use multiple servers of a server cluster to process different types of video image tasks, and improve the utilization rate of the server cluster.

In the first aspect, the embodiment of the present invention provides a distributed computing resource allocation system, including: a central server and a plurality of node servers, wherein the central server is provided with computing unit central ends for different task types, each The node servers are respectively provided with computing unit node terminals corresponding to the computing unit central terminals, and the computing unit central terminal and the corresponding computing unit node terminals form a computing unit;

The central server is used to receive the video image task sent by the user, and distribute the video image task through the computing unit center according to the type of the video image task, and the current task quantity and load situation of the plurality of node servers to the corresponding node server;

The node server is configured to process the video image task through a computing unit node end corresponding to the video image task.

In combination with the first aspect, the embodiment of the present invention provides a first possible implementation manner of the first aspect, wherein the central server includes: a task number determination module configured to determine the number of tasks according to the assigned tasks of each node server in the current server cluster Quantity, to determine the node server with the smallest number of assigned tasks; the first selection module is used to select the node server with the smallest number of tasks as the node server corresponding to the video image task; the second selection module is used when there are multiple node servers When the number of assigned tasks is the least, select the node server with the lowest resource utilization rate among the plurality of node servers as the node server corresponding to the video image task, and the resource utilization rate includes the utilization of at least one of the following server hardware resources Rate: CPU, memory and network bandwidth;

The node server includes: an information reporting module, configured to periodically feed back the current resource utilization rate to the central server.

In combination with the first aspect, the embodiment of the present invention provides a second possible implementation manner of the first aspect, wherein the central server includes: a task adding module configured to search for the task list of the node server corresponding to the video image task , adding the video image task to the number of assigned tasks in the task list; the task deletion module is configured to delete the video image task when receiving the information that the video image task sent by the computing unit node has been completed Deletion of the video image task described in the number of assigned tasks in the task list;

The node server includes: a task feedback module, which is used to send information that the task has been completed to the central server through the computing unit node corresponding to the video image task when the video image task is completed, and the task has been completed The information carries the identifier of the node server where the corresponding computing unit is located.

In combination with the first aspect, the embodiment of the present invention provides a third possible implementation manner of the first aspect. The central server includes: an instruction acquiring module, configured to acquire an instruction to add a computing unit, and the instruction to add a computing unit carries a calculation The unit data packet; the central terminal setting module is used to set the corresponding calculation unit central terminal according to the calculation unit data package, and allocates ports for the set calculation unit central terminal; the sending module is used to configure the calculation unit The data packet and the port number assigned to the port at the central end of the computing unit are sent to the plurality of node servers;

The node server includes: a node end setting module, configured to set the corresponding computing unit node end after receiving the computing unit data packet; a connection establishment module, used to set the port through the port corresponding to the port number The computing unit node end establishes a connection with the computing unit central end.

In combination with the first aspect, the embodiment of the present invention provides a fourth possible implementation manner of the first aspect. The central server includes: a deletion instruction acquisition module, configured to acquire a computing unit deletion instruction, and the computing unit deletion instruction carries There is an identification of the computing unit; the unloading module of the computing unit central terminal is used to unload the computing unit central end corresponding to the computing unit identification according to the computing unit identification carried in the computing unit deletion instruction; the deletion instruction issuing module is used for sending the computing unit deletion instruction to the plurality of node servers;

The node server includes: a computing unit node-end unloading module, configured to, when receiving the computing unit deletion instruction, unload the computation corresponding to the computing unit identifier according to the computing unit identifier in the computing unit deletion command unit node.

In combination with the first aspect, the embodiment of the present invention provides a fifth possible implementation manner of the first aspect. The central server includes: a general task interface, through which all tasks sent by users with different task types are obtained. described video image tasks.

In the second aspect, an embodiment of the present invention provides a task processing method implemented by applying the above-mentioned distributed computing resource allocation system, including:

The central server receives the video image task sent by the user, and according to the type of the video image task, and the current video image task quantity and load situation of the plurality of node servers, distributes the video image task through the central terminal of the computing unit to the corresponding node server;

After receiving the video image task, the node server processes the video image task through a computing unit node corresponding to the video image task.

With reference to the second aspect, the embodiment of the present invention provides a first possible implementation manner of the second aspect, wherein, the central server bases on the type of the video image task and the current video image task of the plurality of node servers Quantity and load situation, distributing the video image task to the corresponding node server through the central end of the computing unit includes:

The central server determines the node server with the least number of assigned tasks according to the number of assigned tasks of each node server in the current server cluster;

The central server selects the node server with the least number of tasks as the node server corresponding to the video image task;

When there are multiple node servers with the least number of assigned tasks, the central server selects the node server with the lowest resource utilization rate among the multiple node servers as the node server corresponding to the video image task, and the resource utilization rate Including the utilization rate of at least one of the following server hardware resources: central processing unit, memory and network bandwidth; wherein, the node server periodically feeds back the current resource utilization rate to the central server.

With reference to the second aspect, the embodiment of the present invention provides a second possible implementation manner of the second aspect, wherein the method further includes:

The central server searches the task list of the node server corresponding to the video image task, and adds the video image task to the number of assigned tasks in the task list;

When receiving the information that the video image task has been completed sent by the computing unit node, the central server deletes the video image task from the number of assigned tasks in the task list;

When the video image task is completed, the node server sends the information that the task has been completed to the central server through the computing unit node end corresponding to the video image task, and the information that the task has been completed carries the corresponding computing unit The ID of the node server where it resides.

In combination with the second aspect, the embodiment of the present invention provides a third possible implementation manner of the second aspect, wherein the method further includes:

The central server acquires an instruction to add a computing unit, and the instruction to add a computing unit carries a data packet of a computing unit;

The central server sets a corresponding computing unit central terminal according to the computing unit data packet, and allocates a port for the set computing unit central terminal;

The central server sends the computing unit data packet and the port number assigned to the central port of the computing unit to the plurality of node servers;

After the node server receives the computing unit data packet, it sets the corresponding computing unit node end;

The node server establishes a connection between the set computing unit node end and the computing unit central end through the port corresponding to the port number.

In combination with the second aspect, the embodiment of the present invention provides a fourth possible implementation manner of the second aspect, wherein the method further includes:

The central server acquires a computing unit deletion instruction, and the computing unit deletion instruction carries a computing unit identifier;

The central server uninstalls the computing unit central end corresponding to the computing unit identifier according to the computing unit identifier carried in the computing unit deletion instruction;

The central server sends the computing unit deletion instruction to the plurality of node servers;

When receiving the computing unit deletion instruction, the node server uninstalls the computing unit node end corresponding to the computing unit identifier according to the computing unit identifier in the computing unit deletion command.

In combination with the second aspect, the embodiment of the present invention provides a fifth possible implementation manner of the second aspect, wherein the central server acquires the video image tasks with different task types sent by the user through the general task interface. .

In the distributed computing resource allocation system and task processing method provided by the embodiments of the present invention, the set central server receives video image tasks sent by users, and according to the current number of tasks and load conditions of multiple node servers in the server cluster, video The image task is assigned to the computing unit node end corresponding to the computing unit center end of any node server in the server cluster through the computing unit center end set by the central server, and the video image task is performed through the computing unit node end corresponding to the video image task Processing; by setting the computing unit central terminal for distributing video image tasks in the central server, and setting the computing unit node end that can handle corresponding types of video image tasks in each server of the server cluster, so that a server cluster can be used Multiple servers can process different types of video and image tasks, and it is not necessary to deploy multiple server clusters to meet the processing requirements of video image tasks, which improves the utilization of server clusters and reduces the construction cost of server clusters. Multiple servers process different types of video image tasks, which improves the utilization rate of server resources in the server cluster and avoids waste of resources.

In order to make the above-mentioned objects, features and advantages of the present invention more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and thus It should be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings based on these drawings without creative work.

FIG. 1 shows a schematic structural diagram of a distributed computing resource allocation system provided by Embodiment 1 of the present invention;

FIG. 2 shows a schematic diagram of task types processed by computing units in a distributed computing resource allocation system provided by Embodiment 1 of the present invention;

FIG. 3 shows a schematic architecture diagram of another distributed computing resource allocation system provided by Embodiment 2 of the present invention;

FIG. 4 shows a schematic diagram of a computing unit node end for performing historical data calculation on a video image task in the distributed computing resource allocation system provided by Embodiment 2 of the present invention;

FIG. 5 shows a schematic diagram of a computing unit node end for in-memory computing of video image tasks in the distributed computing resource allocation system provided by Embodiment 2 of the present invention;

6 shows a schematic diagram of a computing unit node end for performing real-time data parallel computing on video image tasks in the distributed computing resource allocation system provided by Embodiment 2 of the present invention;

Fig. 7 shows a schematic diagram of a node end of a computing unit for performing high-concurrency small-block data calculation on a video image task in the distributed computing resource allocation system provided by Embodiment 2 of the present invention;

FIG. 8 shows a flow chart of the task processing method provided by Embodiment 3 of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. The components of the embodiments of the invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present invention.

Considering that in related technologies, in the existing methods of processing video in units of server clusters, one server cluster can only implement one video processing method. When there are many video processing methods, a large number of video processing methods must be deployed. Only the server cluster can meet the video processing requirements, so the construction cost of the server cluster is greatly increased, and the utilization rate of each server cluster is different, and the server resources in the server cluster with low utilization rate cannot be processed by other video way, resulting in a waste of resources. Based on this, an embodiment of the present invention provides a distributed computing resource allocation system and a distributed processing method.

Example 1

Referring to Fig. 1, the present embodiment provides a distributed computing resource allocation system, including: a central server 100 and a plurality of node servers 110, wherein, the central server 100 is provided with computing unit central terminals 101 for different task types, each The node server 110 is respectively provided with a computing unit node end 111 corresponding to the computing unit central end 101, and the computing unit central end and the corresponding computing unit node end form a computing unit;

The central server 100 is used to receive the video image task sent by the user, and according to the type of the video image task, and the current task quantity and load situation of multiple node servers 110, distribute the video image task to the corresponding node server through the computing unit central terminal 101 110;

The node server 110 is configured to process the video image task through the computing unit node end 111 corresponding to the video image task.

The schematic diagram of the task types processed by the computing unit is shown in Figure 2. Each computing unit will process a specific type of video image task, so when the central server 100 receives the video image task sent by the user, it will first obtain the received According to the task type carried in the video image task, the received video image task is distributed to the computing unit central end 101 of the computing unit processing the corresponding task type according to the task type of the video image task.

The central server 100 and the node server 110 can use any existing servers and terminals that can perform distributed computing resource allocation and distributed computing processing, and will not be repeated here.

Video and image tasks include video tasks and image tasks, where video tasks refer to tasks issued by users to process videos or video images; image tasks refer to tasks issued by users to process videos or video images; due to video tasks and Image tasks are relatively similar in processing, so the computing units set in the distributed computing resource allocation system can simultaneously process video tasks and image tasks.

Video image tasks include but are not limited to: car model comparison, face comparison, video transcoding, image rotation, image defogging and video arming. Wherein, the two processings of image rotation and image defogging may be performed on video images in a video task, or on images in an image task.

The content of the video and image processed by the video image task is acquired through a camera or a video camera, so details will not be repeated here.

Video image tasks can be divided into different task types according to different processing methods for video images. When there are multiple video image tasks that are processed in the same manner, the multiple video image tasks may be divided into one type of video image tasks.

The division of task types for video image tasks can be pre-set when developing the distributed computing resource allocation system, or users of the distributed computing resource allocation system can find that some video image tasks are not suitable after a while. When the division of task types is inaccurate, the task types of these video image tasks can be re-divided through the options of the distributed computing resource allocation system.

For example, when comparing the face and car model in the video image, it is necessary to compare the face or car model in the acquired image with the face and car model stored in the database, so in order to ensure the speed of comparison, the data in the database The face and car model data should be kept in memory for a long time, so that when the face and car model comparison is required, the face and car model can be analyzed in real time, so as to quickly feed back the comparison results to the user, then you can set the video image The task types of face comparison and car model comparison are the same task type, which are processed by the computing unit that can perform in-memory calculations.

Task types include, but are not limited to: in-memory computing, historical data parallel computing, real-time data parallel computing, and high-concurrency small-block data computing.

For different types of video image tasks, they are processed separately by different computing units. When the types of tasks that the distributed computing resource allocation system can handle include in-memory computing, historical data parallel computing, real-time data parallel computing, and high-concurrency small-block data computing, the distributed computing resource allocation system will include: The first computing unit for in-memory computing processing, the second computing unit for parallel computing of historical data, the third computing unit for parallel computing of real-time data, and the fourth computing unit for high concurrent small block data computing.

Each computing unit includes a computing unit central terminal 101 for distributing video image tasks and a computing unit node terminal 111 for processing video image tasks; the computing unit node terminal 111 in each computing unit passes a predetermined The port of the corresponding computing unit central terminal 101 is connected to receive the video image task sent by the computing unit central terminal 101, and when the video image task is completed, the task completion information is fed back to the computing unit central terminal 101.

The distributed computing resource allocation system provided in this embodiment receives the video image task sent by the user through the set central server, and according to the current task quantity and load situation of multiple node servers in the server cluster, the video image task is set by the central server. The central end of the computing unit is assigned to any node in the server cluster, and in the computing unit node end corresponding to the central end of the computing unit in the server, the video image task is processed through the computing unit node end corresponding to the video image task; Set up the computing unit central end that distributes video image tasks, and set the computing unit node end that can handle corresponding types of video image tasks in each server of the server cluster, so that multiple servers in one server cluster can be used to process different type of video image tasks, it is not necessary to deploy multiple server clusters to meet the processing requirements of video image tasks, improve the utilization of server clusters, reduce the construction cost of server clusters, and process different types of video images through multiple servers in a server cluster The video image task improves the utilization rate of server resources in the server cluster and avoids waste of resources.

In related technologies, the server that allocates tasks will receive different types of tasks sent by users through different interfaces according to the different task types of video image tasks received, so in the early development of the server, it is necessary to target different tasks Different types of tasks are used to develop different interfaces. In order to reduce the number of developed interfaces, the central server is preset with a general task interface for obtaining video image tasks with different task types sent by users. The video image tasks sent by users can be passed through the general task The interface is obtained by the central server.

It can be seen from the above description that through the set general task interface, tasks with different task types sent by the user are received, and there is no need to design interfaces for each computing unit separately, which improves the development speed of the software.

When the user sends a video image task to the central server, the user sends the video image task to the central server through the terminal used. The video image task carries the task type of the video image task, and the central server obtains the video image task sent by the user. During the task, according to the task type of the video image task carried in the video image task, in the corresponding relationship between the task type preset in the central server and the identification of the computing unit, determine the computing unit that processes the video image task of the task type, and then Assign the video image task to the computing unit central terminal that processes the video image task of this task type; and the computing unit central terminal will send a server resource application request to the central server, and the central server will obtain the information sent by the computing unit central terminal. After the server resource application request, the video image task will be assigned to the corresponding node server through the central terminal of the computing unit according to the current task quantity and load situation of multiple node servers.

In related technologies, when assigning tasks, the server that allocates tasks will only randomly assign the tasks to any server in the server cluster, which will lead to unbalanced load of each server in the server cluster, thereby reducing the task processing capacity of the server cluster , the central server alleviates the load imbalance of each server in the server cluster by setting the task quantity determination module, the first selection module and the second selection module, and the information reporting module set in the node server. The central server specifically includes: a task quantity determination module , used to determine the node server with the smallest number of assigned tasks according to the number of assigned tasks of each node server in the current server cluster; the first selection module is used to select the node server with the smallest number of tasks as the node server corresponding to the video image task; The second selection module is used to select the node server with the lowest resource utilization rate among the multiple node servers as the node server corresponding to the video image task when there are a plurality of node servers with the least number of assigned tasks, and the resource utilization rate includes at least the following A utilization of server hardware resources: CPU, memory, and network bandwidth.

The node server specifically includes: an information reporting module, which is used to periodically feed back the current resource utilization rate to the central server.

After receiving the current resource utilization data of each server periodically fed back by the node server, the central server will cache the received current resource utilization data of each server. Since the node server periodically feeds back the current resource utilization rate to the central server, when the preset time period arrives, the central server will receive the data of the current resource utilization rate of each server fed back by the node server again. At this time, the central server will The cached resource utilization data is erased, and then the currently received resource utilization data of each server is written into the cache.

The module for determining the number of tasks specifically includes: a task number acquisition unit for obtaining the number of assigned tasks of each node server in the server cluster; a task number comparison unit for comparing the number of assigned tasks of each node server to determine the number of assigned tasks The minimum value of the number; the node server determination unit is used to select the node server corresponding to the minimum value of the number of assigned tasks as the node server with the smallest number of assigned tasks.

When the task number determination module determines that there are currently multiple node servers with the least number of assigned tasks, the second selection module specifically includes: a resource utilization acquisition unit, which is used to obtain the least number of assigned tasks cached in the central server The resource utilization rate data of multiple node servers; the resource utilization rate comparison unit is used to compare the resource utilization rate of multiple node servers with the least number of assigned tasks to determine the minimum value of resource utilization rate; the node server determination unit , which is used to select the node server corresponding to the lowest value of resource utilization as the node server with the least number of assigned tasks.

The server mainly processes video image tasks through the central processing unit, and the resource utilization rate comparison unit preferably compares the resource utilization rates of multiple node servers with the least number of assigned tasks by comparing the central processing units in each server. The utilization rate of each core is used to determine the node server with the lowest resource utilization.

The task quantity comparing unit and the resource utilization comparing unit may adopt any existing quantity comparison method to respectively determine the minimum value of the allocated task quantity and the minimum resource utilization rate, which will not be repeated here.

From the above description, it can be seen that when assigning tasks to each server, the tasks are assigned based on the principle of load balancing. According to the usage of each server in the server cluster, when assigning tasks, each server in the server cluster can be assigned. Unified management and distribution.

In order to allocate processing tasks to each server according to the load of the server, the central server grasps the load of each server in real time by setting a task adding module and a task deleting module and a task feedback module set by the node server. The central server specifically includes: a task adding module, which is used to find the task list of the node server corresponding to the video image task, and adds the video image task to the number of assigned tasks in the task list; When sending the information that the video image task has been completed, delete the video image task from the assigned task number in the task list;

The node server includes: a task feedback module, which is used to send information that the task has been completed to the central server through the computing unit node corresponding to the video image task when the video image task is completed, and the information that the task has been completed carries the node where the corresponding computing unit is located The server's ID.

The identification of each node server and the number of tasks assigned to the node server corresponding to the identification of the node server are recorded in each task list. When the central server determines that the node server with the least number of assigned tasks processes the video image task, the central server will The video image task is added to the task list of the node server with the smallest number of assigned tasks, thereby updating the task list of the node server with the smallest number of assigned tasks.

When receiving the information that the video image task sent by the node end of the computing unit has been completed, the task deletion module queries the node server’s task list corresponding to the node server’s identification according to the identification of the node server recorded in the information that the video image task has been completed In the task list of the node server corresponding to the identifier of the node server, the video image task is deleted from the number of assigned tasks in the task list, so as to update the task list.

After receiving the information that the video image task has been completed, the central server will send the message that the video image task is completed to the user through the general task interface, and notify the user of the storage address of the video image task processing result after the processing is completed, so that the user can pass the storage address Get the processing result of the video image task.

From the above description, it can be seen that by recording the number of tasks assigned by each server in the preset task list, the load of each server can be determined in real time, and the processing tasks can be allocated to each server according to the load of the server, which can make Server load is more balanced.

With the development of video processing and video image processing technology, users have more and more processing requirements for video image tasks, and the existing computing units in the distributed computing resource allocation system cannot process video images of certain types of tasks sent by users. When the task is effectively processed, it is necessary to set up a new computing unit in the central server to process these video image tasks. The terminal setting module and the connection establishment module set a new computing unit in the central server and the node server, and the central server specifically includes: an instruction acquisition module, which is used to obtain a computing unit adding instruction, and the computing unit adding instruction carries a computing unit data packet; the central terminal The setting module is used to set the corresponding computing unit central terminal according to the computing unit data packet, and allocate ports for the set computing unit central terminal; the sending module is used to assign the computing unit data packet and the port assigned to the port of the computing unit central terminal The number is sent to multiple node servers.

The node server specifically includes: a node end setting module, which is used to set the corresponding computing unit node end after receiving the computing unit data packet; The center establishes a connection.

When the maintenance personnel of the distributed computing resource allocation system find that a certain type of video image task is processed by all the existing computing units in the distributed computing resource allocation system, the processing time is relatively long, and the processing results cannot be fed back to the user in time When, according to the processing characteristics of this type of video image task, determine the computing unit that can effectively process this type of video image task, and then obtain the computing unit data packet of the computing unit processing this type of video image task, And form a computing unit adding instruction through the obtained computing unit data packet, and send the formed computing unit adding instruction to the central server, so that the central server can set the corresponding computing unit central terminal according to the computing unit data packet.

The maintenance personnel of the distributed computing resource allocation system can obtain the computing unit data package of the computing unit processing this type of video image task from the system background server, or download the computing unit data packet of the computing unit processing this type of video image task through the network. The computing unit data packet can also be acquired through any existing data packet acquisition method, which will not be repeated here.

Of course, the computing unit central end of each computing unit provided in the central server can also record the processing time of the processed video image task, and when the task processing is completed, the processing time of the video image task and the information that the task has been completed Feedback to the central server together, the central server can compare the processing time of the video image task with the preset processing time threshold, when the processing time of the video image task is greater than the preset processing time threshold, it means that the current computing unit is not suitable To process this type of video image task, the central server will associate the task type of the video image task with other computing units, so that the next time this type of video image task is processed, other computing units can be used When the central server determines that none of the currently set computing units are suitable for processing this type of video image task, it will determine and obtain the information of the computing unit processing this type of video image task from the system background server. Computing unit data packets, and form computing unit adding instructions through the obtained computing unit data packets, and send the formed computing unit adding instructions to the central server, so that the central server can set corresponding computing unit central terminals according to the computing unit data packets.

From the above description, it can be seen that the central server sets the computing unit according to the command to add the computing unit, and the process of adding the computing unit is simple, convenient and fast, flexible and convenient to use, and after adding the new computing unit, the processing of video image tasks can be reduced time and can handle tasks in a timely manner.

With the development of video processing and video image processing technology, some computing units in the distributed computing resource allocation system handle fewer and fewer video image tasks, and even some computing units are no longer used, but these are not The computing unit that is used again will still occupy the resources of the central server and node server, resulting in a waste of server resources. Therefore, by setting the deletion command acquisition module in the central server, the computing unit central end Set the computing unit node-side unloading module to uninstall computing units that are no longer in use. The central server specifically includes: a deletion instruction acquisition module, which is used to obtain a computing unit deletion instruction, and the computing unit deletion instruction carries the identification of the computing unit; The identification is to unload the computing unit central terminal corresponding to the identification of the computing unit; the deletion command sending module is used to send a computing unit deletion command to multiple node servers.

The node server includes: a computing unit node end unloading module, configured to uninstall the computing unit node corresponding to the computing unit identifier according to the computing unit identifier in the computing unit deletion command when receiving the computing unit deletion command.

When the maintenance personnel of the distributed computing resource allocation system find that a certain type of video image task is no longer processed for a long time, they will send a computing unit deletion command carrying the identification of the computing unit to the central server, so that the central server and the node server Delete the computing unit corresponding to the identifier of the computing unit.

Of course, the central server can also record the task assignment time of the latest video image task assignment to the computing unit, and periodically judge whether the time interval between the latest task assignment time of the video image task assignment to the computing unit and the current time reaches If the deletion time of the computing unit is reached, if the deletion time of the computing unit is reached, then a computing unit deletion command carrying the identifier of the computing unit that can be deleted is sent to the central server, so that the central server and the node server delete the corresponding computing unit.

The maintenance personnel of the distributed computing resource allocation system can send computing unit addition instructions and computing unit deletion instructions to the central server through the input device of the central server or the input device of the system background server connected to the central server.

From the above description, it can be seen that the computing units that are not in use can be deleted according to the needs of task processing. Processing of tasks by other processing units will not be affected.

Example 2

Referring to Fig. 3, this embodiment provides another distributed computing resource allocation system, which is expressed in the form of CUMN architecture. The distributed computing resource allocation system includes a center (Center), a user (User), a module (Module), a model (Model) and node (Node) 5 parts. Different business requirements will have different processing methods for data. This system uses models to define different calculation methods, which means modeling. For example, Hadoop uses the Map/Reduce computing model. The CUMN architecture stipulates that each computing model consists of one or several centers and several nodes. The models in the framework are "hot-swappable", and models can be added or deleted arbitrarily in the running state, which greatly improves the scalability and compatibility of the distributed computing resource allocation system.

The model is just a calculation method. What is the specific calculation? This solution defines it as a module, which can be imagined as an algorithm. For example, the grid is used to realize video structure, video concentration, video retrieval, etc., and each function is a module. The grid model only defines the splitting of jobs by means of Map/Reduce, but how to calculate after splitting is the function of the module. In this solution, the modules are also "hot-swappable" design, which can dynamically add and delete modules, which provides convenience for system expansion and upgrade.

All models in this program can be independently produced, such as grids. When accessing the system, you can use the public interface of the framework, or you can use the private interface provided by the model itself. In this solution, the part that interacts with the user is defined as U, the uppercase U represents the public interface of the framework, and the lowercase u represents the private interface of the model. When the public interface of the framework cannot meet the requirements sometimes, the private interface of the model can be used.

The CUMN architecture makes the distributed computing resource allocation system a general framework, and the actual computing is realized by models and modules. The characteristics of video and image analysis are as follows:

Large amount of data: Every city, such as video surveillance and traffic checkpoints, generates tens of terabytes to several petabytes of data every day. When handling cases, the police will delineate the videos of dozens of cameras and need to quickly analyze them;

There are many data types and processing methods: historical video, image analysis, real-time video, image analysis.

A model is a computational processing model. For example, the mainstream Hadoop MapReduce is a computational model for processing big data. However, in the field of video applications, due to the diversity of data types (videos, images, and target features), a certain computing model cannot adapt to all data types. For example, Hadoop is suitable for processing a large file (GB-TB level). There is nothing that can be done with pictures of only a few hundred KB. At the same time, different businesses have different requirements for the response speed of data processing (for example, image processing and face comparison should be returned quickly, and video retrieval can be slower to produce results). In order to cope with various data and business types, this solution proposes a model concept. In this solution, a model is a calculation method for processing a certain data or business. All calculation models define a central terminal (in lowercase c) and a node terminal (in lowercase c). The central terminal of the model is defined by the center of the calculation framework (Uppercase C) management, and assign a unique port number for connection with the node side of the model; the node side of the model is managed by the node side of the framework (with uppercase N).

Assume that model 1 divides each task and calculates it for each node; model 2 calculates a task for each node; model 3 means that all calculations must load data into memory; this makes it easier to understand why there is a concept of model up.

This solution adopts the model hot-swapping mode, and models can be added online as needed. The central end of the newly added model is automatically deployed to the central node server and started. The model central end is started by being assigned a port number, and the model central end program monitors the Port number, waiting for the connection of the node side of the model; the node segment program of the model will be automatically deployed to all node servers within the jurisdiction of the framework. Automatic addition and deployment of models.

The central server and node server of this scheme are in n:N mode, that is, a large number of node servers and no less than one central server, and the central server is more than one to avoid the problem of single point of failure; each central server needs to deploy the center of this scheme framework Management software, each node server needs to deploy the node management software of this solution framework; the center management software is responsible for managing the center-end programs of all models (including assigning a unique port for the center-end program), and the node management software is responsible for managing the node ends of all models program.

The model defines how to use the cluster to work, but what to do specifically. This solution proposes the concept of modules. In order to improve the utilization rate of the model, the calculation process is the same for the model anyway, but the loaded modules are different (such as image enhancement and image rotation). With the concept of modules, the model no longer cares about what it does, but just calls different modules for processing according to needs, and does not care about how to handle it. In this way, only one module needs to be added according to the addition of new services, which is why this solution is called a framework.

C and N in model management not only realize the function of automatic deployment, but also actively collect the resource usage status of each model, so as to realize the unified load balancing of resources in the cluster. Each model must refer to the unified load in the cluster when scheduling resources, and send tasks to nodes with relatively low loads to run.

In the traditional way, multiple computing models are deployed in the same cluster. Since there is no communication between the models, the model thinks that node A is relatively idle, but it does not know that the resources of node A have been fully used by other models.

The distributed computing resource allocation system provides a set of user interfaces to allow users to use all modules in the framework. In order to improve the friendliness of this solution, users do not need to know the concept of models and modules, and can see which companies' algorithms are in the framework through the interface, and use the algorithms to realize their own requests. This requires the framework to automatically identify the model and backbone of the algorithm, which has been explained above and will not be repeated here.

This is also more friendly than simply stacking multiple computing model clusters, because the interface methods and languages of multiple computing model clusters may be different, which is not only inconvenient to use, but also requires an external interface every time a new model is added. development work, but this solution only needs to be developed once, and the addition of model interfaces remains unchanged.

The distributed computing resource allocation system can perform the following computing processing on video image tasks:

Figure 4 is a schematic diagram of the node side of the computing unit that performs historical data calculation for video image tasks. Historical data calculation is the default integrated computing model in this framework. It studies how to divide a problem that requires a very large computing power to be solved into many small ones. part, and then assign these parts to many computers for processing, and finally combine these calculation results to obtain the final result, which corresponds to the grid computing model in this scheme. Another characteristic of the grid computing model is to make full use of idle computing power, even without adding new computing power, it can usually effectively improve the utilization of physical machine hardware.

Figure 5 is a schematic diagram of the computing unit node side for in-memory computing of video image tasks. In-Memory Computing is the default integrated computing model of this framework. In essence, the CPU directly reads data from the memory instead of the hard disk. And calculate and analyze the data. This technology is an acceleration of traditional data processing methods, and is a key application technology for realizing intelligent mass data analysis and implementing data analysis.

In-memory computing is very suitable for processing massive amounts of data and data that requires real-time results. For example, the face features of the resident population database, stolen vehicle information and other data can be stored in the memory at one time, and data analysis can be performed on this basis. When fast data analysis is required, in-memory computing can be quickly completed as required.

In terms of real-time computing, in order to reduce the overhead caused by data movement, the memory adopts the method of data resident memory. At the same time, each computing node only calculates the data stored in its own memory, which basically guarantees zero data loading process within the system response cycle. In order to improve the computing throughput, each node only calculates the data stored in the node as much as possible, so as to minimize the data movement overhead between nodes. The data in this solution adopts a redundant storage strategy. When a node is abnormal, other nodes can automatically load a copy to ensure the availability of the system. Since the amount of data suitable for in-memory computing is not too large, generally around tens of gigabytes to hundreds of gigabytes, the use of copy storage will not bring pressure to the storage system.

Figure 6 is a schematic diagram of the computing unit node side for real-time data parallel computing for video image tasks. Real-time data parallel computing is the default integrated computing model of this framework, which analyzes video (picture) streams in real time, such as video arming. Real-time video (picture) analysis focuses on the timely extraction of various events in real-time video streams.

Figure 7 is a schematic diagram of the node side of the computing unit that performs high-concurrency small-block data calculation for video image tasks. High-concurrency small-block data calculation is the default integrated calculation model of this framework. It is different from the data and methods of parallel computing processing. There are still There are huge amounts of small pieces of data that need to be processed. The timeliness requirements for these data processing are high, and there is a possibility of high concurrency at the same time. A typical application is image processing. This solution designs consistent hash scheduling and balanced processing methods for such calculations, which can not only ensure timely data processing, but also solve the problem of high concurrent requests.

The distributed computing resource allocation system proposed in this embodiment can monitor and dynamically allocate the physical resources in the computing nodes. Each computing node will install the node background program of the framework, and the program will obtain the attribute information of the computing node, including the CPU Main frequency, number of CPU cores, number of CPU threads, total memory, type of operating system, network bandwidth, etc., and report to the center of the framework. At the same time, the node program will also periodically (default is 1 second, configurable) report the utilization rate of the node's CPU, memory, and network to the center, providing reference data for the center to allocate resources.

Each request submitted by the user is defined as a job in the frame, and the job is split into multiple fragments to be executed on multiple physical machines, and each fragment frame is defined as a task. The smallest unit of framework scheduling is a task. The framework divides physical resources into CPU cores (including hyperthreading), and each task runs on a CPU core as an independent process (although the operating system will switch processes between multiple cores, but In terms of resource usage, it is only the computing power of one core)

The distributed computing resource allocation system can run on physical machines and virtual machines, and can also run on Windows, Linux, and Unix operating systems. Physical machines range from computing devices such as mainframes, minicomputers, servers, personal PCs, notebooks, tablet computers, and smart phones. At the same time, integrated graphics cards and coprocessors in physical machines are used to accelerate calculations.

The connection relationship and realized functions of each part in the distributed computing resource allocation system will be further described below.

The distributed computing resource allocation system includes: a center (hereinafter referred to as a central server), a computing node (hereinafter referred to as a node server), a computing unit central terminal and a computing unit node terminal.

The central server is used to receive the pending task sent by the user through the set task interface, and send the pending task to the computing unit central terminal that processes the corresponding task according to the task type of the pending task; upon receiving the task, the central terminal of the computing unit sends When requesting resource allocation, the allocated node servers are determined according to the number of processing tasks and load conditions of each node server in the current distributed computing cluster, and the allocated node servers are fed back to the computing unit center, and according to the current resource allocation As a result, the preset task allocation list is updated; when the task release request fed back by the computing unit center is received, the task list is updated according to the node server identification carried in the task release request, and the task is sent to the user through the task interface. Complete the information; when a new computing unit needs to be added according to the needs of task processing, the central server will obtain the computing unit addition command, and install the newly added computing unit central terminal according to the computing unit data package carried in the computing unit addition command, And assign the computing unit identification to the central end of the newly added computing unit, and then send the computing unit identification and data packets to each node server, so that each node server can install the newly added computing unit node end, and add to the newly added computing unit Ports are assigned to the central end of the unit to establish a connection between the central end of the computing unit and the node end of the computing unit. When a computing unit is no longer in use and needs to be deleted, the central server will obtain the computing unit deletion instruction, and delete the computing unit central terminal corresponding to the computing unit identifier according to the computing unit identifier carried in the computing unit deletion instruction, and then A computing unit deletion command carrying the computing unit identifier is sent to each node server, so that each node server deletes the computing unit corresponding to the computing unit identifier.

The central terminal of the computing unit is set in the central server and is used to receive the pending tasks sent by the central server. After receiving the pending tasks, it sends a resource allocation request to the central server, receives the resource allocation information sent by the central server, and The processed task is sent to the node server specified in the resource allocation information through the preset port to the computing unit node corresponding to the computing unit center to process the task; when the task completion request sent by the computing unit node is received, Send a task release request carrying the identity of the node server to the central server to update the pre-stored task assignment list in the central server; when a new computing unit is added, the central end of the newly added computing unit will monitor each node through the port assigned by the central server The connection establishment instruction of the computing unit node end installed on the server, after receiving the connection establishment instruction sent by the computing unit node end of each node server, establishes the connection with the computing unit node end of each node server.

The node server is used to process the assigned tasks, and periodically feed back the load status of the current node server to the central server, so that the central server can assign tasks to each node server according to the load status of each node server; When computing a unit, it will install the computing unit node of the newly added computing unit according to the data packet sent by the central server, and distribute the computing unit identifier sent by the central server to the installed computing unit node, and inform the newly added computing unit of The port of the node terminal when establishing a connection with the central terminal of the computing unit with the same identification of the computing unit, so that the node side of the newly added computing unit passes through the specified port; when deleting the computing unit, the node server deletes the computing unit according to the command sent by the central server The computing unit identifier carried in , delete the corresponding computing unit node.

The computing unit node end is set in each node server and is used to process the assigned tasks. When the task processing is completed, the task completion information is sent to the corresponding computing unit central end; at the computing unit node end where a new computing unit is added , the newly added computing unit node side establishes a connection with the computing unit central side through the port specified by the node server.

The distributed computing resource allocation system provided in this embodiment receives the video image task sent by the user through the set central server, and according to the current task quantity and load situation of multiple node servers in the server cluster, the video image task is set by the central server. The central end of the computing unit is assigned to any node in the server cluster, and in the computing unit node end corresponding to the central end of the computing unit in the server, the video image task is processed through the computing unit node end corresponding to the video image task; Set up the computing unit central end that distributes video image tasks, and set the computing unit node end that can handle corresponding types of video image tasks in each server of the server cluster, so that multiple servers in one server cluster can be used to process different type of video image tasks, it is not necessary to deploy multiple server clusters to meet the processing requirements of video image tasks, improve the utilization of server clusters, reduce the construction cost of server clusters, and process different types of video images through multiple servers in a server cluster The video image task improves the utilization rate of server resources in the server cluster and avoids waste of resources.

Example 3

Referring to FIG. 8, this embodiment provides a task processing method, which can be implemented by the distributed computing resource allocation system in the above embodiment, and the task processing method includes the following steps:

Step 800, the central server receives the video image task sent by the user, and distributes the video image task to the corresponding node server through the central end of the computing unit according to the type of the video image task, and the current video image task quantity and load situation of multiple node servers ;

Step 802: After receiving the video image task, the node server processes the video image task through the computing unit node corresponding to the video image task.

In related technologies, the server that allocates tasks will receive different types of tasks sent by users through different interfaces according to the different task types of video image tasks received, so in the early development of the server, it is necessary to target different tasks Different types of tasks are used to develop different interfaces. In order to reduce the number of developed interfaces, the central server is preset with a general task interface for obtaining video image tasks with different task types sent by users. The video image tasks sent by users can be passed through the general task The interface is obtained by the central server.

It can be seen from the above description that through the set general task interface, tasks with different task types sent by the user are received, and there is no need to design interfaces for each computing unit separately, which improves the development speed of the software.

In related technologies, when assigning tasks, the server that allocates tasks will only randomly assign the tasks to any server in the server cluster, which will lead to unbalanced load of each server in the server cluster, thereby reducing the task processing capacity of the server cluster , in order to ensure the load balance of each node server, the central server assigns the video image task to the corresponding node server through the central end of the computing unit according to the type of video image task, and the current number and load of multiple node servers. Including the following steps (1) to (3):

(1) The central server determines the node server with the least number of assigned tasks according to the number of assigned tasks of each node server in the current server cluster;

(2) The central server selects the node server with the least number of tasks as the node server corresponding to the video image task;

(3) When there are multiple node servers with the least number of assigned tasks, the central server selects the node server with the lowest resource utilization rate among the multiple node servers as the node server corresponding to the video image task, and the resource utilization rate includes at least one of the following Utilization of server hardware resources: central processing unit, memory and network bandwidth; among them, the node server periodically feeds back the current resource utilization to the central server.

From the above description, it can be seen that when assigning tasks to each server, the tasks are assigned based on the principle of load balancing. According to the usage of each server in the server cluster, when assigning tasks, each server in the server cluster can be assigned. Unified management and distribution.

In order to allocate processing tasks to each server according to the load of the server, the central server assigns the video image tasks to the corresponding nodes through the central end of the computing unit according to the type of video image tasks, and the current number and load of multiple node servers. The node server step also includes the following steps (4) to (6):

(4) the central server searches for the task list of the node server corresponding to the video image task, and adds the video image task in the assigned task quantity of the task list;

(5) When receiving the information that the video image task sent by the computing unit node end has been completed, the central server deletes the video image task from the assigned task quantity in the task list;

(6) Node server When the video image task is completed, the node terminal of the computing unit corresponding to the video image task sends the information that the task has been completed to the central server, and the information that the task has been completed carries the identification of the node server where the corresponding computing unit is located.

From the above description, it can be seen that by recording the number of tasks assigned by each server in the preset task list, the load of each server can be determined in real time, and the processing tasks can be allocated to each server according to the load of the server, which can make Server load is more balanced.

With the development of video processing and video image processing technology, users have more and more processing requirements for video image tasks, and the existing computing units in the distributed computing resource allocation system cannot process video images of certain types of tasks sent by users. When tasks are effectively processed, new computing units need to be set in the central server to process these video image tasks, so in order to add computing units, the task processing method also includes the following steps (1) to (5):

(1) The central server obtains the instruction to add the computing unit, and the instruction to add the computing unit carries the data packet of the computing unit;

(2) The central server sets the corresponding computing unit central end according to the computing unit data packet, and allocates ports for the set computing unit central end;

(3) the central server sends the computing unit data packet and the port number assigned to the port of the computing unit central end to a plurality of node servers;

(4) After receiving the computing unit data packet, the node server sets the corresponding computing unit node end;

(5) The node server establishes a connection between the set computing unit node end and the computing unit central end through the port corresponding to the port number.

From the above description, it can be seen that the central server sets the computing unit according to the command to add the computing unit, and the process of adding the computing unit is simple, convenient and fast, flexible and convenient to use, and after adding the new computing unit, the processing of video image tasks can be reduced time and can handle tasks in a timely manner.

With the development of video processing and video image processing technology, some computing units in the distributed computing resource allocation system handle fewer and fewer video image tasks, and even some computing units are no longer used, but these are not The computing unit used again will still occupy the resources of the central server and the node server and cause a waste of server resources. In order to reasonably use the resources of the central server and the node server, the task processing method also includes the following steps (1) to (4):

(1) The central server obtains the computing unit deletion instruction, and the computing unit deletion command carries the identification of the computing unit;

(2) The central server unloads the central end of the computing unit corresponding to the computing unit identifier according to the computing unit identifier carried in the computing unit deletion instruction;

(3) The central server sends computing unit deletion instructions to multiple node servers.

(4) When receiving the computing unit deletion instruction, the node server uninstalls the computing unit node end corresponding to the computing unit identifier according to the computing unit identifier in the computing unit deletion command.

From the above description, it can be seen that the computing units that are not in use can be deleted according to the needs of task processing. Processing of tasks by other processing units will not be affected.

The task processing method provided by this embodiment receives the video image task sent by the user through the set central server, and sends the video image task through the computing unit center set by the central server according to the current task quantity and load situation of multiple node servers in the server cluster. end assigned to any node in the server cluster in the computing unit node end corresponding to the computing unit center end in the server cluster, the video image task is processed through the computing unit node end corresponding to the video image task; by setting the video image task in the central server The central end of the computing unit that distributes image tasks, and the computing unit node end that can handle the corresponding type of video image tasks are set in each server of the server cluster, so that multiple servers in a server cluster can be used to process different types of video For image tasks, there is no need to deploy multiple server clusters to meet the processing requirements of video image tasks, which improves the utilization of server clusters, reduces the construction cost of server clusters, and processes different types of video image tasks through multiple servers in a server cluster , improve the utilization rate of server resources in the server cluster, and avoid waste of resources.

The computer program product of the task processing method provided by the embodiments of the present invention includes a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the methods in the foregoing method embodiments. For specific implementation, please refer to the method embodiments , which will not be repeated here.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-OnlyMemory), random access memory (RAM, RandomAccessMemory), magnetic disk or optical disk and other media that can store program codes.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (12)

1. A distributed computing resource allocation system, characterized in that it comprises: a central server and a plurality of node servers, wherein the central server is provided with computing unit central terminals for different task types, each of the node servers Computing unit node terminals corresponding to the computing unit central terminals are respectively provided, and the computing unit central terminal and the corresponding computing unit node terminals form a computing unit; The central server is used to receive the video image task sent by the user, and distribute the video image task through the computing unit center according to the type of the video image task, and the current task quantity and load situation of the plurality of node servers to the corresponding node server; The node server is configured to process the video image task through a computing unit node end corresponding to the video image task. 2. The distributed computing resource allocation system according to claim 1, wherein the central server comprises: a task quantity determination module, configured to determine the allocated task quantity according to the allocated task quantity of each node server in the current server cluster. The node server with the least number of tasks; the first selection module is used to select the node server with the least number of tasks as the node server corresponding to the video image task; the second selection module is used when there are multiple node servers. When all are the least, select the node server with the lowest resource utilization rate among the plurality of node servers as the node server corresponding to the video image task, and the resource utilization rate includes the utilization rate of at least one of the following server hardware resources: central processing unit , memory and network bandwidth; The node server includes: an information reporting module, configured to periodically feed back the current resource utilization rate to the central server. 3. The distributed computing resource allocation system according to claim 2, wherein the central server comprises: a task adding module, configured to search for a task list of the node server corresponding to the video image task, and to add a task list in the task Add the video image task to the number of assigned tasks in the list; the task deletion module is used to delete the assigned tasks in the task list when receiving the information that the video image task sent by the computing unit node end has been completed Delete the video image tasks mentioned in the number of tasks; The node server includes: a task feedback module, which is used to send information that the task has been completed to the central server through the computing unit node corresponding to the video image task when the video image task is completed, and the task has been completed The information carries the identifier of the node server where the corresponding computing unit is located. 4. The distributed computing resource allocation system according to claim 1, wherein the central server comprises: an instruction acquisition module, configured to acquire an instruction to add a computing unit, and the instruction to add a computing unit carries a data packet of a computing unit The central terminal setting module is used to set the corresponding computing unit central terminal according to the computing unit data packet, and allocates ports for the set computing unit central terminal; the sending module is used to send the computing unit data packet and sending the port number assigned to the port at the central end of the computing unit to the plurality of node servers; The node server includes: a node end setting module, configured to set the corresponding computing unit node end after receiving the computing unit data packet; a connection establishment module, used to set the port through the port corresponding to the port number The computing unit node end establishes a connection with the computing unit central end. 5. The distributed computing resource allocation system according to claim 1, wherein the central server comprises: a deletion instruction acquisition module, configured to acquire a computing unit deletion instruction, and the computing unit deletion instruction carries a computing unit the identification of the computing unit central end; the unloading module of the computing unit central end is used to uninstall the computing unit central end corresponding to the identification of the computing unit according to the identification of the computing unit carried in the computing unit deletion instruction; the deletion instruction issuing module is used to send the Multiple node servers send the computing unit deletion instructions; The node server includes: a computing unit node-end unloading module, configured to, when receiving the computing unit deletion instruction, unload the computation corresponding to the computing unit identifier according to the computing unit identifier in the computing unit deletion command unit node. 6. The distributed computing resource allocation system according to claim 1, wherein the central server comprises: a general task interface, through which the video images sent by users with different task types are acquired Task. 7. A task processing method implemented by applying the system according to any one of claims 1 to 6, characterized in that, comprising: The central server receives the video image task sent by the user, and according to the type of the video image task, and the current video image task quantity and load situation of the plurality of node servers, distributes the video image task through the central terminal of the computing unit to the corresponding node server; After receiving the video image task, the node server processes the video image task through a computing unit node corresponding to the video image task. 8. task processing method according to claim 7, is characterized in that, described central server is according to the type of described video image task, and the current video image task quantity and the load situation of described multiple node servers, the described The video image task is distributed to the corresponding node server through the central end of the computing unit, including: The central server determines the node server with the least number of assigned tasks according to the number of assigned tasks of each node server in the current server cluster; The central server selects the node server with the least number of tasks as the node server corresponding to the video image task; When there are multiple node servers with the least number of assigned tasks, the central server selects the node server with the lowest resource utilization rate among the multiple node servers as the node server corresponding to the video image task, and the resource utilization rate Including the utilization rate of at least one of the following server hardware resources: central processing unit, memory and network bandwidth; wherein, the node server periodically feeds back the current resource utilization rate to the central server. 9. The task processing method according to claim 8, wherein the method further comprises: The central server searches the task list of the node server corresponding to the video image task, and adds the video image task to the number of assigned tasks in the task list; When receiving the information that the video image task has been completed sent by the computing unit node, the central server deletes the video image task from the number of assigned tasks in the task list; When the video image task is completed, the node server sends the information that the task has been completed to the central server through the computing unit node end corresponding to the video image task, and the information that the task has been completed carries the corresponding computing unit The ID of the node server where it resides. 10. The task processing method according to claim 7, wherein the method further comprises: The central server acquires an instruction to add a computing unit, and the instruction to add a computing unit carries a data packet of a computing unit; The central server sets a corresponding computing unit central terminal according to the computing unit data packet, and allocates a port for the set computing unit central terminal; The central server sends the computing unit data packet and the port number assigned to the central port of the computing unit to the plurality of node servers; After the node server receives the computing unit data packet, it sets the corresponding computing unit node end; The node server establishes a connection between the set computing unit node end and the computing unit central end through the port corresponding to the port number. 11. task processing method according to claim 7, is characterized in that, described method also comprises: The central server acquires a computing unit deletion instruction, and the computing unit deletion instruction carries a computing unit identifier; The central server uninstalls the computing unit central end corresponding to the computing unit identifier according to the computing unit identifier carried in the computing unit deletion instruction; The central server sends the computing unit deletion instruction to the plurality of node servers; When receiving the computing unit deletion instruction, the node server uninstalls the computing unit node end corresponding to the computing unit identifier according to the computing unit identifier in the computing unit deletion command. 12. The task processing method according to claim 7, wherein the central server acquires the video image tasks sent by users with different task types through the general task interface.