CN114816669A - Distributed training method and data processing method of model - Google Patents

Abstract

The disclosure provides a distributed training method and a data processing method for a model, and relates to the technical field of artificial intelligence, in particular to the technical fields of image processing, computer vision and the like. The implementation scheme is as follows: the method comprises the steps of obtaining a plurality of containers on a plurality of nodes, wherein each container in the plurality of containers is provided with a training module, and the training module can respond to a training request corresponding to each training task in a plurality of training tasks to execute the training task; in response to receiving a first request corresponding to a first training task, determining a plurality of training containers corresponding to the first training task from a plurality of containers; and respectively sending first training requests corresponding to the first training task to the plurality of training containers so that the training module in each of the plurality of training containers executes the first training task, thereby training the model corresponding to the first training task.

Description

Distributed training method and data processing method of model

technical field

The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of image processing and computer vision, and in particular to a distributed training method for a model, a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

Background technique

Artificial intelligence is the study of making computers to simulate certain thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both hardware-level technology and software-level technology. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, machine learning/deep learning, big data processing technology, knowledge graph technology and other major directions.

Artificial intelligence technology widely uses trained algorithm models as a means to implement related software technologies. Among them, in the training process of algorithm models, due to the large amount of training data and large calculation scale, distributed training methods are often used to perform algorithm model training. train.

The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered to be recognized in any prior art.

SUMMARY OF THE INVENTION

The present disclosure provides a distributed training method for a model, a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

According to an aspect of the present disclosure, a distributed training method for a model is provided, including: acquiring multiple containers on multiple nodes, each container in the multiple containers is deployed with a training module, and the training module The training task can be executed in response to a training request corresponding to each training task in the plurality of training tasks; in response to receiving the first request corresponding to the first training task, determining from the plurality of containers that the training task corresponds to the first training task multiple training containers corresponding to the task; and respectively sending a first training request corresponding to the first training task to the multiple training containers, so that the training modules in each of the multiple training containers are trained The first training task is executed, thereby training a model corresponding to the first training task.

According to another aspect of the present disclosure, there is provided a data processing method, comprising: acquiring data to be processed; and inputting the data to be processed into a processing model, the processing model being distributed training using the model according to the present disclosure obtained by training the method.

According to another aspect of the present disclosure, there is provided a distributed training apparatus for a model, comprising: a container acquisition unit configured to acquire a plurality of containers on a plurality of nodes, each container in the plurality of containers A training module is deployed, the training module is capable of executing the training task in response to a training request corresponding to each training task in the plurality of training tasks; the determining unit is configured to respond to receiving a training task corresponding to the first training task a first request, for determining a plurality of training containers corresponding to the first training task from the plurality of containers; and a training request unit, configured to respectively send the plurality of training containers corresponding to the first training task the first training request, so that the training module in each of the plurality of training containers executes the first training task, thereby training a model corresponding to the first training task.

According to another aspect of the present disclosure, there is provided a data processing apparatus, comprising: a data acquisition unit configured to acquire data to be processed; and a data input unit configured to input the data to be processed into a processing model , the processing model is obtained by using the distributed training method of the model according to the present disclosure.

According to another aspect of the present disclosure, there is provided an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores memory executable by the at least one processor The instructions are executed by the at least one processor to cause the at least one processor to implement a method according to the above.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to implement the method according to the above.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to the above.

According to one or more embodiments of the present disclosure, by deploying a training module capable of executing multiple training tasks in response to a training request in containers on multiple nodes, and sending a training request to each training container, the training module of each training container is enabled Execute training tasks in response to training requests to realize distributed training of the model; since the training module is deployed in the container of the node, the deployment efficiency of the distributed training platform of the model is high, and the deployed platform is easy to expand, reducing usage and maintenance. cost.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings illustrate the embodiments by way of example and constitute a part of the specification, and together with the written description of the specification serve to explain exemplary implementations of the embodiments. The shown embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numbers refer to similar but not necessarily identical elements.

1 shows a flowchart of a distributed training method of a model according to an embodiment of the present disclosure;

2 shows a flowchart of a process of acquiring multiple containers on multiple nodes in a distributed training method for a model according to an embodiment of the present disclosure;

3 shows a schematic diagram of a training platform in a distributed training method for a model according to an embodiment of the present disclosure;

4 shows a flowchart of a distributed training method for a model according to an embodiment of the present disclosure;

FIG. 5 shows a structural block diagram of a distributed training apparatus for a model according to an embodiment of the present disclosure;

FIG. 6 shows a structural block diagram of a data processing apparatus according to an embodiment of the present disclosure; and

7 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship or importance relationship of these elements, and such terms are only used for Distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases they may refer to different instances based on the context of the description.

The terminology used in the description of the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly dictates otherwise, if the number of an element is not expressly limited, the element may be one or more. Furthermore, as used in this disclosure, the term "and/or" covers any and all possible combinations of the listed items.

In the related art, a cluster-based distributed architecture is used to implement distributed model training. For example, distributed model training is implemented using the distributed architecture of spark, hadoop, and mpi. These distributed architectures often involve cluster-based application deployment and management. Therefore, it is necessary to solve the problem of cluster control to avoid loss of efficiency. However, the deployment and management of clusters involves complex technologies and applications (for example, K8S), which makes the use and maintenance costs of the distributed model training architecture relatively high, and the deployment is inconvenient, making it unsuitable for small users (for example, for training platforms). number of computing devices) is required.

To this end, a distributed model training method and apparatus are provided to solve the above problems.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

Referring to FIG. 1, a distributed training method 200 of a model according to some embodiments of the present disclosure includes:

Step S110: Acquire a plurality of containers on a plurality of nodes, each container in the plurality of containers is deployed with a training module, and the training module can respond to training corresponding to each training task in the plurality of training tasks request the execution of the training task;

Step S120: in response to receiving the first request corresponding to the first training task, determining a plurality of training containers corresponding to the first training task from the plurality of containers; and

Step S130: Send a first training request corresponding to the first training task to the multiple training containers, so that the training module in each of the multiple training containers executes the first training task to train a model corresponding to the first training task.

By deploying training modules capable of executing multiple training tasks in response to training requests in containers on multiple nodes, sending training requests to each training container, and enabling the training modules of each training container to execute training tasks in response to the training requests, the distribution of the model is realized Since the training module is deployed in the container of the node, the deployment efficiency of the distributed training platform is high, and the deployed platform is easy to expand, reducing the use and maintenance costs.

In the embodiment according to the present disclosure, by deploying a training module capable of executing multiple training tasks in response to a training request on a node formed by a computing device, the distributed training of the model can be realized without using cluster-based distributed training. Therefore, it does not involve application deployment and management of clusters, which can reduce the cost of use and maintenance, and the container-based deployment makes deployment convenient and can be applied to small users (for example, the number of computing devices applied to the training platform is small) Usage requirements.

In some embodiments, nodes are multiple computing devices, eg, computers.

In some embodiments, containers on multiple nodes may be created based on the deployment of training modules. Among them, the container is started after creation.

In some embodiments, the training module may be a service deployed on a container, which includes an algorithm service corresponding to multiple training tasks and a response service, the response service may respond to the training request and start the algorithm corresponding to the training request service, so as to carry out the corresponding training tasks.

In some embodiments, the multiple training tasks may be face detection, image classification, etc., which are not limited herein.

In some embodiments, the model corresponding to the training task may be a face detection model, an image classification model, or the like.

In some embodiments, an algorithm service may be a service that encapsulates an algorithm corresponding to a training task. For example, the algorithm may be a face detection algorithm, an image classification algorithm, etc., which is not limited herein.

In some embodiments, the training module may also include other services such that the response service initiates the service to perform the corresponding task based on the response to the training request. Other services, for example, may be data acquisition services for acquiring corresponding training data.

In some embodiments, as shown in FIG. 2 , acquiring multiple containers on multiple nodes includes:

Step S210: obtaining a mirror image of the training module;

Step S220: Mount the image on the first node of the plurality of nodes; and

Step S220: Acquire the container on the first node in response to the first node running the image.

By obtaining the image of the training module and by mounting the image, the node obtains the image of the training module. When the image is run, the container is started, which makes the deployment of the training module efficient and easy to expand.

For example, when a user adds a node, he only needs to mount the image of the training module on the added node and run the image on the node, then the training module can be deployed to perform distributed training of the model, so that the The deployment efficiency of the training module is greatly improved, and it is easy to expand.

Referring to Figure 3, a schematic diagram of a distributed training platform for a model according to some embodiments of the present disclosure is shown.

The distributed training platform 300 of the model includes a plurality of nodes (nodes 310-3n0), and a training module (training module 311-3n1) is deployed on each node through a container, and n is a positive integer.

The process of the distributed training method of the model according to the present disclosure is implemented by the node 310 . The node 310 obtains multiple training nodes from multiple nodes (nodes 310-3n0) in response to obtaining the first request corresponding to the first training task, so as to obtain multiple training containers. And a corresponding training request is generated based on the first request to be sent to the plurality of training containers.

In some embodiments, the first request may be a request received from a user through an input and output device. The first request includes information indicating training nodes corresponding to the first training task, data information, and the like, which is not limited herein.

In one example, the information of the training node may be the IP address of the training node; the data information may be information indicating the training data set required for the first training task and the labeling label of each training data in the training data set.

In some embodiments, the first request includes training parameters related to the training task, and the method further includes:

The training request is generated, the training request indicating the training parameters.

By sending the training parameters determined by the user to the training container in the form of a training request, the training module in the training container executes the training task based on the corresponding training request, realizing the distributed training of the model, and the training task of the distributed training of the model. Easy to expand.

In some embodiments, the training parameters are hyperparameters of the model that are adjusted in the process other than based on loss. Hyperparameters can be, for example, the regularization coefficient λ, the depth of the tree in the decision tree model. By setting hyperparameters, different algorithm combinations can be set to achieve different training tasks.

In some embodiments, as shown in FIG. 4 , the method according to the present disclosure further comprises:

Step S410: Obtain training data information corresponding to the first training task, the training data information indicating the training data set of the first training task and the labeling label of each training data in the training data set; and

Step S420: Send the training data information to the multiple training containers respectively, so that multiple training modules in the multiple training containers obtain the training data respectively.

By obtaining the training data information and sending the training data information to the training container, the training data and the training module are deployed separately, which further simplifies the deployment of the distributed training platform.

In some embodiments, the training data information is obtained based on the data information indicated in the first request.

For example, the first request includes the address of the training data, and the training data information is obtained based on the address of the training data.

In some embodiments, the training data information is obtained by obtaining training data uploaded by the user.

In some embodiments, the training data information includes each training data in the training data set and a label of the training data.

By sending the training data to each training container, the training container is trained based on the received training data and the labels of the training data, which further simplifies the deployment of the distributed training platform.

In some embodiments, the training data information includes each training data obtaining address in the training data set, and the training data can be obtained by accessing based on the obtaining address.

In the case of a large amount of data in the training data set, mount the training data set to the storage module, and send the address of each training data in the training data set, so that the training module in the training container can be accessed based on the address. In order to obtain the training data, the storage space for storing the training data set can be reduced, and the communication link congestion caused by the simultaneous transmission of the training data set to multiple training modules can also be avoided.

For example, the training data set is stored on the network-attached storage (NAS), and the training container is made to access the network based on the acquisition address of each training data by sending the acquisition address of each training data in the training data set on the network-attached storage to the training container. Affiliated storage to obtain the training dataset.

In some embodiments, the method according to the present disclosure further comprises:

Obtain a data sequence corresponding to each training container in the plurality of training containers, the data sequence instructing the training module of the training container to input a plurality of training data in the training data set when executing the first training task and wherein the sending the training data information to the plurality of training containers respectively includes:

The training data information is sent to the training container based on the corresponding data sequence of each training container in the plurality of training containers.

By obtaining the corresponding data sequence of each training container, and sending training data information to the training container based on the corresponding data sequence, the training module can input multiple training data in the training data set based on the corresponding data sequence when the training module executes the first training task. data, that is, the training modules of each training container use the same training data set when performing the first training task, but the order of inputting the training data in the training data set is different, so that the trained model takes the training data into account. The training results of the input order are more robust.

In some embodiments, the training data information includes each training data in the training data set and a label of the training data. When the training data information is sent based on the data sequence corresponding to each training container, the training data set is arranged based on the data sequence. After the training data, each training data in the arranged training data set is transmitted in sequence.

In some embodiments, the training data information includes a label and an acquisition address of each training data in the training data set, and when the training data information is sent based on the corresponding data sequence of each training container, the training data in the training data set is arranged based on the data sequence. After obtaining the address of the data, the obtaining address of each training data in the arranged training data set is sent in sequence.

In some embodiments, a service for obtaining data sequence may also be deployed in the training module, so that the training module of each training container inputs multiple data in the training data set based on its corresponding data sequence when executing the first training task. training data.

For example, a function for obtaining the input order of the training data, eg, a random function, is encapsulated in the training module to sort a plurality of training data in the obtained training data set to obtain their corresponding data order.

In some embodiments, during the training process, the containers communicate with each other to synchronize gradient and parameter information during the training process. For example, they communicate with each other through NCCL (Nvidia Collective multi-GPU Communication Library).

In some embodiments, the method according to the present disclosure further comprises:

An acquisition request is sent to the plurality of training containers to acquire the loss obtained by the training module of each of the plurality of training containers after executing the first training task.

By sending an acquisition request to the training container, the loss in the training process is acquired and displayed to the user to realize real-time monitoring of the distributed training process of the model.

According to another aspect of the present disclosure, a data processing method is also provided, comprising:

obtain data to be processed; and

The data to be processed is input into a processing model obtained by training using the distributed training method of the model according to the present disclosure.

In some embodiments, the processing model may be an object detection model, a character recognition model, etc., which is not limited herein.

In some embodiments, the data to be processed may be image data, audio data, etc., which is not limited herein.

According to another aspect of the present disclosure, a distributed training apparatus for a model is also provided. As shown in FIG. 5 , the apparatus 500 includes: a container acquisition unit 510 configured to acquire a plurality of containers on a plurality of nodes, Each of the plurality of containers is deployed with a training module, and the training module is capable of executing the training task in response to a training request corresponding to each training task in the plurality of training tasks; the determining unit 520 is configured to In response to receiving the first request corresponding to the first training task, determining a plurality of training containers corresponding to the first training task from the plurality of containers; and a training request unit 530 configured to respectively send the plurality of training containers to the plurality of containers; A training container sends a first training request corresponding to the first training task, so that the training module in each training container of the plurality of training containers executes the first training task, thereby training and training the first training task. A model corresponding to the training task.

In some embodiments, the container obtaining unit 510 includes: an image obtaining unit configured to obtain an image of the training module; a mounting unit configured to mount the image to the plurality of nodes on a first node in the ; and an obtaining subunit configured to obtain a container on the first node in response to the first node running the image.

In some embodiments, the first request includes training parameters related to the training task, and the apparatus further includes: a training request generating unit configured to generate the training request, the training request indicating the training parameters.

In some embodiments, it further includes: a training data obtaining unit configured to obtain training data information corresponding to the first training task, the training data information indicating the training data set and all the training data of the first training task a label for each training data in the training data set; and a sending unit configured to send the training data information to the plurality of training containers respectively, so that the plurality of training modules in the plurality of training containers Obtain the training data separately.

In some embodiments, the training data information includes each training data in the training data set and a label of the training data.

In some embodiments, the training data information includes a label and an obtaining address of each training data in the training data set, and the training data can be obtained by accessing based on the obtaining address.

In some embodiments, further comprising: a sequence obtaining unit configured to obtain a data sequence corresponding to each training container in the plurality of training containers; and the sending unit further comprising: a sending subunit configured for sending the training data information to the training container based on the corresponding data sequence of each training container in the plurality of training containers, so that the training module of the training container performs the first training task when the training module The training data set is input according to the corresponding data sequence of the training container.

In some embodiments, the method further includes: a loss request unit configured to send an acquisition request to the plurality of training containers to obtain the execution of the training module of each training container in the plurality of training containers. The loss obtained after the first training task is described.

According to another aspect of the present disclosure, a data processing apparatus is also provided. As shown in FIG. 6 , the apparatus 600 includes: a data acquisition unit 610 configured to acquire data to be processed; and a data input unit 620 configured to use The data to be processed is input into a processing model, and the processing model is obtained by training the distributed training method of the model according to the present disclosure.

According to another aspect of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be used by the at least one processor Instructions to be executed, the instructions being executed by the at least one processor to enable the at least one processor to perform a method according to the present disclosure.

According to another aspect of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to the present disclosure.

According to another aspect of the present disclosure, there is also provided a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to the present disclosure.

Referring to FIG. 7 , a structural block diagram of an electronic device 700 that can function as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7 , the electronic device 700 includes a computing unit 701 that can be programmed according to a computer program stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708 . Various appropriate actions and processes are performed. In the RAM 703, various programs and data necessary for the operation of the electronic device 700 can also be stored. The computing unit 701 , the ROM 702 , and the RAM 703 are connected to each other through a bus 704 . An input/output (I/O) interface 705 is also connected to bus 704 .

Various components in the electronic device 700 are connected to the I/O interface 705 , including: an input unit 706 , an output unit 707 , a storage unit 708 , and a communication unit 709 . The input unit 706 may be any type of device capable of inputting information to the electronic device 700, the input unit 706 may receive input numerical or character information, and generate key signal input related to user settings and/or function control of the electronic device, and This may include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and/or remote control. The output unit 707 may be any type of device capable of presenting information, and may include, but is not limited to, a display, speaker, object/audio output terminal, vibrator, and/or printer. The storage unit 708 may include, but is not limited to, magnetic disks and optical disks. The communication unit 709 allows the electronic device 700 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chips Groups such as Bluetooth™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and/or the like.

Computing unit 701 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 701 performs the various methods and processes described above, eg, method 200 . For example, in some embodiments, method 200 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 700 via the ROM 702 and/or the communication unit 709 . When a computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of method 200 described above may be performed. Alternatively, in other embodiments, computing unit 701 may be configured to perform method 200 in any other suitable manner (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-described methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is limited only by the appended claims and their equivalents. Various elements of the embodiments or examples may be omitted or replaced by equivalents thereof. Furthermore, the steps may be performed in an order different from that described in this disclosure. Further, various elements of the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims (21)

1. A distributed training method for a model, comprising: Obtain a plurality of containers on a plurality of nodes, each of the plurality of containers is deployed with a training module, and the training module is capable of executing the training request in response to a training request corresponding to each of the plurality of training tasks. training tasks; in response to receiving a first request corresponding to a first training task, determining a plurality of training containers corresponding to the first training task from a plurality of containers; and Send a first training request corresponding to the first training task to the plurality of training containers, so that the training module in each training container of the plurality of training containers executes the first training task, thereby A model corresponding to the first training task is trained. 2. The method of claim 1 , wherein the acquiring a plurality of containers on a plurality of nodes comprises: obtain a mirror image of the training module; mounting the image on a first node of the plurality of nodes; and In response to the first node running the image, a container on the first node is obtained. 3. The method of claim 1 or 2, wherein the first request includes training parameters related to the first training task, the method further comprising: The training request is generated, the training request indicating the training parameters. 4. The method of claim 1 or 2, further comprising: obtaining training data information corresponding to the first training task, the training data information indicating a training data set for the first training task and a label for each training data in the training data set; and The training data information is respectively sent to the multiple training containers, so that multiple training modules in the multiple training containers obtain the training data respectively. 5. The method of claim 4, wherein the training data information includes each training data in the training data set and a label of the training data. 6 . The method according to claim 4 , wherein the training data information includes a label and an acquisition address of each training data in the training data set, and the training data can be acquired by accessing based on the acquisition address. 7 . 7. The method of claim 4, further comprising: Obtain a data sequence corresponding to each training container in the plurality of training containers, the data sequence instructing the training module of the training container to input a plurality of training data in the training data set when executing the first training task and wherein the sending the training data information to the plurality of training containers respectively includes: The training data information is sent to the training container based on the corresponding data sequence of each training container in the plurality of training containers. 8. The method of any one of claims 1-7, further comprising: An acquisition request is sent to the plurality of training containers to acquire the loss obtained by the training module of each of the plurality of training containers after executing the first training task. 9. A data processing method, comprising: obtain data to be processed; and The data to be processed is input into a processing model, and the processing model is obtained by training the method according to any one of claims 1-8. 10. A distributed training device for a model, comprising: a container acquisition unit configured to acquire a plurality of containers on a plurality of nodes, each of the plurality of containers is deployed with a training module, the training module can be responsive to a plurality of training tasks with each The training request corresponding to the training task executes the training task; a determining unit configured to, in response to receiving a first request corresponding to a first training task, determine a plurality of training containers corresponding to the first training task from the plurality of containers; and A training request unit, configured to send a first training request corresponding to the first training task to the plurality of training containers, so that the training module in each of the plurality of training containers executes the first training task, thereby training a model corresponding to the first training task. 11. The apparatus of claim 10, wherein the container acquisition unit comprises: an image acquisition unit configured to acquire an image of the training module; a mounting unit configured to mount the image on a first node of the plurality of nodes; and An obtaining subunit is configured to obtain a container on the first node in response to the first node running the image. 12. The apparatus of claim 10 or 11, wherein the first request includes training parameters related to the first training task, the apparatus further comprising: A training request generating unit configured to generate the training request, the training request indicating the training parameter. 13. The apparatus of claim 10 or 11, further comprising: A training data obtaining unit configured to obtain training data information corresponding to the first training task, the training data information indicating the training data set of the first training task and each training data in the training data set label; and The sending unit is configured to send the training data information to the multiple training containers respectively, so that multiple training modules in the multiple training containers obtain the training data respectively. 14. The apparatus of claim 13, wherein the training data information includes each training data in the training data set and a label of the training data. 15. The apparatus according to claim 13, wherein the training data information includes a label and an acquisition address of each training data in the training data set, and the training data can be acquired by accessing based on the acquisition address. 16. The apparatus of claim 13, further comprising: a sequence obtaining unit configured to obtain a data sequence corresponding to each training container in the plurality of training containers, the data sequence instructing a training module of the training container to input the data when performing the first training task The sequence of multiple training data in the training data set; and wherein the sending unit further includes: The training data information is sent to the training container based on the corresponding data sequence of each training container in the plurality of training containers. 17. The apparatus of any of claims 10-16, further comprising: A loss requesting unit, configured to send an acquisition request to the plurality of training containers to acquire the data obtained by the training module of each training container in the plurality of training containers after executing the first training task loss. 18. A data processing apparatus comprising: a data acquisition unit configured to acquire the data to be processed; and A data input unit, configured to input the data to be processed into a processing model, the processing model being obtained by training the method according to any one of claims 1-8. 19. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-9 Methods. 20. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any of claims 1-9. 21. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-9.

Images