CN114925591A - An automatic parallel strategy search method and related equipment based on polyhedral model modeling - Google Patents

Abstract

The invention discloses an automatic parallel strategy searching method based on polyhedral model modeling and related equipment, wherein the method comprises the following steps: obtaining a model calculation graph of a deep learning algorithm according to a model object input by a user; converting the model calculation graph to obtain a converted model calculation graph; carrying out equalization processing on the converted model calculation graph to obtain an equalization calculation graph; according to the equilibrium calculation diagram, creating a polyhedral model example, and outputting a parallel strategy according to the polyhedral model example; and calling the bottom framework to execute the parallel strategy. According to the method, the model calculation diagram is converted and balanced, and after the polyhedral model instance is created under the framework of the polyhedral model, the parallel strategy is automatically output, so that different algorithm logics are modeled under the polyhedral model, the parallel strategy process is automatically output, the efficiency of parallel strategy search is improved, and the distributed training development and efficiency tuning difficulty of the deep learning algorithm are reduced.

Description

An automatic parallel strategy search method and related equipment based on polyhedral model modeling

technical field

The invention relates to the technical field of artificial intelligence, in particular to an automatic parallel strategy search method based on polyhedral model modeling and related equipment.

Background technique

In the past ten years, deep learning technology has continuously refreshed the records of various tasks in the fields of vision, natural language, speech, search, and recommendation. The reason for this, described with a keyword is "large-scale". Large-scale data enables the model to have enough knowledge to memorize, the model with large-scale parameters enables the model itself to be able to memorize more data, and the large-scale high-performance computing power (typically represented by GPU) enables the model to train faster. A hundred times or even a thousand times the improvement. The development of data, models, and computing power has given birth to the field of large-scale deep learning. How to split multi-machine tasks, how to configure cluster training resources, how to balance training speed and convergence speed, how to train models that cannot be trained on a single machine, and elastic training and fault tolerance are the key research issues in this direction. Distributed training is the most effective way to solve the above problems and improve training efficiency. The core purpose of distributed training is to speed up the training speed of the model.

At present, mainstream deep learning frameworks such as TensorFlow (TensorFlow is a symbolic mathematics system based on data flow programming, which is widely used in the programming implementation of various machine learning algorithms), Pytorch (PyTorch is an open source Python machine learning library, based on Torch, use applications such as natural language processing), Mindspore (MindSpore is a new open source deep learning training/inference framework suitable for end-to-end cloud scenarios), PaddlePaddle (PaddlePaddle) is a collection of deep learning core frameworks, tool components and services The platform is an open source deep learning platform with advanced technology and complete functions) with multi-machine distributed training function. It is divided into multiple computing devices for distributed computing), operator parallelism, pipeline parallelism (pipeline technology refers to a quasi-parallel processing implementation technology in which multiple instructions overlap and operate during program execution) and other modes. However, these parallel The mode requires algorithm developers to call the parallel segmentation API provided by the AI framework according to the characteristics of the algorithm model. This method increases the technical difficulty of distributed training of AI algorithms. At the same time, because the algorithm developers do not have sufficient grasp of the characteristics of the AI framework and computing equipment, resulting in The model parallel training efficiency is low, and the specific distributed tuning work increases the difficulty of algorithm development and reduces the efficiency of algorithm research.

In response to this problem, the Mindspore framework proposes an automatic parallel training function for models, the FlexFlow framework also proposes a search strategy based on 4-dimensional parallel strategy space modeling, and the RaNNC framework proposes a pipeline parallel strategy that supports Pytorch front-end automatic However, due to the large scale of the parallel strategy search space (related to the scale of the computational graph and the scale of the resource space), the above work is difficult to be practical in terms of automatic parallel search efficiency. When the enlarged model performs a pipeline parallel strategy search on 4 nodes and 32 cards, the required strategy search time reaches more than 4 hours, which improves the debugging and training time during model training and development to a certain extent, and reduces the efficiency. Therefore, the existing technology still needs to be improved and improved.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide an automatic parallel strategy search method and related equipment based on polyhedral model modeling, aiming to solve the problem that in the prior art, when training a large-scale deep learning model, algorithm developers need to configure themselves Parallel strategy, this will cause problems of low training efficiency and difficult development.

In order to achieve the above object, the present invention has adopted the following technical solutions:

An automatic parallel strategy search method based on polyhedral model modeling, including:

Obtain the model calculation graph of the deep learning algorithm according to the model object input by the user;

Converting the model calculation graph to obtain the converted model calculation graph;

Perform equalization processing on the converted model calculation graph to obtain a balanced calculation graph;

Create a polyhedron model instance according to the equilibrium calculation graph, and output a parallel strategy according to the polyhedron model instance;

The underlying framework is invoked to execute the parallel strategy.

In the automatic parallel strategy search method based on polyhedral model modeling, the step of obtaining the model calculation graph of the deep learning algorithm according to the model object input by the user specifically includes:

Obtain an algorithm model according to the model object input by the user;

After randomly inputting a numerical value to the algorithm model, record the calculation process of the algorithm model to obtain the model calculation diagram;

Or generate a syntax tree after parsing the model object by a python interpreter, and then analyze the syntax tree to obtain the model computation graph.

In the automatic parallel strategy search method based on polyhedral model modeling, the step of converting the model computation graph to obtain the converted model computation graph specifically includes:

The model computation graph is re-expressed with a pre-defined intermediate representation to obtain a converted model computation graph.

In the automatic parallel strategy search method based on polyhedral model modeling, the step of performing equalization processing on the converted model calculation graph to obtain the balanced calculation graph specifically includes:

Set the average computation amount threshold of the node, and compare the computation amount node in the converted model computation graph with the average computation amount threshold;

The adjacent calculation amount nodes smaller than the average calculation threshold are fused, and the calculation amount nodes larger than the average calculation threshold are split to obtain a balanced calculation graph.

In the automatic parallel strategy search method based on polyhedral model modeling, the steps of creating a polyhedral model instance according to the equilibrium calculation graph, and outputting a parallel strategy according to the polyhedral model instance specifically include:

Map the equilibrium calculation graph on the polyhedron model to obtain a polyhedron optimization model;

Input the equilibrium calculation graph into the polyhedron optimization model to obtain an instance of the polyhedron model;

A parallel strategy is output according to the polyhedral model instance and the number of computing resources input by the user.

In the automatic parallel strategy search method based on polyhedral model modeling, the step of invoking the underlying framework to execute the parallel strategy specifically includes:

The execution API of the underlying framework is called to execute the parallel strategy.

In the automatic parallel strategy search method based on polyhedral model modeling, the model object refers to the single-machine training code of the deep learning algorithm predefined by the user.

In the automatic parallel strategy search method based on polyhedral model modeling, the pre-defined intermediate representations include: IRType, IRValue, IRNode and IRGraph.

In the automatic parallel strategy search method based on polyhedral model modeling, the parallel strategy includes a data parallel segmentation dimension and a pipeline parallel segmentation dimension.

In the automatic parallel strategy search method based on polyhedral model modeling, the execution API is an operation manager in the underlying AI framework.

An automatic parallel strategy search system, the automatic parallel strategy search system further comprises:

The calculation graph generation module is used to obtain the model calculation graph of the deep learning algorithm according to the model object input by the user;

a computational graph conversion module, configured to convert the model computational graph to obtain the converted model computational graph;

The calculation graph equalization module is used to equalize the converted model calculation graph to obtain a balanced calculation graph;

a parallel strategy search module, configured to create a polyhedron model instance according to the equilibrium calculation graph, and output a parallel strategy according to the polyhedron model instance;

The parallel strategy execution module is used for invoking the underlying framework to execute the parallel strategy.

A controller comprising: a memory, a processor, and an automatic parallel strategy search program based on polyhedral model modeling stored on the memory and executable on the processor, the polyhedral model-based modeling The modular automatic parallel strategy search program is executed by the processor to implement the steps of the automatic parallel strategy search method based on polyhedral model modeling as described above.

A computer-readable storage medium that stores an automatic parallel strategy search program based on polyhedral model modeling, and the automatic parallel strategy search program based on polyhedral model modeling is executed by a processor. The steps of the described automatic parallel policy search method based on polyhedral model modeling.

Compared with the prior art, the present invention provides an automatic parallel strategy search method and related equipment based on polyhedral model modeling. Model calculation graph of the deep learning algorithm; convert the model calculation graph to obtain a converted model calculation graph; perform equalization processing on the converted model calculation graph to obtain a balanced calculation graph; create a polyhedron according to the balanced calculation graph model instance, and output a parallel strategy according to the polyhedron model instance; invoke the underlying framework to execute the parallel strategy. In the present invention, the generated model calculation graph is converted and equalized, and a polyhedron model instance is created based on the frame of the polyhedron model, so as to automatically output a parallel strategy according to the polyhedron model instance, and the algorithm logic under different frameworks is realized under the polyhedron model. The process of modeling and automatically outputting parallel strategies that can be executed efficiently, thus effectively improving the efficiency of searching for parallel strategies, and reducing the difficulty of distributed training development and efficiency tuning of deep learning algorithms.

Description of drawings

1 is a flowchart of a preferred embodiment of an automatic parallel strategy search method based on polyhedral model modeling provided by the present invention;

Fig. 2 is the flow chart of step S100 in the preferred embodiment of the automatic parallel strategy search method based on polyhedral model modeling provided by the present invention;

3 is a flowchart of step S300 in a preferred embodiment of the automatic parallel strategy search method based on polyhedral model modeling provided by the present invention;

4 is a schematic diagram of node splitting and node aggregation provided by the present invention;

5 is a flowchart of step S400 in a preferred embodiment of the automatic parallel strategy search method based on polyhedral model modeling provided by the present invention;

FIG. 6 is a schematic diagram of computing graph segmentation under the data parallel mode provided by the present invention;

7 is a schematic diagram of a computation graph segmentation in a pipeline parallel mode provided by the present invention;

8 is a schematic block diagram of an automatic parallel strategy search system provided by the present invention;

Fig. 9 is the architecture relationship diagram of the Pytorch framework provided by the present invention and the automatic parallel strategy search system;

FIG. 10 is a schematic diagram of the operating environment of the preferred embodiment of the controller provided by the present invention.

Detailed ways

In order to make the objectives, technical solutions and effects of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

It will be understood by those skilled in the art that the singular forms "a", "an", "the" and "the" as used herein can include the plural forms as well, unless expressly stated otherwise. It should be further understood that the word "comprising" used in the description of the present invention refers to the presence of stated features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components and/or groups thereof. It will be understood that when we refer to an element as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Furthermore, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combination of one or more of the associated listed items.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms, such as those defined in a general dictionary, should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, should not be interpreted in idealistic or overly formal meaning to explain.

First of all, distributed training of deep learning models based on multi-machine and multi-card has become the most important technical solution to accelerate model training efficiency. The main distributed training modes include data parallelism, model parallelism, optimizer parallelism, pipeline parallelism, hybrid parallelism, etc. , These parallel methods are also the parallel functions mainly supported by the current mainstream deep learning frameworks (such as TensorFlow, Pytorch, Mindspore, PaddlePaddle). Different AI frameworks only have differences in ease of use and efficiency. However, almost all of the above parallel methods require algorithm developers. It is implemented by calling the framework API, and at the same time, manual training efficiency tuning is performed, which is very difficult for algorithm developers who do not know much about the implementation mechanism of the underlying AI framework and the characteristics of cluster communication. Complex implementation and debugging bring great efficiency. reduce.

On the other hand, because the search efficiency of automatic parallel strategy is related to the exponential growth of AI model calculation graph and cluster resource scale, the search efficiency of some existing automatic parallel search work is difficult to meet the efficiency requirements of large-parameter models on large-scale clusters. Therefore, it is necessary to Design and implement an efficient multi-machine and multi-card distributed training parallel strategy search method to solve the problem of automatic parallel training strategy search for large models, so that algorithm developers can only focus on the development of algorithm logic, and quickly implement them in AI. Distributed training on a cluster.

In order to solve the above problems in the prior art, the present invention provides an automatic parallel strategy search method and related equipment based on polyhedral model modeling. In the present invention, the generated model calculation graph is converted and equalized, the calculation graph is balanced, and under the framework of the polyhedron model, a polyhedral model instance is created according to the balanced calculation graph, so as to automatically output a parallel strategy according to the polyhedral model instance, and the Under the polyhedron model, the algorithm logic under different frameworks is modeled, and the process of automatically outputting parallel strategies that can be executed efficiently, thus effectively improving the efficiency of searching for parallel strategies, and reducing the distributed training development and efficiency of deep learning algorithms. Tuning difficulty.

The design scheme of the automatic parallel strategy search method based on polyhedral model modeling will be described below through specific exemplary embodiments. It should be noted that the following embodiments are only used to explain the technical scheme of the invention, and do not specifically limit it. :

Referring to Fig. 1, an automatic parallel strategy search method based on polyhedral model modeling provided by the present invention includes:

S100. Obtain a model calculation graph of the deep learning algorithm according to the model object input by the user.

Among them, the model object refers to the stand-alone training code of the deep learning algorithm pre-defined by the user, such as the BERT model and the GPT3 model (GPT-3 is constructed by OpenAI, an independent AI research and deployment company, and is a large-scale natural language model. , currently running on Microsoft Azure), etc.

Specifically, under the frameworks of different deep learning models (such as TensorFlow, Pytorch, Mindspore, PaddlePaddle), the corresponding model objects (the algorithm logic predefined by the user) are different, so it is necessary to adopt the corresponding method based on the user. The input single-machine training code code based on the deep learning algorithm defined by the AI framework obtains the model calculation graph of the deep learning algorithm. Among them, the model calculation graph refers to a representation of deep learning algorithms in the AI framework. Currently, a directed acyclic graph (DAG) is generally used to represent the calculation process of a deep learning algorithm, and the nodes of the calculation graph represent a calculation operation. , the edges represent tensor data dependencies between algorithmic computation operations. The function of the above process of generating the model calculation graph is to represent the calculation process of an algorithm.

Further, referring to FIG. 2, step 100 specifically includes:

S110, obtaining an algorithm model according to the model object input by the user;

S120, after randomly inputting a numerical value to the algorithm model, record the calculation process of the algorithm model, and obtain the model calculation diagram;

S130, or generate a syntax tree after parsing the model object by a python interpreter, and then analyze the syntax tree to obtain the model computation graph.

Specifically, taking the Pytorch framework as an example in the present invention, the model calculation graph defined by the algorithm developer can be obtained by two methods: jit.trace (tracing technology) or jit.script (source code conversion technology); wherein, the jit.trace method It is based on the method of vector calculation and tracking. After obtaining an algorithm model according to the model object input by the user, randomly input a value according to the input type of the algorithm, and then record each calculation process of the algorithm. These recorded calculation processes are constructed as A computational graph; for example, the algorithm model is to input a 32X32 picture, and the output is the classification label of this picture, then the random input refers to the random input of a 32X32 data. The jit.script method is based on source code conversion. Its basic principle is to first parse the model object (user-defined algorithm logic) into a syntax tree (a tree-like structure that describes the truth of the world) through the python interpreter, and then analyze the The syntax tree gets the model computation graph.

Please continue to refer to FIG. 1 . In S200 , the model calculation graph is converted to obtain a converted model calculation graph.

Specifically, since there are many ways to define the computation graphs under different frameworks, that is, the computation graphs under different frameworks represent differences. For example, the Pytorch framework uses the Torch computation graph (Torch Graph), so it is necessary to convert the computation graphs of different frameworks into a single graph. It is a general-purpose intermediate representation of the calculation graph, and usually, the intermediate representation method of the calculation graph is manually defined by experts, which is a process of manually establishing rule mapping, so the intermediate representation conversion of different calculation graphs is based on their own rules. , in the present invention, a new calculation graph representation method is defined for parallel strategy search, so it is necessary to convert the upper-layer framework calculation graph into a calculation graph used in parallel strategy search.

Further, step S200 specifically includes:

S210. Re-represent the model calculation graph with a predefined intermediate representation to obtain a converted model calculation graph.

Specifically, the computation graphs under different frameworks are re-expressed with the intermediate representations predefined by the user, so that the computation graphs under different frameworks are converted into the computation graphs used in the parallel strategy search in the present invention. The intermediate representations are a kind of computation graph. The general method is specifically defined as four class definitions in the C++ implementation; wherein, the pre-defined intermediate representations are: IRType (type), IRValue (value), IRNode (node) and IRGraph (graph).

Please continue to refer to FIG. 1, S300, performing equalization processing on the converted model calculation graph to obtain an equalized calculation graph.

Specifically, because the calculation amount of many different node operations in the calculation graph is different, the calculation graph is unbalanced. In order to obtain small nodes of balanced size when the nodes are segmented in the following steps, and in order to improve the automatic search parallel strategy The efficiency of the calculation graph needs to be balanced, that is, node splitting and node aggregation. Refers to the re-aggregation or splitting of nodes in the calculation graph, which can make the transformed calculation graph achieve a calculation balance in two dimensions and avoid uneven calculation distribution during parallel strategy search.

Further, referring to FIG. 3 , step S300 specifically includes:

S310, setting the average computation amount threshold of the node, and comparing the computation amount nodes in the converted model computation graph with the average computation amount threshold;

S320 , fuse adjacent nodes with calculation amount smaller than the average calculation threshold, and split the nodes with calculation amount larger than the average calculation threshold to obtain a balanced calculation graph.

Specifically, the equalization process mainly includes the aggregation or splitting operation of the nodes in the model calculation graph; before the node aggregation or node differential operation, first, it is necessary to set the average calculation amount threshold of a node, and traverse the transformed nodes according to the order of the nodes. The calculation amount node in the model calculation graph of The node splitting operation is: according to the average calculation threshold, split the calculation node larger than the average calculation threshold into multiple nodes, usually a matrix larger than the average calculation threshold The multiplication operator is split, so that the node calculation amount after fusion and splitting is equivalent to the average calculation amount threshold; after the node aggregation or node split operation is completed, a balanced calculation graph can be obtained.

In the present invention, the large nodes in the calculation graph are split, or the adjacent small nodes are aggregated to obtain the balanced node, that is, the balanced calculation graph is obtained, so that the calculation amount among the nodes is balanced. , which can effectively avoid the problem of uneven calculation distribution during parallel strategy search, and at the same time improve the efficiency of node segmentation.

Among them, node aggregation is to convert the computing process of multiple nodes into one node in representation, for example, three computing nodes are aggregated, and the first three nodes of the aggregation are three IRNode objects respectively, and a new IRNode object is formed after aggregation; In addition, the segmentation of large-scale nodes needs to be determined according to the operator type of the specific node. Not all nodes can be split. For example, matrix multiplication is a node that can be split, and it is also a basic operator commonly used in AI models. The specific operation process of node splitting and node aggregation is shown in Figure 4, which represents the process of splitting large-computation nodes and aggregating adjacent small-computation nodes.

Please continue to refer to FIG. 1, S400, according to the equilibrium calculation graph, create a polyhedron model instance, and output a parallel strategy according to the polyhedron model instance.

Specifically, this process belongs to the training modeling under the model based on polyhedron optimization, and the parallel strategy search process. The training and modeling process refers to initializing under the framework of the established polyhedron optimization model according to the equilibrium calculation graph to create a polyhedron. The optimization model instance is modeled as a specific object; then, the parallel strategy search refers to the optimization model instance based on the polyhedron, according to the specified parallel mode (the specified parallel mode refers to the parallel mode that the developer needs to perform, such as data parallel, Model parallelism, optimizer parallelism, pipeline parallelism, hybrid parallelism, etc.), taking the pipeline parallel strategy search as an example: then according to the pipeline mode specified by the developer, the strategy value of the number of pipelines and the number of mini-batches is searched, and the final strategy value is generated. Parallel strategy. Among them, the parallel mode is a large framework, such as data parallelism, which refers to the process of dividing the training data samples into multiple computing devices for distributed computing during the distributed training of the AI model, and the parallel strategy refers to a specific The distributed training splitting strategy of the algorithm model during execution.

Polyhedral model modeling is a common method for compiler optimization for for loops in compilers. By representing the loop in the polyhedral model space, the parallel computing optimization of the loop can be realized directly through the mapping table of the polyhedral model, so as to improve the training efficiency; in this paper In the invention, the training and calculation process of a deep learning model is represented as a multi-layer loop operation, and then how to represent the training process through a polyhedron model is defined, and finally the common data parallelism, model parallelism, and pipeline parallelism are modeled on the polyhedron. unified under the framework.

Suppose a deep learning model is represented by a directed acyclic graph (DAG) as a computational graph D(N, E), where N is the set of computational graph nodes and E is the set of edges of the computational graph. Therefore, the training calculation process of the deep learning model (in fact, it is also a modeling process) is expressed as follows (expressed in the python programming language), thus realizing the mapping of the for loop to the polyhedron optimization model:

for e in range(Epoch_num): //Epoch_num: The number of training rounds of the model, that is, the range of rounds (domain) of the for loop;

for b in range(Batch_num): //Batch_num: The number of sample batches for one round of model training, that is, the batch number range of the for loop;

for node in nodes: //node: a node of the model; nodes: node collection of the model calculation graph;

out=Forward(node, b)//Forward(): Indicates the forward calculation of the model;

for node in reverse(nodes): //reverse(): Reverse sorting of model computing nodes;

grad=Backward(node, b)//grad: represents the gradient of the model parameters; Backward(): represents the reverse gradient calculation of the model;

Update(node)//update(): Indicates the parameter update process of the model.

Further, referring to FIG. 5 , step S400 specifically includes:

S410. Create a polyhedron optimization model;

S420. Initialize the polyhedron optimization model according to the equilibrium calculation graph to obtain a polyhedron model instance;

S430. Output a parallel strategy according to the polyhedral model instance and the number of computing resources input by the user. Among them, the parallel strategy includes the data parallel segmentation dimension, the pipeline parallel segmentation dimension, and the arrangement of the pipeline.

Specifically, the equilibrium calculation graph is first mapped on the polyhedron model to obtain a polyhedron optimization model, where the "mapping" here can be understood as an affine transformation of the coordinate space, such as translation, rotation and other operations, and then the equilibrium calculation graph is input To the polyhedron optimization model, a polyhedron model instance (specific object) is obtained, that is, the for loop is mapped to the polyhedron optimization model to obtain a polyhedron model instance (finding a distributed optimization calculation model), and then through the geometric linearity Transform to find the data parallel segmentation dimension or pipeline parallel segmentation dimension of distributed training, and combine the number of computing resources input by the user. For example, the computing resources are 8 GPU cards, so the data parallel will be divided from the data parallel segmentation dimension. 8 mini-batches are calculated on 8 cards respectively, and finally the parallel strategy (data parallel segmentation dimension, pipeline parallel segmentation dimension and pipeline arrangement, etc.) is synthesized and output. The specific data parallel segmentation dimension and the pipeline parallel segmentation dimension segmentation process are shown in Figure 6 and Figure 7 respectively, where the abscissa represents the number of sample batches for model training, and the ordinate represents the number of layers of model parameters, such as BERT The basic model calculation process has 24 layers; the dots in the first quadrant represent the calculation process of a sample batch on a certain layer of the model, and the arrows represent the data dependence or time dependence of the model training calculation process; the dotted box indicates that the model is distributed in a distributed manner. An example of segmentation during training; Figure 6 shows an example of calculation graph segmentation in data parallel mode, which is directly segmented from the batch direction; Figure 7 shows an example of calculation graph segmentation in pipeline parallel mode, in a dislocation Divide in the horizontal direction.

In the present invention, the training calculation process of the deep learning model is uniformly described, and the calculation process is described in the polyhedron model. Through the polyhedron model, the commonly used distributed parallel training modes such as data parallelism and pipeline parallelism can pass the obtained polyhedron model. The optimization model is unified, so a feasible parallel strategy can be searched through a simple mapping transformation of the polyhedral model.

Please continue to refer to FIG. 1, S500, call the underlying framework to execute the parallel strategy. The execution API is an operation manager (runtime interface) in the underlying AI framework.

Specifically, in the polyhedron model, the data parallel segmentation dimension or pipeline parallel segmentation dimension of distributed training is found, and the parallel strategy is automatically output in combination with the number of computing resources input by the user. Finally, it is necessary to call the underlying framework to execute the API to execute all the parallel strategy. The API is an interactive interface.

Further, step S500 specifically includes:

S510. Call the execution API of the underlying framework to execute the parallel strategy. The execution API is an operation manager in the underlying AI framework.

Specifically, after creating a polyhedral model instance under the polyhedral model, find the data parallel segmentation dimension of the distributed training or the pipeline parallel segmentation dimension through the instance of the polyhedral model, and automatically output the distribution of the algorithm model in combination with the number of computing resources input by the user Then you need to call the underlying framework execution API to implement the distributed training strategy of the algorithm model.

Further, referring to FIG. 8 , based on the above-mentioned automatic parallel strategy search method based on polyhedral model modeling, the present invention also provides an automatic parallel strategy search system accordingly. The automatic parallel strategy search system also includes: a calculation graph A generation module 100 , a computational graph conversion module 200 , a computational graph equalization module 300 , a parallel strategy search module 400 and a parallel strategy execution module 500 .

Specifically, the calculation graph generation module 100 is used to obtain the model calculation graph of the deep learning algorithm according to the model object input by the user; the calculation graph conversion module 200 is used to convert the model calculation graph to obtain the converted model calculation graph. The model calculation graph of model instance, and output a parallel strategy according to the polyhedron model instance; the parallel strategy execution module 500 is configured to invoke the underlying framework to execute the parallel strategy.

The realization of the automatic parallel strategy search method based on polyhedral model modeling proposed by the present invention can be regarded as a middleware supporting the AI framework to realize the automatic parallel function. Figure 9 shows the relationship between its function and the Pytorch framework. First of all, the user uses the Pytorch front-end API to define the algorithm. In fact, the user does not need to consider the parallel segmentation of the algorithm, but only needs to consider the single-machine implementation logic of the algorithm; therefore, the user only needs to pay attention to the API usage method provided by the Pytorch framework for the algorithm, through the API To implement a stand-alone algorithm, as shown in Figure 9, the vertical column on the left is the basic structure of the Pytorch framework, and the right part is the automatic parallel strategy search system. The main interface between the automatic parallel strategy search system and the Pytorch framework is the calculation graph representation of the algorithm and the operation of the underlying framework. The user is almost unaware of the automatic parallel strategy search system.

Then the user-defined algorithm will output the computation graph through Pytorch's JIT (just-in-time compilation) module, and the computation graph will be input to the automatic parallel strategy search system and then successively input to the computation graph generation module, computational graph conversion module, computational graph equalization module and parallel strategy search module , and finally output the calculation subgraph that is divided according to the parallel strategy, and execute the calculation subgraph by calling the underlying Runtime API of Pytorch (the internal functional API of the Pytorch framework).

Furthermore, the present invention also provides a controller, the controller includes a processor 10 , a memory 20 and a display 30 . FIG. 10 shows only some of the components of the controller, but it should be understood that implementation of all of the illustrated components is not required, and that more or fewer components may be implemented instead.

The memory 20 may in some embodiments be an internal storage unit of the controller, such as a hard disk or memory of the controller. In other embodiments, the memory 20 may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) equipped on the controller. , SD) card, flash memory card (Flash Card) and so on. Further, the memory 20 may also include both an internal storage unit of the controller and an external storage device. The memory 20 is used to store application software and various types of data installed on the controller. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, an automatic parallel strategy search program 40 is stored in the memory 20, and the automatic parallel strategy search program 40 can be executed by the processor 10, thereby realizing the automatic parallel strategy search method based on polyhedral model modeling in the present invention.

In some embodiments, the processor 10 may be a central processing unit (Central Processing Unit, CPU), a microprocessor or other data processing chips, which are used to execute program codes or process data stored in the memory 20, such as Execute the automatic parallel strategy search method based on polyhedral model modeling and the like.

In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display 30 is used for displaying information on the device and for displaying a visual user interface. The components 10-30 of the apparatus communicate with each other via a system bus.

In one embodiment, when the processor 10 executes the automatic parallel strategy search program 40 in the memory 20, the following steps are implemented:

Obtain the model calculation graph of the deep learning algorithm according to the model object input by the user;

Converting the model calculation graph to obtain the converted model calculation graph;

Perform equalization processing on the converted model calculation graph to obtain a balanced calculation graph;

Create a polyhedron model instance according to the equilibrium calculation graph, and output a parallel strategy according to the polyhedron model instance;

The underlying framework is invoked to execute the parallel strategy.

Wherein, the step of obtaining the model calculation graph of the deep learning algorithm according to the model object input by the user specifically includes:

After randomly inputting a numerical value into the algorithm model, record the calculation process of the algorithm model, and obtain the model calculation graph; wherein, the model object refers to the stand-alone training code of the deep learning algorithm predefined by the user.

Or generate a syntax tree after parsing the model object by a python interpreter, and then analyze the syntax tree to obtain the model computation graph.

Wherein, the step of converting the model computation graph to obtain the converted model computation graph specifically includes:

The model computation graph is re-expressed with a pre-defined intermediate representation to obtain a converted model computation graph.

Wherein, the step of performing equalization processing on the converted model calculation graph to obtain a balanced calculation graph specifically includes:

A balanced computation graph is obtained by performing splitting operations on large nodes in the converted model computation graph and performing aggregation operations on adjacent small nodes in the converted model computation graph.

Wherein, the step of creating a polyhedron model instance according to the equilibrium calculation graph, and outputting a parallel strategy according to the polyhedron model instance specifically includes:

Create a polyhedron optimization model; initialize the polyhedron optimization model according to the equilibrium calculation graph to obtain a polyhedron model instance;

A parallel strategy is output according to the polyhedral model instance and the number of computing resources input by the user.

Wherein, the step of invoking the underlying framework to execute the parallel strategy specifically includes:

The execution API of the underlying framework is called to execute the parallel strategy.

Further, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an automatic parallel strategy search program 40, and the automatic parallel strategy search program 40 is executed by the processor to achieve the above-mentioned The steps of the automatic parallel strategy search method based on polyhedron model modeling; since the steps of the automatic parallel strategy search method based on polyhedron model modeling have been described in detail, they will not be repeated here.

To sum up, the present invention provides an automatic parallel strategy search method based on polyhedral model modeling and related equipment. The automatic parallel strategy search method based on polyhedral model modeling includes: obtaining a deep learning method according to a model object input by a user The model calculation graph of the algorithm; convert the model calculation graph to obtain a converted model calculation graph; perform equalization processing on the converted model calculation graph to obtain a balanced calculation graph; according to the balanced calculation graph, create a polyhedron model instance, and Output the parallel strategy according to the polyhedron model instance; call the underlying framework to execute the parallel strategy. In the present invention, the model calculation graph is converted and equalized, and after the polyhedron model instance is created under the frame of the polyhedron model, the parallel strategy is automatically output, so that different algorithm logics are modeled under the polyhedron model and automatically output. The parallel strategy process improves the efficiency of parallel strategy search and reduces the difficulty of distributed training development and efficiency tuning of deep learning algorithms.

It can be understood that for those of ordinary skill in the art, equivalent replacements or changes can be made according to the technical solutions of the present invention and the inventive concept thereof, and all these changes or replacements should belong to the protection scope of the appended claims of the present invention.

Claims (13)

1. an automatic parallel strategy search method based on polyhedron model modeling, is characterized in that, the described automatic parallel strategy search method based on polyhedron model modeling comprises: Obtain the model calculation graph of the deep learning algorithm according to the model object input by the user; Converting the model calculation graph to obtain the converted model calculation graph; Perform equalization processing on the converted model calculation graph to obtain a balanced calculation graph; Create a polyhedron model instance according to the equilibrium calculation graph, and output a parallel strategy according to the polyhedron model instance; The underlying framework is invoked to execute the parallel strategy. 2. the automatic parallel strategy search method based on polyhedral model modeling according to claim 1, is characterized in that, the described step that obtains the model computation graph of deep learning algorithm according to the model object input by user specifically comprises: Obtain an algorithm model according to the model object input by the user; After randomly inputting a numerical value to the algorithm model, record the calculation process of the algorithm model to obtain the model calculation diagram; Or generate a syntax tree after parsing the model object by a python interpreter, and then analyze the syntax tree to obtain the model computation graph. 3. the automatic parallel strategy search method based on polyhedral model modeling according to claim 2, it is characterised in that the described model calculation graph is converted, and the step of obtaining the converted model calculation graph specifically comprises: The model computation graph is re-expressed with a pre-defined intermediate representation to obtain a converted model computation graph. 4. the automatic parallel strategy search method based on polyhedral model modeling according to claim 3, is characterized in that, described with the model calculation graph after conversion is carried out equalization processing, and the step that obtains equalization calculation graph specifically comprises: Set the average computation amount threshold of the node, and compare the computation amount node in the converted model computation graph with the average computation amount threshold; The adjacent calculation amount nodes smaller than the average calculation threshold are fused, and the calculation amount nodes larger than the average calculation threshold are split to obtain a balanced calculation graph. 5. The automatic parallel strategy search method based on polyhedral model modeling according to claim 4, wherein, according to the equilibrium calculation graph, a polyhedral model instance is created, and a parallel strategy is output according to the polyhedral model instance. The steps include: Create a polyhedron optimization model; Initialize the polyhedron optimization model according to the equilibrium calculation graph to obtain a polyhedron model instance; A parallel strategy is output according to the polyhedral model instance and the number of computing resources input by the user. 6. The automatic parallel strategy search method based on polyhedral model modeling according to claim 3, wherein the step of invoking the underlying framework to execute the parallel strategy specifically comprises: The execution API of the underlying framework is called to execute the parallel strategy. 7 . The automatic parallel strategy search method based on polyhedral model modeling according to claim 1 , wherein the model object refers to a single-machine training code of a deep learning algorithm predefined by a user. 8 . 8 . The automatic parallel strategy search method based on polyhedral model modeling according to claim 3 , wherein the predefined intermediate representations include: IRType, IRValue, IRNode and IRGraph. 9 . 9 . The automatic parallel strategy search method based on polyhedral model modeling according to claim 5 , wherein the parallel strategy includes a data parallel segmentation dimension and a pipeline parallel segmentation dimension. 10 . 10 . The automatic parallel strategy search method based on polyhedral model modeling according to claim 6 , wherein the execution API is an operation manager in an underlying AI framework. 11 . 11. An automatic parallel strategy search system, characterized in that the automatic parallel strategy search system comprises: The calculation graph generation module is used to obtain the model calculation graph according to the deep learning algorithm logic defined by the user; a computational graph conversion module, configured to convert the model computational graph to obtain the converted model computational graph; The calculation graph equalization module is used to equalize the converted model calculation graph to obtain a balanced calculation graph; a parallel strategy search module, configured to create a polyhedron model instance according to the equilibrium calculation graph, and output a parallel strategy according to the polyhedron model instance; The parallel strategy execution module is used for invoking the underlying framework to execute the parallel strategy. 12. A controller, characterized in that the controller comprises: a memory, a processor, and an automatic parallel strategy search program based on polyhedral model modeling that is stored on the memory and can run on the processor, The automatic parallel strategy search program based on polyhedral model modeling is executed by the processor to implement the steps of the automatic parallel strategy search method based on polyhedral model modeling according to any one of claims 1-9. 13. A computer-readable storage medium, wherein the computer-readable storage medium stores an automatic parallel strategy search program based on polyhedral model modeling, and the automatic parallel strategy search program based on polyhedral model modeling is processed The steps of implementing the automatic parallel strategy search method based on polyhedral model modeling according to any one of claims 1-9 when the computer is executed.

Images