KR20240028915A - Dropout method and system for efficiently training deep learning models - Google Patents

Abstract

A dropout method and system for efficiently learning a deep learning model are disclosed. According to one embodiment, a dropout method performed by a dropout system includes limiting the number of subnetworks participating in learning during the entire learning period of a deep learning model; training each of the subnetworks selected from the deep learning model by a preset number of learning repetitions based on the limited number of subnetworks; and generating a dropout mask for each matrix multiplication operation of the selected subnetwork.

Description

Dropout method and system for efficiently learning deep learning models {DROPOUT METHOD AND SYSTEM FOR EFFICIENTLY TRAINING DEEP LEARNING MODELS}

The explanation below is about dropout technology to solve overfitting problems in deep learning models.

Dropout is a technique that sets the value of some neurons to 0 with a certain probability for the resulting neurons of each layer of a deep learning model and excludes them from only one training iteration. At this time, in order to set the values of some neurons to 0, a dropout mask consisting of 1 or 0 is created and applied to the resulting neurons of each layer. Dropout avoids co-adaptation between adjacent neurons within a deep learning model and thereby eliminates overfitting, a phenomenon in which high accuracy is not obtained for evaluation data due to excessive learning on the training data. It's a technique.

Because dropout randomly sets the neuron value to 0 with a certain probability for neurons in each layer of the deep learning model at each learning repetition, different subnetworks of the deep learning model are learned at each learning repetition. As a result, subnetworks as different as the total number of learning iterations that make up the learning step are trained for only one learning iteration. Additionally, because the values of neurons are randomly set to 0, there is a problem that it is difficult to effectively exclude neurons that have changed to 0 from calculations on the GPU where deep learning model calculations are performed.

Gyro Dropout, a dropout-based learning technique, can be provided to efficiently learn deep learning models.

Block-level dropout, a GPU-friendly learning technique that is a modification of gyro dropout, can be provided.

The dropout method performed by the dropout system includes limiting the number of subnetworks participating in learning during the entire learning period of the deep learning model; training each of the subnetworks selected from the deep learning model by a preset number of learning repetitions based on the limited number of subnetworks; and generating a dropout mask for each matrix multiplication operation of the selected subnetwork.

The limiting step includes a hyperparameter representing the number of all subnetworks participating in learning during the entire learning period of the deep learning model, and a plurality of hyperparameters representing the number of subnetworks to be simultaneously learned in one learning iteration. It may include the step of setting hyperparameters.

The limiting step may include limiting the number of subnetworks that will participate in each learning iteration of the deep learning model by adjusting the values of the plurality of set hyperparameters.

The learning step includes selecting some sub-networks to participate in learning of the deep learning model based on the limited number of sub-networks, training the selected some sub-networks by a preset learning repetition, and training the deep learning model. It may include selecting another sub-network to participate in learning, and repeating the operation of training the selected other sub-network by a preset number of learning repetitions until the entire learning period of the deep learning model ends.

The learning step may include generating a dropout mask to learn a plurality of consecutive data from each of the selected subnetworks based on the number of data to be simultaneously learned in one learning repetition.

The generating step may include generating the value of consecutive neurons in a specific shape as 0 through each of the selected subnetworks to exclude the neurons from the GPU matrix multiplication operation.

It may include a computer program stored in a non-transitory computer-readable recording medium to execute the dropout method on the dropout system.

The dropout system includes a number limiter that limits the number of subnetworks participating in learning during the entire learning period of the deep learning model; an iterative learning unit that trains each of the subnetworks selected from the deep learning model by a preset number of learning repetitions based on the limited number of subnetworks; and an optimization unit that generates a dropout mask for each matrix multiplication operation of the selected subnetwork.

By pre-selecting a fixed number of subnetworks, implementation is simple and higher accuracy can be achieved during training iterations.

Lost computations can be efficiently pruned on GPUs, improving training throughput without loss of accuracy.

Figure 1 is a diagram for explaining the output of a layer using a dropout technique, according to one embodiment.
Figure 2 is a block diagram for explaining a dropout system, according to one embodiment.
Figure 3 is a flowchart for explaining a dropout method, according to one embodiment.
Figure 4 is a diagram for explaining a learning operation using dropout, according to one embodiment.
Figure 5 is a diagram for explaining a block-by-block dropout operation, according to one embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

Unlike existing dropout, which trains a very large number of subnetworks (models composed of some neurons of a deep learning model) over a short learning period, in the embodiment, a subnetwork selected from the deep learning model based on a limited number of subnetworks We want to improve the ensemble performance of the subnetwork learned in the evaluation stage by learning as many training repetitions as possible. Accordingly, in the embodiment, a dropout operation that improves the efficiency of neural network training will be described.

Figure 1 is a diagram for explaining the output of a layer using a dropout technique, according to one embodiment.

Figure 1 is a diagram showing each dropout technique applied to the output of a layer with 8 neurons and 6 batch sizes. Figure 1(a) is an existing dropout, Figure 1(b) is a first dropout (Gyro Dropout) proposed in the embodiment, and Figure 1(c) is a second dropout proposed in the embodiment. This is an example of out (block-wise dropout).

Among the techniques used to increase the accuracy of the learned model when learning a deep learning model, there is dropout, which is a technique to reduce overfitting, which refers to the difference in inference accuracy between learning and actual evaluation. Existing dropout generates some of the resulting neurons of each layer that constitutes the deep learning model as 0 with a specific probability, as shown in Figure 1(a), and excludes them from learning before inputting them as input to the next layer. Since the existing dropout in Figure 1(a) selects and learns a new subnetwork of the deep learning model at each learning iteration, a significant number of subnetworks are used for learning, and each subnetwork is learned in only one learning iteration. This happens.

If we look at dropout from the ensemble learning perspective, which is one of the artificial intelligence learning theories, each training data trains a different sub-network of the deep learning model (i.e., a model made up of some neurons of the entire deep learning model), and in the evaluation stage, Inference is made by integrating all learned sub-networks. From the perspective of ensemble learning, the greater the diversity of base models participating in learning, the higher the accuracy of the ensemble model, and there is a number of base models for which the accuracy of the ensemble model is highest. Accordingly, the existing dropout technique, which trains a very large number of subnetworks and has a short learning period for each subnetwork, has room for improvement from an ensemble learning perspective.

Figure 1(b) is a dropout technique that trains neurons during learning iterations using a fixed number of subnetworks, instead of randomly removing neurons in every learning iteration. Because each subnetwork is trained stably, it becomes more diverse and the ensemble achieves good generalization. Table 1 shows the algorithm of the gyro dropout algorithm proposed in the embodiment. Before learning, the dropout system first selects subnetworks to participate in learning, trains each of them for a certain number of learning iterations, selects the next subnetwork, and trains it for a certain number of learning iterations. This is repeated until the entire learning step is completed. You can.

Table 1:

Figure 1(c) illustrates the block-wise dropout technique, a GPU-friendly variant of Figure 1(b), where block-wise dropout selects a block of neurons instead of dropping out individual neurons and sets all outputs to 0. . Block-level dropout groups hidden neurons into multiple groups that must be removed together throughout training and drops them all out, allowing for efficient pruning of warp execution on GPUs.

In this way, it can be seen that the existing dropout in Figure 1(a) trains six different subnetworks for six pieces of data in a mini-batch. In other words, each row has a different dropout pattern than every other row. Figure 1(b) shows learning with only two subnetworks because the dropout patterns of the top 3 rows and the bottom 3 rows are the same. In this case, the number of subnetworks to be learned simultaneously is 2.

Through experiments, one or more learning data is learned using the same subnetwork (i.e., when learning is done by dropping out neurons at the same location for each consecutive row as shown in Figure 1(b)), participating in learning during the entire learning period. When the number of subnetworks of the deep learning model is limited, it can be confirmed that the accuracy of the learned model is higher than that of the existing dropout technique. Using these results, the embodiment uses Gyro Dropout, a new learning technique that adjusts the number of subnetworks created in the learning process and extends the learning period of each subnetwork, and Block Unit Drop, a GPU-friendly variant of Gyro Dropout. Out can be provided.

FIG. 2 is a block diagram for explaining a dropout system in one embodiment, and FIG. 3 is a flowchart for explaining a dropout method in one embodiment.

The processor of the dropout system 100 may include a number limiting unit 210, an iterative learning unit 220, and an optimization unit 230. These processor components may be expressions of different functions performed by the processor according to control instructions provided by program code stored in the dropout system. The processor and its components may control the dropout system to perform steps 310 to 330 included in the dropout method of FIG. 3. At this time, the processor and its components may be implemented to execute instructions according to the code of an operating system included in the memory and the code of at least one program.

The processor may load the program code stored in the file of the program for the dropout method into memory. For example, when a program is executed in a dropout system, the processor can control the dropout system to load program code from the program's file into memory under the control of the operating system. At this time, the number limit unit 210, the iterative learning unit 220, and the optimization unit 230 each execute commands of corresponding portions of the program code loaded into the memory to execute subsequent steps 310 to 330. These may be different functional representations of the processor.

In step 310, the number limiter 210 may limit the number of sub-networks participating in learning during the entire learning period of the deep learning model. The number limit unit 210 includes a hyperparameter representing the number of all sub-networks participating in learning during the entire learning period of the deep learning model, and a hyperparameter representing the number of sub-networks to be learned simultaneously in one learning iteration. Hyperparameters can be set. The number limiter 210 may limit the number of sub-networks that will participate in each learning iteration of the deep learning model by adjusting the values of a plurality of set hyperparameters.

In step 320, the iterative learning unit 220 may train each of the subnetworks selected in the deep learning model by a preset number of learning repetitions based on the limited number of subnetworks. The iterative learning unit 220 selects some sub-networks to participate in learning of the deep learning model based on the limited number of sub-networks, trains some of the selected sub-networks by a preset learning repetition, and performs the learning of the deep learning model. The operation of selecting another sub-network to participate and training the other selected sub-network by a preset number of learning repetitions can be repeated until the entire learning period of the deep learning model ends. The iterative learning unit 220 may generate a dropout mask to learn a plurality of consecutive data from each of the selected subnetworks based on the number of data to be learned simultaneously in one learning repetition.

In step 330, the optimizer 230 may generate a dropout mask for each matrix multiplication operation of the selected subnetwork. The optimizer 230 may exclude the neurons from the GPU matrix multiplication operation by generating values of consecutive neurons in a specific shape as 0 through each of the selected subnetworks.

Figure 4 is a diagram for explaining a learning operation using dropout, according to one embodiment.

This is an example showing the output matrix of the first layer. The height of the matrix represents the batch size, and the width of the matrix is the number of neurons. Figures 4(a) and 4(b) will be explained using the example of simultaneously learning four sub-networks at a time. In Figure 4(a) the four subnetworks are within a single deployment, and in Figure 4(b) the four subnetworks span two deployments. The figure on the right shows the forward computation of a three-layer deep learning model with block-wise dropout.

The dropout system is a pre-selected number of subnetworks ( ) and the number of subnetworks to be learned simultaneously in the learning iteration ( ) You can use two hyperparameters to limit the number of subnetworks that will participate in each learning iteration. Figure 4(a) shows =4, This diagram shows the operation of learning a deep learning model for 40 learning iterations through gyro dropout with =16. The matrices shown in Figure 4 all represent the resulting neurons of one layer of the deep learning model, and the 8 rows of the matrix mean that the number of data that the deep learning model simultaneously learns during one learning iteration is 8. Since the number of all subnetworks participating in learning is 16, and the number of subnetworks participating in learning during one learning iteration is 4, a new subnetwork to be learned 4 times can be selected. Additionally, since the number of data that can be learned simultaneously in one learning iteration is 8, a drop mask can be created to learn two consecutive data for each subnetwork.

The dropout system can create a dropout mask so that neurons whose value becomes 0 during the GPU matrix multiplication operation can be excluded from the operation. Block-level dropout is a gyro dropout in which the values of a certain number of consecutive neurons in all subnetworks participating in learning are 0, as shown in Figure 1(c). When droopout excludes the resulting neurons from learning with a certain probability, they are scattered in a disorderly manner like a conventional dropout, and the values of the neurons become 0. In this case, when performing GPU matrix multiplication calculations, the neurons excluded from learning cannot be excluded from the calculation and the calculation is performed. This does not shorten the computation time. However, when block-level dropout is applied, the values of neurons in the horizontal direction are generated as 0 in succession, and because the same subnetwork is used in consecutive rows, the values of square-shaped neurons are generated as 0 as a result, which causes GPU It is efficient in reducing time as it can be effectively excluded from matrix multiplication operations.

Figure 5 is a diagram for explaining a block-by-block dropout operation, according to one embodiment.

To explain block-level dropout, we will explain the GPU execution model and the operation in which matrix multiplication is calculated on the GPU. A GPU has multiple streaming multiprocessors (SMs), each executing multiple hardware threads simultaneously. GPUs adopt a single-instruction-multi-threading (SIMT) model, where red groups execute the same sequence of instructions simultaneously. The execution model is supported by a programming abstract that provides the concept of thread blocks. A thread block is a group of threads assigned to the same streaming multiprocessor and executed in parallel. A thread block consists of hundreds to thousands of threads and is organized in 1D, 2D, or 3D structures to represent multidimensional data in the application domain. The GPU internally splits each block of threads into scheduled units called warps, which are made up of 32 threads executing the same sequence of instructions.

SIMT is inefficient for arbitrarily sparse matrix operations because all threads in a warp must execute the same instruction sequence. For example, let's consider the multiplication of arbitrary sparse matrices on a GPU. When a warp executes multiplication and addition calculations, it cannot skip the multiplication with 0 and spare GPU cycles because all threads in the warp execute the same instructions. In the SIMT execution model, skipping multiplication can only speed up execution if all threads in a warp exclusively multiply by zero.

To hide DRAM latency, General Matrix Multiplication (GEMM) kernels typically apply tiling and pipelining. Instead of computing each element of the output matrix, tiling optimization simultaneously computes the multiplication of groups of elements, called tiles. Pipelining optimization loads tiles from DRAM and overlaps tile calculations.

CUTLASS, a state-of-the-art open source GEMM kernel, also applies two optimizations. In CUTLASS, the tiles of the output matrix are computed by a single thread block, so the tiles are called thread block tiles. Referring to Figure 5, it shows the thread block tile of CUTLASS. The default size of a thread block tile is 128×128. The shape of thread block tiles and other tiles can be determined to achieve unified memory access and maximize memory bandwidth efficiency. To compute the multiplication of a tile, the input matrix is loaded into a 128×8 (or 8×128) shaped input tile to the level 1 cache (i.e. shared memory) (hatched boxes on the left and right). Thread block tiles are processed by a single thread block consisting of 256 threads or 8 warps (marked W0-W7), each computing four warp tiles of the 32×16 shape shown on the right side of Figure 5. While input tiles are loaded, computations with previously loaded tiles are executed, so memory loads and computations are pipelined and overlapped.

The dropout system can determine the tiling and dropout block shape by considering two factors: the integrated memory access of the GEMM kernel and the subnetwork preselection effect.

The GEMM kernel typically leverages the GPU's memory subsystem architecture and pulls data from DRAM into shared memory with a cache line granularity of 128 bytes. Pruning to a smaller size does not fully utilize the memory subsystem and wastes memory bandwidth. Therefore, for efficient memory transactions, the dropout block size should be at least 32 elements (e.g., 128 bytes). At the same time, computations of entire rows or columns of thread block tiles must be pruned so that loading of the corresponding input data (from shared memory to registers) can be skipped. To achieve this, the first dimension of the dropout block must be 128 (elements). Therefore, we consider 128×32 and 32×128 dropout block shapes as candidates. The former assigns a larger value to the batch size (i.e. the height of the thread block tile) dimension, while the latter sets the dimension of the output neuron to be larger. Now, to maximize accuracy, we can consider that the number of subnetworks to be trained simultaneously is quite small. Additionally, using a larger value for the output neuron dimension and a dropout of 32×128 blocks significantly reduces the total number of possible subnetworks. Considering these factors, the candidate dropout block shape can be determined to be 128×32.

Using block-wise dropout, two types are supported: input-based pruning and output-based pruning. As the name suggests, the former prunes neurons that are dropped out in the input matrix, while the latter prunes neurons that are missing in the output matrix. Between the two pruning types, input-based pruning is slightly more efficient than output-based pruning because it loads input data more efficiently. Accordingly, input pruning can be prioritized over output pruning. In the right diagram of Figure 4, the forward computation of a three-layer deep learning model with block-wise dropout is shown. Output pruning is applied to the first layer and input pruning is applied to the last two layers.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (13)

In the dropout method performed by the dropout system,
Limiting the number of subnetworks participating in learning during the entire learning period of the deep learning model;
training each of the subnetworks selected from the deep learning model by a preset number of learning repetitions based on the limited number of subnetworks; and
Generating a dropout mask for each matrix multiplication operation of the selected subnetwork.
A dropout method containing . According to paragraph 1,
The limiting step is,
Setting a plurality of hyperparameters, including a hyperparameter indicating the number of all subnetworks participating in learning during the entire learning period of the deep learning model, and a hyperparameter indicating the number of subnetworks to be learned simultaneously in one learning iteration. step
A dropout method containing . According to paragraph 2,
The limiting step is,
Limiting the number of subnetworks that will participate in each learning iteration of the deep learning model by adjusting the values of the plurality of set hyperparameters
A dropout method containing . According to paragraph 1,
The learning step is,
Based on the limited number of sub-networks, some sub-networks that will participate in learning the deep learning model are selected, some of the selected sub-networks are trained by a preset learning repetition, and other sub-networks that participate in learning the deep learning model are selected. Selecting a network and repeating the operation of training other selected sub-networks by a preset number of learning repetitions until the entire learning period of the deep learning model ends.
A dropout method containing . According to paragraph 1,
The learning step is,
Generating a dropout mask to learn a plurality of consecutive data from each of the selected subnetworks based on the number of data to be learned simultaneously in one learning iteration.
A dropout method containing . According to paragraph 1,
The generating step is,
Excluding the neurons from the GPU matrix multiplication operation by generating the value of consecutive neurons in a specific shape as 0 through each of the selected subnetworks.
A dropout method containing . A computer program stored in a non-transitory computer-readable recording medium for executing the dropout method of any one of claims 1 to 6 on the dropout system. In the dropout system,
A number limiter that limits the number of subnetworks participating in learning during the entire learning period of the deep learning model;
an iterative learning unit that trains each of the subnetworks selected from the deep learning model by a preset number of learning repetitions based on the limited number of subnetworks; and
An optimization unit that generates a dropout mask for each matrix multiplication operation of the selected subnetwork.
Dropout system including. According to clause 8,
The number limit part is,
Setting a plurality of hyperparameters, including a hyperparameter indicating the number of all subnetworks participating in learning during the entire learning period of the deep learning model, and a hyperparameter indicating the number of subnetworks to be learned simultaneously in one learning iteration.
A dropout system characterized in that. According to clause 9,
The number limit part is,
Limiting the number of subnetworks that will participate in each learning iteration of the deep learning model by adjusting the values of the plurality of set hyperparameters
A dropout system characterized in that. According to clause 8,
The iterative learning unit,
Based on the limited number of sub-networks, some sub-networks that will participate in learning the deep learning model are selected, some of the selected sub-networks are trained by a preset learning repetition, and other sub-networks that participate in learning the deep learning model are selected. The operation of selecting a network and training other selected sub-networks by a preset number of learning repetitions is repeated until the entire learning period of the deep learning model ends.
A dropout system characterized in that. According to clause 8,
The iterative learning unit,
Generating a dropout mask to learn a plurality of consecutive data from each of the selected subnetworks based on the number of data to be learned simultaneously in one learning iteration.
A dropout system characterized in that. According to clause 8,
The optimization unit,
Excluding neurons from the GPU matrix multiplication operation by generating the value of consecutive neurons in a specific shape as 0 through each of the selected subnetworks.
A dropout system characterized in that.

Images