KR102531763B1 - Method and Apparatus for Recovery of no-Fine-Tuning Training and Data for Lightweight Neural Networks using Network Pruning Techniques - Google Patents

Abstract

A performance recovery method and device that does not require data and fine-tuned learning for a lightweight neural network through network pruning techniques are presented. The performance recovery method that does not require data and fine-tuning learning for a lightweight neural network through the network pruning technique proposed in the present invention uses the linear relationship between the filter removed from the layer of the pretrained convolutional neural network and the filter preserved. A filter of a pretrained convolutional neural network by representing a damaged filter, defining a loss function due to the damaged filter according to a batch normalization layer and an activation layer, and representing coefficient values that minimize the loss function as element values of a transfer matrix. and restoring the performance of the neural network compressed by pruning by performing an operation of H and a pruning matrix.

Description

Method and Apparatus for Recovery of no-Fine-Tuning Training and Data for Lightweight Neural Networks using Network Pruning Techniques}

The present invention relates to a performance recovery method and apparatus that do not require data and fine-tuning learning for a lightweight neural network through a network pruning technique.

Network pruning is one of the areas of active research to compress overfitted neural networks. Network pruning aims to reduce the number of parameters and reduce the inference speed of a neural network by removing the parameters of the neural network while preserving the performance of the pretrained convolutional neural network as much as possible. Network pruning is mainly divided into two types: unstructured pruning and filter pruning. Weight pruning is a technique that makes a neural network sparse by replacing the parameters themselves with '0' without changing the structure of the neural network. This technique is effective in reducing the number of parameters of the neural network, but does not change the structure of the neural network. Therefore, special hardware and software are required to reduce inference speed. On the other hand, since filter pruning changes the structure of the neural network by removing the filter of the neural network, it is possible to reduce the inference speed without additional hardware and software.

In general, filter pruning includes iterative pruning that repeatedly removes the filter of the neural network in the process of training an untrained neural network and pruning a pretrained convolutional neural network to remove parameters and then performing retraining. There is a one-shot pruning technique that restores performance through However, iterative pruning and one-shot pruning require sufficient data and hardware for learning, and have problems in that the learning process takes a lot of time. In order to solve this problem, the following prior art has been proposed.

The prior art [1] proposed a method of greedily selecting a filter to be removed for each layer of a neural network based on a reconstruction error using original data in order to recover compressed performance with a small learning process. However, the prior art [1] still has a problem in that all original data for re-learning and data for generating reconstruction errors are required. The prior art [2] proposed a method of globally selecting a filter to be removed using a small amount of data in a neural network using the Kullback-Leibler divergence criterion, and with little learning. In order to learn only compressed neural networks, a relearning method using knowledge distillation was proposed. However, it still did not solve the problem of needing data and relearning. In order to recover the performance after pruning the pretrained convolutional neural network without data and retraining, the one-to-one method according to the prior art [3,4] uses the most similar preserved one for each filter removed. Recover the performance loss caused by the filter being removed by finding the filter in , multiplying the filter being removed by the scale, and adding it to the filter being preserved.

The technical problem to be achieved by the present invention is to solve the problem of selecting a filter that is not similar in the second layer of the neural network in the method of restoring a compressed neural network based on the similarity of the prior art. An object of the present invention is to provide a method and apparatus for restoring the performance of a compressed neural network based on a linear combination between a filter to be removed and a filter to be preserved rather than a selection method.

In one aspect, the performance recovery method that does not require data and fine-tuned learning for a neural network that is lightweight through the network pruning technique proposed in the present invention provides a relationship between a filter removed from a layer of a pretrained convolutional neural network and a filter that is preserved. Representing the damaged filter using a linear relationship, defining a loss function due to the damaged filter according to the batch normalization layer and the activation layer, and representing the coefficient values that minimize the loss function as element values of the transfer matrix to pre-learn. and restoring performance of the neural network compressed by pruning by performing an operation of a filter of the convolutional neural network and a pruning matrix.

In the step of representing a damaged filter using a linear relationship between a filter to be removed from a layer of the pretrained convolutional neural network and a filter to be preserved, the pruning matrix used in filter pruning of the pretrained convolutional neural network is transformed into a transfer matrix. By doing so, the information of the removed filter is inherited by the preserved filter, and the removed filter is represented as a linear combination of the preserved filters.

In the step of defining a loss function due to the damaged filter according to the batch normalization layer and the activation layer, after approximating the activation map with a feature map to minimize the loss function, the batch normalization layer and the activation layer for the residual error. A loss function is defined by representing the activation map according to the feature map as a value passed through the batch normalization layer, and a coefficient value that minimizes the residual error, batch normalization error, and activation error of the loss function is found.

The step of restoring the performance of the neural network compressed by pruning by performing the operation of the filter of the pretrained convolutional neural network and the pruning matrix by representing the coefficient values that minimize the loss function as element values of the transfer matrix. Find a transfer matrix that minimizes the reconstruction error of the pretrained convolutional neural network and the compressed neural network by multiplying the minimized coefficient value by the filter to be removed and adding it to the preserved filter, and transfers the coefficient value that minimizes the loss function. Recover the performance of the neural network compressed by pruning without data and learning by performing a 2-mode product operation of the filter of the pretrained convolutional neural network and the pruning matrix represented by the element values of the matrix.

In another aspect, a performance recovery device that does not require data and fine-tuning learning for a lightweight neural network through the network pruning technique proposed in the present invention is a filter removed from a layer of a pretrained convolutional neural network and a preserved A pruning modeling unit that represents a damaged filter using a linear relationship between filters and defines a loss function due to the damaged filter according to the batch normalization layer and the activation layer, and coefficient values that minimize the loss function are represented as element values of the transfer matrix and a reconstruction modeling unit that restores performance of the neural network compressed by pruning by performing an operation of a filter and a pruning matrix of the pretrained convolutional neural network.

Method and apparatus for restoring a compressed neural network to the performance of the original neural network without data and learning using filter pruning according to embodiments of the present invention In the method of restoring a compressed neural network based on the similarity of the prior art by restoring the performance of a compressed neural network based on a linear combination between the filter and the preserved filters, a dissimilar filter is used in the latter layer of the neural network. Problems can be solved by choosing.

1 is a diagram showing layers of a deep neural network according to the prior art.
2 is a diagram showing the configuration of a performance recovery device that does not require data and fine-tuning learning for a neural network that has been reduced through a network pruning technique according to an embodiment of the present invention.
3 is a flowchart illustrating a performance recovery method that does not require data and fine-tuning learning for a neural network that has been reduced through a network pruning technique according to an embodiment of the present invention.
4 is a diagram for explaining a pruning matrix and a transfer matrix according to an embodiment of the present invention.
5 is a diagram illustrating an algorithm for restoring performance of a neural network compressed by pruning without data and learning according to an embodiment of the present invention.
6 is a diagram showing a method for restoring the performance of a convolutional neural network according to an embodiment of the present invention and residual error (RE), batch normalization error (BE), and weighted average reconstruction error (WARE) with NM.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing layers of a deep neural network according to the prior art.

In order to restore the performance of neural networks compressed by pruning without data and learning, the prior art [3, 4] used a method of restoring performance by using the similarity between the filter to be removed and the filter to be preserved. However, as shown in FIG. 1 [4], the recently developed deep neural network has many layers, and the similarity between filters gradually decreases as the layers become deeper. That is, the method of recovering a compressed neural network based on similarity has a problem in that a dissimilar filter can be selected in the latter layer of the neural network [3, 4]. To solve this problem, NM [4] proposed a method of selecting a similar filter using a distance-based measurement method considering the batch normalization and activation function of neural networks. However, this method still has the possibility of incorrectly selecting dissimilar filters based on similarity.

Therefore, in the present invention, the performance of the compressed neural network is restored based on a linear combination between the filter to be removed and the filter to be preserved, rather than a method of selecting one filter most similar to the filter to be removed. The present invention proposes a method and apparatus for restoring a compressed neural network to the performance of an original neural network without data and learning using filter pruning.

2 is a diagram showing the configuration of a performance recovery device that does not require data and fine-tuning learning for a neural network that has been reduced through a network pruning technique according to an embodiment of the present invention.

A performance recovery device that does not require data and fine-tuning learning for a neural network that is lightweight through the proposed network pruning technique includes a pruning modeling unit 210 and a reconstruction modeling unit 220.

번째 레이어에서 제거되는 필터로 인해

번째 레이어의 출력 채널이 제거되고 이로 인해, 그 다음 레이어인

번째 필터의 채널이 연쇄적으로 제거됨으로써 출력의 손상이 발생된다. 이러한 오류(error)는 각 레이어 마다 누적되어 최종적으로 압축된 신경망의 성능하락이 발생한다. 이러한 성능 하락을 복구하기 위해, 본 발명에서는 제거되는 필터와 보존되는 필터간의 선형관계를 이용하고 배치 정규화 레이어와 활성화 레이어를 고려한 손실 함수를 정의한다. 그리고 정의된 손실함수를 최소화하는

를 찾아, 제거되는 필터에 곱하여 보존되는 필터들에게 더함으로써 사전학습된 합성곱 신경망과 압축된 신경망의 재구성 오류를 최소화할 수 있도록 한다. 도 2에서 재구성 모델링부(다시 말해, 재구성된 모델(Restored Model))(220)에 관하여 더욱 상세히 설명하면, 본 발명은 먼저 합성곱 신경망에서 필터 프루닝을 N-mode product[5]을 이용하여 나타내고 원본 신경망과 프루닝된 신경망 사이의 재구성 오류를 최소화하는 문제로 정의하여 데이터와 학습없이 프루닝으로 압축된 신경망을 회복 방법을 제안한다. Referring to FIG. 2, a performance recovery device that does not require data and fine-tuning learning for a lightweight neural network through a network pruning technique according to an embodiment of the present invention will be briefly described. The proposed device is pruning without data and learning. It is possible to recover the performance degradation of the compressed neural network caused by this. First, the performance degradation due to pruning is, as shown in FIG.

Due to the filter being removed in the second layer

The output channel of the first layer is removed, so that the next layer

As the channels of the th filter are sequentially removed, the output is damaged. These errors are accumulated for each layer, resulting in a performance degradation of the final compressed neural network. To recover this performance degradation, the present invention defines a loss function using a linear relationship between a filter to be removed and a filter to be preserved and considering the batch normalization layer and the activation layer. And minimizing the defined loss function

Find , and multiply the filters to be removed and add them to the filters to be preserved, thereby minimizing the reconstruction error of the pretrained convolutional neural network and the compressed neural network. Referring to the reconstruction modeling unit (ie, the restored model) 220 in FIG. 2 in more detail, the present invention first performs filter pruning in a convolutional neural network using N-mode product [5] It is defined as the problem of minimizing the reconstruction error between the original neural network and the pruned neural network, and proposes a recovery method for compressed neural networks without data and learning.

The pruning modeling unit 210 according to an embodiment of the present invention represents a damaged filter using a linear relationship between a filter removed from a layer of a pretrained convolutional neural network and a filter preserved, and according to the batch normalization layer and the activation layer Define a loss function due to the damaged filter. The pruning modeling unit 210 transforms the pruning matrix used in the filter pruning of the pretrained convolution into a transfer matrix so that the information of the filter to be removed is inherited by the filter to be preserved, and the filter to be removed is the filter to be preserved. can be represented as a linear combination of

The pruning modeling unit 210 according to an embodiment of the present invention approximates the activation map with a feature map in order to minimize the loss function (221), and then determines the residual error according to the batch normalization layer and the activation layer. A loss function is defined by representing an activation map as a value of a feature map passed through a batch normalization layer, and coefficient values minimizing residual errors, batch normalization errors, and activation errors of the loss function can be found.

The reconstruction modeling unit 220 restores the performance of the neural network compressed by pruning by representing the coefficient value that minimizes the loss function as an element value of the transfer matrix and performing the operation of the filter of the pretrained convolutional neural network and the pruning matrix. .

The reconstruction modeling unit 220 according to an embodiment of the present invention multiplies the filter to be removed by the coefficient value that minimizes the loss function and adds it to the filter to be preserved to reduce the reconstruction error of the pretrained convolutional neural network and the compressed neural network. Find the transfer matrix that minimizes. Thereafter, the coefficient value that minimizes the loss function is expressed as an element value of the transfer matrix, and a 2-mode product operation is performed between the filter of the pretrained convolutional neural network and the pruning matrix, thereby processing without data and learning. It is possible to recover the performance of compressed neural networks by running.

3 is a flowchart illustrating a performance recovery method that does not require data and fine-tuning learning for a neural network that has been reduced through a network pruning technique according to an embodiment of the present invention.

A performance recovery method that does not require data and fine-tuning learning for a lightweight neural network through the proposed network pruning technique uses a linear relationship between a filter removed from a layer of a pretrained convolutional neural network and a filter that is preserved to repair a damaged filter. Step 310 of representing, step 320 of defining a loss function due to the corrupted filter according to the batch normalization layer and activation layer, and pretrained convolution by representing coefficient values that minimize the loss function as element values of a transfer matrix. A step 330 of restoring performance of the neural network compressed by pruning by performing an operation of a filter of the neural network and a pruning matrix.

In step 310, a damaged filter is represented by using a linear relationship between a filter to be removed from a layer of the pretrained convolutional neural network and a filter to be preserved.

In step 310 according to an embodiment of the present invention, the pruning modeling unit preserves filter information removed by transforming the pruning matrix used in filter pruning of the pretrained convolution into a transfer matrix. Inherit to the filter that becomes, and the filter that is removed can be represented as a linear combination of the filters that are preserved.

In step 320, a loss function due to the corrupted filter according to the batch normalization layer and the activation layer is defined.

In step 320 according to an embodiment of the present invention, the pruning modeling unit approximates the activation map with a feature map to minimize the loss function, and then activates according to the batch normalization layer and the activation layer for the residual error. A loss function is defined by representing a map as a value through which a feature map has passed through a batch normalization layer, and coefficient values minimizing residual errors, batch normalization errors, and activation errors of the loss function can be found.

In step 330, the performance of the neural network compressed by pruning is restored by representing the coefficient value that minimizes the loss function as an element value of the transfer matrix and performing the operation of the filter of the pretrained convolutional neural network and the pruning matrix.

In step 330 according to an embodiment of the present invention, the reconstruction modeling unit reconstructs the pretrained convolutional neural network and the compressed neural network by multiplying the filter to be removed by the coefficient value that minimizes the loss function and adding it to the filter to be preserved. Find the transfer matrix that minimizes the error. Thereafter, the coefficient value that minimizes the loss function is expressed as an element value of the transfer matrix, and a 2-mode product operation is performed between the filter of the pretrained convolutional neural network and the pruning matrix, thereby processing without data and learning. It is possible to recover the performance of compressed neural networks by running.

4 is a diagram for explaining a pruning matrix and a transfer matrix according to an embodiment of the present invention.

4(a) shows a pruning matrix according to an embodiment of the present invention, FIG. 4(b) shows a transfer matrix for LBYL, and FIG. 4(c) shows a transfer matrix for one-to-one.

번째 연산에서 얻은 결과는 다음과 같다: Prior to explaining filter pruning in a pretrained convolutional neural network according to an embodiment of the present invention, a convolutional neural network having L layers

The result obtained from the second operation is:

는 합성곱 연산을 의미하며,

은

번째 레이어의 출력이면서

번째 레이어의 입력 활성화맵이며,

은 합성곱 필터를 의미한다.

은 각각 특징맵, 활성화맵을 나타낸다. 이때 m은 필터의 수를 의미하며 n은 이전 레이어의 활성화맵의 채널 수이며, w×h는 활성화맵의 크기이며, k×k는 필터의 크기를 의미한다. 그리고

는 활성화 레이어이며,

은 배치 정규화 레이어이다. here,

denotes a convolution operation,

silver

As the output of the second layer

It is the input activation map of the second layer,

denotes a convolutional filter.

denotes a feature map and an activation map, respectively. Here, m means the number of filters, n is the number of channels of the activation map of the previous layer, w×h is the size of the activation map, and k×k is the size of the filter. and

is the activation layer,

is the batch normalization layer.

번째 레이어에서 필터 프루닝을 수행하면 사전학습된 합성곱 신경망의 구조가 변경되어, 합성곱 필터의 차원이 변경되기 때문에 손상된 필터

, 특징맵

, 활성화맵

으로 변경된다. 여기서 t는 m보다 작은 남아있는 필터의 수를 의미한다. 이를 사전학습된 합성곱 신경망의 필터

와 프루닝 행렬(pruning matrix)

의 1-모드곱(1-mode product)를 이용하여 도 4(a)와 같이 나타낼 수 있다.

Performing filter pruning in the second layer changes the structure of the pretrained convolutional neural network, which changes the dimensionality of the convolutional filter, resulting in a damaged filter.

, feature map

, the activation map

is changed to where t denotes the number of remaining filters smaller than m. This is the filter of the pretrained convolutional neural network.

and pruning matrix

It can be expressed as shown in FIG. 4 (a) using the 1-mode product of .

번째 필터는 제거되지 않으며, (m-t)개의 필터들은 완전히 제거된다. Here, each

The th filter is not removed, and (mt) filters are completely removed.

번째 레이어에서 제거되는 필터로 인해

번째 레이어의 출력 채널이 제거되고 이로 인해 그 다음 레이어인

번째 필터의 채널이 연쇄적으로 제거됨으로써 출력의 손상이 발생된다. 즉,

번째 필터

는

로 변경된다. 여기서

은

번째 레이어의 필터의 수이며,

는 필터의 크기이다. 필터의 차원이 변경되지만 필터의 수는 변경되지 않기 때문에

번째 레이어의 출력의 차원은 사전학습된 합성곱 신경망의 출력 차원과 동일하다. 하지만 제거된 필터의 채널로 인해 발생되는 오류(error)는 각 레이어마다 누적되어 최종적으로 압축된 신경망의 성능하락이 발생한다. 손상된 필터

는 사전학습된 필터

와 프루닝 행렬

의 2-모드곱(2-mode product)으로 나타낼 수 있다: neural network

Due to the filter being removed in the second layer

The output channels of the first layer are removed, resulting in

As the channels of the th filter are sequentially removed, the output is damaged. in other words,

second filter

Is

is changed to here

silver

is the number of filters in the second layer,

is the size of the filter. Because the dimension of filters changes but the number of filters does not.

The dimension of the output of the second layer is the same as that of the pretrained convolutional neural network. However, errors generated by the channels of the removed filters are accumulated in each layer, resulting in a performance degradation of the final compressed neural network. damaged filter

is the pretrained filter

and the pruning matrix

It can be expressed as a 2-mode product of

은 손상된 특징맵을 의미하며

는 사전학습된 합성곱 신경망의 특징맵이다. here,

denotes a damaged feature map

is the feature map of the pretrained convolutional neural network.

를 도 4(b)와 같이 전달 행렬

로 변형시킴으로써 제거되는 필터의 정보를 보존되는 필터에게 상속시키고자 한다. 즉,

번째 레이어의 필터가 제거되어 연쇄적으로

번째 레이어 필터의 채널이 제거되었을 때, 압축된 신경망의 출력값

이 사전학습된 합성곱 신경망의 출력값

에 근사할 수 있는 전달 행렬

을 찾아 손상된 성능을 복구하고자 한다. A pruning matrix as shown in FIG. 4(a) used in filter pruning according to an embodiment of the present invention.

As shown in Figure 4 (b), the transfer matrix

By transforming into , we want to inherit the information of the filter to be removed to the filter to be preserved. in other words,

The filter of the second layer is removed, cascading

The output value of the compressed neural network when the channel of the second layer filter is removed.

The output of this pretrained convolutional neural network

A transfer matrix that can approximate

to recover the damaged performance.

번째 특징맵 사이의 재구성 오류를 최소화하는 전달 행렬을 찾는 것을 목적으로 하고, 목적식은 다음과 같이 나타낼 수 있다: Therefore, in the present invention, the pretrained convolutional neural network and the compressed neural network as follows

The purpose is to find a transfer matrix that minimizes the reconstruction error between the feature maps, and the objective expression can be expressed as:

을 곱하여 보존되는 필터들 각각에게 더한다.

번째 필터 하나가 제거되었을 때, 전달 행렬

은 다음과 같이 정의된다: However, the feature map is an output value that can be obtained only when there is data. Therefore, the present invention proposes a method for minimizing reconstruction errors between feature maps without data. For this purpose, it is assumed that one filter to be removed can be represented as a linear combination of the filters to be preserved. The weight and coefficient values of the filter that removes the coefficient values obtained at this time

Multiply by and add to each of the preserved filters.

When the first filter is removed, the transfer matrix

is defined as:

는 스칼라로서, 제거되는

번째 필터와 얼마나 관련있는지 정도를 나타낸다. 도 4(b)는 제거되는 필터의 수가 2개인 경우의 예시이다.here

is a scalar, which is removed

Indicates the degree to which it is related to the first filter. 4(b) is an example of a case in which the number of filters to be removed is two.

의

번째 채널만을 고려하게되는 경우 압축된 신경망의 특징맵

은 다음과 같이 나타낼 수 있다: One layer of the convolutional neural network consists of a convolutional filter, a batch normalization layer, and an activation layer. However, first, let's look at how reconstruction errors are made without considering batch normalization and activation layers. For the sake of simplicity in the above objective

of

If only the th channel is considered, the feature map of the compressed neural network

can be expressed as:

번째 레이어에서의

번째 특징맵에 대한 사전학습된 합성곱 신경망과 압축된 신경망의 재구성 오류는 다음과 같이 나타낼 수 있다: then,

in the second layer

The reconstruction error of the pretrained convolutional neural network and the compressed neural network for the feature map can be expressed as:

번째 레이어에서

번째 필터가 제거되었을 때,

번째 레이어의

재구성 오류는 다음이 나타낼 수 있다: from the above expression

in the second layer

When the second filter is removed,

of the second layer

A reconstruction error may indicate:

Based on the above expression, minimizing the objective expression can reduce the problem to minimizing the following expression:

를 만족하는

를 찾아야 한다. 앞서 배치 정규화와 활성화 레이어를 고려하지 않았기 때문에 쉽게, 활성화맵을 특징맵으로 근사할 수 있게 된다

. 따라서 위의 식은 하기식으로 나타낼 수 있다: here

to satisfy

should find Since we did not consider the batch normalization and activation layer earlier, we can easily approximate the activation map as a feature map.

. Therefore, the above equation can be expressed as:

는 데이터없이 구할 수 없는 값이므로, 나머지 부분

을 최소화하는

를 찾아야 하며, 이 부분을 잔차 오류

(Residual Error)라 한다. here

is a value that cannot be obtained without data, so the remainder

to minimize

, and this part is the residual error

(Residual Error).

는 사전학습된 합성곱 신경망의 네트워크의 파라미터만으로 최소화된다. here

Is It is minimized only with the parameters of the network of the pretrained convolutional neural network.

를 통과한 값으로 나타낼 수 있다

. 그리고

은 다음과 같이 표현할 수 있다: Considering a batch normalization layer, the activation map is the feature map of the batch normalization layer.

can be expressed as a value that passes through

. and

can be expressed as:

Organize 1.

,

,

은 배치 정규화 파라미터 이다. here,

,

,

is the batch normalization parameter.

는

로 대체될 수 있다. 그리고 상기 식의 두 번째 항을 배치 정규화 오류

(Batch Normalization Error)로 나타낸다. considering batch normalization

Is

can be replaced with And the second term in the above expression is the batch normalization error

(Batch Normalization Error).

Definition 2.

를 고려하면, 활성화맵은 특징맵이 배치 정규화와 활성화 레이어를 통과한 값으로 나타낼 수 있다

. 그리고

는 다음과 같이 나타낼 수 있다: Finally, the activation layer

Considering , the activation map can be represented by the value of the feature map passing through the batch normalization and activation layer.

. and

can be expressed as:

Theory 1.

는 활성화 오류(Activation Error)로 나타낸다. here,

is represented by an activation error.

는 잔차 오류(

), 배치정규화 오류(

), 활성화 오류(

)를 최소화하는 값으로 정해져야 한다. 하지만 활성화 오류를 줄이기 위해서는 특징맵의 모든 요소에 대해 모든 경우를 고려해야 하고, 이는 데이터없이는 불가능하다. 따라서, 다음 아래의 손실함수를 최소화하는

를 찾고자 한다: Through expansion that considers batch normalization and activation layers, the final reconstruction error can be defined as the following equation above. from the above expression

is the residual error (

), batch normalization error (

), activation error (

) should be set to a value that minimizes However, in order to reduce the activation error, all cases must be considered for all elements of the feature map, which is impossible without data. Therefore, minimizing the loss function

You want to find:

. here,

.

는 다음과 같이 닫힌 형태(closed form)을 얻어 낼 수 있다. that satisfies the above loss function.

can be obtained in closed form as follows.

theory 2.

,

는

번째레이어의

번째 필터의 벡터,

는

번째 레이어의

번째 필터의 벡터,

및

이다. here,

,

Is

of the second layer

vector of the th filter,

Is

of the second layer

vector of the th filter,

and

am.

5 is a diagram illustrating an algorithm for restoring performance of a neural network compressed by pruning without data and learning according to an embodiment of the present invention.

는 제거되는 필터에 대해 각각 위의 닫힌 형태로부터 얻게 되고, 이를 전달 행렬의 요소값으로 정하고 이를 사전학습된 합성곱 신경망의 필터와 2-모드곱(2-mode product) 연산을 수행하게 되면, 데이터와 학습없이 프루닝으로 압축된 신경망의 성능을 복구할 수 있다. 이에 대한 알고리즘을 도 5에 도시하였다. minimizing the loss function

is obtained from the above closed form for each filter to be removed, and when this is set as the element value of the transfer matrix and a 2-mode product operation is performed with the filter of the pretrained convolutional neural network, the data It is possible to recover the performance of compressed neural networks by pruning without training with . An algorithm for this is shown in FIG. 5 .

를 초기화한다(2번째 줄). 그리고 각 레이어에서 보존되는 필터에 대해서는 전달행렬에 대해서는 보존되는 필터에 해당하는 인덱스만 1이고 나머지가 0인 원핫 벡터(one-hot vector)를 입력한다(4-5번째 줄). 그 이후 제거되는 필터에 대해서는 손실함수를 최소화는

를 찾아 전달 행렬의 요소로 입력한다(6-7번째 줄). 하나의 레이어에 대해서 전달 행렬을 구하게 되면 사전학습된 합성곱 신경망의 파라미터와 전달 행렬과의 2-모드곱 를 수행하여 프루닝으로 손실된 성능을 복구시킨다. 이러한 과정을 프루닝을 수행하는 레이어에 대해서 모두 수행한다.For the layer pruning the convolutional neural network, the transfer matrix of each layer

initialize (line 2). And, for the filter that is preserved in each layer, a one-hot vector in which only the index corresponding to the filter that is preserved is 1 and the rest is 0 is input for the transfer matrix (lines 4 to 5). For filters that are removed after that, the loss function is minimized.

Find and enter it as an element of the transfer matrix (lines 6-7). When the transfer matrix is obtained for one layer, the performance lost by pruning is restored by performing 2-mode multiplication between the parameters of the pretrained convolutional neural network and the transfer matrix. All of these processes are performed on the pruning layer.

Tables 1 to 4 are the results of CIFAR and ImageNet data for applying the data proposed in the present invention and the technique for restoring the performance of the pretrained convolutional neural network compressed by the pruning technique without learning.

<Table 1>

<Table 2>

<Table 3>

<Table 4>

The network pruning criteria used in the experiment used L1-norm [6], L2-norm [7], and L2-GM [8] of convolutional filters, and the proposed method is robust even when pruning randomly ( In order to show robustness, an experiment was conducted using a random criterion. The top of each table shows the performance of the pretrained convolutional neural network and data. As an experimental comparison group, NM is a one-to-one compensation method, and Prune represents the performance of a compressed neural network without relearning after pruning. Through Tables 1 to 4, it can be confirmed that the proposed method shows higher average performance than other comparative experimental groups in various data and pretrained convolutional neural networks.

6 is a diagram showing a method for restoring the performance of a convolutional neural network according to an embodiment of the present invention and residual error (RE), batch normalization error (BE), and weighted average reconstruction error (WARE) with NM.

Figure 4 shows the residual error (RE), batch normalization error (BE), and weighted average reconstruction error (WARE) of the proposed method and NM. The weighted average reconstruction error is calculated using 1000 training data. The neural network recovered by the proposed method and the average reconstruction error of the method proposed by NM were compared. As shown in Fig. 4, the residual error of the proposed method has a low error value. The reason for this result is that NM transfers the information of the filter to be removed to one filter that is preserved for the filter to be removed, whereas the proposed method considers the relationship between filters that are preserved for the filter to be removed, resulting in lower residuals. can only have errors The reason why the batch normalization error and the weighted average reconstruction error have similar values in the initial layer of the neural network is that similar filters exist more in the initial layer as shown in FIG. 1 . However, it can be confirmed that the error of NM is higher than the proposed method in the later layers. Through this, it was confirmed that the proposed method maintains a smaller variation between layers compared to NM and has lower values for the three errors.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

<References>

[1] J. Luo, J. Wu, and W. Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5068-5076. IEEE Computer Society, 2017

[2] J. Luo and J. Wu. Neural network pruning with residual-connections and limited-data. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA,USA, June 13-19, 2020, pages 1455-1464. IEEE, 2020.

[3] S. Srinivas and R. V. Babu. Data-free parameter pruning for deep neural networks. In X. Xie, M. W. Jones, and G. K. L. Tam, editors, Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015, pages 31.1-31.12. BMVA Press, 2015.

[4] W. Kim, S. Kim, M. Park, and G. Jeon. Neuron merging: Compensating for pruned neurons. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020 , virtual, 2020.

[5] T. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM Rev., 51(3):455-337 500, 2009

[6] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.

[7] Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang. Soft filter pruning for accelerating deep convo315 lutional neural networks. In J. Lang, editor, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 2234-2240. ijcai.org, 2018

[8] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4340-4349. Computer Vision Foundation/IEEE, 2019.

Claims (8)

representing a damaged filter by using a linear relationship between a filter removed from a layer of a convolutional neural network pretrained through a pruning modeling unit and a filter preserved;
Defining a loss function due to the damaged filter according to a batch normalization layer and an activation layer through a pruning modeling unit; and
Restoring the performance of the neural network compressed by pruning by representing coefficient values that minimize the loss function as element values of the transfer matrix through a reconstruction modeling unit and performing an operation of a filter and a pruning matrix of a pretrained convolutional neural network.
A performance recovery method of a convolutional neural network comprising a. According to claim 1,
Representing a damaged filter by using a linear relationship between a filter removed from a layer of a convolutional neural network pretrained through the pruning modeling unit and a filter preserved,
By transforming the pruning matrix used in filter pruning of the pretrained convolution into a transfer matrix, the information of the filter to be removed is inherited by the filter to be preserved, and the filter to be removed is represented as a linear combination of the filters to be preserved.
Performance recovery method of convolutional neural networks. According to claim 1,
Defining a loss function due to the damaged filter according to the batch normalization layer and the activation layer through the pruning modeling unit,
After approximating the activation map with a feature map to minimize the loss function, the loss function is expressed by the batch normalization layer and the activation map according to the activation layer as the value that the feature map passed through the batch normalization layer for the residual error. Define and find the coefficient value that minimizes the residual error, batch normalization error, and activation error of the loss function
Performance recovery method of convolutional neural networks. According to claim 1,
Recovering the performance of the neural network compressed by pruning by performing the calculation of the filter and the pruning matrix of the pretrained convolutional neural network by representing the coefficient values that minimize the loss function through the reconstruction modeling unit as element values of the transfer matrix ,
Find a transfer matrix that minimizes the reconstruction error of the pretrained convolutional neural network and the compressed neural network by multiplying the filter to be removed by the coefficient value that minimizes the loss function and adding it to the filter to be preserved;
Coefficient values that minimize the loss function are represented as element values of the transfer matrix, and pruning is performed without data and learning by performing a 2-mode product operation of the filter of the pretrained convolutional neural network and the pruning matrix. Recovering the performance of compressed neural networks
Performance recovery method of convolutional neural networks. a pruning modeling unit that represents a damaged filter using a linear relationship between a filter removed from a layer of a pretrained convolutional neural network and a filter preserved, and defines a loss function due to the damaged filter according to a batch normalization layer and an activation layer; and
Reconstruction modeling unit that restores the performance of the neural network compressed by pruning by performing the calculation of the filter and pruning matrix of the pretrained convolutional neural network by representing the coefficient values that minimize the loss function as element values of the transfer matrix
A performance recovery device of a convolutional neural network comprising a. According to claim 5,
The pruning modeling unit,
By transforming the pruning matrix used in filter pruning of the pretrained convolution into a transfer matrix, the information of the filter to be removed is inherited by the filter to be preserved, and the filter to be removed is represented as a linear combination of the filters to be preserved.
A performance recovery device for convolutional neural networks. According to claim 5,
The pruning modeling unit,
After approximating the activation map with a feature map to minimize the loss function, the loss function is expressed by the batch normalization layer and the activation map according to the activation layer as the value that the feature map passed through the batch normalization layer for the residual error. Define and find the coefficient value that minimizes the residual error, batch normalization error, and activation error of the loss function
A performance recovery device for convolutional neural networks. According to claim 5,
The reconstruction modeling unit,
Find a transfer matrix that minimizes the reconstruction error of the pretrained convolutional neural network and the compressed neural network by multiplying the filter to be removed by the coefficient value that minimizes the loss function and adding it to the filter to be preserved;
Coefficient values that minimize the loss function are represented as element values of the transfer matrix, and pruning is performed without data and learning by performing a 2-mode product operation of the filter of the pretrained convolutional neural network and the pruning matrix. Recovering the performance of compressed neural networks
A performance recovery device for convolutional neural networks.

Images