KR102242334B1 - Method and Device for High Resolution Video Frame Rate Conversion with Data Augmentation - Google Patents

Abstract

A method and apparatus for high-speed frame rate conversion based on deep learning for performing frame interpolation with high quality and high speed on a high-resolution video image are provided. A data transformation method according to an embodiment of the present invention includes the steps of: receiving continuous input images to be used for learning an artificial intelligence model that generates an intermediate frame for frame interpolation; A first generating step of generating transformed images by changing image values of input input images; And a second generation step of generating an intermediate frame of the transformed images by changing an image value of an intermediate frame of the input input images.
As a result, it is possible to interpolate high-resolution images such as 4K frames at high speed, and it is possible to generate accurate optical flow maps even in such various images by performing robust learning data transformation on fade-in/out, lighting changes, and zoom images. Thus, high-speed robust frame interpolation for high-resolution images becomes possible.

Description

Method and Device for High Resolution Video Frame Rate Conversion with Data Augmentation}

The present invention relates to a frame rate conversion technology of a high-resolution video, and more particularly, to a method of predicting an optical flow estimation map based on a deep learning technique at a low resolution and restoring it to high resolution to interpolate the frame rate of a high-resolution video at high speed. will be.

1) Outline of video frame rate conversion technique

A moving picture is composed of a set of continuous still images. In video, a still image is called a frame, and the number of frames per unit time is called the frame rate of a video. For example, if 24 frames per second are configured, the frame rate is 24 fps (frame per second). The frame rate is determined by the intention of the photographer, the format of the image, and the limitations of the camera. In order for the observer to feel the image as a continuous screen, a certain frame rate is required, and if it is lower than this, the motion does not seem smooth. This phenomenon may vary depending on the size of the display, lighting, and viewing distance. To improve this, increasing the frame rate of a video by post-processing is called video frame rate conversion.

2) prior art

The simplest way to increase the video frame rate is to repeat the frames. For example, if a 30fps video is increased to a 60fps video, one frame is repeatedly output for each frame. However, in this method, since the amount of information in the video is the same and the continuity of the movement has not changed, the discomfort felt by the observer is the same. To solve this problem, a technology for generating a virtual frame using consecutive frames has been developed. That is, an intermediate image of t+0.5 second is newly generated by using an image between t seconds and t+1 second, and this is called a frame interpolation technique.

Various methods have been developed for frame interpolation, and generally, it goes through a two-step process as follows. The first step is to acquire a motion or optical flow map, and the second step is to generate an intermediate frame based on motion information (Synthesis). In order for the motion of an object to appear smoothly in a video, the motion of the object must be in the middle between the two images. Therefore, it is very important to accurately find the optical flow map that contains the motion information of the object. Various techniques based on this have been proposed.

Recently, deep learning algorithms have appeared and are used in various fields such as computer vision and speech recognition, and are showing superior performance compared to conventional methods. In line with this, various frame interpolation techniques using deep learning have emerged. These techniques are being studied continuously in recent years as they predict high-quality optical flow maps using deep learning and show more superior interpolation results than conventional methods.

3) Problems in the prior art

The problem of the existing method is a problem due to the network structure, and for a large resolution such as 4K (3840x2160) resolution, calculation is not possible due to insufficient GPU memory, or a very slow calculation speed is shown. This phenomenon makes it difficult to apply the frame interpolation method using deep learning to commercial applications that require real-time computation. In addition, high-resolution images generally have a larger motion than low-resolution images. Existing methods tend to generate low-quality optical flow maps for such large movements, which has a problem of causing deterioration of the interpolated image quality.

Also, the problem of the existing method is that the optical flow map is not accurately generated for this type of image because fade-in/out, lighting change, and zoom image are not considered. Therefore, in such an image, there is a disadvantage that the frame interpolation performance is rapidly deteriorated.

The present invention has been conceived to solve the above problems, and an object of the present invention is to provide a frame rate fast conversion method and apparatus based on deep learning that performs frame interpolation with high quality and high speed on a high-resolution video image. .

In accordance with an embodiment of the present invention for achieving the above object, a data transformation method includes the steps of: receiving consecutive input images to be used for learning an artificial intelligence model that generates an intermediate frame for frame interpolation; A first generating step of generating transformed images by changing image values of input input images; And a second generation step of generating an intermediate frame of the transformed images by changing an image value of an intermediate frame of the input input images.

In the first generating step, the first input image is modified by subtracting the adjustment parameter for the first input image among the input images, and the second input image is changed by adding the adjustment parameter for the second input image among the input images. In addition, the second generation step may be to modify the intermediate frame by adding an adjustment parameter that varies according to the time of the intermediate frame.

In the first generating step, the first input image is modified by adding an adjustment parameter to the first input image among the input images, and the second input image is modified by adding the adjustment parameter to the second input image among the input images. In addition, the second generation step may be to modify the intermediate frame by adding an adjustment parameter to the intermediate frame.

The adjustment parameter may be generated with a random value.

The adjustment parameter may be a parameter for adjusting any one of brightness, gamma function, contrast, hue, and saturation.

The data transformation method according to the present invention includes determining a reference patch of input images and an intermediate frame of the input images; Reducing a reference patch of a first input image among input images; Enlarging a reference patch of a second input image among the input images; A first adjustment step of adjusting a reference patch of the reduced first input image to an original size; It may further include a second adjustment step of adjusting the reference patch of the enlarged second input image to the original size.

The determining step may be to determine the location and size of the reference patch.

The data transformation method according to the present invention further includes a step of randomly generating a zoom parameter, wherein the reducing step is to reduce the reference patch of the first input image by using the zoom parameter, and the enlargement step is the second input image The reference patch of may be enlarged using a zoom parameter.

The first adjustment step and the second adjustment step may be performed when it is determined that the reference patch of the enlarged second input image does not deviate from the second input image.

According to another aspect of the present invention, there is provided an input unit for receiving continuous input images to be used for training of an artificial intelligence model that generates an intermediate frame for frame interpolation; And a processor configured to generate transformed images by changing an image value of input input images, and to generate an intermediate frame of transformed images by changing an image value of an intermediate frame of the input input images. A system is provided.

According to another aspect of the present invention, there is provided a method comprising: receiving continuous input images to be used for training of an artificial intelligence model that generates an intermediate frame for frame interpolation; A first generating step of generating transformed images by changing image values of input input images; A second generation step of generating an intermediate frame of the transformed images by changing an image value of an intermediate frame of the input input images; There is provided an artificial intelligence model training method comprising: training an artificial intelligence model with an intermediate frame between the transformed images generated in the first generating step and the transformed images generated in the second generating step.

According to another aspect of the present invention, transformed images are generated by changing image values of consecutive input images to be used for training of an artificial intelligence model that generates an intermediate frame for frame interpolation, and an intermediate frame of the input images A data transformation system for generating an intermediate frame of transformed images by changing an image value of; And a high-speed frame interpolation system for training an artificial intelligence model with an intermediate frame between the transformed images generated by the processor and the transformed images.

As described above, according to the embodiments of the present invention, it is possible to interpolate high-resolution images such as 4K frames at high speed, and by robustly transforming learning data on fade-in/out, lighting changes, and zoom images, It is possible to generate an accurate optical flow map even in an image, enabling high-speed and robust frame interpolation for high-resolution images.

1 is a frame interpolation technique,
2 is a block diagram of a high-speed frame interpolation system;
3 is a method of representing an image in the form of a pyramid,
4 is a detailed block diagram of a low-resolution optical flow prediction unit;
5 is a detailed block diagram of an optical flow resolution improving unit;
6 is an intermediate frame time interval,
7 is a detailed block diagram of an intermediate frame generation unit;
8 is a detailed block diagram of an intermediate frame resolution enhancement unit;
9 is a training data configuration,
10 is a data-transformed training data configuration,
11 is a method of determining a brightness adjustment parameter,
12 is a data transformation process for each sample for learning,
13 is a conceptual diagram of generating a zoom data modification patch;
14 is a zoom data modification patch generation process;
15 is a flowchart of generating a zoom data modification patch;
16 is a block diagram of a data transformation system.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

In an embodiment of the present invention, first, a method of performing high-speed frame interpolation on a high-resolution image using a deep learning network is presented, and secondly, a frame interpolation technique that is robust to lighting changes, fade in/out, and zoom images is provided. This suggests a method of argumentation of data during learning.

1. High-speed frame interpolation

In the high-speed frame interpolation system according to an embodiment of the present invention, as shown in FIG. 2, an optical flow prediction unit 110, a low-resolution optical flow resolution enhancement unit 120, an intermediate frame generation unit 130, and an intermediate frame It is configured to include a resolution enhancement unit 140.

The low-resolution optical flow prediction unit 110 predicts a bidirectional optical flow of temporally consecutive frames given as inputs at a low resolution. In the conventional method, since optical flow prediction is performed at the original resolution, a large amount of memory is required when receiving a high-resolution image as an input, and frame interpolation cannot be performed in a system with a limited hardware environment, and frame interpolation is performed. Even so, it has a problem with a very slow operation speed. In addition, in the case of a high-resolution image, the movement of objects in the image is very large, and the existing methods have a problem of generating blur or ghost deterioration in this case. In an embodiment of the present invention, the above problems are effectively solved by performing optical flow prediction at a low resolution.

3 shows a method of representing an image in the form of a pyramid used in an embodiment of the present invention. Level 1 resolution has the same resolution as that of the original input image, and Level 2 images have a resolution obtained by down-sampling the resolution of Level 1 to a size of 1/2. Finally, Level 3 resolution has a quarter size of Level 1 resolution. That is, if the original input resolution of Level 1 is W (width) x H (height), the resolution of Level 2 is W/2 x H/2, and the resolution of Level 3 is W/4 x H/4.

The low-resolution optical flow prediction unit 110 generates an optical flow map at the level 3 resolution, and the optical flow resolution enhancement unit 120 restores the level 3 resolution optical flow map to the Level 2 resolution. The intermediate frame generator 130 generates an intermediate frame having a level 2 resolution by using an optical flow having a level 2 resolution and an input image having a level 2 resolution. The intermediate frame resolution improving unit 140 restores the intermediate frame of Level 2 resolution to the original resolution of Level 1.

The low-resolution optical flow prediction unit 110 calculates an optical flow map between the two input frames. In general, a receptive field of sufficient size is required in a convolutional neural network to generate high-quality optical flow paths. However, as the receptive field increases, the algorithmic computation complexity of the convolutional neural network increases. Existing methods have a problem of having a slow motion speed for a high-resolution image such as a 4K frame because an optical flow map having the same size as the resolution of the original image is generated.

In an embodiment of the present invention, as shown in FIG. 4, after the resolution reduction unit 111 reduces the resolution of the original input image to Level 3 resolution, the optical flow prediction unit 112 predicts the optical flow.

,

의 L1은 Level 1의 해상도를,

,

의 L3는 Level 3의 해상도를 각각 나타낸다. The resolution reducing unit 111 reduces the original input image of Level 1 resolution to Level 3 resolution. The reduction method may use a bilinear or bicubic filter, and is not limited to a specific method. In Figure 3

,

L1 of is the resolution of Level 1,

,

L3 of represents the resolution of Level 3, respectively.

The optical flow prediction unit 112 calculates an optical flow map between the two input frames in a level 3 low resolution spiritually. Through this, it is possible to easily handle the movement of an object in an image even through a relatively small receptive field, and at the same time, it is possible to greatly reduce the memory and operation time required for calculation operations.

와

를 생성한다. 광학 흐름을 생성하는 방법은 다양한 딥러닝 네트워크 및 기존 방법들이 사용될 수 있으며 특정한 방법에 한정되지 않는다.The optical flow prediction unit 112 is an optical flow map having a size of a quarter of the original input resolution (that is, the size of Level 1).

Wow

Create A method of generating the optical flow may be used various deep learning networks and existing methods, and is not limited to a specific method.

The optical flow resolution improving unit 120 is a device for improving the resolution from a Level 3 optical flow map to a Level 2 optical flow map, and is configured as shown in FIG. 5.

The resolution increasing unit 1 121 increases the level 3 resolution optical flow map to the level 2 resolution optical flow map. The increment method may use a bilinear or bicubic filter, and is not limited to a specific method.

,

사이에 생성할 중간 프레임의 시간을 의미한다.

의 의미는

에서

로 진행할 때의 광학 흐름 지도를

은 그 역을 나타낸다. 도 5의

는 임의의 t에서 보간된 프레임인

에서

으로 진행되는 광학 흐름 지도로 볼 수 있고

도 마찬가지이다.

와

는 다음과 같이 산출 될 수 있다.The optical flow readjustment unit 123 readjusts the optical flow map for the input time t. As shown in Figure 6, t is

,

It means the time of the intermediate frame to be created between.

Means

in

The optical flow map when proceeding to

Represents the reverse. Of Fig. 5

Is the interpolated frame at any t

in

It can be viewed as an optical flow map that proceeds to

The same goes for it.

Wow

Can be calculated as follows.

(1)

(One)

,

를 생성한다. 증가 방법은 bilinear 또는 bicubic 필터 등을 사용할 수 있으며 특정 방법에 한정되지 않는다.The resolution reduction unit 2 (112) reduces the input original image with Level 1 resolution to Level 2 resolution.

,

Create The increment method may use a bilinear or bicubic filter, and is not limited to a specific method.

와

와 Level 2의 입력 영상

,

를 이용하여 임시 중간 프레임

와

를 생성한다.

와

는 식 (2)와 같이 구할 수 있으며, 여기서

은 back-warping 연산을 의미한다.In the warping unit 1 (124), the optical flow map

Wow

And Level 2 input video

,

Temporary Intermediate Frame Using

Wow

Create

Wow

Can be obtained as Equation (2), where

Means back-warping operation.

(2)

(2)

,

, 광학 흐름 재조정부(123)의 출력

,

, 해상도 감소부2(112)의 출력

,

을 이용하여 광학 흐름을 개선한다. 즉, 위의 입력들을

로 묶을 수 있고 광학 흐름 향상부(125)의 입력이 된다. 광학 흐름 향상부(125)는 합성곱 신경망으로 구성되며 개선된 광학 흐름 지도

,

를 생성한다.In the optical flow improving unit 125, the output of the warping unit 1 124

,

, The output of the optical flow readjustment unit 123

,

, The output of the resolution reduction unit 2 (112)

,

To improve optical flow. In other words, the above inputs

It can be bundled with and becomes the input of the optical flow enhancer 125. The optical flow enhancement unit 125 is composed of a convolutional neural network, and an improved optical flow map

,

Create

,

를 이용하여 Level 2 해상도의 중간 프레임을 생성한다. 도 7은 중간 프레임 생성부(130)의 상세 구성을 나타낸다. 도시된 바와 같이, 중간 프레임 생성부(130)는 와핑부2(131)와 중간 프레임 합성부(132)로 구성된다.The intermediate frame generation unit 130 is an output of the optical flow resolution enhancement unit 120

,

Create an intermediate frame of Level 2 resolution using. 7 shows a detailed configuration of the intermediate frame generation unit 130. As shown, the intermediate frame generating unit 130 includes a warping unit 2 131 and an intermediate frame combining unit 132.

,

와 Level 2의 입력 영상

,

를 이용하여 임시 중간 프레임

와

를 생성한다. 이는 식 (3)과 같이 구할 수 있다.In the warping unit 2 (131), the optical flow map

,

And Level 2 input video

,

Temporary Intermediate Frame Using

Wow

Create This can be calculated as Equation (3).

(3)

(3)

The intermediate frame synthesis unit 132 generates a final intermediate frame of Level 2 resolution by using the generated temporary intermediate frame as follows.

과

에 모두 존재한다면 B는 0.5로 설정되며, 만약 p가

에는 존재하지만

에 존재하지 않는다면 B=1 로 설정된다. B는 학습을 통해서 선택 될 수도 있고 고정된 값으로 결정될 수 있다.Here, ⊙ means element-wise multiplication and B means the blending parameter of the input image. For example, the pixel p is the two input images

and

B is set to 0.5 if it is present in both, and if p is

Exists in

If it does not exist, B=1 is set. B can be selected through learning or can be determined as a fixed value.

를 원영상 해상도인 Level1 해상도로 복원한다. 도 8에 도시된 바와 같이, 중간 프레임 해상도 향상부(140)는 해상도 증가부2(141과 142), 와핑부3(143), 해상도 개선부(144)로 구성된다.The intermediate frame resolution enhancement unit 140 selects the final intermediate frame of Level 2 resolution.

Is restored to the original image resolution, Level1 resolution. As shown in FIG. 8, the intermediate frame resolution improving unit 140 includes a resolution increasing unit 2 141 and 142, a warping unit 3 143, and a resolution improving unit 144.

와

,

를 Level 2 해상도에서 Level 1 해상도로 증가시켜

와

,

를 생성한다. 증가 방법은 bilinear 또는 bicubic 필터 등을 사용할 수 있으며 특정 방법에 한정되지 않는다.In the resolution increase section 2 (141 and 142),

Wow

,

From Level 2 resolution to Level 1 resolution

Wow

,

Create The increment method may use a bilinear or bicubic filter, and is not limited to a specific method.

,

,

의 입력 간에 시간 차이가 존재하여 개선 효율이 저하된다. 이를 방지하기 위해

과

는 광학 흐름 지도를 이용하여 와핑부3(143)을 거쳐 t시간의 프레임을 각각 생성하도록 한다. 이는 식 (4)와 같이 표현할 수 있다.The resolution improving unit 144 uses the original image information as an input to input a high-resolution image as an input. If you put the original image right away

,

,

There is a time difference between the inputs of, and the improvement efficiency is lowered. To prevent this

and

Is to generate a frame of time t through the warping unit 3 (143) using an optical flow map. This can be expressed as Equation (4).

(4)

(4)

,

,

를 묶어

을 입력으로 하는 합성곱 신경망으로 구성된다. 해상도 개선부(144)의 출력은 Level 1의 해상도를 갖는

이 전체 네트워크의 최종 출력이 된다.Finally, the resolution improving unit 144

,

,

Tie it up

It is composed of a convolutional neural network with as input. The output of the resolution improving unit 144 is

This becomes the final output of the entire network.

When intermediate frames are generated by the method presented above, accurate frame interpolation is possible because fast movements of high-resolution images can be accurately found at high speed.

2. Data transformation

As described above, a general image frequently flashes, changes lighting, and fades in/out. In such an image, it is difficult to accurately calculate the optical flow map. The main reason for this cause is that there are not many samples in which lighting changes occur in the data during training. In order to include a large number of samples with varying lighting, it is necessary to collect a lot of this data, which requires a lot of cost and effort. In an embodiment of the present invention, this is solved through data transformation during learning.

,

과 한개의 Ground Truth(GT) 영상

으로 구성된다. 딥러닝 네트워크에서 두 개의 입력 영상은 도 2에서의 입력 영상이 되고, 출력 영상은

가 된다. 학습 시

와

의 차이가 최소가 되게 하도록 네트워크를 갱신하게 된다. In general, frame interpolation learning using deep learning is performed using two input images as shown in FIG. 9.

,

And one Ground Truth(GT) video

It consists of. In the deep learning network, the two input images become the input images in FIG. 2, and the output images are

Becomes. When learning

Wow

The network is updated to minimize the difference between the two.

,

,

간에 조명 변화가 일어나지 않는다. 조명 변화가 있는 학습 데이터 샘플이 부족하므로 조명 변화가 발생할 경우 영상이 정확히 보간되지 않게 된다.As shown in Fig. 9, in most of the images, between training data, that is,

,

,

There is no change in lighting between them. Since the training data samples with lighting change are insufficient, the image is not accurately interpolated when lighting change occurs.

In order to solve this problem, as shown in FIG. 10, training data in which lighting changes are forcibly generated through data transformation (argumentation) are generated. Brightness and gamma functions may be used to adjust the brightness. The intensity of the brightness adjustment is determined by the brightness adjustment parameter b. In this case, transform the data as follows.

(5)

(5)

과

사이로 제한되므로 0과 1을 포함하지 않는 0과 1 사이의 값이다. 예를 들어 t=0.5 이면 GT의 밝기 조정 파라미터는 0이 된다. GT의 밝기 조정 파라미터는 식 (6)과 같이 일반화 할 수 있다. 밝기 조정 파라미터의 산출은 항상 선형식일 필요는 없으며 다양한 방식으로 계산이 가능하다.The GT image performs data transformation according to the position of t, and it is assumed that the brightness transformation changes linearly. This can be represented by a graph as shown in FIG. 11. The X-axis represents the time at which the GT is located, and the Y-axis represents the brightness adjustment parameter value. the range of t is

and

It is a value between 0 and 1, not including 0 and 1 because it is limited to between. For example, if t=0.5, the GT's brightness adjustment parameter is 0. The brightness adjustment parameter of GT can be generalized as Equation (6). The calculation of the brightness adjustment parameter does not always have to be linear and can be calculated in various ways.

(6)

(6)

Assuming that the signal range of the image is normalized between 0 and 1, the range of b is randomly selected for every batch, between -1 and 1. Data transformation is performed randomly by a certain probability P without performing data transformation in all training samples. That is, the number and intensity of data transformation are all selected randomly.

In addition, a method of equalizing data transformation to the input image can be applied. This can compensate for the deterioration of performance when samples of a specific brightness image are insufficient. In this case, data transformation is performed as in Equation (7).

(7)

(7)

을 생성시킨다(S210). 랜덤 변수

이

보다 클 경우에만(S220-Yes), 데이터 변형을 수행하도록 한다(박스 점선 부분).

는 0과 1사이의 값으로 데이터 변형이 적용되는 빈도를 결정한다. 만약 데이터 변형을 하도록 결정되면, 조정 파라미터 b를 생성한다(S230). 조정 파라미터 b는 데이터 변형 강도를 결정하게 된다. 조정 파라미터 범위 a는 영상의 범위 데이터 변형 종류 등에 따라 다양하게 결정될 수 있다. 그 다음 조명 변화 데이터 변환을 할지를 랜덤 변수

를 사용하여 결정하고(S240), 조명 변화 학습 변형을 할 경우에는(S250-Yes), 식(5,6)을 사용하여 데이터 변형을 수행하고(S260), 그렇지 않을 경우에는(S250-No), 식 (7)을 사용하여 데이터 변형을 수행한다(S270).12 shows a data transformation process performed for each learning sample above. Random variable with values between 0 and 1

Generates (S210). Random variable

this

If it is larger than (S220-Yes), data transformation is performed (a dotted line in the box).

Is a value between 0 and 1 that determines the frequency at which data transformation is applied. If it is determined to perform data transformation, an adjustment parameter b is generated (S230). The adjustment parameter b will determine the strength of the data transformation. The adjustment parameter range a may be variously determined according to the type of image range data transformation. Then, whether to convert the lighting change data to a random variable

Determine using (S240), and in case of learning transformation of lighting change (S250-Yes), perform data transformation using equations (5,6) (S260), otherwise (S250-No) , Data transformation is performed using Equation (7) (S270).

The above method can be applied to various data transformation functions such as gamma function, contrast, hue, and saturation.

As another example, a data transformation method for effectively responding to a zoom image is presented. In the zoomed image, the size of the object changes as time progresses. In order to interpolate efficiently with respect to such an image, a data modification patch considering zoom is generated as shown in FIG. 13.

로 하고 패치의 크기를

라고 하자. 입력 영상과 GT 영상의 패치의 위치는 동일하다. 도 13에서와 같이 한 개의 입력은 크기를 감소시키고 다른 한 개의 입력은 크기를 증가시키면서 줌 영상에 대응하는 패치를 생성할 수 있다.In general, a part of the video is used as a patch when learning is conducted. For example, the size of the video

And the size of the patch

Let's say. The location of the patch of the input image and the GT image is the same. As shown in FIG. 13, a patch corresponding to the zoom image may be generated while one input decreases the size and the other input increases the size.

을 생성시킨다(S310). 랜덤 변수

가

보다 클 경우에만(S320-Yes) 줌 데이터 변형을 수행하도록 한다.14 and 15 show a detailed zoom data generation patch process. Like the data transformation process above, a random variable with a value between 0 and 1

It generates (S310). Random variable

end

If it is larger than (S320-Yes), the zoom data transformation is performed.

If it is determined to perform zoom data transformation, the zoom parameter s is randomly set (S330). The zoom parameter s means the intensity of the zoom magnification, and if it is 1, it becomes the same as the original patch size. If the zoom parameter is too large, a value between 1 and 1.2 is recommended because there are many cases where the patch size may deviate from the image size.

를 결정한다(S340). 패치의 크기는

이다. 기준 패치의 위치, 패치의 크기 및 줌 파라미터를 이용하여 입력 영상

,

에서 사용될 위치를 식 (8~11)과 같이 결정한다. 식(8)과 식(10)에서

는

를 넘지 않는 가장 큰 짝수를 의미한다.

영상에서 사용될 패치의 크기는 식 (8)과 같이 산출된다. 재계산된 패치 크기인

를 기반으로

의 패치 영역을 식 (9)와 같이 계산한다. 이는 도 14의

영상의 빨간 박스 영역을 의미한다.

영상에서 사용될 패치의 크기는 식 (10)과 같이 산출된다. 재계산된 패치 크기인

를 기반으로

의 패치 영역을 식 (11)와 같이 계산한다. 이는 도 14의

영상의 빨간 박스 영역을 의미한다.

과

의 위치는 랜덤하게 바뀔 수 있다. Next, as shown in Fig. 14, the position of the reference patch

It is determined (S340). The size of the patch is

to be. Input image using the location of the reference patch, the size of the patch, and the zoom parameter

,

Determine the location to be used in the equation (8~11). In equations (8) and (10)

Is

It means the largest even number not exceeding.

The size of the patch to be used in the image is calculated as Equation (8). The recalculated patch size

Based on

The patch area of is calculated as in Equation (9). This is the

Refers to the red boxed area of the image.

The size of the patch to be used in the image is calculated as Equation (10). The recalculated patch size

Based on

The patch area of is calculated as in Equation (11). This is the

Refers to the red boxed area of the image.

and

The position of can be changed randomly.

(8)

(8)

(9)

(9)

(10)

(10)

(11)

(11)

의 크기를 갖는

를 생성한다(S370).

는

로 해상도를 조정하여 줌 데이터 변형된 패치인

를 생성한다. 마찬가지로 식 (10)에서 계산된 좌표를 이용하여

의 크기를 갖는

를 생성하고

로 해상도를 조정하여 줌 데이터 변형된 패치인

를 생성한다(S340). 최종적으로

를 갖는

,

,

를 이용하여 학습을 수행하도록 한다.If the area calculated in Equation (9) is out of the range of the image, the zoom data transformation process is not performed. If the boundary condition is satisfied (S360-Yes), using the coordinates calculated in Equation (9)

Having the size of

Generates (S370).

Is

Zoom data transformed patch by adjusting the resolution with

Create Similarly, using the coordinates calculated in Equation (10)

Having the size of

And create

Zoom data transformed patch by adjusting the resolution with

Generates (S340). Finally

Having

,

,

Use to perform learning.

A system capable of performing the data transformation described so far will be described in detail with reference to 16 below. 16 is a block diagram of a data transformation system according to another embodiment of the present invention.

The data transformation system according to an embodiment of the present invention can be implemented as a computing system including an input unit 210, a processor 220, an output unit 230, and a storage unit 240, as shown in FIG. 16. .

The input unit 210 is a means for receiving images through an external storage medium, an external device, a communication network, or the like, and the processor 220 increases the training images by modifying the input image data.

In transforming the image data, the processor 220 uses the method suggested in the above-described embodiment. The storage unit 240 is an internal storage medium that provides a storage space necessary for the processor 220 to transform image data.

The output unit 230 outputs the training images added by the processor 220 to an external storage medium, an external device, a communication network, or the like.

3. Modification

Until now, a method of performing high-speed frame interpolation on high-resolution images using a deep learning network and a robust frame interpolation technique for lighting changes, fade in/out, and zoom images. This is a preferred implementation for a method of transforming data during training. It has been described in detail with examples.

When inputting a high-resolution image, an optical flow map having the same resolution as the original resolution is generated and frame interpolation is performed, which requires a large amount of memory. Unlike the conventional method having a very slow interpolation speed, the input in the embodiment of the present invention By converting a high-resolution image to a low-resolution image, an optical flow map was created at high speed, and it was restored to the original high resolution, enabling high-resolution images such as 4K frames to be interpolated at high speed.

In addition, in the embodiment of the present invention, it is possible to generate an accurate optical flow map even in such a variety of images by performing robust training data transformation on fade-in/out, lighting change, and zoom images.

Meanwhile, it goes without saying that the technical idea of the present invention can be applied to a computer-readable recording medium containing a computer program for performing functions of the apparatus and method according to the present embodiment. Further, the technical idea according to various embodiments of the present invention may be implemented in the form of a computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and can store data. For example, a computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, or the like. In addition, a computer-readable code or program stored in a computer-readable recording medium may be transmitted through a network connected between computers.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be understood individually from the technical spirit or prospect of the present invention.

110: optical flow prediction unit
120: low-resolution optical flow resolution enhancement unit
130: intermediate frame generation unit
140: intermediate frame resolution enhancement unit

Claims (12)

Receiving consecutive input images to be used for training of an artificial intelligence model that generates an intermediate frame for frame interpolation;
A first generating step of generating transformed images by changing image values of input input images;
A second generation step of generating an intermediate frame of the transformed images by changing an image value of an intermediate frame of the input input images;
Determining a reference patch of the input images and an intermediate frame of the input images;
Reducing a reference patch of a first input image among input images;
Enlarging a reference patch of a second input image among the input images;
A first adjustment step of adjusting a reference patch of the reduced first input image to an original size;
And a second adjustment step of adjusting the reference patch of the enlarged second input image to its original size.
The method according to claim 1,
The first generation step,
For the first input image among the input images, the first input image is modified by subtracting the adjustment parameter,
For the second input image among the input images, the second input image is changed by adding an adjustment parameter,
The second generation step is,
A data transformation method, characterized in that the intermediate frame is transformed by adding an adjustment parameter that varies according to the time of the intermediate frame.
The method according to claim 2,
The first generation step,
The first input image is modified by adding an adjustment parameter to the first input image among the input images,
The second input image is modified by adding an adjustment parameter to the second input image among the input images,
The second generation step is,
A data transformation method, characterized in that the intermediate frame is transformed by adding an adjustment parameter to the intermediate frame.
The method according to claim 2 or 3,
The adjustment parameters are,
Data transformation method, characterized in that generated as a random value.
The method of claim 4,
The adjustment parameters are,
A data transformation method, characterized in that it is a parameter for adjusting any one of brightness, gamma function, contrast, hue, and saturation.
delete The method according to claim 1,
The decision step is,
A data transformation method, characterized in that determining the location and size of the reference patch.
The method according to claim 1,
Randomly generating a zoom parameter; further comprising,
The reduction step is,
Reduce the reference patch of the first input image using the zoom parameter,
The magnification step is,
A data transformation method, characterized in that the reference patch of the second input image is enlarged using a zoom parameter.
The method according to claim 1,
The first adjustment step and the second adjustment step,
A data transformation method, characterized in that it is performed when it is determined that the reference patch of the enlarged second input image does not deviate from the second input image.
An input unit that receives continuous input images to be used for training of an artificial intelligence model that generates an intermediate frame for frame interpolation;
A processor for generating transformed images by changing the image values of the input input images, and generating an intermediate frame of the transformed images by changing the image value of the intermediate frames of the input input images,
The processor is
Determine the reference patch of the input images and the intermediate frame of the input images,
Reduce the reference patch of the first input image among the input images,
Enlarging the reference patch of the second input image among the input images,
Adjust the reference patch of the reduced first input image to its original size,
A data transformation system comprising adjusting a reference patch of the enlarged second input image to an original size.
Receiving consecutive input images to be used for training of an artificial intelligence model that generates an intermediate frame for frame interpolation;
A first generating step of generating transformed images by changing image values of input input images;
A second generation step of generating an intermediate frame of the transformed images by changing an image value of an intermediate frame of the input input images;
Training an artificial intelligence model with an intermediate frame between the transformed images generated in the first generating step and the transformed images generated in the second generating step;
Determining a reference patch of the input images and an intermediate frame of the input images;
Reducing a reference patch of a first input image among input images;
Enlarging a reference patch of a second input image among the input images;
A first adjustment step of adjusting a reference patch of the reduced first input image to an original size;
A second adjustment step of adjusting the reference patch of the enlarged second input image to an original size;
Including; training an artificial intelligence model using the reference patch of the first input image adjusted in the first adjustment step, the reference patch of the second input image adjusted in the second adjustment step, and the reference patch of the intermediate frame determined in the determination step; Artificial intelligence model learning method, characterized in that.
The transformed image is generated by changing the image value of the continuous input images to be used for the training of the artificial intelligence model that generates the intermediate frame for frame interpolation, and the transformed image by changing the image value of the intermediate frame of the input input images. A data transformation system for generating an intermediate frame of the data; And
Including; a high-speed frame interpolation system that trains an artificial intelligence model with an intermediate frame between the transformed images generated by the processor and the transformed images; and
The data transformation system,
Determine the reference patch of the input images and the intermediate frame of the input images,
Reduce the reference patch of the first input image among the input images,
Enlarging the reference patch of the second input image among the input images,
Adjust the reference patch of the reduced first input image to its original size,
Adjust the reference patch of the enlarged second input image to its original size,
High-speed frame interpolation system,
An artificial intelligence model learning system, characterized in that the artificial intelligence model is trained using the adjusted reference patch of the first input image, the adjusted reference patch of the second input image, and the determined reference patch of the intermediate frame.

Images