KR100698640B1 - Frame Interpolation Method Using Warping Transform and Its Apparatus - Google Patents

Abstract

Provided are a frame interpolation method and apparatus therefor using warping transformation.

In accordance with another aspect of the present invention, there is provided a brain interpolation method using a warping transformation, comprising: inputting a continuous video signal including a first frame and a second frame, and estimating a motion between the first frame and the second frame; Generating a motion vector between the first frame and the second frame from the estimated motion; Identifying an object and a background of an interpolation frame to be inserted from the generated motion vector; And warping the object, and inserting the interpolation frame between the first frame and the second frame by applying a block-based frame interpolation method to the background.

According to another aspect of the present invention, there is provided a frame interpolation apparatus using a warping transform, comprising: a motion estimator configured to input a continuous video signal including a first frame and a second frame, and estimate a motion between the first frame and the second frame; A motion vector generator configured to generate a motion vector between the first frame and the second frame from the estimated motion; An identification unit for identifying an object and a background of an interpolation frame to be inserted from the generated motion vector; And an interpolation frame inserter for warping the object and inserting the interpolation frame between the first frame and the second frame by applying a block-based frame interpolation method to the background.

According to the present invention, by applying the frame interpolation method and the warping transform frame interpolation method simultaneously to apply the block-based image processing method, it is possible to effectively reduce the motion blurring phenomenon occurring in the hold type display. This can minimize the deterioration of image quality that can occur during frame interpolation.

Description

Method to interpolate frame using warping transformation and its device {Method to interpolate frame using warping transformation and Apparatus

1 is a flowchart of a frame interpolation method using warping transformation according to the present invention.

2 is a block diagram of a frame interpolation apparatus using warping transformation according to the present invention.

3 is a process of estimating a motion vector according to the present invention.

4A illustrates a motion vector and a motion segmentation process according to the present invention.

4b illustrates an adaptive motion compensation field according to the present invention.

5 is a warping transformation process using the estimated motion vector according to the present invention.

6A shows an image of a previous frame.

6B shows an image of a current frame.

6C illustrates a motion vector of an interpolated frame.

6D illustrates an image obtained by separating an object from a background.

6E illustrates an image of an interpolated frame using only warping transformation.

6F illustrates an image of an interpolation frame according to the present invention.

7A shows an enlarged view of a part of an object part of an image of an interpolation frame based on a conventional block-based frame interpolation method.

FIG. 7B is an enlarged view of a part of the object part of FIG. 6F.

FIG. 7C is an enlarged view of a portion of the object part of FIG. 6E.

8 illustrates a frame rate conversion process according to an embodiment of the present invention.

The present invention relates to image processing, and more particularly, to a frame interpolation method and apparatus using warping transformation.

Hold type displays, such as liquid crystal displays (LCDs) and electro-luminescence displays, which hold the display of all frames until a new image is recorded, are used for moving picture displays. At this time, the motion blurring phenomenon caused by the observer's eye is generated on the movement of the moving object, which results in deterioration of image quality. Frame rate-up conversion is a representative method for removing such a motion blurring phenomenon. This frame rate-up conversion technique reduces motion blurring by creating a new interpolation frame between frames to increase the frame rate. This method has a method of creating an interpolation frame using motion compensation and interpolating interpolation between frames adjacent to the interpolation frame. This motion compensation uses a motion vector detected by a Block Matching Algorithm (BMA) method. The block matching algorithm method divides a first frame of a continuous image into a plurality of first blocks, searches for a second block having the highest correlation from a second frame adjacent to the first frame for each first block, A method of obtaining a motion vector from two blocks to a first block.

In the frame interpolation method using such motion compensation, when the interpolation frame is generated by the motion compensation, the first motion vector obtained with respect to the first reference frame and the second reference frame is interpolated between the interpolation frame surface and the first reference frame. An operation called scale conversion is performed to convert the second motion vector. The interpolation frame is generated by performing motion compensation using the second motion vector obtained by the scale conversion in this way. That is, the end point of the second motion vector is fixed on the first frame, and the position of the block on the interpolation frame plane where the start point of the second motion vector points to the image data of the block indicated by the end point of the second motion vector on the first frame. Copy to As described above, the conventional frame interpolation method is performed by executing motion prediction and motion compensation based on a block-based image processing method.

However, the frame interpolation method using the conventional block-based image processing method cannot express parametric movements such as rotation, enlargement, and reduction of an image. There is a problem that it is easy to occur.

Accordingly, the first technical problem to be achieved by the present invention is to efficiently remove motion blurring caused by blocking artifacts occurring in a hold type display. It is to provide a frame interpolation method that can minimize image degradation.

The second technical problem to be achieved by the present invention is to provide a frame interpolation apparatus using the frame interpolation method.

The present invention to achieve the first technical problem,

Inputting a continuous video signal including a first frame and a second frame and estimating a motion between the first frame and the second frame; Generating a motion vector between the first frame and the second frame from the estimated motion; Identifying an object and a background of an interpolation frame to be inserted from the generated motion vector; And inserting the interpolation frame between the first frame and the second frame by warping the object and applying a block-based frame interpolation method to the background. .

The present invention to achieve the second technical problem,

A motion estimator for inputting a continuous video signal including a first frame and a second frame and estimating a motion between the first frame and the second frame; A motion vector generator configured to generate a motion vector between the first frame and the second frame from the estimated motion; An identification unit for identifying an object and a background of an interpolation frame to be inserted from the generated motion vector; And an interpolation frame inserter for inserting the interpolation frame between the first frame and the second frame by warping the object and applying a block-based frame interpolation method to the background. to provide.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings and examples, but the present invention is not limited thereto.

1 is a flowchart of a frame interpolation method using warping transformation according to the present invention.

First, an image for interpolating a frame is input. The input image is input in units of frames that are temporally contiguous, and a current frame is defined as a second frame, and a frame immediately preceding the temporal frame is defined as a first frame.

In case of LCD monitor whose input image signal is RGB signal, it scans at the frame rate of 60Hz, but if the screen change is large due to the characteristics of large screen and liquid crystal, motion jitter or motion blurring This happens. This quality deterioration can be improved by converting a 60Hz input image into a 90Hz output image through the proposed technique and outputting it to the screen.

It can also be used in the case of converting an image compressed at a low frame rate into a high quality image having a high frame rate. That is, it may be used when the input image is converted into a high-quality image having a high frame rate of a signal that is compressed at a low frame rate of the existing analog or digital signal YCbCr or YPbPr. YCbCr or YPbPr is not a color signal but a color expression method based on Y, which refers to luminance, and color difference signals Cb (Pb) and Cr (Pr). Since the human eye is more sensitive to brightness than color, it is processed using a chrominance signal. In the signal standards, Pb and Pr are terms used to refer to analog color difference signals, and Cb and Cr are terms used when encoded into digital signals. However, they are often referred to interchangeably. In the present invention, the signal of the input image may be used when interpolating an image of 480i, 720i, or 1080i, which is a signal of YCbCr or YPbPr.

Next, the motion between frames is estimated (step 100). Motion Estimation (ME) is the task of finding the most similar block between the previous frame and the current frame. The motion estimation method uses a Block Matching Algorithm (BMA) in consideration of accuracy of motion, real-time processing, and hardware implementation. The video signal to be inserted between the frames is mainly generated using a block matching algorithm.

After estimating the motion between two consecutive frames in time, a motion vector is generated (step 110). A motion vector is assigned to a frame to be interpolated from the estimated motion between successive frames. A motion vector of each block is obtained based on the previous frame and the current frame, and blocks having a large influence on the blocks of the interpolated frame are determined as candidates. In order to generate the motion vector of the frame to be interpolated, the motion vector of the candidate blocks is added to the blocks to be compensated with weights to determine the motion vector of the interpolated frame block.

Next, the background and the object are identified based on the calculated motion vector (step 120). At this time, the rate of change of the motion vector serves as a reference for identifying the background and the object. The rate of change of the motion vector is the gradient of the motion vector between the predetermined block in the first frame and the block in the second frame corresponding to the block. If there is almost no gradient of the motion vector, the background part is motionless. If the motion vector is constant, the background is motionless. On the other hand, blocks with gradient values of similar motion vectors are treated as the same object.

Next, the part determined as the object interpolates the frame by applying the warping transformation (step 130).

The part determined as the background interpolates the frame by applying the conventional block-based frame interpolation method (step 131).

This hybrid method can accommodate both the advantages of the warping transform and the advantages of the block-based frame interpolation technique, and thus can quickly interpolate frames with less blocking artifacts.

Finally, an interpolation frame is generated (step 140).

2 is a block diagram of a frame interpolation apparatus using warping transformation according to the present invention. The frame interpolation apparatus according to the present invention includes a motion estimator 200, a motion vector generator 210, an identifier 220, and an interpolation frame generator 230.

The motion estimator 200 performs backward motion estimation by using the temporally contiguous first and second frames of the input image. Referring to FIG. 3, backward motion estimation refers to backward motion estimation for estimating a motion vector of a frame 320 to be interpolated from a first frame 300 which is temporally previous and a second frame 310 consecutive to it. .

The motion vector generator 210 obtains a motion vector of each block based on the previous frame and the current frame, and determines candidate blocks having a large influence on the blocks of the interpolated frame from the obtained motion vectors. A motion vector of the interpolated frame block is determined by adding the weighted influences to the blocks to be compensated to the motion vectors of the candidate blocks.

The identification unit 220 identifies the object and the background by calculating the slope of the motion vector. The slope of the motion vector is calculated using the motion vector obtained by the motion estimation. By calculating the slope of the motion vector, the image and object are identified by estimating which image area the block belongs to.

The interpolated frame generation unit 230 performs warping transformation on the portion determined as the object and interpolates the image by performing the existing block-based frame interpolation method on the portion determined as the background.

,

,

,

의 각각의 면적이 다. 시간적으로 연속한 제 1 프레임(300)과 제 2 프레임(310) 사이의 움직임 벡터들의 절반값만큼 이동한 값과 대치된다. 수식으로 표현하면 다음과 같다.3 is a process of estimating a motion vector according to the present invention. For example, first, the pixels of the first frame 300 and the second frame 310 consecutively divided are first divided into 5 × 5 blocks. The present invention applies backward motion estimation to estimate the motion vector of the frame 320 to be interpolated from the first frame 300 and the second frame 310 that are temporally previous (330). That is, a motion vector of a block of an interpolation frame is obtained based on a previous frame and a current frame, and a block 325 having a large influence on the block of the interpolation frame is determined as a candidate. The influence of the motion vector of the candidate block on the block to be compensated for is added to the motion vector of the interpolated frame block by adding the values multiplied by the weights 321, 322, 323, and 324. The weight is

,

,

,

Each of the area is c. It is replaced by a value shifted by half of the motion vectors between the temporally continuous first frame 300 and the second frame 310. Expressed as an expression:

는 움직임 벡터의 각각의 후보의 가중치를 나타내며,

는 보간 프레임의

번째 블록의 추정된 움직임 벡터를 나타낸다. 즉, 보간 프레임의

번째 블록의 추정된 움직임 벡터는 각각의 움직임 벡터의 각각의 후보의 가중치와

번째 블록의 추정된 움직임 벡터를 곱한값을 모두 더하고, 이 값에 가중치들의 총합을 나누어서 얻어지는 움직임 벡터의 절반값이 된다And,

Denotes the weight of each candidate in the motion vector,

Of the interpolation frame

It represents the estimated motion vector of the first block. That is, the interpolation frame

The estimated motion vector of the first block is equal to the weight of each candidate of each motion vector.

Add up the product of the estimated motion vector of the first block and add this value to the half of the motion vector obtained by dividing the sum of the weights.

4A illustrates a motion vector and a motion segmentation process according to the present invention.

After calculating the motion vector according to each macro block of the interpolation frame and showing it in each macro block, vectors having similar motions are divided into objects, and blocks having no change are divided into backgrounds. Between the background and the object is a boundary.

4b illustrates an adaptive motion compensation field according to the present invention.

Referring to FIG. 4B, a 2-D macroblock based on motion compensation and a block matching algorithm is illustrated as an interface between an internal region and an object. Referring to FIG. 4A, a portion without a motion of the vector becomes a background 410 in FIG. 4B, and a vector with similar movement becomes an object 430. In FIG. 4B, the object is shown as a 2-D mesh as an inner region. The boundary surface 420 is an overlapped portion in motion estimation using two adjacent frames.

5 is a warping transformation process using the estimated motion vector according to the present invention.

,

,

,

)과 연결된 움직임 벡터(

,

,,

)로부터 와핑변환을 적용하여

,

,

,

를 생성하여 보간 프레임의 블록을 재구성한다. 이 변환은 3×3 행렬에 의해 동차의 3-벡터로 선형 변환시키는 과정이다. 구체적인 수학식은 다음과 같다.5, the center of the four blocks (

,

,

,

) And the motion vector associated with

,

, ,

Apply warping transformation from

,

,

,

Reconstruct the blocks of the interpolated frame by generating. This transformation is the process of linear transformation into a homogeneous three-vector by a 3x3 matrix. The specific equation is as follows.

의 형식으로 표현할 수 있다.

는 동차(Homogeneous) 행렬이라고 정의한다. 여기서 9개의 동차(Homogeneous) 행렬의 각 성분은 보간 프레임의 성분들을 결정하게 된다. 참조 프레임의 네개의 중앙점(

,

,

,

)과 이에 해당하는 보간 프레임의 (

,

,

,

)변형된 점으로부터 동차(Homogeneous) 행렬

를 결정할 수 있고, 이 동차(Homogeneous) 행렬을 적용하여 모든 픽셀의 움직임 벡터가 결정된다. 즉, 종래의 블록 기반 방식의 프레임 보간 방법은 블록내의 모든 픽셀들이 하나의 움직임 벡터를 가지는 방식이고, 본 발명은 와핑 변환을 적용한 움직임 벡터가 선형적으로 부드럽게 변하는 것이다. In short,

It can be expressed in the form of.

Is defined as a homogeneous matrix. Here, each component of the nine homogeneous matrices determines components of the interpolation frame. The four center points of the reference frame (

,

,

,

) And the corresponding interpolation frame's (

,

,

,

Homogeneous matrix from transformed points

The motion vectors of all the pixels are determined by applying this homogeneous matrix. That is, in the conventional block-based frame interpolation method, all pixels in a block have a single motion vector. In the present invention, a motion vector to which a warping transformation is applied is linearly and smoothly changed.

FIG. 6A is an image of a continuous previous frame (first frame) and FIG. 6B (second frame) is an image of the current frame. 6C is a motion vector of an interpolated frame estimated from a temporally continuous previous frame and a current frame. In FIG. 6C, an area indicated by a dot represents a background without movement, and an area displaying a certain direction represents an object with movement.

6D illustrates an image obtained by separating an object and a background from the generated motion vector. Referring to FIG. 6D, a black portion shows a background portion without movement, a gray portion shows an object portion with movement, and a white portion shows a boundary portion between the background and the object.

6E illustrates an image of an interpolated frame to which only warping transform is applied.

6F illustrates an interpolation frame image using a hybrid method in which a conventional block-based frame interpolation method is applied to a background and a frame interpolation method using a warping transform is applied to an object.

7A is an enlarged view of a part of an object part of an image of an interpolation frame based on a conventional block-based frame interpolation method, and FIG. 7B is a part of an image of an interpolation frame according to the present invention, that is, a part of the object part of FIG. 6F. 7C is an enlarged view of an image of an interpolation frame to which only a warping transformation is applied, that is, a part of an object part of FIG. 6E.

7A and 7B, it can be seen that there is no difference in image quality in the background part, whereas the object part with a lot of motion shows that the blocking artifact degree of the interpolated frame to which the present invention is applied is significantly reduced. . FIG. 7C illustrates an image of an interpolation frame generated by warping a background and an object. When comparing the images of FIG. 7B and FIG. 7C, the image quality of the background part and the object part is very similar. This means that the background part without motion does not deteriorate the overall image quality of the interpolated frame even when the conventional block-based frame interpolation method is applied without applying the warping transformation.

An embodiment of the present invention will be described. In the frame interpolation method for improving image quality, an LCD TV is currently scanning at a frame rate of 60 Hz, but motion jitter or motion blur occurs when the screen change is large due to the characteristics of a large screen and a liquid crystal. By converting the input image of 60Hz to the output image of 90Hz through the proposed method and minimizing such degradation of image quality, the present invention can also be applied in real time by hardware implementation through ASIC.

8 illustrates a process of changing an image of 60 Hz into a frame of 90 Hz by applying the frame interpolation method according to the present invention. For example, assuming that an image having a frequency of 60 Hz is input in 20 ms units per frame, an image having a frequency of 90 Hz is input in 13.3 ms units per frame. Referring to FIG. 8, when a frame of an image having a frequency of 60 Hz is input in a 20 ms time unit, a frame of an image having a frequency of 90 Hz to be interpolated should be output in a 13.3 ms time unit. On the other hand, the first frame 810 of 90Hz is placed to be output at the same time as the first frame 800 of the image having a frequency of 60Hz. Then, a second frame 811 of 90 Hz is generated at a two-thirds time between the first frame 800 and the second frame 801 of the image having a frequency of 60 Hz, and the third frame 812 of 90 Hz. ) Is generated at a time interval of 1/3 of the second frame 801 and the third frame 802 at 60 Hz. The fourth frame 813 at 90 Hz is output at the same time as the third frame 802 at 60 Hz, and this process is repeated continuously. According to the present invention, an image having a frequency of 60 Hz can be converted to 90 Hz while minimizing degradation of image quality.

When the frame interpolation method for video format conversion is described, video format conversion between them is required to view programs conforming to various standards on various screens. HDTV, which has recently been spreading, requires a high frame rate. However, most broadcast programs do not support this and use the existing NTSC or PAL method. Therefore, video format conversion through frame interpolation method is essential to view the current broadcasting program broadcasted with low frame rate on HDTV. Application of the present invention enables format conversion of better image quality.

The present invention can be embodied as code that can be read by a computer (including all devices having an information processing function) in a computer-readable recording medium.

The computer-readable recording medium includes all kinds of recording devices in which data is stored which can be read by a computer system. Examples of computer-readable recording devices include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like. Computer-readable recording media are distributed and distributed over networked computer devices. Computer readable code may be stored and executed in such a manner.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible. Therefore, the true technical protection scope of the present invention will be defined by the technical details of the appended claims.

As described above, according to the present invention, a motion blurring generated in a hold type display by simultaneously applying a frame interpolation method and a warping transform frame interpolation method to which a block-based image processing method is applied. ) Effect can be effectively removed to minimize image quality deterioration that can occur during frame interpolation.

Claims (12)

Inputting a continuous video signal including a first frame and a second frame and estimating a motion between the first frame and the second frame; Generating a motion vector between the first frame and the second frame from the estimated motion; Identifying an object and a background of an interpolation frame to be inserted from the generated motion vector; And Warping transformation of the object and inserting the interpolation frame between the first frame and the second frame by applying a block-based frame interpolation method to the background. The method of claim 1, The image signal is a frame interpolation method using warping transform, characterized in that the RGB signal. The method of claim 1, And the video signal is a YCbCr or YPbPr signal. The method of claim 1, The step of estimating the motion is frame interpolation method using warping transform, characterized in that the backward motion estimation. The method of claim 1, Estimating the motion vector, Dividing predetermined pixels of the first frame and the second frame in units of blocks; And Estimating a block-by-block motion between the first frame and the second frame. The method of claim 1, Identifying the object and the background, Calculating a slope of a motion vector between a predetermined block in the first frame and a block in the second frame corresponding to the block; And And identifying an object and a background from the slope of the calculated motion vector. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 1 to 6. A motion estimator for inputting a continuous video signal including a first frame and a second frame and estimating a motion between the first frame and the second frame; A motion vector generator configured to generate a motion vector between the first frame and the second frame from the estimated motion; An identification unit for identifying an object and a background of an interpolation frame to be inserted from the generated motion vector; And And an interpolation frame inserter which inserts the interpolation frame between the first frame and the second frame by warping the object and applying a block-based frame interpolation method to the background. The method of claim 8, The image signal is a frame interpolation apparatus using warping transform, characterized in that the RGB signal. The method of claim 8, And the video signal is a YCbCr or YPbPr signal. The method of claim 8, The estimator of the motion vector, A partition unit for dividing the predetermined pixel of the first frame and the second frame in block units; And And a motion estimator for estimating the motion in units of blocks between the first frame and the second frame. The method of claim 8, The identification unit, An operation unit for calculating a slope of a motion vector between a predetermined block in the first frame and a block in the second frame corresponding to the block; And And an identification unit for identifying an object and a background from the slope of the calculated motion vector.

Images