CN102665061A - Motion vector processing-based frame rate up-conversion method and device - Google Patents

Abstract

The embodiment of the present invention discloses a frame rate up-conversion method and device based on motion vector processing, wherein the method includes: performing scene detection on the input original video sequence to detect whether there is currently a scene change; When the transformation occurs, the original video sequence is processed by interpolation and the processed data is obtained; if no scene change is detected, the original video sequence is processed by extrapolation and the processed data is obtained. the data after processing by using symmetric motion estimation to obtain an initial motion vector field; performing motion vector processing on the motion vectors of the initial motion vector field to obtain a new motion vector field; the motion of the new motion vector field The vector is subjected to adaptive motion compensation processing to obtain the frame to be interpolated. The method and device implementing the embodiments of the present invention can reduce the computational complexity and the complexity of motion estimation, improve the quality of interpolated frames, and meet the requirements of real-time applications.

Description

A frame rate up-conversion method and device based on motion vector processing

technical field

The invention relates to the technical field of video processing, in particular to a frame rate up-conversion method based on motion vector processing and a device thereof.

Background technique

The large-scale application of digital video and multimedia information has brought about the diversity of display formats. Therefore, the conversion between these formats is very important. Frame rate up-conversion, also known as frame rate up-conversion (FRUC), is a technology that can convert low frame rate video sequences into high frame rate video sequences. The frame rate increase has a wide range of applications, such as making the motion of video content look more continuous and natural. The frame rate increase is mainly used to realize the conversion between two different frame rate video scanning formats. One application of FRUC is High Definition Television (HDTV). The resolution of HDTV can reach up to 1920*1080, and the frame rate can reach 60 frames/s. At present, the PAL (Phase Alternating Line) system used by my country and European TV has a frame rate of only 25 frames/s, and the NTSC (national television system committee) system used in North America and other countries is 30 frames/s. Therefore, to convert existing formats to HDTV, frame rate up-conversion mechanisms must be introduced. In addition, FRUC can also be applied to low bit rate video coding. Its idea is to appropriately reduce the frame rate of the original video at the encoding end, and then introduce the FRUC mechanism at the decoding end to increase the frame rate to the original frame rate.

Motion Compensated Interpolation (MCFI) is an algorithm with good effect and wide application in FRUC. This algorithm uses the correlation between adjacent frames, assumes that the motion vector of the middle frame is half of the motion vector of the two frames before and after, and then interpolates the middle frame.

However, when the video sequence moves violently or there is scene switching, the correlation of adjacent frames decreases, and this interpolation method will fail. At the same time, since MCFI is a block-based algorithm, when a segmented block is at the edge of the object and contains several moving objects, the motion vector of the block will be inaccurate, resulting in blurred edges of the interpolated image. Moreover, no matching pixel can be found for the pixel points in the occluded area and the exposed area, so the motion vector obtained by motion estimation is incorrect. In such an area, the MCFI method will interpolate an error block, which will cause a square effect. The use of MCFI will increase the computational complexity and the complexity of motion estimation, and reduce the quality of high-interpolation frames, which is not suitable for real-time applications.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides a frame rate up-conversion method and device based on motion vector processing, which can reduce the computational complexity and the complexity of motion estimation, and can improve the accuracy of interpolated frames. quality, meeting the requirements of real-time applications.

In order to solve the above problems, the present invention proposes a frame rate up-conversion method based on motion vector processing, the method comprising:

Perform scene detection on the input original video sequence to detect whether there is currently a scene change;

If it is detected that a scene change occurs, the original video sequence is processed in an interpolation manner to obtain processed data;

If no scene change is detected, the original video sequence is processed in an extrapolation manner to obtain processed data;

Using symmetric motion estimation to calculate the processed data to obtain the initial motion vector field;

performing motion vector processing on the motion vectors of the initial motion vector field to obtain a new motion vector field;

Perform adaptive motion compensation processing on the motion vectors of the new motion vector field to obtain frames to be inserted.

Preferably, the step of performing scene detection on the input original video sequence and detecting whether there is currently a scene change includes: using the frame difference method to detect the scene change, and according to the statistical absolute difference and the number of blocks whose SAD exceeds a specific threshold Determine whether a scene change has occurred.

Preferably, the step of calculating the processed data by using symmetric motion estimation, and obtaining the initial motion vector field includes: taking the coordinates of the block to be interpolated in the interpolated frame as a reference, symmetrically within the search range of the two reference frames Move the position of the corresponding matching block, and select the pair of blocks with the smallest difference as the best matching block to obtain the motion vector and form the initial motion vector field.

Preferably, the step of performing motion vector processing on the motion vectors of the initial motion vector field to obtain a new motion vector field includes: dividing the corresponding block whose motion vector MV is marked as L1 into sub-blocks; dividing the adjacent block as L2 The paired blocks are merged into large blocks to obtain variable blocks; motion estimation is performed on the variable blocks to obtain new motion vector fields.

Preferably, the step of performing adaptive motion compensation processing on the motion vector of the new motion vector field, and obtaining the frame to be interpolated includes: interpolating the current frame by using the previous frame or the next frame, or using a bidirectional frame The way of filling is to perform adaptive motion compensation processing on the motion vectors of the new motion vector field to obtain the frame to be inserted.

Correspondingly, an embodiment of the present invention also provides a frame rate up-conversion device based on motion vector processing, the device comprising:

The scene detection module is used to carry out scene detection to the input original video sequence, and detects whether there is currently a scene change;

A video sequence processing module, configured to process the original video sequence in an interpolation manner and obtain processed data if the scene detection module detects that a scene change occurs; if the scene detection module does not detect a scene When the transformation occurs, the original video sequence is processed in an extrapolation manner to obtain processed data;

A motion vector acquisition module is used to calculate the data processed by the video sequence processing module by using symmetric motion estimation to obtain an initial motion vector field;

A motion vector processing module, configured to perform motion vector processing on the motion vectors of the initial motion vector field obtained by the motion vector acquisition module to obtain a new motion vector field;

The adaptive interpolation filter is used to perform adaptive motion compensation processing on the motion vectors of the new motion vector field obtained by the motion vector processing module to obtain frames to be interpolated.

Preferably, the scene detection module is also used to detect scene changes using the frame difference method, and judge whether there is a scene change according to the statistical absolute difference and the number of blocks whose SAD exceeds a certain threshold.

Preferably, the motion vector acquisition module is also used to move the position of the corresponding matching block symmetrically within the search range of the two reference frames based on the coordinates of the block to be inserted in the interpolation frame, and select the pair with the smallest difference block as the best matching block to obtain the motion vector and form the initial motion vector field.

Preferably, the motion vector processing module is also used to divide the corresponding block marked as L1 of the motion vector MV into sub-blocks, and merge adjacent blocks marked as L2 into large blocks to obtain variable blocks. The variable block performs motion estimation to obtain a new motion vector field.

Preferably, the adaptive interpolation filter is also used to interpolate the current frame by using the previous frame or the next frame, or to adapt the motion vector of the new motion vector field by using bidirectional frame filling. Motion compensation processing to obtain frames to be interpolated.

In the embodiment of the present invention, different methods are adopted in the three different stages of the frame rate conversion to improve the subjective effect of the interpolated image. First, use scene detection to judge whether to use interpolation method or extrapolation method to process the original video sequence, and then update and check the initial motion vector field obtained by motion estimation to obtain the most reliable motion vector (MV) . Finally, the weight of the interpolation filter is adaptively selected to further eliminate local blurring. The scene detection adopts the frame difference method, which is simple and practical in the frame rate conversion application. Because it only needs to count the number of SADs exceeding a certain threshold, it can reduce the computational complexity and the complexity of motion estimation, and has little impact on the overall complexity. In addition, it can not only improve the quality of interpolated frames, but also Meet the requirements of real-time applications.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

1 is a schematic flow chart of a frame rate up-conversion method based on motion vector processing according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of scene detection according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of symmetric motion estimation according to an embodiment of the present invention;

Fig. 4 is a schematic structural composition diagram of a frame rate up-conversion device based on motion vector processing according to an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

In the embodiment of the present invention, the scene detection function is used to detect the input original video sequence to detect whether there is currently a scene change. If it is detected that the current scene changes, the original video sequence is processed by extrapolation; if no scene change is detected, the original video sequence is processed by interpolation. An initial motion vector field (MVF) is obtained by symmetric motion estimation (bidirectional motion estimation). This initial MVF can roughly reflect the motion trajectory of the object, but in the edge area, occlusion area, and exposure area of the object, some motion vectors (MV) may be unreliable, and the unreliable MV is verified and corrected to be compared. Reliable MV. Finally, an adaptive interpolation filter is selected for processing to obtain the output of the frame to be interpolated.

Fig. 1 is a schematic flow chart of a frame rate up-conversion method based on motion vector processing according to an embodiment of the present invention. As shown in Fig. 1, the method includes:

S101, perform scene detection on the input original video sequence, and detect whether there is currently a scene change; if it is detected that the scene change occurs, then execute S102; if not, if it is detected that the scene change occurs, then execute S103;

S102. Process the original video sequence in an interpolation manner and obtain processed data;

S103, processing the original video sequence by means of extrapolation and obtaining processed data;

S104, using symmetric motion estimation to calculate the processed data to obtain an initial motion vector field;

S105, performing motion vector processing on the motion vectors of the initial motion vector field to obtain a new motion vector field;

S106. Perform adaptive motion compensation processing on the motion vectors of the new motion vector field to obtain frames to be inserted. Among them, S101 includes:

The frame difference method is used to detect scene changes, and whether there is a scene change is judged according to the statistical absolute difference and the number of blocks whose SAD exceeds a certain threshold.

The scene detection of the embodiment of the present invention will be described in detail below with reference to FIG. 2 .

In general, there is a large correlation between adjacent frames of a video sequence. As shown in FIG. 2, there are great correlations between f _n-3 frames and f _n-1 frames, and between f _n+1 frames and f _n+3 frames. However, when there is a lens switching situation, there is usually a huge difference between the two frames at the switching point, such as f _n-1 frame and f _n+1 frame. At this time, if the interpolation method is still used to predict frame f _n by using frame f _n-1 and frame f _n+1 , effective frame interpolation cannot be obtained. In this case, if the extrapolation method is used, that is, frames f _n-3 and f _n-1 or frames f _n+1 and f _n+3 are used to extrapolate frames, a better effect can often be obtained.

In the specific implementation, the frame difference method is used to detect the transformation of the scene, and the number of blocks whose sum of absolute difference (Sum of Absolute Difference, SAD) exceeds a certain threshold (specific threshold) is used to judge whether there is a scene transformation . For every two adjacent frames of the input sequence, calculate the SAD value of the corresponding macroblock according to formula (1).

$\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Wherein, f _n-1 and f _n+1 are two adjacent frames, and B _ij is a macroblock in the frame.

The number of macroblocks whose SAD is greater than a certain threshold (SAD_num) is counted. Assume that the number of macroblocks with SAD greater than the threshold between f _n-3 and f _n-1 is SAD_num0, between f _n-1 and f _n+1 is SAD_num1, and between f _n+1 and f _n+3 is SAD_num2.

If SAD_num1<n, it means that the correlation of adjacent frames is very high, so the method of interpolation is adopted. If SAD_num1>n, it is determined that there is a scene switch between these two adjacent frames. At this time, an extrapolation method is used to estimate the intermediate frame f _n .

When extrapolating, if it satisfies

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; num</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; no</span>}\\ \mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; num</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; no</span>}\\ \mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; num</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; no</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Then use the previous two frames f _n-3 and f _n-1 to extrapolate the current frame f _n .

If the following formula is satisfied

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; num</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; no</span>}\\ \mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; num</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; no</span>}\\ \mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; num</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; no</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Then the current frame f _n is extrapolated using the following two frames f _n+1 and f _n+3 .

If SAD_num0, SAD_num1, and SAD_num2 are all greater than n, it means that there is continuous camera switching. At this time, neither interpolation nor extrapolation can get good results. In this case, the method of frame skipping is adopted. In this way, the quality of each image is guaranteed, and for the entire video, the picture is still very smooth.

Further, S104 includes: taking the coordinates of the block to be inserted in the interpolation frame as a reference, symmetrically moving the position of the corresponding matching block within the search range of the two reference frames, and selecting the pair of blocks with the smallest difference as the best matching block , to obtain the motion vector and form the initial motion vector field.

Traditional motion estimation is usually used in frame rate conversion. This method searches for the best matching block in the other frame image for two known adjacent frames based on a block of one frame image, so as to obtain the motion vector ( MV), and then take MV/2 as the motion vector of the intermediate frame. This approach often results in overlaps and gaps.

In the present invention, the symmetric motion estimation method is used for motion estimation, which can solve the existing problems of conventional motion estimation. Symmetrical motion estimation is based on the coordinates of the block to be inserted in the interpolation frame, and then moves the position of the corresponding matching block symmetrically within a certain search range in the two reference frames. According to the matching criterion of formula (4), the smallest difference is selected The pair of blocks is regarded as the best matching block. Fig. 3 is a schematic diagram of symmetric motion estimation. This kind of symmetrical motion estimation makes each block in the pin to be spliced "seamlessly", so it solves the loopholes and overlapping problems caused by direct motion estimation.

$\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; inter</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; MV</span>}<span\; class="notranslate">\; ]</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; MV</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

However, the MV obtained according to the matching criterion of formula (4) is the MV when the residual error is minimized in order to improve the coding efficiency in video coding, and it is not a real motion track.

In FRUC, in order to make the MV closer to the real motion trajectory, while considering the correlation within the frame, the matching criterion of motion estimation is revised, and formula (5) is used as the matching criterion.

SAD[B _ij ]=SAD_inter[B _ij ]+λSAD_intra[B _ij ] (5)

$\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; intra</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; kl</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; kl</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

SAD_intra in formula (6) represents the absolute difference between the upper horizontal boundary and the left vertical boundary of the current block and their adjacent boundaries, N is the width of a block, and the size of the block is N×N. λ is a weighting coefficient, which is an integer ranging from 0 to 16. It is an empirical value. For sequences with more violent movements, the value of λ should be relatively small, and for sequences with relatively gentle movements, the value of λ should be relatively large.

Further, S105 includes:

Divide the corresponding block whose motion vector MV is labeled L1 into sub-blocks;

Merge adjacent drinking blocks labeled L2 into large blocks to obtain variable blocks;

Perform motion estimation on the variable block to obtain a new motion vector field.

The motion vector processing process of the embodiment of the present invention will be described in detail below.

After symmetric motion estimation, an initial MVF can be obtained. Most of the MVs in this MVF are reliable, but there are still some unreliable MVs. Unreliable MVs will cause errors in corresponding blocks when inserting frames, which will seriously affect the subjective effect. Therefore, motion vector processing is a very important part in MCFI.

Generally speaking, a larger SAD means a larger matching error, and the MV corresponding to such a block is often unreliable. Therefore, SAD is a basic principle for judging whether the MV is reliable. But for some blocks, although the SAD is small, the MV is not reliable. The reason for this situation is that there are many similar or repeated regions in an image, and one may correspond to several best matching blocks. According to the minimum SAD, the reliability of the obtained MV is difficult to guarantee. Such MVs can be judged and processed based on the correlation of the MVs.

In the embodiment of the present invention, the reliability of the MV is classified based on the correlation between the SAD and the MV.

Suppose (w, h) is the position of the current block in all blocks of an image, w is the number of row blocks where the current block is located, h is the number of column blocks where the current block is located, (x _{w, h} , y _{w, h} ) is The motion vector of the current block.

The correlation coefficient between the current block and its surrounding 8 adjacent blocks is defined as

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; =</span>\frac{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 8</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\sqrt{{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}{\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 9</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\sqrt{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

The average correlation coefficient between the current block and surrounding adjacent blocks is defined as

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; avg</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 9</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; )</span>$

All the motion vectors of an image are divided into three categories, the first category is unreliable MV caused by excessive SAD, marked as L1; the second category is caused by the MV of the current block and the MV of the surrounding blocks are not very correlated The unreliable MVs are marked as L2; the remaining 3rd type are reliable MVs and are marked as L3.

The above-mentioned motion vector category is represented by formula (9) as follows:

SAD in formula (9) is the result obtained from formula (5).

After the motion vectors are classified, different methods are used to deal with the unreliable MVs marked L1 and L2 respectively. The processing of motion vectors is as follows:

1. Divide the corresponding block whose MV is marked as L1 into sub-blocks. Since such a block is often the edge of an object, it may contain two different objects, so the block is divided into small blocks, each containing an object, which can make the edge of the object clearer;

2. Merge the adjacent drinking blocks marked L2 into a large block. Since the MV of the L2 class often appears in the area where the surrounding motion vectors are smooth, and only the marked MV is irregular, it must be merged with the surrounding blocks into a large block;

3. After the previous division and merging, the size of the block is no longer the initial fixed size, but a variable block size. For these variable blocks, re-motion estimation is required to obtain a new MVF;

4. Reclassify the reliability of the updated MV;

5. Repeat the above 4 processes until the SAD of the block corresponding to the L1 category is smaller than a certain fixed threshold or the smallest segmented block reaches a block size of 2*2.

Further, S106 may include: performing adaptive motion compensation processing on the motion vector of the new motion vector field by using the previous frame or the subsequent frame to interpolate the current frame, or using bidirectional frame filling to obtain the frame to be interpolated .

The adaptive motion compensation method adopted by the embodiment of the present invention will be described below.

MCFI is actually a linear filtering process, and the interpolation result can be expressed by the following formula:

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; f</span>}}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&lambda;\; f</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; MV</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; MV</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$

是预测出来的帧。S表示一个分割块，MV是该块对应的运动矢量。权值λ一般取0.5。但是这种固定权值的滤波器往往会让局部模糊。本发明按以下公式选择λ：where f _n-1 and f _n+1 are known two adjacent frames,

is the predicted frame. S represents a partition block, and MV is the motion vector corresponding to the block. The weight λ is generally 0.5. However, this fixed-weight filter tends to blur the local area. The present invention selects λ according to the following formula:

If the SAD of the current block exceeds a certain threshold, the current frame is interpolated using the forward frame or the subsequent frame, otherwise, the two-way frame is used for filling.

In the embodiment of the present invention, different methods are adopted in the three different stages of the frame rate conversion to improve the subjective effect of the interpolated image. First, use scene detection to judge whether to use interpolation method or extrapolation method to process the original video sequence, and then update and check the initial motion vector field obtained by motion estimation to obtain the most reliable motion vector (MV) . Finally, the weight of the interpolation filter is adaptively selected to further eliminate local blurring. Scene detection adopts frame difference method, which is simple and practical in frame rate conversion applications. Because it only needs to count the number of SADs exceeding a certain threshold, it can reduce the computational complexity and the complexity of motion estimation, and has little impact on the overall complexity. In addition, it can not only improve the quality of interpolated frames, but also Meet the requirements of real-time applications.

Fig. 4 is a schematic structural composition diagram of a frame rate up-conversion device based on motion vector processing according to an embodiment of the present invention. As shown in Fig. 4 , the device includes:

Scene detection module 40, is used for carrying out scene detection to the original video sequence of input, detects whether there is scene change to take place at present;

The video sequence processing module 41 is used for if the scene detection module 40 detects that the scene change occurs, then adopts the interpolation mode to process the original video sequence and obtains the processed data; if the scene detection module 40 does not detect that the scene change occurs , then use extrapolation to process the original video sequence and obtain the processed data;

The motion vector acquisition module 42 is used to calculate the data processed by the video sequence processing module 41 by using symmetric motion estimation to obtain an initial motion vector field;

A motion vector processing module 43, configured to perform motion vector processing on the motion vector of the initial motion vector field obtained by the motion vector acquisition module 42 to obtain a new motion vector field;

The adaptive interpolation filter 44 is configured to perform adaptive motion compensation processing on the motion vectors of the new motion vector field obtained by the motion vector processing module 43 to obtain frames to be interpolated.

In a specific implementation, the scene detection module 40 is also used to detect a scene change using the frame difference method, and judge whether there is a scene change according to the statistical absolute difference and the number of blocks whose SAD exceeds a certain threshold.

Further, the motion vector acquisition module 42 is also used for taking the coordinates of the block to be inserted in the interpolation frame as a reference, symmetrically moving the position of the corresponding matching block within the search range of the two reference frames, and selecting the pair of blocks with the smallest difference As the best matching block, motion vectors are obtained to form an initial motion vector field.

Further, the motion vector processing module 43 is also used to divide the corresponding block marked as L1 of the motion vector MV into sub-blocks, and merge adjacent corresponding blocks marked as L2 into large blocks to obtain variable blocks, which can be Change the block for motion estimation and obtain a new motion vector field.

Further, the adaptive interpolation filter 44 is also used to interpolate the current frame by using the previous frame or the next frame, or to perform adaptive motion compensation processing on the motion vector of the new motion vector field by using the bidirectional frame to fill , to obtain the frame to be inserted.

For the implementation process and principle of each module function of the frame rate up-conversion device based on motion vector processing in the embodiment of the present invention, please refer to the corresponding process description in the embodiment of the frame rate up-conversion method based on motion vector processing in the present invention, here No longer.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: Read Only Memory (ROM, Read Only Memory), Random Access Memory (RAM, Random Access Memory), disk or CD, etc.

In addition, the frame rate up-conversion method and device based on motion vector processing provided by the embodiments of the present invention have been introduced in detail above. In this paper, specific examples are used to illustrate the principle and implementation of the present invention. The above embodiments The description is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and scope of application. In summary , the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. A frame rate up-conversion method based on motion vector processing, characterized in that the method comprises: Perform scene detection on the input original video sequence to detect whether there is currently a scene change; If it is detected that a scene change occurs, the original video sequence is processed in an interpolation manner to obtain processed data; If no scene change is detected, the original video sequence is processed in an extrapolation manner to obtain processed data; Using symmetric motion estimation to calculate the processed data to obtain the initial motion vector field; performing motion vector processing on the motion vectors of the initial motion vector field to obtain a new motion vector field; Perform adaptive motion compensation processing on the motion vectors of the new motion vector field to obtain frames to be inserted. 2. the frame rate up-conversion method based on motion vector processing as claimed in claim 1, is characterized in that, described original video sequence of input is carried out scene detection, detects whether the step that current scene change takes place comprises: The frame difference method is used to detect scene changes, and whether there is a scene change is judged according to the statistical absolute difference and the number of blocks whose SAD exceeds a certain threshold. 3. The frame rate up-conversion method based on motion vector processing as claimed in claim 1 or 2, wherein said symmetric motion estimation is used to calculate the processed data, and the step of obtaining an initial motion vector field comprises: Based on the coordinates of the block to be inserted in the interpolation frame, the position of the corresponding matching block is moved symmetrically within the search range of the two reference frames, and the pair of blocks with the smallest difference is selected as the best matching block to obtain the motion vector. Form the initial motion vector field. 4. The frame rate up-conversion method based on motion vector processing as claimed in claim 3, wherein the step of carrying out motion vector processing to the motion vector of the initial motion vector field to obtain a new motion vector field comprises: Divide the corresponding block whose motion vector MV is labeled L1 into sub-blocks; Merge adjacent drinking blocks labeled L2 into large blocks to obtain variable blocks; Motion estimation is performed on the variable block to obtain a new motion vector field. 5. The frame rate up-conversion method based on motion vector processing as claimed in claim 4, wherein the step of performing adaptive motion compensation processing on the motion vector of the new motion vector field to obtain the frame to be inserted comprises : performing adaptive motion compensation on the motion vector of the new motion vector field by using the previous frame or the next frame to interpolate the current frame, or using bidirectional frames to fill in, to obtain the frame to be interpolated. 6. A frame rate up conversion device based on motion vector processing, characterized in that the device comprises: The scene detection module is used to carry out scene detection to the input original video sequence, and detects whether there is currently a scene change; A video sequence processing module, configured to process the original video sequence in an interpolation manner and obtain processed data if the scene detection module detects that a scene change occurs; if the scene detection module does not detect a scene When the transformation occurs, the original video sequence is processed in an extrapolation manner to obtain processed data; A motion vector acquisition module is used to calculate the data processed by the video sequence processing module by using symmetric motion estimation to obtain an initial motion vector field; A motion vector processing module, configured to perform motion vector processing on the motion vectors of the initial motion vector field obtained by the motion vector acquisition module to obtain a new motion vector field; The adaptive interpolation filter is used to perform adaptive motion compensation processing on the motion vectors of the new motion vector field obtained by the motion vector processing module to obtain frames to be interpolated. 7. The frame rate up-converting device based on motion vector processing as claimed in claim 6, wherein the scene detection module is also used to detect scene changes using the frame difference method, and exceed a specific threshold according to the statistical absolute difference and SAD The number of blocks of value determines whether there is a scene change. 8. The frame rate up-converting device based on motion vector processing as claimed in claim 6 or 7, wherein the motion vector acquisition module is also used to take the coordinates of the block to be inserted in the interpolation frame as a reference, in two The position of the corresponding matching block is moved symmetrically within the search range in the reference frame, and the pair of blocks with the smallest difference is selected as the best matching block to obtain a motion vector and form an initial motion vector field. 9. The frame rate up-conversion device based on motion vector processing as claimed in claim 8, wherein the motion vector processing module is also used to divide the corresponding block whose motion vector MV is marked as L1 into sub-blocks, and Adjacent blocks labeled L2 are merged into large blocks to obtain variable blocks, and motion estimation is performed on the variable blocks to obtain a new motion vector field. 10. The frame rate up-conversion device based on motion vector processing according to claim 9, wherein the adaptive interpolation filter is also used to interpolate the current frame by using a forward frame or a subsequent frame, Or perform adaptive motion compensation processing on the motion vectors of the new motion vector field in a manner of padding with bidirectional frames to obtain frames to be inserted.

Images