CN104301736B - A UHD frame rate up-conversion system that reduces memory bandwidth requirements - Google Patents

Abstract

The invention discloses an ultra-high-definition frame rate up-conversion system that reduces storage bandwidth requirements, wherein: an input module receives an ultra-high-definition video image stream, obtains a high-definition video image through down-sampling, and stores the original image and the down-sampled image into Off-chip memory; the motion estimation module reads the pixel data of the search window in high-definition resolution from the off-chip memory, and calculates the corresponding motion vector, which is scaled and used as the rough value of the motion vector in ultra-high-definition resolution; after the vector The processing module receives the rough value of the motion vector, further refines and smoothes it to obtain the final value of the motion vector; the interpolation module reads the reference pixel block corresponding to the final value of the motion vector from the off-chip memory, and calculates the interpolated pixel block. The present invention realizes ultra-high-definition frame rate up-conversion by only reading the search window at high-definition resolution, reduces off-chip storage bandwidth requirements, on-chip storage area and memory access bandwidth, and overcomes motion estimation caused by lowering resolution error.

Description

A UHD frame rate up-conversion system that reduces memory bandwidth requirements

technical field

The present invention relates to the field of video processing, in particular to a video processing system used for reducing storage bandwidth requirements in ultra-high-definition frame rate up-conversion.

Background technique

In recent years, people's subjective demand for high-quality visual enjoyment and the objective conditions of the rapid development of semiconductor technology have jointly promoted the vigorous development of the UHD TV industry. However, due to the limitation of the bandwidth of the current transmission system, UHD TV programs can only be transmitted at a lower frame rate. At the same time, the refresh rate of large-screen display devices has been greatly improved, and the video frame rate is lower than the screen refresh rate. This mismatch directly leads to image smearing, pauses, blurring, etc., and the display effect is not good. As an important video post-processing method, the video frame rate up-conversion technology can effectively increase the frame rate of the displayed video and improve the subjective quality of the image on a high refresh rate display.

The bandwidth of the frame rate up-conversion system is usually related to factors such as video resolution, frame rate, and search window size. For the UHD frame rate up-conversion system, the processing object is UHD video images, and the corresponding video resolution and search window are much larger than those of high-definition video. Therefore, the extremely high bandwidth requirement becomes the bottleneck of the ultra-high-definition frame rate up-conversion system . In the existing patented technology (such as publication number CN102067583A), some people use video compression methods to reduce bandwidth requirements. Although such methods can be used in ultra-high-definition frame rate up-conversion systems, they greatly increase the computational complexity of the entire system.

Contents of the invention

In view of the deficiencies in the prior art, the purpose of the present invention is to provide an ultra-high-definition video frame rate up-conversion system, which can greatly reduce storage bandwidth requirements while maintaining image quality.

In order to achieve the above object, the ultra-high-definition frame rate up-conversion system that reduces storage bandwidth requirements described in the present invention includes an input module, a motion estimation module, a vector post-processing module, and an interpolation module, wherein:

The input module is used to receive ultra-high-definition video image streams, and obtain high-definition video images after 1/4 down-sampling; simultaneously store original ultra-high-definition video images and down-sampled high-definition video images into off-chip memory;

The motion estimation module is used to read in the pixel data of the search window under the high-definition resolution from the off-chip memory, and obtain the motion vector under the resolution through the block matching algorithm. Coarse value of the lower motion vector;

The vector post-processing module is coupled to the motion estimation module, and further refines and smoothes the rough value of the motion vector obtained by motion estimation, thereby reducing the error introduced by vector scaling and improving the smoothness and density of the vector field;

The interpolation module is coupled to the vector post-processing module, reads the corresponding reference pixel block from the off-chip memory according to the final value of the motion vector obtained by the vector post-processing module, and calculates the interpolation pixel block.

Preferably, the input module includes digital audio and video interfaces such as HDMI, DP, Diiva and a downsampling sub-module, wherein the digital audio and video interface receives and transmits an ultra-high-definition video image stream, and the down-sampling sub-module performs 1/2 on the ultra-high-definition video image stream. 4 samples, and store the original ultra-high-definition video images and high-definition video images in external memory.

More preferably, the input module saves at least two consecutive frames of original ultra-high-definition video images and two frames of down-sampled high-definition video images to the off-chip memory.

Preferably, the motion estimation module includes an update control module, an on-chip cache, and a block matching calculation submodule, wherein the update control module controls to read in high-definition search window data from the off-chip memory, save it in the on-chip cache, and block match The calculation sub-module performs a matching operation within the range of the search window to obtain a motion vector at a high-definition resolution, and outputs the motion vector to the vector post-processing module after being scaled by 2 times.

Preferably, the vector post-processing module includes a two-stage filtering sub-module, and the filtering sub-module improves the smoothness and density of the vector field step by step according to the correlation of the space-time neighborhood vector field and by decomposing the size of the current matching block The vector post-processing module only performs filtering processing on the rough vector field obtained by the motion estimation module, and the processing process does not involve pixel information.

Preferably, the system only reads in the pixel data of the high-definition resolution search window, and the interpolation module directly reads in the pixels of the ultra-high-definition reference block from the off-chip memory for interpolation. The interpolation module efficiently reads the data of the off-chip ultra-high-definition video image, avoiding repeated data access caused by the search window.

Compared with the prior art, the present invention has the following beneficial effects:

1. Only the pixels of the high-definition resolution search window are read from the external memory, thereby reducing the bandwidth of the off-chip memory;

2. The pixels of the high-definition resolution search window are saved on-chip, thereby reducing the on-chip storage area and on-chip memory access bandwidth;

3. The vector post-processing module reduces the error caused by vector scaling, thus ensuring the quality of the image;

4. The interpolation module directly reads the ultra-high-definition video image data stored in the external memory for interpolation. Since the read-in data corresponds to the image blocks one by one, the minimum bandwidth for reading ultra-high-definition video image data is guaranteed.

To sum up, the present invention can greatly reduce the storage bandwidth requirement of the system, and has very important significance for solving the bandwidth bottleneck of the current ultra-high-definition frame rate up-conversion system.

Description of drawings

Other characteristics, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

Fig. 1 is a system block diagram of an embodiment of the present invention;

Fig. 2 is a flowchart of an embodiment of the present invention.

detailed description

The present invention will be described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

As shown in FIG. 1 , this embodiment provides a system for up-converting ultra-high-definition frame rates that reduces storage bandwidth requirements, including an input module, a motion estimation module, a vector post-processing module, and an interpolation module, wherein:

The input module is used to receive ultra-high-definition video image streams, and obtain high-definition resolution video images after 1/4 down-sampling; simultaneously store original ultra-high-definition video images and down-sampled high-definition video images into off-chip memory ;

The motion estimation module is used to read in the pixel data of the search window under the high-definition resolution from the off-chip memory, and obtain the motion vector under the resolution through the block matching algorithm. Coarse value of the lower motion vector;

The vector post-processing module, coupled to the motion estimation module, further refines and smoothes the motion vector coarse value obtained by motion estimation, thereby reducing the error introduced by vector scaling and improving the smoothness and density of the vector field ;

The interpolation module is coupled to the vector post-processing module, reads the corresponding reference pixel block from the off-chip memory according to the final motion vector, and calculates the interpolation pixel block.

In order to describe in detail a system for up-converting UHD frame rates that reduces storage bandwidth requirements in the present invention, the following example is used to illustrate, and FIG. 2 represents the following process:

Assuming that the system externally transmits a 30fps ultra-high-definition video image stream with the HDMI protocol, the input module receives it with the HDMI interface protocol, and obtains a high-definition video image stream through uniform down-sampling, and at the same time converts the original ultra-high-definition video image and the down-sampled high-definition video image Stored in the off-chip memory; according to the processing speed of the system, the external memory stores 8 frames of ultra-high-definition video images and 8 frames of high-definition video image data.

According to the search window [Sx,Sy] of motion estimation in UHD frame rate up-conversion, the corresponding search window under HD resolution is determined to be [Sx/2,Sy/2]. The update control module in the motion estimation module reads the pixels in the search window from off-chip and stores them in the on-chip cache; the block matching calculation sub-module reads the pixels in the search window from the on-chip cache 16*16 pixel blocks are matched and calculated (such as the absolute value difference criterion) to obtain the motion vector field at high-definition resolution; after scaling it by 2 times, the motion vector corresponding to the 32*32 pixel block of the ultra-high-definition video image is obtained, And output to the vector post-processing module.

After the vector post-processing module receives the vector field output by the motion estimation module, it performs a 2-level vector filtering operation, subdivides the 32*32 pixel block into 16*16, or even 8*8 pixel blocks, and finally obtains a 8* An 8-pixel block is the basic unit of the motion vector field; in an embodiment, the vector filtering operation may be Gaussian filtering, or other filtering methods based on neighborhood smoothness optimization.

After the interpolation module receives the vector field of the 8*8 pixel block, it directly reads the 8*8 pixel block of the reference frame corresponding to the vector from the off-chip memory, and performs interpolation calculation.

The following calculation shows that the example of the present invention has the advantage of reducing storage bandwidth requirements:

The bandwidth of the frame rate up-conversion system is usually related to factors such as video resolution, frame rate, and search window size. In the traditional UHD frame rate up-conversion system, in order to obtain better image quality, the search window is generally required to reach [±192, ±128]; a block group can be processed within the search window, and the block group generally consists of 4 lines *2 columns of 32*32 pixel blocks; when reading off-chip data, the horizontal direction can achieve zero waste through data multiplexing, so the reading bandwidth is related to the repeated data read rate in the vertical direction of the search window, which is 192*2/(32*4)=3.

Therefore, the off-chip bandwidth requirement of the traditional UHD frame rate up-conversion system is

3*3840*2160*2*30bps

Correspondingly, the off-chip bandwidth of the present invention is composed of motion estimation and interpolation bandwidth, which is

(3*1920*1080*2*30+3840*1080*2)bps

Apparently, the present invention can reduce the bandwidth requirement by about 41.7%. At the same time, the on-chip cache area is reduced by 75%, and the access bandwidth is reduced by 75%.

The present invention can realize ultra-high-definition frame rate up-conversion by only reading the search window at high-definition resolution, reducing off-chip storage bandwidth requirements, reducing on-chip storage area and memory access bandwidth, and at the same time overcoming the problem by vector post-processing Reduce the motion estimation error caused by the resolution to achieve a satisfactory visual effect.

The above is a preferred embodiment of the present invention, but it should be noted that the above-mentioned input interface protocol, search window size, block size, and vector filtering method are all implementations of the present invention, and are not limitations of the present invention. That is, according to design requirements, the control module of the present invention can adopt other input interface protocols, can set search windows of other specifications, can design blocks of other sizes, and can also use other vector filtering methods.

Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims (5)

1. An ultra-high-definition frame rate up-conversion system that reduces storage bandwidth requirements, characterized in that the system includes an input module, a motion estimation module, a vector post-processing module and an interpolation module, wherein: The input module is used to receive ultra-high-definition video image streams, obtain high-definition video images after 1/4 down-sampling, and store original ultra-high-definition video images and down-sampled high-definition video images into off-chip memory; The motion estimation module is used to read the pixels of the search window under high-definition resolution from the off-chip memory, and obtain the motion vector under the resolution through the block matching algorithm, and the motion vector is obtained as the motion vector under the ultra-high-definition resolution after scaling by 2 times. Coarse value of the motion vector; The vector post-processing module, coupled to the motion estimation module, further refines and smoothes the motion vector coarse value obtained by motion estimation, thereby reducing the error introduced by vector scaling and improving the smoothness and density of the vector field ; The vector post-processing module includes a two-stage filtering sub-module, and the filtering sub-module improves the smoothness and denseness of the vector field step by step according to the correlation of the space-time neighborhood vector field and by decomposing the size of the current matching block; The vector post-processing module only performs filtering processing on the rough vector field obtained by the motion estimation module, and the processing process does not involve pixel information; After the vector post-processing module receives the vector field output by the motion estimation module, it performs a 2-level vector filtering operation, subdivides the 32*32 pixel block into 16*16, or even 8*8 pixel blocks, and finally obtains a 8* 8-pixel block is the basic unit of the motion vector field; The interpolation module is coupled to the vector post-processing module, receives the final value of the motion vector obtained by the vector post-processing module, reads the reference pixel block corresponding to the final value of the motion vector from the off-chip memory, and calculates the interpolation pixel blocks. 2. A kind of ultra-high-definition frame rate up-conversion system that reduces storage bandwidth requirements as claimed in claim 1, wherein the input module includes a digital audio-video interface and a down-sampling sub-module, wherein the digital audio-video interface receives and transmits The ultra-high-definition video image stream, the down-sampling sub-module performs 1/4 sampling on the ultra-high-definition video image stream, and stores the original ultra-high-definition video image and high-definition video image into an off-chip memory. 3. A kind of ultra-high-definition frame rate up-conversion system that reduces storage bandwidth requirements as claimed in claim 2, wherein the input module at least converts two consecutive frames of original ultra-high-definition video images and two frames of down-sampled high-definition Video images are saved to off-chip memory. 4. A kind of ultra-high-definition frame rate up-conversion system that reduces storage bandwidth requirements as claimed in claim 1, wherein the motion estimation module includes an update control module, an on-chip cache and a block matching calculation submodule, wherein The update control module controls the high-definition search window data to be read from the off-chip memory, and is stored in the on-chip high-speed cache. The block matching calculation sub-module performs matching operations within the range of the search window, thereby obtaining motion vectors at high-definition resolutions. After the motion vector is scaled by 2 times, it is output to the vector post-processing module. 5. A kind of ultra-high-definition frame rate up-conversion system that reduces storage bandwidth requirements according to any one of claims 1-4, wherein the system only reads in the pixel data of the high-definition resolution search window, and the The interpolation module directly reads the pixels of the ultra-high-definition reference block from the off-chip memory for interpolation.