CN107027029B - High-performance video coding improvement method based on frame rate conversion - Google Patents

Abstract

The invention provides a high-performance video coding method combined with frame rate conversion aiming at the HEVC coding standard. The method mainly comprises the steps of carrying out self-adaptive frame extraction on an original video at an encoding end to reduce the transmission data volume; the low frame rate video carries out motion vector estimation on a lost frame by HEVC coding and combining a reference frame motion vector extracted from a code stream and intra-frame block information, and the lost frame is reconstructed by combining the motion vector at a decoding end and an improved bidirectional motion compensation inserted frame based on block coverage. The invention reduces the data amount of transmission and saves the transmission bandwidth due to the self-adaptive frame extraction; the process of motion estimation for the decoded video is saved, so that a large amount of time is saved; the improved block coverage based bi-directional motion compensated frame interpolation guarantees the quality of the reconstructed video. The experimental result shows that compared with HEVC standard coding, the algorithm saves the code rate by about 15% and saves the coding time by about 40% on the premise of ensuring that the PSNR of the reconstructed video is close to the consistency at the medium and low code rate end.

Description

An Improved Method for High Performance Video Coding Based on Frame Rate Conversion

technical field

The invention relates to the problem of reducing the video coding code rate in the field of image communication, and mainly relates to a method for reducing the coding code rate by combining the high-performance video coding standard HEVC and video frame rate conversion.

Background technique

With the popularization and development of high-definition digital television (HDTV) and high-end multimedia systems, people have higher and higher requirements for the subjective enjoyment of video, which puts forward higher requirements for the quality of video program sources. The higher the data volume of the video, the greater the bandwidth required for transmission. At the same time, video coding standards are constantly being updated. After the development and improvement of video coding standards such as MPEG and H.264/AVC, a new generation of video coding standards, HEVC, has also been released. And compared with H.264/AVC, the HEVC encoding standard saves nearly 50% of the encoding bit rate, but for high-definition video, the amount of transmitted data is still huge. Therefore, under the limited bandwidth, how to reduce the amount of transmitted data, transmit the encoded code stream in real time, and reduce the transmission bandwidth as much as possible on the premise of ensuring the quality of the decoded video is still a hot topic in the field of video coding.

Motion information during video compression is crucial for video reconstruction. The present invention improves the acquisition of motion information and the post-processing of motion information in the HEVC improved video compression coding framework based on frame rate conversion. In the standard HEVC coding framework, there is no in-depth research on the information recovery of lost frames. Therefore, it is of great significance to find an efficient video interpolation frame recovery method for improving the compression efficiency. At present, the most widely used methods of lost frame recovery in academia are re-motion estimation of the decoded low frame rate video to obtain the motion vector used in the frame insertion process, and it also has certain effects.

The frame rate conversion method, as a technique to improve the compression rate, can effectively reduce the coding rate when combined with the standard HEVC coding. Sung-Hee Lee et al. proposed to reduce the frame rate of the video at the encoding end to reduce the number of encoded frames, reduce the transmission stream data, and then insert frames at the decoding end to restore the video to the original frame rate. There are two traditional frame rate enhancement techniques: non-motion-compensated frame insertion and motion-compensated frame insertion. The motion information of the object in the original video is not considered in the non-motion compensation interpolation process, but the pixel value of the frame to be inserted is restored by the pixel value of the corresponding position of the two reference frames of the previous frame and the current frame, such as the frame repetition method and the frame average method. ; Motion compensation frame interpolation considers the motion of objects in the scene, and reconstructs the frame to be inserted by combining the motion vector information and the pixel values in the reference frame, and this method can effectively reduce the jitter introduced by the frame repetition method and the introduction of the frame averaging method. ambiguity. Choi et al. specially proposed Bidirectional Motion Compensation Interpolation (BMCI) suitable for frame rate up-conversion technology. According to the source of motion vector information used in the frame insertion process, motion compensation frame insertion can be divided into two types. One is Based on the compression domain, this method extracts the motion vector information of adjacent frames from the encoded code stream for motion compensation frame insertion, which saves the process of re-motion estimation of the encoded video to obtain the motion vector. The amount of calculation and time is greatly reduced; the second type is based on the pixel domain, that is, the motion estimation of the decoded video is performed again, and then the obtained motion vector information is used for frame interpolation reconstruction. Compared with the former, the second method is due to the introduction of The motion estimation process increases the coding complexity, so it will consume a lot of time. Although the effect of the motion-compensated frame interpolation method is obviously better than that of the non-motion-compensated method, the video reconstructed by the traditional motion-compensated frame interpolation method will have holes and block effects.

SUMMARY OF THE INVENTION

In order to solve the problem of holes and block effects in the reconstructed frames in the traditional frame rate boosting process, improve the performance of video compression coding and improve the quality of the reconstructed video, the present invention proposes an improved method for high-performance video compression coding combined with frame rate conversion, which ensures that Under the premise of video quality, the video coding rate is reduced, and it has a good application prospect in the real-time transmission system with limited bandwidth.

The basic idea of the present invention is to perform adaptive frame extraction on the original video at the encoding end, encode and decode the low frame rate video, extract motion vector information and special intra-frame block information in the HEVC encoding process from the code stream information, and At the decoding end, the frame rate of the decoded video is up-sampled and restored to the original frame rate in combination with the motion vector, so as to achieve the purpose of reducing the number of encoded frames and the encoding bit rate.

According to the HEVC video coding standard, the present invention proposes an improved HEVC video compression coding framework combined with frame rate conversion. It mainly includes HEVC standard video encoding for the low frame rate video after adaptive frame extraction, and extracts the relevant motion vector information and the corresponding intra-frame block information from the transmitted code stream information. Due to the special block mode of HEVC , the motion vector information is stored in the code stream information in the form of PU blocks, that is, the pixel positions in the same PU block have the same motion vector. In the IPPP inter-frame configuration encoding mode, the encoder will perform intra-frame prediction at the same time. and inter prediction to obtain residual information. Although the prediction mode of most coding blocks is inter prediction, when the residual information obtained by inter prediction is not accurate enough, the encoder will automatically use the residual obtained by intra prediction. Therefore, if the block information extracted from the code stream information corresponds to an intra-coded block, the motion vector is 0, which will cause holes or block effects if it is used for frame insertion. Estimate the motion information of these intra-frame blocks to ensure that each coding block in the frame has corresponding motion vector information. Filling is performed; smooth filtering is performed on the obtained motion vector, and the decoded video is interpolated by bidirectional motion compensation based on block coverage in combination with the processed motion vector information to restore the original frame rate. Specifically, it mainly includes the following steps:

(1) Adaptive frame extraction is performed on the original frame rate video before encoding, and the maximum number of frames extracted between two adjacent frames is set to 2, and the minimum number of frames to be extracted is 0;

(2) The video after encoding the frame extraction is transmitted to the decoding end, and the relevant motion vector information and the special intra-frame block information in HEVC are extracted from the transmitted code stream information;

(3) carrying out bidirectional motion estimation to the lost frame by the motion vector information obtained in step (2) to obtain an initial motion vector;

(4) The position of the intra-frame coding block in a frame is determined in combination with the intra-frame block information obtained from the code stream information. Since the motion information of the intra-frame coding block cannot be used for the interpolation frame reconstruction, the present invention is suitable for the intra-frame block motion. The vector information is replaced by the weighted average value of the motion vectors of the adjacent blocks in the frame and the blocks at the same position in the adjacent two reference frames, so as to obtain the corrected motion vector of the lost frame;

(5) Through the motion estimation of the motion vector of the intra-frame block, each coding block has its own motion vector information, and each 4*4 block is subjected to motion vector smoothing processing;

(6) In combination with the final motion vector information of the lost frame obtained above, the low frame rate decoded video is reconstructed by inserting a frame by a bidirectional motion compensation frame interpolation method based on block coverage;

In the above technical solution of the present invention, the adaptive frame extraction method judges that at most two frames can be dropped continuously between two adjacent reference frames, and the minimum number of dropped frames is 0. The specific criterion for judging whether a frame is a dropable frame is: :

Among them, f _i-1 (i, j) represents the pixel value of the previous frame at position (i, j), f _i (i, j) represents the pixel value of the current frame at position (i, j), m and n represent the width and height of the video respectively, MSAD _i represents the average value of the absolute error sum of the pixel values at the corresponding positions of the adjacent two frames, and R represents the relative change rate of the adjacent frames. Set the threshold of the maximum number of consecutive frames drawn between two adjacent frames to 2, the MSAD threshold to 35, and the R threshold to 3.

In the above-mentioned technical scheme of the present invention, the traditional bidirectional motion compensation frame insertion calculation method is:

则表示由参考帧运动矢量信息计算得到的待插帧相对于前后参考帧的初始运动矢量。where F _t (x, y) represents the reconstructed value of the frame to be inserted at position (x, y), F _t-1 and F _t+1 sub-tables represent the previous reference frame and the current reference frame, (V _x , V _y ) is the motion vector between the front reference frame and the back reference frame of the frame to be inserted obtained in the code stream,

It represents the initial motion vector of the frame to be inserted relative to the reference frame before and after, calculated from the motion vector information of the reference frame.

In the above-mentioned technical scheme of the present invention, the concrete method for revising the motion vector of the intra-frame coding block in conjunction with the intra-frame block information is:

Among them, MV _intra represents the estimated value of the motion information of the intra-frame block, MV _prev represents the motion vector at the same position as the intra-block in the previous reference frame, and MV _curr represents the motion at the same position as the intra-block in the next reference frame vector, MV _i represents the motion vector of the block adjacent to the intra-frame block, _wi represents the compensation weight, n is the number of adjacent blocks, and the setting rule for the compensation weight _wi is: if the size of the adjacent block is equal to If the intra-frame blocks are the same, the weight is 1. Otherwise, it is determined according to the difference between the division depths of adjacent blocks and intra-frame blocks in HEVC. If the difference is 1, the weight is 0.5. If the difference is 2, the weight is 0.25. analogy. If there are intra-frame blocks in adjacent blocks, it is not included, and if all surrounding blocks are intra-frame blocks, the motion vector of the intra-frame block is set to 0.

In the above technical solution of the present invention, the method for smoothing and filtering the corrected motion vector is:

In the above-mentioned technical scheme of the present invention, the described method for performing frame interpolation and reconstruction on low frame rate decoded video through the bidirectional motion compensation frame interpolation method based on block coverage is as follows:

表示待插块所覆盖参考块的运动矢量。Among them, F(x, y) represents the block reconstruction value of the to-be-interpolated frame at position (x, y), n represents the number of reference blocks covered by the to-be-interpolated block,

Represents the motion vector of the reference block covered by the block to be inserted.

The above-mentioned method according to the present invention can formulate an HEVC video codec that performs the above-mentioned improved method for high-performance video coding based on frame rate conversion.

The present invention is completed based on the analysis of the following ideas:

As an effective means that can improve video coding efficiency, frame rate conversion technology can be combined with standard video coding methods. The specific method of adaptive frame extraction in the present invention is: the basis for judging whether a frame is a discardable frame is the average absolute value of the two frames. Error value (MSAD _i ) and relative rate of change (R), when the values of MSAD _i and R are both greater than the given threshold, it is judged that the i-th frame is a scene jump frame, then the frame is reserved and no frame sampling is performed Processing; otherwise, the frame is regarded as a discardable frame, and after judging one frame, it continues to judge whether the next adjacent frame is a discardable frame, so there is a problem, if several consecutive frames are discardable. If the frame is discarded, the number of discarded frames is too large, which has a great influence on the accuracy of the motion information obtained during video reconstruction, so the video reconstruction quality is not good. The maximum number of frames drawn between frames also sets a threshold. Through experimental comparison, when the maximum number of consecutive frames drawn between two adjacent frames is set to 2 frames, the MSAD threshold is 35, and the R threshold is 3, the effect is better.

The low frame rate video after adaptive frame extraction is encoded by the HEVC encoding framework, and the motion vector information is obtained from the encoded code stream to initially estimate the motion vector of the lost frame, and the initial motion vector is corrected for Bidirectional motion-compensated frame insertion to improve the video quality after frame insertion.

Due to the special block mode of HEVC, the motion vector information is stored in the code stream information in the form of PU blocks, that is, the pixel positions in the same PU block have the same motion vector. In the IPPP inter-frame configuration encoding mode, the encoding The encoder will perform intra-frame prediction and inter-frame prediction at the same time to obtain residual information. Although the prediction mode of most coding blocks is inter-frame prediction, when the residual information obtained by inter-frame prediction is not accurate enough, the encoder will automatically use frame The residual obtained by intra-prediction is used for coding. Therefore, if the block information extracted from the code stream information corresponds to an intra-frame coding block, the motion vector is 0. If it is used for frame insertion, it will cause holes or block effects. Therefore, The motion information of these intra-frame blocks is estimated before inter-frame restoration. In the video, there is not only temporal correlation between sequence frames, but also spatial correlation within each frame, so the motion information of adjacent blocks of the block to be interpolated during frame interpolation can also be used as a reference. Therefore, for intra-frame coding blocks, The motion vector of the block to be inserted is estimated by borrowing the motion information of the adjacent blocks in the spatial domain of the block to be inserted and the block at the same position in the reference frame.

After estimating the motion vector of the intra-frame block, each coding block has its own motion information. Before performing interpolation and restoration on each 4*4 block in the frame to be interpolated, take the eight adjacent blocks of the block and its own motion vector. The average value is used as the final motion vector in the interpolation frame restoration process. If the block to be interpolated is located in the boundary area, the motion vector of the reference block in the boundary is taken for reference. For example, if we need to interpolate the block in the lower right corner of the image at this time, then The corresponding position block is selected in the reference frame, and the left block, the upper side block and the upper left three blocks are used for reference. Boundary block interpolation in other corners and so on.

The process of interpolating and reconstructing low frame rate decoded video through the bidirectional motion compensation interpolating method based on block coverage is as follows: Since most of the information in HEVC, such as motion vectors, are stored and transmitted in the format of different divided blocks, in the video The reconstruction process is also completed in the way of pixel value reconstruction of blocks. Therefore, on the basis of bidirectional frame interpolation, the correlation in the spatial domain of the video frame is fully considered, and the interpolation block is divided into regions. The values are interpolated using different methods, that is, a block to be interpolated in a lost frame covers a reference block containing different motion information. When this happens, the present invention uses an improved block coverage-based bidirectional motion-compensated frame interpolation method.

Compared with the standard HEVC video coding method, the method of the present invention first adopts the adaptive frame extraction method to process the sequence to be coded, and then extracts the relevant motion vector information from the code stream information to estimate the motion vector of the lost frame, and combines the code stream. The intra-frame block information in the frame is used to correct the motion vector of the lost frame, and then the motion vector for the final video reconstruction is obtained by smoothing the motion vector of each 4*4 reconstructed block. Interpolation method for video reconstruction. Through the method of the present invention, under the premise of ensuring the quality of the reconstructed video in the middle and low bit rate segments, not only the amount of code stream data can be reduced, the transmission bandwidth can be saved, but also the coding time can be greatly improved because the re-motion estimation process for lost frames is omitted. efficiency.

Description of drawings

FIG. 1 is a flowchart of an improved method for high-performance video coding based on frame rate conversion according to the present invention.

FIG. 2 is a schematic diagram of adaptive frame extraction in the present invention.

FIG. 3 is a schematic diagram of a frame insertion method for bidirectional motion compensation in the present invention.

FIG. 4 is a schematic diagram of a method for smoothing an initial motion vector in the present invention.

FIG. 5 is a schematic diagram of motion compensation interpolation based on block coverage in the present invention.

Figures 6 to 9 are the comparison diagrams of the rate-distortion curves between the algorithm of the present invention and the HM16.0 standard encoding method, wherein Figures 6 to 9 are the sequences PartyScene, FourPeople, ParkScene, and PeopleOnStreet at QPs of 22, 27, 32, 37, and 42 respectively. , 44, 46, 48, 50 comparison of rate-distortion curves.

Detailed ways

The present invention will be described in further detail below in conjunction with examples. It is necessary to point out that the following examples are only used to further illustrate the present invention and should not be construed as limiting the protection scope of the present invention. The specific implementation of some non-essential improvements and adjustments to the present invention should still fall within the protection scope of the present invention.

The high-performance video improved coding algorithm based on frame rate conversion of the present invention is compared with the coding method of the HEVC standard test model HM16.0 as follows:

1. Open the standard HM16.0 test model, the configuration file is lowdelay_P_main, encode and decode the standard video test sequence when the quantization parameters are 22, 27, 32, 37, 42, 46, 48, and 50, and record the standard HEVC Bit rate and reconstructed video PSNR during video encoding;

2. The method of the present invention will be compared with the bit rate of the reference software HM16.0 of the HEVC video coding standard and the PSNR of the reconstructed video in the above 1. Open the programs of the present invention and the standard method at the same time, set the same configuration file, down-sampling the video sequence through the adaptive frame extraction method before encoding, and the standard algorithm encodes and decodes the original frame rate video sequence. The algorithm of the present invention It encodes and decodes the low frame rate video after the frame rate down-sampling, and extracts the relevant motion information and intra-frame block information from the code stream information. Comparative analysis of three video coding performances: peak signal-to-noise ratio (PSNR), bit rate, and encoding time (where PSNR reflects the objective quality of the reconstructed video, the bit rate reflects the amount of data transmission, and the encoding time reflects the time efficiency of video encoding processing). , the gap of comparative performance is evaluated by the following two indicators:

Among them, Δ code rate represents the percentage of the code rate difference between the algorithm of the present invention and the standard algorithm relative to the standard code rate, that is, the percentage of reduced code rate, and ΔT represents the difference between the algorithm of the present invention and the standard codec time relative to the standard code. percentage of time

3. The encoding objects are standard HEVC test videos, their names, resolutions and frame rates are: PartyScene (832x480, 50 frames/sec), FourPeople (1280x720, 60 frames/sec), ParkScene (1920x1080, 24 frames/sec) seconds), PeopleOnStreet (2560x1600, 30 fps).

4. Use the HM16.0 standard method to encode the original frame rate video sequence in HEVC mode;

5. Use the method of the present invention to encode the same sequence of low frame rate videos, and extract relevant motion vector information and intra-frame block information from the code stream;

6. Using the method of the present invention to perform frame rate upsampling on the decoded low frame rate video;

7. Record the relevant experimental data such as code rate, PSNR, time in the encoding and decoding process of the two methods respectively; wherein the method of the present invention saves the code rate relative to the standard method under the condition that the objective quality of the reconstructed video (PSNR) is close to the same as in Table 1 , the comparison between the present invention and the standard algorithm in terms of time efficiency is shown in Table 2. Statistical results show that, compared with the HEVC standard method, the method of the present invention saves about 15% of the code rate on the premise of ensuring that the PSNR of the reconstructed video is close to the same in the medium and low code rate segment; and also saves nearly 40% in encoding time efficiency.

Table 1 Comparison of the coding rate between the algorithm of the present invention and the HM16.0 standard algorithm

Table 2 Comparison of encoding and decoding time efficiency between the algorithm of the present invention and the HM16.0 standard algorithm

Claims (8)

1. A high-performance video coding improvement method based on frame rate conversion is mainly characterized by comprising the following process steps:

(1) before coding, carrying out self-adaptive frame extraction on an original frame rate video, and setting the maximum frame extraction number between two adjacent reference frames to be 2 and the minimum frame extraction number to be 0;

(2) coding the video after frame extraction, decoding by a decoding end, and extracting related motion vectors and special intra-frame block information in HEVC from transmitted code stream information;

(3) performing bidirectional motion estimation on the lost frame through the motion vector obtained in the step (2) to obtain an initial motion vector;

(4) determining the position of an intra-frame coding block in a frame by combining intra-frame block information acquired from code stream information, wherein the motion information of the intra-frame coding block cannot be used for frame interpolation reconstruction, so that the motion vector information of the intra-frame coding block is replaced by the weighted average value of the motion vectors of adjacent blocks of the intra-frame coding block and blocks at the same position in two adjacent reference frames to obtain a lost frame correction motion vector;

(5) by motion estimation of motion vectors of the intra blocks, each coding block has own motion vector information, and motion vector smoothing processing is carried out on each 4 x4 block;

(6) and combining the final motion vector information of the lost frame obtained in the previous step, and performing interpolation reconstruction on the low frame rate decoded video by a bidirectional motion compensation interpolation method based on block coverage.

2. The method as claimed in claim 1, wherein the step (1) of adaptively extracting frames from the original frame rate video, and the specific method for determining whether a frame is a discardable frame is as follows:

wherein f is_i-1(i, j) represents the pixel value of the previous reference frame at position (i, j), f_i(i, j) represents the pixel value of the current reference frame at position (i, j), m and n represent the width and height of the video, respectively, MSAD_iThen representing the average value of the absolute errors of the pixels at the corresponding positions of two adjacent frames, R representing the relative change rate of the adjacent frames, if MSAD_iIf the value of R is greater than the given threshold value, the ith frame is judged to be a scene jump frame, and the frame is reserved without frame extraction; and otherwise, the frame is regarded as an extractable frame, and after one frame is judged, whether the next adjacent frame is an extractable frame or not is continuously judged.

3. The method of claim 2 wherein the MSAD is set during adaptive frame decimation_iThe threshold is 35, the R threshold is 3, the maximum continuous frame number of two adjacent reference frames is 2, and the minimum is 0.

4. The improved frame rate transform-based high performance video coding method as claimed in any of claims 1 to 3, wherein the motion estimation method for the lost frame is:

wherein (V)_x,V_y) For the motion vector between the front reference frame and the back reference frame of the frame to be inserted obtained in the code stream,

it represents the initial motion vector of the frame to be interpolated relative to the previous and next reference frames calculated from the reference frame motion vector information.

5. The improved frame rate transform-based high performance video coding method as claimed in claim 4, wherein the method for filling the intra block motion vector in combination with the intra block information comprises:

MV_intrarepresenting intra block motion information estimate, MV_prevAnd MV_currRespectively representing motion vector information, MV, at the same position as the intra block in the previous and subsequent reference frames_iRepresenting block motion vectors, w, adjacent to the intra block_iRepresents the compensation weight, and n is the number of adjacent blocks.

6. The method as claimed in claim 5, wherein the compensation weight w is_iThe setting rule of (1) is: if the size of the adjacent block is the same as that of the intra-frame block, the weight is 1, otherwise, the difference is determined according to the difference value of the division depths of the adjacent block and the intra-frame block in HEVC, the weight is 0.5 if the difference is 1, the difference is 2, the weight is 0.25, and the like; if there is an intra block in the neighboring blocks, the intra block motion vector is set to 0 if all the surrounding blocks are intra blocks.

7. The improved frame rate conversion based high performance video coding method of claim 6, wherein a smoothing filtering process is performed for each 4 x4 block of motion vector information before the lost frame is recovered.

8. The improved method of high-performance video coding based on frame rate conversion as claimed in claim 7, wherein the lost frame is recovered by an improved bidirectional motion compensation method based on block covering, the specific method is:

wherein F (x, y) representsBlock reconstruction value of frame to be interpolated at position (x, y), n represents number of reference blocks covered by block to be interpolated, F_t-1(x, y) denotes a frame of time preceding the frame to be interpolated, F_t+1(x, y) represents a frame at a time subsequent to the frame to be interpolated,

representing the block motion vector of the frame to be interpolated at position (x, y).

Images