CN102752588B - Video encoding and decoding method using space zoom prediction - Google Patents

Abstract

The invention discloses a video encoding and decoding method using spatial scaling prediction. The encoding method includes: using scaling information, performing motion estimation on scaled reference frames with different scaling ratios of the original reference frame, and finding the best match; value to estimate the zoom step size and zoom window of the zoomed reference frame; use the reference frame index prediction to predict the numbering mode of the original reference frame and the zoomed reference frame, saving the number of bits required for encoding the reference frame index. When the decoding method decodes each macroblock, it includes: firstly perform reference frame index prediction, and find out the position of the reference frame in the reference frame matrix; perform scaling prediction, find out the reference frame scaling ratio; generate prediction according to the scaling ratio and MV block; and finally perform steps such as motion compensation.

Description

A Video Codec Method Using Spatial Scaling Prediction

technical field

The invention relates to the technical field of digital video coding and decoding, in particular to a coding and decoding method for predicting a current frame by using spatial scaling information of a video sequence.

technical background

The current mainstream video codec standards based on block-matching hybrid coding framework, such as MPEG4, H.264/AVC, AVS, etc., in the inter-frame prediction, mainly consider the motion of the image block in the two-dimensional plane from the space, namely Horizontal and vertical movement. Obtain the motion vector of the current image block through motion estimation based on block matching, find out the corresponding blocks of the current image block in the previous frame or multiple frames, and in the next frame, and use these corresponding blocks to predict the current image block, And code the motion vector into the code stream. In addition, the motion vector is also predicted, that is, the motion vector of some blocks around the current block is used to calculate the predicted motion vector of the current block, and only the difference between the predicted motion vector and the actual motion vector obtained through motion estimation is encoded into the code stream. The number of bits required for encoding is greatly saved.

In most video sequences, the image content does not simply move in a two-dimensional plane, but zooms. The change of the distance of the object in the scene relative to the shooting lens will cause the zooming of the object; stretching the lens of the camera will cause the zooming of the whole scene. Therefore, if there is a scaling relationship between the current block and the corresponding block in the reference frame, the best matching block cannot be obtained by using the motion estimation algorithm in the existing video coding standard. However, if the zoom relationship in the video sequence can be used for auxiliary prediction, the coding quality can be improved very well.

After searching and researching the existing literature, it was found that "Subsampled Block-Matching for Zoom Motion Compensated Prediction" published by Lai-Man Po et al. of City University of Hong Kong in 2010 proposed a video encoding method using zoom information. This paper proposes to use the interpolation image required for sub-pixel prediction in the original codec standard to generate reference frames with certain zoom ratios. At the same time, the reference frame index of the multi-frame reference is used to represent the original reference frame of the multi-frame reference and the scaled reference frame. The method can improve the encoding performance and can be easily integrated into the existing encoding and decoding framework. However, since this method can only use reference frames with a specific zoom ratio, and the video zoom ratio in practical applications is uncertain, this method cannot achieve high coding quality improvement. At the same time, since encoding needs to generate more reference frames, the reference frame index value used will be relatively large, which affects the encoding performance.

Therefore, the present invention proposes a method for generating different zooming reference frames by using different zooming steps and zooming windows through zooming prediction for each macroblock, so that the zooming prediction is more accurate. At the same time, a reference frame index prediction method is proposed to predict the reference frame index numbering method, which reduces the number of bits consumed to represent the reference frame index.

Contents of the invention

The purpose of the present invention is to provide a video encoding and decoding method that utilizes the spatial scaling information of the video sequence to assist compression, and improves the quality and efficiency of video compression without greatly increasing the decoding complexity.

To achieve the above object, according to one aspect of the present invention, a video coding method is provided, in which motion estimation is performed on scaled reference frames with different scaling ratios from the original reference frame to find the best matching block. The method includes: recording the number of original reference frames R, the number of scaling layers L and the minimum scaling step S in each image frame, and the number of scaling layers and the minimum scaling step can be manually set or estimated through previous frames The original reference frame is the unscaled reconstructed frame of several frames before and after the current frame in the original video sequence; an original reference frame of a macroblock in the image frame has a group of L equal scaling steps in total including all The zooming reference frame group including the original reference frame, the zooming step size and zooming window of the zooming reference frame group are obtained through zooming prediction; different macroblocks can have zooming reference frames with different zooming step sizes and zooming windows; the macro The sub-blocks in the block can perform motion estimation in the scaled reference frame group of the original reference frame to find the best matching block; the index value of the reference frame selected by the macroblock can be numbered in different ways, and the numbering method Predicted from the reference frame index.

The scaled reference frame group is composed of a series of scaled reference frames obtained by performing bilinear interpolation scaling transformation with different scaling ratios according to the original reference frame, including the scaled reference frame with a scaling ratio of 1, namely the original reference frame. The zoom ratios of these zoom reference frames are arranged from small to large, and there are equal zoom steps between adjacent zoom ratios. The scaling step and scaling window are obtained by scaling prediction.

The zoom prediction includes: using the zoom ratio average value Z ₀ of the available macroblocks in the left, upper left, upper, and upper right macroblocks of the current macroblock as the zoom ratio prediction value of the current macroblock; the available macroblocks are Refers to macroblocks that exist and whose scaling ratios also exist; according to the size relationship between Z ₀ and 1 and the number of scaling layers L of the current frame, query the scaling window settings in Table 1. Wherein, the current frame scaling layer number L can be expressed as 3n+k (n is a natural number, k=1, 2, 3):

Table 1

It can be seen from Table 1 that when the absolute value of the difference between the predicted zoom ratio Z ₀ and 1 is greater than n times the minimum zoom step size S, there are n+1 zoom reference frames with zoom ratios between 1 and Z ₀ , where Including 1 and not including Z ₀ , the scaling ratio between 1 and Z ₀ is divided into n+1 parts on average, that is, the scaling step size is obtained as When the absolute value of the difference between the zoom ratio prediction value Z ₀ and 1 is less than or equal to n times the minimum zoom step S, the zoom step is set to The setting of the zoom window can also be found from Table 1, that is, the number of zoom reference frames with zoom ratios in different ranges. According to the zoom window and the zoom step, the zoom ratio of each zoom reference frame in a zoom reference frame group can be determined, and a bilinear zoom operation is performed on the original reference frame to generate each zoom reference frame. After the scaled reference frame is generated, the macroblock can perform motion estimation on the scaled reference frame, and only need to use the reference frame index to distinguish between different scaled reference frames and original reference frames. If the macroblock has multiple original reference frames, each original reference frame has a zoomed reference frame group, and the zoom windows and zoom steps of all zoomed reference frame groups are the same, and are all obtained through the above-mentioned zoom prediction .

The reference frame index prediction refers to predicting the numbering manner of the reference frame index. Due to the introduction of the scaling relationship, the original one-dimensional reference frame sequence arranged in time becomes a two-dimensional reference frame matrix arranged in time and scaling relationship. The basic numbering method can prioritize the time dimension, first numbering the reference frames with different zoom ratios at the same time, and then numbering the reference frames at different times; it can also give priority to the zoom relationship, first numbering the reference frames at the same zoom ratio at different times, and then numbering the reference frames at different times The frame of reference for the zoom ratio. The reference frame index prediction method is as follows: judging the scaling ratios of available macroblocks in the left, upper left, upper, and upper right macroblocks of the current macroblock; if the scaling ratios of the available macroblocks are greater than 1 at the same time, or less than 1 at the same time , the time dimension is prioritized, and the reference frames with different zoom ratios at the same moment are numbered first; otherwise, the reference frames with the same zoom ratio and different moments are numbered first.

According to another aspect of the present invention, a video decoding method is provided. The method includes: when decoding a frame of image, first decode the number of scaling layers L and the minimum scaling ratio S of the image; , the reference frame index prediction is performed first, and the position of the reference frame used by the macroblock in the reference frame matrix is calculated according to the reference frame index of the macroblock; according to the scaling ratio and scaling layer of the surrounding macroblocks The number of layers L, the minimum scaling ratio S, and Table 1 perform scaling prediction to obtain the scaling window and scaling step size, and then calculate the scaling ratio of the reference frame according to the position of the reference frame used by the macroblock in the reference frame matrix; then decode For each sub-block of the macroblock, according to the motion loss of each sub-block and the scaling ratio of the reference frame, the predictor of each sub-block is obtained from the image block at the corresponding position of the original reference frame through a bilinear difference scaling operation block; finally the predictor sub-block is used for motion compensation.

The present invention has the following advantages: the zoom information can be used to improve the video coding compression rate; different macroblocks adopt different zoom steps and zoom windows according to the zoom prediction, which can better adapt to the differences in the zoom relationship of different regions in the image and improve the compression rate; The reference frame index is predicted, the number of bits required to represent the reference frame index is reduced, and the coding efficiency is improved.

Description of drawings

Fig. 1 is the flowchart of coding in the embodiment;

Fig. 2 is a schematic diagram of the surrounding macroblocks of the current macroblock used for scaling prediction and reference frame index prediction;

Fig. 3 is the zoom window and the zoom step under different predicted zoom ratios in the embodiment;

Fig. 4 shows different reference frame index numbering methods in an embodiment.

Fig. 5 is a flowchart of decoding in the embodiment.

detailed description

The present invention will be described in detail below in conjunction with the accompanying drawings and embodiments. This embodiment is only an embodiment of the present invention but not all embodiments.

The encoding process of this embodiment is shown in FIG. 1 . When encoding a frame of image, first write the artificially set number of scaling layers L and the minimum scaling step S into the image header. In this embodiment, the number of original reference frames is set to 3, the number of scaling layers is set to 5, and the minimum scaling step is set to 0.05. Motion estimation using scale prediction for each macroblock. First, zoom prediction is performed, and the zoom window and zoom step are predicted according to the zoom ratio of the surrounding macroblocks, the number of zoom layers is 5, and the minimum zoom step is 0.05. According to the zoom window and the zoom step, 3 zoom reference frame groups corresponding to the 3 original reference frames are generated through bilinear interpolation, and a total of 15 zoom reference frames are generated. Motion estimation is performed on each scaled reference frame, and motion vectors of each sub-block of the macroblock on a certain scaled reference frame are found. Predict the reference frame index according to the available macroblocks around the current macroblock, and predict the numbering mode of the reference frame index of the macroblock. Finally, encode the reference frame index and the residuals and motion vectors of each sub-block of the macroblock.

When performing the scaling prediction and reference frame index prediction on each macroblock, first check the states of the macroblocks on the left, upper left, upper and upper right of the current macroblock, as shown in FIG. 2 . First check whether these macroblocks exist; if so, whether the encoding mode of these macroblocks is inter-frame coding, and whether there is a scaling ratio. If these macroblocks exist and have a scaling factor, they are usable macroblocks. For example, in Fig. 2, the current macroblock is macroblock 10, macroblock 11 and macroblock 14 do not exist, macroblock 12 and macroblock 13 exist and have a scaling ratio, and are available macroblocks, then average according to the scaling ratio of macroblock 12 and macroblock 13 The value is used as the predictive scaling ratio of the macroblock 10; and the reference frame index numbering mode of the macroblock 10 is predicted according to the scaling ratios of the macroblock 12 and the macroblock 13.

In this embodiment, the number of scaling layers is 5, which can be expressed as 3*1+2. When the scaling prediction is performed on each macroblock, the scaling ratio obtained from each macroblock prediction and the minimum scaling step size of 0.05 are used to predict the scaling window and scaling step size. As shown in FIG. 3 , different zoom windows and zoom steps can be obtained for different prediction zoom ratios. For macroblock 100, the predictive scaling ratio is 1.4. Since 1.4>1+1*0.05, according to Table 1, it can be obtained that the zoom ratio is between 1 and 1.4 and there is a zoom reference frame, so the calculated zoom step is And further, the scaling ratios of other scaling reference frames in the prediction window are calculated. For the macroblock 200, the predicted scaling ratio is 0.8, and similarly, the scaling step size can be obtained as 0.1 through Table 1. For the macroblock 300, the predictive scaling ratio is 0.95. According to Table 1, the macroblock scaling step is the minimum scaling step size of 0.05, and there are two scaling reference frames with scaling ratios greater than 1 and smaller than 1.

In this embodiment, the original reference frame is set to 3, so for each original reference frame, a scaling operation needs to be performed to generate another 4 scaled reference frames, and a total of 12 scaled reference frames need to be generated, which has a relatively high amount of computation. In this embodiment, when generating the scaled reference frame, according to the maximum search range set by the encoding, only the original reference frame is used to calculate the partial picture covered by the search range on each reference frame through bilinear interpolation, instead of generating a complete One frame scaling reference frame.

In this embodiment, the reference frame index numbering method has a time priority method, that is, the reference frames with different scaling ratios at the same time are numbered first, as shown in Figure 4 (a), starting from 0 in the dotted line direction from small to large. ; There is a zoom priority method, that is, the reference frames with the same zoom ratio are numbered first, as shown in Figure 4(b), and numbered in ascending order from 0 in the direction of the dotted line. When performing the reference frame index prediction for each macroblock, the reference frame index numbering mode of the current macroblock is predicted according to the scaling ratio of available macroblocks around the current block as shown in FIG. 2 . If the scaling ratios of the available macroblocks are all greater than 1 or less than 1, they are numbered in a time-priority manner; otherwise, they are numbered in a scaling-priority manner.

A typical decoding process in this embodiment is shown in FIG. 5 . When decoding a frame of image, first decode to obtain the number of scaling layers L and the minimum scaling step S. Then each macroblock is decoded: perform reference frame index prediction, and predict the reference frame index numbering method according to the scaling ratio of available macroblocks around the current macroblock as shown in Figure 2, if the scaling ratio of available macroblocks is all greater than 1 or all less than 1, it will be numbered according to the time priority method, otherwise it will be numbered according to the zoom priority method; according to the predicted reference frame index numbering method and reference frame index value, the position of the reference frame in the reference frame matrix can be calculated; then Calculate the average value of the zoom ratio of the available macroblocks around the current macroblock as shown in Figure 2 to obtain the predicted zoom ratio; according to the predicted zoom ratio, the number of zoom layers L and the minimum zoom step S, query the zoom window settings in Table 1, and calculate the scaling step; calculate the scaling ratio of the reference frame according to the scaling step and the position of the reference frame in the reference frame matrix; follow the scaling ratio of the reference frame and the corresponding original reference frame and the sub-block in the macroblock Calculating the predicted block of the sub-block with the motion vector; performing motion compensation and other decoding steps according to the predicted block and the residual error obtained from decoding.

Claims (6)

1. A video coding method using spatial scaling prediction, characterized in that: Record the original reference frame number R, zoom layer number L and minimum zoom step size S of the image in a coded image header; the original reference frame is a number of unscaled reconstructions before and after the current frame in the original video sequence frame; the number of layers of the scaling layer is the number of reference frames in the scaled reference frame group obtained by scaling an original reference frame, including the original reference frame itself with a scaling ratio of 1; When zooming different original reference frames of the same macroblock to generate the zoom reference frame group of each original reference frame, the same zoom window and zoom step should be used; the zoom window is the number of zoom reference frames with zoom ratios in different ranges; Different macroblocks can have different zoom windows and zoom steps; The zoom window and zoom step are obtained by zoom prediction; Arrange all reference frames into a two-dimensional reference frame matrix organized by time dimension and zoom dimension, and number the reference frames in the matrix horizontally or vertically as reference frame indexes; The method of numbering the reference frame is obtained by predicting the reference frame index; The scaling prediction method includes: In the left, upper left, upper, and upper right macroblocks of the current macroblock, if the macroblock exists and the zoom ratio of the macroblock also exists, it is an available macroblock; Use the average value Z ₀ of the available macroblock scaling ratios around the current as the scaling ratio prediction value of the current macroblock; According to the predicted value Z ₀ of the zoom ratio, the number of zoom layers L of the current frame, and the minimum zoom step of the current frame, the zoom window is obtained by a table lookup method; Calculate the zoom step according to the zoom window. 2. The video coding method using spatial scaling prediction as claimed in claim 1, wherein the minimum scaling step size S refers to scaling any original reference frame of any macroblock in the image, allowing two The minimum value of the difference between adjacent zoom ratios. 3. The video coding method using spatial scaling prediction as claimed in claim 1, wherein the table look-up method comprises: Expressing the number of scaling layers of the current frame as 3n+k (n is a natural number, k=1, 2, 3); According to the size relationship between the predicted zoom ratio values Z ₀ and 1 of the current macroblock, and the value of k, query the number of zoom reference frames with zoom ratios in different intervals in the table below. 4. The video coding method using spatial scaling prediction as claimed in claim 1, wherein the method for obtaining the scaling step according to the scaling window is: When the absolute value of the difference between the zoom ratio prediction value Z ₀ and 1 is greater than n times the minimum zoom step size S, there are n+1 zoom reference frames with zoom ratios between Z ₀ and 1, including 1 but not Z ₀ , the scaling ratio between 1 and Z ₀ is divided into n+1 parts on average, that is, the scaling step size is obtained as When the absolute value of the difference between the zoom ratio prediction value Z ₀ and 1 is less than or equal to n times the minimum zoom step S, the zoom step is set to 5. The video coding method using spatial scaling prediction according to claim 1, wherein the reference frame index prediction method is: In the left, upper left, upper, and upper right macroblocks of the current macroblock, if the macroblock exists and the scaling ratio of the macroblock also exists, it is an available macroblock; if the scaling ratio of the available macroblock is greater than 1 at the same time, or If it is less than 1 at the same time, the time dimension takes priority, and the reference frames with different zoom ratios at the same moment are numbered first; otherwise, the reference frames with the same zoom ratio and different moments are numbered first. 6. A video decoding method using spatial scaling prediction, characterized in that: First decode the original reference frame number R of the image, the number of scaling layers L and the minimum scaling ratio S; When decoding each macroblock, first perform the reference frame index prediction according to claim 5, and calculate the position of the reference frame used by the macroblock in the reference frame matrix according to the reference frame index; Carry out scaling prediction as claimed in claim 1, obtain scaling window and scaling step size; Calculate the zoom ratio of the reference frame; According to the motion vector of each sub-block of the macroblock and the scaling ratio of the reference frame, the predicted sub-block of each sub-block is obtained from the image block at the corresponding position of the original reference frame through a bilinear difference scaling operation.