CN115379240A - A MMVD prediction method and system for video coding based on direction extension - Google Patents

Abstract

The invention discloses a video coding MMVD prediction method and system based on direction extension, which relates to the technical field of image processing, comprising the steps of: acquiring a motion vector candidate list based on a Merge mode; selecting an initial motion in the motion vector candidate list according to a preset selection rule Vector; set the starting point with the initial motion vector as the benchmark, and use the regular hexagon as the frame to expand the motion vector search direction; take the starting point as the origin, and perform the motion vector search under the reduced step size range according to the expanded search directions; Calculate the rate-distortion cost under the minimum rate-distortion criterion according to the motion vector set obtained by searching; select the motion vector with the smallest rate-distortion cost as the best prediction motion vector of the current prediction unit. In the present invention, the search directions in the basic MMVD technology of the prior art are expanded to six based on the regular hexagonal frame, which can better predict object motion information and obtain better coding effects.

Description

A MMVD prediction method and system for video coding based on direction extension

technical field

The present invention relates to the technical field of image processing, in particular to a video coding MMVD prediction method and system based on direction extension.

Background technique

H.266/VVC is the latest international video coding standard jointly finalized in July 2020 by ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG) after nearly 4 years of improvement. . With the improvement of video resolution, VVC is the most advanced video coding standard. Compared with its predecessor High Efficiency Video Coding (HEVC) standard, under the same picture quality, VVC can save about 50% of the coding bit rate. In order to improve the accuracy of inter-frame prediction, VVC's inter-frame mode not only optimizes and expands the original Merge mode, but also adds some inter-frame prediction tools that have not appeared in previous video coding standards, such as the extended Merge mode (Extended merge prediction), Affine motion compensated prediction (Affine motion compensated prediction), Bi-directional optical flow technology (Bi-directional optical flow) and merge mode with motion vector difference (Merge mode with MVD, MMVD), etc. However, these newly proposed tools still have many imperfections.

Nowadays, with the development of science and technology and the complexity of object movement, the flexibility of equipment movement is increasing, and the directions in which objects can move are gradually increasing. It is difficult to accurately predict most objects only for translation prediction in the four directions of up, down, left, and right. Therefore, it is necessary to design a new type of direction model that can predict the motion characteristics of more objects for optimization.

Contents of the invention

In order to better adapt to the needs of video changes, while avoiding the increase of bit rate, the present invention proposes a video coding MMVD prediction method based on direction extension, including steps:

S1: Obtain a motion vector candidate list based on the Merge mode;

S2: Select an initial motion vector in the motion vector candidate list according to a preset selection rule;

S3: Set the starting point based on the initial motion vector, and use the regular hexagon as the frame to expand the search direction of the motion vector;

S4: With the starting point as the origin, search for motion vectors with a reduced step size range according to the expanded search directions;

S5: Calculate the rate-distortion cost under the minimum rate-distortion criterion according to the motion vector set obtained by searching;

S6: Select the motion vector with the smallest rate-distortion cost as the best predicted motion vector of the current prediction unit.

Further, in the step S3, the starting point is the pointing position of the initial motion vector in the direction of the previous frame and the direction of the next frame of the current inter-frame image.

Further, in the step S3, the regular hexagon is a regular hexagon in the horizontal direction, and the starting point is the center point of the regular hexagon.

Further, in the step S3, the expanded search direction is the direction of the line connecting the center point of the regular hexagon and each corner point of the regular hexagon.

Further, in the step S4, the motion vector search is performed within the step size range from the initial pixel size to the preset pixel size, and the step size is increased by a multiple of 2 to search for the motion vector.

The present invention also proposes a video coding MMVD prediction system based on direction extension, including:

An initial selection unit, configured to obtain a motion vector candidate list based on the Merge mode, and select an initial motion vector in the motion vector candidate list according to a preset selection rule;

The direction expansion unit is used to set the starting point based on the initial motion vector, and to expand the motion vector search direction with the regular hexagon as the frame;

The vector search unit is used for taking the starting point as the origin, and performing the motion vector search under the reduced step size range according to each search direction after expansion;

A rate-distortion calculation unit, configured to calculate the rate-distortion cost under the minimum rate-distortion criterion according to the motion vector set obtained by searching;

The vector selection unit is configured to select the motion vector with the smallest rate-distortion cost as the best prediction motion vector of the current prediction unit.

Further, in the direction extension unit, the starting point is the pointing position of the initial motion vector in the direction of the previous frame and the direction of the next frame of the current inter-frame image.

Further, in the direction extension unit, the regular hexagon is a regular hexagon in the horizontal direction, and the starting point is the center point of the regular hexagon.

Further, in the direction extension unit, the extended search direction is the direction of the line connecting the center point of the regular hexagon and each corner point of the regular hexagon.

Further, in the vector search unit, the motion vector search is a motion vector search in which the step size is increased by a multiple of 2 within the step size range from the initial pixel size to the preset pixel size.

Compared with the prior art, the present invention at least contains the following beneficial effects:

(1) The method and system for video coding MMVD prediction based on direction extension described in the present invention expands the search directions in MMVD technology from 4 to 6 on the basis of the prior art, so that it can better Predict object motion information to obtain better coding effect;

(2) While expanding the search direction, through the reduction of the step size search range, the increase in the encoding bit rate caused by the motion vector search obtained by too many motion vectors due to the direction amplification is reduced;

(3) The regular hexagon in the horizontal direction is used as the framework for direction amplification. On the basis of retaining the horizontal movement prediction of the original cross model, the expansion of the search direction further improves the horizontal direction, which is more intense than the vertical movement. predictive performance.

Description of drawings

Fig. 1 is a kind of step diagram of the video coding MMVD prediction method based on direction expansion;

Fig. 2 is a kind of structural diagram of the video coding MMVD prediction system based on direction expansion;

Fig. 3 is a schematic diagram of the search direction of a regular hexagonal motion vector in the horizontal direction;

Fig. 4 is a schematic diagram of a search direction of a regular hexagonal motion vector in the vertical direction.

Detailed ways

The following are specific embodiments of the present invention and in conjunction with the accompanying drawings, the technical solutions of the present invention are further described, but the present invention is not limited to these embodiments.

Embodiment one

An inter-frame prediction technology called Merge was proposed for the first time in the HEVC coding standard. By establishing a motion vector (MV) candidate list with a size of 5 for the current prediction unit (PU) block, using the correlation of motion vectors in the space domain and the time domain, the adjacent coding blocks in the spatial domain and the adjacent coding blocks in the temporal domain are processed. Analysis of motion information, obtaining candidate motion vectors, and then performing inter-frame prediction. VVC has improved on the basis of HEVC and optimized the Merge mode. First, the length of the motion vector candidate list is extended from the initial 5 candidate motion vectors to 6 candidate motion vectors; then the Merge technology is improved, and the time domain and space domain adjacent to the current prediction unit are retained. The candidate motion vector selection method of the coding block, and several new inter-frame candidate list selection methods are added to fill the candidate list, such as: constructing MVP (HMVP) based on historical information, pairwise average MVP; finally, still using the minimum The rate-distortion (RD) criterion is used to calculate the rate-distortion cost, and after comparison, an optimal MV is selected as the best predicted MV for the current PU.

However, the motion information obtained in Merge mode may be inaccurate, because the motion information of adjacent coding blocks may not be the real motion trend of the coding block to be predicted, so MMVD technology was proposed. The MMVD technology was proposed for the first time in the VVC coding standard. It is an inter-frame prediction technology that belongs to the Merge technology at the encoding end and the decoding end. The biggest difference between it and the Merge mode is that the MMVD mode needs to transmit MVD to the decoding end. . At the same time, considering the problem of algorithm efficiency, only the first four candidates in the Merge candidate list are used as the initial motion vector for the time being. In the original MMVD technology, the motion vector search is carried out with a step size ranging from 1/4 pixel to 32 pixels, and the step size increases with a multiple of 2 as the number of searches increases, and stops increasing when it reaches 32 pixels; so the original The specific implementation process of the MMVD mode is as follows: firstly, the motion vector candidate list is obtained based on the Merge mode, and a certain number of initial motion vectors are selected according to the coding bit rate requirements, and the direction of the previous frame of the current inter-frame image is selected based on the initial motion vector and the pointing position of the next frame direction as the starting point; secondly, search the variable step length in the four directions (that is, the cross model) of the starting point; finally, the predicted value is obtained through motion compensation, and the rate distortion is calculated through the predicted value cost, so as to compare the rate-distortion cost at each search point to obtain the best combination of motion vector, direction and step size for transmission.

Compared with the Merge mode, the MMVD technology has been improved, and a refined search is performed around the motion vector. However, in real life, due to the continuous improvement of the flexibility of object movement, the movement direction of many objects has gradually changed from simple translational movement to more complex and changeable direction movement. Only the translational movement in the four directions of up, down, left, and right cannot meet the daily video coding needs. Therefore, in order to better handle the complex motion of objects, the MMVD technology needs to be extended in the search direction to better adapt to the needs of video changes. Based on this, as shown in Figure 1, the present invention proposes a video coding MMVD prediction method based on direction extension, including steps:

S1: Obtain a motion vector candidate list based on the Merge mode;

S2: Select an initial motion vector in the motion vector candidate list according to a preset selection rule;

S3: Set the starting point based on the initial motion vector, and use the regular hexagon as the frame to expand the search direction of the motion vector;

S4: With the starting point as the origin, search for motion vectors with a reduced step size range according to the expanded search directions;

S5: Calculate the rate-distortion cost under the minimum rate-distortion criterion according to the motion vector set obtained by searching;

S6: Select the motion vector with the smallest rate-distortion cost as the best predicted motion vector of the current prediction unit.

For inter-frame image coding with more complex and changeable direction motion, increasing the search direction is obviously the best way to deal with the complexity of the motion, so the present invention first thinks of the means of increasing the search direction. On this basis, considering that the motion vector has directionality, and under natural circumstances, the intensity of horizontal motion in video frame-by-frame images is obviously higher than that in vertical direction. According to this characteristic, on the basis of avoiding too many search directions, the present invention proposes to expand the search direction of the motion vector with a regular hexagon in the horizontal direction as the frame. Wherein, as shown in Figure 3, the starting point described in the step S3 is the center point of the regular hexagon, and the search direction after expansion is the direction of the line connecting the center point of the regular hexagon and each corner point on the regular hexagon, that is Based on the positive semi-axis of the X-axis as the starting line, the motion vector search within a certain step range is carried out in six directions of 0°, 60°, 120°, 180°, 240°, and 300° in order to obtain better The encoding effect and save the encoding bit rate. As shown in Figure 4, compared with the regular hexagon in the vertical direction for motion vector search, the regular hexagon in the horizontal direction not only retains the motion prediction in the horizontal direction of the original cross model, but also predicts two motions in the vertical direction. It is replaced by four motion predictions pointing in different directions, which is more in line with the characteristics of image motion in natural situations.

It can be seen that compared with the original MMVD model, the motion vector is searched only in four directions of the conventional cross model, up, down, left, and right, and then the inter-frame prediction optimal prediction motion vector is determined according to the search results. The six-direction motion vector search based on the regular hexagon proposed by the present invention has a wider coverage, so the prediction result is more accurate, and it can handle more flexible and changeable image motion situations.

Since there are reference images in two directions of the previous frame and the next frame in the MMVD model, and each reference image is expanded into four directions, and each direction has eight search points, which will lead to the complexity of the search algorithm higher degree. With the increase of the search direction, if the motion vector search is still performed in the original MMVD model, it can be predicted that the computational complexity of the search will increase again. The most direct way to reduce the complexity of the algorithm is to reduce the number of search points. The size of the search range directly affects the number of search points. A larger search range can cover more search points, which can improve the decoded video quality to a certain extent. However, an excessively large search range will increase the computational complexity of video encoding, and will lose a lot of time, thereby affecting the real-time performance of the video. A search range that is too small may cause problems with local optimal solutions, and cannot solve the problem of inter-frame prediction well. Therefore, the size of the search range needs to be balanced according to actual usage requirements in terms of algorithm performance and algorithm time complexity.

Based on the above, in order to reduce the computational complexity of the algorithm, the present invention proposes an improved way to reduce the range of the search step, and performs a secondary improvement on the basis of expanding the regular hexagonal search direction in the horizontal direction. According to the center bias characteristic of motion, on the basis of retaining two reference images in the original MMVD model for inter-frame prediction in two different directions of the previous frame direction and the next frame direction, the range of the search step is reduced. In a preferred embodiment, only four search points are searched in each direction, that is, only the part whose step size is within the range of 1/4 pixel to 2 pixels is searched, so as to reduce the search complexity.

In the Skip and Merge modes, the MMVD technology uses three parameters: the starting point, the motion step size, and the motion direction to represent the motion vector. Its candidate list selection process is also the same as the Merge motion vector candidate list generation method in the VVC standard. The MMVD proposed by the present invention retains these characteristics during optimization, and only improves the process of guessing and selecting the final motion vector, adding optional directions. The interframe coding process of the whole MMVD of the present invention can roughly be divided into three steps:

Step1: Acquire the motion vector candidate list based on the Merge mode, and select the appropriate motion vector as the initial motion vector (the first two motion vectors in the candidate list in this embodiment);

Step2: Construct a new MMVD motion vector set for the selected initial motion vector in the case of motion step reduction and search direction expansion;

Step3: Based on the minimum rate-distortion criterion, select the best predicted motion vector in the motion vector set and perform inter-frame coding.

Among them, the new MMVD motion vector set construction steps are as follows:

First, select the first two digits in the Merge motion vector candidate list to check and use them as the initial motion vector. For the initial motion vector, the positions pointed to in the reference frames before and after the motion vector are selected as the starting point for searching. Then, select a direction to search the motion vector within a certain step range. In the same direction, each time a step point is encountered, a candidate motion vector is formed. This process is repeated until the two initial motion vectors are traversed in 6 directions and 4 steps, respectively, and 24 new motion vectors of two adjacent frames of reference images before and after are obtained as motion vector sets.

Embodiment two

The effect of the present invention can be further illustrated by the following simulation experiment: the experiment is based on the H.266 encoding standard test software VTM, the computer used is configured as AMD Ryzen R75800H CPU, 3.2GHz, the running memory is 16GB, and the operating system is Windows 10 (64 bits) , the operating environment is Microsoft Visual Studio 2019. The test material was selected from Common Test Conditions (CTC) for Standard Dynamic Range (SDR) Video. The test sequences here are divided into five levels, namely B, C, D, E, and F, where B, C, and D represent 1920×1080, 832×480, and 416×240 resolution videos, respectively. The simulation test condition mode is low-latency B (LPB) mode, and four quantization parameters (QP) 22, 27, 32, 37 are used to encode the standard test video sequence. All the results of this experiment are obtained by encoding 50 frames of each video sequence. The encoding performance is measured by the widely used BD-Rate value, and the encoding and decoding time is represented by EncT and DecT.

The BD-Rate value indicates the change of the bit rate when the image quality PSNR is consistent. When its value is negative, it means that the encoding performance is improved and the code rate is saved; otherwise, it means that the code rate is lost and the performance is degraded. The values of EncT and DecT represent the ratio of the total encoding and decoding time of the improved algorithm to the total encoding and decoding time of the standard algorithm. The results of this simulation experiment are shown in Table 3 and Table 4:

Table 3 shows the overall performance comparison of the proposed methods in terms of coding efficiency and coding complexity.

Table 3: Performance and time results of different sequences based on the method of the present invention

Table 4: Performance results of different sequences based on the method of the present invention

The method proposed by the invention generally improves the coding performance of the MMVD technology, and at the same time balances the time complexity. From the perspective of performance, through the changes of the three columns of YUV data in Table 3, most of the values of YUV have been reduced; and from the average point of view, the average performance has been improved. In all test sequences, the luminance component Y saves an average of 0.24% of the code rate, the highest is Class D, saving 0.49% of the code rate, and the lowest is Class B, which also saves 0.15% of the code rate; while the chrominance component U It can save an average bit rate of 0.26%. Although the performance in Class E is not so good, it has achieved a performance improvement of 0.31%~0.44% for Class B, Class D and Class F; although the chroma component V only saves on average The code rate is 0.01%, but looking at Class D alone, it has achieved a performance improvement of 0.62%. Relatively speaking, the single performance improvement is relatively large; while the performance is improving, the change in time complexity is analyzed. From Table 3 EncT It can be seen from the two columns of data of DncT and DncT that the encoding and decoding time has also been slightly reduced, with an average reduction of 6% and 5% respectively. From the perspective of sub-video sequence groups, although the time complexity results are not very good for groups B and C with larger video sizes, but for small-sized test sequences, the time is increased by 10% to 15%. the complexity.

Table 4 shows the performance comparison of the proposed method in different sequences in terms of coding efficiency. The data provided is based on Table 3 and refines the performance YUV test results. It can be seen from Table 4 that the brightness component is in Class The Y performance of the luminance component of F is not so significant in the ChinaSpeed sequence, and the other sequences have improved; by analyzing the results of Class E, it can be found that this scheme only partially improves the Y component in the FourPeople and KristenAndSara sequences. Degree components U and V, the improvement results are not obvious, while the remaining RaceHorses sequences of the Class E group have no obvious improvement results on the U component; in addition, there are basically two or more YUV components in the remaining test sequence groups. A decrease of more than two values means that coding gain is obtained and performance is enhanced. It is worth noting that the proposed method can maintain a relatively stable coding efficiency improvement in some video sequences, such as BasketballPass, BlowingBubbles, and PartyScene. The test results of the luminance component and chrominance component of these sequences have achieved gains greater than 0.30%.

Embodiment Three

In order to better understand the technical content of the present invention, this embodiment illustrates the present invention in the form of an inter-system structure, as shown in Figure 2, a video coding MMVD prediction system based on direction extension, including:

An initial selection unit, configured to obtain a motion vector candidate list based on the Merge mode, and select an initial motion vector in the motion vector candidate list according to a preset selection rule;

The direction expansion unit is used to set the starting point based on the initial motion vector, and to expand the motion vector search direction with the regular hexagon as the frame;

The vector search unit is used for taking the starting point as the origin, and performing the motion vector search under the reduced step size range according to each search direction after expansion;

A rate-distortion calculation unit, configured to calculate the rate-distortion cost under the minimum rate-distortion criterion according to the motion vector set obtained by searching;

The vector selection unit is configured to select the motion vector with the smallest rate-distortion cost as the best prediction motion vector of the current prediction unit.

Further, in the direction extension unit, the starting point is the pointing position of the initial motion vector in the direction of the previous frame and the direction of the next frame of the current inter-frame image.

Further, in the direction extension unit, the regular hexagon is a regular hexagon in the horizontal direction, and the starting point is the central point of the regular hexagon.

Further, in the direction extension unit, the extended search direction is the direction of the line connecting the center point of the regular hexagon and each corner point of the regular hexagon.

Further, in the vector search unit, the motion vector search is performed within the step size range from the initial pixel size to the preset pixel size, and the step size is increased by a multiple of 2.

In summary, a method and system for MMVD prediction of video coding based on direction extension described in the present invention expands the search directions in MMVD technology from 4 to 6 on the basis of the prior art, so that it can Better predict the motion information of the object, so as to obtain better coding effect. While amplifying the search direction, by reducing the search range of the step size, the encoding bit rate increase caused by too many motion vectors obtained by the motion vector search due to the direction amplification is reduced.

The regular hexagon in the horizontal direction is used as the framework for direction amplification, and on the basis of retaining the horizontal movement prediction of the original cross model, the prediction performance in the horizontal direction, which is more severe than the vertical movement, is further improved through the expansion of the search direction .

It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relationship between the components in a certain posture (as shown in the figure). Relative positional relationship, movement conditions, etc., if the specific posture changes, the directional indication will also change accordingly.

In addition, in the present invention, descriptions such as "first", "second", "one" and so on are used for descriptive purposes only, and should not be understood as indicating or implying their relative importance or implicitly indicating the indicated technical features quantity. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

In the present invention, unless otherwise specified and limited, the terms "connection" and "fixation" should be understood in a broad sense, for example, "fixation" can be a fixed connection, a detachable connection, or an integral body; It can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediary, and it can be an internal communication between two elements or an interaction relationship between two elements, unless otherwise clearly defined. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention according to specific situations.

In addition, the technical solutions of the various embodiments of the present invention can be combined with each other, but it must be based on the realization of those skilled in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered as a combination of technical solutions. Does not exist, nor is it within the scope of protection required by the present invention.

Claims (10)

1. A video coding MMVD prediction method based on direction extension, characterized in that, comprising steps: S1: Obtain a motion vector candidate list based on the Merge mode; S2: Select an initial motion vector in the motion vector candidate list according to a preset selection rule; S3: Set the starting point based on the initial motion vector, and use the regular hexagon as the frame to expand the search direction of the motion vector; S4: With the starting point as the origin, search for motion vectors with a reduced step size range according to the expanded search directions; S5: Calculate the rate-distortion cost under the minimum rate-distortion criterion according to the motion vector set obtained by searching; S6: Select the motion vector with the smallest rate-distortion cost as the best predicted motion vector of the current prediction unit. 2. a kind of video coding MMVD prediction method based on direction extension as claimed in claim 1, is characterized in that, in described S3 step, starting point is initial motion vector in the previous frame direction of current inter-frame image and next frame The pointing position of the direction. 3. a kind of video coding MMVD prediction method based on direction extension as claimed in claim 1, is characterized in that, in described S3 step, regular hexagon is the regular hexagon on horizontal direction, and starting point is the center of regular hexagon point. 4. a kind of video coding MMVD prediction method based on direction extension as claimed in claim 3, it is characterized in that, in described S3 step, the search direction after extension is each corner point on regular hexagon central point and regular hexagon direction of connection. 5. a kind of video coding MMVD prediction method based on direction extension as claimed in claim 1, is characterized in that, in described S4 step, motion vector search is in the step size range of initial pixel size to preset pixel size, Carry out the motion vector search with the step size increasing by 2 times. 6. A video coding MMVD prediction system based on direction extension, characterized in that, comprising: An initial selection unit, configured to obtain a motion vector candidate list based on the Merge mode, and select an initial motion vector in the motion vector candidate list according to a preset selection rule; The direction expansion unit is used to set the starting point based on the initial motion vector, and to expand the motion vector search direction with the regular hexagon as the frame; The vector search unit is used for taking the starting point as the origin, and performing the motion vector search under the reduced step size range according to each search direction after expansion; A rate-distortion calculation unit, configured to calculate the rate-distortion cost under the minimum rate-distortion criterion according to the motion vector set obtained by searching; The vector selection unit is configured to select the motion vector with the smallest rate-distortion cost as the best prediction motion vector of the current prediction unit. 7. a kind of video coding MMVD prediction system based on direction extension as claimed in claim 6, is characterized in that, in described direction extension unit, starting point is initial motion vector in the previous frame direction of current inter-frame image and next frame direction The pointing position in the frame direction. 8. a kind of video coding MMVD prediction system based on direction extension as claimed in claim 6, is characterized in that, in described direction extension unit, regular hexagon is the regular hexagon on the horizontal direction, and starting point is regular hexagon center point. 9. a kind of video coding MMVD prediction system based on direction extension as claimed in claim 8, is characterized in that, in described direction extension unit, the search direction after extension is regular hexagon center point and each angle on regular hexagon The connection direction of the points. 10. A kind of video coding MMVD prediction system based on directional extension as claimed in claim 6, characterized in that, in the vector search unit, the motion vector search is within the step size range from the initial pixel size to the preset pixel size , carry out the motion vector search with step size 2 times increasing.

Images