CN111479110A - Fast Affine Motion Estimation Method for H.266/VVC - Google Patents

Abstract

The present invention proposes a fast affine motion estimation method for H.266/VVC, the steps of which are: calculating the texture complexity of the current CU by using the standard deviation, and classifying the current CU into a static area or a non-static area according to the texture complexity area; for CUs in static areas, skip affine motion estimation, directly use motion estimation to predict the current CU, and select the best prediction direction mode through rate-distortion optimization; for CUs in non-static areas, use trained The Random Forest Classifier RFC model classifies the current CU and outputs the best predicted direction pattern. For the CU in the static area, the present invention skips affine motion estimation, which reduces the computational complexity; for the CU in the non-static area, the present invention directly predicts the prediction direction mode through the model trained in advance, avoiding affine motion estimation , thereby reducing the complexity of the affine motion estimation module.

Description

Fast Affine Motion Estimation Method for H.266/VVC

technical field

The invention relates to the technical field of image processing, in particular to a fast affine motion estimation method for H.266/VVC.

Background technique

In today's information age, the demand for video services such as 3D images, ultra-high-definition video, and virtual reality is increasing, and the encoding and transmission of high-definition video has increasingly become a research hotspot. With the development and improvement of the H.266/VVC standard, the improvement of video processing efficiency has also driven the development of the video industry, laying the foundation for the development of a new generation of video coding technology. Highly dense data brings huge challenges to bandwidth and storage. The current mainstream video coding standards cannot meet the current emerging applications. Therefore, a new generation of video coding standards H.266/VVC came into being to meet people's demand for video clarity. , fluency and real-time requirements. The International Organization for Standardization ISO/IEC MPEG and ITU-T VCEG established a Joint Video Exploration Team (JVET), which is responsible for the development of the next-generation video coding standard H.266/Versatile Video Coding (VVC). H.266/VVC is formulated for high-definition video of 4K and above, and the bit depth is mainly 10 bits, which is different from the positioning of H.265/HEVC, which causes the maximum block size of the current encoder to become 128. The processed pixels are all 10-bit, even if the input 8-bit sequence is converted to 10-bit processing.

H.266/VVC uses a hybrid coding technology framework, and the image division continues to develop from a single, fixed division to a diverse and flexible division structure, which can more efficiently adapt to the encoding and decoding processing of high-resolution images. In addition, H.266/VVC extends the new elements of the original H.265/HEVC encoder such as inter-intra prediction, prediction signal filtering, transformation, quantization/scaling, entropy coding and other new elements for the new generation of video data, and Taking into account the characteristics of the new generation of video coding standards, a new model prediction mode has been added. Specifically, H.266/VVC follows the motion estimation, motion compensation and motion vector prediction techniques of high-efficiency video coding H.265/HEVC inter-frame coding, and introduces some new techniques on this basis. For example, the Merge mode is extended to add a prediction motion vector based on history, and new prediction methods such as affine transformation technology, adaptive motion vector precision method, and 1/16 sampling precision motion prediction compensation are added. The introduction of many advanced coding tools greatly improves the coding efficiency of the new generation video coding standard H.266/VVC. However, it also significantly increases the computational complexity of H.266/VVC inter-frame coding due to the rate-distortion cost calculation, thereby significantly reducing the coding speed of the new generation of video.

The main principle of inter-frame prediction is to find a best matching block in the previously coded image for each pixel block of the current image. This process is called motion estimation ME, and the image used for prediction is called the reference image. The block is the best matching block in the reference image, that is, the reference pixel block, the displacement from the reference block to the current pixel block is called the motion vector MV, and the difference between the current pixel block and the reference block is called the prediction residual. Among them, the motion estimation ME algorithm is the most critical algorithm in the H.266/VVC video coding process. It occupies more than half of the calculation amount and most of the computing time of the entire video coding, and is the dominant factor determining the video compression efficiency. Motion estimation ME has become a research hotspot in video compression technology by effectively removing temporal redundancy between consecutive images. To improve compression efficiency, recent video codecs try to estimate motion of different shapes and sizes. Furthermore, by adding a multi-type tree, motion estimation ME can be performed on very thin blocks (eg, width is one-eighth the height). Therefore, among the various modules of multi-type trees (MTT), motion estimation ME is the tool with the highest coding complexity in VVC. Since the more advanced inter prediction scheme is recursively performed in the finely divided blocks of the MTT, the computational complexity of the motion estimation ME increases even more than that in HEVC, because in Future Video Coding (FVC), the ME also tries to simulate the New technologies such as shot motion estimation. Affine motion estimation AME, characterized by non-translational motions such as rotation and scaling, is effective in rate distortion (RD) performance at the expense of higher coding complexity. In the whole motion estimation ME processing time, the computational complexity of affine motion estimation AME accounts for a large part, so it is very important to reduce its complexity. Therefore, to reduce the complexity of the VTM encoder, it is necessary to speed up the AME module.

In fact, many literatures have done a lot of research on the high complexity of H.265/HEVC inter-frame prediction. J. Xiong et al. proposed a fast CU selection algorithm based on vertebral motion divergence, which can skip individual inter-frame CUs in H.265/HEVC in advance. L. Shen et al. proposed an adaptive inter-mode decision algorithm for H.265/HEVC, which jointly exploited inter-layer and spatiotemporal correlations, proposed early skip mode decision based on statistical analysis, and based on prediction size Correlation mode decision and rate-distortion (RD) cost-dependent mode decision three methods. H. Lee et al. proposed an early skip mode decision method using its distortion characteristics after calculating the RD cost of the 2N×2N Merge mode. Aiming at the high correlation between texture video and depth map content, Q. Zhang et al. proposed an advanced decision coding unit depth level and adaptive mode decision method to reduce the computational complexity of video coding. Q.Hu et al. proposed a fast inter mode decision algorithm based on Neyman-Pearson rule, which includes early SKIP mode decision and fast CU size decision to reduce H.265/HEVC complexity. Z. Pan et al. proposed a fast reference frame selection algorithm based on content similarity to reduce the computational complexity of inter prediction based on multiple reference frames. Based on the optimal motion vector selection correlation between prediction modes of different sizes, Z. Pan et al. propose a fast motion estimation ME method to reduce the coding complexity of H.265/HEVC encoder. J. Zhang et al. proposed a two-stage fast inter-frame CU decision method based on Bayesian method and conditional random field to reduce the coding complexity of HEVC encoder. T.S. Kim et al. proposed a fast motion estimation algorithm based on HEVC, which supports a highly flexible block partition structure. By searching a narrow area around multiple accurate motion vector predictions, the algorithm greatly reduces its computational complexity. C. Ma et al. proposed a neural network-based arithmetic coding method to encode the inter prediction information in HEVC. L. Shen et al. proposed a fast mode decision algorithm to reduce the computational complexity of the encoder. The proposed method utilizes the optimized encoder after adjusting three parameters, namely the conditional probability of the SKIP/Merge mode, the motion characteristics and the mode complexity. D. Wang et al. proposed a fast depth-level and inter-mode prediction algorithm. The algorithm uses inter-layer correlation, spatial correlation and their degree of correlation to speed up HEVC inter-frame coding. The above fast inter-frame methods can effectively reduce the computational complexity of H.265/HEVC while ensuring the coding performance. However, these methods are not designed for H.266/VVC encoders, which use new inter-frame prediction techniques such as more advanced affine motion compensation prediction, extended Merge mode, adaptive Motion vector accuracy, triangulation mode and other technologies. Based on this, the spatial and inter-layer correlation between H.266/VVC and H.265/HEVC-based inter-frame prediction must be quite different, so it is necessary to re-study the low-complexity inter-frame coding based on H.266/VVC. method.

In view of the high complexity of H.266/VVC inter-frame coding, very few literatures have explored it. S. Park et al. proposed a method to effectively limit the reference frame search range for normal motion estimation and affine motion estimation, which mainly exploits the dependencies within the H.266/VVC prediction structure. The method reduces the coding complexity by minimizing the maximum value of the reference frame search range of the CU based on the prediction information of the parent node. Z. Wang et al. proposed an early termination scheme based on a confidence interval based quadtree plus binary tree (QTBT) partition structure, and established a rate distortion (RD) model based on the motion divergence field, to estimate the rate-distortion RD cost of each partition mode; and based on this model, the block partitioning of H.266/VVC is terminated early to eliminate unnecessary partition iterations and make H.266/VVC coding performance and coding complexity. strike a good balance. Z. Wang et al. proposed a fast QTBT partition decision algorithm for convolutional neural networks (CNN) for H.266/VVC inter-frame coding. Architecture of Convolutional Neural Network CNNs and exploit temporal correlation to control misprediction risk to improve the robustness of Convolutional Neural Networks CNN schemes. D. García-Lucas et al. proposed a pre-analysis algorithm for extracting frame motion information, which is used in the motion estimation module to accelerate H.266/VVC encoders. S. Park et al. proposed a fast H.266/VVC inter-frame coding method to effectively reduce the coding complexity of affine motion estimation in VTM when using multi-type tree MTT. The method consists of two procedures: early termination of the scheme and reduction of the number of reference frames for affine motion estimation. H. Gao et al. proposed a low-complexity decoder-side motion vector refinement scheme, by searching for the block with the smallest matching cost in the previously decoded reference pictures, the initial motion vector MV is optimized from the Merge mode, and is added into the decoder-side motion vector refinement method based on bilateral matching. N. Tang et al. proposed a fast block division algorithm for H.266/VVC inter-frame coding, which uses three frame differences to determine whether the current block is a static object; when the current block is static, no further splitting is required, so it is terminated early Partitioning to improve inter-coding speed. However, little work has been done to alleviate the complexity of AME for affine motion estimation in VVC. For VTM, there is a lot of room to further reduce motion estimation ME complexity in multi-type tree MTT structure, especially in affine motion estimation AME.

SUMMARY OF THE INVENTION

In view of the deficiencies in the above background technologies, the present invention proposes a fast affine motion estimation method for H.266/VVC, which solves the technical problem of high coding complexity of affine motion estimation in VTM.

The technical scheme of the present invention is realized as follows:

A fast affine motion estimation method for H.266/VVC, the steps are as follows:

S1, utilize the standard deviation to calculate the texture complexity SD of the current CU, and divide the current CU into a static area or a non-static area according to the texture complexity SD;

S2. For the CU in the static area, skip the affine motion estimation AME, directly use the motion estimation CME to predict the current CU, and select the best prediction direction mode through the rate-distortion optimization method;

S3. For the CU in the non-static area, use the trained random forest classifier RFC model to classify the current CU, and output the best prediction direction mode.

The method for calculating the texture complexity SD of the current CU using the standard deviation is:

Among them, W represents the width of the CU, H represents the height of the CU, and P(a, b) represents the pixel value at the position (a, b) in the CU.

The method of using the motion estimation CME to predict the current CU, and selecting the best prediction direction mode through the rate-distortion optimization method is:

S21. The current CU firstly predicts Uni-L0 in one direction, then predicts Uni-L1 in one direction, and finally predicts Bi in two directions;

S22, utilization distortion optimization respectively calculates the rate-distortion cost of current CU in step S21 through uni-directional prediction Uni-L0, uni-directional prediction Uni-L1 and bi-directional prediction Bi;

S23. Use the prediction mode with the least rate-distortion cost as the best prediction direction mode.

The calculation methods of the rate-distortion cost of the unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi are:

表示所有可用参考列表集合，

表示参考列表集，L0和L1表示两个参考帧列表，φ(j)表示参考列表中的参考帧，J(·)为率失真代价函数，且

D(·)表示CU编码的失真程度，λ表示拉格朗日乘子，R(·)表示CU编码消耗的比特数。in,

represents the set of all available reference lists,

represents the reference list set, L0 and L1 represent two reference frame lists, φ(j) represents the reference frames in the reference list, J( ) is the rate-distortion cost function, and

D(·) represents the distortion degree of CU coding, λ represents the Lagrangian multiplier, and R(·) represents the number of bits consumed by CU coding.

The training method of the random forest classifier RFC model in the step S3 is:

S31. Select Traffic, Kimono, BQSquare, RaceHorseC, and FourPeople video sequences at different resolutions from the general test sequence, encode the first M frames on the VTM, and simultaneously record the shape of the CU in the VTM, the texture complexity of the CU and the CU The three prediction direction modes are used as the data set, the data set includes the sample set S and the test set T, wherein, the three prediction direction modes include unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi;

将生成的每个训练集

作为根节点，生成对应的决策树{T₁,T₂,...,T_K}，其中，i＝1,2,…,K表示第i个训练样本，K表示训练样本集的大小；S32. Use the Bootstrap method to resample the sample set S to generate K training sample sets

will generate each training set

As the root node, the corresponding decision tree {T ₁ , T ₂ ,...,T _K } is generated, where i=1, 2,..., K represents the ith training sample, and K represents the size of the training sample set;

S33. Start training from the root node, randomly select m feature attributes on each intermediate node of the decision tree, calculate the Gini index coefficient of each feature attribute, and select the feature attribute with the smallest Gini index coefficient as the optimal split of the current node attribute, with the minimum Gini index coefficient as the splitting threshold, divide m feature attributes into left subtree and right subtree;

S34, repeat step S33, train K' times, until K' decision tree training is completed, and each decision tree grows completely without pruning;

S35. The generated multiple decision trees are the random forest classifier RFC model, and the random forest classifier RFC model is used to discriminate and classify the test set T, and the classification result adopts the voting method, and the category with the most output from the K' decision trees is used as the The category of the test set T, and the best prediction direction mode of the current CU is obtained.

The method for obtaining the data set in the step S31 is:

S31.1, using motion estimation CME to predict the video sequence;

S31.2. Use a 4-parameter affine motion model to perform affine prediction on the video sequence predicted in step S31.1, wherein the affine prediction includes unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi ;

S31.3, using the 6-parameter affine motion model to perform radiation prediction on the video sequence after the affine prediction in step S31.2;

S31.4. Calculate the rate-distortion cost of the affine prediction performed in steps S31.2 and S31.3 respectively, and set the prediction mode corresponding to the smallest rate-distortion cost as the prediction direction mode of the video sequence.

The characteristic attributes include two-dimensional Haar wavelet transform horizontal coefficients, two-dimensional Haar wavelet transform vertical coefficients, two-dimensional Haar wavelet transform angle coefficients, angular second moment, contrast, entropy, inverse difference moment, minimum difference sum and gradient .

In the 4-parameter affine motion model, the motion vector of the sample position (x, y) in the CU is:

Among them, (mv _0x , mv _0y ) is the motion vector of the upper left control point, (mv _1x , mv _1y ) is the motion vector of the upper right control point, and W represents the width of the CU;

In the 6-parameter affine motion model, the motion vector of the sample position (x, y) in the CU is:

Among them, (mv _2x , mv _2y ) is the motion vector control point in the lower left corner, and H represents the height of the CU.

The beneficial effects that the technical solution can produce: the present invention first uses the standard deviation SD to divide the CU into a static area and a non-static area. If the CU belongs to the static area, the probability of selecting the SKIP mode for inter-frame prediction is higher, and SKIP tends to be selected. Affine prediction is not required for static regions where inter-prediction mode is performed. Therefore, the affine motion estimation AME module can be terminated in advance in the static region, and the best direction mode of the current CU is the best direction mode of the motion estimation CME; if the CU If it belongs to a non-static area, the inter-frame prediction mode of the CU is determined according to the random forest classification model, and finally the optimal prediction direction mode is obtained in advance; therefore, the present invention reduces the computational complexity and saves the coding time, thereby realizing H.266/ Fast encoding for VVC.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

Fig. 1 is the flow chart of the present invention;

Fig. 2 is the prediction direction mode complexity distribution diagram of the present invention;

Fig. 3 is the 4-parameter affine model of the present invention;

Fig. 4 is the 6-parameter affine model of the present invention;

5 is an overall process diagram of the motion estimation ME of the present invention;

Fig. 6 is the overall running time comparison result diagram of the method of the present invention and the FAME method.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

As shown in FIG. 1 , an embodiment of the present invention provides a fast affine motion estimation method for H.266/VVC, and the specific steps are as follows:

S1. In the image encoding process, the image content of a single area is often encoded by using a larger CU. Conversely, regions with rich details are usually encoded with smaller CUs. It can be seen from this that whether the CU uses the SKIP mode to perform inter-frame prediction is determined by using the texture complexity of the coding block. In the process of image encoding, areas with single image content tend to be encoded using SKIP mode for inter-frame prediction, while areas with rich details are less likely to use SKIP mode for inter-frame prediction. The variance of a CU represents the degree of energy dispersion between the two pixels of the current block, so the texture complexity of a block can be roughly measured by its standard deviation SD. Therefore, the standard deviation is used to calculate the texture complexity SD of the current CU, and According to the texture complexity SD, the current CU is divided into static area or non-static area; the formula of standard deviation is:

Among them, W represents the width of the CU, H represents the height of the CU, and P(a, b) represents the pixel value at the position (a, b) in the CU. Since the texture complexity of neighboring blocks is related to CU, the threshold for classification is derived from the texture complexity of neighboring blocks. According to a large amount of experimental data, it is reasonable to take the minimum value among the standard deviations SD of adjacent blocks of a CU as Th _static . CUs can be classified by thresholds. If the current standard deviation SD is less than the threshold Th _static , it indicates that the current CU is a static area. Conversely, if the value of the standard deviation SD is greater than Th _static , the current CU belongs to the non-static region.

S2. Existing video coding standards (such as H.265/HEVC) use motion vectors MV covering translational motion for motion estimation CME. However, affine motion estimation AME can not only predict translational motion, but also linear transform motion, such as scaling and rotation. Affine motion estimation AME predicts motion more accurately than motion estimation CME if the camera is zoomed or rotated to capture video. In H.266/VVC, the affine motion estimation AME is the same as the motion estimation CME, starting from the uni-prediction Uni-prediction L0, then the uni-prediction Uni-prediction L1, and finally the bi-prediction Bi-prediction. After calculating the three prediction direction modes, the rate distortion optimization (RDO) method selects the best prediction direction mode. Figure 2 shows the distribution of the complexity of affine motion estimation AME inter prediction mode, and uni-prediction Uni-prediction L0 requires more prediction time than Uni-prediction L1. If the reference frames of Uni-prediction L0 and Uni-prediction L1 are different, the coding complexity of Uni-prediction is the same. Otherwise, if the reference frame between the uni-prediction Uni-prediction L0 and the uni-prediction L1 is the same, copy the motion vector MV of the uni-prediction Uni-prediction L1 from the motion vector of the uni-prediction Uni-prediction L0, To avoid redundant affine motion estimation AME processing. Therefore, uni-prediction Uni-prediction L0 prediction consumes more prediction time than Uni-prediction L1. Although the greatest coding complexity is required in unidirectional prediction, bidirectional prediction mode has a higher probability of being the best inter prediction mode. In the affine motion estimation AME module, calculating the rate-distortion RD cost is an important reason for the complexity.

其中，D(·)表示CU编码的失真程度，λ表示拉格朗日乘子，R(·)表示CU编码消耗的比特数。因为用单向预测Uni-L0和单向预测Uni-L1表示的两个参考帧列表用于运动预测，所以应该用两个列表测试用于单向预测的运动估计ME过程，从而在两个列表中生成所有可用的帧。In order to obtain the best motion vector MV and the best reference frame, the encoder searches multiple available reference frames, uses the Lagrangian multiplier method to calculate the rate-distortion RD cost J( ) and compares the cost of the prediction results, Lagrangian The rate-distortion RD cost function J(·) calculated by the daily multiplier method is expressed as:

Among them, D(·) represents the distortion degree of CU coding, λ represents the Lagrangian multiplier, and R(·) represents the number of bits consumed by CU coding. Since the two reference frame lists denoted by unidirectional prediction Uni-L0 and unidirectional prediction Uni-L1 are used for motion prediction, the motion estimation ME process for unidirectional prediction should be tested with two lists, so that in both lists to generate all available frames.

For the CU in the static area, skip the affine motion estimation AME, directly use the motion estimation CME to predict the current CU, and select the best prediction direction mode through the rate-distortion optimization method; the specific method is:

S21. The current CU first goes through the one-way prediction Uni-L0, then goes through the one-way prediction Uni-L1, and finally goes through the two-way prediction Bi.

S22, the utilization distortion optimization calculates the rate-distortion cost of the one-way prediction Uni-L0, the one-way prediction Uni-L1 and the two-way prediction Bi respectively;

The rate-distortion costs of the unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi are:

表示所有可用参考列表集合，

表示参考列表集，L0和L1表示两个参考帧列表，φ(j)表示参考列表中的参考帧，J(·)为率失真代价函数。in,

represents the set of all available reference lists,

represents the reference list set, L0 and L1 represent two reference frame lists, φ(j) represents the reference frames in the reference list, and J(·) is the rate-distortion cost function.

S23. Use the prediction mode with the least rate-distortion cost as the best prediction direction mode.

S3. For the CU in the non-static area, the condition of skipping the affine motion estimation AME process is not satisfied, and the trained random forest classifier RFC model is used to classify the current CU, and output the best prediction direction mode to further reduce Computational complexity. The random forest algorithm generates K self-help sample sets based on Bootstrap resampling, and the data of each sample set grows into a decision tree; at the node of each tree, based on the random subspace method RSM, randomly selected from M' feature vectors Extract m (m<<M') features. According to a certain node splitting algorithm, the optimal attribute is selected from the m feature attributes for branch growth; finally, K' decision trees are combined for mode voting. After the random forest classifier is generated, the random forest classifier model is tested. Each tree in the forest will independently determine the classification result, and the final decision is to take the classification category with the most identical judgments. The formula is as follows:

Among them, H(t) represents the combined classification model, h _i (t) is the single classification tree model, t represents the feature attribute of the decision tree, Y represents the output variable, and I( ) represents the collective indicative function (that is, when the When there is a certain classification result, the function value is 1, otherwise it is 0).

When traversing the CU, the characteristics of the CU and the prediction direction mode of the CU are recorded, so as not to interfere with the normal encoding process. Multiple training sets are generated by resampling through the Bagging ensemble method. Samples are randomly and equally extracted from the original training sample set, and K new training sample sets are generated by repeated and replacement extraction, and finally K new training sample sets are obtained. After the sample is extracted, it enters the training module of the follow-up forest classifier model. Table 1 shows the relevant training parameter settings for random forest classifier RFC model establishment.

Table 1 Training parameter configuration

According to the parameters in Table 1, the training method of the random forest classifier RFC model is:

S31. One of the keys to training the classifier is the selection of the sample set. From the general test sequence, select Traffic, Kimono, BQSquare, RaceHorseC and FourPeople video sequences at different resolutions that can cover rich texture complexity. M=50 frames before encoding, and simultaneously record the shape of the CU in the VTM, the texture complexity of the CU, and the three prediction direction modes of the CU as a data set. The data set includes a sample set S=20 and a test set T=30, among which, three The prediction direction modes include unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi;

In VTM, the blocks of affine motion are also predicted in three ways: Uni-L0 for unidirectional prediction, Uni-L1 for unidirectional prediction, and Bi for bidirectional prediction. Meanwhile, affine prediction also includes 4-parameter and 6-parameter affine models. Both unidirectional prediction or bidirectional prediction of Affine Motion Estimation AME module require related reference frames, which increases the coding complexity of VTM. When only the number of reference frames required by each affine motion estimation AME module is calculated, the affine motion estimation AME process requires twice as many reference frames as the motion estimation CME process. The entire motion estimation ME process is shown in Figure 5. As can be seen from Figure 5, the method for obtaining the data set in step S31 is:

S31.1, use motion estimation CME to predict the video sequence, and the prediction method is the same as that of step S21;

S31.2. Use a 4-parameter affine motion model to perform affine prediction on the video sequence predicted in step S31.1, wherein the affine prediction includes unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi ;

As shown in Figure 3, the motion vector of the sample position (x, y) in the CU of the 4-parameter affine motion model is:

Among them, (mv _0x , mv _0y ) is the motion vector of the upper left control point, (mv _1x , mv _1y ) is the motion vector of the upper right control point, and W represents the width of the CU;

S31.3, using the 6-parameter affine motion model to perform radiation prediction on the video sequence after the affine prediction in step S31.2;

As shown in Figure 4, the motion vector of the sample position (x, y) in the block of the 6-parameter affine motion model is:

Among them, (mv _2x , mv _2y ) is the motion vector control point in the lower left corner, and H represents the height of the CU.

S31.4. Calculate the rate-distortion cost of the affine prediction performed in steps S31.2 and S31.3 respectively, and set the prediction mode corresponding to the smallest rate-distortion cost as the prediction direction mode of the video sequence.

将生成的每个训练集

作为根节点，生成对应的决策树{T₁,T₂,...,T_K}，其中，i＝1,2,…,K表示第i个训练样本，K表示训练样本集的大小；S32. Use the Bootstrap method to resample the sample set S to generate K training sample sets

will generate each training set

As the root node, the corresponding decision tree {T ₁ , T ₂ ,...,T _K } is generated, where i=1, 2,..., K represents the ith training sample, and K represents the size of the training sample set;

S33. Start training from the root node, randomly select m feature attributes on each intermediate node of the decision tree, calculate the Gini index coefficient of each feature attribute, and select the feature attribute with the smallest Gini index coefficient as the optimal split of the current node attribute, with the minimum Gini index coefficient as the splitting threshold, divide m feature attributes into left subtree and right subtree;

The effectiveness of machine learning is highly correlated with the diversity and correlation of training datasets. Although the random forest classifier RFC can handle ultra-high dimensional feature data, picking out truly relevant feature vectors can better generalize the classification model. Since the prediction direction mode of the CU is related to the texture, texture direction and motion state of the image, these are used as the classification basis, that is, as the feature vector of the random forest classifier model. The characteristic attributes selected by the present invention include two-dimensional Haar wavelet transform horizontal coefficient (2D Haar wavelet transform horizontal coefficient, HL), two-dimensional Haar wavelet transform vertical coefficient (2D Haar wavelet transform vertical coefficient, LH), two-dimensional Haar wavelet transform Transform angle coefficient (2D Haar wavelet transform angle coefficient, HH), angular second moment (ASM), contrast (contrast, CON), entropy (entropy, ENT), inverse difference moment (inverse difference moment, IDM), The Sum of Absolute Difference (SAD) and the gradient (gradient) are used as the feature attributes of the random forest classifier model, and the feature attributes are calculated as follows:

The horizontal coefficient HL of the two-dimensional Haar wavelet transform of the image represents the texture in the horizontal direction of the image. The larger the value, the richer the texture in the horizontal direction, and the smaller the value, the flatter the texture in the horizontal direction; the vertical coefficient of the two-dimensional Haar wavelet transform of the image LH represents the texture in the vertical direction of the image. The larger the value, the richer the texture in the vertical direction. The smaller the value, the flatter the texture in the vertical direction. The two-dimensional Haar wavelet transform angle coefficient HH of the image represents the texture in the vertical direction of the image. The larger the value, the richer the texture in the 45° direction. The smaller the value, the flatter the texture in the 45° direction. The two-dimensional Haar wavelet transform horizontal coefficient HL, the two-dimensional Haar wavelet transform vertical coefficient LH, and the two-dimensional Haar wavelet transform angle coefficient HH are expressed as:

Among them, W represents the width of the CU, H represents the height of the CU, and P(a,b) represents the pixel value at the position (a,b).

Angular second moment ASM reflects the uniformity of gray distribution and texture thickness. The larger the value, the more uniform the image texture distribution is; the contrast CON reflects the texture depth of the image, the larger the value, the greater the texture depth; the entropy ENT represents the amount of information in the image , the larger the value, the greater the amount of information in the image; the inverse difference moment IDM reflects the size of the local texture change of the image, the texture of different regions of the image is relatively uniform, and the change is slow, the second-order moment of angle ASM, contrast CON, entropy ENT and inverse difference moment IDEs are represented as:

In the motion estimation algorithm based on block matching, there are many judgment criteria for the best matching block. We use the minimum difference and SAD. The smaller the SAD, the closer the reference block is to the current prediction block. The minimum difference and SAD are expressed as:

Among them, P _k (a, b) represents the value of the current pixel, (a, b) represents the coordinates of the current pixel, P _k-1 (a+i', b+j') is the reference pixel value, (a+i ',b+j') represents the coordinates of the reference pixel.

The gradient represents the texture direction of the CU, using the gradients of the horizontal and vertical directions of the luma samples as feature attributes. The gradients in the horizontal and vertical directions are expressed as:

G _x (a,b)=P(a+1,b)-P(a,b)+P(a+1,b+1)-P(a,b+1),

G _y (a,b)=P(a,b)-P(a,b+1)+P(a+1,b)-P(a+1,b+1),

where G _x (a, b) and G _y (a, b) represent the gradient components of the current pixel in the horizontal and vertical directions, respectively. (a,b) represents the coordinates of the pixel, and P(a,b) represents the pixel value.

S34, repeat step S33, train K'=25 times, until K' decision tree training is completed, and each decision tree grows completely without pruning;

S35. The generated multiple decision trees are the random forest classifier RFC model, and the random forest classifier RFC model is used to discriminate and classify the test set T, and the classification result adopts the voting method, and the category with the most output from the K' decision trees is used as the According to the category of the test set T, the best prediction direction mode of the current CU is obtained, which reduces the computational complexity of the affine motion estimation AME module.

To evaluate the method of the present invention, simulation tests were carried out on the latest H.266/VVC encoder (VTM 7.0). The test video sequence is encoded with default parameters in the "Random Access" configuration. BDBR reflects the compression performance of the present invention, and the reduction in time reflects the reduction in complexity. Table 2 presents the coding characteristics of the present invention, the total coding time of the present invention is reduced to 87% on average, and the affine motion estimation AME time is reduced to 56% on average. Therefore, the present invention can effectively save coding time, and the loss of RD performance can be ignored.

Table 2 Coding characteristics of the present invention

It can be seen from Table 2 that the RD performance and the saved encoding running time of the present invention compared with VTM. For different test videos, the experimental results may fluctuate, but the method proposed in the present invention is effective. Compared with VTM, the present invention can effectively reduce the complexity of AME module for affine motion estimation, and has good RD performance.

Affine motion estimation AME module time is measured according to different Quantization parameters (QP). When the quantization parameter QP is 22, it can be seen from Fig. 6 that the affine motion estimation AME module time for all video sequences totals about 36 hours. However, in the method of the present invention, the time of the affine motion estimation AME module is reduced by about 9 hours. It can be seen that this trend is similar under other quantization parameters QPs. Therefore, it is more intuitively observed from Fig. 6 that the proposed method reduces the encoding time of the affine motion estimation AME module, thereby reducing the computational complexity.

The technical solution of the present invention is described in detail above with reference to the accompanying drawings. The technical solution of the present invention proposes a fast affine motion estimation method for H.266/VVC, which effectively reduces the affine motion estimation AME coding in VTM the complexity. First, the standard deviation SD is used to divide the CU into a static area and a non-static area. If the CU belongs to a static area, the probability of selecting the SKIP mode for inter-frame prediction is higher, and the static area that tends to select the SKIP mode for inter-frame prediction does not need to be Affine prediction, therefore, the affine motion estimation AME module can be terminated early in static regions, and the best direction mode of the current CU is the best direction mode of the motion estimation CME. If the CU belongs to a non-static area, the inter-frame prediction mode of the CU is determined according to the random forest classification model, and the optimal prediction direction mode is finally obtained in advance. Therefore, the present invention reduces computational complexity and saves coding time, thereby realizing fast coding of H.266/VVC.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (8)

1. a fast affine motion estimation method for H.266/VVC, is characterized in that, its steps are as follows: S1, utilize the standard deviation to calculate the texture complexity SD of the current CU, and divide the current CU into a static area or a non-static area according to the texture complexity SD; S2. For the CU in the static area, skip the affine motion estimation AME, directly use the motion estimation CME to predict the current CU, and select the best prediction direction mode through the rate-distortion optimization method; S3. For the CU in the non-static area, use the trained random forest classifier RFC model to classify the current CU, and output the best prediction direction mode. 2. The fast affine motion estimation method for H.266/VVC according to claim 1, wherein the method for calculating the texture complexity SD of the current CU using the standard deviation is:

Among them, W represents the width of the CU, H represents the height of the CU, and P(a, b) represents the pixel value at the position (a, b) in the CU. 3. The fast affine motion estimation method for H.266/VVC according to claim 1, wherein, the current CU is predicted by the motion estimation CME, and the best rate-distortion optimization method is selected. The way to predict the direction pattern is: S21. The current CU firstly predicts Uni-L0 in one direction, then predicts Uni-L1 in one direction, and finally predicts Bi in two directions; S22, utilization distortion optimization respectively calculates the rate-distortion cost of current CU in step S21 through uni-directional prediction Uni-L0, uni-directional prediction Uni-L1 and bi-directional prediction Bi; S23. Use the prediction mode with the least rate-distortion cost as the best prediction direction mode. 4. The fast affine motion estimation method for H.266/VVC according to claim 3, wherein the rate-distortion cost of the unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi The calculation methods are:

in,

represents the set of all available reference lists,

represents the reference list set, L0 and L1 represent two reference frame lists, φ(j) represents the reference frames in the reference list, J( ) is the rate-distortion cost function, and

D(·) represents the distortion degree of CU coding, λ represents the Lagrangian multiplier, and R(·) represents the number of bits consumed by CU coding. 5. the fast affine motion estimation method for H.266/VVC according to claim 1, is characterized in that, the training method of the random forest classifier RFC model in described step S3 is: S31. Select Traffic, Kimono, BQSquare, RaceHorseC, and FourPeople video sequences at different resolutions from the general test sequence, encode the first M frames on the VTM, and simultaneously record the shape of the CU in the VTM, the texture complexity of the CU and the CU The three prediction direction modes are used as the data set, the data set includes the sample set S and the test set T, wherein, the three prediction direction modes include unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi; S32. Use the Bootstrap method to resample the sample set S to generate K training sample sets

will generate each training set

As the root node, the corresponding decision tree {T ₁ , T ₂ ,...,T _K } is generated, where i=1, 2,..., K represents the ith training sample, and K represents the size of the training sample set; S33. Start training from the root node, randomly select m feature attributes on each intermediate node of the decision tree, calculate the Gini index coefficient of each feature attribute, and select the feature attribute with the smallest Gini index coefficient as the optimal split of the current node attribute, with the minimum Gini index coefficient as the splitting threshold, divide m feature attributes into left subtree and right subtree; S34, repeat step S33, train K' times, until K' decision tree training is completed, and each decision tree grows completely without pruning; S35. The generated multiple decision trees are the random forest classifier RFC model, and the random forest classifier RFC model is used to discriminate and classify the test set T, and the classification result adopts the voting method, and the category with the most output from the K' decision trees is used as the The category of the test set T, and the best prediction direction mode of the current CU is obtained. 6. The fast affine motion estimation method for H.266/VVC according to claim 5, wherein the method for obtaining the data set in the step S31 is: S31.1, using motion estimation CME to predict the video sequence; S31.2. Use a 4-parameter affine motion model to perform affine prediction on the video sequence predicted in step S31.1, wherein the affine prediction includes unidirectional prediction Uni-L0, unidirectional prediction Uni-L1 and bidirectional prediction Bi ; S31.3, using the 6-parameter affine motion model to perform radiation prediction on the video sequence after the affine prediction in step S31.2; S31.4. Calculate the rate-distortion cost of the affine prediction performed in steps S31.2 and S31.3 respectively, and set the prediction mode corresponding to the smallest rate-distortion cost as the prediction direction mode of the video sequence. 7. The fast affine motion estimation method for H.266/VVC according to claim 5, wherein the characteristic attributes comprise two-dimensional Haar wavelet transform horizontal coefficients, two-dimensional Haar wavelet transform vertical coefficients, Two-dimensional Haar wavelet transform angle coefficient, angle second moment, contrast, entropy, inverse difference moment, minimum difference sum and gradient. 8. The fast affine motion estimation method for H.266/VVC according to claim 6, wherein, in the 4-parameter affine motion model, the motion vector of the sample position (x, y) in the CU is:

Among them, (mv _0x , mv _0y ) is the motion vector of the upper left control point, (mv _1x , mv _1y ) is the motion vector of the upper right control point, and W represents the width of the CU; In the 6-parameter affine motion model, the motion vector of the sample position (x, y) in the CU is:

Among them, (mv _2x , mv _2y ) is the motion vector control point in the lower left corner, and H represents the height of the CU.

Images