CN113630601B - An affine motion estimation method, device, equipment and storage medium - Google Patents

Abstract

The application provides an affine motion estimation method, an affine motion estimation device, affine motion estimation equipment and a storage medium. Wherein the method comprises the following steps: initializing a current block to obtain attribute information of the current block; determining an initial predicted motion vector of the current block according to the attribute information; carrying out affine motion estimation on the current block based on the initial predicted motion vector to obtain an affine motion vector; based on a linear equation set for calculating the affine motion vector, the affine motion vector is iteratively updated along the direction of decreasing the mean square error gradient, and the optimal affine motion vector of the current block is obtained after iteration is completed.

Description

An affine motion estimation method, device, equipment and storage medium

Technical field

The present application relates to the field of video coding and decoding technology, and in particular to an affine motion estimation method, device, equipment and storage medium.

Background technique

The main function of video coding and decoding technology is to pursue the highest possible video reconstruction quality and the highest possible compression ratio within the available computing resources, such as Advanced Video Coding (AVS).

Affine-based motion compensation technology is a displacement transformation model used for irregular movements such as fade-in/fade-out, rotation, scaling, etc., which solves the problem of inaccurate motion compensation in translation transformation models. Affine-based motion compensation technology includes AffineMerge mode (affine merge mode) and affine motion estimation mode (Affine MotionEstimation), both of which are included in the inter-frame mode selection process. Affine motion estimation is used together with ordinary motion estimation to calculate the rate-distortion cost RDCost. As a new tool for inter-frame prediction, affine motion estimation increases the time complexity of the encoder and requires more hardware resources.

Summary of the invention

The purpose of this application is to provide an affine motion estimation method, device, equipment and storage medium to reduce the complexity of affine motion estimation and facilitate hardware implementation.

The first aspect of this application provides an affine motion estimation method, including:

Initialize the current block and obtain the attribute information of the current block;

Determine an initial predicted motion vector of the current block according to the attribute information;

Perform affine motion estimation on the current block based on the initial predicted motion vector to obtain an affine motion vector;

Based on the linear equation system for calculating the affine motion vector, the affine motion vector is iteratively updated along the direction of mean square error gradient descent, and after the iteration is completed, the optimal affine motion vector of the current block is obtained.

A second aspect of this application provides an affine motion estimation device, including:

The initialization module is used to initialize the current block and obtain the attribute information of the current block;

Determining module, configured to determine the initial predicted motion vector of the current block according to the attribute information;

An affine motion estimation module, used for performing affine motion estimation on the current block based on the initial predicted motion vector to obtain an affine motion vector;

An iterative update module, configured to iteratively update the affine motion vector along the direction of mean square error gradient descent based on the linear equation system for calculating the affine motion vector, and obtain the optimal affine motion vector of the current block after the iteration is completed. .

A third aspect of the present application provides an affine motion estimation device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor runs the computer program Executed to implement the method described in the first aspect of this application.

A fourth aspect of the present application provides a computer-readable medium on which computer-readable instructions are stored, and the computer-readable instructions can be executed by a processor to implement the method described in the first aspect of the present application.

Compared with the existing technology, the affine motion estimation method provided by this application initializes the current block to obtain the attribute information of the current block; determines the initial predicted motion vector of the current block based on the attribute information; based on the initial predicted motion The vector performs affine motion estimation on the current block to obtain the affine motion vector; based on the linear equation system for calculating the affine motion vector, the affine motion vector is iteratively updated along the direction of mean square error gradient descent. After the iteration is completed The optimal affine motion vector of the current block is obtained. Compared with the existing technology, the gradient-based fast affine motion estimation of this application can reduce the complexity of affine motion estimation and is easy to implement in hardware.

BRIEF DESCRIPTION OF THE DRAWINGS

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be construed as limiting the application. Also throughout the drawings, the same reference characters are used to designate the same components. In the attached picture:

Figure 1 shows a flow chart of an affine motion estimation method provided by this application;

Figure 2 shows a flow chart of a specific affine motion estimation method provided by this application;

Figure 3 shows a flow chart for calculating rate distortion cost provided by this application;

Figure 4 shows a schematic diagram of a pipeline structure for affine motion compensation provided by this application;

Figure 5 shows a flow chart for iteratively updating affine motion vectors provided by this application;

Figure 6 shows a schematic diagram of the division operations of the first three levels of the solution step of the system of equations;

Figure 7 shows a schematic diagram of an affine motion estimation device provided by this application;

Figure 8 shows a schematic diagram of an affine motion estimation device provided by this application;

Figure 9 shows a schematic diagram of a computer-readable storage medium provided by this application.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a thorough understanding of the disclosure, and to fully convey the scope of the disclosure to those skilled in the art.

It should be noted that, unless otherwise stated, the technical terms or scientific terms used in this application should have the usual meanings understood by those skilled in the art to which this application belongs.

In addition, terms such as "first" and "second" are used to distinguish different objects and are not used to describe a specific order. Furthermore, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes Other steps or units inherent to such processes, methods, products or devices.

To further illustrate the solution of the embodiment of the present application, the following description will be made in conjunction with the accompanying drawings. It is understandable that in the following embodiments, the same or corresponding contents can be referenced to each other, and for the sake of simplicity of description, no further description will be given later.

First, some technical terms used in this application are introduced as follows:

affine motion estimation, that is, affine motion estimation.

affine motion compensation, that is, affine motion compensation.

CU: coding unit, that is, coding unit.

ME: motion estimation, that is, motion estimation.

mv: motion vector, that is, motion vector.

mvd: motion vector, which is the motion vector difference.

RDO: rate distortion optimization, that is, rate distortion optimization.

satd: sum of absolute transformed difference, sum of absolute transformed difference.

Embodiments of the present application provide an affine motion estimation method and device, an affine motion estimation device, and a computer-readable storage medium, which will be described below with reference to the accompanying drawings.

Please refer to Figure 1, which shows a flow chart of an affine motion estimation method provided by some embodiments of the present application. This method can be applied to the AVS3 hardware encoder.

As shown in Figure 1, the above affine motion estimation method may include the following steps S101 to S104:

Step S101: Initialize the current block and obtain the attribute information of the current block;

Specifically, the attribute information of the current block includes information such as the size of the current block and the position of the current block in the current frame. This step mainly involves data splitting. After data partitioning, each data block can be processed in parallel.

Step S102: determining an initial predicted motion vector of the current block according to the attribute information;

Please refer to FIG. 2 , which shows a flow chart of a specific affine motion estimation method provided by some embodiments of the present application. Specifically, step S102 can be implemented as:

According to the attribute information, obtain the translation motion vector of the current block and the affine motion vector of the adjacent block;

Performing motion compensation on the current block according to the affine motion vector of the adjacent block, and calculating the first rate distortion cost after motion compensation;

Performing motion compensation on the current block according to the translation motion vector of the current block, and calculating a second rate distortion cost after motion compensation;

Perform motion compensation on the current block according to the zero vector, and calculate the third-rate distortion cost after motion compensation;

The minimum rate distortion cost is selected from the first rate distortion cost, the second rate distortion cost and the third rate distortion cost, and the motion vector corresponding to the minimum rate distortion cost is used as the initial predicted motion vector of the current block.

In some implementations of the present application, the above-mentioned method of motion compensation for the current block may be as follows:

Divide the current block into multiple sub-blocks;

A pipeline structure is used to perform motion compensation on the plurality of sub-blocks.

Please refer to FIG. 3 , which shows a flow chart of calculating rate-distortion cost provided by some embodiments of the present application.

Specifically, in Figure 3, the function of motion compensation is to interpolate the predicted pixels after motion based on the current coordinates, motion vectors and reference pixels. Currently, motion compensation is an optimization bottleneck in the number of clock cycles and hardware resources in affine motion estimation. This application designs a pipeline structure for affine motion compensation, and its microstructure is shown in Figure 4.

In Figure 4, taking the 32x32 CU size as an example, there are 64 4x4 sub-block pipeline structures in the CU for motion compensation. Among them, the motion compensation of each sub-block is divided into four steps. The first step is to calculate the coordinates of the reference block corresponding to the sub-block (shown as coordcalc in Figure 4); the second step is to obtain the pixel value of the reference block according to the coordinates of the reference block (shown as split ref in Figure 4); the third step In the first step, the predicted pixel is calculated based on the pixel value interpolation of the reference block (indicated as mc in Figure 4); in the fourth step, the predicted pixel is stored (indicated as write pred in Figure 4). The above four steps of each 4x4 sub-block are performed in a serial manner; the motion compensation between sub-blocks is performed according to the pipeline structure.

If motion compensation is performed block by block in units of 4x4 sub-blocks, the hardware speed will be slow; if the motion compensation process of all 4x4 sub-blocks is parallelized, it will cause a problem of excessive hardware area. Therefore, this application adopts the above motion compensation pipeline structure , to achieve a balance between hardware area and hardware speed.

Step S103: Perform affine motion estimation on the current block based on the initial predicted motion vector to obtain an affine motion vector;

Step S104: Based on the linear equation system for calculating the affine motion vector, iteratively update the affine motion vector along the direction of mean square error gradient descent, and obtain the optimal affine motion vector of the current block after the iteration is completed.

Specifically, iteratively updating the affine motion vector is a key step in affine motion estimation. Iteratively updating the affine motion vector along the direction of mean square error (MSE) gradient descent, and the optimal affine motion vector that finally converges is affine Motion vector output by motion estimation.

Figure 5 shows a flow chart for iteratively updating affine motion vectors in this application. Among them, this application optimizes the solution steps of the system of equations. Before optimization, the solution step of the system of equations includes 7 levels of 64-bit division operations. Each level of division operation consumes more cycles. Figure 6 shows a schematic diagram of the first three levels of division operations in the step of solving the system of equations.

This application uses the following method to optimize the cycle of the solution step of the system of equations. The first is to merge the division operations of the previous stage into the last stage, that is, to reduce the number of division operation stages. However, because the division calculation is not performed in the previous stage, it will cause the intermediate variable data to overflow, so the accuracy of the data involved in the operation needs to be reduced first. This application replaces both the divisor and the dividend of the division operation with numbers corresponding to the nearest power of 2 (for example, replace 9 with the 3rd power of 2, that is, 8). The division operation can be replaced with a shift operation. , although this optimization method will cause some loss of algorithm accuracy, it greatly reduces the number of cycles and saves hardware resources.

In the process of affine motion vector prediction based on gradient-based affine motion estimation, a lot of division operations are involved, which affects the hardware speed. Since the divisor and the dividend are both variables, the calculation method of this application is to find the 2 adjacent divisor and dividend respectively. Power numbers use shift operations instead of division operations to increase hardware speed and reduce hardware resources.

The affine motion estimation method provided by this application initializes the current block to obtain attribute information of the current block; determines the initial predicted motion vector of the current block based on the attribute information; and performs affine on the current block based on the initial predicted motion vector. Motion estimation is used to obtain the affine motion vector; based on the linear equation system for calculating the affine motion vector, the affine motion vector is iteratively updated along the direction of the mean square error gradient descent. After the iteration is completed, the optimal affine motion vector of the current block is obtained. Compared with the existing technology, the gradient-based affine motion estimation of this application can reduce the complexity of affine motion estimation and is easy to implement in hardware.

In the above embodiment, an affine motion estimation method is provided. Correspondingly, the present application also provides an affine motion estimation device. Please refer to FIG. 7 , which shows a schematic diagram of an affine motion estimation device provided by some embodiments of the present application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment. The device embodiments described below are merely illustrative.

As shown in Figure 7, the affine motion estimation device 10 includes:

Initialization module 101 is used to initialize the current block and obtain the attribute information of the current block;

Determining module 102, configured to determine the initial predicted motion vector of the current block according to the attribute information;

An affine motion estimation module 103 is used to perform affine motion estimation on the current block based on the initial predicted motion vector to obtain an affine motion vector;

The iterative update module 104 is configured to iteratively update the affine motion vector along the direction of mean square error gradient descent based on the linear equation system for calculating the affine motion vector, and obtain the optimal affine motion of the current block after the iteration is completed. Vector.

According to some implementations of the present application, the determining module 102 is specifically configured to:

According to the attribute information, obtaining a translation motion vector of a current block and an affine motion vector of an adjacent block;

Performing motion compensation on the current block according to the affine motion vector of the adjacent block, and calculating the first rate distortion cost after motion compensation;

Performing motion compensation on the current block according to the translation motion vector of the current block, and calculating a second rate distortion cost after motion compensation;

Perform motion compensation on the current block according to the zero vector, and calculate the third-rate distortion cost after motion compensation;

The minimum rate distortion cost is selected from the first rate distortion cost, the second rate distortion cost and the third rate distortion cost, and the motion vector corresponding to the minimum rate distortion cost is used as the initial predicted motion vector of the current block.

According to some implementations of the present application, the determination module 102 performs motion compensation on the current block in the following manner:

Divide the current block into multiple sub-blocks;

A pipeline structure is used to perform motion compensation on the plurality of sub-blocks.

According to some implementations of the present application, the attribute information includes the size of the current block and the position of the current block in the current frame.

The affine motion estimation device provided in the embodiment of the present application is based on the same inventive concept as the affine motion estimation method provided in the aforementioned embodiment of the present application and has the same beneficial effects.

The embodiments of the present application also provide an affine motion estimation device corresponding to the affine motion estimation method provided in the aforementioned embodiments, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the aforementioned affine motion estimation method.

Please refer to Figure 8, which shows a schematic diagram of an affine motion estimation device provided by some embodiments of the present application. As shown in Figure 8, the affine motion estimation device 20 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the affine motion estimation method provided by any of the aforementioned embodiments of the present application when running the computer program.

The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

The bus 202 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. The memory 201 is used to store a program, and the processor 200 executes the program after receiving the execution instruction. The motion estimation method disclosed in any of the aforementioned embodiments of the present application can be applied to the processor 200, Or implemented by processor 200.

The processor 200 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 200 . The above-mentioned processor 200 can be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logical block diagram disclosed in the embodiment of this application can be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 201. The processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

The affine motion estimation device provided by the embodiments of the present application and the affine motion estimation method provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented.

An embodiment of the present application also provides a computer-readable storage medium corresponding to the motion estimation method provided in the aforementioned embodiment. Please refer to Figure 9, which shows that the computer-readable storage medium is a CD 30 on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the affine motion estimation method provided in any of the aforementioned embodiments.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the affine motion estimation method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features can be equivalently replaced; and these modifications or substitutions do not deviate from the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present application. The scope shall be covered by the claims and description of this application.

Claims (5)

1. An affine motion estimation method comprising:

initializing a current block to obtain attribute information of the current block; the attribute information includes the size of the current block and the position of the current block in the current frame;

determining an initial predicted motion vector of the current block according to the attribute information;

carrying out affine motion estimation on the current block based on the initial predicted motion vector to obtain an affine motion vector;

based on a linear equation set for calculating the affine motion vector, iteratively updating the affine motion vector along the direction of decreasing the mean square error gradient, and obtaining an optimal affine motion vector of the current block after iteration is completed;

the determining the initial prediction motion vector of the current block according to the attribute information comprises the following steps:

according to the attribute information, obtaining a translational motion vector of the current block and an affine motion vector of an adjacent block;

performing motion compensation on the current block according to affine motion vectors of adjacent blocks, and calculating a first rate distortion cost after motion compensation;

performing motion compensation on the current block according to the translational motion vector of the current block, and calculating a second rate distortion cost after the motion compensation;

performing motion compensation on the current block according to the zero vector, and calculating a third rate distortion cost after motion compensation;

selecting a minimum rate distortion cost from the first rate distortion cost, the second rate distortion cost and the third rate distortion cost, and taking a motion vector corresponding to the minimum rate distortion cost as an initial prediction motion vector of the current block;

the manner of motion compensation for the current block is as follows:

dividing the current block into a plurality of sub-blocks;

the method for performing motion compensation on the plurality of sub-blocks by adopting a pipeline structure specifically comprises the following steps:

the first step, calculating coordinates of a reference block corresponding to the sub-block;

step two, obtaining pixel values of the reference block according to the coordinates of the reference block;

thirdly, calculating a predicted pixel based on pixel value interpolation of the reference block;

fourth, storing the predicted pixel;

the four steps of each sub-block are performed in a serial manner; and performing respective motion compensation between the sub-blocks according to a running water structure.

2. The method of claim 1, wherein in the solving of the system of linear equations, both the divisor and the dividend of the division operation are replaced with numbers corresponding to powers of 2 nearest thereto to replace the division operation with a shift operation.

3. An affine motion estimation device, characterized by comprising:

the initialization module is used for initializing the current block to obtain attribute information of the current block; the attribute information includes the size of the current block and the position of the current block in the current frame;

a determining module, configured to determine an initial predicted motion vector of the current block according to the attribute information;

the affine motion estimation module is used for carrying out affine motion estimation on the current block based on the initial prediction motion vector to obtain an affine motion vector;

the iteration updating module is used for iteratively updating the affine motion vector along the direction of the gradient decline of the mean square error based on the linear equation set for calculating the affine motion vector, and obtaining the optimal affine motion vector of the current block after the iteration is completed;

the determining module is specifically configured to:

according to the attribute information, obtaining a translational motion vector of the current block and an affine motion vector of an adjacent block;

performing motion compensation on the current block according to affine motion vectors of adjacent blocks, and calculating a first rate distortion cost after motion compensation;

performing motion compensation on the current block according to the translational motion vector of the current block, and calculating a second rate distortion cost after the motion compensation;

performing motion compensation on the current block according to the zero vector, and calculating a third rate distortion cost after motion compensation;

selecting a minimum rate distortion cost from the first rate distortion cost, the second rate distortion cost and the third rate distortion cost, and taking a motion vector corresponding to the minimum rate distortion cost as an initial prediction motion vector of the current block;

the mode of the determining module for performing motion compensation on the current block is as follows:

dividing the current block into a plurality of sub-blocks;

the method for performing motion compensation on the plurality of sub-blocks by adopting a pipeline structure specifically comprises the following steps:

the first step, calculating coordinates of a reference block corresponding to the sub-block;

step two, obtaining pixel values of the reference block according to the coordinates of the reference block;

thirdly, calculating a predicted pixel based on pixel value interpolation of the reference block;

fourth, storing the predicted pixel;

the four steps of each sub-block are performed in a serial manner; and performing respective motion compensation between the sub-blocks according to a running water structure.

4. An affine motion estimation device comprising: memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor executes to implement the method according to any of claims 1 to 2 when the computer program is run.

5. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, is to implement the method of any of claims 1 to 2.