CN111247804A - Method and device for image processing - Google Patents

Abstract

A method and a device for processing an image are provided, the method comprises the following steps: acquiring a motion vector CPMV of a control point of an image block; and obtaining a motion vector of the sub image block in the image block according to the CPMV of the image block, wherein the motion vector has integer pixel precision. By making the motion vector of the sub-image block serving as the image processing unit be integer-pixel precision, the motion compensation process of the sub-image block can be made to not involve sub-pixels, so that the bandwidth pressure generated by the Affinie prediction technology can be reduced to a certain extent.

Description

Method and device for image processing

Copyright notice

The disclosure of this patent document contains material that is subject to copyright protection. This copyright belongs to the copyright owner. The copyright owner has no objection to the reproduction by anyone of the patent document or the patent disclosure as it exists in the official records and archives of the Patent and Trademark Office.

technical field

The present application relates to the field of image processing, and more particularly, to a method and apparatus for image processing.

Background technique

The general idea of inter-frame prediction in video coding is to use the temporal correlation between adjacent video frames, use the previously encoded reconstructed frame as a reference frame, and predict the current frame through motion estimation and motion compensation. Remove temporal redundancy information from video. The general process of inter-frame prediction includes motion estimation (Motion Estimation, ME) and motion compensation (Motion Compensation, MC). The current coding block of the current frame searches for the most similar block in the reference frame as the prediction block of the current block, and the relative displacement between the current block and its similar blocks is a motion vector (Motion Vector, MV). The process of motion estimation is the process of obtaining a motion vector after searching and comparing the current coding block of the current frame in the reference frame. Motion compensation is the process of obtaining predicted frames using MV and reference frames. The predicted frame obtained by motion compensation may be different from the original current frame, so it is necessary to pass the difference (residual) between the predicted frame and the current frame to the decoding end after transformation, quantization and other processes. The information of MV and reference frame is passed to the decoding end. The decoding end can reconstruct the current frame through the MV, the reference frame, and the difference between the predicted frame and the current frame.

Due to the continuity of natural object motion, the motion vector of an object between two adjacent frames is not necessarily exactly integer pixel units. To improve the accuracy of motion vectors, sub-pixel accuracy is proposed. For example, in the high efficiency video coding (HEVC) standard, a motion vector with 1/4 pixel precision is used for motion estimation of the luminance component. However, there are no samples at fractional pixels in digital video. Generally speaking, in order to achieve 1/K pixel accuracy estimation, the values of these fractional pixels must be approximately interpolated, that is, the line direction and the reference frame. K-fold interpolation is performed in the column direction, that is, the prediction block is searched in the reference frame after interpolation. In the process of interpolating the current block, the pixels in the current block and the pixels in the adjacent areas need to be used.

Typically, only traditional motion models (eg, translational motion) are considered in the inter prediction process. However, in the real world, there are many forms of motion, such as zooming, rotating, perspective motion and other irregular motions. In order to consider the above multi-motion forms, in VTM-3.0, the affine motion compensation prediction (Affine motion compensation prediction, which can be referred to as Affine for short) technology is introduced. In Affine mode, the affine motion field of an image block can be derived from motion vectors of two control points (four parameters) or three control points (six parameters).

In Affine mode, motion vectors with 1/4 pixel precision, 1/16 pixel precision or other sub-pixel precision can be used for motion estimation of the image processing unit. The image processing unit of the Affine technology is a sub-CU (may be called a sub-block), and the size of the sub-CU is 4×4 (unit: pixel), which will cause the Affine technology to generate greater bandwidth pressure.

SUMMARY OF THE INVENTION

The present application provides an image processing method and apparatus, which can reduce the bandwidth pressure caused by the Affine prediction technology to a certain extent.

In a first aspect, an image processing method is provided, the method comprising: acquiring a motion vector CPMV of a control point of an image block; acquiring a motion vector of a sub-image block in the image block according to the CPMV of the image block, the The motion vectors are described with integer pixel precision.

In a second aspect, an apparatus for image processing is provided, the apparatus comprising: a first acquiring unit for acquiring a motion vector CPMV of a control point of an image block; a second acquiring unit for acquiring according to the first acquiring unit The CPMV of the image block is obtained, and the motion vector of the sub-image block in the image block is obtained, and the motion vector is of integer pixel precision.

In a third aspect, an image processing apparatus is provided, the encoding apparatus includes a memory and a processor, the memory is used for storing instructions, the processor is used for executing the instructions stored in the memory, and the memory is stored in the memory. Execution of the instructions causes the processor to perform the method provided by the first aspect.

In a fourth aspect, a chip is provided, the chip includes a processing module and a communication interface, the processing module is configured to control the communication interface to communicate with the outside, and the processing module is further configured to implement the method provided in the first aspect.

A fifth aspect provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, enables the computer to implement the method in the first aspect or any possible implementation manner of the first aspect .

A sixth aspect provides a computer program product comprising instructions, which when executed by a computer cause the computer to implement the method provided by the first aspect.

In the solution provided by the present application, by making the motion vector of the sub-image block as the image processing unit to be of integer pixel precision, the motion compensation process of the sub-image block does not involve sub-pixels, thereby reducing the amount of noise generated by the Affine prediction technology to a certain extent. Bandwidth pressure.

Description of drawings

Figure 1 is a schematic diagram of a video coding architecture.

Figure 2 is a schematic diagram of 1/4 pixel interpolation.

3(a) and 3(b) are schematic diagrams of the four-parameter Affine model and the six-parameter Affine model, respectively.

Figure 4 is a schematic diagram of the Affine motion vector field.

FIG. 5 is a comparison diagram of reference pixels required by the prior art Affine mode and HEVC mode.

FIG. 6 is a schematic flowchart of an image processing method according to an embodiment of the present application.

FIG. 7 is another schematic flowchart of an image processing method according to an embodiment of the present application.

FIG. 8 is still another schematic flowchart of an image processing method according to an embodiment of the present application.

FIG. 9 is a schematic flowchart of an apparatus for image processing according to an embodiment of the present application.

FIG. 10 is another schematic flowchart of an apparatus for image processing according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein in the specification of the application are for the purpose of describing specific embodiments only, and are not intended to limit the application.

To facilitate understanding of the solutions according to the embodiments of the present application, several related concepts are first described below.

1. Inter prediction

As shown in Figure 1, the video coding framework mainly includes several parts: intra-frame prediction, inter-frame prediction, transformation, quantization, entropy coding, and loop filtering.

This application mainly improves the inter prediction part.

The general idea of inter-frame prediction is to use the temporal correlation between adjacent frames of the video, use the reconstructed frame as a reference frame, and perform motion estimation (Motion Estimation, ME) and motion compensation (Motion Compensation, MC) for the current frame. Prediction is performed to remove the temporal redundancy information of the video.

The current frame mentioned in this article, in the encoding scenario, represents the frame currently being encoded, and in the decoding scenario, represents the frame currently being decoded.

The reconstructed frame mentioned in this paper, in the encoding scenario, refers to the frame that has been previously encoded, and in the decoding scenario, refers to the frame that has been previously decoded.

For a frame of image, the whole frame of image is not directly processed during the encoding process, and the whole frame of image is usually divided into image blocks for processing.

As an example, first divide the whole frame of image into coding areas (Coding Tree Unit, CTU), for example, the size of the CTU is 64×64 or 128×128 (unit: pixel), and then the CTU can be further divided into square or rectangular Coding unit (Coding Unit, CU). During the encoding process, the CU is processed.

The size of the image block mentioned in this article is in pixels.

The general flow of inter prediction is as follows.

For the current image block in the current frame (hereinafter simply referred to as the current block), the most similar block in the reference frame is found as the prediction block of the current block. The relative displacement between the current block and similar blocks is called a Motion Vector (MV). Motion estimation refers to the process of obtaining a motion vector after searching and comparing the current block of the current frame in the reference frame. Motion compensation refers to the process of obtaining a prediction block by using a reference block and a motion vector obtained by motion estimation.

The predicted block obtained in the process of inter-frame prediction may be different from the original current block. Therefore, it is necessary to calculate the difference between the predicted block and the current block, and the difference may be called residual. After transforming, quantizing, and entropy coding the residual, the coded bit stream is obtained.

At the encoding end, after the image encoding is completed, that is, after the bit stream obtained by entropy encoding, the bit stream and encoding mode information, such as inter-frame prediction mode, motion vector information and other information, are stored or sent to the decoding end.

At the decoding end, after obtaining the entropy coded bit stream, entropy decoding is performed on the bit stream to obtain the corresponding residual; then, the prediction block is obtained according to the coding mode information such as the motion vector obtained by decoding; finally, the prediction block is obtained according to the residual and prediction block, obtain the value of each pixel in the current block, that is, reconstruct the current block, and so on, reconstruct the current frame.

As shown in FIG. 1 , in the encoding process, steps such as inverse quantization and inverse transformation may also be included. Inverse quantization refers to the reverse process of quantization. The inverse transformation refers to the reverse process of the transformation process.

Inter-frame prediction mainly includes forward prediction, backward prediction, bi-prediction and so on. The forward prediction is to predict the current frame by using the previous reconstructed frame (which may be referred to as a historical frame) of the current frame. Backward prediction is the prediction of the current frame using the frame following the current frame (which may be called a future frame). Bi-prediction may be bi-directional prediction, ie using both "historical frames" and "future frames" to predict the current frame. Bi-prediction can also be prediction in two directions, eg, using two "historical frames" to predict the current frame, or using two "future frames" to predict the current frame.

2. Sub-pixel precision motion estimation

In actual scenes, due to the continuity of motion of natural objects, the motion vector of objects between two adjacent frames is not necessarily exactly integer pixel units. Therefore, it is necessary to improve the accuracy of motion estimation to sub-pixel level (also known as 1/K pixel precision). For example, in the HEVC standard, a motion vector with 1/4 pixel precision is used for motion estimation of the luminance component.

However, there is no sample value at 1/K pixel in digital video. Usually, in order to achieve motion estimation with 1/K pixel precision, the value of 1/K pixel point is approximately interpolated. K-fold interpolation is performed in the direction and column direction, and the search is performed in the image after the interpolation. The process of interpolating the current block needs to use the pixels in the current block and the pixels in the adjacent areas.

As an example, the process of 1/4 pixel interpolation is shown in Figure 2. For an image block of size 8×8, 4×8, 4×4, or 8×4, the 3 pixels on the left and 4 pixels on the right outside the block are used to generate interpolation points pixel value. As shown in Figure 2, for an image block of size 4×4, a _0,0 and d _0,0 are 1/4 pixels, b _0,0 and h _0,0 are half pixels, and c _{0, 0} and n _0,0 are 3/4 pixels. Suppose that the current block is a 2×2 block, A _0,0 to A _1,0 and A _0,0 to A _0,1 are 2×2 blocks. In order to calculate all the interpolated points in this 2x2 block, some points outside the 2x2 need to be used, including 3 on the left, 4 on the right, 3 on the top, and 4 on the bottom.

3. Affine motion compensated prediction (Affine motion compensated prediction, hereinafter referred to as Affine).

Affine is an inter-frame prediction technology.

In the HEVC standard, the inter prediction process only takes into account traditional motion models (eg, translational motion). However, in the real world, there are many forms of motion, such as zooming, rotating, perspective motion and other irregular motions. In order to take into account the above motion forms, in VTM-3.0, Affine technology is introduced.

As shown in Fig. 3, an Affine mode motion field can be controlled by two control points (four parameters) (as shown in Fig. 3(a)) or three control points (six parameters) (as shown in Fig. 3(b)) Motion vector export.

Hereinafter, the MV (control point motion vector) of the control point is abbreviated as CPMV.

The processing unit of Affine is not a CU, but a sub-block (sub-CU) obtained by dividing the CU, and the size of each sub-CU is 4×4. In Affine mode, each sub-CU has one MV. It can be understood that, unlike a normal CU, a CU in Affine mode does not have only one MV, and the CU has as many MVs as there are sub-CUs in a CU.

As an example, the MV of a sub-CU in one CU is derived by CPMV calculation of two control points or three control points as shown in FIG. 3 . For example, for the four-parameter Affine motion model, the MV of the sub-CU at the (x, y) position is calculated by the following formula:

For another example, for the six-parameter Affine motion model, the MV of the sub-CU located at the (x, y) position is calculated by the following formula:

Where (mv _0x , mv _0y ) is the MV of the upper left control point, (mv _1x , mv _1y ) is the MV of the upper right control point, (mv _2x , mv _2y ) is the MV of the lower left control point. W in the above formula represents the width of the CU where the sub-CU is located, and H represents the height of the CU where the sub-CU is located.

After the calculation of the above formula (1), a schematic diagram of a motion vector in a CU is shown in FIG. 4 , and each square represents a sub-CU with a size of 4×4. After the calculation of the above formula, the MV of all sub-CUs will be converted into a representation of 1/16 pixel precision, that is to say, the highest precision of the MV of a sub-CU is 1/16 pixel.

After calculating the MV of each sub-CU, the prediction block of each sub-CU is obtained through the process of motion compensation. The size of the sub-CUs of the chrominance component and the luminance component are both 4×4, and the motion vector of the 4×4 block of the chrominance component is obtained by averaging the corresponding four motion vectors of the luminance component of 4×4.

In the encoding process of the Affine mode, the CPMV information is written in the code stream, and the MV information of each sub-CU does not need to be written.

4. Adaptive Motion Vector Resolution (AMVR)

AMVR technology can enable the CU to have integer-pixel precision and sub-pixel precision motion vectors. The integer pixel precision may be, for example, 1 pixel precision, 2 pixel precision, or the like. The sub-pixel precision may be, for example, 1/2 pixel precision, 1/4 pixel precision, 1/8 pixel precision, or 1/16 pixel precision.

For example, for each CU that uses Affine AMVR technology (in some cases, the CU may not use Affine AMVR), the encoding end adaptively determines its corresponding MV precision, and writes the decision result into the code stream and transmits it to the decoding end.

The integer pixel precision or sub-pixel precision mentioned in Affine AMVR technology refers to the pixel precision of CPMV, not the pixel precision of sub-CU.

For the CPMV of an integer pixel, the motion estimation process of the CU is the process of the whole pixel, but the MV of the sub-CU obtained after the calculation of the above formula (1) or formula (2) may be 1/4 pixel precision or other sub-CUs. pixel precision.

If the MV of the sub-CU is sub-pixel precision, the motion compensation process of the sub-CU will involve sub-pixels, and since the size of the sub-CU is 4×4, this will cause a large bandwidth pressure to the Affine prediction process.

On VVC's latest reference software VTM-4.0, the applicant selected the official pass-through test data as the test sequence, and carried out a simulation. The simulation results are shown in Figure 5.

As shown in Figure 5, the left box represents the worst-case (MV of 1/4 pixel precision) of HEVC bi-directional inter-prediction CU of 8×8, and the required number of reference pixels is (8+7) ×(8+7)×2=450. The box on the right represents the CU of 4×4 bidirectional inter-frame prediction in the worst case (1/16 and 1/4 pixel precision MV) in the Affine mode of VVC, and the number of required reference pixels is (4+7 )×(4+7)×2×4=968.

It can be seen from Fig. 5 that, compared with HEVC, the existing Affine mode increases the reference pixels by 115%, which causes a large bandwidth pressure.

In view of the above problems, the present application proposes an image processing method and apparatus, which can reduce the bandwidth pressure generated by the Affine technology to a certain extent.

The present application applies to the technical field of digital video coding, and is specifically used for the inter-frame prediction part of a video codec. The present application can be applied to codecs conforming to international video coding standards H.264/HEVC and Chinese AVS2 standards, etc., as well as codecs conforming to next-generation video coding standards VVC or AVS3, etc.

The present application may be applied to the inter-frame prediction part of a video codec, that is, the image processing method according to the embodiment of the present application may be performed by an encoding apparatus or a decoding apparatus.

FIG. 6 is a schematic flowchart of a method 600 for image processing provided by the present application. The method 600 includes the following steps.

610. Obtain the motion vector (CPMV) of the control point of the image block.

The manner of acquiring the CPMV of the image block will be described below, which will not be described here for the time being.

620. Acquire a motion vector of a sub-image block in the image block according to the CPMV of the image block, where the motion vector is of integer pixel precision.

In other words, based on the CPMV of the image block, the motion vector of the sub-image block in the image block is obtained, and the pixel precision of the motion vector of the sub-image block is made to be an integer pixel precision.

Sub-image blocks referred to in this application represent processing units of image processing or video processing. The width and/or height of the sub-image block may be less than 8 pixels. For example, the size of the sub-image block is 4×4 (pixels).

The sub image block may be a block obtained by dividing the image block. It can be understood that if the size of the image block and the sub-image block are the same, the sub-image block can be regarded as the image block itself.

The sub-image block may be a square block, such as a block with a size of 4×4 or 8×8, or a rectangular block, such as a block with a size of 2×4 or 4×8.

The size of the image block mentioned in this application may be 16×16, 16×8, 16×4, 8×16, 4×8, 8×8, 8×4, 4×8 and other sizes.

It should be understood that the motion vector of the sub-image block as the processing unit is of integer pixel precision. Therefore, the motion compensation process of the sub-image block does not involve sub-pixels, so that the bandwidth pressure generated by the video inter-frame prediction process can be reduced.

According to the CPMV of the image block, the process of obtaining the motion vector of the sub-image block in the image block may include: calculating and obtaining the motion vector of the sub-image block according to the motion vectors of two or three control points of the image block, and making the obtained motion vector The pixel precision of the motion vector of the sub-image block is integer pixel precision.

As an example, the motion vector of the sub-image block can be calculated according to the formula (1) or formula (2) described above.

Optionally, in some embodiments, if the pixel precision of the motion vector of the sub-image block obtained directly based on the CPMV of the image block is an integer pixel precision, then this motion vector is the motion vector of the sub-image block to be acquired by the present application. .

For example, as a possible implementation manner, an algorithm is used to calculate the motion vector of the sub-image block according to the CPMV of the image block, and the algorithm can ensure that the pixel precision of the calculated motion vector of the sub-image block is an integer pixel.

Optionally, in some embodiments, if directly based on the CPMV of the image block, the pixel precision of the motion vector of the sub-image block obtained by calculation is sub-pixel precision, for example, 1/4 pixel precision, 1/8 pixel precision or 1/4 pixel precision. /16 pixel precision, it is also necessary to process the currently calculated motion vector to change it from sub-pixel precision to integer pixel precision.

Optionally, step 620 includes the following steps 1) and 2).

1) Calculate the first motion vector of the sub-image block according to the CPMV of the image block, and the first motion vector is of sub-pixel precision.

For example, according to the formula (1) or formula (2) described above, the first motion vector of the sub-image block is calculated based on the CPMV, and the pixel precision of the calculated first motion vector is sub-pixel.

2) Process the first motion vector into a second motion vector with integer pixel precision.

As a possible implementation manner of step 2): obtain the second motion vector according to the first motion vector of the sub-image block, so that the end point of the second motion vector is the closest integer pixel to the end point of the first motion vector.

For example, the closest integer pixel point may be an integer pixel point above, below, to the left or to the right of the end point of the first motion vector.

As an example, through the following formula, the second motion vector (MV2x, MV2y) of the sub-image block is obtained by calculation according to the first motion vector (MV1x, MV1y) of the sub-image block.

If MV1x>=0, MV2x=((MV1x+(1<<(shift-1)))>>shift)<<shift;

If MV1x<0, MV2x=-((-MV1x+(1<<(shift-1)))>>shift)<<shift;

If MV1y>=0, MV2y=((MV1y+(1<<(shift-1)))>>shift)<<shift;

If MV1y<0, MV2y=-((-MV1y+(1<<(shift-1)))>>shift)<<shift,

Formula (3).

The value of shift is related to the storage precision of the motion vector in the encoding software platform. For example, in the current VTM-4.0 reference software, the storage precision of the motion vector is 1/16 precision, so the value of shift can be set to 4.

As another example, the second motion vector (MV2x, MV2y) of the sub-image block is obtained according to the first motion vector (MV1x, MV1y) of the sub-image block by the following formula.

If MV1x>=0, MV2x=(MV1x>>shift)<<shift;

If MV1x<0, MV2x=-(((-MV1x)>>shift)<<shift);

If MV1y>=0, MV2y=(MV1y>>shift)<<shift;

If MV1y<0, MV2y=-(((-MV1y)>>shift)<<shift), formula (4).

The meaning of shift is the same as the meaning of shift described above.

"<<" in formula (3) and formula (4) means left shift, and ">>" means right shift.

It should be noted that the present application does not limit the manner in which the pixel precision of the motion vector is converted from the sub-pixel level to the integer pixel level. For example, the second motion vector with integer pixel precision can also be obtained according to the first motion vector according to other feasible transformation algorithms from subpixels to integer pixels.

When the size of the smallest CU (corresponding to the image block in the embodiment of the present application) processed in the current Affine technology is 16×16, the motion estimation process will not bring pressure on the bandwidth. Therefore, the motion estimation process does not need to be Revise. In this case, the pixel precision of the CPMV of the image block may be an integer pixel or a sub-pixel. If the pixel accuracy of the CPMV of the image block is sub-pixel, then the pixel accuracy of the motion vector of the sub-image block calculated according to the CPMV of the image block is also sub-pixel; if the pixel accuracy of the CPMV of the image block is an integer pixel, according to the image block The pixel precision of the motion vector of the sub-image block calculated by the CPMV of , may also be sub-pixel, for example, the pixel precision of the motion vector of the sub-image block calculated according to formula (1) or formula (2) may be sub-pixel.

As can be seen from the above, in the existing Affine technology, the pixel precision of the sub-image block, that is, the motion vector of the processing unit, may be sub-pixel, which will cause the motion compensation process to involve sub-pixels, which will increase the bandwidth pressure of the Affine technology.

In the solution provided by the present application, by making the motion vector of the sub-image block as the image processing unit to be of integer pixel precision, the motion compensation process of the sub-image block does not involve sub-pixels, thereby reducing the amount of noise generated by the Affine prediction technology to a certain extent. Bandwidth pressure.

It should be understood that the problem of bandwidth pressure can also be alleviated to a certain extent by enlarging the size of the sub-image block as a processing unit, but this will reduce the image compression performance. In the present application, by processing the motion vector of the sub-image block as the processing unit to the integer pixel precision, the motion compensation of the integer pixel precision can be ensured, so that the problem of bandwidth pressure can be solved on the one hand, and better image compression can also be ensured on the other hand. performance.

According to the solution provided in this application, the existing Affine technology can be improved, that is, the motion vector of the Sub-CU in the Affine mode is processed into integer pixel precision, so that the bandwidth pressure generated by the Affine technology can be reduced.

In addition to being applicable to the Affine technology, the solution provided in this application can also be applied to other similar technologies that may appear in the future. For example, the pixel precision of the motion vector includes integer pixel precision and sub-pixel precision, and the size of the image processing unit Smaller, for example, 4×4.

It should be understood that the solution provided by this application can be used to improve the quality of compressed video and improve the hardware friendliness of codecs, and is of great significance to the compression processing of videos such as broadcast television, video conferences, and online videos.

Optionally, in some embodiments, the methods provided by the embodiments of the present application further include: processing the CPMV of the image block to integer pixel precision.

This embodiment can ensure that the CPMV of the image block is of integer pixel precision.

An embodiment of processing the CPMV of the image block to integer pixel precision will be described below.

Optionally, as shown in FIG. 7 , in some embodiments, step 610 includes the following steps 611 , 612 and 613 .

611. Obtain a motion information candidate list of the image block.

For example, the motion vectors of adjacent blocks in the spatial domain and/or the temporal domain of the image block are obtained, and based on the motion vectors of these adjacent blocks, a motion information candidate list of the image block is constructed.

612. Process the motion vectors in the motion information candidate list into integer pixel precision.

For example, the above-described formula (3) or formula (4) can be used to process the motion vectors in the motion information candidate list to integer pixel precision.

Neighboring blocks refer to neighboring blocks used to construct the motion information candidate list of the image block, eg, neighboring blocks in the temporal and/or spatial domains. The present application does not limit the manner of determining adjacent blocks.

613. Obtain the CPMV of the image block according to the motion vector in the motion information candidate list that is processed as an integer pixel precision.

Affine inter prediction mode can be divided into Affine merge mode and Affine inter mode.

The embodiment shown in FIG. 7 can be applied to the Affine inter mode and also can be applied to the Affine merge mode.

Optionally, in the embodiment shown in FIG. 7 , the inter-frame prediction mode of the image block is an Affine merge mode.

In the Affine merge mode, a CPMV can be directly selected from the motion information candidate list as the CPMV of the image block. That is, step 613 includes: selecting a CPMV from the motion information candidate list of the image block as the CPMV of the image block.

Because the motion vectors of adjacent blocks used to construct the motion information candidate list are processed to integer pixel precision, selecting the CPMV from the motion information candidate list directly as the CPMV of the image block can ensure that the CPMV of the image block is an integer pixel.

As an example, the general flow of inter-frame prediction in Affine merge mode includes the following steps. In this example, the image block is taken as the CU as an example.

Step 1-1, obtain the motion vector (MV) of the adjacent block from the adjacent block in the spatial domain and/or the adjacent block in the temporal domain. In this process, MVs of adjacent blocks in Affine mode and MVs of adjacent blocks in traditional mode are obtained, CPMVs are obtained by combining the MVs of these adjacent blocks, and a motion information candidate list of the CU is constructed from these CPMVs.

Step 1-2: Process the motion vectors in the motion information candidate list of the CU into integer pixel precision.

Step 1-3, select a combination from the motion information candidate list (this combination may include two or three CPMVs, representing CPMVs of two control points and three control points) as the CPMVs of the CU.

In the Affine merge mode, the CPMVs selected in the motion information candidate list are used as the CPMVs of the current CU, no motion estimation is required, and there is no concept of MVD in the Affine inter mode (described below). That is to say, in the Affine merge mode, only the index of the CPMVs selected from the motion information candidate list (one CU only needs to write one index) needs to be written into the code stream, and there is no need to transmit the MVD.

Regarding the adjacent block mentioned in step 1-1, the inter prediction mode of the adjacent block can be the traditional inter prediction mode or the affine mode, so the MV obtained from the adjacent block may be integer pixel precision or may be Subpixel accuracy.

In this embodiment, the motion vectors of the adjacent blocks of the current image block are processed to the integer pixel precision, so that the CPMV of the image block can be guaranteed to be the integer pixel precision.

As mentioned above, the embodiment shown in FIG. 7 can also be applied to the Affine inter mode. In order to better understand the embodiments of the present application, before describing the embodiment in which the embodiment shown in FIG. 7 is applied to the Affine inter mode, a general flow of the Affine Inter mode is described first.

As an example, the general flow of the Affine Inter mode includes the following steps. In this example, the image block is taken as the CU as an example.

Step 2-1: Obtain motion vectors of adjacent blocks from adjacent blocks in the spatial domain and/or adjacent blocks in the temporal domain. In this process, the motion vectors of the adjacent blocks in the Affine mode and the motion vectors of the adjacent blocks in the traditional mode are obtained; CPMVs are obtained by combining the obtained motion vectors, and the motion information candidate list of the CU is constructed from these CPMVs.

Step 2-2, select a combination from the motion information candidate list constructed in step 2-1 (this combination may contain two or three CPMVs, representing the CPMVs of two control points and three control points), as the current CU The predicted MV (Motion vector prediction, MVP) (that is, the predicted CPMVs of the current CU).

Step 2-3, perform motion estimation with the current entire CU as a unit, and obtain the CPMVs of the current CU.

Step 2-4: Calculate the difference between the CPMVs selected in step 2-2 and the CPMVs estimated by motion in step 2-3 to obtain a motion vector difference (Motion Vector Difference, MVD).

In Affine Inter mode, the index of the selected CPMVs and the MVD need to be written into the code stream.

In the Affine Inter mode, the motion estimation process is performed in units of CUs (corresponding to the image blocks in the embodiments of the present application), and the motion compensation process is performed in 4×4 sub-CUs (corresponding to the sub-images in the embodiments of the present application) block) as a unit.

Regarding the adjacent block mentioned in step 2-1, the inter prediction mode of the adjacent block may be the traditional inter prediction mode or the affine mode, so the MV obtained from the adjacent block may be integer pixel precision or may be Subpixel accuracy.

In the Affine Inter mode, the encoding end will select different pixel precisions of the motion vector of the CU, and this process may be called an Adaptive Motion Vector Resolution (Adaptive Motion Vector Resolution, AMVR) decision.

The pixel precision of AMVR decisions is essentially the pixel precision of the MVD, that is, the pixel precision of the CU's CPMVs, not the pixel precision of the sub-CU's MVs.

In the existing Affine Inter mode, the range of pixel precision for AMVR decisions includes but is not limited to: 1/16 pixel precision, 1/8 pixel precision, 1/4 pixel precision, 1/2 pixel precision, 1 pixel precision, 2 Pixel precision, 4 pixel precision, etc. In other words, a CU can have multiple CPMVs with different pixel precisions. For example, a CU can have three different CPMVs: integer pixel, 1/4 pixel precision, and 1/16 pixel precision.

Optionally, in the embodiment shown in FIG. 7 , the inter-frame prediction mode of the image block is the Affine Inter mode, and step 611 includes acquiring the motion information candidate list of the image block; step 612 includes obtaining the motion information candidate list in the motion information candidate list. The motion vector processing is integer pixel precision; Step 613 includes: selecting the predicted CPMV of the image block from the motion information candidate list of the image block to obtain the MVD of the image block, and the predicted CPMV of the image block and the MVD of the image block to obtain the image block. The CPMV of the image block.

As shown in FIG. 8 , in this embodiment, step 610 may further include step 614 , performing a motion vector precision decision of N pixels for the image block, where N is a positive integer.

That is, a motion vector precision decision (AMVR decision) with integer pixel precision is performed on the image block.

It can be understood that by performing AMVR decision on the image block with integer pixel accuracy, the pixel accuracy of the MVD of the image block can be guaranteed to be an integer pixel, and the pixel accuracy of the CPMV of the image block can also be guaranteed to be an integer pixel. In this way, it can be ensured that no sub-pixels are involved in the motion estimation process of the image block, so that the bandwidth pressure can be reduced to a certain extent.

In this embodiment, when using Affine AMVR to make motion vector precision decisions, all pixel precisions are not decided, but the decision of 1/M (M>1) pixel precision is skipped, that is, only N pixel precisions are made. decision.

It should be understood that, in this embodiment, when the motion vector precision index is written into the code stream, since the pixel precision options are reduced, the number of bits (bit number) written into the code stream is correspondingly reduced, and it is even unnecessary to write motion representation The number of bits for the vector precision index. For example, the original pixel precision options include three: integer pixel, 1/4 pixel and 1/16 pixel, then at least 2 bits of information are required to represent these three pixel precisions, for example, "0" is used to represent 1/4 pixel, "10" means 1/16 of a pixel, and "11" means a whole pixel. In this embodiment, "0" can be used to represent an integer pixel, so only 1-bit data needs to be written in the code stream, or, an integer pixel precision can be agreed upon in the protocol, so there is no need to write the motion vector precision index This saves signaling overhead and reduces bandwidth pressure.

It should be noted that when the inter-frame prediction mode of the image block is the Affine inter mode, the embodiment of performing the motion vector precision decision of N (N is a positive integer) pixel for the image block is the same as the embodiment shown in FIG. 8 . It can be implemented in combination, or can be implemented independently from the embodiment shown in FIG. 8 .

Optionally, as shown in FIG. 8 , in some embodiments, the inter prediction mode of the image block is Affine inter mode, and step 610 includes: acquiring the CPMV of the image block, and performing motion vector precision of N pixels on the image block. decision, N is a positive integer.

It should be understood that by making a motion vector precision decision of N pixels for the image block, whether the motion vectors of adjacent blocks of the image block are processed to integer pixel precision, the CPMV of the image block can be guaranteed to be integer pixel precision.

It should also be understood that, in the Affine Inter mode, by processing the pixel precision of the CPMV of the image block to the integer pixel precision, the motion estimation of the integer pixel precision can be guaranteed, which helps to reduce the bandwidth pressure.

It can be seen from the above that, in the Affine merge mode, the implementation manner of processing the pixel precision of the CPMV of the image block to the integer pixel precision is: processing the motion vector in the motion information candidate list to the integer pixel precision.

In the Affine inter mode, the implementation of processing the pixel precision of the CPMV of the image block to the integer pixel precision is as follows: processing the motion vector in the motion information candidate list to the integer pixel precision, and processing the image block with the integer pixel precision AMVR decision.

Or, in the Affine inter mode, the implementation manner of processing the pixel precision of the CPMV of the image block to the integer pixel precision is: performing AMVR decision of the image block with the integer pixel precision.

In the above-mentioned embodiments involving processing motion vectors of adjacent blocks to integer pixel precision, the motion vectors of adjacent blocks may be processed to integer pixel precision in the manner shown in the above formula (3) or formula (4). Other feasible algorithms or methods for converting sub-pixels to whole pixels can also be used to process the motion vectors of adjacent blocks to whole-pixel precision. This application does not limit this.

Optionally, in some embodiments, when the size of an image block is smaller than a threshold, the CPMV of the image block is processed to integer pixel precision.

The threshold can be determined according to actual needs. For example, the threshold is 16 pixels.

For example, when the height and/or width of an image block is less than 16 pixels, the CPMV of the image block is processed to integer pixel precision.

As can be seen from the Affine Inter mode described above, in the Affine Inter mode, motion estimation in units of image blocks will be performed. For example, when the height and width of the image block are equal to or greater than 16 pixels, even the sub-pixel precision motion estimation process will not cause large bandwidth pressure. In this case, the CPMV of the image block may not be processed to make it becomes integer pixel precision.

However, if the height and/or width of the image block is less than 16 pixels, for example, the size of the image block is 4×8, 8×4, 4×16 or 16×4, the motion estimation process with sub-pixel precision may cause large bandwidth pressure. In this case, the CPMV of the image block can be processed to integer pixel precision.

Optionally, in some embodiments, the prediction mode of the image block is Affine Inter mode, and the height and/or width of the image block is less than 16 pixels, the method according to this embodiment of the present application further includes: performing integer pixel precision on the image block AMVR decision.

This embodiment can ensure a motion estimation process with integer pixel precision, so as to avoid causing a large bandwidth pressure.

In addition, when the motion vector precision index of the image block that satisfies the condition that the height and/or width is less than 16 pixels is written into the code stream, the number of bits written into the code stream can be reduced due to the reduction of pixel precision options.

For example, for a CU whose height and width are greater than or equal to 16 pixels, select AMVR pixel precision among three modes: integer pixel, 1/4 pixel and 1/16 pixel, for example, use "0" for 1/4 pixel, "10" " for 1/16 of a pixel, and "11" for a whole pixel. For CUs whose height and/or width are less than 16 pixels, because there is only one option of AMVR pixel precision, it is not necessary to write the AMVR pixel precision index into the code stream, for example, integer pixel precision can be adopted by agreement.

The embodiments of the present application may be applied to different inter-frame prediction modes, for example, forward prediction, backward prediction, or bi-prediction. In other words, the inter-frame prediction mode of the sub-image block mentioned in the embodiments of the present application may be any of the following: forward prediction, backward prediction, and bi-prediction.

For example, if the inter-frame prediction mode of the sub-image block is forward prediction, the motion vector of the sub-image block obtained in the forward prediction process is processed as an integer pixel.

For another example, if the inter-frame prediction mode of the sub-image block is backward prediction, the motion vector of the sub-image block obtained by the backward prediction process is processed as an integer pixel.

For another example, if the inter-frame prediction mode of the sub-image block is bi-prediction, the motion vector of the sub-image block obtained in the bi-prediction process is processed as an integer pixel.

Optionally, the inter-frame prediction mode of the sub-image block is bi-prediction, but for only one prediction process in the bi-prediction, the method provided by the embodiment of the present application is used to process the motion vector of the sub-image block to integer pixel precision.

For example, the CPMV of the image block is the CPMV of the image block obtained by forward prediction in the bi-prediction process, or the CPMV of the image block obtained by the backward prediction in the bi-prediction process.

In other words, for example, if the inter-frame prediction mode of the sub-image block is bi-prediction, the motion vector of the sub-image block obtained by one prediction process of the bi-prediction process is processed as an integer pixel. This one prediction process may be a forward prediction process in bi-prediction, or a backward prediction process in bi-prediction.

It can be seen from the above that, in the solution provided by the present application, by making the motion vector of the sub-image block as the image processing unit to be of integer pixel precision, the motion compensation process of the sub-image block can not involve sub-pixels, so that the Affine prediction can be reduced to a certain extent. Technology-generated bandwidth pressure.

Further, by processing the pixel precision of the CPMV of the image block to the integer pixel precision, in the Affine Inter mode, the motion estimation of the integer pixel precision can be guaranteed, which helps to reduce the bandwidth pressure.

Therefore, the solution provided by the present application can reduce the bandwidth pressure caused by the inter-frame prediction process, and can also ensure a certain compression performance.

The method embodiments of the present application are described above, and the device embodiments of the present application will be described below. It should be understood that the description of the apparatus embodiment corresponds to the description of the method embodiment. Therefore, for the content that is not described in detail, reference may be made to the foregoing method embodiment, which is not repeated here for brevity.

As shown in FIG. 9 , an embodiment of the present application provides an apparatus 900 for image processing. The apparatus 900 includes the following units.

The first obtaining unit 910 is configured to obtain the motion vector CPMV of the control point of the image block.

The second obtaining unit 920 is configured to obtain, according to the CPMV of the image block obtained by the first obtaining unit 910, a motion vector of a sub-image block in the image block, and the motion vector is of integer pixel precision.

In the solution provided by the present application, by making the motion vector of the sub-image block as the image processing unit to be of integer pixel precision, the motion compensation process of the sub-image block does not involve sub-pixels, thereby reducing the amount of noise generated by the Affine prediction technology to a certain extent. Bandwidth pressure.

Optionally, in some embodiments, the second obtaining unit 920 is configured to: calculate a first motion vector of the sub-image block according to the CPMV of the image block, where the first motion vector is sub-pixel precision; One motion vector is processed as a second motion vector with integer pixel precision.

Optionally, in some embodiments, the second obtaining unit 920 is configured to obtain a second motion vector according to the first motion vector of the sub-image block, so that the end point of the second motion vector is the same as that of the first motion vector. The closest integer pixel to the end point.

For example, the second obtaining unit 920 is configured to, through formula (3) or formula (4), process the first motion vector into a second motion vector whose pixel precision is an integer pixel.

Optionally, in some embodiments, the height and/or width of the sub-image block is 4 pixels.

Optionally, in some embodiments, the first obtaining unit 910 is configured to: obtain a motion information candidate list of the image block, and process the motion vectors in the motion information candidate list into integer pixel precision; The list is processed as a motion vector with integer pixel precision, and the CPMV of the image block is obtained.

Optionally, in some embodiments, the apparatus 900 further includes: a processing unit 930, configured to perform a motion vector precision decision of N pixels for the image block, where N is a positive integer.

Optionally, in some embodiments, the height and/or width of the image block is less than 16 pixels.

Optionally, in some embodiments, the inter-frame prediction mode of the sub-image block is any one of the following: forward prediction, backward prediction, and bi-prediction.

Optionally, in some embodiments, the inter-frame prediction mode of the sub-image block is bi-prediction, wherein the CPMV of the image block is the CPMV of the image block obtained by forward prediction in the bi-prediction process, or the bi-prediction The CPMV of the image block obtained by backward prediction in the process.

Optionally, the apparatus 900 for image processing in this embodiment may be an encoder, and the apparatus 900 may further include functional modules for implementing video encoding related processes.

Optionally, the apparatus 900 for image processing in this embodiment may be a decoder, and the apparatus 900 may further include functional modules for implementing video decoding related processes.

As shown in FIG. 10 , an embodiment of the present invention further provides an apparatus 1000 for image processing. The apparatus 1000 includes a processor 1010 and a memory 1020, the memory 1020 is used to store instructions, the processor 1010 is used to execute the instructions stored in the memory 1020, and the execution of the instructions stored in the memory 1020 makes the processor 1010 A method for performing the above method embodiments.

Specifically, the encoding apparatus 1000 further includes a communication interface 1030 for transmitting signals with external devices.

Optionally, the image processing apparatus 1000 in this embodiment is an encoder, and the communication interface 1030 is configured to receive image or video data to be processed from an external device. Alternatively, the communication interface 1030 is further configured to send the encoded code stream to the decoding end.

Optionally, the apparatus 1000 for image processing in this embodiment is a decoder, and the communication interface 1030 is configured to receive an encoded code stream from an encoding end.

An embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a computer, the computer causes the computer to execute the method of the above method embodiments.

Embodiments of the present invention also provide a computer program product containing instructions, characterized in that, when the instructions are executed by a computer, the instructions cause the computer to execute the methods of the above method embodiments.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions according to the embodiments of the present invention result in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. Computer instructions may be stored on or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server, or data center over a wire (e.g. coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) means to another website site, computer, server or data center. A computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like that includes an integration of one or more available media. Useful media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, digital video disc (DVD)), or semiconductor media (eg, solid state disk (SSD)), and the like.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (21)

1. a method for image processing, is characterized in that, comprises: Obtain the motion vector CPMV of the control point of the image block; According to the CPMV of the image block, the motion vector of the sub-image block in the image block is obtained, and the motion vector is of integer pixel precision. 2. The method according to claim 1, wherein, based on the CPMV of the image block, obtaining the motion vector of the sub-image block in the image block, comprising: Calculate the first motion vector of the sub-image block according to the CPMV of the image block, where the first motion vector is sub-pixel precision; The first motion vector is processed into a second motion vector with integer pixel precision. 3. The method according to claim 1 or 2, wherein the sub-image block has a height and/or a width of 4 pixels. 4. The method according to any one of claims 1 to 3, wherein obtaining the motion vector CPMV of the control point of the image block, comprising: obtaining a motion information candidate list of the image block; processing the motion vectors in the motion information candidate list to integer pixel precision; The CPMV of the image block is acquired according to the motion vector in the motion information candidate list processed as an integer pixel precision. 5. The method according to any one of claims 1 to 4, wherein the method further comprises: A motion vector precision decision of N pixels is performed on the image block, where N is a positive integer. 6. The method according to any one of claims 1 to 5, wherein the image block has a height and/or a width of less than 16 pixels. 7. The method according to any one of claims 1 to 6, wherein the inter-frame prediction mode of the sub-image block is any one of the following: forward prediction, backward prediction, and bi-prediction. 8. The method according to any one of claims 1 to 6, wherein the inter-frame prediction mode of the sub-image block is bi-prediction, wherein the CPMV of the image block is the forward direction in the bi-prediction process The CPMV of the image block obtained by prediction, or the CPMV of the image block obtained by backward prediction in the bi-prediction process. 9. The method according to claim 2, wherein processing the first motion vector into a second motion vector with integer pixel precision, comprising: According to the first motion vector, the second motion vector is acquired, so that the end point of the second motion vector is an integer pixel point closest to the end point of the first motion vector. 10. A device for image processing, comprising: The first acquisition unit, for acquiring the motion vector CPMV of the control point of the image block; A second obtaining unit, configured to obtain a motion vector of a sub-image block in the image block according to the CPMV of the image block obtained by the first obtaining unit, where the motion vector is of integer pixel precision. 11. The apparatus according to claim 10, wherein the second acquiring unit is used for: Calculate the first motion vector of the sub-image block according to the CPMV of the image block, where the first motion vector is sub-pixel precision; The first motion vector is processed into a second motion vector with integer pixel precision. 12. The apparatus according to claim 10 or 11, wherein the sub-image block has a height and/or a width of 4 pixels. 13. The apparatus according to any one of claims 10 to 12, wherein the first acquiring unit is used for: obtaining a motion information candidate list of the image block; processing the motion vectors in the motion information candidate list to integer pixel precision; The CPMV of the image block is acquired according to the motion vector in the motion information candidate list processed as an integer pixel precision. 14. The device according to any one of claims 10 to 13, wherein the device further comprises: A processing unit, configured to perform motion vector precision decision of N pixels on the image block, where N is a positive integer. 15. The apparatus according to any one of claims 10 to 14, wherein the image block has a height and/or a width of less than 16 pixels. 16. The apparatus according to any one of claims 10 to 15, wherein the inter-frame prediction mode of the sub-image block is any one of the following: forward prediction, backward prediction, and bi-prediction. 17. The apparatus according to any one of claims 10 to 16, wherein the inter-frame prediction mode of the sub-image block is bi-prediction, wherein the CPMV of the image block is a forward direction in a bi-prediction process The CPMV of the image block obtained by prediction, or the CPMV of the image block obtained by backward prediction in the bi-prediction process. 18. The apparatus according to claim 11, wherein the second obtaining unit is configured to obtain the second motion vector according to the first motion vector, so that the end point of the second motion vector is The closest integer pixel to the end point of the first motion vector. 19. An image processing device, comprising: a memory and a processor, wherein the memory is used to store instructions, the processor is used to execute the instructions stored in the memory, and the processing of the instructions stored in the memory is processed. Executed such that the processor is configured to execute the method of any one of claims 1 to 9. 20. A computer storage medium, characterized in that a computer program is stored thereon, and when the computer program is executed by a computer, the computer causes the computer to perform the method according to any one of claims 1 to 9. 21. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9.

Images