CN110944171B - An image prediction method and device - Google Patents

Abstract

The embodiment of the present application discloses an image prediction method and related products. The method includes: analyzing the code stream to obtain a multi-hypothesis information index of the current image block to be decoded; obtaining the first multi-hypothesis information corresponding to the current image block from the multi-hypothesis information list according to the multi-hypothesis information index, the first multi-hypothesis information includes The motion information index and the first identification, the motion information index of the first multi-hypothesis information indicates the multi-hypothesis motion information of the current image block to be decoded, the first identification indicates the first division method of the multi-hypothesis method, and the first multi-hypothesis information also includes the first A parameter of a division method; according to the first identifier and the parameter of the first division method, use the first division method to process a plurality of hypothetical predicted image blocks, so as to obtain a predicted image of the current image block to be decoded piece. In this way, different image blocks can use different division methods to obtain the prediction block of the image block, thereby improving the efficiency of image prediction and reducing the decoding time.

Description

An image prediction method and device

technical field

The present application relates to the technical field of video coding and decoding, and more specifically, relates to an image prediction method and device.

Background technique

Digital video capabilities can be incorporated into a wide variety of devices, including digital television, digital broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, electronic Book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radiotelephones (so-called "smart phones"), video teleconferencing devices, video streaming devices and its analogues. Digital video devices implement video compression techniques, for example, in the standards defined by MPEG-2, MPEG-4, ITU-TH.263, ITU-T H.264/MPEG-4 Part 10 Advanced Video Coding (AVC), video Coding standard The video compression techniques described in the H.265/High Efficiency Video Coding (HEVC) standard and extensions to such standards. Video devices may more efficiently transmit, receive, encode, decode, and/or store digital video information by implementing such video compression techniques.

Video compression techniques perform spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (i.e., a video frame or a portion of a video frame) may be divided into several image blocks, which may also be referred to as tree blocks, coding units (coding units, CUs) and/or or coded nodes. Image blocks in a to-be-intra-coded (I) slice of an image are coded using spatial prediction with respect to reference samples in neighboring blocks in the same image. Image blocks in a slice of a picture to be inter-coded (P or B) may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. An image may be referred to as a frame, and a reference image may be referred to as a reference frame.

Among them, various video coding standards including the HEVC standard propose predictive coding modes for image blocks, that is, to predict a current block to be coded based on coded video data blocks. In intra prediction mode, the current block is predicted based on one or more previously decoded neighboring blocks in the same picture as the current block; in inter prediction mode, the current block is predicted based on already decoded blocks in a different picture piece.

The current method in the inter-frame prediction mode has low prediction efficiency, thus affecting the execution time of the entire decoding process.

Contents of the invention

The present application provides an image prediction method and device, which can improve the efficiency of image prediction, and further reduce the time of the entire decoding process.

In a first aspect, an image prediction method is provided, the method comprising: analyzing a code stream to obtain a multi-hypothesis information index of the current image block to be decoded; obtaining the current image block from a multi-hypothesis information list according to the multi-hypothesis information index Corresponding to the first multi-hypothesis information, the first multi-hypothesis information includes a motion information index and a first identifier, the motion information index of the first multi-hypothesis information indicates the multi-hypothesis motion information of the current image block to be decoded, the first Identify the first division method indicating the multi-hypothesis method, the first multi-hypothesis information also includes the parameters of the first division method; perform motion compensation according to the motion information of the multi-hypothesis encoding of the current image block to be decoded to obtain multiple hypotheses the predicted image block; according to the first identifier and the parameters of the first division method, use the first division method to process the plurality of hypothetical predicted image blocks, so as to obtain the predicted image block of the current image block to be decoded.

In a possible implementation manner, the multi-hypothesis information list includes at least one piece of the first multi-hypothesis information, and at least one piece of second multi-hypothesis information, where the second multi-hypothesis information includes a motion information index and a second identifier, and the second The motion information index of the multi-hypothesis information indicates the multi-hypothesis motion information of an image block, the second identifier indicates the second division method of the multi-hypothesis method, and the second multi-hypothesis information also includes parameters of the second division mode.

In a possible implementation manner, the first identifier and the second identifier are different values of the same flag bit.

In a possible implementation manner, the first division method is triangle division, and the parameter of the first division method indicates the division direction of the triangle division; the second division method is square division, and the parameter of the second division method indicates the The weighting factor for the square division.

It should be understood that the first division method and the second division method are different division methods, and each division method is used to divide a plurality of assumed predicted image blocks, that is, used to divide the original assumed predicted image block and the additional Hypothetical predicted image blocks. For the specific methods and ways of dividing and predicting image blocks in the multi-hypothesis method, please refer to the content of JVET-K0144 proposal (triangular division) and JVET-K0257 proposal (square division).

In a possible implementation manner, the first division method is triangle division, and the parameter of the first division method indicates the division direction of the triangle division, or the first division method is square division, and the parameter of the first division method indicates The weighting factor for this square division.

In a possible implementation manner, the method further includes: according to the motion information index of the first multi-hypothesis information, acquiring the multi-hypothesis motion of the current image block to be decoded from the candidate motion information list of the current image block to be decoded Information, the multi-hypothesis motion information of the current image block to be decoded includes the motion information of the original hypothesis and the motion information of the additional hypothesis.

In a possible implementation manner, the motion information index of the first multi-hypothesis information includes a first index and a second index, and according to the motion information index of the first multi-hypothesis information, the candidate of the current image block to be decoded Obtaining the multi-hypothesis motion information of the current image block to be decoded from the motion information list includes: obtaining the motion information of the original hypothesis from the first candidate motion information list of the current image block according to the first index, and obtaining the motion information of the original hypothesis according to the first index The second index is to obtain the motion information of the additional hypothesis from the second candidate motion information list of the current image block.

In this way, using the above method, since the code stream carries the division identifier indicating the multi-hypothesis method, the decoder can determine which division method to use in the multi-hypothesis method to predict the image, that is, for different image blocks, it can be Different division methods are used to obtain the prediction block of the image block, so that the image block can use a division method more suitable for its own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

In a second aspect, an image prediction device is provided, and the device includes a module for performing the method in any one of the implementation manners of the above-mentioned first aspect.

In a third aspect, an image prediction device is provided, including: a non-volatile memory and a processor coupled to each other, and the processor invokes the program code stored in the memory to execute any one of the implementations in the first aspect. some or all steps of the method.

In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores program code, wherein the program code includes part or all of the method for executing any one of the implementation manners in the first aspect Step instructions.

In a fifth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is instructed to execute some or all steps of the method in any implementation manner in the first aspect.

In a sixth aspect, an image prediction device is provided, which includes: a memory for storing video data in the form of a code stream, the video data including one or more image blocks; a video decoder for parsing the code stream to obtain the current The multi-hypothesis information index of the image block to be decoded; according to the multi-hypothesis information index, the first multi-hypothesis information corresponding to the current image block is obtained from the multi-hypothesis information list, and the first multi-hypothesis information includes the motion information index and the first Identification, the motion information index of the first multi-hypothesis information indicates the multi-hypothesis motion information of the current image block to be decoded, the first identification indicates the first division method of the multi-hypothesis method, and the first multi-hypothesis information also includes the first A parameter of a division method; perform motion compensation according to the motion information of the multi-hypothesis encoding of the current image block to be decoded to obtain multiple hypothetical prediction image blocks; according to the first identifier and the parameters of the first division method, use the The first division manner processes the plurality of hypothetical predicted image blocks to obtain a predicted image block of the current image block to be decoded.

In a possible implementation manner, the multi-hypothesis information list includes at least one piece of the first multi-hypothesis information, and at least one piece of second multi-hypothesis information, where the second multi-hypothesis information includes a motion information index and a second identifier, and the second The motion information index of the multi-hypothesis information indicates the multi-hypothesis motion information of an image block, the second identifier indicates the second division method of the multi-hypothesis method, and the second multi-hypothesis information also includes parameters of the second division mode.

In a possible implementation manner, the first identifier and the second identifier are different values of the same flag bit.

In a possible implementation manner, the first division method is triangle division, and the parameter of the first division method indicates the division direction of the triangle division; the second division method is square division, and the parameter of the second division method indicates the The weighting factor for the square division.

It should be understood that the first division method and the second division method are different division methods, and each division method is used to divide a plurality of assumed predicted image blocks, that is, used to divide the original assumed predicted image block and the additional Hypothetical predicted image blocks. For the specific methods and ways of dividing and predicting image blocks in the multi-hypothesis method, please refer to the content of JVET-K0144 proposal (triangular division) and JVET-K0257 proposal (square division).

In a possible implementation manner, the first division method is triangle division, and the parameter of the first division method indicates the division direction of the triangle division, or the first division method is square division, and the parameter of the first division method indicates The weighting factor for this square division.

In a possible implementation manner, the video decoder is further configured to: acquire the motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded according to the motion information index of the first multi-hypothesis information. Multi-hypothesis motion information, the multi-hypothesis motion information of the current image block to be decoded includes the motion information of the original hypothesis and the motion information of the additional hypothesis.

In a possible implementation manner, the motion information index of the first multi-hypothesis information includes a first index and a second index, and according to the motion information index of the first multi-hypothesis information, the current to-be-decoded image block In terms of acquiring the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list, the video decoder is further configured to: acquire the original motion information from the first candidate motion information list of the current image block according to the first index The assumed motion information, and according to the second index, obtain the additional assumed motion information from the second candidate motion information list of the current image block.

In this way, using the above-mentioned equipment, since the code stream carries the division identifier indicating the multi-hypothesis method, the decoder can determine which division method to use in the multi-hypothesis method to predict the image, that is, for different image blocks, it can be Different division methods are used to obtain the prediction block of the image block, so that the image block can use a division method more suitable for its own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

Description of drawings

FIG. 1 is a conceptual block diagram of an encoding system 10;

FIG. 2 is a schematic block diagram of an example of a video encoder 20;

FIG. 3 is a schematic block diagram of an example of a video decoder 30;

FIG. 4 is an illustration of an example of a video encoding system 40;

FIG. 5 is a schematic diagram of a device 500 for implementing the method of the present application;

Fig. 6 is a schematic diagram of an image prediction method;

Fig. 7 is a schematic diagram of an image prediction device.

Detailed ways

The technical solution in this application will be described below with reference to the accompanying drawings.

Hereinafter, specific embodiments of the present invention and application examples using the specific embodiments of the present invention will be described with reference to the drawings. Video coding generally refers to the processing of sequences of pictures that form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" may be used as synonyms. Video encoding as used in this application (or this disclosure) means video encoding or video decoding. Video encoding is performed on the source side and typically involves processing (eg, by compressing) raw video pictures to reduce the amount of data needed to represent the video pictures (and thus more efficiently store and/or transmit). Video decoding is performed at the destination and typically involves inverse processing relative to the encoder to reconstruct the video picture. The "encoding" of video pictures (or collectively referred to as pictures, which will be explained below) involved in the embodiments should be understood as involving "encoding" or "decoding" of video sequences. The combination of encoding part and decoding part is also called codec (encoding and decoding).

In the case of lossless video coding, the original video picture can be reconstructed, ie the reconstructed video picture has the same quality as the original video picture (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed by, for example, quantization, to reduce the amount of data required to represent a video picture without being able to fully reconstruct the video picture at the decoder side, i.e. the quality of the reconstructed video picture is compared to the original video picture of low or poor quality.

Several video coding standards of H.261 belong to "lossy hybrid video codecs" (ie, combining spatial and temporal prediction in the sample domain with 2D transform coding in the transform domain for applying quantization). Each picture of a video sequence is usually partitioned into a non-overlapping set of blocks, usually coded at the block level. In other words, the encoder side usually processes at the block (video block) level, i.e. encodes the video, for example, generates a predicted block through spatial (intra-picture) prediction and temporal (inter-picture) prediction, from the current block (currently processed or to be processed block) minus the predicted block to obtain the residual block, transform the residual block in the transform domain and quantize the residual block to reduce the amount of data to be transmitted (compressed), and the decoder side will be inversely processed relative to the encoder Partially applied to an encoded or compressed block to reconstruct the current block for representation. Additionally, the encoder replicates the decoder processing loop such that the encoder and decoder generate the same predictions (eg, intra and inter predictions) and/or reconstructions for processing, ie encoding, subsequent blocks.

As used herein, the term "block" may be a portion of a picture or frame. For ease of description, refer to the Video Coding Joint Working Group (Joint Collaboration Team on Video Coding, JCT) of ITU-T Video Coding Experts Group (Video Coding Experts Group, VCEG) and ISO/IEC Motion Picture Experts Group (Motion Picture Experts Group, MPEG). -High-Efficiency Video Coding (HEVC) developed by VC) describes embodiments of the present invention. Those of ordinary skill in the art understand that embodiments of the present invention are not limited to HEVC. Can refer to CU, PU, and TU. In HEVC, a CTU is split into multiple CUs by using a quadtree structure represented as a coding tree. The decision whether to encode a region of a picture using inter-picture (temporal) or intra-picture (spatial) prediction is made at the CU level. Each CU can be further split into one, two or four PUs according to the PU split type. The same prediction process is applied within a PU and relevant information is transferred to the decoder on a PU basis. After obtaining the residual block by applying a prediction process based on the PU split type, the CU can be partitioned into transform units (TUs) according to other quadtree structures similar to the coding tree used for the CU. In the latest development of video compression technology, quad-tree and binary tree (QTBT) are used to divide the frame to divide the encoding block. In the QTBT block structure, a CU can be square or rectangular in shape. In XXX, the coding tree unit (CTU) is first divided by a quadtree structure. The quadtree leaf nodes are further divided by the binary tree structure. Binary tree leaf nodes are called coding units (CUs), and the segments are used for prediction and transform processing without any other partitioning. This means that CU, PU and TU have the same block size in the QTBT coded block structure. Meanwhile, it is also proposed to use multiple partitions together with the QTBT block structure, such as ternary tree partitioning.

Embodiments of the encoder 20 , decoder 30 and codec system 10 , 40 are described below based on FIGS. 1 to 4 (before the embodiment of the invention is described in more detail based on FIG. 6 ).

FIG. 1 is a conceptual or schematic block diagram illustrating an exemplary encoding system 10 , such as a video encoding system 10 that may utilize techniques of the present application (disclosure). Encoder 20 (e.g., video encoder 20) and decoder 30 (e.g., video decoder 30) of video encoding system 10 represent examples of devices that may be used to perform techniques for image prediction according to various examples described in this application . As shown in FIG. 1 , an encoding system 10 includes a source device 12 for providing encoded data 13 , eg, an encoded picture 13 , to eg a destination device 14 that decodes the encoded data 13 .

The source device 12 includes an encoder 20 and, additionally and optionally, a picture source 16 , such as a preprocessing unit 18 such as a picture preprocessing unit 18 , and a communication interface or communication unit 22 .

Picture source 16 may include or be any kind of picture capture device, for example capturing real world pictures, and/or any kind of picture or commentary (for screen content coding, some text on the screen is also considered a picture to be coded or part of an image) generating device, such as a computer graphics processor for generating computer-animated pictures, or for acquiring and/or providing real-world pictures, computer-animated pictures (for example, screen content, virtual reality (VR) images), and/or any combination thereof (for example, augmented reality (AR) images).

A (digital) picture is or can be regarded as a two-dimensional array or matrix of sample points with luminance values. The sampling points in the array may also be called pixels (short for picture element) or pixels (pel). The number of samples in the array or picture in the horizontal and vertical directions (or axes) defines the size and/or resolution of the picture. In order to represent a color, three color components are usually used, that is, a picture can be represented as or contain three sample arrays. In the RBG format or color space, the picture includes corresponding red, green and blue sample arrays. However, in video coding, each pixel is usually represented in a luminance/chrominance format or color space, for example, YCbCr, including a luminance component indicated by Y (which can also be indicated by L sometimes) and two chrominances indicated by Cb and Cr portion. The luminance (luma for short) component Y represents the brightness or grayscale level intensity (e.g. both are the same in a grayscale picture), while the two chrominance (chroma for short) components Cb and Cr represent the chrominance or color information components . Correspondingly, a picture in YCbCr format includes a luma sample array of luma sample values (Y), and two chroma sample arrays of chrominance values (Cb and Cr). A picture in RGB format can be converted or converted to YCbCr format and vice versa, this process is also known as color transformation or conversion. If the picture is black and white, the picture may only include an array of luma samples.

Picture source 16 (e.g., video source 16) may be, for example, a camera for capturing pictures, a memory such as a picture memory that includes or stores previously captured or generated pictures, and/or acquires or receives pictures of any kind (internal or external) interface. The camera may be, for example, an integrated camera local or integrated in the source device, and the memory may be local or an integrated memory, eg integrated in the source device. The interface may be, for example, an external interface that receives pictures from an external video source, such as an external picture capture device, such as a camera, external memory, or an external picture generation device, such as an external computer graphics processor, computer or server. The interface can be any kind of interface according to any proprietary or standardized interface protocol, eg wired or wireless interface, optical interface. The interface for acquiring the picture data 17 may be the same interface as the communication interface 22 or a part of the communication interface 22 .

Different from the pre-processing unit 18 and the processing performed by the pre-processing unit 18 , the picture or picture data 17 (eg, video data 16 ) may also be referred to as a raw picture or raw picture data 17 .

The preprocessing unit 18 is configured to receive (raw) picture data 17 and perform preprocessing on the picture data 17 to obtain a preprocessed picture 19 or preprocessed picture data 19 . For example, preprocessing performed by preprocessing unit 18 may include retouching, color format conversion (eg, from RGB to YCbCr), color grading, or denoising. It is understood that the pre-processing unit 18 may be an optional component.

An encoder 20 (eg, video encoder 20 ) is used to receive preprocessed picture data 19 and provide encoded picture data 21 (details will be described further below, eg, based on FIG. 2 or FIG. 4 ). In an example, the encoder 20 can be used to carry the identifier indicating the division mode of the multi-hypothesis method and the parameters of the division mode in the code stream of the image block during the encoding process. For details, please refer to the corresponding paragraph.

The communication interface 22 of the source device 12 may be used to receive and transmit the encoded picture data 21 to other devices, e.g. the destination device 14 or any other device, for storage or direct reconstruction, or for storing the encoded picture data 21 accordingly. The encoded picture data 21 is processed before encoding the data 13 and/or transmitting the encoded data 13 to another device, such as the destination device 14 or any other device for decoding or storage.

Destination device 14 includes a decoder 30 (eg, video decoder 30 ), which may additionally or optionally include a communication interface or unit 28 , a post-processing unit 32 , and a display device 34 .

The communication interface 28 of the destination device 14 is used to receive the encoded picture data 21 or the encoded data 13, for example, directly from the source device 12 or any other source, such as a storage device, such as a coded picture data store equipment.

Communication interface 22 and communication interface 28 may be used to communicate directly via a direct communication link between source device 12 and destination device 14 or via any type of network to transmit or receive encoded picture data 21 or encoded data 13 Links such as direct wired or wireless connections, any kind of network such as wired or wireless networks or any combination thereof, or any kind of private and public networks, or any combination thereof.

The communication interface 22 may, for example, be used to package the encoded picture data 21 into a suitable format, such as packets, for transmission over a communication link or network.

The communication interface 28 , which forms a counterpart to the communication interface 22 , may for example be used to decapsulate the encoded data 13 to obtain the encoded picture data 21 .

Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 in FIG. 1 for encoded picture data 13, or as bidirectional communication interfaces, and Can be used eg to send and receive messages to establish connections, acknowledge and exchange any other information related to communication links and/or data transfers eg encoded picture data transfers.

The decoder 30 is adapted to receive encoded picture data 21 and provide decoded picture data 31 or a decoded picture 31 (details will be described further below, eg based on FIG. 3 or FIG. 5 ). In one example, the decoder 30 can be used to implement the image prediction method described in this application.

The post-processor 32 of the destination device 14 is configured to post-process decoded picture data 31 (also referred to as reconstructed picture data), e.g., decoded picture 131, to obtain post-processed picture data 33, e.g., post-processed Image 33. Post-processing performed by post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, trimming, or resampling, or any other processing, e.g., to prepare decoded picture data 31 for processing by The display device 34 displays.

The display device 34 of the destination device 14 is used to receive the post-processed picture data 33 for displaying the picture eg to a user or viewer. The display device 34 may be or include any kind of display for presenting the reconstructed picture, eg, an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital A digital light processor (DLP) or other display of any kind.

Although FIG. 1 depicts source device 12 and destination device 14 as separate devices, device embodiments may include both source device 12 and destination device 14 or the functionality of both, i.e. source device 12 or corresponding functionality and the destination device 14 or corresponding functionality. In such embodiments, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and/or software, or using separate hardware and/or software, or any combination thereof .

It will be apparent to those skilled in the art based on the description that the functionality of the different units or the presence and (exact) division of the functionality of the source device 12 and/or destination device 14 shown in FIG. 1 may vary depending on the actual device and application.

Both encoder 20 (e.g., video encoder 20) and decoder 30 (e.g., video decoder 30) may be implemented as any of a variety of suitable circuits, e.g., one or more microprocessors, digital signal processors, (digital signal processor, DSP), application-specific integrated circuit (ASIC), field-programmable gate array (field-programmable gate array, FPGA), discrete logic, hardware, or any combination thereof. If the techniques are implemented partially in software, a device may store instructions for the software in a suitable non-transitory computer-readable storage medium and may execute the instructions in hardware using one or more processors to perform the techniques of this disclosure . Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered as one or more processors. Each of video encoder 20 and video decoder 30 may be incorporated into one or more encoders or decoders, either of which may be integrated as a combined encoder/decoder in a corresponding device Part of the codec (codec).

Source device 12 may be referred to as a video encoding device or video encoding device. Destination device 14 may be referred to as a video decoding device or video decoding device. Source device 12 and destination device 14 may be examples of video encoding devices or video encoding apparatus.

Source device 12 and destination device 14 may comprise any of a variety of devices, including any class of handheld or stationary devices, such as notebook or laptop computers, mobile phones, smart phones, tablets or tablet computers, video cameras, desktop Computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may not use Or use any class of OS.

In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Accordingly, source device 12 and destination device 14 may be wireless communication devices.

In some cases, the video encoding system 10 shown in FIG. 1 is merely an example, and the techniques of the present application may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily involve any data communication between encoding and decoding devices. . In other examples, data may be retrieved from local storage, streamed over a network, etc. A video encoding device may encode and store data to memory, and/or a video decoding device may retrieve and decode data from memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other but merely encode data to memory and/or retrieve data from memory and decode data.

It should be understood that for each of the examples described above with reference to video encoder 20, video decoder 30 may be used to perform the reverse process. With regard to signaling syntax elements, the video decoder 30 may be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, video encoder 20 may entropy encode one or more syntax elements defining... into an encoded video bitstream. In such instances, video decoder 30 may parse such syntax elements and decode the related video data accordingly.

Encoder & Encoding Method

FIG. 2 shows a schematic block diagram of an example of a video encoder 20 for implementing the techniques of the present application (disclosure). In the example of FIG. 2, the video encoder 20 includes a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a buffer 216, a loop filter A decoder unit 220, a decoded picture buffer (decoded picture buffer, DPB) 230, a prediction processing unit 260, and an entropy encoding unit 270. Prediction processing unit 260 may include inter prediction unit 244 , intra prediction unit 254 , and mode selection unit 262 . Inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG. 2 may also be called a hybrid video encoder or a video encoder according to a hybrid video codec.

For example, the residual calculation unit 204, the transform processing unit 206, the quantization unit 208, the prediction processing unit 260, and the entropy encoding unit 270 form the forward signal path of the encoder 20, while for example the inverse quantization unit 210, the inverse transform processing unit 212, the The structure unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (decoded picture buffer, DPB) 230, and the prediction processing unit 260 form the backward signal path of the encoder, wherein the backward signal path of the encoder corresponds to to the signal path of the decoder (see decoder 30 in FIG. 3).

The encoder 20 receives, eg via an input 202, a picture 201 or a block 203 of a picture 201, eg a picture in a sequence of pictures forming a video or a video sequence. The picture block 203 can also be called the current picture block or the picture block to be coded, and the picture 201 can be called the current picture or the picture to be coded (especially when the current picture is distinguished from other pictures in video coding, other pictures such as the same video sequence That is, previously encoded and/or decoded pictures in the video sequence also including the current picture).

segmentation

Embodiments of encoder 20 may include a partitioning unit (not shown in FIG. 2 ) for partitioning picture 201 into a plurality of blocks, such as block 203 , typically into a plurality of non-overlapping blocks. A split unit can be used to use the same block size for all pictures in a video sequence and a corresponding grid defining the block size, or to change the block size between pictures or subsets or groups of pictures and split each picture into the corresponding block.

In one example, prediction processing unit 260 of video encoder 20 may be used to perform any combination of the partitioning techniques described above.

Like the picture 201 , the block 203 is also or can be regarded as a two-dimensional array or matrix of samples with luminance values (sample values), although its size is smaller than that of the picture 201 . In other words, block 203 may comprise, for example, one array of samples (e.g., a luma array in the case of a black and white picture 201) or three arrays of samples (e.g., one luma array and two chrominance arrays in the case of a color picture) or according to An array of any other number and/or class of color formats to apply. The number of sampling points in the horizontal and vertical directions (or axes) of the block 203 defines the size of the block 203 .

The encoder 20 shown in FIG. 2 is used to encode a picture 201 block by block, for example, performing encoding and prediction on each block 203 .

residual calculation

The residual calculation unit 204 is configured to calculate the residual block 205 based on the picture block 203 and the predicted block 265 (further details of the predicted block 265 are provided below), for example, by subtracting the predicted The sample values of the block 265 to obtain the residual block 205 in the sample domain.

transform

The transform processing unit 206 is configured to apply a transform such as discrete cosine transform (discretecosine transform, DCT) or discrete sine transform (discrete sine transform, DST) on the sample values of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as transform residual coefficients and represent the residual block 205 in the transform domain.

The transform processing unit 206 may be configured to apply an integer approximation of DCT/DST, such as the transform specified for HEVC/H.265. Such integer approximations are usually scaled by some factor compared to the orthogonal DCT transform. In order to maintain the norm of the forward and inverse transformed residual blocks, an additional scaling factor is applied as part of the transformation process. The scaling factor is usually chosen based on certain constraints, for example, the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, the trade-off between accuracy and implementation cost, etc. For example, specifying a specific scaling factor for the inverse transform at the decoder 30 side by eg the inverse transform processing unit 212 (and at the encoder 20 side for the corresponding inverse transform by eg the inverse transform processing unit 212), and correspondingly, may be performed at the encoder The side 20 specifies the corresponding scaling factor for the forward transform through the transform processing unit 206 .

Quantify

A quantization unit 208 is configured to quantize the transform coefficients 207 , eg by applying scalar quantization or vector quantization, to obtain quantized transform coefficients 209 . Quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209 . The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207 . For example, n-bit transform coefficients may be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting a quantization parameter (quantization parameter, QP). For example, with scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. A suitable quantization step size can be indicated by a quantization parameter (quantization parameter, QP). For example, a quantization parameter may be an index to a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (smaller quantization step size), and a larger quantization parameter may correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may involve division by a quantization step size and corresponding quantization or inverse quantization, eg performed by inverse quantization 210, or may involve multiplication by a quantization step size. Embodiments according to some standards such as HEVC may use quantization parameters to determine the quantization step size. In general, the quantization step size can be calculated based on the quantization parameter using a fixed-point approximation of an equation involving division. An additional scaling factor can be introduced for quantization and dequantization to recover the norm of the residual block that might have been modified by the scale used in the fixed-point approximation of the equations for the quantization step size and quantization parameter. In one example implementation, inverse transform and inverse quantized scaling may be combined. Alternatively, a custom quantization table can be used and signaled from the encoder to the decoder in e.g. the bitstream. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.

The inverse quantization unit 210 is configured to apply the inverse quantization of the quantization unit 208 on the quantized coefficients to obtain the dequantized coefficients 211, e.g., based on or using the same quantization step size as the quantization unit 208, applying the quantization scheme applied by the quantization unit 208 The inverse quantization scheme. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, corresponding to the transform coefficients 207, although the losses due to quantization are generally not the same as the transform coefficients.

The inverse transform processing unit 212 is configured to apply an inverse transform of the transform applied by the transform processing unit 206, for example, an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain in the sample domain Inverse transform block 213 . The inverse transform block 213 may also be referred to as the inverse transform dequantized block 213 or the inverse transform residual block 213 .

The reconstruction unit 214 (e.g. summer 214) is used to add the inverse transform block 213 (i.e. reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the sample domain, e.g. The sample values of the reconstructed residual block 213 are added to the sample values of the prediction block 265 .

Optionally, a buffer unit 216 (or simply "buffer" 216), such as a line buffer 216, is used to buffer or store the reconstructed block 215 and corresponding sample values, for eg intra prediction. In other embodiments, the encoder may be configured to use the unfiltered reconstructed blocks and/or corresponding sample values stored in the buffer unit 216 for any kind of estimation and/or prediction, such as intra predict.

For example, an embodiment of encoder 20 may be configured such that buffer unit 216 is used not only for storing reconstructed blocks 215 for intra prediction 254, but also for loop filter unit 220 (not shown in FIG. 2 ). output), and/or such that buffer unit 216 and decoded picture buffer unit 230 form one buffer. Other embodiments may be used to use filtered blocks 221 and/or blocks or samples from decoded picture buffer 230 (neither shown in FIG. 2 ) as input or basis for intra prediction 254 .

A loop filter unit 220 (or "loop filter" 220 for short) is used to filter the reconstructed block 215 to obtain a filtered block 221 to smooth pixel transformation or improve video quality. The loop filter unit 220 is intended to represent one or more loop filters, such as deblocking filters, sample-adaptive offset (sample-adaptive offset, SAO) filters or other filters, such as bilateral filters, auto An adaptive loop filter (ALF), or a sharpening or smoothing filter, or a collaborative filter. Although loop filter unit 220 is shown in FIG. 2 as an in-loop filter, in other configurations, loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as a filtered reconstructed block 221 . The decoded picture buffer 230 may store the reconstructed encoded block after the loop filter unit 220 performs a filtering operation on the reconstructed encoded block.

Embodiments of the encoder 20 (respectively, the loop filter unit 220) may be used to output loop filter parameters (e.g., SAO information), for example, directly or by the entropy encoding unit 270 or any other The entropy encoding unit outputs after entropy encoding, for example, so that the decoder 30 can receive and apply the same loop filter parameters for decoding.

The decoded picture buffer (DPB) 230 may be a reference picture memory storing reference picture data for the video encoder 20 to encode video data. The DPB 230 may be formed by any one of various memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (synchronous DRAM, SDRAM), magnetoresistive RAM (magnetoresistive RAM, MRAM), resistive RAM (resistive RAM, RRAM)) or other types of memory devices. DPB 230 and buffer 216 may be provided by the same memory device or by separate memory devices. In a certain example, a decoded picture buffer (DPB) 230 is used to store the filtered block 221 . The decoded picture buffer 230 may further be used to store other previously filtered blocks of the same current picture or a different picture such as a previous reconstructed picture, such as the previously reconstructed and filtered block 221, and may provide a complete previously Reconstructed, ie decoded pictures (and corresponding reference blocks and samples) and/or partially reconstructed current pictures (and corresponding reference blocks and samples), eg for inter prediction. In a certain example, a decoded picture buffer (DPB) 230 is used to store the reconstructed block 215 if the reconstructed block 215 is reconstructed without in-loop filtering.

Prediction processing unit 260, also referred to as block prediction processing unit 260, is adapted to receive or acquire block 203 (current block 203 of current picture 201) and reconstructed picture data, e.g. references from the same (current) picture from buffer 216 samples and/or reference picture data 231 of one or more previously decoded pictures from a decoded picture buffer 230, and for processing such data for prediction, i. Predicted block 265 of predicted block 255 .

The mode selection unit 262 may be used to select a prediction mode (such as an intra or inter prediction mode) and/or the corresponding prediction block 245 or 255 used as the prediction block 265 for computing the residual block 205 and reconstructing the reconstructed block 215.

Embodiments of the mode selection unit 262 may be used to select a prediction mode (e.g., from those supported by the prediction processing unit 260) that provides the best match or the smallest residual (minimum residual means better compression in transmission or storage), or provide minimal signaling overhead (minimum signaling overhead means better compression in transmission or storage), or consider or balance both of the above. The mode selection unit 262 can be used to determine the prediction mode based on rate distortion optimization (RDO), that is, to select the prediction mode that provides the minimum rate distortion optimization, or to select the prediction mode that the relevant rate distortion at least satisfies the prediction mode selection standard .

The prediction processing performed by an instance of encoder 20 (eg, by prediction processing unit 260 ) and the mode selection (eg, by mode selection unit 262 ) are explained in detail below.

As mentioned above, the encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra prediction modes and/or inter prediction modes.

The set of intra prediction modes may include 35 different intra prediction modes, for example, non-directional modes such as DC (or mean) mode and planar mode, or directional modes as defined in H.265, or may include 67 different intra prediction modes, eg non-directional modes such as DC (or mean) mode and planar mode, or directional modes as defined in the developing H.266.

The set of (possible) inter prediction modes depends on the available reference pictures (i.e., e.g. the aforementioned at least partially decoded pictures stored in the DBP 230) and other inter prediction parameters, e.g. on whether to use the entire reference picture or only the reference A part of the picture, eg a search window area around the area of the current block, is searched for the best matching reference block, and/or eg depending on whether pixel interpolation such as half-pixel and/or quarter-pixel interpolation is applied.

In addition to the above prediction modes, skip mode and/or direct mode may also be applied.

The prediction processing unit 260 may further be used to partition the block 203 into smaller block partitions or sub-blocks, for example, by iteratively using quad-tree (quad-tree, QT) partitioning, binary-tree (binary-tree, BT) partitioning or Triple-tree (TT) partitioning, or any combination thereof, and for performing prediction, for example, for each of the block partitions or sub-blocks, wherein the mode selection includes selecting the tree structure of the partitioned block 203 and selecting the tree structure to apply to the block A prediction mode for each of the partitions or sub-blocks.

The inter prediction unit 244 may include a motion estimation (motion estimation, ME) unit (not shown in FIG. 2 ) and a motion compensation (motion compensation, MC) unit (not shown in FIG. 2 ). The motion estimation unit is adapted to receive or acquire a picture block 203 (current picture block 203 of a current picture 201) and a decoded picture 231, or at least one or more previously reconstructed blocks, e.g. one or more other/different previously reconstructed blocks The reconstructed blocks of picture 231 are decoded for motion estimation. For example, the video sequence may comprise the current picture and the previously decoded picture 31, or in other words the current picture and the previously decoded picture 31 may be part of, or form, a sequence of pictures forming the video sequence.

For example, encoder 20 may be configured to select a reference block from multiple reference blocks of the same or different pictures in multiple other pictures, and provide a reference picture (or a reference picture index) to a motion estimation unit (not shown in FIG. 2 ). ...) and/or provide the offset (spatial offset) between the location of the reference block (X, Y coordinates) and the location of the current block as an inter prediction parameter. This offset is also called a motion vector (MV).

The motion compensation unit is configured to obtain, eg receive, inter prediction parameters, and perform inter prediction based on or using the inter prediction parameters to obtain an inter prediction block 245 . Motion compensation performed by a motion compensation unit (not shown in FIG. 2 ) may involve fetching or generating a predictive block based on motion/block vectors determined by motion estimation (possibly performing interpolation to sub-pixel precision). Interpolation filtering can generate additional pixel samples from known pixel samples, potentially increasing the number of candidate predictive blocks that can be used to encode a picture block. Upon receiving the motion vector for the PU of the current picture block, motion compensation unit 246 may locate the predictive block to which the motion vector points in a reference picture list. Motion compensation unit 246 may also generate syntax elements associated with blocks and video slices for use by video decoder 30 in decoding picture blocks of the video slice.

The intra prediction unit 254 is configured to acquire, eg receive, the picture block 203 (current picture block) of the same picture and one or more previously reconstructed blocks, eg reconstructed neighboring blocks, for intra estimation. For example, the encoder 20 may be configured to select an intra prediction mode from a plurality of (predetermined) intra prediction modes.

Embodiments of the encoder 20 may be used to select the intra prediction mode based on optimization criteria, such as based on the smallest residual (eg, the intra prediction mode that provides the most similar prediction block 255 to the current picture block 203 ) or the smallest rate-distortion.

The intra-prediction unit 254 is further configured to determine an intra-prediction block 255 based on the intra-prediction parameters as the selected intra-prediction mode. In any case, after selecting the intra prediction mode for the block, the intra prediction unit 254 is also configured to provide the intra prediction parameters to the entropy coding unit 270, i.e., to provide an indication of the selected intra prediction mode for the block Information. In one example, intra-prediction unit 254 may be configured to perform any combination of the intra-prediction techniques described below.

The entropy coding unit 270 is used to use an entropy coding algorithm or scheme (for example, a variable length coding (variable lengthcoding, VLC) scheme, a context adaptive VLC (context adaptive VLC, CAVLC) scheme, an arithmetic coding scheme, a context adaptive binary arithmetic coding (context adaptive binary arithmetic coding, CABAC), syntax-based context-adaptive binary arithmetic coding (syntax-based context-adaptive binary arithmetic coding, SBAC), probability interval partitioning entropy (probability interval partitioningentropy, PIPE) coding or other entropy coding methods or technique) applied to one or all (or not) of the quantized residual coefficients 209, inter prediction parameters, intra prediction parameters and/or loop filter parameters to obtain The encoded picture data 21 is output in the form of a bitstream 21 . The encoded bitstream may be transmitted to video decoder 30 or archived for later transmission or retrieval by video decoder 30 . Entropy encoding unit 270 may also be used to entropy encode other syntax elements of the current video slice being encoded.

Other structural variants of the video encoder 20 are available for encoding video streams. For example, a non-transform based encoder 20 may directly quantize the residual signal without the transform processing unit 206 for certain blocks or frames. In another embodiment, encoder 20 may have quantization unit 208 and inverse quantization unit 210 combined into a single unit.

FIG. 3 shows an exemplary video decoder 30 for implementing the technique of the present application, ie, the image prediction method. The video decoder 30 is configured to receive encoded picture data (eg, encoded bitstream) 21 , eg encoded by the encoder 20 , to obtain a decoded picture 231 . During the decoding process, video decoder 30 receives video data, such as an encoded video bitstream representing picture blocks of an encoded video slice, and associated syntax elements from video encoder 20 .

In the example of FIG. 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (such as a summer 314), a buffer 316, a loop filter 320, an The decoded picture buffer 330 and the prediction processing unit 360 . Prediction processing unit 360 may include inter prediction unit 344 , intra prediction unit 354 , and mode selection unit 362 . In some examples, video decoder 30 may perform a decoding pass that is substantially the inverse of the encoding pass described with reference to video encoder 20 of FIG. 2 .

The entropy decoding unit 304 is configured to perform entropy decoding on the encoded picture data 21 to obtain, for example, quantized coefficients 309 and/or decoded encoding parameters (not shown in FIG. 3 ), for example, inter-frame prediction, intra-frame prediction parameters Any or all of (decoded) , loop filter parameters, and/or other syntax elements. The entropy decoding unit 304 is further configured to forward the inter prediction parameters, intra prediction parameters and/or other syntax elements to the prediction processing unit 360 . Video decoder 30 may receive syntax elements at the video slice level and/or the video block level.

The inverse quantization unit 310 may be functionally the same as the inverse quantization unit 110, the inverse transform processing unit 312 may be functionally the same as the inverse transform processing unit 212, the reconfiguration unit 314 may be functionally the same as the reconfiguration unit 214, and the buffer 316 may be functionally Like buffer 216 , loop filter 320 may be functionally the same as loop filter 220 , and decoded picture buffer 330 may be functionally the same as decoded picture buffer 230 .

The prediction processing unit 360 may include an inter prediction unit 344 and an intra prediction unit 354, wherein the inter prediction unit 344 may be functionally similar to the inter prediction unit 244, and the intra prediction unit 354 may be functionally similar to the intra prediction unit 254 . The prediction processing unit 360 is generally configured to perform block prediction and/or obtain a predicted block 365 from the encoded data 21, and to receive or obtain prediction related parameters and/or information about the Information about the selected prediction mode.

When the video slice is coded as an intra-coded (I) slice, intra prediction unit 354 of prediction processing unit 360 is used to perform the prediction based on the signaled intra prediction mode and the previous decoded block from the current frame or picture. data to generate prediction blocks 365 for picture blocks of the current video slice. When a video frame is encoded as an inter-coded (i.e., B or P) slice, inter prediction unit 344 (e.g., motion compensation unit) of prediction processing unit 360 is used to Other syntax elements generate prediction blocks 365 for video blocks of the current video slice. For inter prediction, the predicted block can be generated from one reference picture within one reference picture list. Video decoder 30 may construct reference frame lists based on the reference pictures stored in DPB 330 using default construction techniques: list 0 and list 1 .

The prediction processing unit 360 is configured to determine prediction information for a video block of the current video slice by parsing motion vectors and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, prediction processing unit 360 uses some of the received syntax elements to determine the prediction mode (e.g., intra or inter prediction), inter prediction slice type (e.g., B slice, P slice or GPB slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each inter-coded video block of the slice, each inter-coded video block for the slice, The inter-prediction status and other information of the video blocks are inter-coded to decode the video blocks of the current video slice.

Inverse quantization unit 310 may be used to inverse quantize (ie, inverse quantize) the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 304 . The inverse quantization process may include using quantization parameters calculated by video encoder 20 for each video block in a video slice to determine the degree of quantization that should be applied and likewise determine the degree of inverse quantization that should be applied.

An inverse transform processing unit 312 is used to apply an inverse transform (eg, an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to generate a residual block in the pixel domain.

Reconstruction unit 314 (e.g. summer 314) is used to add inverse transform block 313 (i.e. reconstructed residual block 313) to prediction block 365 to obtain reconstructed block 315 in the sample domain, e.g. by adding The sample values of the reconstructed residual block 313 are added to the sample values of the prediction block 365 .

A loop filter unit 320 is used (either during the encoding loop or after the encoding loop) to filter the reconstructed block 315 to obtain a filtered block 321 to smooth pixel transitions or improve video quality. In one example, loop filter unit 320 may be configured to perform any combination of the filtering techniques described below. The loop filter unit 320 is intended to represent one or more loop filters, such as deblocking filters, sample-adaptive offset (sample-adaptive offset, SAO) filters or other filters, such as bilateral filters, auto An adaptive loop filter (ALF), or a sharpening or smoothing filter, or a collaborative filter. Although loop filter unit 320 is shown in FIG. 3 as an in-loop filter, in other configurations, loop filter unit 320 may be implemented as a post-loop filter.

The decoded video blocks 321 in a given frame or picture are then stored in a decoded picture buffer 330 that stores reference pictures for subsequent motion compensation.

The decoder 30 is configured to output the decoded picture 31 for presentation to or viewing by a user, eg, via an output 332 .

Other variants of video decoder 30 may be used to decode the compressed bitstream. For example, decoder 30 may generate an output video stream without loop filter unit 320 . For example, the non-transform based decoder 30 may directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another embodiment, video decoder 30 may have inverse quantization unit 310 and inverse transform processing unit 312 combined into a single unit.

FIG. 4 is an illustrative diagram of an example of a video encoding system 40 including encoder 20 of FIG. 2 and/or decoder 30 of FIG. 3 , according to an exemplary embodiment. System 40 may implement a combination of various techniques of the present application. In the illustrated embodiment, video encoding system 40 may include imaging device 41, video encoder 20, video decoder 30 (and/or a video encoder implemented by logic circuitry 47 of processing unit 46), antenna 42, One or more processors 43 , one or more memories 44 and/or a display device 45 .

As shown, imaging device 41 , antenna 42 , processing unit 46 , logic circuitry 47 , video encoder 20 , video decoder 30 , processor 43 , memory 44 and/or display device 45 are capable of communicating with each other. As discussed, although video encoding system 40 is depicted with video encoder 20 and video decoder 30 , video encoding system 40 may include only video encoder 20 or only video decoder 30 in different examples.

In some examples, video encoding system 40 may include antenna 42 as shown. For example, antenna 42 may be used to transmit or receive an encoded bitstream of video data. Additionally, in some examples, video encoding system 40 may include display device 45 . Display device 45 may be used to present video data. In some examples, logic circuitry 47 may be implemented by processing unit 46 as shown. The processing unit 46 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. The video encoding system 40 may also include an optional processor 43, which may similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. In some examples, the logic circuit 47 may be implemented by hardware, such as dedicated hardware for video encoding, etc., and the processor 43 may be implemented by general-purpose software, an operating system, and the like. In addition, the memory 44 can be any type of memory, such as a volatile memory (for example, a static random access memory (Static Random Access Memory, SRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.) or a nonvolatile memory memory (for example, flash memory, etc.) and the like. In a non-limiting example, memory 44 may be implemented by cache memory. In some examples, logic circuitry 47 may access memory 44 (eg, to implement an image buffer). In other examples, logic circuitry 47 and/or processing unit 46 may include memory (eg, cache, etc.) for implementing an image buffer or the like.

In some examples, video encoder 20 implemented by logic circuitry may include an image buffer (eg, implemented by processing unit 46 or memory 44 ) and a graphics processing unit (eg, implemented by processing unit 46 ). A graphics processing unit may be communicatively coupled to the image buffer. Graphics processing unit may include video encoder 20 implemented by logic circuitry 47 to implement the various modules discussed with reference to FIG. 2 and/or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

Video decoder 30 may be implemented in a similar manner by logic circuitry 47 to implement the various modules discussed with reference to decoder 30 of FIG. 3 and/or any other decoder system or subsystem described herein. In some examples, logic implemented video decoder 30 may include an image buffer (implemented by processing unit 2820 or memory 44 ) and a graphics processing unit (eg, implemented by processing unit 46 ). A graphics processing unit may be communicatively coupled to the image buffer. Graphics processing unit may include video decoder 30 implemented by logic circuitry 47 to implement the various modules discussed with reference to FIG. 3 and/or any other decoder system or subsystem described herein.

In some examples, antenna 42 of video encoding system 40 may be used to receive an encoded bitstream of video data. As discussed, an encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., as discussed herein, such as data related to encoding partitions (e.g., transform coefficients or quantized transform coefficients , (as discussed) an optional indicator, and/or data defining an encoding split). Video encoding system 40 may also include video decoder 30 coupled to antenna 42 and for decoding the encoded bitstream. A display device 45 is used to present video frames.

FIG. 5 is a simplified block diagram of an apparatus 500 that may be used as either or both of source device 12 and destination device 14 in FIG. 1 , according to an exemplary embodiment. Apparatus 500 can implement the technology of the present application for image prediction, and apparatus 500 can be in the form of a computing system comprising multiple computing devices, or in the form of a single computer such as a mobile phone, tablet computer, laptop computer, notebook computer, desktop computer, etc. The form of the computing device.

The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices that are capable of manipulating or processing information, existing or to be developed in the future. As shown, while the disclosed embodiments may be practiced using a single processor, such as processor 502, speed and efficiency advantages may be realized using more than one processor.

In one embodiment, the memory 504 in the apparatus 500 may be a read only memory (Read Only Memory, ROM) device or a random access memory (random access memory, RAM) device. Any other suitable type of storage device may be used as memory 504 . Memory 504 may include code and data 506 accessed by processor 502 using bus 512 . Memory 504 may further include an operating system 508 and application programs 510 comprising at least one program that permits processor 502 to perform the methods described herein. For example, application programs 510 may include applications 1 through N that further include a video encoding application that performs the methods described herein. Apparatus 500 may also contain additional memory in the form of secondary memory 514, which may be, for example, a memory card for use with a mobile computing device. Because video communication sessions may contain large amounts of information, such information may be stored in whole or in part in secondary memory 514 and loaded into memory 504 for processing as needed.

Apparatus 500 may also include one or more output devices, such as display 518 . In one example, display 518 may be a touch-sensitive display that combines a display with touch-sensitive elements operable to sense touch input. Display 518 may be coupled to processor 502 via bus 512 . Other output devices that permit a user to program or otherwise use apparatus 500 may be provided in addition to, or as an alternative to, display 518 . When the output device is or contains a display, the display can be implemented in different ways, including through a liquid crystal display (LCD), cathode-ray tube (CRT) display, plasma display, or light emitting diode (light emitting diode) display. diode, LED) display, such as organic LED (organic LED, OLED) display.

The apparatus 500 may also include or be in communication with an image sensing device 520, such as a camera or any other image sensing device 520 existing or to be developed in the future that can sense an image, such as is the image of the user running the device 500 . The image sensing device 520 may be placed directly facing a user operating the apparatus 500 . In an example, the position and optical axis of image sensing device 520 may be configured such that its field of view includes an area immediately adjacent to display 518 and from which display 518 is visible.

Apparatus 500 may also include or be in communication with a sound sensing device 522 such as a microphone or any other sound sensing device existing or to be developed in the future that can sense sounds in the vicinity of apparatus 500 . Sound sensing device 522 may be placed directly facing a user operating apparatus 500 and may be used to receive sounds, such as speech or other sounds, made by the user while operating apparatus 500 .

Although processor 502 and memory 504 of device 500 are shown in FIG. 5 as being integrated in a single unit, other configurations may also be used. Operation of processor 502 may be distributed among multiple directly coupleable machines, each having one or more processors, or across a local area or other network. The memory 504 may be distributed among multiple machines, such as a network-based memory or memory among multiple machines running the apparatus 500 . Although only a single bus is shown here, the bus 512 of the device 500 may be formed by multiple buses. Further, slave memory 514 may be directly coupled to other components of apparatus 500 or accessible via a network, and may comprise a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Accordingly, apparatus 500 may be implemented in a variety of configurations. The terms involved in this application are briefly described below:

Intra-frame predictive coding: A coding method in which the surrounding pixel values are used to predict the current pixel value, and then the prediction error is coded.

Coded picture: A coded representation of a picture that contains all coding tree units of the picture.

motion vector (MV): A two-dimensional vector used for inter prediction that provides an offset from a coordinate in a decoded picture to a coordinate in a reference picture.

Prediction Block: A rectangular M×N block of samples on which the same prediction is applied.

Prediction process: The predictor is used to provide an estimate of the data element (eg, sample value or motion vector) currently being decoded.

Predicted value: Specifies a value or combination of previously decoded data elements (eg, sample values or motion vectors) used in the decoding process of subsequent data elements.

Reference Frame: A picture or frame that is either a short-term reference picture or a long-term reference picture. A reference frame contains samples that can be used in decoding order for inter prediction in the decoding process of subsequent pictures.

Inter-frame prediction: According to the pixels in the reference frame of the current block, the position of the pixel used for prediction in the reference frame is indicated by the motion vector, and the predicted image of the current block is generated.

Bi-predictive (B) slice: A slice that can be decoded by predicting sample values for each block with up to two motion vectors and reference indices using intra prediction or inter prediction.

CTU: Coding tree unit (coding tree unit), an image is composed of multiple CTUs, a CTU usually corresponds to a square image area, including the brightness pixels and chrominance pixels in this image area (or only brightness pixels , or may only contain chroma pixels); the CTU also contains syntax elements that indicate how to divide the CTU into at least one coding unit (CU), and decode each coding unit to obtain a reconstructed image.

CU: coding unit, corresponding to an A×B rectangular area in the image, including A×B brightness pixels or/and its corresponding chrominance pixels, A is the width of the rectangle, B is the height of the rectangle, A and B can be the same It can also be different, and the values of A and B are usually integer powers of 2, such as 128, 64, 32, 16, 8, and 4. A coding unit includes a predicted image and a residual image, and the predicted image and the residual image are added to obtain a reconstructed image of the coding unit. The predicted image is generated through intra-frame prediction or inter-frame prediction, and the residual image is generated through inverse quantization and inverse transform processing of transform coefficients.

VTM: A new codec reference software developed by the JVET organization.

Fusion coding (merge): An inter-frame coding coding method, the motion vector of which is not directly transmitted in the code stream. The current block can select the corresponding fusion candidate from the merge candidate list according to the merge index, and use the motion information of the fusion candidate as the motion information of the current block, or scale the motion information of the fusion candidate as Motion information for the current block.

The image prediction method described in this application is related to inter-frame prediction, and the multi-hypothesis method is an inter-frame prediction method. The inter-frame prediction is briefly described below. In video coding and decoding, inter-frame prediction is mainly used to eliminate temporal and spatial redundancy in video.

Inter prediction is a prediction technique based on motion compensation. In inter-frame predictive coding, since there is a certain time-domain correlation of the same objects in adjacent frames of the image, each frame of the image sequence can be divided into many non-overlapping blocks, and the motion of all pixels in the block is considered to be the same . The main processing is to determine the motion information of the current block, obtain the reference image block from the reference frame according to the motion information, and generate the prediction image of the current block. Motion information includes inter-frame prediction direction, reference frame index (reference index, ref_idx), motion vector (motion vector, MV), etc., inter-frame prediction indicates that the current block uses forward prediction, backward prediction or bidirectional prediction through the inter-frame prediction direction Which of the prediction directions is specified by the reference frame index (reference index), the reference frame (reference frame) is indicated by the motion vector, and the reference block (reference block) of the current block (current block) in the reference frame is relative to the current block in the current frame position offset. The motion vector indicates the displacement vector of the reference image block used to predict the current block relative to the current block in the reference frame, so one motion vector corresponds to one reference image block.

When coding, video coding standards such as H.265/HEVC and H.266/VVC divide a frame of image into non-overlapping coding tree units (Coding Tree Unit, CTU), and a CTU is divided into one or more coding tree units. Unit (CodingUnit, CU). A CU contains encoding information, including information such as prediction mode and transform coefficients. Decoding end: perform corresponding prediction, inverse quantization, inverse transformation and other decoding processing on the CU according to the encoding information, and generate a reconstructed image corresponding to the CU.

In the code stream, motion information occupies a large amount of data. In order to reduce the amount of data required, motion information is usually transmitted in a predictive manner. Overall, it can be divided into two modes: inter and merge:

Inter mvp mode: The transmitted motion information includes: inter-frame prediction direction (forward, backward or bidirectional), reference frame index, motion vector predictor index, and motion vector residual value. For the motion vector information in the motion information, the method of transmitting the difference between the actual motion vector and the motion vector predictor (motion vector predictor, MVP) is usually adopted, and the encoding end transfers the motion vector residual value (motion vector difference, MVD) to the decoder. The motion vector prediction may contain multiple predictors. Generally, the motion vector prediction candidate list (mvp candidate list) is constructed in the same way in the encoding segment and the decoding segment, and the motion vector predictor index (MVP index) is passed to decoder side.

Merge mode: Use the same method to construct a fusion motion information candidate list (merge candidate list) in the encoding segment and decoding segment, and pass the index to the decoding end. The merge index is transmitted in the code stream. The motion information in the motion information candidate list (candidate list) is usually obtained from its spatial adjacent blocks or temporal blocks in the reference frame, where the candidate motion information obtained from the motion information of the image blocks adjacent to the current block is called spatial Candidate (spatial candidate), the motion information of the corresponding position image block obtained from the current block in the reference image is called temporal candidate (temporal candidate).

Bi-directional Optical flow (BIO) is an inter-frame coding method based on block-based motion compensation technology for pixel-level motion vector optimization and improvement (JVET-E1001 proposal, JVET-K0255 proposal, JVET-K0119 proposal proposals and other related technical proposals). Bidirectional optical flow technology can provide improved motion vectors at the pixel level without additional tedious searches or additional bitstream information. The two-way optical flow technology combines optical flow field and Hermite interpolation analysis for the motion trajectory of each pixel, and improves the motion vector by minimizing the intersection point between the motion trajectory and the reference frame plane. In the process, the horizontal and vertical gradients of each pixel in the prediction block need to be used, and the gradient and a special 6-tap filter are interpolated.

Decoder-side motion vector refinement (DMVR) is an interframe coding method that adjusts existing motion vectors by searching at the decoding end (JVET-E1001 proposal, JVET-K0275 proposal and other related technical proposals ). The motion vector improvement technology at the decoding end is used on coding blocks that use bi-directional mode for inter-frame prediction. At the decoding end, first use the motion vector MV0 of list0 and the motion vector MV1 of list1 transmitted in the initial code stream to obtain a bidirectional template through two prediction blocks generated by motion compensation. Use the bidirectional template to search around the positions indicated by the initial motion vectors MV0 and MV1 in the reference frames of list0 and list1, and find the position with the smallest matching distortion within the search range as the new MV0 and MV1 and two new prediction blocks. The final prediction block is generated using the new prediction block weights.

In HEVC and previous standards, reference frames are divided into forward and backward groups, which are placed in two reference frame lists (reference picture list), generally named list0 and list1. The inter-frame prediction direction indicates which prediction direction the current block uses in forward prediction, backward prediction or bidirectional prediction, and selects and uses different reference frame lists list0, list1 or list0 and list1 according to the direction of prediction. For the selected reference frame list, the reference frame is indicated by the reference frame index. In the selected reference frame, the position offset of the prediction block of the current block relative to the current block in the current frame is indicated by the motion vector. Then according to the prediction direction, use the prediction blocks obtained from the reference frames in list0, list1 or list0 and list1 to generate the final prediction block. Wherein, when the prediction direction is unidirectional, the prediction block obtained from the reference frame in list0 or list1 is directly used; when the prediction direction is bidirectional, the prediction block obtained from the reference frame in list0 and list1 is weighted and averaged Synthesize the final prediction block. In the Multi-Hypothesis Inter Prediction method, the prediction block or the final prediction block is used as the original hypothesis (Hypothesis) of the current block, and the number of assumptions is increased by adding new motion information or prediction information to improve coding. performance. The JVET-K0269 and JVET-K0257 proposals use the multi-hypothesis inter-frame coding method, but each implements the multi-hypothesis inter-frame coding in different ways.

The multi-hypothesis coding method in the JVET-K0269 proposal adds additional assumptions by additionally transmitting motion information on the basis of interframe coding in the original HEVC or VVC VTM. Each additional hypothesis needs to transmit whether there is an additional hypothesis flag (additional hypothesis flag), the reference frame number of the additional hypothesis (ref idx add hyp), the motion vector predictor information of the additional hypothesis (mvp add hyp flag), the motion of the additional hypothesis Vector of difference values, and weighting coefficients for additional hypotheses (add hyp weight idx). The additional hypothetical reference frame numbers, motion vector predictor information, and motion vector difference values are the same as in HEVC and previous standards, but the reference frame list is generated differently. The additional hypothetical reference frame list is generated by alternately inserting list0 and list1 in HEVC or VVC VTM, and repeated reference frames are no longer added to the list. After the prediction block (extra hypothesis) is obtained through the motion information of the additional hypothesis, it is compared with the original final prediction block (original hypothesis) by using the weight coefficient of the additional hypothesis. There are two weighting coefficients for additional assumptions, 1/4 and -1/8.

The image prediction method described in this application is used to predict at least one image block using inter-frame prediction in an image using the current method, so as to obtain a predicted image block. Subsequently, further decoding processing may be performed based on the obtained predicted image block to obtain a reconstructed image of the image block, and the processing at the encoding end and the decoding end are the same. The following uses the decoding end as an example for description. For the implementation of the encoding end, please refer to the encoding flow described above and the method flow of the decoding end described below. For example, the coding end obtains the multi-hypothesis coding information of the image block to be coded in the multi-hypothesis coding information list by traversing. parameters. This division method is used when the multi-hypothesis coding method is used to divide the prediction blocks of multiple hypotheses of a picture to be coded. For example, if the division method is triangle division, the parameter is the direction of the triangle division, and if the division mode is square division, the parameter is the weighting coefficient.

Triangulation can refer to the multi-hypothesis encoding method in the JVET-K0144 proposal. On the basis of inter-frame coding in the original HEVC or VVCVTM, when the current coding block is merge or skip, the coding block is divided into two triangular coding blocks along the diagonal direction, and each triangular coding block is used from the fusion motion information candidate The individual sports information obtained in the list. The combination of the division direction of the two triangular coding blocks and the selection of the fusion motion information candidate is selected from the preset multi-hypothesis fusion motion information candidate selection and division combination list using a combination number, and the number is transmitted in the code stream. There is a certain overlapping area between the two triangular coding blocks, and the weighting coefficient related to the distance is used for weighting.

For square division, please refer to the JVET-K0257 proposal. In the proposal, a combination coding method is used to realize multi-hypothesis coding, combining merge with inter MVP, merge, and intra coding. When the current original hypothesis/first hypothesis is the inter MVP mode, and the one-way method is used (that is, only list0 or list1), use the merge mode to generate additional hypotheses. When the current original hypothesis/first hypothesis is in the merge mode, you can use the fusion motion information candidate selected by the original hypothesis/first hypothesis in the fusion motion information candidate list and the next fusion motion information candidate as the motion information of the additional hypothesis to generate additional assumption. When the current original hypothesis/first hypothesis is the merge mode, the intra coding mode can also be used to generate additional hypotheses, and there is an additional syntax element to indicate whether to use this mode. After the prediction block (extra hypothesis) is obtained, it is compared with the original final prediction block using weighting coefficients. In the JVET-K0257 proposal, the weighting coefficients and methods are related to the combination of multiple hypotheses.

The generation process of the reference frame list is an existing technology, and the process can be carried out in the same way as HEVC (that is, when the slice (SLICE) starts, list0 and list1 are constructed), and other generation methods of the reference frame list can also be used (such as The method in the JVET-K0269 proposal), the present invention is not limited. In order to improve efficiency, multi-hypothesis encoding methods (such as JVET-K0269 proposal and JVET-K0257 proposal) add additional predictions other than the original set of forward, backward, or bidirectional prediction (the first hypothesis), which is called in the present invention For additional assumptions, may include second assumptions, third assumptions, etc. In the present invention, the original set of forward, backward, or bidirectional predictions is called the original hypothesis or the first hypothesis.

The image prediction method described in this application will be described below with reference to FIG. 6 . The image prediction method includes:

602: Analyze the code stream to obtain the multi-hypothesis information index of the current image block to be decoded.

602 corresponds to step 1 mentioned below. The multi-hypothesis information index is the multi-hypothesis mode information mentioned below.

604: Obtain first multi-hypothesis information corresponding to the current image block from the multi-hypothesis information list according to the multi-hypothesis information index, the first multi-hypothesis information includes a motion information index and a first identifier, and the first multi-hypothesis information The motion information index indicates the multi-hypothesis motion information of the current image block to be decoded, the first identifier indicates the first division mode of the multi-hypothesis method, and the first multi-hypothesis information also includes parameters of the first division mode.

In a possible implementation manner, the multi-hypothesis information list includes at least one piece of the first multi-hypothesis information, and at least one piece of second multi-hypothesis information, where the second multi-hypothesis information includes a motion information index and a second identifier, and the second The motion information index of the multi-hypothesis information indicates the multi-hypothesis motion information of an image block, the second identifier indicates the second division method of the multi-hypothesis method, and the second multi-hypothesis information also includes parameters of the second division mode.

In a possible implementation manner, the first identifier and the second identifier are different values of the same flag bit.

In a possible implementation manner, the first division method is triangle division, and the parameter of the first division method indicates the division direction of the triangle division; the second division method is square division, and the parameter of the second division method indicates the The weighting factor for the square division. Of course, it is also possible that the second division method is triangular division, the parameter of the second division method indicates the division direction of the triangle division, the first division method is square division, and the parameter of the first division method indicates the weight of the square division coefficient.

It should be understood that the first division method and the second division method are different division methods, and each division method is used to divide a plurality of assumed predicted image blocks, that is, used to divide the original assumed predicted image block and the additional Hypothetical predicted image blocks. For the specific methods and ways of dividing and predicting image blocks in the multi-hypothesis method, please refer to the content of JVET-K0144 proposal (triangular division) and JVET-K0257 proposal (square division).

In a possible implementation manner, the first division method is triangle division, and the parameter of the first division method indicates the division direction of the triangle division, or the first division method is square division, and the parameter of the first division method indicates The weighting factor for this square division.

606: Perform motion compensation according to the multi-hypothesis encoded motion information of the image block currently to be decoded, so as to obtain multiple hypothetical prediction image blocks.

604 and 606 correspond to step 2 below.

608: According to the first identifier and the parameters of the first division mode, use the first division mode to process the multiple hypothetical predicted image blocks to obtain a predicted image block of the current image block to be decoded.

608 corresponds to step 3 below. The predicted image block of the current image block to be decoded may also be referred to as the final inter-frame predicted image of the current image block to be decoded.

In a possible implementation manner, the method further includes: according to the motion information index of the first multi-hypothesis information, acquiring the multi-hypothesis motion of the current image block to be decoded from the candidate motion information list of the current image block to be decoded Information, the multi-hypothesis motion information of the current image block to be decoded includes the motion information of the original hypothesis and the motion information of the additional hypothesis.

In a possible implementation manner, the motion information index of the first multi-hypothesis information includes a first index and a second index, and according to the motion information index of the first multi-hypothesis information, the candidate of the current image block to be decoded Obtaining the multi-hypothesis motion information of the current image block to be decoded from the motion information list includes: obtaining the motion information of the original hypothesis from the first candidate motion information list of the current image block according to the first index, and obtaining the motion information of the original hypothesis according to the first index The second index is to obtain the motion information of the additional hypothesis from the second candidate motion information list of the current image block.

In this way, using the above method, since the code stream carries the division identifier indicating the multi-hypothesis method, the decoder can determine which division method to use in the multi-hypothesis method to predict the image, that is, for different image blocks, it can be Different division methods are used to obtain the prediction block of the image block, so that the image block can use a division method more suitable for its own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

The process of obtaining a decoded data block by using the image prediction method corresponding to FIG. 6 is described in detail below. For further implementation details of the above image prediction method, reference may be made to the detailed description below. In the following process, the current block is the current image block described above.

The decoding process includes steps 1 to 4. A block on which decoding processing is being performed is called a current block.

Step 1: Parse the code stream of the current block to obtain the prediction mode information of the current block

Specifically, the mode information of the current block includes but not limited to merge, skip, inter mode information, multi-hypothesis mode information and other coding information.

Step 2: Obtain the motion information of the current block according to the prediction mode information of the current block;

If it is obtained from the multi-hypothesis mode information that the current block adopts the multi-hypothesis coding mode, the multi-hypothesis information of the current block is obtained.

Specifically include the following:

Generate a candidate list of fused motion information. Specifically, it includes: adding the spatial candidate and the temporal candidate of the current block to the fusion motion information candidate list of the current block, and the method is the same as that in HEVC. As shown in FIG. 1 , spatial fusion candidates include A0, A1, B0, B1, and B2, and temporal fusion candidates include T0 and T1. In VTM, temporal fusion candidates also include candidates provided by Adaptive Temporal Motion Vector Prediction (ATMVP) techniques. The present invention does not relate to the process related to generating the fusion motion information candidate list. This process can be carried out by using methods in HEVC or VTM, or other methods for generating fusion motion information candidate lists, such as the method in the JVET-K0257 proposal.

Then obtain the motion information of two hypotheses and the multi-hypothesis motion compensation information, the decoding end: according to the multi-hypothesis information index carried in the code stream, from the multi-hypothesis information list (hereinafter also referred to as the combination list of multi-hypothesis combination mode and motion information) Determine the multi-hypothesis information corresponding to the current block. Each multi-hypothesis information in the multi-hypothesis information list includes: motion index, specifically the fusion of the fusion motion information candidate list index of the first hypothesis (that is, the original hypothesis above) and the second hypothesis (that is, the additional hypothesis above) The motion information candidate list index, the division mode identifier (that is, the first identifier or the second identifier above) and the division mode information (that is, the parameter of the first division method or the parameter of the second division method above).

The motion information of the first hypothesis is confirmed from the fusion motion information candidate list according to the fusion motion information candidate list index of the first hypothesis in the multi-hypothesis information corresponding to the current block. The motion information of the second hypothesis is confirmed from the fusion motion information candidate list according to the fusion motion information candidate list index of the second hypothesis in the multi-hypothesis information corresponding to the current block. Wherein, the division mode identification is used to indicate that the division mode is triangular division or square division. When the division mode identification indicates that the division mode is triangular division, the division mode information indicates the division direction of the triangular division mode; when the division mode identification indicates that the division mode is square When dividing, the division mode information indicates the weighting coefficient of the square division.

Wherein, the division mode identification is used to indicate that the division mode is triangular division or square division. When the division mode identification indicates that the division mode is triangular division, the division mode information indicates the division direction of the triangular division mode; when the division mode identification indicates that the division mode is square When dividing, the division mode information indicates the weighting coefficient of the square division.

The multi-hypothesis information list, that is, combination[], includes but is not limited to the following examples, one of which includes multiple combinations, and each combination combination includes {parameters of triangle division mode or square division mode, the first hypothesis fused motion information candidate list index, fused motion information candidate list index of the second assumption, division mode flag}. Of course, the present application does not limit the sequence of these four parameters in each combination, that is, the sequence of indicated information may be different in examples not listed.

In this application, when the square is divided, the first hypothetical fused motion information candidate list index and the second hypothetical fused motion information candidate list index use a preset index combination, including but not limited to the combination of index, index+1, and also includes The combination of index+m, index+n, where m and n are both less than the length of the fusion motion information candidate list, and m is not equal to n. And, because the triangulation parameter list saves a variety of specific implementation methods, for example There can be about 40 types, and a part of implementation methods can be randomly selected or specified as parameters for triangulation in the multi-hypothesis information list described in this application.

Each of Examples 1 to 4 includes 10 pieces of multi-hypothetical information.

Example one:

Example two:

Example three:

Example four:

If all image blocks to be decoded are only divided into triangles, there are many division methods (more than 40 types), and the processing complexity of triangles is higher than that of squares, resulting in low efficiency of obtaining the predicted image block of the current block. However, if all the image blocks to be decoded only use the square division mode, the combination of motion information of the additional hypothesis is limited, and both the original hypothesis and the additional hypothesis use square division, the way of multi-hypothesis fusion is relatively simple, and the encoding and decoding effect is not good. However, in this application, a combination only needs 10 specific division methods (one piece of multi-hypothesis information corresponds to a division method), which can meet the encoding and decoding requirements, which reduces the complexity compared with triangular division alone, and can also ensure decoding efficiency and decoding performance.

However, with the method of the present application, since the code stream carries the division identifier indicating the multi-hypothesis method, the decoder can determine which division method to use in the multi-hypothesis method to predict the image, that is, for different image blocks, it can be Different division methods are used to obtain the prediction block of the image block, so that the image block can use a division method more suitable for its own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

If the current block does not use the multi-hypothesis coding mode according to the multi-hypothesis mode information, execute:

If the current block is in merge/skip mode, generate a candidate list of fusion motion information. Specifically, it includes: adding the spatial candidate and the temporal candidate of the current block to the fusion motion information candidate list of the current block, and the method is the same as that in HEVC. As shown in FIG. 1 , spatial fusion candidates include A0, A1, B0, B1, and B2, and temporal fusion candidates include T0 and T1. In VTM, temporal fusion candidates also include candidates provided by Adaptive Temporal Motion Vector Prediction (ATMVP) techniques. The present invention does not relate to the process related to generating the fusion motion information candidate list, which can be carried out by using the method in HEVC or VTM, or other methods for generating the fusion motion information candidate list. If the current block is in the Inter MVP mode, generating a motion vector prediction candidate list is a prior art, and can be carried out by using methods in HEVC or VTM, or other methods for generating a motion vector prediction candidate list, which are not limited by the present invention.

Then acquire the motion information of the first hypothesis, and the decoder: if the current block is in the merge/skip mode, determine the motion information of the current block according to the fusion index carried in the code stream. If the current block is in the Inter MVP mode, the motion information of the current block is determined according to the inter-frame prediction direction, reference frame index, motion vector predictor index, and motion vector residual value transmitted in the code stream.

Step 3: Perform inter-frame prediction according to the motion information of the first hypothesis and the second hypothesis, as well as the division mode identifier and division mode information, to obtain a predicted image of the current block.

Wherein, the current block adopts a multi-hypothesis coding mode obtained according to the multi-hypothesis mode information.

Predicted images of the first hypothesis and the second hypothesis are respectively obtained by using the motion information of the first hypothesis and the second hypothesis. Decoder: Use the reference frame direction, reference frame number and motion vector in the motion information of the first hypothesis and the second hypothesis to locate the reference blocks of the first hypothesis and the second hypothesis from the reference frame, and according to the multi-hypothesis combination mode information The prediction blocks of the first hypothesis and the second hypothesis are obtained. The reference frame direction is forward prediction means that the current coding unit selects a reference image from the forward reference image set to obtain the reference block. The reference frame direction is backward prediction means that the current coding unit selects a reference image from the backward reference image set to obtain the reference block. The direction of the reference frame is bidirectional prediction, which refers to selecting a reference image from the forward and backward reference image sets to obtain the reference block. When the bidirectional prediction method is used, there are two reference blocks in the current coding unit, each of which needs to be indicated by a motion vector and a reference frame index.

Then determine the pixel value of the pixel in the prediction block of the current block according to the pixel value of the pixel in the reference block. More specifically, do one of the following:

Method 1: When using the triangular multi-hypothesis mode, obtain the triangular prediction blocks of the first hypothesis and the second hypothesis according to the multi-hypothesis direction, and use the preset weighting coefficients to generate the final prediction image from the first hypothesis and the second hypothesis. The weighting coefficients can be Reference is made to JVET-K0144, and the present invention is not limited thereto. When using the square multi-hypothesis mode, obtain the square prediction blocks of the first hypothesis and the second hypothesis, and use the preset weighting coefficients to generate the final prediction image from the first hypothesis and the second hypothesis. The weighting coefficients can be carried out by referring to JVET-K0257, The present invention is not limited.

Method 1: When using the triangular multi-hypothesis mode, obtain the square prediction blocks of the first hypothesis and the second hypothesis according to the multi-hypothesis direction, and use the preset weighting coefficients to generate the final prediction image from the first hypothesis and the second hypothesis. The weighting coefficients can be Refer to JVET-K0144 using a triangular weighting coefficient matrix, which is not limited in the present invention. When using the square multi-hypothesis mode, obtain the square prediction blocks of the first hypothesis and the second hypothesis, and use the preset weighting coefficients to generate the final prediction image from the first hypothesis and the second hypothesis. The weighting coefficients can refer to JVET-K0257. The present invention is not limited.

If it is obtained from the multi-hypothesis mode information that the current block does not adopt the multi-hypothesis coding mode, the motion information of the first hypothesis is used to perform inter-frame prediction to obtain a predicted image of the current block.

Decoder: Obtain the prediction block from the reference frame by using the reference frame direction, reference frame number and motion vector. The reference frame direction is forward prediction means that the current coding unit selects a reference image from the forward reference image set to obtain the reference block. The reference frame direction is backward prediction means that the current coding unit selects a reference image from the backward reference image set to obtain the reference block. The direction of the reference frame is bidirectional prediction, which refers to selecting a reference image from the forward and backward reference image sets to obtain the reference block. When the bidirectional prediction method is used, there are two reference blocks in the current coding unit, each of which needs to be indicated by a motion vector and a reference frame index. Then determine the pixel value of the pixel in the prediction block of the current block according to the pixel value of the pixel in the reference block. The method in HEVC or VTM can be used for the above, and other methods for generating a motion vector prediction candidate list can also be used, which is not limited in the present invention.

Step 4: Add the final inter-frame prediction image of the current block (that is, the prediction image block of the current image block to be decoded above) and the residual image to obtain the reconstructed image of the current block;

Decoder: If there is a residual in the current block, add the residual information and the predicted image to obtain the reconstructed image of the current block; if there is no residual in the current block, the predicted image is the reconstructed image of the current block.

The above process is an existing technology, for example, the same method as HEVC or VTM can be used, and other motion compensation and image reconstruction methods can also be used, which is not limited in the present invention.

The present application also describes an image prediction device, which can be used to implement the above image prediction method.

In an implementation manner, the image prediction device includes: a memory for storing video data in the form of a code stream, the video data including one or more image blocks; a video decoder for executing the above-mentioned various image prediction methods ( For example, corresponding to FIG. 6 ) and the decoding method, in one implementation mode, the video decoder is used to analyze the code stream to obtain the multi-hypothesis information index of the current image block to be decoded; according to the multi-hypothesis information index, from the multi-hypothesis information list In the method, the first multi-hypothesis information corresponding to the current image block is obtained, the first multi-hypothesis information includes a motion information index and a first identifier, and the motion information index of the first multi-hypothesis information indicates the number of the current image block to be decoded Hypothesis motion information, the first identifier indicates the first division method of the multi-hypothesis method, and the first multi-hypothesis information also includes the parameters of the first division mode; perform motion according to the motion information of the multi-hypothesis encoding of the current image block to be decoded Compensating to obtain a plurality of hypothetical predicted image blocks; according to the first identifier and the parameters of the first division method, using the first division method to process the plurality of hypothetical predicted image blocks to obtain the current image block to be decoded predicted image blocks.

Certainly, the video decoder may also execute various other implementation manners of the method corresponding to FIG. 6 , which will not be repeated here. In addition, obviously, the video decoder can be implemented by the processor in FIG. 5 running the code in the memory in FIG. 5 .

In another implementation manner, as shown in FIG. 7 , the image prediction device 700 includes a parsing module 701 for parsing a code stream to obtain a multi-hypothesis information index of the current image block to be decoded.

The first query module 702 is configured to obtain the first multi-hypothesis information corresponding to the current image block from the multi-hypothesis information list according to the multi-hypothesis information index, and the first multi-hypothesis information includes the motion information index and the first multi-hypothesis information An identifier, the motion information index of the first multi-hypothesis information indicates the multi-hypothesis motion information of the current image block to be decoded, the first identifier indicates the first division method of the multi-hypothesis method, and the first multi-hypothesis The assumption information also includes parameters of the first division manner.

The motion compensation module 703 is configured to perform motion compensation according to the multi-hypothesis coded motion information of the current image block to be decoded, so as to obtain multiple hypothetical prediction image blocks.

A division processing module 704, configured to use the first division mode to process the plurality of hypothetical predicted image blocks according to the first identifier and the parameters of the first division mode, so as to obtain the current image block to be decoded predicted image blocks.

The above-mentioned modules are only schematically divided, and the functions of multiple modules may be realized by one module in actual implementation.

The image prediction device 700 can execute the method corresponding to FIG. 6 and other various implementation manners of the foregoing methods, which will not be repeated here.

Using the above-mentioned image prediction device, since the code stream carries the division identifier indicating the multi-hypothesis method, the decoder can determine which division method to use in the multi-hypothesis method to predict the image, that is, for different image blocks, you can use Different division methods are used to obtain the prediction block of the image block, so that the image block can use a division method more suitable for its own characteristics, thereby improving the efficiency of image prediction and reducing the decoding time.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (22)

1. An image prediction method, characterized in that, the image prediction method comprises: Analyze the code stream to obtain the multi-hypothesis information index of the current image block to be decoded; According to the multi-hypothesis information index, the first multi-hypothesis information corresponding to the current image block is obtained from the multi-hypothesis information list, the first multi-hypothesis information includes a motion information index and a first identifier, and the first multi-hypothesis information The motion information index indicates the multi-hypothesis motion information of the current image block to be decoded, the first identifier indicates the first division method of the multi-hypothesis method, and the first multi-hypothesis information also includes the first division mode parameters; performing motion compensation according to the multi-hypothesis coded motion information of the current image block to be decoded, so as to obtain multiple hypothetical prediction image blocks; According to the first identifier and the parameters of the first division mode, the first division mode is used to process the plurality of hypothetical predicted image blocks, so as to obtain the predicted image block of the current image block to be decoded. 2. The method according to claim 1, wherein the multi-hypothesis information list includes at least one piece of the first multi-hypothesis information, and at least one second piece of multi-hypothesis information, and the second multi-hypothesis information includes motion An information index and a second identification, the motion information index of the second multi-hypothesis information indicates the multi-hypothesis motion information of an image block, the second identification indicates the second division method of the multi-hypothesis method, and the second multi-hypothesis The assumption information also includes parameters of the second division manner. 3. The method according to claim 2, wherein the first identifier and the second identifier are different values of the same flag bit. 4. The method according to claim 2 or 3, wherein the first division method is triangle division, the parameter of the first division method indicates the division direction of the triangle division, and the second division method is square division, and the parameter of the second division mode indicates the weighting coefficient of the square division. 5. The method according to any one of claims 1 to 3, wherein the first division method is triangular division, and the parameter of the first division mode indicates the division direction of the triangular division, or the second division method is triangular division. One division mode is square division, and the parameter of the first division mode indicates the weighting coefficient of the square division. 6. The method according to claim 1, further comprising: According to the motion information index of the first multi-hypothesis information, obtain the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded, the current image block to be decoded The multi-hypothesis motion information of includes the motion information of the original hypothesis and the motion information of the additional hypothesis. 7. The method according to claim 6, wherein the motion information index of the first multi-hypothesis information includes a first index and a second index, and the motion information according to the first multi-hypothesis information Indexing, obtaining the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded, including: According to the first index, obtain the motion information of the original hypothesis from the first candidate motion information list of the current image block, and according to the second index, obtain the second candidate motion information of the current image block Get the motion information of the additional hypothesis in the list. 8. An image prediction device, characterized in that the image prediction device comprises: A memory for storing video data in the form of a code stream, the video data including one or more image blocks; A video decoder, configured to parse the code stream to obtain the multi-hypothesis information index of the current image block to be decoded; According to the multi-hypothesis information index, the first multi-hypothesis information corresponding to the current image block is obtained from the multi-hypothesis information list, the first multi-hypothesis information includes a motion information index and a first identifier, and the first multi-hypothesis information The motion information index indicates the multi-hypothesis motion information of the current image block to be decoded, the first identifier indicates the first division method of the multi-hypothesis method, and the first multi-hypothesis information also includes the first division mode parameters; performing motion compensation according to the multi-hypothesis coded motion information of the current image block to be decoded, so as to obtain multiple hypothetical prediction image blocks; According to the first identifier and the parameters of the first division mode, the first division mode is used to process the plurality of hypothetical predicted image blocks, so as to obtain the predicted image block of the current image block to be decoded. 9. The image prediction device according to claim 8, wherein the multi-hypothesis information list includes at least one piece of the first multi-hypothesis information, and at least one piece of second multi-hypothesis information, the second multi-hypothesis information Including a motion information index and a second identifier, the motion information index of the second multi-hypothesis information indicates the multi-hypothesis motion information of an image block, the second identifier indicates the second division method of the multi-hypothesis method, the first The two-multiple-hypothesis information also includes parameters of the second division method. 10. The image prediction device according to claim 9, wherein the first identifier and the second identifier are different values of the same flag bit. 11. The image prediction device according to claim 9 or 10, wherein the first division method is triangular division, the parameter of the first division mode indicates the division direction of the triangular division, and the second The division mode is square division, and the parameter of the second division mode indicates the weighting coefficient of the square division. 12. The image prediction device according to any one of claims 8 to 10, wherein the first division method is triangle division, and the parameter of the first division method indicates the division direction of the triangle division, or the The first division method is square division, and the parameter of the first division method indicates the weighting coefficient of the square division. 13. The image prediction device according to claim 8, wherein the video decoder is further used for: According to the motion information index of the first multi-hypothesis information, obtain the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded, the current image block to be decoded The multi-hypothesis motion information of includes the motion information of the original hypothesis and the motion information of the additional hypothesis. 14. The image prediction device according to claim 13, wherein the motion information index of the first multi-hypothesis information includes a first index and a second index, and in the The motion information index, obtaining the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded, the video decoder is also used for: According to the first index, obtain the motion information of the original hypothesis from the first candidate motion information list of the current image block, and according to the second index, obtain the second candidate motion information of the current image block Get the motion information of the additional hypothesis in the list. 15. An image prediction device, characterized in that the image prediction device comprises: The analysis module is used to analyze the code stream to obtain the multi-hypothesis information index of the current image block to be decoded; The first query module is configured to obtain the first multi-hypothesis information corresponding to the current image block from the multi-hypothesis information list according to the multi-hypothesis information index, the first multi-hypothesis information includes a motion information index and a first identifier, The motion information index of the first multi-hypothesis information indicates the multi-hypothesis motion information of the current image block to be decoded, the first identifier indicates the first division method of the multi-hypothesis method, and the first multi-hypothesis information also Including parameters of the first division method; A motion compensation module, configured to perform motion compensation according to the motion information of the multi-hypothesis encoding of the current image block to be decoded, so as to obtain multiple hypothetical prediction image blocks; A division processing module, configured to use the first division mode to process the plurality of hypothetical predicted image blocks according to the first identifier and the parameters of the first division mode, so as to obtain the current image block to be decoded Predict image blocks. 16. The image prediction device according to claim 15, wherein the multi-hypothesis information list includes at least one piece of the first multi-hypothesis information, and at least one piece of second multi-hypothesis information, the second multi-hypothesis information Including a motion information index and a second identifier, the motion information index of the second multi-hypothesis information indicates the multi-hypothesis motion information of an image block, the second identifier indicates the second division method of the multi-hypothesis method, the first The two-multiple-hypothesis information also includes parameters of the second division method. 17. The image prediction device according to claim 16, wherein the first identifier and the second identifier are different values of the same flag bit. 18. The image prediction device according to claim 16 or 17, wherein the first division method is triangular division, the parameter of the first division mode indicates the division direction of the triangular division, and the second The division mode is square division, and the parameter of the second division mode indicates the weighting coefficient of the square division. 19. The image prediction device according to any one of claims 15 to 17, wherein the first division method is triangle division, and the parameter of the first division method indicates the division direction of the triangle division, or the The first division method is square division, and the parameter of the first division method indicates the weighting coefficient of the square division. 20. The image prediction device according to claim 15, characterized in that, the image prediction device further comprises a second query module, the second query module is used for the motion information based on the first multi-hypothesis information Indexing, obtaining the multi-hypothesis motion information of the current image block to be decoded from the candidate motion information list of the current image block to be decoded, the multi-hypothesis motion information of the current image block to be decoded includes motion information of the original hypothesis and additional Hypothetical motion information. 21. The image prediction device according to claim 20, wherein the motion information index of the first multi-hypothesis information includes a first index and a second index, and the second query module is configured to index, obtaining the motion information of the original assumption from the first candidate motion information list of the current image block, and according to the second index, obtaining the motion information from the second candidate motion information list of the current image block Motion information for additional hypotheses. 22. A computer storage medium, wherein an executable instruction set is stored in the computer storage medium, and when the instruction set is run on a computer, the computer is made to execute the method according to any one of claims 1 to 7.

Images