RU2823558C2 - Encoder, decoder and corresponding methods using adaptive loop filter - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to video encoding and decoding. Result is achieved by the fact that the encoding method implemented by the decoding device comprises a step for obtaining a bit stream, in which at least one bit in the bit stream represents a syntax element for the current block, wherein the syntax element indicates a truncation index of the adaptive loop filter (ALF); parsing the bitstream to obtain a syntax element value for the current block, wherein the syntax element is encoded using a fixed-length code; applying adaptive contour filtering to the current block based on the value of the syntax element for the current block. Here, the fixed-length code means that all possible values of the syntax element are signalled using the same number of bits.

EFFECT: high efficiency of video compression.

21 cl, 24 dwg, 3 tbl

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application claims priority. US Provisional Patent Application No. 62/843,431, filed May 4, 2019. The disclosure of the above patent application is incorporated herein by reference in its entirety.

TECHNICAL FIELD

Embodiments of the present application (disclosure) generally relate to the field of image processing and, more specifically, to filtering samples of blocks in a picture.

BACKGROUND OF THE ART

Video encoding (video encoding and decoding) is used in a wide range of digital video applications, such as digital broadcast television, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat, video conferencing, DVD and Blu-ray discs, systems for collecting and editing video content and video cameras for security applications.

The amount of video data required to present even a relatively short video can be significant, which can lead to difficulties when the data must be streamed or otherwise transmitted over a communications network with limited bandwidth. Thus, video data is typically compressed before being transmitted over modern telecommunications networks. Video size can also be an issue when the video is stored on a storage device, as memory resources may be limited. Video compression devices often use software and/or hardware at the source to encode video data before transmission or storage, thereby reducing the amount of data required to represent digital video images. The compressed data is then received at its destination by a video decompression device, which decodes the video data. With limited network resources and ever-increasing demands for higher video quality, improved compression and decompression techniques are desired that improve compression ratios with little or no loss in picture quality.

SUMMARY OF THE INVENTION

Embodiments of the present application provide apparatus and methods for encoding and decoding according to independent claims.

The above and other goals are achieved by the invention according to independent claims. Additional forms of implementation are obvious from the dependent claims, description and figures.

According to a first aspect of the present disclosure, there is provided an encoding method implemented by a decoding apparatus, comprising:

obtaining a bitstream, wherein at least one bit in the bitstream represents a syntax element for the current block, wherein the syntax element indicates a truncation index of the truncation value for the adaptive loop filter (ALF); parsing the bitstream to obtain the value of a syntax element for the current block, wherein the syntax element is encoded using a fixed-length code; Apply adaptive contour filtering to the current block based on the value of the syntax element for the current block. Here, a fixed-length code means that all possible values of a syntax element are signaled using the same number of bits. This provides a simpler way to signal truncation parameters. Additionally, encoding efficiency is improved.

In a possible implementation of the method according to the first aspect, as such, the value of the syntax element for the current block is obtained using only at least one bit.

In a possible implementation of the method according to any previous implementation or the first aspect as such, at least one bit means two bits.

In a possible implementation of the method according to any previous implementation or the first aspect as such, at least one bit in the bit stream represents the value of a syntax element.

In a possible implementation of the method according to any previous implementation or the first aspect as such, the syntax element is for an adaptive chrominance edge filter or an adaptive luma edge filter.

In a possible implementation of the method according to any previous implementation or the first aspect as such, the trim value is used to determine a trim range used to determine the trim range used to limit (or truncate) the difference between the target sample value and the nearest sample value, and the limited value difference samples (or the trimmed difference of the sample values) is used to modify the sample target value during the ALF process.

In a possible implementation of the method according to any previous implementation or the first aspect as such, applying adaptive contour filtering to the current block based on the value of the syntax element comprises obtaining a truncation value based on the value of the syntax element; using a trim value to limit (or truncate) the difference between the current block's sample target value and the nearest sample value; multiplying the limited sample difference (or truncated sample difference) by an adaptive loop filter (ALF) factor; using the result of the multiplication to modify the sample target value.

In a possible implementation of the method according to any previous implementation or the first aspect as such, the truncation value is determined using the truncation index specified by the syntax element and the mapping between the truncation indices and the truncation values.

In a possible implementation of the method according to any previous implementation or the first aspect as such, the fixed-length code comprises a binary representation of an unsigned integer using at least one bit. In other words, at least one bit is a binary representation of the value of the syntax element, and the value of the syntax element is an unsigned integer.

In an exemplary implementation of the method according to any previous implementation or the first aspect, as such, the syntax element is applied to a set of blocks, and the current block is one block in the set of blocks.

In a possible implementation of the method according to any previous implementation or the first aspect as such, the syntax element is at the slice level.

According to a second aspect of the present disclosure, an encoding method implemented by a decoding apparatus is provided, comprising:

obtaining a bitstream in which at least one bit in the bitstream represents a syntax element for the current block, wherein the syntax element is an adaptive loop filter (ALF) clipping value index and/or an ALF coefficient parameter; parsing the bitstream to obtain a syntax element value for the current block, wherein the syntax element value for the current block is obtained using only at least one bit of the syntax element; Apply adaptive contour filtering to the current block based on the value of the syntax element for the current block.

In a possible implementation of the method according to the second aspect, the syntax element is itself encoded using a fixed length code.

In a possible implementation of the method according to the previous implementation, the fixed-length code comprises a binary representation of an unsigned integer using at least one bit. In other words, at least one bit is a binary representation of the value of the syntax element, and the value of the syntax element is an unsigned integer.

In a possible implementation of the method according to the second aspect as such or any previous implementation thereof, the syntax element itself determines the meaning of the syntax element.

In a possible implementation of the method according to the second aspect per se or any previous implementation thereof, at least one bit in the bit stream represents the value of the syntax element.

In a possible implementation of the method according to the second aspect as such or any previous implementation thereof, the truncation value index ALF indicates the truncation index of the truncation value for the adaptive loop filter (ALF).

In a possible implementation of the method according to the second aspect as such or any previous implementation thereof, the ALF coefficient parameter is used to obtain the ALF coefficient.

In a possible implementation of the method according to the second aspect as such or any previous implementation thereof, the value of the syntax element for the current block is obtained using only at least one bit of the syntax element, meaning the value of the syntax element is determined by the syntax element itself.

In a possible implementation of the method according to the second aspect per se or any previous implementation thereof, the syntax element is applied to a set of blocks, and the current block is one block in the set of blocks.

In a possible implementation of the method according to the second aspect as such or any previous implementation thereof, the syntax element is at the slice level.

In a possible implementation of the method according to the second aspect as such or any previous implementation thereof, the ALF coefficient parameter is used to determine the ALF coefficient.

In a possible implementation of the method according to the first aspect or the second aspect as such or any previous implementation thereof, the syntax element is an ALF truncation value index, and at least one bit representing the syntax element is two bits.

In a possible implementation of the method according to the previous implementation, the truncation value index ALF identifies one truncation value from four truncation values.

In an exemplary implementation of the method according to the first or second aspect as such or any previous implementation thereof, the truncation value index value ALF is used to determine a truncation range, wherein the truncation range is used in the adaptive loop filtering process.

According to a third aspect of the present disclosure, an encoding method implemented by an encoding apparatus is provided, comprising:

determining a syntax element value for the current block, wherein the syntax element specifies a truncation index of the truncation value for the adaptive loop filter (ALF); generating a bit stream based on the value of the syntax element, wherein at least one bit in the bit stream represents the syntax element, wherein the syntax element is encoded using a fixed length code.

In a possible implementation of the method according to the third aspect, as such, the at least one syntax element bit is obtained using only the value of the syntax element for the current block.

In a possible implementation of the method according to the third aspect as such or any previous implementation thereof, the value of a syntax element corresponding to the minimum difference (eg, mean square error or rate distortion index) between the reconstructed block (or filtered block) of the current block and the original signal of the current block, and the reconstructed the block (or filtered block) is the result using the syntax element value, and the minimum difference is lower than any other difference corresponding to any other possible syntax element value.

In a possible implementation of the method according to the third aspect per se or any previous implementation thereof, at least one bit in the bit stream represents the value of the syntax element.

In a possible implementation of the method according to the third aspect as such or any previous implementation thereof, the trim value is used to determine a trim range used to limit (or truncate) the difference between the target sample value and the nearest sample value, and the limited sample value difference (or truncated difference sample values) is used to modify the sample target value during the ALF process.

In a possible implementation of the method according to the third aspect as such or any previous implementation thereof, the fixed-length code comprises a binary representation of an unsigned integer using at least one bit. In other words, at least one bit is a binary representation of the value of the syntax element, and the value of the syntax element is an unsigned integer.

In a possible implementation of the method according to the third aspect per se or any previous implementation thereof, the syntax element is applied to a set of blocks, and the current block is one block in the set of blocks.

In a possible implementation of the method according to the third aspect as such or any previous implementation thereof, the syntax element is at the slice level.

According to a fourth aspect of the present disclosure, an encoding method implemented by an encoding apparatus is provided, the method comprising:

determining a syntax element value for the current block, wherein the syntax element is an adaptive loop filter (ALF) clipping value index and/or an ALF filter coefficient parameter; generating a bit stream based on the value of the syntax element, wherein at least one bit in the bit stream represents the syntax element, wherein the at least one bit of the syntax element is obtained using only the value of the syntax element for the current block.

In a possible implementation of the method according to the fourth aspect as such or any previous implementation thereof, the syntax element is encoded using a fixed length code.

In a possible implementation of the method according to the previous implementation, the fixed-length code comprises a binary representation of an unsigned integer using at least one bit. In other words, at least one bit is a binary representation of the value of the syntax element, and the value of the syntax element is an unsigned integer.

In a possible implementation of the method according to the fourth aspect per se or any previous implementation thereof, at least one bit in the bit stream represents the value of the syntax element.

In a possible implementation of the method according to the fourth aspect per se or any previous implementation thereof, the syntax element is applied to a set of blocks, and the current block is one block in the set of blocks.

In a possible implementation of the method according to the fourth aspect per se or any previous implementation thereof, the syntax element is at the slice level.

In a possible implementation of the method according to the fourth aspect as such or any previous implementation thereof, the ALF coefficient parameter is used to determine the ALF coefficient.

In a possible implementation of the method according to the third aspect or the fourth aspect as such or any previous implementation thereof, the syntax element is an index of the ALF truncation value, and at least one bit representing the syntax element is two bits.

In a possible implementation of the method according to the previous implementation of the fourth aspect, the truncation value index ALF identifies one truncation value from four truncation values.

In a possible implementation of the method according to the third aspect or the fourth aspect as such or any previous implementation thereof, the truncation value index value ALF is used to determine a truncation range, wherein the truncation range is used in the adaptive loop filtering process.

According to a fifth aspect of the present disclosure, a decoder is provided comprising processing circuitry for performing a method according to the first or second aspect or any implementation thereof.

According to a sixth aspect of the present disclosure, an encoder is provided comprising processing circuitry for performing a method according to the third or fourth aspect or any implementation thereof.

According to a seventh aspect of the present disclosure, there is provided a computer program product comprising program code for performing a method according to any one of the first to fourth aspects, or any implementation thereof.

According to an eighth aspect of the present disclosure, there is provided a non-transitory computer-readable medium carrying program code that, when executed by a computer device, causes the computer device to perform a method according to any one of the first to fourth aspects or any implementation thereof.

According to the ninth aspect of the present disclosure, a decoder is provided comprising:

one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures a decoder to execute a method in accordance with the first or second aspect or any implementation thereof.

According to the tenth aspect of the present disclosure, an encoder is provided comprising:

one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures an encoder to perform the method of the third or fourth aspect or any other implementation thereof.

According to an eleventh aspect of the present disclosure, a decoder is provided comprising:

an entropy decoding unit configured to obtain a bit stream, wherein at least one bit in the bit stream represents a syntax element for the current block, wherein the syntax element indicates a truncation index of a truncation value for the adaptive loop filter (ALF);

wherein the entropy decoding unit is further configured to parse the bitstream to obtain a syntax element value for the current block, wherein the syntax element is encoded using a fixed length code; and a filtering block configured to apply adaptive loop filtering to the current block based on the value of the syntax element for the current block.

According to the twelfth aspect of the present disclosure, a decoder is provided comprising:

an entropy decoding unit configured to obtain a bitstream, wherein at least one bit in the bitstream represents a syntax element for the current block, wherein the syntax element is an adaptive loop filter (ALF) clipping value index or an ALF coefficient parameter;

wherein the entropy decoding unit is further configured to parse the bit stream to obtain a syntax element value for the current block, wherein the syntax element value for the current block is obtained using only at least one bit of the syntax element; and a filtering block configured to apply adaptive loop filtering to the current block based on the value of the syntax element for the current block.

According to the thirteenth aspect of the present disclosure, an encoder is provided comprising:

a determination block configured to determine the value of a syntax element for the current block, wherein the syntax element indicates a truncation index of the truncation value for the adaptive loop filter (ALF); an entropy encoding unit configured to generate a bit stream based on the value of the syntax element, wherein at least one bit in the bit stream represents the syntax element, wherein the syntax element is encoded using a fixed length code.

According to the fourteenth aspect of the present disclosure, an encoder is provided comprising:

a determination block configured to determine the value of a syntax element for the current block, wherein the syntax element is an ALF truncation value index or an adaptive loop filter (ALF) filter coefficient parameter; an entropy encoding block configured to generate a bit stream based on a syntax element value, wherein at least one bit in the bit stream represents a syntax element, wherein the at least one syntax element bit is obtained using only the syntax element value for the current block.

According to the fifteenth aspect of the present disclosure, an encoding method implemented by a decoding apparatus is provided, comprising:

obtaining a bit stream, where n bits in the bit stream represent a syntax element indicating a truncation index of a truncation value for an adaptive loop filter (ALF), where n is an integer greater than or equal to 0;

parsing the bit stream to obtain the value of a syntax element for the current block, wherein the value of the syntax element is a binary representation of an unsigned integer using n bits; Apply adaptive contour filtering to the current block based on the value of the syntax element for the current block.

In a possible implementation form of the method according to the fifteenth aspect, as such, the syntax element may be a slice-level syntax element.

According to a sixteenth aspect of the present disclosure, an encoding method implemented by an encoding apparatus is provided, comprising:

determining a value of a syntax element indicating a truncation index of the truncation value for the adaptive loop filter (ALF), where n is an integer greater than or equal to 0; generating a bitstream containing n bits based on the value of the syntax element, wherein the binary representation of an unsigned integer using n bits is the value of the syntax element.

In a possible implementation form of the method according to the sixteenth aspect, as such, the syntax element may be a slice-level syntax element.

According to the seventeenth aspect of the present disclosure, a decoder is provided comprising:

an entropy decoding unit configured to obtain a bit stream, where n bits in the bit stream represent a slice-level syntax element indicating a truncation index of a truncation value for an adaptive loop filter (ALF), where n is an integer greater than or equal to 0; wherein the entropy decoding unit is further configured to parse the bit stream to obtain a syntax element value for the current block, wherein the syntax element value is a binary representation of an unsigned integer using n bits; a filtering block configured to apply adaptive loop filtering to the current block based on the value of the syntax element for the current block.

According to the eighteenth aspect of the present disclosure, an encoder is provided comprising:

a determiner configured to determine a value of a slice-level syntax element indicating a truncation index of a truncation value for an adaptive loop filter (ALF), where n is an integer greater than or equal to 0; an entropy encoding unit configured to generate a bit stream containing n bits based on the value of the syntax element, wherein a binary representation of an unsigned integer using n bits is the value of the syntax element.

According to the nineteenth aspect of the present disclosure, a decoder is provided including processing circuitry for performing the method according to the fifteenth aspect or any implementation thereof.

According to the twentieth aspect of the present disclosure, an encoder is provided comprising processing circuitry for executing the method of the sixteenth aspect or any implementation thereof.

According to the twenty-first aspect of the present disclosure, there is provided a computer program product containing program code for executing the method of the fifteenth aspect or the sixteenth aspect or any implementation thereof.

According to the twenty-second aspect of the present disclosure, there is provided a non-transitory computer-readable medium carrying program code that, when executed by a computer device, causes the computer device to perform the method of any of the fifteenth aspect or the sixteenth aspect or any implementation thereof.

According to the twenty-third aspect of the present disclosure, a decoder is provided comprising:

one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures a decoder to execute the method of the fifteenth aspect or any implementation thereof.

According to the twenty-fourth aspect of the present disclosure, there is provided an encoder comprising:

one or more processors; and a non-volatile computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures an encoder to execute the method of the sixteenth aspect or any implementation thereof.

According to a twenty-fifth aspect of the present disclosure, a non-volatile storage medium is provided comprising a bitstream comprising n bits, wherein an unsigned binary integer representation using n bits is the value of a syntax element, and the syntax element indicates a truncation index of the truncation value for an adaptive loop filter (ALF) ), where n is an integer greater than or equal to 0.

According to a twenty-sixth aspect of the present disclosure, a non-volatile storage medium is provided comprising a bit stream, wherein at least one bit in the bit stream represents a syntax element, wherein the syntax element is encoded using a fixed length code and indicates a truncation index of a truncation value for an adaptive loop filter (ALF). ).

In a possible implementation form of the method according to the twenty-sixth aspect as such, in which the syntax element itself determines the meaning of the syntax element.

According to a twenty-seventh aspect of the present disclosure, a non-volatile storage medium is provided comprising a bitstream, wherein at least one bit in the bitstream represents a syntax element, wherein the syntax element is an adaptive loop filter (ALF) clipping value index or an ALF filter coefficient parameter, and at least one bit of the syntax element is obtained using only the value of the syntax element.

According to the twenty-eighth aspect of the present disclosure, a non-volatile storage medium is provided containing a bitstream encoded in a manner of any aspect or any implementation thereof.

Details of one or more embodiments are set forth in the accompanying drawings and in the following description. Other features, objects and advantages will be apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following embodiments, the inventions are described in more detail with reference to the accompanying figures and drawings, in which:

FIG. 1A is a block diagram showing an example of a video encoding system configured to implement embodiments of the invention; FIG.

FIG. 1B is a block diagram showing another example of a video encoding system configured to implement embodiments of the invention; FIG.

FIG. 2 is a block diagram showing an example of a video encoder configured to implement embodiments of the invention; FIG.

FIG. 3 is a block diagram showing an example of a structure of a video decoder configured to implement embodiments of the invention; FIG.

FIG. 4 is a block diagram illustrating an example of encoding equipment or decoding equipment;

FIG. 5 is a block diagram illustrating another example of encoding equipment or decoding equipment;

Figure 6 shows the shapes of the ALF filter, chromaticity diamond 5×5, luminance diamond 7×7;

FIG. 7 illustrates the classification of subsampled ALF blocks;

FIG. 8 illustrates VTM-5.0 ALF luminance and chrominance truncation parameter signaling;

FIG. 9 illustrates modified VTM-5.0 ALF luma and chrominance truncation parameter signaling where the truncation parameters are transmitted using a 2-bit fixed length code; FIG.

FIG. 10 is a flowchart illustrating a method according to the first aspect of the present disclosure; FIG.

FIG. 11 is a flow diagram illustrating a method according to the second aspect of the present disclosure; FIG.

FIG. 12 is a flowchart illustrating a method according to the third aspect of the present disclosure; FIG.

FIG. 13 is a flowchart illustrating a method according to the fourth aspect of the present disclosure; FIG.

FIG. 14 is a block diagram illustrating a decoder according to a fifth aspect of the present disclosure; FIG.

FIG. 15 is a block diagram illustrating an encoder according to a sixth aspect of the present disclosure; FIG.

FIG. 16 is a block diagram illustrating a decoder according to a ninth aspect of the present disclosure; FIG.

FIG. 17 is a block diagram illustrating an encoder according to a tenth aspect of the present disclosure; FIG.

FIG. 18 is a block diagram illustrating a decoder according to an eleventh aspect of the present disclosure; FIG.

FIG. 19 is a block diagram illustrating a decoder according to the twelfth aspect of the present disclosure; FIG.

FIG. 20 is a block diagram illustrating an encoder according to the thirteenth aspect of the present disclosure; FIG.

FIG. 21 is a block diagram illustrating an encoder according to the fourteenth aspect of the present disclosure; FIG.

FIG. 22 is a block diagram showing an exemplary structure of a content delivery system 3100 that implements a content delivery service; FIG. And

FIG. 23 is a block diagram showing the structure of an exemplary terminal device.

The following identical reference numerals refer to identical or at least functionally equivalent features unless expressly indicated otherwise.

IMPLEMENTATION OF THE INVENTION

In the following description, reference is made to the accompanying drawings, which form a part of this disclosure and which show, by way of illustration, specific aspects of embodiments of the invention or specific aspects in which embodiments of the present invention may be used. It is understood that embodiments of the present invention may be used in other aspects and contain structural or logical changes not shown in the figures. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

For example, it will be understood that the disclosure associated with the described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or a plurality of specific method steps are described, the corresponding apparatus may include one or a plurality of blocks, such as functional blocks, for performing the described one or multiple method steps (e.g., one block performing one or a plurality of steps, or a plurality of blocks , each of which performs one or more of a plurality of steps), even if such one or more units are not expressly described or illustrated in the drawings. On the other hand, for example, if a particular device is described in terms of one or a plurality of blocks, such as functional blocks, the corresponding method may include one step for performing the functionality of the one or multiple blocks (for example, one step performing the functionality of the one or multiple blocks , or a plurality of steps, each of which performs the functionality of one or more of a plurality of blocks), even if such one or multiple steps are not explicitly described or illustrated in the figures. It is further understood that features of various exemplary embodiments and/or aspects described herein may be combined with each other unless specifically stated otherwise.

Video encoding generally refers to the processing of the sequence of pictures that form a video or video sequence. Instead of the term "picture", the terms "frame" or "image" can be used interchangeably in the field of video encoding. Video encoding (or encoding in general) contains two parts: video encoding and video decoding. Video encoding is performed at the source side, typically involving processing (eg, by compression) of the source video images to reduce the amount of data required to represent the video images (for more efficient storage and/or transmission). Video decoding is performed on the receiver (destination) side and usually contains reverse processing compared to the encoder for reconstructing video images. Embodiments relating to "encoding" of video pictures (or pictures in general) are to be understood as referring to "encoding" or "decoding" video pictures or corresponding video sequences. The combination of an encoding part and a decoding part is also called CODEC (encoding and decoding).

In the case of lossless video encoding, the original video images can be restored, i.e. The restored video images are of the same quality as the original video images (provided there are no transmission losses or other data loss during storage or transmission). In the case of lossy video encoding, additional compression is performed, for example by quantization, to reduce the amount of data representing video images that cannot be fully reconstructed in the decoder, i.e. the quality of the restored video images is lower or worse compared to the quality of the original video images.

Several video encoding standards belong to the group of "hybrid lossy video codecs" (i.e., they combine spatial and temporal prediction in the sample domain and 2D transform encoding to apply quantization in the transform domain). Each picture of a video sequence is usually divided into a set of non-overlapping blocks, and encoding is usually performed at the block level. In other words, in an encoder, video is typically processed, that is, encoded at the block (video block) level, for example, using spatial (within a picture) prediction and/or temporal (between pictures) prediction to generate a prediction block, subtracting the prediction block from the current block ( block that is currently being processed/to be processed) to obtain a residual block, transforming the residual block and quantizing the residual block in the transform domain to reduce the amount of data that has to be transmitted (compression), whereas in the decoder the reverse processing is compared to applied by the encoder to the encoded or compressed block to recover the current block for presentation. In addition, the encoder duplicates the decoder's processing loop so that both will generate identical predictions (eg, intra and outer prediction) and/or reconstruction for processing, ie, encoding, of subsequent blocks.

In the following embodiments, video encoding systems 10, video encoder 20, and video decoder 30 are described based on FIGS. from 1 to 3.

FIG. 1A is a schematic block diagram illustrating an exemplary encoding system 10, such as video encoding system 10 (or encoding system 10 for short), which may use the techniques of this present application. Video encoder 20 (or encoder 20 for short) and video decoder 30 (or decoder 30 for short) of video encoding system 10 are examples of devices that may be configured to perform methods in accordance with various examples described herein.

As shown in FIG. 1A, the encoding system 10 includes a source device 12 configured to provide encoded picture data 21, for example, to a destination device 14 for decoding the encoded picture data 13.

The source device 12 contains an encoder 20 and can additionally, i.e. optionally comprising a picture source 16, a preprocessor (or pre-processing unit) 18, such as a picture preprocessor 18, and a communications interface or communications unit 22.

The picture source 16 may contain or be any type of picture capturing device, such as a camera for capturing a real world picture, and/or any type of picture generating device, such as a computer graphics processor for generating a computer animated picture, or any other device for receiving and/or providing real-world pictures, computer-generated pictures (eg, screen content, virtual reality (VR) pictures), and/or any combination thereof (eg, augmented reality (AR) pictures). The source of the picture can be any type of memory or storage that stores any of the above pictures.

In contrast to the preprocessor 18 and the processing performed by the preprocessing unit 18, the picture or picture data 17 may also be referred to as a raw picture or raw picture data 17.

The preprocessor 18 is configured to receive (raw) picture data 17 and perform preprocessing on the picture data 17 to produce a preprocessed picture 19 or preprocessed picture data 19. Preprocessing performed by preprocessor 18 may, for example, include cropping, color format conversion (eg, RGB to YCbCr), color correction, or denoising. It can be understood that the pre-processing module 18 may be an optional component.

Video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21 (further details will be described below, for example based on FIG. 2).

The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any additional processed version thereof) over the communication channel 13 to another device, for example, to the destination device 14 or any other device, for storage or direct restoration (reconstruction).

The recipient device 14 includes a decoder 30 (eg, video decoder 30) and may additionally, i.e. optionally comprising a communication interface or communication unit 28, a post processor 32 (or post processing unit 32) and a display device 34.

The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or any additional processed version thereof), for example, directly from the source device 12 or from any other source, such as a storage device, such as an encoded picture data storage device and provide the encoded picture data 21 to the decoder 30.

The communication interface 22 and the communication interface 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network , such as a wired or wireless network or any combination thereof, or any private and public network, or any combination thereof.

The communications interface 22 may, for example, be configured to package the encoded picture data 21 into an appropriate format, such as packets, and/or process the encoded picture data using any type of transmission encoding or processing for transmission over a communications or communication channel. net.

The communication interface 28, which is analogous to the communication interface 22, may, for example, be configured to receive transmitted data and process the transmission data using any kind of appropriate decoding or transmission processing and/or decompression to obtain encoded picture data 21.

Both communication interface 22 and communication interface 28 can be configured as unidirectional communication interfaces, as shown by the arrow for communication channel 13 in FIG. 1A pointing from source device 12 to destination device 14, or bidirectional communication interfaces, and can be configured , for example, for sending and receiving messages, for example, for establishing a connection, for confirming and exchanging any other information related to the communication channel and/or data transfer, for example, the transfer of encoded picture data.

The decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded picture 31 (further details will be described below, for example, based on FIG. 3 or FIG. 5.

The post-processor 32 of the recipient device 14 is configured to post-process decoded picture data 31 (also referred to as reconstructed picture data), such as decoded picture 31, to obtain post-processed picture data 33, such as post-processed picture 33. The post-processing performed by post-processing unit 32 may comprise, for example, color format conversion (eg, from YCbCr to RGB), color correction, cropping or resampling, or any other processing, for example, to prepare decoded picture data 31 for display, for example, by display device 34.

The display device 34 of the recipient device 14 is configured to receive post-processed picture data 33 for displaying the picture, for example, to a user or viewer. The display device 34 may be or include any kind of display for presenting the reconstructed picture, such as an integrated or external display or monitor. The displays may, for example, comprise liquid crystal displays (LCDs), organic light emitting diodes (OLEDs), plasma displays, projectors, micro-LED displays, liquid crystal on silicon (LCoS), digital light processor (DLP), or any other type of display.

Although FIG. 1A illustrates source device 12 and destination device 14 as separate devices, embodiments of the devices may also include both or both functionality, source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such embodiments, source device 12 or related functionality and destination device 14 or related functionality may be implemented using the same hardware and/or software, or using separate hardware and/or software, or any combination thereof. .

As will be apparent to one skilled in the art from the description, the presence and (precise) separation of functionality of various blocks or functionality in source device 12 and/or destination device 14, as shown in FIG. 1A, may vary depending on the actual device and applications.

Encoder 20 (eg, video encoder 20) or decoder 30 (eg, video decoder 30) or both encoder 20 and decoder 30 may be implemented by the processing circuitry shown in FIG. 1B, such as one or more microprocessors, digital signal processors ( DSP), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding, or any combination thereof. Encoder 20 may be implemented through processing circuitry 46 to implement various modules, as discussed with respect to encoder 20 of FIG. 2 and/or any other encoder system or subsystem described herein. Decoder 30 may be implemented through processing circuit 46 to implement various modules, as discussed with respect to decoder 30 of FIG. 3 and/or any other decoder system or subsystem described herein. The processing circuit may be configured to perform various operations, which will be described below. As shown in FIG. 5, if the techniques are partially implemented in software, the apparatus may store instructions for the software on a suitable non-transitory computer readable storage medium and may execute the instructions in hardware using one or more processors to execute the techniques of this disclosure. Any one of video encoder 20 and video decoder 30 can be integrated as part of a combined encoder/decoder (CODEC) in one device, for example, as shown in Fig. 1B.

Source device 12 and destination device 14 may comprise any of a wide range of devices, including any type of portable or desktop devices, such as laptop or laptop computers, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes , televisions, display devices, digital media players, game consoles, video streaming devices (eg, content service servers or content delivery servers), broadcast receiver device, broadcast transmitter device, etc., and may contain any kind of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

In some cases, the video encoding system 10 illustrated in FIG. 1A is merely an example, and the methods of the present application may be applied to video encoding settings (such as video encoding or video decoding) that do not necessarily involve transmitting any data between encoding and decoding devices. . In other examples, data is retrieved from local memory, streamed over a network, or the like. A video encoding apparatus may encode and store data into memory, and/or a video decoding apparatus may retrieve and decode data from memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data into memory and/or retrieve and decode data from memory.

For ease of description, embodiments of the present invention are described herein, for example, with reference to High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC) reference software, a next-generation video encoding standard being developed by the Joint Collaboration on Video Coding (JCT-VC) Expert Group. ITU-T video encoding (VCEG) and ISO/IEC Moving Picture Expert Group (MPEG). One of ordinary skill in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

Encoder and encoding method

FIG. 2 shows a schematic block diagram of an exemplary video encoder 20 that is configured to implement the methods of the present application. In the example of FIG. 2, video encoder 20 includes an input 201 (or input interface 201), a remainder calculation block 204, a transform processing block 206, a quantization block 208, an inverse quantization block 210 and an inverse transform processing block 212, a reconstruction block 214, a contour block 220 filter, decoded picture buffer (DPB) 230, mode selection unit 260, entropy encoding unit 270, and output 272 (or output interface 272). The mode selection block 260 may include an inter prediction block 244, an intra prediction block 254, and a separating block 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG. 2 may also be called a hybrid video encoder or a video encoder according to a hybrid video codec.

The remainder calculation block 204, transform processing block 206, quantization block 208, mode selection block 260 may relate to the formation of the forward signal path of the encoder 20, while the inverse quantization block 210, inverse transform processing block 212, restoration block 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter prediction block 244, and intra prediction block 254 may relate to generating a return signal path of the video encoder 20, wherein the return signal path of the video encoder 20 corresponds to the signal path of the decoder (see video decoder 30 in FIG. 3) . The inverse quantization unit 210, the inverse transform processing unit 212, the restoration unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 244, and the intra prediction unit 254 are also relevant to forming the “on-chip decoder” of the video encoder 20.

Pictures and picture division (pictures and blocks)

Encoder 20 may be configured to receive, for example, through input 201, a picture 17 (or picture data 17), such as a picture of a sequence of pictures forming a video or video sequence. The received picture or picture data may also be a preprocessed picture 19 (or preprocessed picture data 19). For simplicity, the following description refers to picture 17. Picture 17 may also be referred to as the current picture or the picture that is to be encoded (particularly in video encoding, to distinguish the current picture from other pictures, such as previously encoded and/or decoded pictures of the same video sequence , that is, a video sequence that also contains the current picture).

A (digital) picture is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. A selection in an array may also be referred to as a pixel (short form of picture element) or pel. The number of samples in the horizontal and vertical direction (or axis) of the array or picture determines the size and/or resolution of the picture. Three color components are usually used to represent color, i.e. the picture can be presented or include three arrays of samples. In RBG format, or color space, a picture contains a corresponding array of red, green, or blue samples. However, in video encoding, each pixel is typically represented in a luminance-chrominance format or color space, such as YCbCr, which contains a luminance component denoted Y (sometimes L is also used instead) and two chrominance components denoted Cb and Cr. The luminance (or short chroma) component Y represents the brightness or intensity of the gray level (eg, as in a grayscale picture), while the two chrominance (or short chrominance) components Cb and Cr represent the chrominance or color information components. Accordingly, a picture in YCbCr format contains an array of brightness samples with values (Y) of brightness samples and two arrays of chrominance samples with values (Cb and Cr) of chromaticity. Pictures in RGB format can be converted or converted to YCbCr format and vice versa, the process is also known as color conversion or conversion. If the picture is monochrome, it can only contain an array of brightness samples. Accordingly, a picture can be, for example, an array of luminance samples in a monochrome format or an array of luminance samples and two corresponding arrays of chrominance samples in 4:2:0, 4:2:2 and 4:4:4 color formats.

Embodiments of video encoder 20 may include a picture splitter (not shown in FIG. 2) configured to divide picture 17 into multiple (usually non-overlapping) picture blocks 203. These blocks may also be called root blocks, macroblocks (H.264/AVC) or coding tree blocks (CTB) or coding tree units (CTU) (H.265/HEVC and VVC). The picture division block may be configured to use the same block size for all pictures in a video sequence and a corresponding grid defining the block size, or to vary the block size between pictures or subsets or groups of pictures and divide each picture into appropriate blocks.

In additional embodiments, the video encoder may be configured to receive block 203 of picture 17 directly, such as one, more, or all of the blocks making up picture 17. Picture block 203 may also be referred to as the current picture block or picture block to be encoded.

Like picture 17, picture block 203 is again, or can be viewed as, a two-dimensional array or matrix of samples with intensity values (sample values), albeit smaller in size than picture 17. In other words, block 203 may contain, for example, a single array of samples (e.g. , a brightness array in the case of a monochrome picture 17 or a brightness or chrominance array in the case of a color picture) or three arrays of samples (for example, brightness and two chrominance arrays in the case of color picture 17) or any other number and/or type of arrays depending on the color used format. The number of samples in the horizontal and vertical direction (or axis) of block 203 determines the size of block 203. Accordingly, the block may be, for example, an MxN array of discrete arrays (M-column by N-row) or an MxN array of transform coefficients.

Embodiments of video encoder 20, as shown in FIG. 2, may be configured to encode picture 17 block by block, for example, encoding and prediction are performed on a block-by-block basis 203.

Calculating the remainder

Residual calculation block 204 may be configured to calculate residual block 205 (also referred to as residual 205) based on picture block 203 and prediction block 265 (more details about prediction block 265 are provided later), for example, by subtracting the value of the samples of prediction block 265 from the values of the samples. picture block 203, sample by sample (pixel by pixel), to obtain a residual block 205 in the sample region.

Conversion

Transform processing unit 206 may be configured to apply a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as residual transform coefficients and represent the residual block 205 in the transform domain.

Transform processing block 206 may be configured to apply DCT/DST integer approximations, such as the transforms defined for H.265/HEVC. Compared to the orthogonal DCT transform, such integer approximations are usually scaled by some factor. To preserve the norm of the residual block that is processed by the forward and inverse transforms, additional scaling factors are applied as part of the transform process. Scaling factors are usually chosen based on some constraints, for example, scaling factors are powers of two for shift operations, bit depth of transform factors, tradeoff between accuracy and implementation cost, etc. Particular scaling factors, for example, are indicated for the inverse transform, for example, by the inverse transform processing unit 212 (and the corresponding inverse transform, for example, by the inverse transform processing unit 312 in the video decoder 30) and corresponding scaling factors for the forward transform, for example, by the transform processing unit 206 in the encoder 20 can be determined accordingly.

Embodiments of video encoder 20 (respectively transform processing block 206) may be configured to output transform parameters, such as the type of transform or transforms, for example, directly or encoded or compressed by entropy encoding block 270, so that, for example, video decoder 30 can receive and use conversion parameters for decoding.

Quantization

Quantization unit 208 may be configured to quantize transform coefficients 207 to obtain quantized coefficients 209, for example, by applying scalar quantization or vector quantization. Quantized coefficients 209 may also be referred to as quantized transform coefficients 209 or quantized residual coefficients 209.

The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform factor may be rounded to an m-bit transform factor during quantization, where n is greater than m. The degree of quantization can be changed by adjusting the quantization parameter (QP). For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be specified by the quantization parameter (QP). The quantization parameter may, for example, be an index to a predefined set of applicable quantization step sizes. For example, small quantization parameters may correspond to fine quantization (small quantization step sizes), and large quantization parameters may correspond to coarse quantization (large quantization step sizes), or vice versa. Quantization may include dividing by the quantization step size and corresponding and/or inverse dequantization, for example, by inverse quantization block 210 may include multiplying by the quantization step size. Embodiments according to some standards, such as HEVC, may be configured to use a quantization parameter to determine the quantization step size. Typically, the quantization step size can be calculated based on the quantization parameter using a fixed point approximation of the equation involving division. Additional scaling factors can be introduced for quantization and dequantization to recover the residual block norm that may have been changed due to the scaling used in the fixed point approximation of the above equation for the quantization step size and the quantization parameter. In one exemplary implementation, inverse transform scaling and dequantization may be combined. Alternatively, customized quantization tables can be used and the signal passed from the encoder to the decoder, for example in a bitstream. Quantization is a lossy operation in which the loss increases with the size of the quantization steps.

Embodiments of video encoder 20 (respectively quantization block 208) may be configured to output quantization parameters (QPs), for example, directly or encoded through entropy encoding block 270, so that, for example, video decoder 30 can receive and apply quantization parameters for decoding.

Inverse quantization

The inverse quantization block 210 is configured to apply the inverse quantization of the quantization block 208 to the quantized coefficients to obtain the dequantized coefficients 211, for example, by applying the inverse quantization scheme applied by the quantization block 208 based on or using the same quantization step size as the quantization block 208. The dequantized coefficients 211 may also be referred to as the dequantized residual coefficients 211 and correspond, although they are typically not identical to the transform coefficients due to quantization loss, to the transform coefficients 207 .

Reverse conversion

The inverse transform processing unit 212 is configured to apply an inverse transform to the transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) or other inverse transforms, to obtain the reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the sampling area. The reconstructed residual block 213 may also be referred to as a transform block 213.

Recovery

A reconstruction block 214 (e.g., an adder or adder 214) is configured to add a transform block 213 (i.e., a reconstructed residual block 213) to a prediction block 265 to obtain a reconstructed block 215 in the sample region, for example, by adding, sample by sample, the sample values of the reconstructed residual block 213 and the sample values of the prediction block 265.

Filtration

Loop filter block 220 (or “loop filter” 220 for short) is configured to filter the reconstructed block 215 to obtain a filtered block 221, or more generally to filter the reconstructed samples to obtain filtered samples. The edge filter block, for example, is configured to smooth pixel transitions or otherwise improve video quality . The edge filter block 220 may include one or more edge filters, such as a deblocking filter, an adaptive offset filter (SAO), or one or more other filters, such as a two-way filter, an adaptive edge filter (ALF), sharpening filters, smoothing filters, or joint filters, or any combination of them. Although the loop filter block 220 is shown in FIG. 2 as a loop filter, in other configurations the loop filter block 220 may be implemented as a post-loop filter. Filtered block 221 may also be referred to as filtered reconstructed block 221.

Embodiments of video encoder 20 (respectively, loop filter block 220) may be configured to output loop filter parameters (such as adaptive sample offset information), for example, directly or encoded through entropy encoding block 270, so that, for example, decoder 30 can receive and apply the same loop filter parameters or corresponding loop filters for decoding.

Buffer of decoded pictures

Decoded picture buffer (DPB) 230 may be a memory that stores reference pictures, or generally reference picture data, for encoding video data by video encoder 20. DPB 230 may be formed by any of a variety of storage devices, such as dynamic random access memory (DRAM). DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer (DPB) 230 may be configured to store one or more filtered blocks 221. The decoded picture buffer 230 may be further configured to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221, of the same current picture. or different pictures, for example, previously reconstructed pictures, and can provide fully previously restored, i.e. decoded, pictures (and corresponding reference blocks and samples) and/or a partially reconstructed current picture (and corresponding reference blocks and samples), for example, for an external predictions. The decoded picture buffer (DPB) 230 may also be configured to store one or more unfiltered reconstructed blocks 215 or, in general, unfiltered reconstructed samples, for example, if the reconstructed block 215 is not filtered by a loop filter block 220 or any other additional processed version of the reconstructed blocks, or samples.

Mode selection (separation and prediction)

The mode selection unit 260 includes a separating unit 262, an inter prediction unit 244, and an intra prediction unit 254, and is configured to receive or obtain original picture data, for example, the original block 203 (the current block 203 of the current picture 17), and reconstructed picture data, for example, filtered and/or unfiltered reconstructed samples or blocks of the same (current) picture and/or from one or more previously decoded pictures, for example, from decoded picture buffer 230 or other buffers (eg, a line buffer, not shown). The reconstructed picture data is used as reference picture data for prediction, for example, inter-prediction or intra-prediction to obtain a prediction unit 265 or a predictor 265.

The mode selection block 260 may be configured to determine or select a split for the current block prediction mode (including no split) and prediction mode (eg, intra or inter prediction mode) and generate a corresponding prediction block 265 that is used to calculate the residual block 205 and reconstruct restored block 215.

Embodiments of the mode selector 260 may be configured to select a partition and prediction mode (e.g., from those supported or available for the mode selector 260) that provide the best match, or in other words, the minimum remainder (the minimum remainder means the best compression for transmission or storage), or minimum signaling overhead (minimum signaling overhead means better compression for transmission or storage), or which accounts for or balances both. The mode selector 260 may be configured to determine a division and prediction mode based on rate/distortion optimization (RDO), that is, selecting a prediction mode that provides minimum distortion at a certain rate. Terms such as “best”, “minimum”, “optimal”, etc. in this context do not necessarily refer to an overarching "best", "minimum", "optimal", etc., but may also refer to the fulfillment of a selection or termination criterion, such as when a value exceeds or falls below a threshold, or other restrictions, potentially leading to "suboptimal selection" but reducing complexity and processing time.

In other words, partitioner 262 may be configured to partition block 203 into smaller sections or sub-blocks (which form blocks again), for example, iteratively using quadratic partitioning (QT), binary partitioning (BT), or ternary tree partitioning (TT). or any combination thereof, and to perform, for example, prediction for each of the sections or subblocks of the block, wherein the mode selection comprises selecting a tree structure of the partitioned block 203 and the prediction modes are applied to each of the sections or subblocks of the block.

Next, the division (e.g., by division block 260) and prediction processing (by inter prediction block 244 and intra prediction block 254) performed by the exemplary video encoder 20 will be explained in more detail.

Separation

Divider 262 may divide (or split) the current block 203 into smaller pieces, such as smaller blocks of square or rectangular size. These smaller blocks (which may also be called subblocks) can be further divided into even smaller sections. This is also called tree partitioning or hierarchical tree partitioning, in which the root block, for example at level 0 of the root tree (hierarchy level 0, depth 0), can be recursively partitioned, for example divided into two or more blocks of the next lower tree level. for example, nodes at tree level 1 (hierarchy level 1, depth 1), where these blocks can be again divided into two or more blocks of the next lower level, for example tree level 2 (hierarchy level 2, depth 2), etc. d. until the split is completed, for example because a completion criterion is met, such as the maximum tree depth or minimum block size being reached. Blocks that are not further divided are also called leaf blocks or leaf nodes of a tree. A tree using two partitions is called a binary tree (BT), a tree using three partitions is called a ternary tree (TT), and a tree using four partitions is called a quadtree (QT).

As mentioned earlier, the term "block" as used herein can be a part, particularly a square or rectangular part of a picture. With reference to, for example, HEVC and VVC, a block may be or correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU) and a transform unit (TU) and/or corresponding blocks, for example a coding tree block (CTB), coding block (CB), transform block (TB) or prediction block (PB).

For example, a coding tree unit (CTU) may be or contain a CTB of luma samples, two corresponding CTBs of chroma samples of a picture that has three arrays of samples, or a CTB of samples of a monochrome picture or a picture that is encoded using three separate color planes and the syntactic structures used for encoding samples. Accordingly, a code tree block (CTB) may be an N×N block of samples for some value of N, such that dividing a component by the CTB is a division. A coding unit (CU) may be or contain a luma sample encoding unit, two corresponding chrominance sample encoding units of a picture that has three sample arrays, or a monochrome sample encoding unit or a picture that is encoded using three separate color planes and syntactic structures used for encoding samples. Accordingly, a code block (CB) may be an M×N block of samples for some values of M and N, so that dividing the CTB into code blocks is a division.

In embodiments, for example, according to HEVC, a coding tree unit (CTU) can be partitioned into CUs using a quadtree structure designated as a coding tree. The decision of whether to encode a picture region using extra-picture (temporal) or intra-picture (spatial) prediction is made at the CU level. Each CU can be further partitioned into one, two or four PUs according to the PU partition type. Within a single PU, the same prediction process is applied, and the relevant information is passed to the PU-based decoder. After obtaining the residual block by applying a prediction process based on the partition type on the PU, the CU can be divided into transformation units (TUs) according to another quadtree structure similar to the encoding tree for the CU.

In embodiments, for example, according to the latest video encoding standard currently under development, called Versatile Video Coding (VVC), quadrant and binary tree (QTBT) partitioning is used to divide the encoding block. In a block structure, a QTBT CU can be either square or rectangular in shape. For example, a coding tree unit (CTU) is first divided into a quadtree structure. The leaf nodes of a quadtree are further separated by a binary tree or ternary (or ternary) tree structure. The leaf nodes of a partitioning tree are called coding units (CUs), and this partitioning is used to process prediction and transformation without any further partitioning. This means that CU, PU and TU have the same block size in the QTBT coding block structure. In parallel with the block structure of QTBT, the use of multiple partitioning, such as triple tree partitioning, has been proposed.

In one example, the mode selector 260 of the video encoder 20 may be configured to perform any combination of the division methods described herein.

As described above, video encoder 20 is configured to determine or select the best or optimal prediction mode from a set of (predetermined) prediction modes. The set of prediction modes may comprise, for example, intra prediction modes and/or inter prediction modes.

Internal prediction

The set of intra prediction modes may contain 35 different intra prediction modes, for example, non-directional modes, such as DC (or average) mode and planar mode, or directional modes, for example, as defined in HEVC, or may contain 67 different intra prediction modes, for example , non-directional modes such as DC (or medium) mode and planar mode, or directional modes such as those defined for VVC.

The intra prediction block 254 is configured to use reconstructed samples of neighboring blocks of the same current picture to generate an intra prediction block 265 according to an intra prediction mode of a set of intra prediction modes.

The intra prediction block 254 (or more generally the mode selection block 260) is further configured to output intra prediction parameters (or more generally information indicating the selected intra prediction mode for the block) to the entropy encoding block 270 in the form of syntax elements 266 for including the encoded picture in the data 21 so that, for example, the video decoder 30 can receive and use the prediction parameters for decoding.

External prediction

The set (or possible) inter prediction modes depend on the available reference pictures (i.e., previous at least partially decoded pictures, e.g., stored in DBP 230) and other inter prediction parameters, e.g., whether the entire reference picture is or only a portion, eg, a search window area around the current block area, of the reference picture is used to search for the most suitable reference block and/or, eg, whether pixel interpolation, such as half/half-pixel and/or quarter-pixel interpolation, is applied or not.

In addition to the above prediction modes, a skip mode and/or a direct mode may be used.

The inter-prediction block 244 may include a motion estimation (ME) block and a motion compensation (MC) block (both not shown in FIG. 2). The motion estimator may be configured to receive or obtain a picture block 203 (a current picture block 203 for a current picture 17) and a decoded picture 231, or at least one or a plurality of previously reconstructed blocks, such as reconstructed blocks of one or a plurality of others. distinct previously decoded pictures 231 for motion estimation. For example, the video sequence may contain the current picture and previously decoded pictures 231, or, in other words, the current picture and the previously decoded picture 231 may be part of or form a sequence of pictures making up the video sequence.

The encoder 20 may, for example, be configured to select a reference block from a plurality of reference blocks of the same or different pictures from a plurality of other pictures and provide a reference picture (or reference picture index) and/or an offset (spatial offset) between the position (x, y coordinates ) of the reference block and the position of the current block as inter-prediction parameters to the motion estimation block. This displacement is also called motion vector (MV).

The motion compensation unit is configured to receive, for example, receiving an inter prediction parameter and performing inter prediction based on or using the inter prediction parameter to obtain an inter prediction unit 265. The motion compensation performed by the motion compensation block may include obtaining or generating a prediction block based on a motion vector/block determined by motion estimation, possibly performing interpolations with sub-pixel precision. Interpolation filtering can generate additional pixel samples from known pixel samples, thereby potentially increasing the number of candidate prediction blocks that can be used to encode a picture block. After receiving the motion vector for the PU of the current picture block, the motion compensation unit may determine the location of the prediction block pointed to by the motion vector in one of the reference picture lists.

The motion compensation module may also generate syntax elements associated with video blocks and slices for use by video decoder 30 when decoding the picture blocks of the video slice.

Entropy encoding

Entropy encoding block 270 is configured to apply, for example, an entropy encoding algorithm or scheme (e.g., variable length encoding (VLC) scheme, context adaptive VLC (CAVLC), arithmetic encoding scheme, binarization, context adaptive binary arithmetic encoding (CABAC) , based on the syntax of context-adaptive binary arithmetic coding (SBAC), probability interval entropy encoding (PIPE), or other entropy encoding method or technique) or bypass (uncompressed) quantized coefficients 209, inter prediction parameters, intra prediction parameters, contour parameters filter and/or other syntax elements to obtain encoded picture data 21, which may be output through output 272, for example, in the form of an encoded bitstream 21, so that, for example, video decoder 30 can receive and use parameters for decoding. The encoded bitstream 21 may be transmitted to video decoder 30 or stored in memory for later transmission or retrieval by video decoder 30.

Other structural variations of video encoder 20 may be used to encode the video stream. For example, non-transform encoder 20 may quantize the residual signal directly without transform processing unit 206. In another implementation, encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

Decoder and decoding method

FIG. 3 shows an example of a video decoder 30 that is configured to implement the methods of the present application. Video decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bitstream 21), e.g., encoded by encoder 20, to obtain a decoded picture 331. The encoded picture data or bitstream contains information for decoding the encoded picture data, e.g., data that represents the picture blocks of the encoded video slice and associated syntax elements.

In the example of FIG. 3, decoder 30 includes an entropy decoding block 304, an inverse quantization block 310, an inverse transform processing block 312, a reconstruction block 314 (e.g., adder 314), a loop filter 320, a decoded picture buffer (DBP) 330, an external block 344. prediction and intra prediction block 354. The inter prediction unit 344 may be or include a motion compensation unit. Video decoder 30 may, in some examples, perform a decoding step that is generally the reverse of the encoding step described with respect to video encoder 100 in FIG. 2.

As described with respect to the encoder 20, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-prediction unit 344, and the intra-prediction unit 354 also relate to forming the “on-chip decoder”. » video encoder 20. Accordingly, the inverse quantization block 310 may be identical in function to the inverse quantization block 110, the inverse transform processing block 312 may be identical in function to the inverse transform processing block 212, the restoration block 314 may be identical in function to the reconstruction block 214, loop filter 320 may be identical in function to loop filter 220, and decoded picture buffer 330 may be identical in function to decoded picture buffer 230. Therefore, the explanations provided for the corresponding blocks and functions of the video encoder 20 apply correspondingly to the corresponding blocks and functions of the video decoder 30.

Entropy Decoding

An entropy decoding block 304 is configured to parse the bit stream 21 (or generally the encoded picture data 21) and perform, for example, entropy decoding on the encoded picture data 21 to obtain, for example, quantized coefficients 309 and/or decoded encoding parameters ( not shown in FIG. 3), for example, any or all of the inter prediction parameters (e.g., reference picture index and motion vector), intra prediction parameter (e.g., intra prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and/or other syntax elements. The entropy decoding block 304 may be configured to employ decoding algorithms or schemes corresponding to the encoding schemes as described with respect to the entropy encoding module 270 of the encoder 20. The entropy decoding block 304 may be further configured to provide inter prediction parameters, an intra prediction parameter, and/or other syntax elements to mode selection block 360 and other parameters to other blocks of decoder 30. Video decoder 30 may receive syntax elements at the video slice level and/or video block level.

Inverse quantization

An inverse quantization block 310 may be configured to receive quantization parameters (QPs) (or, more generally, information related to inverse quantization) and quantized coefficients from the encoded picture data 21 (e.g., by parsing and/or decoding, e.g., by block 304 entropy decoding) and apply, based on the quantization parameters, inverse quantization to the decoded quantized coefficients 309 to obtain dequantized coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include using a quantization parameter determined by video encoder 20 for each video block in the video slice to determine the degree of quantization and, likewise, the degree of inverse quantization to be applied.

Reverse conversion

The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain reconstructed residual blocks 213 in the sample region. The reconstructed residual blocks 213 may also be referred to as transform blocks 313. The conversion may be an inverse transform, such as an inverse DCT, an inverse DST, an integer integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may be further configured to receive transform parameters or corresponding information from the encoded picture data 21 (e.g., by parsing and/or decoding, e.g., by entropy decoding unit 304) to determine the transform to be applied to the dequantized coefficients 311.

Recovery

The reconstruction block 314 (e.g., adder or adder 314) may be configured to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the sample region, for example, by adding the sample values of the reconstructed residual block 313 and the sample values of the prediction block 365.

Filtration

A loop filter block 320 (either in the encoding loop or after the encoding loop) is configured to filter the reconstructed block 315 to produce a filtered block 321, for example, to smooth pixel transitions or otherwise improve video quality. The edge filter block 320 may include one or more edge filters, such as a de-blocking filter, an adaptive offset filter (SAO), or one or more other filters, such as a two-way filter, an adaptive edge filter (ALF), sharpening filters, anti-aliasing filters or collaborative filters, or any combination thereof. Although loop filter block 320 is shown in FIG. 3 as a loop filter, in other configurations, loop filter block 320 may be implemented as a post-loop filter.

Buffer of decoded pictures

The decoded video picture blocks 321 are then stored in a decoded picture buffer 330, which stores the decoded pictures 331 as reference pictures for subsequent motion compensation for other pictures and/or for output or display.

The decoder 30 is configured to output the decoded picture 311, for example, through an output 312 for presentation or viewing to the user.

Prediction

The inter-prediction block 344 may be identical to the inter-prediction block 244 (in particular, the motion compensation module), and the intra-prediction block 354 may be identical to the inter-prediction block 254 in function and performs division or partitioning decisions and prediction based on the division parameters and/or or prediction or corresponding information received from the encoded picture data 21 (eg, by parsing and/or decoding, eg, by entropy decoding unit 304). The mode selector 360 may be configured to perform prediction (intra or inter prediction) for each block based on the reconstructed pictures, blocks, or corresponding samples (filtered or unfiltered) to obtain a prediction block 365.

When a video slice is encoded as an intra-encoded (I) slice, the intra prediction block 354 of the mode selection block 360 is configured to generate a prediction block 365 for a picture block of the current video slice based on the intra prediction signal mode and data from previously decoded current picture blocks. When a video picture is encoded as an externally encoded (i.e., B or P) slice, an external prediction block 344 (e.g., a motion compensation block) of the mode selection block 360 is configured to create prediction blocks 365 for the video block of the current video slice based on motion vectors and other syntax elements. received from the entropy decoding unit 304. For inter-prediction, prediction blocks may be created from one of the reference pictures within one of the reference picture lists. Video decoder 30 may construct reference picture lists, List 0 and List 1, using default construction techniques based on reference pictures stored in DPB 330.

The mode selector 360 is configured to determine prediction information for a video block of the current video slice by parsing motion vectors and other syntax elements, and uses the prediction information to create prediction blocks for the current video block to be decoded. For example, mode selector 360 uses some of the received syntax elements to determine the prediction mode (eg, intra or inter prediction) used to encode video slice video blocks, the type of inter prediction slice (eg, B-slice, P-slice, or GPB slice ), design information for one or more reference picture lists for a slice, motion vectors for each outer-coded video block of the slice, inter-prediction status for each outer-coded video block of the slice, and other information for decoding video blocks in the current video slice.

Other embodiments of the video decoder 30 may be used to decode the encoded picture data 21. For example, decoder 30 may produce an output video stream without loop filtering block 320. For example, the non-transform decoder 30 may perform inverse quantization of the residual signal directly without detransforming processing unit 312 for some blocks or frames. In another implementation, video decoder 30 may have an inverse quantization block 310 and an inverse transform processing block 312 combined into one block.

It should be understood that in encoder 20 and decoder 30, the processing result of some current step may be further processed and then output to the next step. For example, after interpolation filtering, motion vector acquisition, or contour filtering, an additional operation such as Clip or offset, can be performed on the result of interpolation filtering, motion vector, or contour filtering processing.

It should be noted that additional operations may be applied to the derived motion vectors of the current block (including, but not limited to, affine mode checkpoint motion vectors, affine mode, planar, ATMVP mode subblock motion vectors, temporary motion vectors, and the like). For example, the value of a motion vector is limited to a predefined range according to its representing bit. If the representing bit of the motion vector is bitDepth, then the range is -2^(bitDepth-1) ~ 2^(bitDepth-1)-1, where "^" means exponentiation. For example, if bitDepth is 16, the range is -32768~32767; if bitDepth is 18, the range is -131072~131071. For example, the value of a derived motion vector (eg, MVs of four 4x4 subblocks in one 8x8 block) is limited such that the maximum difference between the integer parts of the four MVs of a 4x4 subblock is no more than N pixels, eg, no more than 1 pixel. Here are two ways to constrain the motion vector according to bitDepth.

Method 1: Removing MSB (Most Significant Bit) Overflow via Streaming Operations

where mvx represents the horizontal component of the motion vector of the image block or sub-block, mvy represents the vertical component of the motion vector of the image block or sub-block, and ux and uy indicate an intermediate value.

For example, if the value of mvx is -32769, after applying formulas (1) and (2), the resulting value is 32767. In a computer system, decimal numbers are stored as two's complement. The two's complement for -32769 is 1,0111,1111,1111,1111 (17 bits), then the MSB is discarded, so the resulting two's complement is 0111,1111,1111,1111 (decimal is 32767), which is the same as the output from the application of formulas (1) and (2).

Operations can be applied during the summation of mvp and mvd, as shown in formulas (5) to (8).

Method 2: Remove MSB overflow by truncating the value

where vx represents the horizontal component of the motion vector of the image block or sub-block, vy represents the vertical component of the motion vector of the image block or sub-block; x, y and z respectively correspond to the three input values of the MV clipping process, and the definition of the Clip3 function is as follows:

FIG. 4 is a schematic diagram of a video encoding apparatus 400 according to an embodiment of the present disclosure. Video encoding apparatus 400 is suitable for implementing the disclosed embodiments that are described herein. In an embodiment, video encoding device 400 may be a decoder, such as video decoder 30 of FIG. 1A, or an encoder such as video encoder 20 of FIG. 1A.

The video encoding device 400 includes input ports 410 (or input ports 410) and receiver (Rx) blocks 420 for receiving data; a processor, logical unit, or central processing unit (CPU) 430 for processing data; transmitter (Tx) blocks 440 and output ports 450 (or output ports 450) for data transmission; and a memory 460 for storing data. Video encoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components connected to input ports 410, receiver units 420, transmitter units 440, and output ports 450 to provide optical or optical input or output. electrical signals.

The processor 430 is implemented in hardware and software. Processor 430 may be implemented as one or more CPU chips, cores (eg, a multi-core processor), FPGA, ASIC, and DSP. Processor 430 communicates with input ports 410, receiver units 420, transmitter units 440, output ports 450, and memory 460. Processor 430 includes an encoding module 470. Encoding module 470 implements the disclosed embodiments described above. For example, encoding module 470 implements, processes, prepares, or provides various encoding operations. Therefore, the inclusion of the encoding module 470 provides a significant improvement in the functionality of the video encoding device 400 and allows the video encoding device 400 to be transformed into a different state. Alternatively, encoding module 470 is implemented as instructions stored in memory 460 and executed by processor 430.

Memory 460 may include one or more disks, tape drives, and solid-state drives and may be used as an overflow storage device for storing programs when such programs are selected for execution and for storing instructions and data that are read during execution of the programs. Memory 460 may be, for example, volatile and/or non-volatile and may be read only memory (ROM), random access memory (RAM), ternary memory associative memory (TCAM), and/or static random access memory (SRAM).

FIG. 5 is a simplified block diagram of apparatus 500 that may be used as one or both of source device 12 and sink device 14 of FIG. 1 according to an exemplary embodiment .

The processor 502 in the hardware 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or plurality of devices capable of manipulating or processing information that currently exists or will be developed in the future. Although the disclosed implementations may be practiced with a single processor, as shown, for example, with processor 502, speed and efficiency benefits may be achieved by using more than one processor.

Memory 504 in hardware 500 may be read-only memory (ROM) or random access memory (RAM) in some implementation. Memory 504 may be any other suitable type of storage device. Memory 504 may include code and data 506 accessed by processor 502 using bus 512. Memory 504 may further include an operating system 508 and application programs 510, wherein application programs 510 include at least one program which allows the processor 502 to perform the methods described herein. For example, application programs 510 may include applications 1 through N, which further include a video encoding application that performs the methods described herein.

The apparatus 500 may also include one or more output devices, such as a display 518. The display 518 may be, in one example, a touch-sensitive display that combines a display with a touch-sensitive element capable of sensing touch inputs. . Display 518 may be coupled to processor 502 via bus 512.

Although depicted here as a single bus, bus 512 of apparatus 500 may be composed of multiple buses. In addition, secondary storage 514 may be directly coupled to other components of hardware 500 or may be accessible over a network and may comprise a single plug-in unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, the apparatus 500 can be implemented in a wide variety of configurations.

Loop filter

There are a total of three loop filters in the VTM3. In addition to the deblocking filter and SAO (two loop filters in HEVC), VTM3 uses an adaptive loop filter (ALF). The filter order in VTM3 is deblocking, SAO and ALF filter.

A.L.F.

VTM5 uses an adaptive loop filter (ALF) with block filter adaptation. For the luminance component, one of 25 filters is selected for each 4 × 4 block based on the direction and activity of local gradients.

Filter shape:

JEM uses two filter diamond shapes (as shown in Figure 6) for the luminance component. A 7 × 7 diamond shape is used for the luma component, and a 5 × 5 diamond shape is used for the chrominance component.

Block classification:

For the luminance component, each 4 × 4 block belongs to one of 25 classes. The classification index C is derived based on its directionality D and quantized activity value in the following way :

To calculate And First, gradients in the horizontal, vertical, and two diagonal directions are calculated using the one-dimensional Laplacian:

Where are the indices? And refer to the coordinates of the top left sample in the 4 × 4 block, and R(i, j) indicates the reconstructed sample at the coordinate .

To simplify block classification, a one-dimensional Laplacian calculation with subsampling is used. As shown in Fig. 7(a)-(d), the same subsampled positions are used to calculate the gradient in all directions.

Then the maximum and minimum values the gradients of the horizontal and vertical directions are specified as:

,

The maximum and minimum gradient values of the two diagonal directions are set as:

,

To get the directional value , these values are compared with each other and with two threshold values And

STAGE 1 . If And are true installed in .

STEP 2. If , continue from step 3; otherwise continue from step 4.

STEP 3. If , installed in , otherwise D is set to 1.

STEP 4. If , D is set to , otherwise is set to 3.

The activity A value is calculated as:

A is further quantized to the range 0 to 4 inclusive, and the quantized value is denoted .

For chromaticity components in a picture, the classification method is not applied, i.e. For each chrominance component, one set of ALF coefficients is applied.

Geometric transformations of filter coefficients

Before filtering each 4 × 4 luma block, geometric transformations such as rotation or diagonal and vertical flip are applied to the filter coefficients f(k, l) depending on the gradient values computed for that block. This is equivalent to applying these transformations to the samples in the filter support region. The idea is to make the different blocks that ALF is applied to more similar by aligning their directionality.

Three geometric transformations are introduced, including diagonal, vertical flip and rotation:

Diagonal: fDk,

Vertical flip:

Rotation: fRk,

Where the size of the filter, and 0≤k, l≤K-1 are the coordinates of the coefficients, so that the location 0,0 is in the upper left corner, and the location K-1, K-1 is in the lower right corner. Transformations are applied to the filter coefficients f(k, l) depending on the gradient values computed for that block. The relationship between the transformation and the four gradients in the four directions is summarized in the following table.

Table 1 Display of gradient calculated for one block and transformations Gradient values Conversion g _d2< g _d1 and g _h< g _v No conversion g _d2< g _d1 and g _v< g _h Diagonal g _d1< g _d2 and g _h< g _v Vertical flip g _d1< g _d2 and g _v< g _h Rotation

Filter parameters alarm

In VTM3, the ALF filter parameters are communicated in the slice header. Up to 25 sets of luminance filter coefficients can be transmitted. To reduce bit overhead, filter coefficients of different classifications can be combined.

The filtration process can be controlled at the CTB level. The flag is always signaled to indicate whether ALF is applied to the luminance CTB. For each chroma CTB, a flag may be signaled indicating whether ALF applies to the chroma CTB, depending on the value of alf_chroma_ctb_present_flag.

The filter coefficients are quantized at a rate of 128. To further limit the complexity of the multiplication, bitstream matching is applied, whereby the coefficient value of the central position must be in the range from 0 to 2 ^8, and the coefficient values of the remaining positions must be in the range from -2 ⁷ to 2 ⁷ -1, inclusive.

Filtration process

On the decoder side, when ALF is enabled for CTB, each sample R(i, j) in the CU is filtered, resulting in a sample value , as shown below, where L denotes the filter length, f _{m, n} represents the filter coefficient, and f(k, l) denotes the decoded filter coefficients.

Or filtering can also be expressed as

where the samples I (x+i, y+j) are the input samples, O(x, y) is the filtered output sample (i.e. the result of the filtering), and denotes the filter coefficients, where the samples I (x+i, y+j) are the input samples, O(x, y) are the filtered output samples (i.e. the filtering result), and denotes the filter coefficients. In practice, VTM is implemented using integer arithmetic for fixed-point precision calculations:

where L denotes the length of the filter, and filter coefficients with fixed-point accuracy.

Since VTM5 (ITU JVET-N0242), ALF is performed in a non-linear manner. Equation 21 can be formulated as follows:

Where same filter coefficients as in equation (22) [except which is equal to 1 in equation (23), whereas it is equal to in equation (21).

The filter is further modified by introducing nonlinearity to make ALF more efficient by using a truncation function to reduce the influence of neighboring sample values ((Ix+i, y+j) when they are too different from the filtered value ( ) of the current sample.

In VTM5 the ALF filter is modified as follows:

Where is the truncation function, and k(i, j) are the truncation parameters, which depend on the coefficient (i, j) of the filter. Truncation Options are specified for each ALF filter, one cutoff value is signaled per filter coefficient. This means that up to 12 cutoff values can be passed in the bitstream for the luma filter and up to 6 cutoff values for the chrominance filter.

To limit signaling overhead and encoder complexity, the estimation of truncation values is reduced to a small set of possible values. VTM5 uses only 4 possible fixed values, which are the same for INTER and INTRA tile groups.

Because the variance of local differences is often higher for luma than for chrominance, two different sets of cutoff values are used for the luma and chrominance filters. The truncation values also include the maximum sample value (here 1024 for 10-bit depth) in each set, so truncation can be disabled if it is not needed.

The sets of truncation values used in VTM5 are presented in Table 2. The 4 values were selected by approximately equally dividing in the logarithmic domain the full range of sample values (encoded at 10 bits) for luma and the range from 4 to 1024 for chrominance.

More precisely, the brightness table of truncation values was obtained using the following formula:

where M=2 ¹⁰ and N=4.

Similarly, chromaticity tables of truncation values are obtained using the following formula:

where M=2 ¹⁰ , N=4 and A=4.

INTRA/INTER tile group LUMA (brightness) { 1024, 181, 32, 6 } CHROME (chroma) { 1024, 161, 25, 4 }

Table 2: Authorized truncation values

Other loop filter

There are a total of three in-circuit filters in VVC. In addition to the deblocking filter and SAO (two loop filters in HEVC), an adaptive loop filter (ALF) is used. The ALF consists of a luma ALF, a chrominance ALF, and a cross-component ALF (CC-ALF). The ALF filtering process is designed such that luma ALF, chrominance ALF and CC-ALF can be executed in parallel. The filtering order in VVC is deblocking, SAO and ALF filter. SAO in VVC is the same as in HEVC.

A new process has been added to VVC called luminance matching with chrominance scaling (this process was previously known as adaptive in-loop converter). LMCS modifies sample values before encoding and after reconstruction by redistributing codewords across the entire dynamic range. This new process is executed before the lock is released.

Adaptive Loop Filter

VVC uses an adaptive loop filter (ALF) with block filter adaptation. For the luminance component, one of 25 filters is selected for each 4x4 block based on the direction and activity of local gradients.

Filter shape:

Two filter diamond shapes are used (as shown in Figure 6). A 7x7 diamond shape is used for the luma component, and a 5x5 diamond shape is used for the chrominance components.

Block classification:

For the luminance component, each 4 × 4 block belongs to one of 25 classes. The classification index C is derived based on its directionality D and quantized activity value in the following way:

To calculate And First, gradients in the horizontal, vertical, and two diagonal directions are calculated using the one-dimensional Laplacian:

Where are the indices? And refer to the coordinates of the top left sample in the 4 × 4 block, and R(i, j) indicates the reconstructed sample at the coordinate .

To simplify block classification, a one-dimensional Laplacian calculation with subsampling is used. As shown in Fig. 7(a)-(d), the same subsampled positions are used to calculate the gradient in all directions.

Then the maximum and minimum values the gradients of the horizontal and vertical directions are specified as:

The maximum and minimum values of the gradient of the two diagonal directions are given as:

To display the directional value , these values are compared with each other and with two threshold values And

STAGE 1 . If both And are true installed in .

STEP 2. If , continue from step 3; otherwise continue from step 4.

STEP 3. If , D is set to , otherwise D is set .

STEP 4. If , D is set to , otherwise D is set .

Meaning activity is calculated as:

is further quantized to the range 0 to 4 inclusive, and the quantized value is denoted .

For chromaticity components in a picture, the classification method is not applied, i.e. For each chrominance component, one set of ALF coefficients is applied.

Geometric transformations of filter coefficients and trim values

Before filtering each 4x4 luma block to the filter coefficients f(k, l) and to the corresponding filter cutoff values geometric transformations are applied, such as rotation or diagonal and vertical flip, depending on the gradient values calculated for that block. This is equivalent to applying these transformations to the samples in the filter support region. The idea is to make the different blocks that ALF is applied to more similar by aligning their directionality.

Three geometric transformations are introduced, including diagonal, vertical flip and rotation:

Diagonal:

Vertical flip:

Rotation:

Where is the filter size, and 0≤k, l≤K-1 are the coordinates of the coefficients, so the location (0,0) is in the upper left corner, and the location is located in the lower right corner. Transformations are applied to the filter coefficients f(k, l) and to the values truncation depending on the gradient values calculated for this block. The relationship between the transformation of the four gradients in the four directions is summarized in Table 1 above.

Filter parameters alarm

The ALF filter parameters are signaled in the adaptation parameter set (APS). Up to 25 sets of luma filter coefficients and clipping value indices and up to eight sets of chrominance filter coefficients and clipping value indexes can be signaled in one APS. To reduce bit overhead, filter coefficients of different classifications can be combined for the luminance component. The slice header signals the APS indexes used for the current slice.

The truncation value indices, which are decoded from the APS, allow the truncation values to be determined using a table of truncation values for both luma and chroma components. These truncation values depend on the internal bit depth. More precisely, the truncation values are obtained using the following formula:

with B equal to the internal bit depth, α is a predefined constant value equal to 2.35, and N equal to 4 is the number of valid truncation values in the VVC. AlfClip is then rounded to the nearest value in power of 2 format.

Up to 7 APS indices can be signaled in the slice header to indicate the luma filter sets that are used for the current slice. Additional control of the filtration process can be carried out at the CTB level. The flag is always signaled to indicate whether ALF is applied to the luminance CTB. The brightness CTB can select a filter set from 16 fixed filter sets and filter sets from APS. The filter bank index is signaled to the luminance CTB to indicate which filter bank is being applied. 16 fixed filter sets are predefined and hard-coded in both the encoder and decoder.

For the chroma component, an APS index is reported in the slice header to indicate the chroma filter sets used for the current slice. At the CTB level, the filter index is signaled for each chroma CTB if more than one chroma filter is installed in the APS.

The filter coefficients are quantized at a rate of 128. To limit the complexity of the multiplication, bitstream matching is applied so that the value of the off-center position coefficient must be in the range -2 ⁷ to 2 ⁷ -1 inclusive. ^. The center position coefficient is not signaled in the bit stream and is assumed to be 128.

Filtration process

On the decoder side, when ALF is enabled for CTB, each sample R(i, j) in the CU is filtered, resulting in a sample value as below,

where f(k, l) denotes the decoded filter coefficients, K(x,y) is the truncation function, and denotes the decoded truncation parameters. Variables k and l vary between And , where L denotes the length of the filter. Truncation function , which corresponds to the function The truncation operation introduces nonlinearity to make ALF more efficient by reducing the influence of neighboring sample values that are too different from the current sample value.

The selected truncation values are encoded in the alf_data syntax element using a Golomb encoding scheme corresponding to the truncation value index in Table 1 above. This encoding scheme is similar to the encoding scheme for the filter index. Alf_data may be in adaptation_parameter_set_rbsp(), and adaptation_parameter_set_rbsp() may be referenced by the slice header.

The syntax details are shown in the table below.

The semantics of the newly introduced syntax elements is as follows:

alf_luma_clip equal to 0 indicates that a linear adaptive contour filter is applied to the luma component. alf_luma_clip equal to 1 indicates that a nonlinear adaptive contour filter can be applied to the luma component.

alf_chroma_clip equal to 0 indicates that a linear adaptive contour filter is applied to the chroma components; alf_chroma_clip equal to 1 indicates that a nonlinear adaptive contour filter is applied to the chroma component. If not, alf_chroma_clip is output as 0.

alf_luma_clip_min_eg_order_minus1 plus 1 specifies the minimum exp-Golomb code order for signaling luma clipping indices. The value of alf_luma_clip_min_eg_order_minus1 must be in the range from 0 to 6 inclusive.

alf_luma_clip_eg_order_increase_flag[ i] equal to 1 indicates that the minimum order of the exp-Golomb code for signaling luma truncation indices is increased by 1. alf_luma_clip_eg_order_increase_flag[i] equal to 0 indicates that the minimum order of the exp-Golomb code for signaling luma truncation indices is not increased by 1.

The order of the expGoOrderYClip[i] exp-Golomb code used to decode the alf_luma_clip_idx[sigFiltIdx][j] values is derived as follows:

expGoOrderYClip[i]=alf_luma_clip_min_eg_order_minus1+1+alf_luma_clip_eg_order_increase_flag[i]

alf_luma_clip_idx[ sigFiltIdx][j] specifies the clipping index of the clipping value to use before multiplying by the jth coefficient of the luminance signal filter specified by sigFiltIdx. When alf_luma_clip_idx[sigFiltIdx][j] is missing, it is assumed to be 0 (no clipping).

The order k of the exp-Golomb binarization uek(v) is derived as follows:

golombOrderIdxYClip []={0, 0, 1, 0, 0, 1, 2, 1, 0, 0, 1, 2}

k=expGoOrderYClip [golombOrderIdxYClip [j]]

The filterClips variable [sigFiltIdx] [j] with sigFiltIdx=0..alf_luma_num_filters_signalled_minus1, j=0..11 is initialized as follows: The NumYClipValue variable is set to 4.

For i=0..NumYClipValue - 1

alf_luma_clipping_value[i]=Round (2 ^{(BitDepthY * (NumYClipValue - i)/NumYClipValue)} )

filterClips[sigFiltIdx][j]=alf_luma_clipping_value[alf_luma_clip_idx[sigFiltIdx][j]]

AlfClip Luminosity Filter Clip Values_L With AlfClip elements_L[filtIdx][j], with filtIdx=0..NumAlfFilters - 1 and j=0..11 are output as follows

AlfClip _L [filtIdx][j]=filterClips[alf_luma_coeff_delta_idx[filtIdx]][j]

alf_chroma_clip_min_eg_order_minus1 plus 1 specifies the minimum exp-Golomb code order for signaling chroma clipping indices. The value of alf_chroma_clip_min_eg_order_minus1 must be in the range from 0 to 6 inclusive.

alf_chroma_clip_eg_order_increase_flag[ i], equal to 1, indicates that the minimum exp-Golomb code order for signaling chroma truncation indices is increased by 1. alf_chroma_clip_eg_order_increase_flag[i], equal to 0, indicates that the minimum order of the exp-Golomb code for signaling chroma truncation indices is not signaled increases by 1.

The order of expGoOrderC[i] of the exp-Golomb code used to decode the alf_chroma_clip_idx[j] values is derived as follows:

expGoOrderC[i]=alf_chroma_clip_min_eg_order_minus1+1+alf_chroma_clip_eg_order_increase_flag[i]

alf_chroma_clip_idx[ j] specifies the clipping index of the clipping value to use before multiplying by the jth chroma filter coefficient. When alf_chroma_clip_idx[j] is missing, it is assumed to be 0 (no clipping).

The order k of the exp-Golomb binarization uek(v) is derived as follows:

golombOrderIdxC []={0, 0, 1, 0, 0, 1}

k=expGoOrderC [golombOrderIdxC [j]]

AlfClipC chroma filter clipping values with AlfClip elements_C [j]_,with j=0..5 are output as follows:

The NumCClipValue variable is set to 4.

For i=0..NumCClipValue - 1

alf_chroma_clipping_value[i]=Round (2 ^{(BitDepthC - 8)} * 2 ^{(8 * (NumCClipValue - i - 1)/(NumCClipValue -1))} )

AlfClip _C [j]=alf_chroma_clipping_value[alf_chroma_clip_idx[j]]

ALF syntax specification as per VVC specification

Adaptive Loop Filtering Process

1.1 General provisions

The inputs to this process are arrays of samples of reconstructed pictures before the adaptive contour filter recPictureL, recPictureCb and recPictureCr.

The outputs of this process are modified arrays of samples of reconstructed pictures after the adaptive contour filter alfPictureL, alfPictureCb and alfPictureCr.

The sample values in the modified sample arrays of the reconstructed pictures after the adaptive contour filter alfPictureL, alfPictureCb and alfPictureCr are initially set equal to the sample values in the sample arrays of the reconstructed pictures before the adaptive contour filter recPictureL, recPictureCb and recPictureCr, respectively.

When the value of tile_group_alf_enabled_flag is 1, for each encoding tree unit with luma encoding tree block location (rx, ry), where rx=0..PicWidthInCtbs - 1 and ry=0..PicHeightInCtbs - 1, the following process is applied:

When the value of alf_ctb_flag[0][rx][ry] is equal to 1, the encoding tree block filtering process for luma samples as specified in clause 1.2 is called with recPictureL, alfPictureL and the luma encoding tree block location (xCtb, yCtb) set to ( rx<<CtbLog2SizeY, ry<<CtbLog2SizeY ) as input, and the output is a modified filtered picture alfPictureL.

When the value of alf_ctb_flag[1][rx][ry] is 1, the encoding tree block filtering process for chroma samples as specified in clause 1.1 is called with recPicture set to recPictureCb, alfPicture set to alfPictureCb, and the chroma encoding tree block location ( xCtbC, yCtbC) set to (rx<<(CtbLog2SizeY - 1), ry<<(CtbLog2SizeY - 1)) as input, and the output is a modified filtered picture alfPictureCb.

When the value of alf_ctb_flag[2][rx][ry] is 1, the encoding tree block filtering process for chroma samples as specified in clause 1.4 is called with recPicture set to recPictureCr, alfPicture set to alfPictureCr, and Chroma Tree Block Location ( xCtbC, yCtbC) is set to (rx<<(CtbLog2SizeY - 1), ry<<(CtbLog2SizeY - 1)) as input, and the output is a modified filtered picture alfPictureCr.

1.2 Process of filtering encoding tree blocks for brightness samples

Inputs to this process:

restored array of brightness picture samples recPictureL before the adaptive contour filtering process,

filtered reconstructed array of brightness picture samples alfPictureL, Data

luma location (xCtb, yCtb) indicating the top-left sample of the current luma encoding tree block relative to the top-left sample of the current picture.

The output of this process is a modified filtered array of samples of the reconstructed brightness picture alfPictureL.

The inference process for item 1.3 of the filter index is called with the location (xCtb, yCtb) and the reconstructed array of brightness picture samples recPictureL as input, and with filterIdx [x] [y] and transposeIdx [x] [y] with x, y= 0..CtbSizeY - 1 as outputs.

To output filtered reconstructed luma samples alfPictureL[x][y], each reconstructed luma sample within the current luma encoding tree block recPictureL[x][y] is filtered as follows with x, y=0..CtbSizeY - 1:

The array of luminance filter coefficients f [j] corresponding to the filter specified in filterIdx [x] [y] is output as follows with j=0..12:

f[j]=AlfCoeffL [filterIdx [x] [y]] [j].

The array of luma filter cutoff values c[j] corresponding to the filter specified in filterIdx [x] [y] is output as follows with j=0..11:

c[j]=AlfClip _L [filtIdx[x][y]][j]

The luminance filter coefficients filterCoeff are derived as a function of transposeIdx[x][y] as follows:

If transposeIndex [x] [y]=1,

filterCoeff []={f [9], f [4], f [10], f [8], f [1], f [5], f [11], f [7], f [3], f [0], f [2], f [6], f [12]}

filterClip []={c [9], c [4], c [10], c [8], c [1], c [5], c [11], c [7], c [3], c [0], c [2], c [6]}

Otherwise, if transposeIndex[x][y]=2,

filterCoeff []={f [0], f [3], f [2], f [1], f [8], f [7], f [6], f [5], f [4], f [9], f [10], f [11], f [12]}

filterClip []={c [0], c [3], c [2], c [1], c [8], c [7], c [6], c [5], c [4], c [9], c [10], c [11]}

Otherwise, if transposeIndex[x][y]=3,

filterCoeff []={f [9], f [8], f [10], f [4], f [3], f [7], f [11], f [5], f [1], f [0], f [2], f [6], f [12]}

filterClip []={c [9], c [8], c [10], c [4], c [3], c [7], c [11], c [5], c [1], c [0], c [2], c [6]}

Otherwise

filterCoeff []={f [0], f [1], f [2], f [3], f [4], f [5], f [6], f [7], f [8], f [9], f [10], f [11], f [12]}

filterClip []={c [0], c [1], c [2], c [3], c [4], c [5], c [6], c [7], c [8], c [9], c [10], c [11]}

The locations (hx, vy) for each of the corresponding luminance samples (x, y) within a given recPicture array of luminance samples are output as follows:

hx=Clip3 (0, pic_width_in_luma_samples - 1, xCtb+x)

vy=Clip3 (0, pic_height_in_luma_samples - 1, yCtb+y)

The sum of the variables is displayed as follows:

The sum of the variables is displayed as follows:

The modified filtered sample of the reconstructed brightness picture alfPictureL [xCtb+x] [yCtb+y] is output as follows:

alfPictureL[xCtb+x][yCtb+y]=Clip3(0, (1<<BitDepthY) - 1, sum).

1.3 Output process for ALF transpose and filter index for luminance samples

Inputs to this process:

a luma location (xCtb, yCtb) indicating the top left sample of the current luma encoding tree block relative to the top left sample of the current picture,

restored array of brightness picture samples recPictureL before the adaptive contour filtering process.

The results of this process are:

classification filter index array filterIdx [x] [y] with x, y=0..CtbSizeY - 1,

array of transpose indices transposeIdx [x] [y] with x, y=0..CtbSizeY - 1.

The locations (hx, vy) for each of the corresponding luminance samples (x, y) within a given recPicture array of luminance samples are output as follows:

hx=Clip3 (0, pic_width_in_luma_samples - 1, x)

vy=Clip3 (0, pic_height_in_luma_samples - 1, y)

The classification filter index array filterIdx and the transpose index array transposeIdx are obtained using the following ordered steps:

The variables filterH [x] [y], filterV [x] [y], filterD0 [x] [y] and filterD1 [x] [y] with x, y=- 2..CtbSizeY+1 are output as follows:

If both x and y are even numbers or both x and y are odd numbers, the following applies:

filtH[x][y]=Abs( (recPicture[hxCtb+x, vyCtb+y]<<1) - recPicture[hxCtb+x-1, vyCtb+y] -recPicture[hxCtb+x+1, vyCtb+y ] )

filtV[x][y]=Abs( (recPicture[hxCtb+x, vyCtb+y]<<1) - recPicture[hxCtb+x, vyCtb+y-1] - recPicture[hxCtb+x, vyCtb+y+1 ] )

filtD0[x][y]=Abs( (recPicture[hxCtb+x, vyCtb+y]<<1) - recPicture[hxCtb+x-1, vyCtb+y-1] - recPicture[hxCtb+x+1, vyCtb +y+1] )

filtD1[x][y]=Abs( (recPicture[hxCtb+x, vyCtb+y]<<1) - recPicture[hxCtb+x+1, vyCtb+y-1] - recPicture[hxCtb+x-1, vyCtb +y+1] )

Otherwise, filterH[x][y], filterV[x][y], filterD0[x][y], and filterD1[x][y] are set to 0.

Variables varTempH1[x][y], varTempV1[x][y], varTempD01[x][y], varTempD11[x][y] and varTemp[x][y] with x, y=0..(CtbSizeY - 1)>>2 are output as follows:

sumH[x][y]=ΣiΣj filtH[(x<<2)+i][(y<<2)+j] with i, j=-2..5

sumV[x][y]=ΣiΣj filtV[(x<<2)+i][(y<<2)+j] with i, j=-2..5

sumD0[x][y]=ΣiΣj filtD0[(x<<2)+i][(y<<2)+j] with i, j=-2..5

sumD1[x][y]=ΣiΣj filtD1[(x<<2)+i][(y<<2)+j] with i, j=-2..5

sumOfHV[x][y]=sumH[x][y]+sumV[x][y]

The variables dir1 [x] [y], dir2 [x] [y] and dirS [x] [y] with x, y=0..CtbSizeY - 1 are output as follows:

The variables hv1, hv0 and dirHV are output as follows:

If sumV[x>>2][y>>2] is greater than sumH[x>>2][y>>2], the following applies:

hv1=sumV[x>>2][y>>2]

hv0=sumH[x>>2][y>>2]

dirHV=1

Otherwise the following applies:

hv1=sumH[x>>2][y>>2]

hv0=sumV[x>>2][y>>2]

dirHV=3

The variables d1, d0 and dirD are output as follows:

If sumD0[x>>2][y>>2] is greater than sumD1[x>>2][y>>2], the following applies:

d1=sumD0[x>>2][y>>2]

d0=sumD1[x>>2][y>>2]

dirD=0

Otherwise the following applies:

d1=sumD1[x>>2][y>>2]

d0=sumD0[x>>2][y>>2]

dirD=2

Variables hvd1, hvd0 are output as follows:

hvd1=(d1 * hv0>hv1 * d0) ? d1 : hv1

hvd0=(d1 * hv0>hv1 * d0) ? d0 : hv0

The variables dirS [x] [y], dir1 [x] [y] and dir2 [x] [y] are obtained as follows:

dir1[x][y]=(d1 * hv0>hv1 * d0) ? dirD : dirHV

dir2[x][y]=(d1 * hv0>hv1 * d0) ? dirHV : dirD

dirS[x][y]=(hvd1>2 * hvd0) ? 1 : ((hvd1 * 2>9 * hvd0) ? 2 : 0)

The variable avgVar [x] [y] with x, y=0..CtbSizeY - 1 is output as follows:

varTab[]={ 0, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4 }

avgVar[x][y]=varTab[Clip3(0, 15, (sumOfHV[x>>2][y>>2] * 64)>>(3+BitDepthY))]

The classification filter index array filterIdx [x] [y] and the transpose index array transposeIdx [x] [y] with x=y=0..CtbSizeY - 1 are output as follows:

transposeTable []={0, 1, 0, 2, 2, 3, 1, 3}

transposeIdx[x][y]=transposeTable[dir1[x][y] * 2+(dir2[x][y]>>1)]

filtIdx[x][y]=avgVar[x][y]

When dirS[x][y] is not equal to 0, filterIdx[x][y] is modified as follows:

filtIdx[x][y]+=(((dir1[x][y] & 0x1)<<1)+dirS[x][y]) * 5.

1.4 Process of filtering encoding tree blocks for chrominance samples

Inputs to this process:

the reconstructed array of recPicture chromaticity picture samples before the adaptive contour filtering process,

filtered reconstructed array of alfPicture chromaticity picture samples,

a chroma location (xCtbC, yCtbC) indicating the top-left sample of the current block of the chroma encoding tree relative to the top-left sample of the current picture.

The result of this process is a modified filtered reconstructed array of alfPicture chromaticity picture samples.

The size of the current chroma tree block ctbSizeC is output as follows:

ctbSizeC=CtbSizeY/SubWidthC

To output filtered reconstructed alfPicture [x] [y] chroma samples, each reconstructed chroma sample within the current recPicture [x] [y] chroma encoding tree block is filtered as follows with x, y=0..ctbSizeC - 1:

The location (hx, vy) for each of the corresponding chroma samples (x, y) within a given recPicture array of chroma samples is output as follows:

hx=Clip3 (0, pic_width_in_luma_samples/SubWidthC - 1, xCtbC+x)

vy=Clip3 (0, pic_height_in_luma_samples/SubHeightC - 1, yCtbC+y)

The sum of the variables is displayed as follows:

sum=AlfCoeff _C [0] * (Clip3(-AlfClip _C [0], AlfClip _C [0], recPicture[h _x , v _y+2 ] - curr)+Clip3 (-AlfClip _C [0], AlfClip _C [ 0], recPicture[h _x , v _{y - 2} ] - curr))+
AlfCoeff _C [1] * (Clip3(-AlfClip _C [1], AlfClip _C [1], recPicture[h _x+1 , v _y+1 ] - curr)+Clip3 (-AlfClip _C [1], AlfClip _C [ 1],recPicture[h _{x - 1} , v _{y - 1} ] - curr))+
AlfCoeff _C [2] * (Clip3(-AlfClip _C [2], AlfClip _C [2], recPicture[h _x , v _y+1 ] - curr)+Clip3 (-AlfClip _C [2], AlfClip _C [2] ,recPicture[h _x , v _{y - 1} ] - curr))+AlfCoeff _C [3] * (Clip3(-AlfClip _C [3], AlfClip _C [3], recPicture[h _{x - 1} , v _y+1 ] - curr)+Clip3 (-AlfClip _C [3], AlfClip _C [3],recPicture[h _x+1 , v _{y - 1} ] - curr)+
AlfCoeff _C [4] * (Clip3(-alfClip _C [4], alfClip _C [4], recPicture[h _x+2 , v _y ] - curr)+Clip3 (-AlfClip _C [4], AlfClip _C [4] ,recPicture[h _{x - 2} , v _y ] - curr)+
AlfCoeff _C [5] * (Clip3(-AlfClip _C [5], AlfClip _C [5], recPicture[h _x+1 , v _y ] - curr)+Clip3 (-AlfClip _C [5], AlfClip _C [5] , recPicture[h _{x - 1} , v _y ] - curr))

sum=curr+((sum+64)>>7)

A modified filtered sample of the reconstructed chromaticity picture alfPicture [xCtbC+x] [yCtbC+y] is output as follows:

alfPicture[xCtbC+x][yCtbC+y]=Clip3(0, (1<<BitDepthC) - 1, sum).

As described above and as shown in FIG. 8, the ALF luma and chrominance trim parameters are transmitted using K-th order exponential Golomb codes similar to the ALF filter coefficients.

Using K-order exponential Golomb codes for truncation parameters may be inefficient in terms of encoding efficiency, since the signaled truncation parameter is simply an index into the table of truncation values (see Table 2 above). The index value ranges from 0 to 3.

Therefore, passing index values from 0 to 3 using K-th order exponential Golomb codes in the same way as ALF filter coefficients uses additional syntax elements, alf_luma_clip_min_eg_order_minus1, alf_luma_clip_eg_order_increase_flag [i] to determine the value of K (the order of the exponential Golomb code to be used ) and then the alf_luma_clip_idx syntax element is signaled using a K-th order exponential Golomb code. Therefore, this signaling method is complex and also inefficient in terms of encoding efficiency. Therefore, a simpler way to signal truncation parameters is desirable.

In one embodiment of the proposed solution (Solution 1), as shown in FIG. 9, the truncation parameters are signaled using fixed length codes, and therefore syntax elements alf_luma_clip_min_eg_order_minus1, alf_luma_clip_eg_order_increase_flag [i] are not used. Syntax element alf_luma_clip_idxsignaled using a fixed length code of 2 bits.This method has the advantage that the truncation parameters are signaled in a very simple way, the encoding efficiency is increased since several syntax elements related to K-th order exponential holomb codes are no longer transmitted.

The modified alf_data syntax looks like this:

alf_data() { Descriptor alf_chroma_idc that (v) alf_luma_clip u(1) if(alf_choma_idc) alf_chroma_clip u(1) alf_luma_num_filters_signalled_minus1 tb(v) if(alf_luma_num_filters_signalled_minus1>0) { for(filtIdx=0; filtIdx<NumAlfFilters; filtIdx++) alf_luma_coeff_delta_idx[filtIdx] tb(v) } alf_luma_coeff_delta_flag u(1) if (!alf_luma_coeff_delta_flag && alf_luma_num_filters_signalled_minus1>0) alf_luma_coeff_delta_prediction_flag u(1) alf_luma_min_eg_order_minus1 ue(v) for(i=0; i<3; i++) alf_luma_eg_order_increase_flag[i] u(1) if (alf_luma_coeff_delta_flag) { for(sigFiltIdx=0; sigFiltIdx<=alf_luma_num_filters_signalled_minus1; sigFiltIdx++) alf_luma_coeff_flag[sigFiltIdx] u(1) } for(sigFiltIdx=0; sigFiltIdx<=alf_luma_num_filters_signalled_minus1; sigFiltIdx++) { if (alf_luma_coeff_flag[sigFiltIdx]) { for (j=0; j<12; j++) { alf_luma_coeff_delta_abs[sigFiltIdx][j] uek(v) if(alf_luma_coeff_delta_abs[sigFiltIdx][j]) alf_luma_coeff_delta_sign[sigFiltIdx][j] u(1) } } } if(alf_luma_clip) { for (sigFiltIdx=0; sigFiltIdx<=alf_luma_num_filters_signalled_minus1; sigFiltIdx++) { if (alf_luma_coeff_flag[sigFiltIdx]) { for (j=0; j<12; j++) { if(filterCoefficients[sigFiltIdx][j]) alf_luma_clip_idx[sigFiltIdx][j] u(2) } } } } if (alf_chroma_idc>0) { for(i=0; i<2; i++) for(j=0; j<6; j++) { alf_chroma_coeff_abs[j] uek(v) if(alf_chroma_coeff_abs[j]>0) alf_chroma_coeff_sign[j] u(1) } } if (alf_chroma_idc>0 && alf_chroma_clip) { for(j=0; j<6; j++) { if(alf_chroma_coeff_abs[j]) alf_chroma_clip_idx[j] u(2) } } }

As an alternative solution (Solution 2), a truncated unary encoding may also be used to signal the index of the truncate parameter.

As an alternative solution (Solution 3), if the number of truncation parameters is changed from a fixed value of 4 to another number greater than 4, the fixed length (v) code "v value" is increased accordingly. For example, if the number of truncation parameters is increased from 4 to 5 or 6, then the fixed-length code uses 3 bits to signal the truncation parameter.

As an alternative solution (Solution 4), the ALF filter coefficients are signaled using a fixed length code instead of a K-th order exponential Golomb code.

FIG. 10 is a flow diagram illustrating a method according to the first aspect of the present disclosure. The method comprises the steps of obtaining a bitstream, wherein at least one bit in the bitstream represents a syntax element for the current block, 1001, wherein the syntax element is an adaptive loop filter (ALF) truncation value index indicating a truncation index of the truncation value to use before multiplied by the ALF coefficient; parsing the bit stream to obtain the value of a syntax element for the current block, wherein the syntax element is encoded using a fixed length code, 1002; applying adaptive loop filtering to the current block based on the value of the syntax element for the current block, 1003.

FIG. 11 is a flow diagram illustrating a method according to the second aspect of the present disclosure. According to a second aspect, an encoding method implemented by a decoding device is provided, the method comprising the step of:

obtaining a bitstream, wherein at least one bit in the bitstream represents a syntax element for the current block, 1101, wherein the syntax element is an adaptive loop filter (ALF) clipping value index or an ALF coefficient parameter; parsing the bitstream to obtain a syntax element value for the current block, wherein the syntax element value for the current block is obtained using only at least one bit of the syntax element 1102; applying adaptive contour filtering to the current block based on the value of the syntax element for the current block, 1103.

FIG. 12 is a flow diagram illustrating a method according to a third aspect of the present disclosure. According to a third aspect, an encoding method implemented by an encoding device is provided, a method comprising: determining a syntax element value for a current block, wherein the syntax element specifies a truncation index of the truncation value to be used before multiplying by an adaptive loop filter (ALF) factor, 1201; generating a bit stream based on the value of the syntax element, wherein at least one bit in the bit stream represents the syntax element, wherein the syntax element is encoded using a fixed length code, 1202.

FIG. 13 is a flow chart illustrating a method according to the fourth aspect of the present disclosure. According to a fourth aspect, an encoding method implemented by an encoding device is provided, the method comprising: determining a syntax element value for a current block, wherein the syntax element is an adaptive loop filter (ALF) clipping value index or an ALF filter coefficient parameter, 1301; generating a bitstream based on the value of a syntax element, wherein at least one bit in the bitstream represents a syntax element, wherein at least one bit of the syntax element is obtained using only the value of the syntax element for the current block, 1302.

FIG. 14 is a block diagram illustrating a decoder according to a fifth aspect of the present disclosure. According to a fifth aspect of the present disclosure, a decoder 1400 is provided including processing circuitry 1401 for performing a method according to the first or second aspect or any implementation thereof.

FIG. 15 is a block diagram illustrating an encoder according to a sixth aspect of the present disclosure. According to a sixth aspect of the present disclosure, an encoder 1500 is provided including a processing circuit 1501 for performing the method of the third or fourth aspect or any implementation thereof.

FIG. 16 is a block diagram illustrating a decoder according to a ninth aspect of the present disclosure. According to a ninth aspect of the present disclosure, a decoder 1600 is provided, comprising: one or more processors 1601; and a non-transitory computer-readable medium 1602 coupled to the processors 1601 and storing a program for execution by the processors 1601, wherein the program, when executed by the processors 1601, configures the decoder 1600 to perform the method of the first or second aspect, or any implementation thereof.

FIG. 17 is a block diagram illustrating an encoder according to a tenth aspect of the present disclosure. According to a tenth aspect of the present disclosure, there is provided an encoder 1700 comprising: one or more processors 1701; and a non-transitory computer-readable medium 1702 coupled to the processors 1701 and storing a program for execution by the processors 1701, the program, when executed by the processors 1701, configuring the encoder 1700 to perform the method of the third or fourth aspect, or any implementation thereof.

FIG. 18 is a block diagram illustrating a decoder according to an eleventh aspect of the present disclosure. According to an eleventh aspect of the present disclosure, a decoder 1800 is provided, comprising: an entropy decoding block 1801 (may be an entropy decoding block 304) configured to receive a bitstream 1811, wherein at least one bit in the bitstream 1811 represents a syntax element for the current block, wherein wherein the syntax element is a truncation value index ALF indicating a truncation index of the truncation value to be used before multiplying by an adaptive loop filter (ALF) coefficient; the entropy decoding unit 1801 is further configured to parse the bit stream 1811 to obtain a syntax element value 1812 for the current block, wherein the syntax element is encoded using a fixed length code; and a filtering block 1803 (may be a loop filter 320) configured to apply adaptive loop filtering to the current block based on the syntax element value 1812 for the current block.

FIG. 19 is a block diagram illustrating a decoder according to the twelfth aspect of the present disclosure. According to a twelfth aspect of the present disclosure, a decoder 1900 is provided, comprising: an entropy decoding module 1901 (may be an entropy decoding module 304) configured to receive a bit stream 1911, at least one bit in the bit stream 1911 representing a syntax element for the current block, wherein the element syntax is the ALF truncation value index or the ALF factor parameter; the entropy decoding unit 1801 is further configured to parse the bitstream to obtain a syntax element value 1912 for the current block, wherein the syntax element value for the current block is obtained using only at least one bit of the syntax element; and a filtering block 1903 (may be a loop filter 320) configured to apply adaptive loop filtering to the current block based on the syntax element value 1912 for the current block.

FIG. 20 is a block diagram illustrating an encoder according to the thirteenth aspect of the present disclosure. According to a thirteenth aspect of the present disclosure, an encoder 2000 is provided, comprising: a determination block 2001 (may be a contour filter 220) configured to determine a syntax element value 2012 for the current block, wherein the syntax element specifies a truncation index of the truncation value to be used before multiplying by an adaptive contour factor filter (ALF); an entropy encoding unit 2002 (may be an entropy encoding unit 270) configured to generate a bitstream 2011 based on a syntax element value 2012, wherein at least one bit in the bitstream 2011 represents a syntax element, wherein the syntax element is encoded using a code fixed length.

FIG. 21 is a block diagram illustrating an encoder according to the fourteenth aspect of the present disclosure. According to a fourteenth aspect of the present disclosure, there is provided an encoder 2100 comprising: a determination block 2101 (may be a loop filter 220) configured to determine a syntax element value 2112 for the current block, wherein the syntax element is an ALF trim value index or an ALF filter coefficient parameter; an entropy encoding block 2102 (may be an entropy encoding block 270) configured to generate a bit stream 2111 based on a syntax element value 2112, wherein at least one bit in the bit stream 2111 represents a syntax element, wherein at least one bit of the syntax element is obtained by using only the value of the syntax element for the current block.

The present disclosure provides the following additional embodiments:

Option 1 implementation. An encoding method implemented by a decoding device, containing:

obtaining a bitstream, wherein at least one bit in the bitstream corresponds to a syntax element for a current block (or set of blocks, one block in the set of blocks being the current block);

parsing the bitstream to obtain a syntax element value for the current block, wherein the obtained syntax element value for the current block relates only to at least one bit;

Filters the current block based on the value of the syntax element for the current block.

Option 2 implementation. The method of Embodiment 1, wherein the value of the syntax element is encoded according to a fixed-length code (a fixed-length code means that all possible values of the syntax element are reported using the same number of bits).

Option 3 implementation. The method of Embodiment 1, wherein the value of a syntax element is encoded according to a truncated unary code (truncated unary means that the most frequently occurring value of a given syntax element is signaled using the least number of bits, and the least occurring value of a syntax element is signaled using the largest number bits).

Option 4 implementation. The method as in any one of Embodiments 1 to 3, wherein the syntax element is an adaptive loop filter truncation index parameter.

Option 5 implementation. The method as in any one of Embodiments 1 to 3, wherein the syntax element is an adaptive loop filter coefficient parameter.

Option 6 implementation. The method as in any one of Embodiments 1 to 5, wherein the value of the syntax element is used to determine a filter coefficient, the filter coefficient is used in the filtering process.

Option 7 implementation. The method as in any one of Embodiments 1 to 5, wherein the value of the syntax element is used to determine a truncation range, the truncation range is used in the filtering process (the truncation range is used to limit the number of modifications allowed on a given sample by a neighboring sample).

Embodiment 8. A decoder (30) containing processing circuitry for performing the method according to any one of embodiments 1-7.

Option 9 implementation. A computer program product comprising program code for performing the method according to any one of embodiments 1-7.

Embodiment 10. Decoder containing:

one or more processors; And

a non-volatile computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures a decoder to perform the method according to any of embodiments 1-7.

The following will be an explanation of the applications of the encoding method as well as the decoding method as shown in the above-mentioned embodiments and the system using them.

FIG. 22 is a block diagram showing a content delivery system 3100 for implementing a content distribution service. This content delivery system 3100 includes a capture device 3102, a terminal device 3106, and optionally includes a display 3126. The capture device 3102 communicates with the terminal device 3106 via a communication line 3104. The communication link may include a communication channel 13 described above. Communication link 3104 includes, but is not limited to, WIFI, Ethernet, cable, wireless (3G/4G/5G), USB, or any combination thereof, or the like.

The capture device 3102 generates data and can encode the data using the encoding method shown in the above embodiments. Alternatively, capture device 3102 may distribute the data to a streaming server (not shown in the Figures), and the server encodes the data and transmits the encoded data to terminal device 3106. Capture device 3102 includes, but is not limited to, a camera, smartphone, or tablet. , a computer or laptop, a video conferencing system, a PDA, a vehicle-mounted device, or a combination of any of these, or the like. For example, capture device 3102 may include source device 12 described above. When the data includes video, the video encoder 20 included in the capture device 3102 may actually perform video encoding processing. When the data includes audio (ie, speech), an audio encoder included in the capture device 3102 may actually perform audio encoding processing. For some practical scenarios, capture device 3102 distributes encoded video and audio data by multiplexing them together. For other practical scenarios, such as in a video conferencing system, encoded audio data and encoded video data are not multiplexed. The capture device 3102 distributes the encoded audio data and the encoded video data to the terminal device 3106 separately.

In the content delivery system 3100, the terminal device 310 receives and reproduces encoded data. Terminal device 3106 may be a device capable of receiving and retrieving data, such as a smartphone or tablet 3108, computer or laptop 3110, network video recorder (NVR)/digital video recorder (DVR) 3112, television 3114, set-top box (STB) 3116, video conferencing system 3118 , a video surveillance system 3120, a personal digital assistant (PDA) 3122, a vehicle-mounted device 3124, or a combination thereof, or the like, capable of decoding the above encoded data. For example, terminal device 3106 may include receiver device 14 described above. When the encoded data includes video, the video decoder 30 included in the terminal device is given priority to perform video decoding. When the encoded data includes audio, an audio decoder included in the terminal device is given priority to perform audio decoding processing.

For a terminal device with its own display, such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR)/digital video recorder (DVR) 3112, a television 3114, a personal digital assistant (PDA) 3122, or a vehicle-mounted device 3124, the terminal device can transmit the decoded data to its display. For a non-display terminal device, such as the STB 3116, video conferencing system 3118, or video surveillance system 3120, it communicates with an external display 3126 to receive and display the decoded data.

When each device in this system performs encoding or decoding, a picture encoding device or a picture decoding device may be used, as shown in the above embodiments.

FIG. 23 is a diagram showing the structure of an exemplary terminal device 3106. After the terminal device 3106 receives a stream from the capture device 3102, a protocol processing unit 3202 analyzes the transmission protocol of the stream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hypertext Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG-DASH, Real Time Transport Protocol (RTP), Exchange Protocol real-time messaging (RTMP) or any combination thereof, or the like.

After the protocol processing unit 3202 processes the stream, a stream file is generated. The file is output to a demultiplexing block 3204. A demultiplexer 3204 may separate the multiplexed data into encoded audio data and encoded video data. As described above, in other practical scenarios, such as in a video conferencing system, encoded audio data and encoded video data are not multiplexed. In this situation, the encoded data is transmitted to the video decoder 3206 and audio decoder 3208 without using the demultiplexer 3204.

Through demultiplexing processing, an elementary video stream (ES), audio ES, and optionally subtitles are generated. The video decoder 3206, which includes the video decoder 30 described in the above embodiments, decodes the ES video using the decoding method as shown in the above embodiments to generate a video frame and supplies this data to the synchronous block 3212. The audio decoder 3208 decodes the ES audio to generate audio frame and supplies this data to synchronous block 3212. Alternatively, the video frame may be stored in a buffer (not shown in FIG. 23) before it is supplied to synchronous block 3212. Likewise, the audio frame may be stored in a buffer (not shown in FIG. 23) before it is fed into the 3212 synchronous block.

A synchronization block 3212 synchronizes a video frame and an audio frame and supplies video/audio to a video/audio display 3214. For example, a synchronization block 3212 synchronizes the presentation of video and audio information. Information may be encoded in syntax using timestamps relating to the presentation of the encoded audio and video data, as well as timestamps relating to the delivery of the data stream itself.

If subtitles are included in the stream, subtitle decoder 3210 decodes the subtitles and synchronizes them with the video frame and audio frame and transmits the video/audio/subtitles to the video/audio/subtitle display 3216.

The present invention is not limited to the above system, and either the picture encoding device or the picture decoding device in the above embodiments may be included in another system, such as an automobile system.

Although embodiments of the present invention have been primarily described in terms of video encoding, it should be noted that embodiments of encoding system 10, encoder 20, and decoder 30 (and thus system 10), as well as other embodiments described herein, may also be configured to process or encode a still picture, i.e. processing or encoding a single picture independently of any preceding or subsequent picture, as in video encoding. In general, only the inter prediction (encoder) and 344 (decoder) blocks 244 may not be available if the encoding for picture processing is limited to a single picture 17. All other functions (also referred to as tools or techniques) of video encoder 20 and video decoder 30 may equally used for processing static images, for example, remainder calculation 204/304, transform 206, quantization 208, inverse quantization 210/310, (inverse) transform 212/312, division 262/362, intra prediction 254/354 and/or contour filtering 220, 320, entropy encoding 270 and entropy decoding 304.

Embodiments of, for example, encoder 20 and decoder 30, and the functions described herein, for example, with reference to encoder 20 and decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted through a communication medium in the form of one or more instructions or code and executed by a hardware processing unit. Computer-readable media may include computer-readable media that corresponds to a tangible medium, such as a storage medium, or a communication medium, including any medium that enables transmission of a computer program from one location to another, such as according to a communications protocol. Thus, computer readable media may typically correspond to (1) tangible computer readable media that is durable, or (2) a communication medium such as a signal or carrier wave. Storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementing the techniques described in this disclosure. The computer program product may include computer-readable media.

By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage device, magnetic disk storage device or other magnetic storage devices, flash memory or any other storage medium, which can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Additionally, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave communications, then coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave communications are included in the definition of media. However, it should be understood that computer readable and storage media do not include connections, carrier waves, signals, or other short-lived media, but are instead directed to long-term, tangible storage media. The disk and disc used in this document include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where discs ) typically reproduce data magnetically, while discs (discs) reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer readable media.

The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term “processor” as used herein can refer to any of the above structure or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within specialized hardware and/or software modules configured for encoding and decoding or included in a bundled codec. In addition, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or hardware components, including a wireless telephone, an integrated circuit (IC), or a set of ICs (eg, a chipset). This disclosure describes various components, modules, or blocks to highlight functional aspects of devices configured to implement the disclosed technologies, but not necessarily required to be implemented by different hardware units. Rather, as described above, the various blocks may be combined into a codec hardware block or provided by a collection of interacting hardware blocks, including one or more processors as described above, together with suitable software and/or firmware.

Claims (35)

1. An encoding method implemented by a decoding device, containing: obtaining a bitstream, wherein at least one bit in the bitstream represents a syntax element for the current block, wherein the syntax element indicates a truncation index of a truncation value for the adaptive loop filter (ALF); parsing the bitstream to obtain the value of a syntax element for the current block, wherein the syntax element is encoded using a fixed-length code; Apply adaptive contour filtering to the current block based on the value of the syntax element for the current block. 2. The method of claim 1, wherein the fixed-length code comprises a binary representation of an unsigned integer using at least one bit. 3. The method of claim 1 or 2, wherein the syntax element is applied to a set of blocks and the current block is one block in the set of blocks. 4. The method according to claim 1 or 2, in which the syntax element is at the slice level. 5. Method according to any one of paragraphs. 1-4, in which the syntax element is an index of the truncation value ALF, and at least one bit representing the syntax element means two bits. 6. The method of claim 5, wherein the truncation value index ALF identifies one truncation value from the four truncation values. 7. Method according to any one of paragraphs. 1-6, in which the truncation value index value ALF is used to determine the truncation range, wherein the truncation range is used in the adaptive loop filtering process. 8. An encoding method implemented by an encoding device, containing: determining a syntax element value for the current block, wherein the syntax element specifies a truncation index of the truncation value for the adaptive loop filter (ALF); generating a bit stream based on the value of the syntax element, wherein at least one bit in the bit stream represents the syntax element, wherein the syntax element is encoded using a fixed length code. 9. The method of claim 8, wherein the fixed-length code comprises a binary representation of the unsigned integer using at least one syntax element bit. 10. The method of claim 8 or 9, wherein the syntax element is applied to a set of blocks and the current block is one block in the set of blocks. 11. The method of claim 8 or 9, wherein the syntax element is at the slice level. 12. Method according to any one of paragraphs. 8-11, in which the syntax element is an ALF truncation value index and at least one bit representing the syntax element means two bits. 13. The method of claim 12, wherein the truncation value index ALF identifies one truncation value from the four truncation values. 14. Method according to any one of paragraphs. 8-13, in which the truncation value index value ALF is used to determine a truncation range, wherein the truncation range is used in the adaptive loop filtering process. 15. A decoder (30) comprising a processing circuit for performing the method according to any one of claims. 1-7. 16. An encoder (20) containing a processing circuit for performing the method according to any one of claims. 8-14. 17. A non-transitory computer-readable medium carrying program code that, when executed by a computer device, causes the computer device to perform the method according to any one of claims. 1-14. 18. Decoder containing: one or more processors; And a non-volatile computer-readable medium coupled to the processors and storing a program portion for execution by the processors, wherein the program portion, when executed by the processors, configures a decoder to perform the method of any one of claims. 1-7. 19. Encoder containing: one or more processors; And a non-transitory computer-readable medium coupled to the processors and storing a program portion for execution by the processors, wherein the program portion, when executed by the processors, configures an encoder to perform the method of any one of claims. 8-14. 20. Decoder containing: an entropy decoding block configured to obtain a bitstream, wherein at least one bit in the bitstream represents a syntax element for the current block, wherein the syntax element indicates a truncation index of a truncation value for the adaptive loop filter (ALF); the entropy decoding unit is further configured to parse the bitstream to obtain a syntax element value for the current block, wherein the syntax element is encoded using a fixed length code; And a filtering block configured to apply adaptive loop filtering to the current block based on the value of the syntax element for the current block. 21. Encoder containing: a determination block configured to determine the value of a syntax element for the current block, wherein the syntax element indicates a truncation index of the truncation value for the adaptive loop filter (ALF); an entropy encoding unit configured to generate a bit stream based on the value of the syntax element, wherein at least one bit in the bit stream represents the syntax element, wherein the syntax element is encoded using a fixed length code.