HK40092888B - Method for decoding inter-prediction video block of video stream and electronic equipment - Google Patents

Description

Methods and electronic devices for decoding inter-frame predicted video blocks of a video stream

Incorporation

This international PCT application is based on and claims priority to U.S. non-provisional patent application No. 17/708,801, filed March 30, 2022, which in turn claims priority to U.S. provisional patent application No. 63/270,397, filed October 21, 2021, and U.S. provisional patent application No. 63/289,122, filed December 13, 2021, both entitled "Adaptive Resolution of Motion Vector Difference". The entire contents of the earlier applications are incorporated herein by reference.

Technical Field

This disclosure relates generally to video encoding/decoding, and more particularly to methods and electronic devices for decoding inter-frame predicted video blocks of a video stream.

Background Technology

The background description provided herein is intended to provide a general explanation of the contents of this disclosure. Certain works of the inventors (i.e., those already described in this background section) and content in the specification relating to prior art prior to the filing date are not, whether explicitly or implicitly, considered to be prior art relative to this disclosure.

Video encoding and decoding can be performed using inter-frame picture prediction with motion compensation. Uncompressed digital video can comprise a series of pictures, each with a spatial size of, for example, a 1920×1080 luminance sample and an associated full-sampled or subsampled chrominance sample. This series of pictures can have a fixed or variable picture rate (alternatively referred to as the frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, and 4:2:0 chrominance subsampling with 8 bits per pixel per color channel requires bandwidth close to 1.5 Gbit/s. One hour of such video would require more than 600 GB of storage space.

One objective of video encoding and decoding is to reduce redundancy in the uncompressed input video signal through compression. Compression can help reduce the aforementioned bandwidth and/or storage space requirements, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations thereof can be employed. Lossless compression refers to a technique that reconstructs an exact copy of the original signal from the compressed original signal through a decoding process. Lossy compression refers to an encoding/decoding process in which the original video information is not fully preserved during encoding and is not fully recovered during decoding. When using lossy compression, the reconstructed signal may differ from the original signal, but despite the loss of some information, the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. In the case of video, lossy compression is widely used in many applications. The tolerable amount of distortion depends on the application. For example, users of some consumer video streaming applications may tolerate higher distortion than users of movie or television broadcasting applications. The compression ratio achievable through a specific coding algorithm can be selected or adjusted to reflect various levels of distortion tolerance: higher tolerable distortion generally allows the coding algorithm to produce higher loss and higher compression ratio.

Video encoders and decoders can utilize techniques from multiple categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

Video codec techniques can include techniques called intra-frame coding. In intra-frame coding, sample values are represented without reference to samples or other data from previously reconstructed reference images. In some video codecs, images are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the image is called an intra-frame picture. Intra-frame pictures and their derivatives (e.g., independent decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in an encoded video stream and video session, or as a still image. Samples of blocks following intra-frame prediction can then be transformed into the frequency domain, and the transformed coefficients thus generated can be quantized before entropy coding. Intra-frame prediction represents a technique that minimizes the sample values in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block for a given quantization step size.

For example, traditional intra-frame coding techniques known from MPEG-2 generation coding technologies do not use intra-frame prediction. However, some newer video compression techniques include those that attempt to encode/decode blocks based on, for example, surrounding sample data and/or metadata obtained during spatially adjacent encoding and/or decoding, preceding the data blocks encoded or decoded intra-frame in the decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. It should be noted that, at least in some cases, intra-frame prediction uses only reference data from the current frame being reconstructed, and not reference data from other reference frames.

Intra-prediction can take many different forms. When more than one such technique is available in a given video coding technique, the technique in use is called an intra-prediction mode. One or more intra-prediction modes can be provided in a particular codec. In some cases, a mode may have sub-modes and/or may be associated with various parameters, and the mode/sub-mode information and intra-coding parameters for the video block may be encoded separately or included together in the mode codeword. Which codeword is used for a given mode, sub-mode, and/or parameter combination can affect the coding efficiency gain through intra-prediction, and the entropy coding technique used to convert the codeword into a bitstream can also have an impact.

H.264 introduced a certain intra-frame prediction mode, which was improved in H.265 and further refined in new coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Typically, for intra-frame prediction, neighboring sample values that have become available are used to form prediction blocks. For example, available values from a specific set of neighboring samples can be copied into the prediction block along some direction and/or line. References to the directions used can be encoded in the bitstream or predicted on their own.

Referring to Figure 1A, a subset of nine prediction directions is depicted in the lower right corner, representing nine of the 33 possible intra-frame prediction directions of H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes specified in H.265). The point (101) where the arrow converges indicates the sample being predicted. The arrows indicate the directions along which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples in the upper right corner at a 45-degree angle to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples in the lower left corner of sample (101) at a 22.5-degree angle to the horizontal.

Referring again to Figure 1, a 4×4 square block (104) of samples is depicted in the upper left (indicated by bold dashed lines). The square block (104) contains 16 samples, each labeled with “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is in the lower right corner. Example reference samples following a similar numbering scheme are also shown. Reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to block (104). In H.264 and H.265, predicted samples adjacent to the block being reconstructed are used.

Intra-frame image prediction for block 104 can begin by copying reference sample values from neighboring samples according to the prediction direction indicated by a signal. For example, suppose the encoded video stream includes signaling for the prediction direction indicated by the arrow (102) for block 104, i.e., predicting samples from one or more prediction samples at the upper right, at a 45-degree angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted based on reference sample R08.

In some cases, the values of multiple reference samples can be combined, for example, by interpolation, in order to calculate the reference sample, especially when the direction is not divisible by 45 degrees.

As video coding techniques continue to evolve, the number of possible directions increases. For example, in H.264 (2003), nine different directions were available for intra-frame prediction. In H.265 (2013), this increased to 33 directions, and at the time of this disclosure, JEM/VVC/BMS supports up to 65 directions. Experimental studies have been conducted to help identify the most suitable intra-frame prediction directions, and some techniques in entropy coding can be used to encode those most suitable directions with a small number of bits, accepting a certain bit cost for the direction. Furthermore, sometimes the direction itself can be predicted from neighboring directions used in the intra-frame prediction of already decoded adjacent blocks.

Figure 1B shows a schematic diagram (180) depicting 65 intra-frame prediction directions according to JEM to illustrate the increase in the number of prediction directions in various coding techniques developed over time.

The way bits representing intra-prediction directions in an encoded video bitstream are mapped to prediction directions can vary depending on the video coding technique; for example, it can range from a simple, direct mapping of the prediction direction to the intra-prediction mode, to a mapping to a codeword, to a complex adaptive scheme involving the most probable mode, and similar techniques. However, in all cases, for intra-prediction, there may be some directions that are statistically less likely to occur in the video content compared to others. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, those less likely directions will be represented by more bits than the more likely directions.

Inter-frame image prediction, or inter-prediction, can be based on motion compensation. In motion compensation, sample data from a previously reconstructed image or a portion thereof (the reference image), after being spatially offset along a direction indicated by a motion vector (hereafter referred to as MV), can be used to predict a newly reconstructed image or image portion (e.g., a patch). In some cases, the reference image can be the same as the image currently being reconstructed. MV can have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference image being used (similar to a temporal dimension).

In some video compression techniques, the current MV applicable to a region of sample data can be predicted based on other MVs, such as those related to other regions of sample data that are spatially adjacent to the region being reconstructed and precede the current MV in decoding order. This significantly reduces the total amount of data required to encode the MV by eliminating redundancy in related MVs, thereby increasing compression efficiency. MV prediction works effectively, for example, because when encoding an input video signal (called natural video) from a camera, there is a statistical probability that a larger region than the region applicable to a single MV moves along similar directions in the video sequence. Therefore, in some cases, the larger region can be predicted using similar motion vectors derived from the MVs of neighboring regions. This makes the actual MV for a given region similar to or the same as the MV predicted based on the surrounding MVs. Consequently, after entropy coding, the MV can be represented with fewer bits than when directly encoding the MV (rather than predicting the MV based on neighboring MVs). In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, such as when rounding errors occur while calculating the predicted value based on multiple surrounding MVs, the MV prediction itself can be lossy.

H.265/HEVC (ITU-T H.265 Recommendation, “High Efficiency Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms specified in H.265, this paper describes the technique referred to below as “spatial combining”.

Specifically, referring to Figure 2, the current block (201) includes samples that have been discovered by the encoder during the motion search process, and these samples can be predicted based on previous blocks of the same size that have generated spatial offsets. Alternatively, the MV can be derived from the metadata associated with one or more reference images, rather than being directly encoded. For example, using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206 respectively), the MV is derived from the metadata of the nearest reference image (in decoding order). In H.265, MV prediction can use predictions from the same reference image being used by adjacent blocks.

Summary of the Invention

This disclosure generally relates to video coding, and more particularly to a method and system for providing adaptive resolution of motion vector difference in inter-frame prediction of video blocks. In an example implementation, a method for decoding inter-frame predicted video blocks of a video stream is disclosed. The method may include: receiving the video stream; determining that a motion vector difference (MVD) between a motion vector associated with the inter-frame predicted video block and a reference motion vector is written into the video stream, wherein the reference motion vector corresponds to only one of a reference picture in a reference frame list 0 and a reference frame list 1, unless the MVD is written jointly for both reference pictures; obtaining from the video stream an indication of a size range of the MVD within a plurality of predefined motion vector difference size ranges; determining the pixel resolution of the MVD based on the size range; identifying additional MVD information in the video stream based on the pixel resolution; extracting the additional MVD information from the video stream; and decoding the inter-frame predicted video block based on the pixel resolution, the additional MVD information, the reference motion vector, and the reference frame associated with the motion vector.

In the example implementation above, the pixel resolution is 2 ^{^n} pixels, where n is an integer ranging from -6 to 11 and inclusive.

In any of the above example implementations, the plurality of predefined motion vector difference magnitude ranges are associated with pixel resolutions in a predefined manner in a non-ascending order, wherein higher pixel resolutions are associated with smaller pixel resolution values.

In any of the above example implementations, obtaining an indication of the size range of the MVD includes: extracting a first predefined syntax element from the video stream, the first predefined syntax element indicating the MVD category of the MVD in a predefined MVD category set, with lower MVD categories corresponding to smaller MVD size ranges; and determining the size range of the MVD based on the MVD category.

In any of the above example implementations, determining the pixel resolution of the MVD based on the size range includes: determining whether the size range is higher than a preset MVD range threshold level; when the size range is higher than the preset MVD range threshold level, determining the pixel resolution to be an integer number of pixels; and when the size range is not higher than the preset MVD range threshold level, determining the pixel resolution to be a fractional number of pixels.

In any of the above example implementations, identifying additional MVD information in the video stream based on the pixel resolution includes: parsing the video stream according to a second predefined syntax element to obtain the integer pixel portion of the MVD; and, when it is determined that the pixel resolution is a fractional pixel, further parsing the video stream according to at least a third predefined syntax element to obtain the fractional pixel portion of the MVD.

In any of the above example embodiments, the MVD range threshold level includes the lowest or second lowest MVD in the predefined set of MVD categories.

In any of the above example implementations, each MVD in the predefined set of MVD categories with a size range higher than the preset MVD range threshold level is associated with a single allowed integer MVD pixel value.

In any of the above example implementations, the single allowed integer pixel value includes the pixel value corresponding to the higher value in the corresponding size range.

In any of the above example embodiments, the single allowed integer pixel value includes the pixel value corresponding to the midpoint value in the corresponding size range.

In any of the above example implementations, determining the pixel resolution of the MVD based on the size range includes: determining whether the size range is lower than, includes, or is higher than a preset MVD threshold value; when the size range is determined to be higher than the preset MVD threshold value, determining the pixel resolution to be an integer number of pixels; and when the size range is determined to be lower than the preset MVD threshold value, determining the pixel resolution to be a fractional number of pixels.

In any of the above example implementations, the method further includes: when determining that the size range includes the preset MVD threshold size value: extracting a second predefined syntax element from the video stream, the second predefined syntax element indicating an MVD size offset relative to the starting size of the MVD size range; obtaining an integer size of the MVD based on the MVD size range and the MVD size offset; determining that the pixel resolution is a decimal when the integer size of the MVD is not higher than the preset MVD threshold size value; and determining that the pixel resolution is a non-decimal when the integer size of the MVD is higher than the preset MVD threshold size value.

In any of the above example implementations, when the pixel resolution is determined to be a decimal, identifying additional MVD information in the video stream based on the pixel resolution includes: parsing the video stream according to a third predefined syntax element to obtain the decimal part of the MVD.

In any of the above example implementations, the MVD threshold value is less than 4 pixels.

In any of the above example implementations, the MVD pixel resolution associated with the plurality of predefined motion vector difference size ranges differs from one size range to another.

In some other example implementations, a method for decoding inter-frame predicted video blocks of a video stream is disclosed. The method may include: receiving the video stream; determining that a motion vector difference (MVD) between a motion vector associated with the inter-frame predicted video block and a reference motion vector is written into the video stream, wherein the reference motion vector corresponds to only one of a reference image in a reference frame list 0 and a reference frame list 1, unless the MVD is written jointly for both reference images; extracting an integer portion of the size of the MVD from the video stream; determining the pixel resolution of the MVD based on the integer portion of the size of the MVD; identifying additional MVD information in the video stream based on the pixel resolution; and decoding the inter-frame predicted video block based on the pixel resolution, the integer portion of the size of the MVD, the additional MVD information, the reference motion vector, and the reference frame associated with the motion vector.

In the example implementation above, the pixel resolution of the multiple MVDs depends on the size of the multiple MVDs in descending order.

In any of the above example implementations, determining the pixel resolution of the MVD based on the integer part of the MVD size includes: determining whether the integer part of the MVD size is higher than a preset MVD threshold size; when the integer part of the MVD size is higher than the preset MVD threshold size, determining the pixel resolution to be an integer number of pixels; and when the integer part of the MVD size is not higher than the preset MVD threshold size, determining the pixel resolution to be a fractional number of pixels.

Various aspects of this disclosure also provide a video encoding or decoding device or apparatus, including circuitry configured to perform any of the methods described above.

Various aspects of this disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and/or encoding, cause the computer to perform the methods described above for video decoding and/or encoding.

Attached Figure Description

Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

Figure 1A illustrates a schematic diagram of an exemplary subset of intra-frame prediction direction patterns.

Figure 1B illustrates an exemplary intra-frame prediction direction.

Figure 2 shows a schematic diagram of the current block and its surrounding spatial merging candidate blocks for motion vector prediction in an example.

Figure 3 shows a simplified block diagram of a communication system (300) according to an example embodiment.

Figure 4 shows a simplified block diagram of a communication system (400) according to an example embodiment.

Figure 5 shows a simplified block diagram of a video decoder according to an example embodiment.

Figure 6 shows a simplified block diagram of a video encoder according to an example embodiment.

Figure 7 shows a block diagram of a video encoder according to another example embodiment.

Figure 8 shows a block diagram of a video decoder according to another example embodiment.

Figure 9 illustrates a scheme for dividing coded blocks according to an example embodiment of the present disclosure.

Figure 10 illustrates another scheme for code block partitioning according to an example embodiment of this disclosure.

Figure 11 illustrates another scheme for code block partitioning according to an example embodiment of this disclosure.

Figure 12 shows an example of dividing a basic block into coded blocks according to an example partitioning scheme.

Figure 13 illustrates an exemplary ternary partitioning scheme.

Figure 14 illustrates an exemplary quadtree/binary tree coding block partitioning scheme.

Figure 15 illustrates a scheme for dividing a coded block into multiple transform blocks according to an example embodiment of the present disclosure, and the encoding order of these transform blocks.

Figure 16 illustrates another scheme for dividing a coded block into multiple transform blocks according to an example embodiment of the present disclosure, and the encoding order of these transform blocks.

Figure 17 illustrates another scheme for dividing a coded block into multiple transform blocks according to an example embodiment of the present disclosure.

Figure 18 shows a flowchart of a method according to an example embodiment of the present disclosure.

Figure 19 shows another flowchart of a method according to an example embodiment of the present disclosure.

Figure 20 shows a schematic diagram of a computer system according to an example embodiment of the present disclosure.

Detailed Implementation

Throughout the specification and claims, terms may have subtle meanings implied or implied in the context that go beyond their expressly stated meanings. The phrases “in one embodiment” or “in some embodiments” as used herein do not necessarily refer to the same embodiment, and the phrases “in another embodiment” or “in other embodiments” as used herein do not necessarily refer to different embodiments. Similarly, the phrases “in one implementation” or “in some implementations” as used herein do not necessarily refer to the same implementation, and the phrases “in another implementation” or “in other embodiments” as used herein do not necessarily refer to different implementations. For example, the claimed subject matter includes combinations of all or some exemplary embodiments/implementations.

Generally, terms can be understood at least in part from their usage in the context. For example, terms such as “and,” “or,” or “and/or” as used herein can include a variety of meanings that can depend at least in part on the context in which the terms are used. Typically, “or,” when used to relate a list such as A, B, or C, is intended to mean A, B, and C (used here for inclusion) and A, B, or C (used here for exclusion). Furthermore, the terms “one or more” or “at least one,” as used herein, can be used, at least in part on the context, to describe any feature, structure, or characteristic in a singular sense, or to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms such as “a,” “an,” or “the” can also be understood to convey either a singular or a plural usage, at least in part on the context. Furthermore, terms such as “based on” or “determined by” can be understood to not necessarily convey a set of exclusive factors, but may allow for the presence of other factors that are not necessarily explicitly described, which also depends at least in part on the context. Figure 3 shows a simplified block diagram of a communication system (300) according to an embodiment of this disclosure. The communication system (300) includes multiple terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) can perform unidirectional data transmission. For example, terminal device (310) can encode video data (e.g., video data from a video picture stream captured by terminal device (310)) for transmission over the network (350) to another terminal device (320). The encoded video data is transmitted in the form of one or more encoded video streams. Terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to recover video pictures, and display the video pictures based on the recovered video data. Unidirectional data transmission can be implemented in applications such as media services.

In another example, the communication system (300) includes a pair of second terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which may be implemented, for example, during a video conferencing application. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., video data from a video picture stream captured by the terminal device) for transmission over a network (350) to the other terminal device (330) and (340). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other terminal device (330) and (340), and may decode the encoded video data to recover video pictures, and may display the video pictures on an accessible display device based on the recovered video data.

In the example of Figure 3, terminal devices (310), (320), (330), and (340) can be implemented as servers, personal computers, and smartphones, but the applicability of the basic principles of this disclosure is not limited thereto. Embodiments of this disclosure can be implemented on desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and/or the like. Network (350) refers to any number or type of network that transmits encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired (connected) and/or wireless communication networks. Communication networks (350) can exchange data in circuit-switched channels, packet-switched channels, and/or other types of channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and/or the Internet. For the purposes of this discussion, unless explicitly stated herein, the architecture and topology of the network (350) may be irrelevant to the operation of this disclosure.

As an example of an application of the disclosed subject matter, Figure 4 illustrates the placement of a video encoder and a video decoder in a video streaming environment. The disclosed subject matter is equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

The video streaming system may include a video capture subsystem (413), which may include, for example, a video source (401) of a digital camera, which creates an uncompressed video picture stream or image (402). In one example, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream compared to encoded video data (404) (or encoded video bitstream), and the video picture stream (302) may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. Compared to the uncompressed video picture stream (402), the encoded video data (404) (or encoded video bitstream (404)) depicted as thin lines to emphasize its lower data volume can be stored on a streaming server (405) for future use or directly stored on a downstream video device (not shown). One or more streaming client subsystems, such as client subsystems (406) and (408) in FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an uncompressed output video picture stream (411) that can be presented on a display (412) (e.g., a screen) or other presentation device (not depicted). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, encoded video data (404), video data (407), and video data (409) (e.g., video streams) can be encoded according to certain video coding/compression standards. Examples of such standards include ITU-T H.265. In embodiments, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed topics are applicable in the context of VVC and can be used with other video coding standards.

It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

In the following text, Figure 5 shows a block diagram of a video decoder (510) according to any embodiment of the present disclosure. The video decoder (510) may be disposed in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of the video decoder (410) in the example of Figure 4.

The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence may be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence may be associated with multiple video frames or images. Encoded video sequences may be received from a channel (501), which may be a hardware/software link leading to a storage device storing the encoded video data or a streaming source transmitting the encoded video data. The receiver (531) may receive encoded video data and other data, such as encoded audio data and/or auxiliary data streams, that may be forwarded to their respective processing circuitry (not depicted). The receiver (531) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (515) may be provided between the receiver (531) and the entropy decoder/parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be located externally to and separate from the video decoder (510) (not depicted). In other applications, the buffer memory (not depicted) may be located externally to the video decoder (510) to, for example, prevent network jitter, and another additional buffer memory (515) may be located internally to, for example, handle broadcast timing. When the receiver (531) receives data from a store/forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer memory (515) may not be required, or it may be made smaller. For use on packet networks such as the Internet, a buffer memory (515) of sufficient size may be required, and the size of the buffer memory (515) may be relatively large. Such a buffer memory may be implemented with an adaptive size and may be implemented at least partially in an operating system or a similar component (not depicted) external to the video decoder (510).

The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (510) and potential information for controlling a presentation device such as a display (512) (e.g., a screen), which may or may not be integral to the electronic device (530), but may be coupled to the electronic device (530), as shown in Figure 5. Control information for the presentation device may be in the form of Supplemental Enhancement Information (SEI) messages or fragments of Video Usability Information (VUI) parameter sets (not depicted). The parser (520) may perform parsing/entropy decoding on the encoded video sequence received by the parser (520). The entropy coding of the encoded video sequence may be performed according to video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) can extract a set of subgroup parameters from the encoded video sequence for use in the video decoder, based on at least one parameter corresponding to a subgroup. Subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

The parser (520) can perform entropy decoding/parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

Depending on the type of encoded video frames or portions thereof (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different processing or functional units. Which units are involved and how they are involved can be controlled by the parser (520) through subgroup control information parsed from the encoded video sequence. For simplicity, the flow of such subgroup control information between the parser (520) and the various processing or functional units described below is not depicted.

In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the following disclosure adopts the conceptual subdivision into multiple functional units.

The first unit may include a scaler/inverse transform unit (551). The scaler/inverse transform unit (551) can receive quantization transform coefficients as symbols (521) from the parser (520) and control information, including information indicating which type of inverse transform to use, block size, quantization factor/parameter, quantization scaling matrix, etc. The scaler/inverse transform unit (551) can output a block including sample values, which can be input into the aggregator (555).

In some cases, the output samples of the scaler/inverse transform (551) may belong to intra-coded blocks; that is, blocks that do not use prediction information from previously reconstructed images, but can use prediction information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use information from surrounding blocks that have been reconstructed and stored in the current picture buffer (558) to generate blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers partially reconstructed and/or fully reconstructed current images. In some implementations, the aggregator (555) may add the prediction information generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler/inverse transform unit (551) based on each sample.

In other cases, the output samples of the scaler/inverse transform unit (551) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (553) may access the reference image memory (557) to extract samples for inter-frame image prediction. After motion compensation is performed on the extracted samples according to the symbols (521) belonging to the block, these samples may be added by the aggregator (555) to the output of the scaler/inverse transform unit (551) (the output of unit 551 may be referred to as residual samples or residual signals), thereby generating output sample information. The extraction of prediction samples by the motion compensation prediction unit (553) from the address in the reference image memory (557) may be controlled by motion vectors, and these motion vectors may be provided to the motion compensation prediction unit (553) in the form of symbols (521), which may have, for example, an X component, a Y component (offset), and a reference image component (time). Motion compensation may also include interpolation of sample values extracted from the reference image memory (557) when using subsample precise motion vectors, and may also be associated with motion vector prediction mechanisms, etc.

The output samples of the aggregator (555) can be employed by various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (521) from the parser (520) in the loop filter unit (556). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values. Various types of loop filters can be included in various orders as part of the loop filter unit 556, as will be described in further detail below.

The output of the loop filter unit (556) can be a sample stream that can be output to the presentation device (512) and stored in the reference image memory (557) for future inter-frame image prediction.

Once fully reconstructed, certain encoded images can be used as reference images for future inter-frame image prediction. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (520)) is identified as the reference image, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

The video decoder (510) can perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under that configuration file. To conform to the standard, the complexity of the encoded video sequence may be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

In some exemplary embodiments, the receiver (531) may receive supplemental (redundant) data along with the encoded video. This supplemental data may be a portion of the encoded video sequence. The supplemental data may be used by the video decoder (510) to properly decode the data and/or more accurately reconstruct the original video data. The supplemental data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

Figure 6 is a block diagram of a video encoder (603) according to an exemplary embodiment disclosed in this application. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., transmission circuitry). The video encoder (603) may be used in place of the video encoder (403) in the example of Figure 4.

The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of Figure 6), which can capture video images that will be encoded by the video encoder (603). In another example, the video source (601) may be implemented as part of the electronic device (620).

A video source (601) can provide a sequence of source video samples to be encoded by a video encoder (603) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCB, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device capable of storing previously prepared video. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. The video data can be provided as multiple individual pictures or images, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. The relationship between pixels and samples will be readily understood by those skilled in the art. The following focuses on describing samples.

According to some exemplary embodiments, a video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units described below. For simplicity, coupling is not depicted in the figures. Parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of images (GOP) layout, maximum motion vector search range, etc. The controller (650) may be used with other suitable functions related to a video encoder (603) optimized for a particular system design.

In some exemplary embodiments, the video encoder (603) may be configured to operate within an encoding loop. As an oversimplification, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded within the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data, even though the embedded decoder 633 processes the encoded video stream through the source encoder 630, which does not have entropy coding (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols in entropy coding and the encoded video stream can be lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since the decoding of the symbol stream produces bit-accurate results independent of the decoder location (local or remote), the contents of the reference image memory (634) also correspond bit-accurately between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the encoder's prediction section are exactly the same as the sample values that the decoder will "see" during the prediction phase. This fundamental principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is used to improve coding quality.

The operation of the “local” decoder (633) can be the same as that of the “remote” decoder, for example, the video decoder (510) which has been described in detail above in conjunction with Figure 5. However, referring briefly to Figure 5 again, since symbols are available and the entropy encoder (645) and parser (520) are able to encode/decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (510), which includes the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633) or in the encoder.

It can be observed that any decoder technique, except for parsing/entropy decoding which may only exist in the decoder, must also exist in the corresponding encoder in essentially the same functional form. For this reason, the disclosed topics sometimes focus on decoder operations, which cooperate with the decoding part of the encoder. Therefore, the description of encoder techniques can be simplified, as encoder techniques are inverses of the fully described decoder techniques. Only certain areas or aspects of the encoder are described in more detail below.

During operation, in some exemplary implementations, the source encoder (630) may perform motion-compensated predictive coding, which predictively encodes the input image by referencing one or more previously encoded images from the video sequence designated as "reference images." In this manner, the encoding engine (632) encodes the differences (or residuals) in the color channels between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image. The term "residual" and its adjective form "residual" are used interchangeably.

The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture based on symbols created by the source encoder (630). The operation of the encoding engine (632) can be a lossy process. When the encoded video data can be decoded on the video decoder (not shown in Figure 6), the reconstructed video sequence is often a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that can be performed by the video decoder on the reference picture, and can store the reconstructed reference picture in a reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that shares common content (no transmission errors) with the reconstructed reference picture that will be obtained by the remote video decoder.

The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (635), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units using techniques such as Huffman coding, variable-length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware/software link to a storage device that will store the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and/or auxiliary data streams (source not shown).

The controller (650) manages the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Variations of I-pictures and their corresponding applications and characteristics are well understood by those skilled in the art.

A predictive picture (P-picture) can be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values for each block.

A bidirectional predictive picture (B-picture) can be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

Source images can typically be spatially subdivided into multiple sample coding blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and coded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the block can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction. For other purposes, source images or intermediate images can be subdivided into other types of blocks. The partitioning of coding blocks and other types of blocks may or may not follow the same pattern, as described in further detail below.

The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

In one exemplary embodiment, the transmitter (640) may transmit additional data while transmitting encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal/spatial/SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

The captured video can be presented as multiple source images (video images) in a time-series format. Intra-frame image prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given image, while inter-frame prediction utilizes (temporal or other) correlations between images. For example, a specific image being encoded/decoded can be divided into blocks, and this specific image being encoded/decoded is called the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference image.

In some exemplary embodiments, bidirectional prediction techniques can be used for inter-frame image prediction. According to this bidirectional prediction technique, two reference images are used, such as a first reference image and a second reference image that precede the current image in the video in decoding order (but may be past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted jointly using a combination of the first and second reference blocks.

In addition, merging mode techniques can be used for inter-frame image prediction to improve coding efficiency.

According to some exemplary embodiments of this disclosure, predictions such as inter-frame picture prediction and intra-frame picture prediction are performed on a block-by-block basis. For example, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression. The CTUs in the pictures may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU may include three parallel coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively partitioned into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU may be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs. Each of the one or more 32×32 blocks may be further partitioned into four 16×16 pixel CUs. In some exemplary embodiments, each CU may be analyzed during encoding to determine the prediction type used for the CU among various prediction types, such as inter-frame prediction or intra-frame prediction. Based on temporal and/or spatial predictability, a Cubic Queues (CUs) can be divided into one or more Prediction Units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations during encoding (encoding/decoding) are performed on a per-prediction-block basis. The division of a CU into PUs (or PBs for different color channels) can be performed in various spatial modes. For example, a luma or chroma PB may comprise a matrix of values for samples (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

Figure 7 illustrates a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is used to receive a processing block (e.g., a prediction block) of sample values within a current video image in a video image sequence, and to encode the processing block into an encoded image that is part of an encoded video sequence. The exemplary video encoder (703) can be used in place of the video encoder (403) in the example of Figure 4.

For example, a video encoder (703) receives a matrix of sample values for a processing block, such as an 8×8 sample prediction block. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether to optimally encode the processing block using intra-frame mode, inter-frame mode, or bidirectional prediction mode. When it is determined that the processing block should be encoded in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processing block into an encoded picture; and when it is determined that the processing block should be encoded in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques respectively to encode the processing block into an encoded picture. In some exemplary embodiments, a merging mode can be used as an inter-frame picture prediction sub-mode, wherein motion vectors are derived from one or more motion vector predictors without the aid of encoded motion vector components outside the predictors. In some other exemplary embodiments, motion vector components applicable to the subject block may be present. Therefore, the video encoder (703) may include components not explicitly shown in FIG7, such as a mode decision module for determining the prediction mode of the processing block.

In the example of FIG7, the video encoder (703) includes an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in the exemplary arrangement of FIG7.

The inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and later images in display order), generate inter-frame prediction information (e.g., redundancy information description based on inter-frame coding techniques, motion vectors, merging mode information), and compute inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image based on encoded video information, decoded using decoding unit 633, which is embedded in the exemplary encoder 620 of FIG. 6 (shown as the residual decoder 728 of FIG. 7, as described in further detail below).

The intra encoder (722) is configured to receive samples of the current block (e.g., the processed block), compare the block with encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information based on one or more intra coding techniques). The intra encoder (722) may compute intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

A general controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on that general control data. In one example, the general controller (721) determines the prediction mode of a block and provides control signals to a switch (726) based on that prediction mode. For example, when the prediction mode is an intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; and when the prediction mode of a block is an inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

A residual calculator (723) can be configured to calculate the difference (residual data) between the received block and the prediction result of a block selected from the intra encoder (722) or the inter encoder (730). A residual encoder (724) can be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) can be configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform the inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, which can be buffered in a memory circuit (not shown) and used as a reference image.

The entropy encoder (725) can be configured to format the bitstream to include encoded blocks and perform entropy encoding. The entropy encoder (725) is configured to include various types of information in the bitstream. For example, the entropy encoder (725) can be configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Residual information may be absent when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

Figure 8 illustrates an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In one example, the video decoder (810) can be used instead of the video decoder (410) in the example of Figure 4.

In the example of FIG8, the video decoder (810) includes an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together as shown in the exemplary arrangement of FIG8.

An entropy decoder (871) can be used to reconstruct certain symbols from an encoded picture, which represent the syntax elements constituting the encoded picture. Such symbols may include, for example, the mode encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, merged sub-mode, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can be identified for use by the intra-frame decoder (872) or the inter-frame decoder (880) for prediction, residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (872). Residual information may be provided to the residual decoder (873) via inverse quantization.

The inter-frame decoder (880) can be configured to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

The intra-frame decoder (872) can be configured to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

The residual decoder (873) can be configured to perform inverse quantization to extract the dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) can also use certain control information (to include quantizer parameters (QP)) which can be provided by the entropy decoder (871) (the data path is not depicted because this is only low-data-volume control information).

The reconstruction module (874) can be configured to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module, as appropriate) in the spatial domain to form a reconstructed block, which forms part of the reconstructed image, which in turn forms part of the reconstructed video. It should be noted that other suitable operations, such as deblocking, can also be performed to improve visual quality.

It should be noted that any suitable technology can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In some exemplary embodiments, one or more integrated circuits may be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In another embodiment, one or more processors executing software instructions may be used to implement the video encoder (403), video encoder (603), and video encoder (603), as well as the video decoder (410), video decoder (510), and video decoder (810).

Turning to block partitioning for encoding and decoding, partitioning can generally begin with basic blocks and can follow a predefined set of rules, a specific pattern, a partition tree, or any partitioning structure or scheme. Partitioning can be hierarchical and recursive. After dividing or partitioning the basic blocks according to any of the example partitioning processes described below, or other processes or combinations thereof, a final set of partitions or coding blocks can be obtained. Each of these partitions can be at one of the different partitioning levels in the partitioning hierarchy and can have various shapes. Each partition can be called a coding block (CB). For the various example partitioning implementations further described below, each generated CB can be of any allowed size and partitioning level. Such partitions are called coding blocks because they can form units for which some basic encoding/decoding can be performed, and encoding/decoding parameters can be optimized, determined, and written into the encoded video stream. The highest or deepest level in the final partition represents the depth of the tree-based coding block partitioning structure. Coding blocks can be luma coding blocks or chroma coding blocks. The CB tree structure for each color can be called a coding block tree (CBT).

The coding blocks for all color channels can be collectively referred to as coding units (CUs). The hierarchical structure of all color channels can be collectively referred to as coding tree units (CTUs). The partitioning patterns or structures of different color channels within a CTU may be the same or different.

In some implementations, the partitioning tree schemes or structures used for the luma and chroma channels may not need to be the same. In other words, the luma and chroma channels can have separate coding tree structures or patterns. Furthermore, whether the luma and chroma channels use the same or different coding tree structures, and the actual coding tree structure used, can depend on whether the slice being encoded is a P-slice, a B-slice, or an I-slice. For example, for an I-slice, the chroma and luma channels can have separate coding tree structures or coding tree structure patterns, while for P-slices or B-slices, the luma and chroma channels can share the same coding tree scheme. When applying separate coding tree structures or patterns, the luma channel can be divided into CBs using one coding tree structure, and the chroma channel can be divided into chroma CBs using another coding tree structure.

In some exemplary implementations, predetermined partitioning patterns can be applied to the basic blocks. As shown in Figure 9, an exemplary 4-way partitioning tree can start from a first predefined level (e.g., a 64×64 block level or other size, as the basic block size), and the basic blocks can be partitioned hierarchically down to a predefined lowest level (e.g., a 4×4 level). For example, the basic block can have four predefined partitioning options or patterns indicated by 902, 904, 906, and 908, where the partition designated as R is allowed for recursive partitioning, such that the same partitioning option indicated in Figure 9 can be repeated at lower scales until the lowest level (e.g., the 4×4 level). In some implementations, additional restrictions can be applied to the partitioning scheme of Figure 9. In the implementation of Figure 9, rectangular partitions (e.g., 1:2/2:1 rectangular partitions) can be allowed, but these rectangular partitions are not allowed to be recursive, while square partitions are allowed to be recursive. If needed, the recursive partitioning according to Figure 9 will generate a final set of coded blocks. The coded tree depth can be further defined to indicate the partitioning depth from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., 64×64 blocks) can be set to 0, and the coding tree depth increases by 1 after the root block is further divided once according to Figure 9. For the above scheme, the maximum or deepest level from the 64×64 basic block to the 4×4 minimum partition will be 4 (starting from level 0). This partitioning scheme can be applied to one or more color channels. Each color channel can be partitioned independently according to the scheme of Figure 9 (e.g., the partitioning pattern or option in the predefined pattern can be determined independently for each color channel at each hierarchical level). Alternatively, two or more color channels can share the same hierarchical pattern tree of Figure 9 (e.g., the same partitioning pattern or option in the predefined pattern can be selected for two or more color channels at each hierarchical level).

Figure 10 illustrates another exemplary predefined partitioning pattern that allows recursive partitioning to form a partition tree. As shown in Figure 10, an exemplary 10-way partitioning structure or pattern can be predefined. The root block can start from a predefined level (e.g., from a 128×128 level or a 64×64 level basic block). The example partitioning structures in Figure 10 include various 2:1/1:2 and 4:1/1:4 rectangular partitions. The partitioning type of the three sub-partitions 1002, 1004, 1006, and 1008 indicated in the second row of Figure 10 can be referred to as “T-shaped” partitions. The “T-shaped” partitions 1002, 1004, 1006, and 1008 can be referred to as left T-shaped, upper T-shaped, right T-shaped, and lower T-shaped, respectively. In some example implementations, further subdivision of any of the rectangular partitions in Figure 10 is not allowed. The coding tree depth can be further defined to indicate the partitioning depth from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., a 128×128 block) can be set to 0, and the coding tree depth increases by 1 after the root block is further divided once according to Figure 10. In some implementations, only full square partitions in 1010 are allowed to be recursively partitioned to the next level of the partition tree according to the pattern in Figure 10. In other words, recursive partitioning is not allowed for square partitions within T-patterns 1002, 1004, 1006, and 1008. If needed, the recursive partitioning process according to Figure 10 will generate the final set of coding blocks. This scheme can be applied to one or more color channels. In some implementations, more flexibility can be added when using partitions below the 8×8 level. For example, 2×2 chroma inter-frame prediction can be used in some cases.

In some other exemplary implementations of coded block partitioning, quadtree structures can be used to partition basic blocks or intermediate blocks into quadtree partitions. This quadtree partitioning can be applied hierarchically and recursively to any square partition. Whether a basic block, intermediate block, or partition is further quadtree-partitioned can be adapted to various local features of the basic block, intermediate block, or partition. Quadtree partitioning at image boundaries can be further adjusted. For example, implicit quadtree partitioning can be performed at image boundaries such that the block will remain quadtree-partitioned until its size fits the image boundaries.

In some other example implementations, hierarchical binary partitioning from the base block can be used. For such a scheme, the base block or intermediate block can be divided into two partitions. The binary partition can be horizontal or vertical. For example, a horizontal binary partition can divide the base block or intermediate block into equal right and left partitions. Similarly, a vertical binary partition can divide the base block or intermediate block into equal upper and lower partitions. This binary partitioning can be hierarchical and recursive. At each base block or intermediate block, it can be decided whether the binary partitioning scheme should continue, and if so, whether to use a horizontal or vertical binary partition. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or two dimensions). Optionally, further partitioning can stop once a predefined partition level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of the partitions can be no less than 1:4 (or greater than 4:1). Therefore, a vertical strip partition with an aspect ratio of 4:1 can only be further divided into two vertically bifurcated partitions, each with an aspect ratio of 2:1.

In several other examples, as shown in Figure 13, a ternary partitioning scheme can be used to partition basic blocks or any intermediate blocks. The ternary pattern can be implemented vertically as shown at 1302 in Figure 13, or horizontally as shown at 1304 in Figure 13. While the exemplary partition ratio (vertically or horizontally) in Figure 13 is shown as 1:2:1, other ratios can be predefined. In some implementations, two or more different ratios can be predefined. This ternary partitioning scheme can be used to complement quadtree or binary partitioning structures because this ternary partitioning captures objects located at the center of a block within a contiguous partition, whereas quadtrees and binary trees always partition along the block center, thus dividing the object into separate partitions. In some implementations, the width and height of the example ternary partition are always powers of 2 to avoid additional transformations.

The partitioning schemes described above can be combined in any way at different partitioning levels. As an example, the quadtree and binary partitioning schemes described above can be combined to partition the basic block into a quadtree-binary-tree (QTBT) structure. In this scheme, the basic block or intermediate block/partition can be a quadtree partition or a binary partition, depending on a set of predefined conditions, if specified. Figure 14 shows a specific example. In the example of Figure 14, a basic block is first divided into four partitions by a quadtree, as shown in 1402, 1404, 1406, and 1408. Subsequently, each generated partition is either partitioned into four further partitions by the quadtree at the next level (e.g., 1408), or partitioned into two further partitions by a binary partition (e.g., horizontally or vertically, such as 1402 or 1406, both of which are symmetrical), or not partitioned at all (e.g., 1404). For square partitions, binary or quadtree partitions can be recursively allowed, as shown in the overall partitioning pattern example in 1410 and the corresponding tree structure/representation in 1420, where solid lines represent quadtree partitions and dashed lines represent binary tree partitions. Flags can be used for each binary partition node (non-leaf binary partition) to indicate whether the binary partition is horizontal or vertical. For example, as shown in 1420, consistent with the partitioning structure in 1410, flag "0" can indicate a horizontal binary partition, and flag "1" can indicate a vertical binary partition. For quadtree partitions, it is not necessary to indicate the partition type, because quadtree partitions always divide blocks or partitions horizontally and vertically to produce 4 sub-blocks/partitions of equal size. In some implementations, flag "1" can indicate a horizontal binary partition, and flag "0" can indicate a vertical binary partition.

In some example implementations of QTBT, quadtrees and binary partitioning rule sets can be represented by the following predefined parameters and their associated corresponding functions:

- CTU size: The size of the root node of the quadtree (the size of the basic block).

- MinQTSize: The minimum allowed size of a quadtree leaf node

- MaxBTSize: The maximum allowed size of the root node of a binary tree.

- MaxBTDepth: Maximum allowed binary tree depth

- MaxBTSize: The minimum allowed size of a binary leaf node

In some example implementations of the QTBT partitioning structure, the CTU size can be set to 128×128 luma samples and two corresponding 64×64 chroma sample blocks (when considering and using example chroma subsampling), MinQTSize can be set to 16×16, MaxBTSize can be set to 64×64, MinBTSize (for width and height) can be set to 4×4, and MaxBTDepth can be set to 4. Quadtree partitioning can first be applied to the CTU to generate quadtree leaf nodes. The size of the quadtree leaf nodes can range from their minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If a node is 128×128, it will not be partitioned by the binary tree first because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, nodes not exceeding MaxBTSize can be partitioned by the binary tree. In the example of Figure 14, the basic block is 128×128. According to a predefined set of rules, the basic block can only be partitioned by the quadtree. The basic block is partitioned at a depth of 0. Each of the four resulting partitions is 64×64, not exceeding MaxBTSize, and can be further partitioned into quadtrees or binary trees at level 1. This process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further partitioning is not considered. When the width of a binary tree node equals MinBTSize (i.e., 4), further horizontal partitioning is not considered. Similarly, when the height of a binary tree node equals MinBTSize, further vertical partitioning is no longer considered.

In some example implementations, the QTBT scheme described above can be configured to support the flexibility of having the same QTBT structure for luma and chroma, or having separate QTBT structures. For example, for P-slices and B-slices, the luma and chroma CTBs in a CTU can share the same QTBT structure. However, for I-slices, the luma CTB can be partitioned into CBs using a QTBT structure, and the chroma CTB can be partitioned into chroma CBs using a different QTBT structure. This means that CUs can be used to refer to different color channels in an I-slice; for example, an I-slice can consist of coded blocks for the luma component or coded blocks for the two chroma components, and a CU in a P-slice or B-slice can consist of coded blocks for all three color components.

In some other implementations, the QTBT scheme can be supplemented with the ternary partitioning scheme described above. This implementation can be referred to as a multi-type-tree (MTT) structure. For example, in addition to binary partitioning of nodes, one of the ternary partitioning patterns in Figure 13 can be selected. In some implementations, only square nodes can be ternary partitioned. Additional flags can be used to indicate whether the ternary partition is horizontal or vertical.

The design of two- or multi-level trees, such as QTBT implementations and QTBT implementations supplemented by ternary partitioning, is likely primarily for the purpose of reducing complexity. Theoretically, the complexity of traversing a tree is TD , where T represents the number of partition types and D is the depth of the tree. A trade-off can be made by using multiple types (T) while simultaneously reducing the depth (D).

In some implementations, the CB can be further subdivided. For example, for the purpose of intra-frame or inter-frame prediction during encoding and decoding, the CB can be further divided into multiple prediction blocks. In other words, the CB can be further divided into different sub-partitions, in which separate prediction decisions/configurations can be made. In parallel, to depict the level at which the transform or inverse transform of the video data is performed, the CB can be further divided into multiple transform blocks (TBs). The schemes for dividing the CB into PBs and TBs can be the same or different. For example, each partitioning scheme can be performed using its own process, such as based on various features of the video data. In some example implementations, the PB and TB partitioning schemes may be independent. In other example implementations, the PB and TB partitioning schemes and boundaries may be interrelated. For example, in some implementations, TBs can be partitioned after PB partitioning; specifically, each PB is determined after coded block partitioning and can then be further partitioned into one or more TBs. For example, in some implementations, a PB can be divided into one, two, four, or other numbers of TBs.

In some implementations, the luma and chroma channels can be processed differently to divide a basic block into coded blocks and further into prediction blocks and/or transform blocks. For example, in some implementations, for the luma channel, dividing the coded block into prediction blocks and/or transform blocks is permitted, while for the chroma channel, this division may not be permitted. Therefore, in such implementations, the transform and/or prediction of the luma block can be performed only at the coded block level. For another example, the minimum transform block size for the luma and chroma channels can be different; for example, the coded block for the luma channel can be divided into smaller transform and/or prediction blocks than the chroma channel. For yet another example, the maximum depth at which a coded block is divided into transform and/or prediction blocks can differ between the luma and chroma channels; for example, the coded block for the luma channel can be divided into deeper transform and/or prediction blocks than the chroma channel. For a specific example, the luma coding block can be divided into multiple transform blocks of various sizes, which can be represented recursively, with a maximum of two levels of recursion. Transform block shapes such as squares, 2:1/1:2, and 4:1/1:4, and transform block sizes ranging from 4×4 to 64×64 are allowed. However, for the chroma block, only the largest possible transform block specified for the luma block is permitted.

In some example implementations that divide coded blocks into PBs, the depth, shape, and/or other characteristics of the PB divisions may depend on whether the PB is intra-frame coded or inter-frame coded.

Dividing a coded block (or prediction block) into transform blocks can be implemented in various example schemes, including but not limited to recursive or non-recursive quadtree partitioning and predefined pattern partitioning, with additional consideration given to transform blocks at the boundaries of the coded or prediction blocks. Typically, the resulting transform blocks can be at different partitioning levels, can have different sizes, and do not need to be square (e.g., they can be rectangles with some allowed size and aspect ratio). Other examples are described in more detail below with reference to Figures 15, 16, and 17.

However, in some other implementations, CBs obtained via any of the above partitioning schemes can be used as basic or minimal coding blocks for prediction and/or transform. In other words, no further partitioning is performed for inter-frame/intra-frame prediction and/or transform purposes. For example, CBs obtained according to the QTBT scheme described above can be directly used as units for performing prediction. Specifically, this QTBT structure eliminates the concept of multiple partitioning types; that is, it eliminates the separation of CU, PU, and TU, and provides greater flexibility for the CU/CB partitioning shape described above. In this QTBT block structure, CU/CB can have a square or rectangular shape. The leaf nodes of this QTBT are used as units for prediction and transform processing without any further partitioning. This means that in this exemplary QTBT coding block structure, CU, PU, and TU have the same block size.

The various CB partitioning schemes described above, as well as the further partitioning of CB into PBs and/or TBs (excluding PB/TB partitioning), can be combined in any way. The following specific implementations are provided as non-limiting examples.

The following describes a specific example implementation of coded block and transform block partitioning. In such an example implementation, basic blocks can be partitioned into coded blocks using recursive quadtree partitioning or the predefined partitioning patterns described above (e.g., the patterns in Figures 9 and 10). At each level, whether further quadtree partitioning of a particular partition should continue can be determined by the characteristics of the local video data. The resulting CBs can be at various quadtree partitioning levels and have various sizes. The decision on whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode picture regions can be made at the CB level (or CU level, for all three color channels). Each CB can be further partitioned into one, two, four, or other numbers of PBs according to a predefined PB partitioning type. Within a PB, the same prediction processing can be applied, and relevant information can be sent to the decoder based on the PB. After obtaining the remaining blocks by applying the prediction process based on the PB partitioning type, the CB can be partitioned into TBs according to another quadtree structure similar to the CB. In this particular implementation, CBs or TBs can be, but are not limited to, squares. Furthermore, in this particular example, for inter-frame prediction, PBs can be squares or rectangles, while for intra-frame prediction, PBs can only be squares. The encoded block can be divided into, for example, four square TBs. Each TB can be further recursively divided (using quadtrees) into smaller TBs, called a Residual Quadtree (RQT).

Another example implementation for partitioning basic blocks into CBs, PBs, and/or TBs is further described below. For example, instead of using multiple partitioning unit types such as those shown in Figures 9 or 10, a quadtree with an embedded multi-type tree structure (e.g., a QTBT or a QTBT with ternary partitioning as described above) can be used. The separation of CBs, PBs, and TBs (i.e., partitioning CBs into PBs and/or TBs, and PBs into TBs) may be abandoned unless the size of the CBs is too large for the maximum transform length, in which case further partitioning of such CBs may be necessary. This example partitioning scheme can be designed to support greater flexibility in the shape of CB partitions, allowing both prediction and transformation to be performed at the CB level without further partitioning. In such a coding tree structure, CBs can have square or rectangular shapes. Specifically, coding tree blocks (CTBs) can first be partitioned using a quadtree structure. Then, the leaf nodes of the quadtree can be further partitioned using an embedded multi-type tree structure. An example of an embedded multi-type tree structure using binary or ternary partitioning is shown in Figure 11. Specifically, the exemplary multi-type tree structure in Figure 11 includes four partition types, referred to as vertical binary partition (SPLIT_BT_VER) (1102), horizontal binary partition (SPLIT_BT_HOR) (1104), vertical ternary partition (SPLIT_TT_VER) (1106), and horizontal ternary partition (SPLIT_TT_HOR) (1108). CBs then correspond to the leaves of the multi-type tree. In this example implementation, unless the CB is too large for the maximum transform length, the partition is used for both prediction and transform processing without any further division. This means that, in most cases, in a quadtree with an embedded multi-type tree coding block structure, the CB, PB, and TB have the same block size. Exceptions occur when the maximum supported transform length is less than the width or height of the color component of the CB. In some implementations, in addition to binary or ternary partitions, the embedded mode in Figure 11 may also include quadtree partitions.

Figure 12 illustrates a specific example of a quadtree with an embedded multi-type tree-coded block structure for partitioning a basic block (including quadtree, binary tree, and ternary partitioning options). More specifically, Figure 12 shows the basic block 1200 partitioned into four square partitions 1202, 1204, 1206, and 1208 by a quadtree. For each quadtree partition, a decision is made to further partition using the multi-type tree structure and quadtree of Figure 11. In the example of Figure 12, partition 1204 is not further partitioned. Partitions 1202 and 1208 are each partitioned using another quadtree. For partition 1202, the upper-left, upper-right, lower-left, and lower-right partitions of the second-level quadtree partition are partitioned using a quadtree, a horizontal binary partition 1104 of Figure 11, no partitioning, and a horizontal ternary partition 1108 of Figure 11, respectively. Partition 1208 employs another quadtree partitioning method. The upper-left, upper-right, lower-left, and lower-right partitions of the second-level quadtree partitioning are respectively divided into three levels: vertical ternary partitioning 1106 (Figure 11), no partitioning, no partitioning, and horizontal binary partitioning 1104 (Figure 11). The upper-left sub-partition of 1208 is further divided into two sub-partitions based on horizontal binary partitioning 1104 and horizontal ternary partitioning 1108 (Figure 11). Partition 1206 adopts a second-level partitioning pattern following vertical binary partitioning 1102 (Figure 11), dividing into two partitions, and then performing a third-level partitioning according to horizontal ternary partitioning 1108 and vertical binary partitioning 1102 (Figure 11). Based on horizontal binary partitioning 1104 (Figure 11), a fourth-level partitioning is further applied to one of them.

For the specific example above, the maximum luma transform size can be 64×64, and the supported maximum chroma transform size can be different from the luma, for example, 32×32. Even though the example CBs in Figure 12 are not typically further subdivided into smaller PBs and/or TBs, when the width or height of the luma-coded block or chroma-coded block is greater than the maximum transform width or height, the luma-coded block or chroma-coded block can be automatically subdivided in the horizontal and/or vertical directions to meet the transform size constraints in that direction.

In the specific example above used to divide a basic block into CBs, as mentioned above, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P-slices and B-slices, the luma and chroma CTBs in a CTU can share the same coding tree structure. For example, for I-slices, luma and chroma can have separate coding block tree structures. When using separate block tree structures, the chroma CTB can be divided into chroma CBs using one coding tree structure and the chroma CTB into chroma CBs using another coding tree structure. This means that a CU in an I-slice may consist of coding blocks for one luma component or two chroma components, while a CU in a P-slice or B-slice always consists of coding blocks for all three color components, unless the video is monochrome.

When a coded block is further divided into multiple transform blocks, these transform blocks can be ordered in the bitstream in various sequences or scanning methods. The following describes in more detail example implementations of dividing a coded block or prediction block into transform blocks and the encoding order of these transform blocks. In some example implementations, as mentioned above, transform partitioning can support transform blocks of multiple shapes, such as 1:1 (square), 1:2/2:1, and 1:4/4:1, with transform block sizes ranging from 4×4 to 64×64. In some implementations, if the coded block is less than or equal to 64×64, transform block partitioning can be applied only to the luma component, such that for the chroma block, the transform block size is the same as the coded block size. Otherwise, if the coded block width or height is greater than 64, both the luma and chroma coded blocks can be implicitly partitioned into multiples of min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform blocks, respectively.

In some example implementations of transform block partitioning, for intra-frame and inter-frame coding blocks, the coding block can be further divided into multiple transform blocks, with a partitioning depth of up to a predetermined number of levels (e.g., 2 levels). The transform block partitioning depth and size can be correlated. For some example implementations, the mapping from the transform block size at the current depth to the transform block size at the next depth is shown in Table 1 below.

Table 1 Transform Block Size Settings

Based on the example mapping in Table 1, for a 1:1 square block, the next-level transform partition can create four 1:1 square sub-transform blocks. Transform block partitioning can stop at, for example, 4×4. Therefore, the size of a transform block at the current depth of 4×4 corresponds to the same size 4×4 at the next depth. In the examples in Table 1, for a 1:2/2:1 non-square block, the next-level transform partition can create two 1:1 square sub-transform blocks, while for a 1:4/4:1 non-square block, the next-level transform partition can create two 1:2/2:1 sub-transform blocks.

In some example implementations, additional constraints can be imposed on the luma component of an intra-coded block relative to the transform block partitioning. For example, for each level of transform partitioning, all sub-transform blocks can be constrained to have equal sizes. For instance, for a 32×16 coded block, a level 1 transform partition creates two 16×16 sub-transform blocks, and a level 2 transform partition creates eight 8×8 sub-transform blocks. In other words, the second-level partitioning must be applied to all level 1 sub-blocks to maintain equal transform unit sizes. Figure 15 shows an example of partitioning an intra-coded square block into transform blocks according to Table 1, and the encoding order indicated by arrows. Specifically, 1502 shows a square coded block. 1504 shows four equal-sized transform blocks divided into a first-level partition according to Table 1, with the encoding order of these four transform blocks indicated by arrows. 1506 shows 16 equal-sized transform blocks divided into a second-level partition according to Table 1, with the encoding order of these 16 transform blocks indicated by arrows.

In some example implementations, the aforementioned restrictions on intra-frame coding may not apply to the luma component of inter-frame coded blocks. For example, after the first-level transform segmentation, any sub-transform block can be further independently segmented into more than one level. Therefore, the resulting transform blocks can be of the same size or different sizes. Figure 16 shows an example of segmenting an inter-frame coded block into multiple transform blocks with their coding order. In the example of Figure 16, according to Table 1, inter-frame coded block 1602 is divided into two levels of transform blocks. At the first level, the inter-frame coded block is segmented into four transform blocks of equal size. Then, only one (not all) of the four transform blocks is further segmented into four sub-transform blocks, resulting in a total of seven transform blocks of two different sizes, as shown in 1604. An example coding order of these seven transform blocks is shown by the arrow in 1604 of Figure 16.

In some example implementations, additional constraints can be applied to the transform block for one or more chroma components. For instance, the transform block size can be the same as the coding block size, but not smaller than a predefined size, such as 8×8, for one or more chroma components.

In some other example implementations, for coded blocks with a width (W) or height (H) greater than 64, the luma and chroma coded blocks can be implicitly divided into multiples of min(W, 64) × min(H, 64) and min(W, 32) × min(H, 32), respectively. Here, in this disclosure, "min(a, b)" can return the smaller value between a and b.

Figure 17 further illustrates another alternative example scheme for dividing a coded block or prediction block into multiple transform blocks. As shown in Figure 17, instead of using recursive transform partitioning, a set of predefined partitioning types can be applied to the coded block based on its transform type. In the specific example shown in Figure 17, one of six example partitioning types can be applied to divide the coded block into various numbers of transform blocks. This scheme for generating transform block partitions can be applied to either coded blocks or prediction blocks.

More specifically, the partitioning scheme in Figure 17 provides up to six example partitioning types for any given transform type (transform type refers to, for example, the type of the main transform, such as ADST, etc.). In this scheme, a transform partitioning type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In the examples, the transform partitioning type assigned to a coding block or prediction block can be determined based on its transform type. A specific transform partitioning type can correspond to a transform block split size and mode, as shown by the six transform partitioning types in Figure 17. The correspondence between various transform types and various transform partitioning types can be predefined. An example is shown below, where uppercase labels indicate the transform partitioning type that can be assigned to a coding block or prediction block based on rate-distortion cost:

• PARTITION_NONE (No partitioning): Allocate a transformation size equal to the block size.

• PARTITION_SPLIT: Allocate a transform size with a width that is half the width of the block size and a height that is half the height of the block size.

• PARTITION_HORZ (Horizontal Partition): Allocates a transform size with the same width as the block size and a height that is half the height of the block size.

• PARTITION_VERT (Vertical Partition): Allocates a transform size with a width that is half the width of the block size and a height that is the same as the height of the block size.

• PARTITION_HORZ4 (Horizontal 4-partition): Allocate a transform size with the same width as the block size and a height that is one-quarter of the block size height.

• PARTITION_VERT4 (Vertical 4-partition): Allocate a transform size with a width that is one-quarter the width of the block size and a height that is the same as the height of the block size.

In the example above, as shown in Figure 17, the transform partition type includes a uniform transform size for all partitioned transform blocks. This is just an example and is not restrictive. In some other implementations, mixed transform block sizes can be used for partitioned transform blocks within a specific partition type (or pattern).

PBs (or CBs, also called PBs when not further divided into prediction blocks) obtained according to any of the above partitioning schemes can become a single block for encoding via intra-frame or inter-frame prediction. For inter-frame prediction of the current PB, a residual between the current block and the prediction block can be generated, encoded, and included in the encoded bitstream.

Inter-frame prediction can be implemented, for example, in single-reference mode or composite reference mode. In some implementations, a skip flag can be included in the bitstream of the current block (or higher) to indicate whether the current block is inter-coded and not skipped. If the current block is inter-coded, another flag can be included in the bitstream as a signal to indicate whether the current block uses single-reference mode or composite reference mode. For single-reference mode, a single reference block can be used to generate the prediction block for the current block. For composite reference mode, two or more reference blocks can be used, for example, by weighted averaging to generate the prediction block. Composite reference mode can be referred to as more than one reference mode, two reference modes, or multiple reference modes. One or more reference blocks can be identified using one or more reference frame indices and additionally using one or more corresponding motion vectors indicating the positional (e.g., horizontal and vertical pixel) offset between the reference blocks and the current block. For example, the inter-frame prediction block for the current block can be generated from a single reference block identified by a motion vector in a reference frame in single-reference mode, while for composite reference mode, the prediction block can be generated by a weighted average of two reference blocks in two reference frames indicated by two motion vectors. Motion vectors can be encoded and included in the bitstream in various ways.

In some implementations, the encoding or decoding system may have a decoded picture buffer (DPB). Some images/pictures can be stored in the DPB awaiting display (in the decoding system), and some images/pictures in the DPB can be used as reference frames for inter-frame prediction. In some implementations, reference frames in the DPB can be labeled as short-term or long-term references to the current image being encoded or decoded. For example, short-term reference frames may include frames used to perform inter-frame prediction on blocks in the current frame or inter-frame prediction on blocks in a predetermined number (e.g., 2) of subsequent video frames closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that can be used to predict image blocks in more than a predefined number of frames that are far removed from the current frame in decoding order. This information regarding the labeling of short-term and long-term reference frames may be referred to as a Reference Picture Set (RPS), and this information may be added to the header of each frame in the encoded bitstream. Each frame in an encoded video stream can be identified by a Picture Order Counter (POC), which is numbered either in an absolute manner or in order associated with a group of images starting from, for example, frame i, according to the playback sequence.

In some example implementations, one or more reference picture lists containing identifiers of short-term and long-term reference frames used for inter-frame prediction can be formed based on information in the RPS. For example, a single picture reference list, denoted as L0 reference (or reference list 0), can be formed for unidirectional inter-frame prediction, while two picture reference lists can be formed for bidirectional inter-frame prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1) for the two prediction directions. The reference frames included in the L0 and L1 lists can be ordered in various predetermined ways. The lengths of the L0 and L1 lists can be written into the video bitstream. When multiple reference frames used to generate a prediction block by weighted averaging are located on the same side of the block to be predicted in a composite prediction mode, unidirectional inter-frame prediction can be in single-reference mode or composite reference mode. Bidirectional inter-frame prediction can be in composite mode only because bidirectional inter-frame prediction involves at least two reference blocks.

In some implementations, a merge mode (MM) can be implemented for inter-frame prediction. Typically, in a merge mode, the motion vectors in a single-reference prediction or one or more motion vectors in a composite-reference prediction of the current PB can be derived from other motion vectors, rather than being computed and written independently. For example, in a coding system, the current motion vector of the current PB can be reduced to the difference between the current motion vector and one or more other already encoded motion vectors (called reference motion vectors). This difference, rather than the entire current motion vector, can be encoded and included in the bitstream, and this difference can be linked to the reference motion vector. Accordingly, in a decoding system, the motion vector corresponding to the current PB can be derived based on the decoded motion vector difference and the decoded reference motion vector linked to it. As a specific form of inter-frame prediction in general merge mode (MM), this motion vector difference-based inter-frame prediction can be called Merge Mode with Motion Vector Difference (MMVD). Therefore, general MM or specific MMVD can be implemented to improve coding efficiency by leveraging the correlation between motion vectors associated with different PBs. For example, adjacent PBs may have similar motion vectors. For another example, for blocks of similar location/position in space, motion vectors can be correlated in time (between frames).

In some example implementations, the MM flag can be included in the bitstream during encoding processing to indicate whether the current PB is in merge mode. Alternatively or additionally, the MMVD flag can be included and written into the bitstream during encoding to indicate whether the current PB is in MMVD mode. MM and/or MMVD flags or indicators can be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, image level, etc. For a specific example, the current CU may include both the MM flag and the MMVD flag, and the MMVD flag may be written immediately after the skip flag and the MM flag to specify whether MMVD mode is used for the current CU.

In some example implementations of MMVD, a merged candidate list can be formed for the block being predicted, used for motion vector prediction. The merged candidate list can contain a predetermined number (e.g., 2) of MV predictor candidate blocks whose motion vectors can be used to predict the current motion vector. MVD candidate blocks can include blocks selected from neighboring blocks and/or temporary blocks (e.g., blocks at the same position in the previous or next frame of the current frame). These options represent blocks with similar or identical motion vectors to the current block in terms of spatial or temporal location. The size of the MV predictor candidate list can be predetermined. For example, the list might contain two candidate blocks. To be included in the merged candidate list, a candidate block might need to have one (or more) reference frames identical to the current block, must exist (e.g., boundary checks are required when the current block is near the edge of a frame), and must have been encoded during encoding and/or decoded during decoding. In some implementations, if the merge candidate list is available and the above conditions are met, it can first be filled with spatially adjacent blocks (scanned in a specific predefined order), and then temporary blocks can be used to fill the list if space is still available. For example, these adjacent candidate blocks can be selected from the blocks to the left and top of the current block. The merge MV predictor candidate list can be written to the bitstream.

In some implementations, the actual merge candidate for the reference motion vector used to predict the motion vector of the current block can be written into the bitstream. When the merge candidate list contains two candidates, a one-bit flag called the merge candidate flag can be used to indicate the selection of the reference merge candidate. For the current block predicted in composite mode, each of the multiple motion vectors predicted using the MV predictor can be associated with a reference motion vector from the merge candidate list.

In some example implementations of MMVD, after selecting a merging candidate and using it as the basic motion vector predictor for the motion vector to be predicted, the motion vector difference (MVD, or delta MV, representing the difference between the motion vector to be predicted and the reference candidate motion vector) can be computed in the encoding system. This MVD can include information representing the magnitude and direction of the MV difference, which can be written into the bitstream. The magnitude and direction of the motion difference can be written into the bitstream in various ways.

In some example implementations of MMVD, a distance index can be used to specify the magnitude of the motion vector difference and indicate one of a set of predefined offsets representing a predefined motion vector difference relative to a starting point (reference motion vector). The MV offset, indicated by the index, can then be added to the horizontal or vertical component of the starting (reference) motion vector. The offset that the horizontal or vertical component of the reference motion vector should have is determined by exemplary direction information from the MVD. Table 2 specifies examples of predefined relationships between distance indices and predefined offsets.

Table 2 shows an example relationship between distance index and predefined MV offset.

In some exemplary implementations of MMVD, the direction index can be further written into the bitstream and used to represent the direction of the MVD relative to the reference motion vector. In some implementations, the direction can be limited to either a horizontal or vertical direction. Examples of 2-bit direction indices are shown in Table 3. In the examples in Table 3, the interpretation of the MVD can vary depending on the information of the start/reference MVs. For example, when the start/reference MV corresponds to a single prediction block or a double prediction block, where both reference frame lists point to the same side of the current image (i.e., the POC of both reference images is greater than or less than the POC of the current image), the symbols in Table 3 can specify the sign (direction) of the MV offset added to the start/reference MV. When the start/reference MV corresponds to a double prediction block of two reference images on different sides of the current image (i.e., the POC of one reference image is greater than the POC of the current image, and the POC of the other reference image is less than the POC of the current image), and the difference between the reference POC in image reference list 0 and the current frame is greater than the difference between the reference POC in image reference list 1 and the current frame, the symbols in Table 3 can specify the sign of the MV offset added to the reference MV corresponding to the reference image in image reference list 0, and the sign of the offset of the MV corresponding to the reference image in image reference list 1 can have the opposite value (opposite sign for offset). Otherwise, if the difference between the reference POC in image reference list 1 and the current frame is greater than the difference between the reference POC in image reference list 0 and the current frame, then the symbols in Table 3 can specify the sign of the MV offset added to the reference MV associated with image reference list 1, and the sign of the offset of the reference MV associated with image reference list 0 has the opposite value.

Table 3 shows an example implementation of the sign of the MV offset specified by the direction index.

Directional IDX 00 01 10 11 x-axis (horizontal) + - N/A N/A y-axis (perpendicular) N/A N/A + -

In some example implementations, MVD can be scaled based on the difference between POCs in each direction. If the difference between POCs in two lists is the same, no adjustment is needed. Otherwise, if the difference between POCs in reference list 0 is greater than the difference between POCs in reference list 1, the MVD of reference list 1 is adjusted. If the difference between POCs in reference list 1 is greater than the difference between POCs in list 0, the MVD of list 0 can be adjusted in the same way. If the starting MV is a single prediction, the MVD is added to either the available MV or the reference MV.

In some example implementations of MVD encoding and writing for bidirectional composite prediction, symmetric MVD encoding can be implemented in addition to encoding and writing the two MVDs separately, such that only one MVD needs to be written, while the other MVD can be derived from the written MVD. In this implementation, motion information including reference image indices for both list-0 and list-1 is written to the bitstream. However, only the MVD associated with, for example, reference list-0 is written, while the MVD associated with reference list-1 is not written but derived. Specifically, at the slice level, a flag called "mvd_l1_zero_flag" can be included in the bitstream to indicate whether reference list-1 has not been written to the bitstream. If this flag is 1, indicating that reference list-1 is equal to zero (and therefore not written), the bidirectional prediction flag called "BiDirPredFlag" can be set to 0, meaning there is no bidirectional prediction. Otherwise, if `mvd_l1_zero_flag` is zero, and if the most recent reference picture in list-0 and the most recent reference picture in list-1 form a forward and backward reference picture pair or a backward and forward reference picture pair, then `BiDirPredFlag` can be set to 1, and the reference pictures in lists-0 and-1 are both short-term reference pictures. Otherwise, `BiDirPredFlag` is set to 0. `BiDirPredFlag` being 1 indicates that the symmetric mode flag is additionally written to the bitstream. When `BiDirPredFlag` is 1, the decoder can extract the symmetric mode flag from the bitstream. For example, the symmetric mode flag can be written at the CU level (if needed), and it indicates whether the symmetric MVD coding mode is being used for the corresponding CU. When the symmetric mode flag is 1, it indicates that the symmetric MVD encoding mode is used, and only the reference picture indices of lists -0 and -1 (referred to as "mvp_l0_flag" and "mvp_l1_flag") and the MVD associated with list -0 (referred to as "MVD0") are written, and another motion vector difference "MVD1" is derived instead of being written. For example, MVD1 can be derived as -MVD0. Therefore, in the exemplary symmetric MVD mode, only one MVD is written to the bitstream. In some other exemplary implementations of MV prediction, for single-reference mode and composite-reference mode MV prediction, a coordination scheme can be used to implement the general merge mode MMVD and some other types of MV prediction. Various syntax elements can be used to indicate how the MV of the current block is predicted.

For example, for single-reference mode, the following MV prediction patterns can be written into the bitstream:

NEARMV directly uses one of the motion vector predictors (MVPs) in the list indicated by the Dynamic Reference List (DRL) index, without using any MVD.

NEWMV – Uses one of the motion vector predictors (MVPs) in a list written by the DRL index as a reference and applies an increment to the MVP (e.g., using MVD).

GLOBALMV – Uses motion vectors based on frame-level global motion parameters.

Similarly, for a composite reference frame prediction mode that uses two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes can be written into the bitstream:

NEAR_NEARMV – For each of the two MVs to be predicted, use one of the motion vector predictors (MVPs) from the list written by the DRL index, instead of using MVD.

NEAR_NEWMV – To predict the first of two motion vectors, one of the motion vector predictors (MVPs) in the list written by the DRL index is used as the reference MV, without using MVD; to predict the second of two motion vectors, one of the motion vector predictors (MVPs) in the list written by the DRL index is used as the reference MV, combined with the additionally written incremental MV (MVD).

NEW_NEARMV – To predict the second of two motion vectors, one of the motion vector predictors (MVPs) in the list written by the DRL index is used as the reference MV, without using MVD; to predict the first of two motion vectors, one of the motion vector predictors (MVPs) in the list written by the DRL index is used as the reference MV, combined with an additional incremental MV (MVD).

NEW_NEWMV – Uses one of the motion vector predictors (MVPs) in the list written by the DRL index as a reference MV, and combines it with an additional incremental MV to predict each of the two MVs.

GLOBAL_GLOBALMV – Uses the MV for each reference based on frame-level global motion parameters.

Therefore, the term "NEAR" refers to MV prediction using a reference MV without using MVD as the general merging mode, while the term "NEW" refers to MV prediction involving using a reference MV and offsetting it with the written MVD as the MMVD mode. For composite inter-frame prediction, the aforementioned reference base motion vector and motion vector increments can typically be different or independent between the two references, even if they are correlated, and this correlation can be used to reduce the amount of information required to write the two motion vector increments. In this case, joint writing of the two MVDs can be achieved and indicated in the bitstream.

The dynamic reference list (DRL) above can be used to store a set of indexed motion vectors. This set of motion vectors is dynamically stored and considered as candidate motion vector predictors.

In some example implementations, predefined MVD resolutions are allowed. For example, a motion vector precision (or accuracy) of 1/8 pixel may be permitted. The MVDs described above in various MV prediction modes can be constructed and written to the bitstream in various ways. In some implementations, various syntax elements can be used to represent the motion vector differences described above in reference frame list 0 or list 1.

For example, a syntax element called "mv_joint" can specify which components of the motion vector difference it is associated with are non-zero. For MVD, this is a joint write for all non-zero components. For example, mv_joint has the following values.

0 can indicate that there is no non-zero MVD along the horizontal or vertical direction;

1 can represent that non-zero MVD exists only in the horizontal direction;

2 can represent that non-zero MVD exists only in the vertical direction;

3 can indicate that there is a non-zero MVD in both the horizontal and vertical directions.

When the "mv_joint" syntax element of the MVD signal indicates that there are no non-zero MVD components, no further MVD information can be written. If the "mv_joint" syntax indicates that there are one or two non-zero components, additional syntax elements can be written for each non-zero MVD component, as described below.

For example, a syntax element called "mv_sign" can be used to additionally specify whether the corresponding motion vector difference component is positive or negative.

For another example, the syntax element called "mv_class" can be used to specify the class of motion vector differences in a predefined set of classes for the corresponding non-zero MVD component. For instance, predefined classes of motion vector differences can be used to divide the continuous size space of motion vector differences into non-overlapping ranges, each range corresponding to one MVD class. Therefore, the written MVD class indicates the size range of the corresponding MVD component. In the example implementation shown in Table 4, higher classes correspond to motion vector differences with larger size ranges. In Table 4, the symbol (n, m) is used to represent the range of motion vector differences greater than n pixels and less than or equal to m pixels.

Table 4. Categories of Motion Vector Difference Magnitude

In some other examples, a syntax element called "mv_bit" can be further used to specify the integer portion of the offset between the non-zero motion vector difference component and the starting size of the corresponding written MV class. The number of bits required in "mv_bit" for writing the entire range of each MVD class can vary as a function of the MV class. For example, in the implementation in Table 4, MV_CLASS 0 and MV_CLASS 1 might only require a single bit to represent an integer pixel offset of 1 or 2 starting from MVD 0. Each higher MV_CLASS might require progressively increasing by one "mv_bit" bit compared to the previous MV_CLASS.

In some other examples, a syntax element called "mv_fr" can be further used to specify the first two decimal places of the motion vector difference for the corresponding non-zero MVD component, while a syntax element called "mv_hp" can be used to specify the third decimal place (high-resolution bit) of the motion vector difference for the corresponding non-zero MVD component. Two "mv_fr" bits effectively provide 1/4 pixel MVD resolution, while "mv_hp" bits can further provide 1/8 pixel resolution. In some other implementations, multiple "mv_hp" bits can be used to provide a finer MVD pixel resolution than 1/8 pixel. In some example implementations, additional flags can be written to the bitstream at one or more different levels to indicate whether 1/8 pixel or higher MVD resolutions are supported. If the MVD resolution is not applied to a particular coding unit, the syntax elements above for the corresponding unsupported MVD resolution may not be written to the bitstream.

In the example implementation above, fractional resolution may be independent of different categories of MVD. In other words, regardless of the magnitude of the motion vector difference, a predefined number of "mv_fr" and "mv_hp" bits can be used to write fractional MVDs of non-zero MVD components, thus providing similar options for motion vector resolution.

However, in some other example implementations, the resolution of motion vector differences across various MVD size categories can be differentiated. Specifically, for larger MVDs in higher MVD categories, higher resolution MVDs may not provide a statistically significant improvement in compression efficiency. Therefore, for a larger range of MVD sizes corresponding to higher MVD size categories, MVDs can be encoded with decreasing resolutions (integer pixel resolution or fractional pixel resolution). Similarly, for typically large MVD values, MVDs can be encoded with decreasing resolutions (integer pixel resolution or fractional pixel resolution). This MVD resolution, which depends on either the MVD category or the MVD size, is often referred to as adaptive MVD resolution. Adaptive MVD resolution can be implemented in various ways, as described in the following example implementations, to achieve better overall compression efficiency. In particular, since statistical observations show that processing MVD resolutions similar to smaller or lower-level MVDs in an adaptive manner to larger or higher-level MVDs may not significantly increase inter-prediction residual coding efficiency, the number of bits of signaling reduced by focusing on lower-precision MVDs may be greater than the additional number of bits required for inter-frame prediction residuals as a result of such lower-precision MVDs.

In some typical implementations, the pixel resolution or precision of the MVD may or may not increase with the increase of the MVD category. Decreasing the pixel resolution of the MVD corresponds to a coarser MVD (or a larger step size from one MVD level to the next). In some implementations, the correspondence between MVD pixel resolution and MVD category can be specified, predefined, or preconfigured, and therefore does not need to be written into the encoded bitstream.

In some example implementations, the MV categories in Table 4 can each be associated with different MVD pixel resolutions.

In some example implementations, each MVD category may be associated with a single allowed resolution. In other implementations, one or more MVD categories may be associated with two or more optional MVD pixel resolutions. Therefore, the signal in the bitstream for such MVD category is followed by additional signaling to indicate which optional pixel resolution to select for the current MVD component.

In some example implementations, the adaptively allowed MVD pixel resolutions may include, but are not limited to, 1/64 pixel, 1/32 pixel, 1/16 pixel, 1/8 pixel, 1-4 pixels, 1/2 pixel, 1 pixel, 2 pixels, 4 pixels… (in descending order of resolution). Therefore, each ascending MVD category can be associated with one of these resolutions in a non-ascending manner. In some implementations, an MVD category can be associated with two or more resolutions, and a higher resolution may be lower than or equal to a lower resolution of a previous MVD category. For example, if MV_CLASS_3 in Table 4 can be associated with optional 1-pixel and 2-pixel resolutions, then the highest resolution that MV_CLASS_4 in Table 4 can be associated with would be 2 pixels. In some other implementations, the highest allowed resolution of an MV category may be higher than the lowest allowed resolution of a previous (lower) MV category. However, the allowed average resolution of ascending MV categories may simply be non-ascending.

In some implementations, when fractional pixel resolutions higher than 1/8 of a pixel are allowed, the “mv_fr” and “mv_hp” signaling can be extended accordingly to a total of more than 3 fractional digits.

In some example implementations, fractional pixel resolution may be allowed only for MVD classes that are at or below the threshold MVD class. For example, fractional pixel resolution may be allowed only for MVD-CLASS 0, and not for any of the other MV classes in Table 4. Similarly, fractional pixel resolution may be allowed only for MVD classes that are at or below any of the other MV classes in Table 4. For other MVD classes above the threshold MVD class, only integer pixel resolution of the MVD is allowed. In this way, for MVDs written with MVD classes at or above the threshold MVD class, it may not be necessary to write fractional resolution signaling such as "mv-fr" and/or "mv-hp" bits. For MVD classes with a resolution less than 1 pixel, the number of bits in the "mv-bit" signaling can be further reduced. For example, for MV_CLASS_5 in Table 4, the MVD pixel offset range is (32, 64], so 5 bits are needed to write the entire range for a 1-pixel resolution. However, if MV_CLASS_5 is associated with a 2-pixel MVD resolution, the "mv-bit" may need to be 4 bits instead of 5 bits, and neither "mv-fr" nor "mv-hp" needs to be written after "mv_class" is written as "MV_CLASS_5".

In some example implementations, MVDs with integer pixel values below a threshold may only allow fractional pixel resolution. For example, MVDs with values less than 5 pixels may only allow fractional pixel resolution. Corresponding to this example, fractional resolution is allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4, but not for all other MV categories. As another example, MVDs with values less than 7 pixels may only allow fractional pixel resolution. Corresponding to this example, fractional resolution is allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4 (range less than 5 pixels), but not for MV_CLASS_3 and higher (range greater than 5 pixels). For MVDs belonging to MV_CLASS_2 with a pixel range of 5 pixels, fractional pixel resolution may or may not be allowed depending on the "m-bit" value. If the "m-bit" value is written as 1 or 2 (making the integer part of the written MVD 5 or 6, calculated as the start of the pixel range of MV_CLASS_2 with an offset of 1 or 2, represented by "m-bit"), then fractional pixel resolution is allowed. Otherwise, if the "m-bit" value is written as 3 or 4 (making the integer part of the written MVD 7 or 8), then fractional pixel resolution may not be allowed.

In some other implementations, for MV classes equal to or higher than a threshold MV class, only a single MVD value may be allowed. For example, such a threshold MV class could be MV_CLASS2. Therefore, MV_CLASS_2 and above may be allowed only with a single MVD value and no fractional pixel resolution. The single allowed MVD value for these MV classes may be predefined. In some examples, the allowed single value may be the higher end of the respective range for these MV classes in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 may be higher than or equal to the threshold class_2, and the single allowed MVD values for these classes may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively. In other examples, the allowed single value may be the middle value of the respective range for these MV classes in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 can be higher than the category thresholds, and the individual allowed MVD values for these categories can be predefined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Any other value within the range can also be defined as a single allowed resolution for each MVD category.

In the above implementation, when the written "mv_class" is equal to or higher than the predefined MVD level threshold, the "mv_class" signaling alone is sufficient to determine the MVD value. Then, "mv_class" and "mv_sign" are used to determine the size and direction of the MVD.

When MVD is written to only one reference frame (from reference frame list 0 or list 1, but not both), or when written to two reference frames jointly, the accuracy (or resolution) of MVD can depend on the category of the associated motion vector difference in Table 3 and/or the size of MVD.

In some other implementations, the pixel resolution or precision of the MVD may or may not increase with the MVD size. For example, the pixel resolution may depend on the integer part of the MVD size. In some implementations, fractional pixel resolution is allowed only for MVD sizes less than or equal to an amplitude threshold. For the decoder, the integer part of the MVD size can be extracted from the bitstream first. The pixel resolution can then be determined, and it can then be determined whether any fractional MVDs exist in the bitstream and need to be parsed (e.g., if fractional pixel resolution is not allowed for a particular extracted integer MVD size, then the bitstream to be extracted may not contain fractional MVD bits). The example implementations above related to adaptive MVD pixel resolution dependent on MVD category apply to adaptive MVD pixel resolution dependent on MVD size. For a particular example, MVD categories above or containing a size threshold may only be allowed one predefined value.

The various example implementations above are for single-reference mode. These implementations also apply to NEW_NEARMV, NEAR_NEWMV, and/or NEW_NEWMV mode examples in composite prediction under MMVD. These implementations are generally suitable for MVD writes.

Figure 18 illustrates a flowchart 1800 of an example method following the principles described above for implementing adaptive MVD resolution. The example decoding method flow begins at S1801. In S1810, a video stream is received, and it is determined that a motion vector difference (MVD) between a motion vector associated with the inter-frame predicted video block and a reference motion vector is written into the video stream, wherein the reference motion vector corresponds to only one of the reference frames in reference frame list 0 and reference frame list 1, unless the MVD is written jointly for both reference frames. In S1820, an indication of the size range of the MVD within a plurality of predefined motion vector difference size ranges is obtained from the video stream. In S1830, the pixel resolution of the MVD is determined based on the size range. In S1840, additional MVD information in the video stream is identified based on the pixel resolution. In S1850, the additional MVD information is extracted from the video stream. In S1860, the inter-frame predicted video block is decoded based on the pixel resolution, the additional MVD information, the reference motion vector, and the reference frame associated with the motion vector. The example method flow ends at S1899.

Figure 19 illustrates another flowchart 1900 of the example method following the principles described above for implementing adaptive MVD resolution. The example method flow begins at S1901. In S1910, a video stream is received, and it is determined that the motion vector difference (MVD) between the motion vector associated with the inter-frame predicted video block and a reference motion vector is written into the video stream, wherein the reference motion vector corresponds to only one of the reference frames in reference frame list 0 and reference frame list 1, unless the MVD is written jointly for both reference frames. In S1920, the integer portion of the size of the MVD is extracted from the video stream. In S1930, the pixel resolution of the MVD is determined based on the integer portion of the size of the MVD. In S1940, additional MVD information in the video stream is identified based on the pixel resolution. In S1950, the inter-frame predicted video block is decoded based on the pixel resolution, the integer portion of the size of the MVD, the additional MVD information, the reference motion vector, and the reference frame associated with the motion vector. The example method flow ends at S1999.

In the embodiments and implementations of this disclosure, any steps and/or operations may be combined or arranged in any number or order as needed. Two or more steps and/or operations may be performed in parallel. The embodiments and implementations in this disclosure may be used individually or in combination in any order. Furthermore, each method (or embodiment), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments in this disclosure may be applied to luma blocks or chroma blocks. The term block may be interpreted as a prediction block, coding block, or coding unit, i.e., CU. The term block may also be used here to refer to a transform block. In the following, when referring to block size, it may refer to the width or height of the block, or the maximum value of the width and height, or the minimum value of the width and height, or the area size (width * height), or the aspect ratio of the block (width:height, or height:width).

The above-described techniques can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media. For example, Figure 20 illustrates a computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter.

Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpreter codes, microcode, etc.

The instructions can be executed on various types of computers or their components, including personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

The components for the computer system (2000) shown in Figure 20 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of the computer software used to implement embodiments of this disclosure. The configuration of the components should also not be construed as having any dependencies or requirements relating to any one or a combination of components shown in the exemplary embodiments of the computer system (2000).

Computer systems (2000) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not depicted). Human-computer interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images acquired from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), etc.

The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard (2001), mouse (2002), touchpad (2003), touch screen (2010), data glove (not shown), joystick (2005), microphone (2006), scanner (2007), camera (2008).

Computer systems (2000) may also include certain human-machine interface output devices. Such human-machine interface output devices may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell/taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback from touchscreens (2010), data gloves (not shown), or joysticks (2005), but may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers (2009), headphones (not shown)), visual output devices (e.g., screens including CRT screens, LCD screens, plasma screens, OLED screens (2010), each with or without touchscreen input, each with or without tactile feedback—some of which are capable of outputting two-dimensional or more three-dimensional visual outputs via devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

Computer systems (2000) may also include human-accessible storage devices and their associated media: for example, optical media including CD/DVD ROM/RW (2020) having media such as CD/DVD (2021), finger drives (2022), removable hard disk drives or solid-state drives (2023), conventional magnetic media such as magnetic tapes and floppy disks (not shown), devices based on dedicated ROM/ASIC/PLD such as security dongles (not shown), etc.

Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter currently disclosed does not cover transmission media, carrier waves, or other transient signals.

The computer system (2000) may also include interfaces (2054) to one or more communication networks (2055). Networks may be, for example, wireless networks, wired networks, or optical networks. Networks may further be local area networks, wide area networks, metropolitan area networks, vehicle and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial television including CAN buses, etc. Some networks typically require external network interface adapters (e.g., USB ports of the computer system (2000)) to connect to certain general-purpose data ports or peripheral buses (2049); other network interfaces are typically integrated into the core of the computer system (2000) by connecting to the system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system). The computer system (2000) can use any of these networks to communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., CANbus connected to certain CANbus devices), or bidirectional, such as connecting to other computer systems using a local area network (LAN) or wide area network (WAN) digital network. As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

The aforementioned human-machine interface devices, human-machine accessible storage devices, and network interfaces can be attached to the kernel (2040) of a computer system (2000).

The kernel (2040) may include one or more central processing units (CPU) (2041), graphics processing units (GPUs) (2042), dedicated programmable processing units (2043) in the form of field-programmable gate areas (FPGAs), hardware accelerators (2044) for certain tasks, graphics adapters (2050), etc. These devices, along with read-only memory (ROM) (2045), random access memory (2046), and internal mass storage such as internal non-user-accessible hard drives, SSDs, etc. (2047), may be connected via a system bus (2048). In some computer systems, the system bus (2048) may be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly connected to the kernel's system bus (2048) or connected via a peripheral bus (2049) to the kernel's system bus (1848). In one example, a screen (2010) may be connected to a graphics adapter (2050). Peripheral bus architectures include PCI, USB, etc.

The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) can execute certain instructions, which can be combined to form the aforementioned computer code. This computer code can be stored in ROM (2045) or RAM (2046). Transient data can also be stored in RAM (2046), while permanent data can be stored, for example, in internal mass storage (2047). Fast storage and retrieval of any storage device can be achieved using a cache, which can be closely associated with one or more CPUs (2041), GPUs (2042), mass storage (2047), ROM (2045), RAM (2046), etc.

Computer-readable media may have computer code thereon for performing various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

As a non-limiting example, a computer system having an architecture (2000), particularly a kernel (2040), can provide functionality by having one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and some non-transitory memory of the kernel (2040), such as internal kernel mass storage (2047) or ROM (2045). Software implementing various embodiments of this disclosure can be stored in such devices and executed by the kernel (2040). Depending on specific needs, the computer-readable media may include one or more storage devices or chips. The software can cause the kernel (2040), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute a particular process or a particular portion of a particular process described herein, including defining data structures (2046) stored in RAM and modifying such data structures according to a software-defined process. Additionally or alternatively, the computer system may provide functionality through hard-wired or otherwise embodied logic in circuitry (e.g., the accelerator (2044)), which may replace or operate with the software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, logic circuitry embodying the execution, or both. This disclosure includes any suitable combination of hardware and software.

While exemplary embodiments have been described in this disclosure, modifications, substitutions, and various alternative equivalents fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and thus fall within its spirit and scope.

Appendix A: Abbreviations

JEM: Joint Exploration Model

VVC: Next-Generation Video Coding

BMS: Benchmark Set

MV: Motion Vector

HEVC: High-Efficiency Video Coding

SEI: Supplemental Enhancement Information

VUI: Video Availability Information

GOPs: Image Groups

TUs: Transformation Unit

PUs: Prediction Units

CTUs: Coding Tree Units

CTBs: Coded Tree Blocks

PBs: Predicted Blocks

HRD: Assuming a reference decoder

SNR: Signal-to-noise ratio

CPUs: Central Processing Unit

GPUs: Graphics Processing Units

CRT: Cathode Ray Tube

LCD: Liquid Crystal Display

OLED: Organic Light Emitting Diode

CD: CD-ROM

DVD: Digital Video Disc

ROM: Read-Only Memory

RAM: Random Access Memory

ASIC: Application-Specific Integrated Circuit

PLD: Programmable Logic Device

LAN: Local Area Network

GSM: Global System for Mobile Communications

LTE: Long Term Evolution

CANBus: Controller Area Network Bus

USB: Universal Serial Bus

PCI: Peripheral Device Interconnect

FPGA: Field Programmable Gate Domain

SSD: Solid State Drive

IC: Integrated Circuit

HDR: High Dynamic Range

SDR: Standard Dynamic Range

JVET: Joint Video Exploration Team

MPM: Most Likely Pattern

WAIP: Wide-angle Intra-frame Prediction

CU: Encoding Unit

PU: Prediction Unit

TU: Transformation Unit

CTU: Coding Tree Unit

PDPC: Location-Related Prediction Combination

ISP: Intra-frame sub-block division

SPS: Sequence Parameter Settings

PPS: Image Parameter Set

APS: Adaptive Parameter Set

VPS: Video Parameter Set

DPS: Decoding Parameter Set

ALF: Adaptive Loop Filter

SAO: Sampling Adaptive Offset

CC-ALF: Adaptive Loop Filter Across Components

CDEF: Constrained Directional Enhancement Filter

CCSO: Cross-component sample offset

LSO: Local Sampling Offset

LR: Loop Recovery Filter

AV1: Open Media Alliance Video 1

AV2: Open Media Alliance Video 2

MVD: Motion Vector Difference

CfL: Predicting chromaticity from luminance

SDT: Semi-decoupled tree

SDP: Semi-decoupling partitioning

SST: Semi-separated tree

SB: Super Block

IBC (or IntraBC): Intra-block copy

CDF: Cumulative Density Function

SCC: Screen Content Encoding

GBI: Generalized Dual Prediction

BCW: CU-level weighted bidirectional prediction

CIIP: Combined Intra-Inter-Frame Prediction

POC: Image Serial Number

RPS: Reference Image Gallery

DPB: Decode Image Buffer

MMVD: Motion Vector Differential Combining Mode

Claims (14)

1. A method for decoding inter-frame predicted video blocks of a video stream, characterized in that the method comprises: Receive the video stream; The motion vector difference (MVD) between the motion vector associated with the inter-frame predicted video block and the reference motion vector is determined to be written into the video stream, wherein the reference motion vector corresponds to only one of the reference frames in reference frame list 0 and reference frame list 1, unless the MVD is written jointly for both reference frames; Obtaining an indication of the size range of the MVD within a plurality of predefined motion vector difference size ranges from the video stream, the indication of the size range of the MVD includes: extracting a first predefined syntax element from the video stream, the first predefined syntax element indicating the MVD category of the MVD in a predefined MVD category set, the lower MVD category corresponding to a smaller MVD size range; and determining the size range of the MVD based on the MVD category; The pixel resolution of the MVD is determined based on the size range of the MVD, wherein the pixel resolution of the MVD depends on the MVD category in descending order; Additional MVD information in the video stream is identified based on the pixel resolution of the MVD, wherein the additional MVD information indicates the optional pixel resolution selected by the MVD; Extract the additional MVD information from the video stream; and The inter-frame predicted video block is decoded based on the pixel resolution of the MVD, the additional MVD information, the reference motion vector, and the reference frame associated with the motion vector. 2. The method according to claim 1, wherein the pixel resolution of the MVD is 2 ^{^n} pixels, where n is an integer, and the value ranges from -6 to 11, including -6 and 11. 3. The method according to claim 1, wherein determining the pixel resolution of the MVD based on the size range of the MVD comprises: Determine whether the size range of the MVD is higher than a preset MVD range threshold level; When the size range of the MVD is determined to be higher than the preset MVD range threshold level, the pixel resolution of the MVD is determined to be an integer number of pixels; and When it is determined that the size range of the MVD is not higher than the preset MVD range threshold level, the pixel resolution of the MVD is determined to be a fraction of pixels. 4. The method according to claim 3, wherein identifying additional MVD information in the video stream based on the pixel resolution of the MVD includes: The video stream is parsed according to the second predefined syntax element to obtain the integer pixel portion of the MVD; and When it is determined that the pixel resolution of the MVD is a fraction of pixels, the video stream is further parsed according to at least a third predefined syntax element to obtain the fractional pixel portion of the MVD. 5. The method according to claim 3, wherein the preset MVD range threshold level includes the lowest MVD or the second lowest MVD in the predefined MVD category set. 6. The method according to claim 3, wherein each MVD in the predefined MVD category set whose size range is higher than the preset MVD range threshold level is associated with a single allowed integer MVD pixel value. 7. The method according to claim 6, wherein the single allowed integer pixel value includes the pixel value corresponding to the higher value in the corresponding size range. 8. The method according to claim 6, wherein the single allowed integer pixel value includes a pixel value corresponding to the midpoint value in the corresponding size range. 9. The method according to claim 4, wherein determining the pixel resolution of the MVD based on the size range of the MVD comprises: Determine whether the size range of the MVD is lower than, includes, or is higher than a preset MVD threshold value; When the size range of the MVD is determined to be higher than the preset MVD threshold value, the pixel resolution of the MVD is determined to be an integer number of pixels; and When it is determined that the size range of the MVD is not higher than the preset MVD threshold size value, the pixel resolution of the MVD is determined to be a fraction of pixels. 10. The method according to claim 9, further comprising: when determining that the size range of the MVD includes the preset MVD threshold size value: Extract a second predefined syntax element from the video stream, the second predefined syntax element indicating the MVD size offset relative to the starting size of the MVD size range; The integer size of the MVD is obtained based on the size range of the MVD and the size offset of the MVD; When the integer value of the MVD is not higher than the preset MVD threshold value, the pixel resolution of the MVD is determined to be a decimal; and When the integer value of the MVD is higher than the preset MVD threshold value, the pixel resolution of the MVD is determined to be a non-decimal. 11. The method according to claim 10, wherein when the pixel resolution of the MVD is determined to be a decimal, the step of identifying additional MVD information in the video stream based on the pixel resolution of the MVD includes: The video stream is parsed according to a third predefined syntax element to obtain the fractional part of the MVD. 12. The method according to claim 9, wherein the preset MVD threshold value is less than 4 pixels. 13. An electronic device for decoding inter-frame predicted video blocks of a video stream, characterized in that the electronic device includes a memory for storing computer instructions and a processor communicating with the memory, wherein the processor, when executing the computer instructions, is configured to cause the electronic device to perform the method of any one of claims 1-12. 14. A method for processing a video stream, wherein the video stream is decoded based on the method of any one of claims 1 to 12.