CN118694964A - Unequally weighted planar motion vector export - Google Patents

Abstract

The present invention provides an unequally weighted planar motion vector derivation. A method of planar motion vector derivation is provided that can employ unequally weighted combinations of neighboring motion vectors. In some embodiments, motion vector information associated with a bottom right pixel or block adjacent to a current coding unit can be derived based on motion information associated with a top row or top adjacent row of a current coding unit and motion information associated with a left column or left adjacent column of the current coding unit. Weighted or unweighted combinations of such values can be combined in a planar mode prediction model to derive associated motion information for bottom and/or right adjacent pixels or blocks.

Description

Unequally weighted planar motion vector export

This application is a divisional application of the patent application with application number 201980025081.1 (PCT/US2019/027560) filed on October 10, 2020, application date on April 15, 2019, and titled "Unequal Weight Planar Motion Vector Derivation".

Priority claim

This application claims priority under 35 U.S.C. §119(e) to earlier-filed U.S. Provisional Application No. 62/657,831, filed on April 15, 2018, the entire contents of which are hereby incorporated by reference.

Technical Field

The present disclosure relates to the field of video coding, and more particularly to increased coding efficiency and a reduced memory burden associated with a number of stored co-located pictures.

Background Art

The technical improvements of the evolving video coding standards show a trend of improving coding efficiency to achieve higher bit rates, higher resolutions and better video quality. The Joint Video Exploration Team has developed a new video coding scheme called JVET and is developing an updated video coding scheme called Versatile Video Coding (VVC) - the full content of VVC version ⁷ of the draft 2 of the standard entitled Versatile Video Coding (Draft 2) published by JVET on October 1, 2018 is incorporated herein by reference. Similar to other video coding schemes like HEVC (High Efficiency Video Coding), JVET and VVC are both block-based hybrid spatial and temporal prediction coding schemes. However, relative to HEVC, JVET and VVC include many modifications to the bitstream structure, syntax, constraints, and mappings for generating decoded pictures. JVET has been implemented in the Joint Exploration Model (JEM) encoder and decoder, but VVC is not expected to be implemented until early 2020.

Current and anticipated video coding schemes typically utilize simple assumptions of neighboring pixel/block similarity to determine the value of predicted intensity values for neighboring pixels or blocks of pixels. The same process can be implemented for associated motion vectors (MVs). However, this assumption may lead to erroneous results. A system and method for unequal weighted planar motion vector derivation is needed.

Summary of the invention

A system of one or more computers may be configured to perform a specific operation or action by installing software, firmware, hardware, or a combination thereof on the system, which software, firmware, hardware, or a combination thereof causes the system to perform a specific described action in operation. One or more computer programs may be configured to perform a specific operation or action by including instructions that, when executed by a data processing device, cause the device to perform the described action. A general aspect includes: identifying a coding unit having a top adjacent row, a left adjacent column, a bottom adjacent row, and a right adjacent column; determining motion information associated with a lower right adjacent pixel located at the intersection of the bottom adjacent row and the right adjacent column based at least in part on motion information associated with the top row and the left adjacent column; determining motion information associated with the right adjacent column based at least in part on the motion information associated with the lower right adjacent pixel; and encoding the coding unit. Other embodiments of this aspect may include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

Implementations may include one or more of the following features: determining motion information associated with the bottom-neighboring row based at least in part on the motion information associated with the bottom-right neighboring pixel; wherein a planar coding mode is employed; determining a first weight value associated with the top-neighboring row, and determining a second weight value associated with the left-neighboring column, wherein the step of determining the motion information associated with the bottom-right neighboring pixel is based at least in part on a combination of the first weight value and the motion information associated with the top-neighboring row, and a combination of the second weight value and the motion information associated with the left-neighboring column. Implementations of the described technology may further include hardware, methods or processes, or computer software on a computer-accessible medium.

In addition, a general aspect may include a system for video encoding, comprising: storing in a memory an encoding unit having a top adjacent row, a left adjacent column, a bottom adjacent row, and a right adjacent column; determining and storing in the memory motion information associated with a bottom right adjacent pixel located at an intersection of the bottom adjacent row and the right adjacent column based at least in part on motion information associated with the top adjacent row and the left adjacent column; determining and storing in the memory motion information associated with the right adjacent column based at least in part on the motion information associated with the bottom right adjacent pixel; and encoding the encoding unit. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the actions of the method.

Implementations may include one or more of the following features: The system for video encoding further includes: determining and storing in a memory motion information associated with the bottom adjacent row based at least in part on the motion information associated with the bottom right adjacent pixel. The system for video encoding further includes: determining and storing in a memory a first weight value associated with the top adjacent row, and determining and storing in a memory a second weight value associated with the left adjacent column, wherein the step of determining the motion information associated with the bottom right adjacent pixel is based at least in part on a combination of the first weight value and the motion information associated with the top adjacent row, and a combination of the second weight value and the motion information associated with the left adjacent column. Implementations of the described technology may include hardware, methods or processes, or computer software on a computer accessible medium.

BRIEF DESCRIPTION OF THE DRAWINGS

Further details of the invention are explained with the aid of the accompanying drawings, in which:

FIG. 1 depicts the partitioning of a frame into a plurality of coding tree units (CTUs).

2a-2c illustrate an exemplary partitioning of a CTU into coding units (CUs).

FIG. 3 depicts a quadtree plus binary tree (QTBT) representation of the CU partition of FIG. 2 .

FIG4 depicts a simplified block diagram for CU encoding in a JVET or VVC encoder.

FIG5 depicts possible intra prediction modes for the luma component in JVET or VVC.

FIG6 depicts a simplified block diagram for CU encoding in a JVET or VVC decoder.

7a-7b depict graphical representations of the horizontal and vertical prediction calculators, respectively.

FIG. 8 depicts an exemplary implementation of the weight parameter S[n] where the sum of the width and height is 256.

FIG. 9 depicts an exemplary implementation of the weight parameter S[n] where the sum of the width and height is 512.

FIG. 10 depicts a block flow diagram of a system and method for unequal weighted planar motion vector derivation.

11 depicts an embodiment of a computer system adapted and configured to provide variable template sizes for template matching.

FIG. 12 depicts an embodiment of a video encoder/decoder adapted and configured to provide variable template sizes for template matching.

DETAILED DESCRIPTION

FIG. 1 depicts the partitioning of a frame into a plurality of coding tree units (CTUs) 100. A frame may be an image in a video sequence. A frame may include a matrix or a collection of matrices representing intensity measures in an image in terms of pixel values. Thus, a collection of these matrices may generate a video sequence. Pixel values may be defined to represent color and brightness in full-color video encoding, where pixels are divided into three channels. For example, in a YCbCr color space, a pixel may have a brightness value Y representing the intensity of a gray level in an image, and two chrominance values Cb and Cr representing the degree of difference in color from gray to blue and red. In other embodiments, pixel values may be represented by values in different color spaces or models. The resolution of a video may determine the number of pixels in a frame. Higher resolutions may represent more pixels and better image clarity, but may also result in higher bandwidth, storage, and transmission requirements.

JVET can be used to encode and decode frames of a video sequence. JVET is a video coding scheme developed by the Joint Video Exploration Team. Several versions of JVET have been implemented in JEM (Joint Exploration Model) encoders and decoders. Similar to other video coding schemes like HEVC (High Efficiency Video Coding), JVET is a block-based hybrid spatial and temporal prediction coding scheme. During encoding using JVET, a frame is first divided into square blocks called CTU 100, as shown in Figure 1. For example, CTU 100 can be a block of 128×128 pixels.

FIG. 2a depicts an exemplary partitioning of a CTU 100 into CUs 102. Each CTU 100 in a frame may be partitioned into one or more CUs (coding units) 102. As described below, a CU 102 may be used for prediction and transformation. Unlike HEVC, in JVET, a CU 102 may be rectangular or square and may be encoded without further partitioning into prediction units or transform units. A CU 102 may be as large as its root CTU 100, or a smaller subdivision of a root CTU 100 as small as a 4×4 block.

In JVET, CTU 100 may be partitioned into CUs 102 according to a quadtree plus binary tree (QTBT) scheme, where CTU 100 may be recursively partitioned into square blocks according to a quadtree, and then those square blocks may be recursively partitioned horizontally or vertically according to a binary tree. Parameters may be set to control the partitioning according to QTBT, such as CTU size, minimum size of quadtree and binary tree leaf nodes, maximum size of binary tree root node, and maximum depth of the binary tree. In VVC, CTU 100 may also be partitioned into CUs using ternary tree partitioning.

As a non-limiting example, FIG. 2a shows a CTU 100 partitioned into CUs 102, where the solid lines represent quadtree partitioning and the dashed lines represent binary tree partitioning. As shown, the binary tree partitioning allows for horizontal and vertical partitioning to define the structure of the CTU and its subdivision into CUs. FIGS. 2b and 2c depict alternative non-limiting examples of ternary tree partitioning of a CU, where the subdivision of the CU is unequal.

FIG3 depicts a QTBT representation of the partitioning of FIG2. A quadtree root node represents CTU 100, where each child node in the quadtree portion represents one of four square blocks partitioned from a square parent block. The square block represented by the quadtree leaf node can then be partitioned zero or more times using a binary tree, where the quadtree leaf node is the root node of the binary tree. At each level of the binary tree portion, the block can be partitioned vertically or horizontally. A flag set to "0" indicates a horizontal partition of the block, while a flag set to "1" indicates a vertical partition of the block.

After quadtree partitioning and binary tree partitioning, the blocks represented by the leaf nodes of the QTBT represent the final CU 102 to be encoded, for example, using inter-prediction or intra-prediction encoding. For slices or full frames encoded with inter-prediction, different partitioning structures can be used for luma and chroma components. For example, for inter slices, the CU 102 can have coding blocks (CBs) for different color components, such as one luma CB and two chroma CBs. For slices or full frames encoded with intra-prediction, the partitioning structure can be the same for luma and chroma components.

FIG4 depicts a simplified block diagram for CU encoding in a JVET encoder. The main stages of video encoding include segmentation as described above to identify CU 102, followed by encoding CU 102 using prediction at 404 or 406, generating residual CU 410 at 408, transforming at 412, quantizing at 416, and entropy encoding at 420. The encoder and encoding process shown in FIG4 also includes a decoding process described in more detail below.

Given the current CU 102, the encoder can obtain the predicted CU 402 in the spatial domain using intra prediction at 404 or in the temporal domain using inter prediction at 406. The basic idea of predictive coding is to transmit a difference or residual signal between the original signal and the prediction for the original signal. At the receiver side, the original signal can be reconstructed by adding the residual and the prediction, as will be described below. Because the correlation of the difference signal is lower than the original signal, fewer bits are required for its transmission.

A slice coded entirely with an intra-predicted CU, e.g., an entire picture or a portion of a picture, may be an I slice that can be coded without reference to other slices, and thus may be a possible point at which decoding can begin. A slice coded with at least some inter-predicted CUs may be a predicted (P) or bi-predicted (B) slice that can be decoded based on one or more reference pictures. P slices may use intra-prediction and inter-prediction together with previously coded slices. For example, P slices may be compressed further than I slices using inter-prediction, but the coding of previously coded slices is required to encode them. B slices may use intra-prediction or inter-prediction, using interpolated predictions from two different frames, to use data from previous and/or subsequent slices for their coding, thereby improving the accuracy of the motion estimation process. In some cases, P slices and B slices may also or may alternatively be coded using intra-block copying, where data from other parts of the same slice is used.

As will be discussed below, the CU 434 may be intra-predicted or inter-predicted based on a reconstructed CU 102 from a previously encoded CU 102 (eg, a neighboring CU 102 or a CU 102 in a reference picture).

When CU 102 is encoded in the spatial domain using intra prediction at 404, an intra prediction mode may be found that best predicts pixel values of CU 102 based on samples from neighboring CUs 102 in the picture.

When encoding the luma component of a CU, the encoder can generate a list of candidate intra prediction modes. Although HEVC has 35 possible intra prediction modes for the luma component, in JVET, there are 67 possible intra prediction modes for the luma component, and in VVC, there are 85 prediction modes. These modes include planar mode, DC mode, 65 directional modes shown in Figure 5, and 18 wide-angle prediction modes. Planar mode uses a three-dimensional plane of values generated from neighboring pixels, DC mode uses the average value of neighboring pixels, and directional mode uses values copied from neighboring pixels in the direction indicated by the solid line. The wide-angle prediction mode can be used with non-square blocks.

When generating a list of candidate intra prediction modes for the luma component of a CU, the number of candidate modes on the list may depend on the size of the CU. The candidate list may include: a subset of 35 modes of HEVC with the lowest SATD (Sum of Absolute Transform Difference) cost; a new directional mode added for JVET adjacent to the candidate found from the HEVC mode; and a set of six most probable modes (MPMs) from CU 102 identified based on the intra prediction mode for previously coded neighboring blocks and a mode in the default mode list.

When encoding the chrominance components of a CU, a list of candidate intra prediction modes may also be generated. The list of candidate modes may include modes generated from luma samples using cross-component linear model projections, intra prediction modes found for luma CBs, especially co-located positions in chroma blocks, and chroma prediction modes previously found for neighboring blocks. The encoder may find the candidate modes with the lowest rate-distortion cost on the list and use those intra prediction modes when encoding the luma and chroma components of the CU. Syntax may be encoded in the bitstream indicating the intra prediction mode used to encode each CU 102.

After the best intra prediction modes for CU 102 have been selected, the encoder can use those modes to generate the predicted CU 402. When the selected mode is a directional mode, a 4-tap filter can be used to improve directional accuracy. The top or left columns or rows of the prediction block can be adjusted using a boundary prediction filter, such as a 2-tap or 3-tap filter.

The predicted CU 402 may be further smoothed using a position-dependent intra prediction combining (PDPC) process that uses unfiltered samples of neighboring blocks to adjust the predicted CU 402 generated based on filtered samples of neighboring blocks, or an adaptive reference sample smoothing using a 3-tap or 5-tap low-pass filter to process reference samples.

When CU 102 is encoded in the temporal domain using inter prediction at 406, a set of motion vectors (MVs) may be found that point to the sample points in the reference picture that best predict the pixel values of CU 102. Inter prediction exploits temporal redundancy between slices by representing the displacement of blocks of pixels in a slice. The displacement is determined based on the pixel values in the previous or following slices through a process called motion compensation. The motion vectors and associated reference indices representing the displacement of pixels relative to a particular reference picture may be provided to the decoder in the bitstream along with the residual between the original pixel and the motion compensated pixel. The decoder may use the residual and the signaled motion vector and reference indices to reconstruct the pixel block in the reconstructed slice.

In JVET, a motion vector may be stored with an accuracy of 1/16 pixel, and the difference between the motion vector and the predicted motion vector of the CU may be encoded with a quarter-pixel or integer-pixel resolution.

In JVET, motion vectors may be found for multiple sub-CUs within CU 102 using various techniques, such as advanced temporal motion vector prediction (ATMVP), spatio-temporal motion vector prediction (STMVP), affine motion compensated prediction, pattern-matched motion vector derivation (PMMVD), and/or bidirectional optical flow (BIO).

Using ATMVP, the encoder can find a temporal vector pointing to a corresponding block in a reference picture for CU 102. The temporal vector can be found based on the motion vector and reference picture found for a previously encoded neighboring CU 102. Using the reference block pointed to by the temporal vector for the entire CU 102, a motion vector can be found for each sub-CU within CU 102.

STMVP can find the motion vector for the sub-CU by scaling and averaging the motion vectors found for neighboring blocks previously coded using inter-frame prediction, and together find the temporal vector.

A motion vector field may be predicted for each sub-CU in a block based on the two control motion vectors found for the corners of the block using affine motion compensated prediction. For example, a motion vector for a sub-CU may be derived based on the corner motion vectors found for each 4×4 block within CU 102.

PMMVD can use bilateral matching or template matching to find an initial motion vector for the current CU 102. Bilateral matching can look at the current CU 102 and reference blocks in two different reference pictures along the motion trajectory, while template matching can look at the corresponding blocks identified by the template in the current CU 102 and the reference picture. The initial motion vector found for the CU 102 can then be refined for each sub-CU one by one.

BIO can be used when performing inter-frame prediction based on earlier and later reference pictures using bidirectional prediction, and BIO allows finding a motion vector for a sub-CU based on the difference gradient between the two reference pictures.

In some cases, local illumination compensation (LIC) may be used at the CU level to find values for the scaling factor parameters and the offset parameters based on samples neighboring the current CU 102 and corresponding samples neighboring the reference block identified by the candidate motion vector. In JVET, the LIC parameters may vary and be signaled at the CU level.

For some of the above methods, the motion vector found for each sub-CU of the CU can be signaled to the decoder at the CU level. For other methods, such as PMMVD and BIO, motion information is not signaled in the bitstream to save overhead, and the decoder can derive the motion vector through the same process.

After the motion vectors for CU 102 have been found, the encoder may use those motion vectors to generate a predicted CU 402. In some cases, when motion vectors have been found for individual sub-CUs, overlapped block motion compensation (OBMC) may be used when generating the predicted CU 402 by combining those motion vectors with motion vectors previously found for one or more neighboring sub-CUs.

When using bidirectional prediction, JVET can use decoder-side motion vector refinement (DMVR) to find motion vectors. DMVR allows motion vectors to be found based on two motion vectors found for bidirectional prediction using a bidirectional template matching process. In DMVR, a weighted combination of the prediction CU 402 generated using each of the two motion vectors can be found, and the two motion vectors can be refined by replacing them with a new motion vector that best points to the combined prediction CU 402. The two refined motion vectors can be used to generate the final prediction CU 402.

At 408 , once the prediction CU 402 has been found, either at 404 with intra prediction or at 406 with inter prediction, the encoder may subtract the prediction CU 402 from the current CU 102 to find the residual CU 410 , as described above.

The encoder may use one or more transform operations at 412 to transform the residual CU 410 into transform coefficients 414 that express the residual CU 410 in the transform domain, for example, using a discrete cosine block transform (DCT transform) to convert the data into the transform domain. Compared to HEVC, JVET allows more types of transform operations, including DCT-II, DST-VII, DST-VII, DCT-VIII, DST-I, and DCT-V operations. The allowed transform operations may be grouped into subsets, and indications of which subsets and which specific operations in those subsets are used may be signaled by the encoder. In some cases, a large block size transform may be used to zero high-frequency transform coefficients in CUs 102 larger than a certain size so that only low-frequency transform coefficients are maintained for those CUs 102.

In some cases, a mode-dependent non-independent secondary transform (MDNSST) may be applied to the low-frequency transform coefficients 414 after the forward kernel transform. The MDNSST operation may use a Hypercube-Givens transform (HyGT) based on rotated data. When used, an index value identifying a specific MDNSST operation may be signaled by the encoder.

At 416, the encoder may quantize the transform coefficients 414 into quantized transform coefficients 416. The quantization of each coefficient may be calculated by dividing the coefficient value by a quantization step size, which is derived from a quantization parameter (QP). In some embodiments, Qstep is defined as 2 ^(QP-4)/6 . Quantization may aid in data compression because the high-precision transform coefficients 414 may be converted into quantized transform coefficients 416 having a limited number of possible values. Thus, quantization of the transform coefficients may limit the amount of bits generated and transmitted by the transform process. However, although quantization is a lossy operation and the loss of quantization cannot be recovered, the quantization process trades off the quality of the reconstructed sequence against the amount of information required to represent the sequence. For example, a lower QP value may produce a better quality decoded video, although a larger amount of data may be required to represent and transmit. Conversely, a high QP value may produce a lower quality reconstructed video sequence, but with lower data and bandwidth requirements.

JVET can utilize a variance-based adaptive quantization technique, which allows each CU 102 to use a different quantization parameter for its encoding process (rather than using the same frame QP when encoding each CU 102 of a frame). The variance-based adaptive quantization technique adaptively reduces the quantization parameter for certain blocks while increasing the quantization parameter in other blocks. In order to select a specific QP for a CU 102, the variance of the CU is calculated. In short, if the variance of a CU is higher than the average variance of the frame, a higher QP can be set for the CU 102 than the QP of the frame. If a CU 102 exhibits a lower variance than the average variance of the frame, a lower QP can be assigned.

At 420, the encoder can find the final compressed bits 422 by entropy encoding the quantized transform coefficients 418. Entropy coding is intended to eliminate statistical redundancy of the information to be transmitted. In JVET, the quantized transform coefficients 418 can be encoded using CABAC (context adaptive binary arithmetic coding), which uses a probability metric to eliminate statistical redundancy. For CU 102 with non-zero quantized transform coefficients 418, the quantized transform coefficients 418 can be converted into binary. Each bit ("bin") of the binary representation can then be encoded using a context model. CU 102 can be decomposed into three regions, each region having its own set of context models for pixels within the region.

Multiple scanning passes may be performed to encode bins. During the pass for encoding the first three bins (bin0, bin1, and bin2), an index value indicating which context model to use for the bin may be found by finding the sum of the positions of the bin in up to five previously encoded adjacent quantized transform coefficients 418 identified by the template.

The context model can be based on the probability of a binary bit value being "0" or "1". When encoding the value, the probabilities in the context model can be updated based on the actual number of values "0" and "1" encountered. While HEVC uses a fixed table to reinitialize the context model for each new picture, in JVET, the probabilities of the context model for a new inter-prediction picture can be initialized based on the context model developed for a previously coded inter-prediction picture.

The encoder may generate a bitstream containing entropy coded bits 422 of the residual CU 410, prediction information such as a selected intra prediction mode or motion vector, an indicator of how the CU 102 is partitioned from the CTU 100 according to the QTBT structure, and/or other information about the encoded video. The bitstream may be decoded by a decoder as described below.

In addition to using the quantized transform coefficients 418 to find the final compressed bits 422, the encoder can also use the quantized transform coefficients 418 to generate a reconstructed CU 434 by following the same decoding process that the decoder will use to generate the reconstructed CU 434. Thus, once the transform coefficients have been calculated and quantized by the encoder, the quantized transform coefficients 418 can be transmitted to the decoding loop in the encoder. After quantizing the transform coefficients of the CU, the decoding loop allows the encoder to generate the same reconstructed CU 434 as the decoder generated during the decoding process. Therefore, when performing intra-frame prediction or inter-frame prediction on the new CU 102, the encoder can use the same reconstructed CU 434 that the decoder would use for neighboring CUs 102 or reference pictures. The reconstructed CU 102, reconstructed slice, or complete reconstructed frame can serve as a reference for other prediction stages.

At the decoding loop of the encoder (and see below for the same operation in the decoder), to obtain pixel values for the reconstructed image, a dequantization process may be performed. To dequantize a frame, for example, the quantized value of each pixel of the frame is multiplied by a quantization step size, such as described above (Qstep), to obtain reconstructed dequantized transform coefficients 426. For example, in the decoding process shown in FIG. 4, in the encoder, the quantized transform coefficients 418 of the residual CU 410 may be dequantized at 424 to find the dequantized transform coefficients 426. If an MDNSST operation is performed during encoding, the operation may be reversed after dequantization.

At 428, the dequantized transform coefficients 426 may be inversely transformed, for example, by applying a DCT to the values to obtain a reconstructed image, to find a reconstructed residual CU 430. At 432, the reconstructed residual CU 430 may be added to the corresponding prediction CU 402 found at 404 using intra prediction or at 406 using inter prediction to find a reconstructed CU 434.

At 436, one or more filters may be applied to the reconstructed data during the decoding process (in the encoder or, as described below, in the decoder) at the picture level or CU level. For example, the encoder may apply a deblocking filter, a sample adaptive offset (SAO) filter, and/or an adaptive loop filter (ALF). The decoding process of the encoder may implement filters to estimate the optimal filter parameters that can address potential artifacts in the reconstructed image and transmit them to the decoder. Such improvements improve the objective and subjective quality of the reconstructed video. In deblocking filtering, pixels near the sub-CU boundary may be modified, while in SAO, pixels in the CTU 100 may be modified using edge offsets or band offsets. JVET's ALF may use a filter having a circularly symmetric shape for each 2×2 block. Indications of the size and identity of the filter used for each 2×2 block may be signaled.

If the reconstructed pictures are reference pictures, they may be stored in a reference buffer 438 for inter-prediction of future CUs 102 at 406 .

During the above steps, JVET allows the use of content adaptive cropping operations to adjust color values to match between the upper and lower cropping boundaries. The cropping boundaries may change for each slice, and the parameters identifying the boundaries may be signaled in the bitstream.

6 depicts a simplified block diagram for CU encoding in a JVET decoder. The JVET decoder may receive a bitstream containing information about an encoded CU 102. The bitstream may indicate how the CU 102 of a picture is partitioned from the CTU 100 according to the QTBT structure, prediction information for the CU 102 (e.g., intra prediction mode or motion vector), and bits 602 representing an entropy-coded residual CU.

At 604, the decoder may decode the entropy coded bits 602 using the CABAC context model signaled by the encoder in the bitstream. The decoder may use the parameters signaled by the encoder to update the probabilities of the context model in the same way that the probabilities of the context model were updated during encoding.

After inverting the entropy encoding at 604 to find quantized transform coefficients 606, the decoder may dequantize them at 608 to find dequantized transform coefficients 610. If an MDNSST operation was performed during encoding, the operation may be inverted by the decoder after dequantization.

At 612, the dequantized transform coefficients 610 may be inversely transformed to find a reconstructed residual CU 614. At 616, the reconstructed residual CU 614 may be added to a corresponding prediction CU 626 found at 622 using intra prediction or at 624 using inter prediction to find a reconstructed CU 618.

At 620, one or more filters may be applied to the reconstructed data at the picture level or the CU level. For example, the decoder may apply a deblocking filter, a sample adaptive offset (SAO) filter, and/or an adaptive loop filter (ALF). As described above, an in-loop filter located in the decoder loop of the encoder may be used to estimate the optimal filter parameters to improve the objective and subjective quality of the frame. These parameters are transmitted to the decoder to filter the reconstructed frame at 620 to filter the reconstructed frame with the filtered reconstructed frame in the encoder.

After generating the reconstructed pictures by finding the reconstructed CU 618 and applying the signaled filters, the decoder may output the reconstructed pictures as output video 628. If the reconstructed pictures are to be used as reference pictures, they may be stored in a reference buffer 630 for inter prediction of future CUs 102 at 624.

Planar mode is often the most frequently used intra-coding mode in VVC, HEVC and JVET. Figures 7a and 7b show the VVC, HEVC and JVET planar predictor generation process for horizontal prediction calculation (Figure 7a) 700 and vertical prediction calculation (Figure 7b) 710 for a coding unit (block) with height H=8 702 and width W=8 704, where the (0,0) 706 coordinate corresponds to the upper left position within the coding CU.

Planar mode in VVC, HEVC, and JVET (HEVC Planar) generates a first-order approximation of the prediction of the current coding unit (CU) by forming a plane based on the intensity values of neighboring pixels. Due to the raster scan coding order, the reconstructed left column neighboring pixels and the reconstructed top row neighboring pixels can be used for the current CU instead of the right column neighboring pixels and the bottom row neighboring pixels. The plane prediction subprocess of VVC, HEVC, and JVET sets the intensity values of all right column neighboring pixels to the same as the intensity values of the top right neighboring pixels, and the intensity values of all bottom row pixels to the same as the intensity values of the bottom left neighboring pixels.

Once the neighboring pixels surrounding the prediction block are defined, the horizontal and vertical predictors (P _h (x,y) and P _v (x,y) respectively) for each pixel within the CU may be determined according to the following equations:

P _h (x,y)＝(W-1-x)*R(-1,y)+(x+1)*R(W,-1)

P _v (x,y)＝(H-1-y)*R(x,-1)+(y+1)*R(-1,H)

in

R(x,y) represents the intensity value of the reconstructed adjacent pixel at the (x,y) coordinate;

W is the block width; and

H is the block height.

From these values, the final plane predictor P(x,y) is calculated by averaging the horizontal predictor and the vertical predictor, with specific adjustments made when the current CU is non-square according to the following equation:

This planar prediction concept can be applied to determine MVs at a fine granularity. In VVC and JVET, each 4×4 sub-block within a CU may have its own MV. If MV(x,y) is assumed to be the MV of the sub-block containing pixel (x,y) and the planar intra prediction concept described earlier, the reconstructed pixel R(x,y) may be converted to an MV at the sub-block level, MV(x,y), noting in particular that for intra planes, R(x,y) is a 1D intensity, while for inter planes, P(x,y) is a 2D MV. That is, the horizontal and vertical predictors _Ph (x,y) and _Pv (x,y) may be converted to horizontal and vertical MV predictors, _MVh (x,y) and _MVv (x,y), respectively. The final planar MV, MV(x,y), may then be calculated by averaging the horizontal and vertical predictors.

In some implementations, multiple reference slices may be used for temporal prediction. Thus, neighboring subblocks may use difference references for their associated MVs. For simplicity, one reference may be used when combining more than one MV in the case where more than one reference is employed. In some implementations, one possible option for combining multiple references is to select the reference of the neighboring subblock that is closest to the coded picture in terms of POC distance.

Plane derivation of MVs may require MVs of CUs surrounding neighboring sub-blocks. However, in some implementations, it is possible that some neighboring sub-blocks may be considered unavailable for plane derivation because they may not have suitable MVs. By way of non-limiting example, the neighboring sub-blocks may be coded in intra mode, may not use an appropriate reference list, or may not use an appropriate reference slice. In these cases, a default MV or an alternative MV may be used instead of a specific MV for the neighboring sub-block. In this case, an alternative MV may be used, for example, based on the MV of the first available adjacent neighbor. In an alternative implementation in which the neighboring sub-block MV does not use an appropriate reference slice presented, one possible option is to scale the available MV to the desired reference slice using a weighting factor according to a ratio of temporal distances.

The present disclosure presents a system and method for performing the following operations: deriving the MV of the bottom-right neighboring sub-block of the current CU (lifting process), and then using the derived MV of the bottom-right neighboring sub-block together with the MVs of other corner neighboring sub-blocks (e.g., top-right neighboring sub-block, bottom-left neighboring sub-block) to calculate the MVs of the bottom row and right column neighboring sub-blocks.

In some embodiments, the bottom right sub-block MV derivation process may be a weighted average of the top right and bottom left neighboring sub-blocks, as defined in the equation presented below:

In an alternative implementation, a flat plane may be assumed, and based on the MVs of the top-left, top-right, and bottom-left neighboring sub-blocks, the MV may be derived based on the following equation:

MV(W,H)＝MV(W,-1)+MV(-1,H)-MV(-1,-1)

Wherein the position (0,0) represents the upper left sub-block position of the current block, W is the width of the current block and H is the height of the current block. MV(x,y) may thus represent the MV of the reconstructed sub-block containing the pixel at the position (x,y) and the estimated/predicted MV at the sub-block containing the position (x,y).

Yet another non-limiting example may derive the bottom-right sub-block MV based on using the MV of the sub-block at the co-located position containing the pixel (W, H) in the co-located reference (similar to the TMVP derivation).

In the case of deriving the MV of the bottom right neighboring sub-block, MV(W,H), the MV of the bottom row neighboring sub-block, _MVb (x,H) and the MV of the right column neighboring sub-block, _MVr (W,y) can be calculated. If linear interpolation is assumed, the MV can be defined as follows:

However, in alternative embodiments, models other than linear interpolation may be used to correlate the MVs.

Once the motion vectors of neighboring sub-blocks surrounding the current CU are defined, the horizontal and vertical MVs (MV _h (x,y) and MV _v (x,y), respectively) of each sub-block within the CU may be determined according to the following equations:

MV _h (x,y)＝(W-1-x)*MV(-1,y)+(x+1)*MV _r (W,y)

MV _v (x,y)＝(H-1-y)*MV(x,-1)+(y+1)*MV _b (x,H)

where MV _h (x,y) and MV _v (x,y) are scaled versions of the horizontal and vertical MV predictors. However, these factors can be compensated in the final MV predictor calculation step.

In some implementations, the top right and bottom left sub-block positions may be set to MV(W-1,-1) and MV(-1,H-1), respectively. In such implementations, the interpolation for the intermediate predictor may be described, for example, by the following equations:

MV _h (x,y)＝(W-1-x)*MV(-1,y)+(x+1)*MV _r (W-1,y)

MV _v (x,y)＝(H-1-y)*MV(x,-1)+(t+1)*MV _b (x,H-1)

Some implementations may utilize unequal weight combinations for final planar MV derivation. Unequal weights may be employed to exploit differences in the accuracy of input intensities in the final interpolation process. Specifically, larger weights may be applied to sub-block positions that are closer to more reliable neighboring sub-block positions. In VVC and JVET, the processing order follows the raster scan at the CTU level and the z scan for CUs within the CTU. Therefore, the top row and left column neighboring sub-blocks are actually reconstructed sub-blocks and are more reliable than the estimated bottom row and right column neighboring sub-blocks. An example application of using unequal weights employed at the final MV derivation is described in the following equation:

Also, the example of unequal weight assignment shown in the above equation can be generalized to a general equation as shown below:

where A(x,y) and B(x,y) are position-dependent weighting factors for the horizontal and vertical predictors, respectively, c(x,y) is a position-dependent rounding factor, and D(x,y) is a position-dependent scaling factor.

It should be noted that unequal weight assignments may also be used in the horizontal and vertical MV predictor calculation stages, and depending on codec design considerations, the bottom right position adjustment and unequal weight assignment components may be used together or separately. In some implementations, the weighting factors and lifting process may be modified according to picture type (I, P, B, and/or any other known, convenient, and/or desired type), temporal layer, color component (Y, Cb, Cr, and/or any other known, convenient, and/or desired color component).

In some embodiments, a special merge mode with a unique merge candidate indicator can be used to signal the use of unequal weight plane MV derivation. In embodiments where this special merge mode is selected, the merged sub-block MV can be calculated according to the following equation:

The equations associated with the bottom right sub-block position adjustment process and the unequal weight assignment process, respectively, as presented above, involve division operations, which may be costly in terms of computational complexity. However, these division operations can generally be converted into scaling operations to make them more efficient and implementation friendly, as presented in the following equations:

MV(W,H)＝((W*MV(W,-1)+H*MV(-1,H))*S[W+H])>>ShiftDenom

MV(W,H)＝((H*MV(W,-1)+W*MV(-1,H))*S[W+H])>>ShiftDenom

MV _b (x,H)＝(((W-1-x)*MV(-1,H)+(x+1)*MV(W,H))*S[W])>>ShiftDenom

MV _r (W,y)＝(((H-1-y)*MV(W,-1)+(y+1)*MV(W,H))*S[H])>>ShiftDenom

MV(x,y)＝((H*MV _h (x,y)*(y+1)+W*WV _v (x,y)*(x+1))*S[x+y+2] )>>(ShiftDenom+log ₂ W+log ₂ H)

where S[n] is the weight factor for parameter n, and ShiftDenom is the factor for the shift down operation. Specifically, S[n] is the factor is an approximation of , and can be described as:

An example 800 of S[n] where the sum of width and height is 256 and ShiftDenom is 10 is depicted in FIG. 8 , and another example 900 of S[n] where the sum of width and height is 512 and ShiftDenom is 10 is depicted in FIG. 9 .

In the examples presented in Figures 8 and 9, a memory size of 2570 bits (257 entries of 10 bits each for Figure 8) and 5130 bits (513 entries of 10 bits each for Figure 9) is required to fill the weight table. This memory size may be excessive and the memory burden is heavy, and therefore it may be beneficial for efficiency and memory management to reduce this memory requirement. The following are two examples of two possible ways to achieve a reduction in the size and memory burden of S[n].

A non-limiting example of S[n] is presented below, where the sum of width and height is 128 and ShiftDenom is 10.

S[n]＝{341,256,205,171,146,128,114,102,93,85,79,73,68,

64,60,57,54,51,49,47,45,43,41,39,38,37,35,34,33,

32,31,30,29,28,28,27,26,26,25,24,24,23,23,22,22,

21,21,20,20,20,19,19,19,18,18,18,17,17,17,17,16,

16,16,16,15,15,15,15,14,14,14,14,14,13,13,13,13,

13,13,12,12,12,12,12,12,12,12,11,11,11,11,11,11,

11,11,10,10,10,10,10,10,10,10,10,10,9,9,9,9,

9,9,9,9,9,9,9,9,9,8,8,8,8,8,8,8,8}

Another non-limiting example of S[n] is shown below, where the sum of width and height is 128 and ShiftDenom is 9.

S[n]＝{171,128,102,85,73,64,57,51,47,43,39,37,34,32,30,28,27,26,24,23,22,21,20,20, 19,18,18,17,17,16,16,15,15,14,14,13,13,13,12,12,12,12,11,11,11,11,10,10,10, 10,10,9,9,9,9,9,9,9,8,8,8,8,8,8,8,8,7,7,7,7,7,7,7,7, 7,7,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,5,5,5,5,5,5,5,5, 5,5,5,5,5,5,5,5,5,5,5,5,4,4,4,4,4,4,4,4,4,4,4,4,4, 4,4}

In the above example, only 126 of the necessary 129 entries need to be stored because the first two entries (1/0 and 1/1) are not used in the presented system and method. In addition, the third entry representing 1/2 in the above example has values 512 and 256 and can be handled separately during the weight calculation. Accordingly, the weighted average calculation shown in the equation presented above:

MV _b (x,H)＝(((W-1-x)*MV(-1,H)+(x+1)*MV(W,H))*S[W])>>ShiftDenom

MV _r (W,y)＝(((H-1-y)*MV(W,-1)+(y+1)*MV(W,H))*S[H])>>ShiftDenom

can be modified as shown in the following equation:

MV _b (x,H)＝(((W-1-x)*MV(-1,H)+(x+1)*MV(W,H))*S[W-3])>>ShiftDenom

MV _r (W,y)＝(((H-1-y)*MV(W,-1)+(y+1)*MV(W,H))*S[H-3])>>ShiftDenom

However, if A simple shift conversion of the identity does not provide an accurate output, thus resulting in poor coding efficiency. The inefficiency may be due to the conversion process, which may allow errors to accumulate linearly with distance. In some embodiments, this error can be reduced by exploiting the fact that the weights of the horizontal and vertical predictors are complementary in the following equation:

Accordingly, the weighting can be calculated based on the weight of the horizontal or vertical predictor (whichever is more accurate). This can be achieved by introducing the parameters horWeight and verWeight into the following equation,

This yields the following equation:

MV(x,y)＝(H*MV _h (x,y)*horWeight+W*MV _v (x,y)*verWeight)>>(ShiftDenom+log ₂ W+log ₂ H)

Alternatively, the following equation may be employed when using the simplified table as identified above.

horWeight=verWeight=(1＜＜(ShiftDenom-1)), where x=y=0

In some embodiments, the table size used to store the weights of the parameters S[n] can be further reduced because the unequal weight plane MV derivation operates at the sub-block level rather than the pixel level, because the sub-block level can be configured at a coarser granularity (e.g., 4×4 in VVC and JVET).

Therefore, when the sub-block size is N×N, MV(x,y) can be mapped to the sub-block coordinates MV(x/N,y/N). In the case where the size is reduced from W×H to (W/N)×(H/N), the maximum size of the table is correspondingly lower, so a smaller table size can be used. Accordingly, the above equation can be reformulated and presented as:

FIG. 10 depicts a flow chart 1000 of a system and method for unequal weighted planar motion vector derivation. In step 1002, a CU is received, and then in step 1004, motion information associated with the CU is determined. Then in step 1006, motion information associated with a lower right neighboring pixel or block is derived based on the motion information associated with the CU. Then in steps 1008 and 1010, motion information may be derived and/or limited according to the systems and methods described herein based at least in part on the motion information associated with the CU and the derived motion information associated with the lower right neighboring pixel or block. Although FIG. 10 depicts step 1008 before step 1010, in some embodiments, steps 1008 and 1010 may occur in parallel and/or step 1010 may be before step 1008. In step 1012, it is determined whether an unequal weighting technique is used to derive the relevant motion vector. If it is determined in step 1012 that weighting is not employed, the system may continue encoding in step 1014 as described herein. However, if it is determined in step 1012 that the motion information was derived using a weighted combining technique, then the indicator may be set in step 1016 and the system may continue encoding in step 1014 .

The execution of the instruction sequence required to implement the embodiment can be performed by the computer system 1100 shown in Figure 11. In the embodiment, the execution of the instruction sequence is performed by a single computer system 1100. According to other embodiments, two or more computer systems 1100 coupled by a communication link 1115 can coordinate with each other to execute the instruction sequence. Although only a description of one computer system 1100 will be given below, it should be understood that any number of computer systems 1100 can be used to practice the embodiment.

A computer system 1100 according to an embodiment will now be described with reference to Figure 11, which is a block diagram of functional components of computer system 1100. As used herein, the term computer system 1100 is used broadly to describe any computing device that can store and independently run one or more programs.

Each computer system 1100 may include a communication interface 1114 coupled to the bus 1106. The communication interface 1114 provides two-way communication between the computer systems 1100. The communication interface 1114 of each computer system 1100 transmits and receives electrical, electromagnetic, or optical signals, including data streams representing various types of signal information (e.g., instructions, messages, and data). The communication link 1115 links one computer system 1100 to another computer system 1100. For example, the communication link 1115 may be a LAN, in which case the communication interface 1114 may be a LAN card; or the communication link 1115 may be a PSTN, in which case the communication interface 1114 may be an integrated services digital network (ISDN) card or a modem; or the communication link 1115 may be the Internet, in which case the communication interface 1114 may be a dial-up, cable, or wireless modem.

The computer system 1100 may transmit and receive messages, data, and instructions, including programs, i.e., applications, code, through its respective communication links 1115 and communication interfaces 1114. The received program code may be executed by the respective processor 1107 as it is received, and/or stored in the storage device 1110, or in other associated non-volatile storage media, for later execution.

In an embodiment, the computer system 1100 works in conjunction with a data storage system 1131, which is, for example, a data storage system 1131 containing a database 1132 that is easily accessible by the computer system 1100. The computer system 1100 communicates with the data storage system 1131 through a data interface 1133. The data interface 1133 coupled to the bus 1106 transmits and receives electrical, electromagnetic or optical signals, including data streams representing various types of signal information (e.g., instructions, messages, and data). In an embodiment, the functions of the data interface 1133 can be performed by the communication interface 1114.

The computer system 1100 includes a bus 1106 or other communication mechanism for transmitting instructions, messages, and data (collectively referred to as information), and one or more processors 1107 coupled to the bus 1106 for processing information. The computer system 1100 also includes a main memory 1108, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1106 for storing dynamic data and instructions to be executed by the processor 1107. The main memory 1108 may also be used to store temporary data, i.e., variables or other intermediate information during the execution of instructions by the processor 1107.

The computer system 1100 may also include a read only memory (ROM) 1109 or other static storage device coupled to the bus 1106 for storing static data and instructions for the processor 1107. A storage device 1110, such as a magnetic disk or optical disk, may also be provided and coupled to the bus 1106 for storing data and instructions for the processor 1107.

The computer system 1100 may be coupled to a display device 1111, such as but not limited to a cathode ray tube (CRT) or a liquid crystal display (LCD), via the bus 1106 for displaying information to a user. An input device 1112, such as alphanumeric keys and other keys, may be coupled to the bus 1106 for transmitting information and command selections to the processor 1107.

According to one embodiment, each computer system 1100 performs specific operations by executing one or more sequences of one or more instructions contained in the main memory 1108 by its respective processor 1107. Such instructions can be read into the main memory 1108 from another computer-usable medium, such as ROM 1109 or storage device 1110. Execution of the sequences of instructions contained in the main memory 1108 causes the processor 1107 to perform the processes described herein. In alternative embodiments, hard-wired circuits may be used in place of or in combination with software instructions. Therefore, the embodiments are not limited to any specific combination of hardware circuitry and/or software.

As used herein, the term "computer-usable medium" refers to any medium that provides information or can be used by the processor 1107. Such media can take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media, that is, media that can retain information without power, include ROM 1109, CD ROM, magnetic tape, and disk. Volatile media, that is, media that cannot retain information without power, include main memory 1108. Transmission media include coaxial cables, copper wires, and optical fibers, including wires with bus 1106. Transmission media can also take the form of carrier waves (that is, electromagnetic waves that can be modulated in frequency, amplitude, or phase to transmit information signals). In addition, transmission media can take the form of sound waves or light waves (such as those generated during radio wave and infrared data communications).

In the above description, each embodiment has been described with reference to the specific elements of each embodiment. However, it will be apparent that various modifications and changes may be made to it without departing from the broader essence and scope of the embodiment. For example, the reader will understand that the specific ordering and combination of the process actions shown in the process flow charts described herein are merely exemplary, and different or additional process actions or different combinations or orderings of process actions may be used to practice these embodiments. Therefore, the description and drawings should be regarded as exemplary and not restrictive.

It should also be noted that the present invention can be implemented in various computer systems. The various techniques described herein can be implemented in hardware or software or a combination of the two. Preferably, these techniques are implemented in a computer program executed on a programmable computer, each of which includes a processor, a storage medium (including volatile and non-volatile memory and/or storage element) readable by the processor, at least one input device and at least one output device. The data program code input using the input device can be applied to perform the functions described above and generate output information. The output information is applied to one or more output devices. Preferably, each program is implemented in a high-level program programming language or an object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly language or machine language. In any case, the language can be a compiled or interpreted language. Each such computer program is preferably stored on a storage medium or device (e.g., ROM or disk) that can be read by a general or special programmable computer, so as to configure and operate the computer to execute the above-mentioned program when the computer reads the storage medium or device. It can also be considered that the system is implemented as a computer-readable storage medium configured with a computer program, wherein the storage medium so configured enables the computer to operate in a specific predefined manner. Furthermore, the storage element of the exemplary computing application may be a relational or sequential (flat file) type computing database capable of storing data in various combinations and configurations.

FIG. 12 is a high-level view of a source device 1212 and a destination device 1210 that may incorporate features of the systems and devices described herein. As shown in FIG. 12 , an exemplary video encoding system 1210 includes a source device 1212 and a destination device 1214, wherein, in this example, the source device 1212 generates encoded video data. Thus, the source device 1212 may be referred to as a video encoding device. The destination device 1214 may decode the encoded video data generated by the source device 1212. Thus, the destination device 1214 may be referred to as a video decoding device. The source device 1212 and the destination device 1214 may be examples of video encoding devices.

Destination device 1214 may receive the encoded video data from source device 1212 via channel 1216. Channel 1216 may include a medium or device capable of moving the encoded video data from source device 1212 to destination device 1214. In one example, channel 1216 may include a communication medium that enables source device 1212 to transmit the encoded video data directly to destination device 1214 in real-time.

In this example, source device 1212 can modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol), and can transmit the modulated video data to destination device 1214. The communication medium may include a wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a portion of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or other devices that facilitate communication from source device 1212 to destination device 1214. In another example, channel 1216 may correspond to a storage medium that stores the encoded video data generated by source device 1212.

12, source device 1212 includes a video source 1218, a video encoder 1220, and an output interface 1222. In some cases, output interface 1228 may include a modulator/demodulator (modem) and/or a transmitter. In source device 1212, video source 1218 may include a source, such as a video capture device (e.g., a camera), a video archive containing previously captured video data, a video feed interface for receiving video data from a video content provider, and/or a computer graphics system for generating video data, or a combination of these sources.

The video encoder 1220 can encode captured, pre-captured or computer-generated video data. The input image can be received by the video encoder 1220 and stored in the input frame memory 1221. The general processor 1223 can load information from here and perform encoding. The program for driving the general processor can be loaded from a storage device, such as the exemplary memory module depicted in Figure 12. The general processor can use the processing memory 1222 for encoding, and the output of the general processor to the encoded information can be stored in a buffer, such as an output buffer 1226.

The video encoder 1220 may include a resampling module 1225 that may be configured to encode (e.g., encode) video data in a scalable video coding scheme that defines at least one base layer and at least one enhancement layer. As part of the encoding process, the resampling module 1225 may resample at least some of the video data, wherein the resampling may be performed in an adaptive manner using a resampling filter.

The encoded video data, e.g., an encoded bitstream, may be transmitted directly to the destination device 1214 via the output interface 1228 of the source device 1212. In the example of FIG. 12, the destination device 1214 includes an input interface 1238, a video decoder 1230, and a display device 1232. In some cases, the input interface 1228 may include a receiver and/or a modem. The input interface 1238 of the destination device 1214 receives the encoded video data via the channel 1216. The encoded video data may include various syntax elements representing the video data generated by the video encoder 1220. Such syntax elements may be included with the encoded video data transmitted on the communication medium, stored on a storage medium, or stored on a file server.

The encoded video data may also be stored on a storage medium or file server for later access by the destination device 1214 for decoding and/or playback. For example, the encoded bitstream may be temporarily stored in an input buffer 1231 and then loaded into a general purpose processor 1233. A program for driving the general purpose processor may be loaded from a storage device or memory. The general purpose processor may use processing memory 1232 to perform decoding. The video encoder 1230 may also include a resampling module 1235 similar to the resampling module 1225 employed in the video encoder 1220.

12 depicts a resampling module 1235 separate from a general purpose processor 1233, but those skilled in the art will appreciate that the resampling function may be performed by a program executed by a general purpose processor, and that the processing in the video encoder may be performed using one or more processors. The decoded image may be stored in an output frame buffer 1236 and then sent to an input interface 1238.

Display device 1238 can be integrated with destination device 1214 or can be external thereto. In some examples, destination device 1214 can include an integrated display device and can also be configured to interface with an external display device. In other examples, destination device 1214 can be a display device. Typically, display device 1238 displays decoded video data to a user.

The video encoder 1220 and the video decoder 1230 can operate according to a video compression standard. ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) are studying the potential need for standardization of future video coding technology, the compression capability of which significantly exceeds the current high-efficiency video coding HEVC standard (including its current and near-term extensions to screen content coding and high dynamic range coding). The working groups are working together to carry out this exploration activity (called the Joint Video Exploration Team (JVET)) to evaluate the compression technology designs proposed by their experts in this field. The recent development of JVET is described in "Algorithm Description of Joint Exploration Test Model 5 (JEM 5)" by J. Chen, E. Alshina, G. Sullivan, J. Ohm, J. Boyce, JVET-E1001-V2.

Additionally or alternatively, the video encoder 1220 and the video decoder 1230 may operate in accordance with other patents or industry standards that work with the disclosed JVET features. Thus, the other standards are, for example, the ITU-T H.264 standard, otherwise known as MPEG-4, Part 10, Advanced Video Coding (AVC), or extensions of such standards. Thus, although newly developed for JVET, the techniques of the present disclosure are not limited to any particular coding standard or technique. Other examples of video compression standards and techniques include MPEG-2, ITU-T H.263, and proprietary or open source compression formats and related formats.

The video encoder 1220 and the video decoder 1230 can be implemented in hardware, software, firmware, or any combination thereof. For example, the video encoder 1220 and the decoder 1230 can use one or more processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, or any combination thereof. When the video encoder 1220 and the decoder 1230 are partially implemented in software, the device can store instructions for the software in a suitable non-transitory computer-readable storage medium, and can use one or more processors in hardware to execute the instructions to perform the technology of the present disclosure. Each of the video encoder 1220 and the video decoder 1230 can be included in one or more encoders or decoders, and any of the encoders or decoders can be integrated as part of a combined encoder/decoder (CODEC) in the corresponding device.

Aspects of the subject matter described herein may be described in the general context of computer-executable instructions (e.g., program modules) executed by a computer, such as the general-purpose processors 1223 and 1233 described above. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Aspects of the subject matter described herein may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

Examples of memory include random access memory (RAM), read-only memory (ROM), or both. The memory can store instructions, such as source code or binary code, for performing the techniques described above. The memory can also be used to store variables or other intermediate information during the execution of instructions to be executed by a processor such as processors 1223 and 1233.

The storage device may also store instructions, such as source code or binary code, for performing the techniques described above. The storage device may additionally store data used and manipulated by the computer processor. For example, the storage device in the video encoder 1220 or the video decoder 1230 may be a database accessed by the computer system 1223 or 1233. Other examples of storage devices include random access memory (RAM), read-only memory (ROM), hard drive, disk, optical disk, CD-ROM, DVD, flash memory, USB memory card, or any other computer-readable medium.

The memory or storage device may be an example of a non-transitory computer-readable storage medium used by or in conjunction with a video encoder and/or decoder. The non-transitory computer-readable storage medium contains instructions for controlling a computer system that is to be configured to perform the functions described in a particular embodiment. The instructions, when executed by one or more computer processors, may be configured to perform the functions described in a particular embodiment.

Moreover, it is noted that some embodiments have been described as processes that can be depicted as flow charts or block diagrams. Although each may describe the operations as sequential processes, multiple operations in these operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. The process may have additional steps not included in the accompanying drawings.

Specific embodiments may be implemented in a non-transitory computer-readable storage medium for use by or in conjunction with an instruction execution system, device, system, or machine. The computer-readable storage medium contains instructions for controlling a computer system to perform the method described in the specific embodiments. The computer system includes one or more computing devices. The instructions, when executed by one or more computer processors, may be configured to perform the functions described in the specific embodiments.

As used in the specification herein and throughout the claims that follow, "a," "an," and "the" include plural references unless the context clearly indicates otherwise. Also, as used in the specification herein and throughout the claims that follow, the meaning of "in" includes "in" and "on," unless the context clearly indicates otherwise.

Although the exemplary embodiments of the present invention have been described in detail with the above structural features and/or method actions specific language, it is to be understood that those skilled in the art will readily recognize that many additional modifications are possible in the exemplary embodiments without substantially departing from the novel teachings and advantages of the present invention. In addition, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the above-mentioned specific features or actions. Therefore, these and all such modifications are intended to be included within the scope of the present invention as interpreted according to the width and scope of the appended claims.

Claims (4)

1. A method of decoding video included in a bitstream by a decoder, comprising:

(a) Receiving the bitstream, wherein the bitstream comprises at least four slices indicating how to divide a coding tree unit into coding units according to a quadtree plus multi-tree structure, the quadtree plus multi-tree structure allowing dividing a square parent node with quadtree division, the quadtree division dividing the square parent node in half in both horizontal and vertical directions to define leaf nodes shaped as squares, each leaf node having the same size;

(b) Wherein the quadtree plus multi-tree structure allows a first one of the leaf nodes to be partitioned based on a symmetric binary tree partition that partitions the first one of the leaf nodes of the quadtree partition in a horizontal direction or a vertical direction into half, generating two rectangular blocks of the same size as leaf nodes, wherein the two rectangular blocks generated by the symmetric binary tree partition are rectangular coding units, wherein the two rectangular blocks generated by the symmetric binary tree partition are not prediction units, wherein the two rectangular blocks generated by the symmetric binary tree partition are not transform units;

(c) Wherein the quadtree plus multi-tree structure allows a second one of the leaf nodes to be partitioned based on an asymmetric tree partition that partitions the second one of the leaf nodes of the quadtree partition in a horizontal direction or a vertical direction, generating three rectangular blocks as leaf nodes, wherein a size of two rectangular blocks is different from a size of a third one of the three blocks, wherein the three rectangular blocks generated by the asymmetric tree partition are rectangular encoding units, wherein the three rectangular blocks generated by the asymmetric tree partition are not prediction units, wherein the three rectangular blocks generated by the asymmetric tree partition are not transformation units;

(d) Identifying final coding units to be decoded represented by leaf nodes of the quadtree plus multi-tree structure, wherein a plurality of the final coding units are rectangles, wherein each of the rectangular final coding units is a decision point whether to perform inter-picture or intra-picture prediction;

(e) The following two are received: (1) A first motion vector associated with a rectangular coding unit of a B slice of a current frame of the video, wherein one of the rectangular final coding units is the rectangular coding unit included in a bi-directionally predicted slice of the current frame of the video that references a temporally previous reference slice relative to a temporally previous reference frame of the current frame of the rectangular coding unit, and (2) a second motion vector associated with the rectangular coding unit of the B slice of the current frame of the video that references a temporally future reference slice relative to a temporally future frame of the current frame of the rectangular coding unit;

(f) Wherein the rectangular coding unit of the B slices of the current frame of the video has a top adjacent row, a left adjacent column, a bottom adjacent row, and a right adjacent column, each of the top adjacent row, the left adjacent column, the bottom adjacent row, and the right adjacent column being in the current frame;

(g) Applying optical flow between the temporally future reference stripe and the temporally previous reference stripe to perform sample-based motion modification using a corrected motion vector based on at least one of the second motion vector and the first motion vector;

(h) The rectangular encoding unit is decoded based on the sample-based motion modification as a result of applying the optical flow between the temporally preceding reference stripe and the temporally future reference stripe.

2. A method of encoding video included in a bitstream by an encoder, comprising:

(a) Providing the bitstream, wherein the bitstream comprises at least four slices indicating how to divide a coding tree unit into coding units according to a quadtree plus multi-tree structure, the quadtree plus multi-tree structure allowing dividing a square parent node with quadtree division, the quadtree division dividing the square parent node in half in both horizontal and vertical directions to define leaf nodes shaped as squares, each leaf node having the same size;

(b) Wherein the quadtree plus multi-tree structure allows a first one of the leaf nodes to be partitioned based on a symmetric binary tree partition that partitions the first one of the leaf nodes of the quadtree partition in a horizontal direction or a vertical direction into half, generating two rectangular blocks of the same size as leaf nodes, wherein the two rectangular blocks generated by the symmetric binary tree partition are rectangular coding units, wherein the two rectangular blocks generated by the symmetric binary tree partition are not prediction units, wherein the two rectangular blocks generated by the symmetric binary tree partition are not transform units;

(c) Wherein the quadtree plus multi-tree structure allows a second one of the leaf nodes to be partitioned based on an asymmetric tree partition that partitions the second one of the leaf nodes of the quadtree partition in a horizontal direction or a vertical direction, generating three rectangular blocks as leaf nodes, wherein a size of two rectangular blocks is different from a size of a third one of the three blocks, wherein the three rectangular blocks generated by the asymmetric tree partition are rectangular encoding units, wherein the three rectangular blocks generated by the asymmetric tree partition are not prediction units, wherein the three rectangular blocks generated by the asymmetric tree partition are not transformation units;

(d) Wherein the bitstream is configured for identifying final coding units to be decoded represented by leaf nodes of the quadtree plus multi-tree structure, wherein a plurality of the final coding units are rectangles, wherein each of the rectangular final coding units is a decision point whether to perform inter-picture or intra-picture prediction;

(e) The following two are provided: (1) A first motion vector associated with a rectangular coding unit of a B slice of a current frame of the video, wherein one of the rectangular final coding units is the rectangular coding unit included in a bi-directionally predicted slice of the current frame of the video that references a temporally previous reference slice relative to a temporally previous reference frame of the current frame of the rectangular coding unit, and (2) a second motion vector associated with the rectangular coding unit of the B slice of the current frame of the video that references a temporally future reference slice relative to a temporally future frame of the current frame of the rectangular coding unit;

(f) Wherein the rectangular coding unit of the B slices of the current frame of the video has a top adjacent row, a left adjacent column, a bottom adjacent row, and a right adjacent column, each of the top adjacent row, the left adjacent column, the bottom adjacent row, and the right adjacent column being in the current frame;

(g) Wherein the bitstream is configured for applying optical flow between the temporally future reference stripe and the temporally previous reference stripe to perform sample-based motion modification using a corrected motion vector based on at least one of the second motion vector and the first motion vector;

(h) The rectangular encoding unit is encoded based on the sample-based motion modification as a result of applying the optical flow between the temporally preceding reference stripe and the temporally future reference stripe.

3. A non-transitory computer readable storage medium containing instructions for controlling a computer system, which when executed, perform the method of claim 1.

4. A non-transitory computer readable storage medium containing instructions for controlling a computer system, which when executed, perform the method of claim 2.