CN110557639B - Application of interleaved prediction - Google Patents

Abstract

A method of processing a video block, comprising: since the block satisfies a condition, determining to apply interleaving prediction to the block, determining a prediction block based on a first intermediate prediction block and a second intermediate prediction block, and using the prediction block to generate an encoding or encoding of the block Decode representation. The first intermediate prediction block is generated from a first group of sub-blocks obtained by dividing the block according to the first division mode, and the second intermediate prediction block is generated from a second group of sub-blocks obtained by dividing the block according to the second division mode. At least one of the sub-blocks in the second group has a different size than the sub-blocks in the first group.

Description

Application of Interleaving Prediction

CROSS-REFERENCE TO RELATED APPLICATIONS

In accordance with the provisions of the applicable Patent Law and/or the Paris Convention, this application promptly requires (1) the prior Chinese patent application filed under International Patent Application No. PCT/CN2018/089242 on May 31, 2018 and ( 2) The priority and interest of the prior Chinese patent application filed under International Patent Application No. PCT/CN2019/070058 on January 2, 2019, which were subsequently abandoned after filing. The entire disclosures of International Patent Application Nos. PCT/CN2018/089242 and PCT/CN2019/070058 are incorporated herein by reference as part of this disclosure.

technical field

This application document relates to video coding technologies, devices and systems.

Background technique

Motion compensation (MC) is a technique in video processing that, given previous and/or future frames, predicts frames in a video by taking into account the motion of the camera and/or objects in the video. Motion compensation can be used in the encoding of video data to achieve video compression.

SUMMARY OF THE INVENTION

This document discloses methods, systems, and apparatuses related to subblock-based motion prediction in video motion compensation.

In an exemplary aspect, a method of processing a video block is disclosed, the method comprising: since the block satisfies a condition, determining to apply interleaving prediction to the block, determining a prediction block based on a first intermediate prediction block and a second intermediate prediction block, and An encoded or decoded representation of the block is generated using the predicted block. The first intermediate prediction block is generated from a first set of sub-blocks obtained by dividing the block according to the first division mode, and the second intermediate prediction block is generated from a second set of sub-blocks obtained by dividing the block according to the second division mode. At least one sub-block in the second group has a different size than the sub-block in the first group.

In another exemplary aspect, an apparatus includes a processor configured to implement the methods described herein.

In yet another exemplary aspect, the various techniques described herein can be implemented as a computer program product stored on a non-transitory computer readable medium, the computer program product comprising program code for implementing the methods described herein .

In yet another exemplary aspect, a video decoding apparatus may implement the methods described herein.

The details of one or more embodiments are set forth in the accompanying drawings, the description, and the claims.

Description of drawings

FIG. 1 is a schematic diagram illustrating an example of subblock-based prediction.

Figure 2 shows an example of an affine motion field of a block described by two control point motion vectors.

Figure 3 shows an example of an affine motion vector field for each sub-block of a block.

FIG. 4 shows an example of motion vector prediction of block 400 in AF_INTER mode.

FIG. 5A shows an example of a selection order of candidate blocks of a current coding unit (CU).

FIG. 5B shows another example of candidate blocks of the current CU in AF_MERGE mode.

6 shows an example of an optional temporal motion vector prediction (ATMVP) motion prediction process for a CU.

FIG. 7 shows an example of one CU with four sub-blocks and adjacent blocks.

8 is a flowchart of an example method of video processing.

9 shows an example of a functional block diagram of a video encoder or video decoder.

Figure 10 shows an example of bidirectional matching used in the frame rate up-conversion (FRUC) method.

Figure 11 shows an example of template matching used in the FRUC method.

Figure 12 shows an example of unidirectional motion estimation (ME) in the FRUC method.

13 shows an example of interleaving prediction with two partition modes in accordance with the disclosed technique.

14A illustrates an example partitioning pattern in which a block is partitioned into 4x4 sub-blocks in accordance with the disclosed technique.

14B illustrates an example partitioning pattern in which a block is partitioned into 8x8 sub-blocks in accordance with the disclosed technique.

14C illustrates an example partitioning pattern in which a block is partitioned into 4x8 sub-blocks in accordance with the disclosed technique.

14D illustrates an example partitioning pattern in which a block is partitioned into 8x4 sub-blocks in accordance with the disclosed technique.

14E illustrates an example partitioning pattern in which a block is partitioned into non-uniform sub-blocks in accordance with the disclosed technique.

14F illustrates another example partitioning pattern in which a block is partitioned into non-uniform sub-blocks in accordance with the disclosed technique.

14G illustrates yet another example partitioning pattern in which a block is partitioned into non-uniform sub-blocks in accordance with the disclosed technique.

15A is an example flow diagram of a method for improving bandwidth usage and prediction accuracy of a block-based motion prediction video system in accordance with the disclosed techniques.

15B is another example flow diagram of a method for improving bandwidth usage and prediction accuracy of a block-based motion prediction video system in accordance with the disclosed techniques.

16 is a schematic diagram illustrating an example of the architecture of a computer system or other control device that may be used to implement various portions of the disclosed technology.

17 illustrates a block diagram of an example embodiment of a mobile device that may be used to implement portions of the disclosed techniques.

Detailed ways

Global motion compensation is one of the variants of motion compensation techniques in video compression and can be used to predict camera motion. However, moving objects within a frame of a video file are not adequately represented by various implementations of global motion compensation. Local motion estimation, such as block motion compensation, can be used to interpret moving objects within a frame, where the frame is divided into blocks of pixels for performing motion prediction.

Subblock-based prediction, developed based on block motion compensation, was first introduced into the video coding standard through High Efficiency Video Coding (HEVC) Annex I (3D-HEVC).

FIG. 1 is a schematic diagram illustrating an example of a prediction-based sub-block. Using subblock-based prediction, a block 100 such as a coding unit (CU) or prediction unit (PU) is divided into several non-overlapping subblocks 101 . Different sub-blocks may be assigned different motion information, such as reference indices or motion vectors (MV). Motion compensation is then performed separately for each sub-block.

To explore future video coding technologies beyond HEVC, the Video Coding Experts Group (VCEG) and the Moving Picture Experts Group (MPEG) jointly established the Joint Video Exploration Team (JVET) in 2015. JVET took a number of methods and added them to reference software called the Joint Exploration Model (JEM). In JEM, subblock-based prediction is employed in various coding techniques, such as affine prediction, optional temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), bidirectional optical flow (BIO), and Frame Rate Up-Conversion (FRUC), which are discussed in detail below.

Affine prediction

In HEVC, only translational motion models are applied for motion compensated prediction (MCP). However, cameras and objects may have various motions such as zoom in/out, rotation, perspective motion, and/or other irregular motions. On the other hand, JEM applies a simplified affine transform motion-compensated prediction.

Figure 2 shows an example of an affine motion field of block 200 described by two control point motion vectors V ₀ and V ₁ . The motion vector field (MVF) of block 200 can be described by the following equation:

As shown in FIG. 2, (v _0x , v _0y ) is the motion vector of the upper left control point, and (v _1x , v _1y ) is the motion vector of the upper right control point. To simplify motion compensated prediction, subblock-based affine transform prediction can be applied. The sub-block size M×N is derived as follows:

Here, MvPre is the motion vector fractional precision (eg, 1/16 in JEM). (v _2x , v _2y ) is the motion vector of the lower left control point, which is calculated according to equation (1). If desired, M and N can be adjusted down to be divisors of w and h, respectively.

FIG. 3 shows an example of an affine MVF for each sub-block of block 300 . To derive the motion vector for each MxN sub-block, the motion vector for the center sample of each sub-block can be calculated according to equation (1) and rounded to motion vector fractional precision (eg, 1/16 in JEM). A motion compensated interpolation filter can then be applied to generate predictions for each sub-block using the derived motion vectors. After MCP, the high precision motion vector of each sub-block is rounded and saved to the same precision as the normal motion vector.

In JEM, there are two affine motion modes: AF_INTER mode and AF_MERGE mode. For CUs with both width and height greater than 8, AF_INTER mode can be applied. In the bitstream, an affine flag at the CU level is signaled to indicate whether AF_INTER mode is used. In AF_INTER mode, adjacent blocks are used to construct a motion vector pair {(v ₀ ,v ₁ )|v ₀ ={v _A ,v _B ,v _c },v ₁ ={v _D ,v _E }} Candidate list.

FIG. 4 shows an example of motion vector prediction (MVP) of block 400 in AF_INTER mode. As shown in Figure 4, v0 is selected from the motion vectors of sub-blocks A, B or C. The motion vectors of adjacent blocks may be scaled according to the reference list. The motion vector may also be scaled according to the relationship between the picture order count (POC) referenced by neighboring blocks, the POC referenced by the current CU, and the POC of the current CU. _The method for selecting v1 from adjacent subblocks D and E is similar. When the number of candidate lists is less than 2, the list is populated by duplicating motion vector pairs consisting of each AMVP candidate. When the candidate list is larger than 2, the candidates may first be sorted according to adjacent motion vectors (eg, based on the similarity of the two motion vectors in a pair of candidates). In some implementations, the first two candidates are kept. In some embodiments, a rate-distortion (RD) cost check is used to determine which motion vector pair candidate to select as control point motion vector prediction (CPMVP) for the current CU. An index indicating the position of the CPMVP in the candidate list may be signaled in the bitstream. After the CPMVP of the current affine CU is determined, affine motion estimation is applied, and the control point motion vector (CPMV) is found. The difference between CPMV and CPMVP is then signaled in the bitstream.

When a CU is applied in AF_MERGE mode, it takes the first block encoded in affine mode from the valid adjacent reconstructed blocks. FIG. 5A shows an example of the selection order of candidate blocks of the current CU 500 . As shown in FIG. 5A , the selection order may be from left ( 501 ), top ( 502 ), top right ( 503 ), bottom left ( 504 ) to top left ( 505 ) of the current CU 500 . FIG. 5B shows another example of candidate blocks of the current CU 500 in AF_MERGE mode. If the adjacent lower left block 501 is coded in affine mode, as shown in FIG. 5B , the motion vectors v2, v3 and v4 containing the upper left, upper right and lower left corners of the CU of sub-block 501 are derived. The motion vector v0 in the upper left corner of the current CU 500 is calculated based on v2, v3 and v4. The motion vector v1 at the upper right of the current CU can be calculated accordingly.

After calculating the CPMV v0 and v1 of the current CU according to the affine motion model in Equation (1), the MVF of the current CU can be generated. To identify whether the current CU is coded using AF_MERGE mode, an affine flag may be signaled in the bitstream when at least one adjacent block is coded in affine mode.

Optional Temporal Motion Vector Prediction (ATMVP)

In the ATMVP method, the Temporal Motion Vector Prediction (TMVP) method is modified by extracting sets of motion information (including motion vectors and reference indices) from blocks smaller than the current CU.

6 shows an example of an ATMVP motion prediction process for CU 600. The ATMVP method predicts motion vectors for sub-CUs 601 within CU 600 in two steps. The first step is to identify the corresponding block 651 in the reference picture 650 with the temporal vector. The reference picture 650 is also referred to as a motion source picture. The second step is to divide the current CU 600 into sub-CUs 601, and obtain the motion vector and reference index of each sub-CU from the block corresponding to each sub-CU.

In the first step, the reference picture 650 and the corresponding block are determined from the motion information of the spatially neighboring blocks of the current CU 600 . To avoid repeated scan processing of adjacent blocks, the first MERGE candidate in the MERGE candidate list of the current CU 600 is used. The first available motion vector and its associated reference index are set to the temporal vector and the index of the motion source picture. In this way, the corresponding block can be more accurately identified than TMVP, where the corresponding block (sometimes referred to as a collocated block) is always located in the lower right corner or center position relative to the current CU.

In the second step, the corresponding block of the sub-CU 651 is identified by the temporal vector in the motion source picture 650 by adding the temporal vector to the coordinates of the current CU. For each sub-CU, the motion information for the sub-CU is derived using the motion information of its corresponding block (eg, the smallest motion grid covering the center sample). After the motion information of the corresponding NxN block is identified, it is converted into the motion vector and reference index of the current sub-CU in the same way as the TMVP of HEVC, where motion scaling and other procedures are applied. For example, the decoder checks whether the low-latency condition is met (eg, the POC of all reference pictures of the current picture is less than the POC of the current picture), and may use the motion vector MVx (eg, the motion vector corresponding to the reference picture list X) to predict Motion vector MVy for each sub-CU (eg, X equals 0 or 1 and Y equals 1-X).

Space-Time Motion Vector Prediction (STMVP)

In the STMVP method, the motion vectors of sub-CUs are derived recursively in raster scan order. FIG. 7 shows an example of one CU with four sub-blocks and adjacent blocks. Consider an 8x8 CU 700 that includes four 4x4 sub-CUs A (701), B (702), C (703), and D (704). Adjacent 4x4 blocks in the current frame are labeled a (711), b (712), c (713), and d (714).

The motion derivation of sub-CU A begins by identifying its two spatial neighbors. The first neighbor is the NxN block above sub-CU A 701 (block c 713). If this block c (713) is not available or intra-coded, then check the other NxN blocks above sub-CU A (701) (from left to right, starting at block c 713). The second neighbor is a block to the left of sub-CU A 701 (block b 712). If block b (712) is not available or is intra-coded, other blocks to the left of sub-CU A 701 are checked (from top to bottom, starting at block b 712). The motion information obtained from neighboring blocks for each list is scaled to the first reference frame of the given list. Next, the temporal motion vector prediction (TMVP) of the sub-block A701 is derived according to the same procedure as the TMVP specified in HEVC. The motion information for the collocated block at block D 704 is extracted and scaled accordingly. Finally, after retrieving and scaling the motion information, all available motion vectors are averaged separately for each reference list. The average motion vector is designated as the motion vector of the current sub-CU.

Frame Rate Up Conversion (FRUC)

For a CU, the FRUC flag may be signaled when its MERGE flag is true. When the FRUC flag is false, the MERGE index can be signaled and the regular MERGE mode used. When the FRUC flag is true, another FRUC mode flag may be signaled to indicate which method (eg, bidirectional matching or template matching) will be used to derive the motion information for the block.

At the encoder side, the decision whether to use FRUCMERGE mode for the CU is based on the RD cost selection made on the normal MERGE candidates. For example, multiple matching modes (eg, bidirectional matching and template matching) of the CU are checked by using RD cost selection. The mode that results in the lowest cost is further compared with other CU modes. If the FRUC match mode is the most efficient mode, then for the CU, the FRUC flag is set to true and the associated match mode is used.

In general, the motion derivation process in FRUC MERGE mode has two steps: first, a CU-level motion search is performed, and then a sub-CU-level motion refinement is performed. At the CU level, based on bidirectional matching or template matching, the initial motion vector for the entire CU is derived. First, a list of MV candidates is generated, and the candidate resulting in the lowest matching cost is selected as the starting point for further CU-level refinement. A local search based on bidirectional matching or template matching is then performed near the starting point. The MV result with the minimum matching cost is taken as the MV value of the entire CU. Then, starting from the derived CU motion vector, the motion information is further refined at the sub-CU level.

For example, the following derivation process is performed for WxH CU motion information derivation. In the first stage, the MV of the entire W×H CU is derived. In the second stage, the CU is further divided into MxM sub-CUs. The value of M is calculated according to (13), and D is the pre-defined division depth, which is set to 3 by default in JEM. The MV value of each sub-CU is then derived.

Figure 10 shows an example of bidirectional matching used in the frame rate up-conversion (FRUC) method. Bidirectional matching is used to obtain motion information for the current CU by finding the closest match between two blocks along the motion trajectory of the current CU (1000) in two different reference pictures (1010, 1011). The temporal distance between the motion vectors MV0 (1001) and MV1 (1002) pointing to the two reference blocks and the current picture and the two reference pictures (eg, TD0 (1003) and TD1 (1004)) under the continuous motion trajectory assumption proportional. In some embodiments, bidirectional matching becomes mirror-based bidirectional MV when the current picture 1000 is temporarily located between two reference pictures (1010, 1011) and the temporal distance of the current picture to both reference pictures is the same.

Figure 11 shows an example of template matching used in the FRUC method. Template matching may be used to find the closest match between a template in the current picture (eg, the top and/or left neighboring blocks of the current CU) and a block in the reference picture 1110 (eg, the same size as the template) Obtain the motion information of the current CU 1100. In addition to the FRUC MERGE pattern described above, template matching can also be applied to the AMVP pattern. In both JEM and HEVC, AMVP has two candidates. Through the template matching method, new candidates can be derived. If the newly derived candidate via template matching is different from the first existing AMVP candidate, insert it at the very beginning of the AMVP candidate list, and then set the list size to 2 (e.g. by removing the second existing AMVP candidate) candidate). When applied to AMVP mode, only CU-level search is applied.

The MV candidates set at the CU level may include the following: (1) the original AMVP candidates, if the current CU is in AMVP mode, (2) all MERGE candidates, (3) several MVs in the interpolated MV field (described later), and the top and the left adjacent motion vector.

When using bidirectional matching, each valid MV of a MERGE candidate can be used as input to generate MV pairs that are assumed to be bidirectional matching. For example, a valid MV for a MERGE candidate at reference list A is (MVa, ref _a ). The reference picture ref _b of its paired bidirectional MV is then found in another reference list B such that ref _a and ref _b are temporally on different sides of the current picture. If reference ref _b in reference list B is not available, then reference ref _b is determined to be _a different reference than reference ref a, and its temporal distance to the current picture is the smallest distance in list B. After the reference ref _b is determined, MVb is derived by scaling MVa based on the temporal distance between the current picture and the reference ref _a and the reference ref _b .

In some implementations, four MVs from the interpolated MV field may also be added to the CU-level candidate list. More specifically, the interpolated MVs at positions (0, 0), (W/2, 0), (0, H/2) and (W/2, H/2) of the current CU are added. When applying FRUC in AMVP mode, the original AMVP candidates are also added to the CU-level MV candidate set. In some implementations, at the CU level, 15 MVs for AMVP CUs and 13 MVs for MERGE CUs may be added to the candidate list.

MV candidates set at the sub-CU level include MVs determined from CU-level searches, (2) top, left, top left and top right adjacent MVs, (3) scaled versions of MVs collocated in reference pictures, (4) One or more ATMVP candidates (eg, up to four) and (5) one or more STMVP candidates (eg, up to four). The scaled MV from the reference picture is derived as follows. The reference pictures in both lists are traversed. The MV at the collocated position of the sub-CU in the reference picture is scaled to be the reference of the starting CU-level MV. ATMVP and STMVP candidates can be the top four. At the sub-CU level, one or more MVs (eg, up to 17) are added to the candidate list.

Generation of Interpolated MV Fields

Before encoding the frame, an interpolated motion field for the entire picture is generated based on unidirectional ME. This motion field can then be used as an MV candidate at the CU level or sub-CU level.

In some embodiments, the motion field of each reference picture in the two reference lists is traversed on a 4x4 block level. FIG. 12 shows an example of unidirectional motion estimation (ME) 1200 in the FRUC method. For each 4x4 block, if the motion associated with the block passes through a 4x4 block in the current picture and the block is not assigned any interpolated motion, the motion of the reference block is scaled according to the temporal distances TD0 and TD1 to The current picture (in the same way as the MV scaling of TMVP in HEVC), and the scaling motion is assigned to the block in the current frame. If no scaled MV is assigned to a 4x4 block, the motion of the block is marked as unavailable in the interpolated motion field.

Interpolation and matching costs

Motion compensated interpolation is required when motion vectors point to fractional sample locations. To reduce complexity, bilinear interpolation is used instead of regular 8-tap HEVC interpolation for both bidirectional matching and template matching.

The calculation of the matching cost is a little different at different steps. When selecting candidates from the CU-level candidate set, the matching cost can be the sum-difference (SAD) of bidirectional matching or template matching. After determining the starting MV, the matching cost C of the bidirectional matching search at the sub-CU level is calculated as follows:

Here, w is a weight coefficient. In some embodiments, w may be empirically set to 4. MV and ^MVs indicate the current MV and the starting MV, respectively. It is still possible to use SAD as a matching cost for pattern matching to search at the sub-CU level.

In FRUC mode, the MV is derived by using only luma (luma) samples. The derived motion will be used for luma (luma) and chroma (chroma) for MC inter prediction. After the MV is determined, a final MC is performed using an 8-tap (8-taps) interpolation filter for luma and a 4-tap (4-taps) interpolation filter for chroma.

MV refinement is a pattern-based MV search, with either bidirectional matching cost or template matching cost as the criterion. In JEM, two search modes are supported—unrestricted center-biased diamond search (UCDBS) and adaptive cross search, with MV refinement at CU level and sub-CU level, respectively. For both CU-level and sub-CU-level MV refinements, the MV is directly searched at one-quarter luma sample MV precision, followed by eighth luma sample MV refinement. The search range of MV refinement for CU and sub-CU steps is set to 8 luma samples.

In bidirectional matching MERGE mode, bidirectional prediction is applied because the motion information of the CU is derived based on the closest match between two blocks along the current CU motion trajectory in two different reference pictures. In template matching MERGE mode, the encoder can choose from list 0 unidirectional prediction, list 1 unidirectional prediction, or bidirectional prediction for the CU. The selection can be based on template matching costs as follows:

If costBi<=factor*min(cost0, cost1)

then use bidirectional prediction;

else if cost0 <= cost1

then use the one-way prediction in list 0;

otherwise,

Use the one-way prediction in Listing 1;

Here, cost0 is the SAD matched by the list 0 template, cost1 is the SAD matched by the list 2 template, and costBi is the SAD matched by the bidirectional template. For example, when the value of factor is equal to 1.25, it means that the selection process is shifted towards bidirectional prediction. Inter prediction direction selection can be applied to CU-level template matching processing.

The sub-block-based prediction techniques discussed above can be used to obtain more accurate motion information for each sub-block when the sub-block size is smaller. However, smaller sub-blocks impose higher bandwidth requirements in motion compensation. On the other hand, for smaller sub-blocks, the derived motion information may be inaccurate, especially when there is some noise in the block. Therefore, having a fixed sub-block size within a block may be sub-optimal.

Described herein are techniques that may be used in various embodiments to use non-uniform and/or variable sub-block sizes to address bandwidth and precision issues introduced by fixed sub-block sizes. These techniques (also known as interleaving prediction) use different methods of dividing blocks in order to obtain motion information more reliably without increasing bandwidth consumption.

Using interleaved prediction techniques, the block is divided into sub-blocks with one or more division patterns. The partition mode indicates the method of dividing the block into sub-blocks, including the size of the sub-block and the location of the sub-block. For each partition mode, a corresponding prediction block may be generated by deriving motion information of each sub-block based on the partition mode. Therefore, in some embodiments, multiple prediction blocks may be generated by multiple partition modes even for one prediction direction. In some embodiments, only one partition mode may be applied for each prediction direction.

13 shows an example of interleaving prediction with two partition modes in accordance with the disclosed technique. The current block 1300 may be divided into multiple modes. For example, as shown in FIG. 13, the current block is divided into mode 0 (1301) and mode 1 (1302). Two prediction blocks P ₀ (1303) and P ₁ (1304) are generated. By calculating the weighted sum of P ₀ (1303) and P ₁ (1304), the final prediction block P (1305) of the current block 1300 can be generated.

In general, given X partition modes, X prediction blocks (denoted as P ₀ , P ₁ , . . . , P _X-1 ) of the current block can be generated by sub-block-based prediction in X partition modes. The final prediction for the current block (denoted as P) can be generated as:

Here, (x, y) are the coordinates of the pixels in the block, and _wi (x, y) are the weight coefficients of P _i . By way of example rather than limitation, the weights can be expressed as:

N is a non-negative value. Alternatively, the displacement operation in equation (8) can also be expressed as:

The sum of the weights is a power of 2, and the weighted sum P can be computed more efficiently by performing a shift operation instead of a floating point division.

Partition patterns can have different sub-block shapes, sizes or positions. In some embodiments, the partitioning pattern may include irregular sub-block sizes. Figures 14A-14G show examples of several partition patterns for 16x16 blocks. In Figure 14A, the block is divided into 4x4 sub-blocks according to the disclosed technique. This mode is also used in JEM. 14B illustrates an example of a partitioning pattern for partitioning a block into 8x8 sub-blocks in accordance with the disclosed techniques. 14C shows an example of a partitioning pattern for partitioning a block into 8x4 sub-blocks in accordance with the disclosed technique. 14D illustrates an example of a partitioning pattern for partitioning a block into 4x8 sub-blocks in accordance with the disclosed techniques. In Figure 14E, a portion of a block is divided into 4x4 sub-blocks according to the disclosed technique. Pixels on block boundaries are divided into smaller sub-blocks of size such as 2×4, 4×2 or 2×2. Some sub-blocks can be merged to form larger sub-blocks. Figure 14F shows an example of adjacent sub-blocks (eg, 4x4 sub-blocks and 2x4 sub-blocks) that are combined to form larger sub-blocks of size 6x4, 4x6, or 6x6. In Figure 14G, a portion of the block is divided into 8x8 sub-blocks. Whereas, pixels at block boundaries are divided into smaller sub-blocks such as 8x4, 4x8 or 4x4.

In subblock-based prediction, the shape and size of the subblock may be determined based on the shape and/or size of the coding block and/or coding block information. The coding block information may include an coding algorithm used on the block and/or sub-block, such as whether the motion compensated prediction is (1) an affine prediction method, (2) an optional temporal motion vector prediction method, (3) space-time A motion vector prediction method, (4) a bidirectional optical flow method, or (5) a frame rate up-conversion method. For example, in some embodiments, when the size of the current block is M×N, the size of the sub-block is 4×N (or 8×N, etc.), ie, the sub-block has the same height as the current block. In some embodiments, when the size of the current block is M×N, the size of the sub-block is M×4 (or M×8, etc.), that is, the sub-block has the same width as the current block. In some embodiments, when the size of the current block is MxN (where M>N), the size of the sub-block is AxB, where A>B (eg, 8x4). Alternatively, the size of the sub-block is BxA (eg, 4x8).

In some embodiments, the size of the current block is MxN. When M×N<=T (or min(M,N)<=T, or max(M,N)<=T, etc.), the size of the sub-block is A×B; when M×N>T (or When min(M, N)>T, or max(M, N)>T, etc.), the size of the sub-block is C×D, where A<=C and B<=D. For example, if MxN<=256, the size of the sub-block may be 4x4. In some implementations, the size of the sub-block is 8x8.

In some embodiments, whether to apply interleaved prediction may be determined based on the direction of inter prediction. The direction indicates whether the first or second intermediate prediction block is predicted backward in time or forward in time. For example, in some embodiments, interleaved prediction may be suitable for bidirectional prediction, but not for unidirectional prediction.

As another example, when applying multiple hypotheses, interleaved prediction can be applied to one prediction direction when there is more than one reference block. Multiple hypotheses may indicate that multiple video frames are used to make prediction blocks. When multiple video frames are used to make a prediction block, interleaved prediction can be applied to one prediction direction. The prediction direction may be a forward or backward prediction direction. The forward prediction direction refers to the occurrence of multiple video frames in the video sequence before the prediction block, and the backward direction refers to the occurrence of multiple reference frames in the video sequence after the prediction block.

In some embodiments, how to apply interleaving prediction may also be determined based on the inter prediction direction. In some embodiments, a bidirectionally predicted block with subblock-based prediction is divided into subblocks with two different partitioning modes for two different reference lists. In bidirectional prediction, a first reference list and a second reference list may be obtained, where the first reference list and the second reference list represent frames that are temporally forward and backward away from the prediction block. More specifically, the first reference list may include a first set of blocks in a first direction relative to a prediction block in the video sequence. The first set of blocks can be used to create prediction blocks. The second reference list may include a second set of blocks in a second direction relative to the prediction block in the video sequence. The second set of blocks can be used to create prediction blocks. The first direction and the second direction may be opposite, that is, one direction may be forward in time and the other direction may be backward in time away from the prediction block. The first reference list may be partitioned into a first group of sub-blocks according to a first partition mode. The second reference list may be partitioned into a second set of sub-blocks according to a second partition mode, wherein the first mode and the second mode are different.

For example, when predicting from reference list 0 (L0), the bidirectional prediction block is divided into 4x8 sub-blocks, as shown in Fig. 14D. When predicting from reference list 1 (L1), the same block is divided into 8x4 sub-blocks, as shown in Fig. 14C. The final prediction P is calculated as:

Here, P ⁰ and P ¹ are the predicted values from L0 and L1, respectively. w ⁰ and w ¹ are weighted values from L0 and L1, respectively. As shown in equation (16), the weighting value can be determined as: w ⁰ (x, y)+w ¹ (x, y)=1<<N (where N is a non-negative integer value). Since each direction prediction uses fewer subblocks (eg, 4×8 subblocks instead of 8×8 subblocks), the computation requires less bandwidth compared to existing subblock-based methods. By using larger sub-blocks, the prediction results are also less susceptible to noise interference.

In some embodiments, for the same reference list, a unidirectionally predicted block with subblock-based prediction is partitioned into subblocks with two or more different partitioning modes. For example, for the prediction of list L (L=0 or 1), ^PL is calculated as follows:

是用第i个划分模式生成的预测，并且

是

的加权值。例如，当XL为2时，列表L应用两种划分模式。在第一种划分模式中，将块划分为如图14D所示4×8子块，在第二种划分模式中，将块划分为如图14C所示的8×4子块。Here, XL is the number of division modes of the list L.

is the prediction generated with the ith partition mode, and

Yes

weighted value. For example, when XL is 2, list L applies two division modes. In the first division mode, the block is divided into 4×8 sub-blocks as shown in FIG. 14D , and in the second division mode, the block is divided into 8×4 sub-blocks as shown in FIG. 14C .

In some embodiments, a bidirectional prediction block based on subblock prediction is considered a combination of two unidirectional prediction blocks from L0 and L1, respectively. Predictions from each list can be derived as described in the example above. The final prediction P can be calculated as:

Here parameters a and b are two additional weights applied to the two intra prediction blocks. In this particular example, both a and b can be set to 1. Similar to the example above, the bandwidth usage is better than existing subblock-based methods or with Existing subblock-based methods are the same. At the same time, prediction results can be improved by adopting larger sub-blocks.

In some embodiments, a separate non-uniform pattern may be used in each uni-directional prediction block. For example, for each list L (eg, L0 or L1 ), the blocks are divided into different patterns (eg, as shown in FIG. 14E or FIG. 14F ). Using a smaller number of subblocks reduces bandwidth requirements. The inhomogeneity of the sub-blocks also increases the robustness of the prediction results.

In some embodiments, for a multi-hypothesis coding block, there may be multiple prediction blocks generated by different partition modes for each prediction direction (or reference picture list). The final prediction can be generated using multiple prediction blocks and applying additional weights. For example, the additional weight can be set to 1/M, where M is the total number of prediction blocks generated.

In some embodiments, the encoder can determine whether and how to apply interleaving prediction. The encoder can then perform operations at sequence level, picture level, view level, slice level, coding tree unit (CTU) (also known as largest coding unit (LCU)) level, CU level, PU level, tree unit (TU) level, The slice level, slice group level or region level (possibly including multiple CU/PU/TU/LCU) sends information corresponding to the determination to the decoder. This information can be in sequence parameter set (SPS), view parameter set (VPS), picture parameter set (PPS), slice header (SH), picture header, sequence header, slice level, slice group level, CTU/LCU, CU, Signaling in the first block of a PU, TU or region.

In some implementations, interleaved prediction is applicable to existing subblock methods, such as affine prediction, ATMVP, STMVP, FRUC, or BIO. In this case, no additional signaling costs are required. In some implementations, new subblock MERGE candidates generated by interleaving prediction may be inserted into the MERGE list, eg, interleaving prediction+ATMVP, interleaving prediction+STMVP, interleaving prediction+FRUC, and so on.

In some embodiments, a flag may be signaled to indicate whether to use interleaving prediction. Signaling the marker may include encoding the marker in the video information. In one example, if the current block is affine inter-coded, a flag a may be signaled to indicate whether to use interleaved prediction. In another example, if the current block is affine MERGE coded and unidirectional prediction is applied, this flag may be signaled to indicate whether interleaved prediction is used. In a third example, if the current block is affine MERGE encoded, this flag may be signaled to indicate whether to use interleaved prediction.

In some embodiments, interleaved prediction may always be used if the current block is affine MERGE coded and unidirectional prediction is applied. In some embodiments, interleaved prediction may always be used if the current block is affine MERGE encoded.

In some embodiments, a flag indicating whether to use interlace prediction may be inherited without signaling. In one example, inheritance can be used if the current block is affine MERGE encoded. In another example, a flag may be inherited from the flag of an adjacent block from which the affine model is inherited. In the third example, the flag is inherited from a predefined adjacent block, such as the adjacent block to the left or above. In a fourth example, the flags can be inherited from the first encountered affine coded adjacent block. If no adjacent blocks are affine coded, the flag can be inferred to be zero. In other words, if no adjacent blocks are affine coded, interleaving prediction is not applied. In the fifth example, the flag can only be inherited if the current block applies unidirectional prediction. In the sixth example, the flag can only be inherited if the current block and the adjacent block from which it is to be inherited are in the same CTU. In the seventh example, the flag can only be inherited if the current block and the adjacent block from which it is to be inherited are in the same CTU row. In the eighth example, when the affine model is derived from temporally adjacent blocks, the flags cannot be inherited from the flags of the adjacent blocks. In the ninth example, flags cannot be inherited from flags of adjacent blocks that are not in the same LCU or LCU row or video data processing unit (eg, 64x64 or 128x128). How the flag is signaled and/or derived may depend on the block size and/or coding information of the current block. The encoded information includes information encoded in the video.

In some embodiments, if the reference picture is the current picture, no interleaving prediction is applied. In one example, if the reference frame is the current frame containing the prediction block, a flag indicating whether to use interleaved prediction is not signaled. The reference frame is used as the basis for predicting the prediction block.

In some embodiments, the partitioning mode to be used by the current block may be derived based on information from spatially and/or temporally neighboring blocks. For example, both the encoder and the decoder may employ a set of predetermined rules to obtain partition patterns based on temporal adjacency (eg, previously used partition patterns for the same block) or spatial adjacency (eg, partition patterns used by adjacent blocks) , rather than relying on the encoder to send the relevant information.

In some embodiments, the weighting value w may be fixed. For example, all partition modes can be weighted equally: w _i (x,y)=1. In some embodiments, the weighting value may be determined based on the location of the block and the partitioning mode used. For example, w _i (x, y) may be different for different (x, y). In some embodiments, the weighting values may further depend on sub-block prediction based coding techniques (eg, affine or ATMVP) and/or other coding information (eg, skip or non-skip modes and/or MV information).

In some embodiments, the encoder may determine the weighting value and calculate the weighting value at the sequence level, picture level, slice level, CTU/LCU level, CU level, PU level or region level (possibly including multiple CU/PU/TU/LCU) Send these values to the decoder. The weighting value may be signaled in the first block of a sequence parameter set (SPS), picture parameter set (PPS), slice header (SH), CTU/LCU, CU, PU or region. In some embodiments, the weighted values may be derived from the weighted values of spatially and/or temporally neighboring blocks.

It should be noted that the interleaving prediction techniques disclosed herein may be applied to one, part or all of subblock prediction based coding techniques. For example, interleaved prediction techniques can be applied to affine prediction, while other subblock prediction-based coding techniques (eg, ATMVP, STMVP, FRUC, or BIO) do not use interleaved prediction. As another example, all affine, ATMVP and STMVP apply the interleaving prediction techniques disclosed herein.

15A is an example flow diagram of a method 1500 for improving motion prediction in a video system in accordance with the disclosed techniques. The method 1500 includes, at 1502, selecting a set of pixels from a video frame to form a block. The method 1500 includes, at 1504, partitioning the block into a first set of sub-blocks according to the first mode. The method 1500 includes generating, at 1506, a first inter-prediction block based on the first set of sub-blocks. The method 1500 includes, at 1508, dividing the block into a second set of sub-blocks according to the second mode. At least one sub-block in the second group has a different size than one sub-block in the first group. The method 1500 includes generating, at 1510, a second inter-prediction block based on the second set of sub-blocks. The method 1500 also includes determining, at 1512, a prediction block based on the first inter-prediction block and the second inter-prediction block.

In some embodiments, (1) affine prediction method, (2) optional temporal motion vector prediction method, (3) space-time motion vector prediction method, (4) bidirectional optical flow method, or (5) frame At least one of the rate up-conversion methods generates the first inter-prediction block or the second inter-prediction block.

In some embodiments, the sub-blocks in the first or second group have a rectangular shape. In some embodiments, the sub-blocks in the first set of sub-blocks have non-uniform shapes. In some embodiments, the sub-blocks in the second set of sub-blocks have non-uniform shapes.

In some embodiments, the method includes determining the first mode or the second mode based on the size of the block. In some embodiments, the method includes determining the first mode or the second mode based on information from a second block that is temporally or spatially adjacent to the block.

In some embodiments, for motion prediction of the block in the first direction, partitioning the block into a first set of sub-blocks is performed. In some embodiments, for motion prediction of the block in the second direction, partitioning the block into the second set of sub-blocks is performed.

In some embodiments, for motion prediction of a block in the first direction, partitioning the block into a first set of sub-blocks and partitioning the block into a second set of sub-blocks is performed. In some embodiments, the method further comprises: performing motion prediction on the block in the second direction by dividing the block into a third set of sub-blocks according to a third mode; generating a third intermediate prediction block based on the third set of sub-blocks; Divide the block into a fourth group of sub-blocks according to a fourth mode, wherein at least one sub-block in the fourth group is different in size from the sub-block in the third group; generate a fourth intermediate prediction block based on the fourth group of sub-blocks; generate a fourth intermediate prediction block based on the fourth group of sub-blocks; The three intermediate prediction blocks and the fourth intermediate prediction block determine the second prediction block; and the third prediction block is determined based on the prediction block and the second prediction block.

In some embodiments, the method includes sending information of a first mode and a second mode for partitioning the block to an encoding device in a block-based motion prediction video system. In some embodiments, transmitting the information of the first mode and the second mode is performed at one of: (1) sequence level, (2) picture level, (3) view level, (4) slice level, (5) encoding Tree unit, (6) maximum coding unit level, (7) coding unit level, (8) prediction unit level, (10) tree unit level, or (11) region level.

In some embodiments, determining the prediction result comprises: applying a first set of weights to a first intermediate prediction block to obtain a first weighted prediction block; applying a second set of weights to a second intermediate prediction block to obtain a second weighted prediction block ; and calculating a weighted sum of the first weighted prediction block and the second weighted prediction block to obtain the prediction block.

In some embodiments, the first set of weights or the second set of weights includes fixed weight values. In some embodiments, the first set of weights or the second set of weights is determined based on information from another block that is temporally or spatially adjacent to the block. In some embodiments, the first set of weights or the second set of weights is determined using an encoding algorithm used to generate the first prediction block or the second prediction block. In some implementations, at least one value in the first set of weights is different from another value in the first set of weights. In some implementations, at least one value in the second set of weights is different from another value in the second set of weights. In some implementations, the sum of the weights is equal to a power of two.

In some embodiments, the method includes transmitting the weights to an encoding device in a block-based motion prediction video system. In some embodiments, transmission weighting is performed at one of: (1) sequence level, (2) picture level, (3) view level, (4) slice level, (5) coding tree unit, (6) maximum coding Unit level, (7) coding unit level, (8) prediction unit level, (10) tree unit level, or (11) region level.

15B is an example flow diagram of a method 1550 for improving block-based motion prediction in a video system in accordance with the disclosed techniques. The method 1550 includes, at 1552, selecting a set of pixels from the video frame to form a block. The method 1550 includes dividing the block into a plurality of sub-blocks at 1554 based on a size of the block or information of another block that is spatially or temporally adjacent to the block. At least one sub-block of the plurality of sub-blocks is different in size from other sub-blocks. The method 1550 also includes generating, at 1556, a motion vector prediction by applying an encoding algorithm to the plurality of sub-blocks. In some embodiments, the encoding algorithm includes (1) an affine prediction method, (2) an optional temporal motion vector prediction method, (3) a space-time motion vector prediction method, (4) a bidirectional optical flow method, or (5) ) at least one of frame rate up-conversion methods.

As described further herein, the encoding process may avoid checking the affine mode of a block split from a parent block, which itself is encoded using a mode different from the affine mode.

Table 1 illustrates example performance results using regular 2x2 affine prediction for random access (RA) configurations.

Table 1 Example test results for 2x2 affine prediction

Y U V EncT DecT Category A1 -0.11% -0.18% -0.09% 139% 111% Category A2 -0.9% -0.85% -0.68% 142% 125% Category B -0.58% -0.51% -0.67% 136% 114% Category C -0.26% -0.24% -0.24% 129% 108% Category D -0.54% -0.52% -0.53% 130% 118% Category F -0.89% -1.02% -0.97% 125% 108% overall -0.47% -0.44% -0.44% 136% 114%

Table 2 illustrates example performance results obtained by applying interleaved prediction to unidirectional prediction in accordance with embodiments of the present technology. Table 3 illustrates example performance results obtained by applying interleaved prediction to bidirectional prediction in accordance with embodiments of the present technology.

Table 2 Example test results for interleaved prediction in one-way prediction

Y U V EncT DecT Category A1 -0.05% -0.14% -0.02% 101% 100% Category A2 -0.55% -0.17% -0.11% 102% 101% Category B -0.33% -0.17% -0.20% 101% 101% Category C -0.15% -0.16% -0.04% 100% 100% Category D -0.21% -0.09% -0.02% 106% 106% Category F -0.39% -0.40% -0.39% 102% 102% overall -0.27% -0.16% -0.11% 101% 101%

Table 3 Example test results for interleaved prediction in bidirectional prediction

Y U V EncT DecT Category A1 -0.09% -0.18% -0.12% 103% 102% Category A2 -0.74% -0.40% -0.28% 104% 104% Category B -0.37% -0.39% -0.35% 103% 102% Category C -0.22% -0.19% -0.13% 102% 102% Category D -0.42% -0.28% -0.32% 103% 102% Category F -0.60% -0.64% -0.62% 102% 102% overall -0.38% -0.30% -0.23% 103% 102%

As shown in Tables 2 and 3, interleaved prediction achieves major coding gains with lower complexity compared to conventional 2x2 affine prediction based coding. In particular, interleaved prediction applied to bidirectional prediction achieves a coding gain of 0.38% compared to the 2x2 affine method (0.47%). The encoding time and decoding time are 103% and 102%, respectively, compared to 136% and 114% in the 2x2 affine method.

16 is a schematic diagram illustrating an example of the structure of a computer system or other control device 1600 that may be used to implement various portions of the disclosed technology. In FIG. 16 , computer system 1600 includes one or more processors 1605 and memory 1610 connected by interconnect 1625 . Interconnect 1625 may represent any one or more separate physical buses, point-to-point connections, or both, connected by appropriate bridges, adapters, or controllers. Thus, interconnect 1625 may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), IIC (I2C) bus or Institute of Electrical and Electronics Engineers (IEEE) Standard 674 bus (sometimes referred to as "FireWire").

The processor 1605 may include a central processing unit (CPU) to control, for example, the overall operation of the host computer. In some embodiments, the processor 1605 does this by executing software or firmware stored in the memory 1610 . The processor 1605 may be or include one or more programmable general purpose or special purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), etc., or a combination of these devices.

Memory 1610 may be or include the main memory of the computer system. Memory 1610 represents any suitable form of random access memory (RAM), read only memory (ROM), flash memory, etc., or a combination of these devices. In use, memory 1610 may contain, among other things, a set of machine instructions that, when executed by processor 1605, cause processor 1605 to perform operations to implement embodiments of the disclosed technology.

Also connected to processor 1605 via interconnect 1625 is (optional) network adapter 1615. Network adapters 1615 provide computer system 1600 with the ability to communicate with remote devices, such as storage clients and/or other storage servers, and may be, for example, Ethernet adapters or Fibre Channel adapters.

17 illustrates a block diagram of an example embodiment of a mobile device 1700 that may be used to implement portions of the disclosed techniques. Mobile device 1700 may be a laptop, smartphone, tablet, video camera, or other device capable of processing video. The mobile device 1700 includes a processor or controller 1701 to process data, and a memory 1702 in communication with the processor 1701 to store and/or buffer the data. For example, the processor 1701 may include a central processing unit (CPU) or a microcontroller unit (MCU). In some implementations, the processor 1701 may include a field programmable gate array (FPGA). In some implementations, mobile device 1700 includes or communicates with a graphics processing unit (GPU), video processing unit (VPU), and/or wireless communication unit to implement various visual and/or communication data processing functions of the smartphone device. For example, memory 1702 may include and store processor-executable code that, when executed by processor 1701, configures mobile device 1700 to perform various operations, such as receiving information, commands and/or data, processing information and data, and Send or provide the processed information/data to another data device, such as an actuator or an external display. To support various functions of mobile device 1700 , memory 1702 may store information and data, such as instructions, software, values, images, and other data processed or referenced by processor 1701 . For example, various types of random access memory (RAM) devices, read only memory (ROM) devices, flash memory devices, and other suitable storage media may be used to implement the storage function of memory 1702. In some implementations, mobile device 1700 includes an input/output (I/O) unit 1703 to interface processor 1701 and/or memory 1702 with other modules, units or devices. For example, I/O unit 1703 may interface with processor 1701 and memory 1702 to utilize various wireless interfaces compatible with typical data communication standards, eg, between one or more computers and user equipment in the cloud. In some implementations, mobile device 1700 can interface with other devices through I/O unit 1703 using a wired connection. Mobile device 1700 may also interface with other external interfaces (eg, data storage) and/or visual or audio display device 1704 to retrieve and transmit data that may be processed by a processor, stored in memory, or transmitted by display device 1704 or an output unit of an external device. Displayed data and information. For example, display device 1704 may display a video frame modified based on MVP (eg, a video frame including prediction block 1305 as shown in FIG. 13 ) in accordance with the disclosed techniques.

In some embodiments, a video decoder device may implement a video decoding method wherein the video decoding is performed using the improved block-based motion prediction described herein. The method may include forming a video block using a set of pixels from a video frame. A block may be partitioned into a first set of sub-blocks according to a first pattern. The first intermediate prediction block may correspond to the first group of sub-blocks. The block may comprise a second set of sub-blocks according to the second mode. At least one of the sub-blocks in the second group has a different size than one of the sub-blocks in the first group. The method may also determine the prediction block based on the first intermediate prediction block and a second intermediate prediction block generated from the second set of sub-blocks. Other features of the method may be similar to method 1500 described above.

In some embodiments, a decoder-side method of video decoding may utilize block-based motion prediction to improve predicted video quality by using blocks of video frames, where a block corresponds to a set of pixel blocks. Based on the size of the block or information from another block that is spatially or temporally adjacent to the block, the block may be divided into multiple sub-blocks, wherein at least one of the multiple sub-blocks is of a different size than the other sub-blocks. The decoder may use motion vector prediction generated by applying an encoding algorithm to multiple sub-blocks. Additional features of the method are described with reference to Figure 15B and the corresponding description.

In some embodiments, the video decoding method may be implemented using a decoding apparatus implemented on a hardware platform as described in FIGS. 16 and 17 .

8 is a flowchart of an example method 800 of video encoding or decoding. Method 800 includes determining (802) to apply interleaving prediction to the block because the block satisfies the condition. The method 800 includes determining (804) a prediction block based on the first inter-prediction block and the second inter-prediction block. The method 800 includes generating (806) an encoded or decoded representation of the block using the predicted block. For example, a video encoder or transcoder may perform encoding at 806 and a video decoder may perform generation of a decoded representation at 806 . The first intermediate prediction block is generated from a first set of sub-blocks obtained by dividing the block according to the first mode, and the second intermediate prediction block is generated from a second set of sub-blocks obtained by dividing the block according to the second mode, and the second At least one sub-block in the group has a different size than the sub-block in the first group.

In some embodiments, the block satisfies the condition that the block is encoded using bidirectional predictive coding. In some embodiments, a block satisfies the condition that the block is predicted using multiple hypothesis prediction, and wherein interleaved prediction is applied to the prediction directions available for multiple reference blocks. In some embodiments, the block is partitioned into a first set of sub-blocks and the block is partitioned into a second set of sub-blocks for motion prediction of the block in the first direction. In some embodiments, the encoded representation may include the first mode and the second mode of information.

In some embodiments, the condition includes encoding the block using bidirectional prediction instead of unidirectional prediction, wherein bidirectional prediction is based on previous and subsequent video frames, and unidirectional prediction is based on previous or subsequent video frames only. In some embodiments, the condition is that the block is bidirectionally predicted, and wherein the first intermediate prediction block is generated from the first set of sub-blocks using a first reference list of blocks, and the second intermediate prediction block is a second intermediate prediction block using a second The reference list is generated from the second set of subblocks. In some embodiments, the condition is that the block is uni-directionally predicted, and wherein the first intermediate prediction block is generated from the first set of sub-blocks using a reference list L0 or L1 of blocks, and the second intermediate prediction block is generated using the block The reference list L0 or L1 is generated from the second set of sub-blocks. In some embodiments, the condition is that the block is bidirectionally predicted, and wherein the first intermediate prediction block is generated from the first set of sub-blocks using a first reference list of blocks, and the second intermediate prediction block is a second intermediate prediction block using a second The reference list is generated from the second set of subblocks. The method may further include generating one or more third intermediate prediction blocks from one or more third sets of sub-blocks of the block using the first reference list, and generating one or more fourth intermediate prediction blocks from the one or more fourth groups of blocks using the second reference list The group of sub-blocks generates one or more fourth intermediate prediction blocks, wherein the prediction blocks are determined based on the one or more third intermediate prediction blocks and/or the one or more fourth intermediate prediction blocks.

In some embodiments, the condition is that the block is a multiple-hypothesis coded block. The method may also include generating one or more additional intermediate prediction blocks from one or more sets of additional sub-blocks of the block using the reference list for the block, wherein the prediction block is determined based on the one or more additional intermediate prediction blocks of.

Other features and variations of method 800 may be similar to those described with reference to Figures 15A and 15B.

9 shows a functional block diagram of an example apparatus 1900 implementing the interleaving prediction techniques disclosed herein. For example, apparatus 1900 may be a video encoder or transcoder that receives video 1902 . The received video 1902 may be in the form of compressed video or uncompressed video. Video 1902 may be received through a network interface or from a storage device. Video 1902 (either in uncompressed or compressed form) may correspond to a video frame of a certain size. Apparatus 1900 may perform preprocessing 1904 operations on video 1902 . Preprocessing 1904 may be optional and may include things such as decryption, color space conversion, quality enhancement filtering, and the like. Encoder 1906 may convert video 1902 into an encoded representation that may be optionally post-processed by post-processing block 1910 to produce an output video. For example, encoder 1906 may perform interleaved prediction on blocks of video 1902. A block can represent an area of video of any size, but is typically chosen to have a fixed number of horizontal and vertical dimensions in terms of number of pixels (eg, 128x128 or 16x16, etc.). In some cases, a block may represent a coding unit. Optional post-processing blocks may include filtering, encryption, packing, etc. The output video 1910 may be stored on a storage device, or may be transmitted over a network interface.

From the foregoing, it should be understood that specific embodiments of the presently disclosed technology have been described herein for ease of illustration, but that various modifications may be made without departing from the scope of the present invention. Therefore, the technology disclosed in the present invention is not limited by the limitations of the claims, except for the following.

The embodiments, modules, and functional operations disclosed and otherwise described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more of them a combination of. The disclosed embodiments and other embodiments can be implemented as one or more computer program products, ie, one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or for controlling the execution of, data processing apparatus. operate. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a composition of matter that affects a machine-readable propagated signal, or a combination of one or more thereof. The term "data processing apparatus" includes all apparatus, equipment and machines for processing data, including, for example, programmable processors, computers, or groups of multiple processors or computers. In addition to hardware, the apparatus may include code that creates an execution environment for the computer program, eg, code making up processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is a human-generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiving device.

A computer program (also called a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component , subroutines, or other units suitable for use in a computing environment. Computer programs do not necessarily correspond to files in the file system. Programs may be stored in sections of files that hold other programs or data (for example, one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordination files (for example, storing a or files of multiple modules, subroutines, or portions of code). A computer program can be deployed for execution on one or more computers, located at one site or distributed across multiple sites, interconnected by a communications network.

The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, eg, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more of any kind of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks to receive data from or transfer data therefrom Transfer to one or more mass storage devices, or both. However, a computer does not necessarily have such a device. Computer readable media suitable for storage of computer program instructions and data include all forms of non-volatile memory, media and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable Magnetic disks; magneto-optical disks; and CDROM and DVD-ROM optical disks. The processor and memory may be supplemented by, or incorporated into, special purpose logic circuitry.

While this patent document contains many details, these should not be construed as limitations on the scope of any invention or claims, but rather as descriptions of features of particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various functions that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although the above features may be described as functioning in certain combinations, even as originally claimed, in some cases one or more features of a claim combination may be deleted from the combination and the claim combination Can point to sub-combinations or variants of sub-combinations.

Similarly, although a drawing depicts operations in a particular order, it should not be understood that such operations must be performed in the particular order or sequence shown, or that all illustrated operations be performed, to obtain desired results. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

Only some implementations and examples have been described, and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims (27)

1. A method of processing a video block using interleaved prediction, the method comprising: determining a prediction block for the block based on the first inter-prediction block and the second inter-prediction block because the block satisfies one or more conditions; and generating an encoded or decoded representation of the block using the predicted block; Wherein, the first intermediate prediction block is generated from a first group of sub-blocks obtained by dividing the block according to a first division mode, and the second intermediate prediction block is obtained from dividing the block according to a second division mode The resulting second set of sub-blocks is generated, wherein the first division mode is different from the second division mode; The encoding or decoding represents at the sequence parameter set (SPS), view parameter set (VPS), picture parameter set (PPS), slice header (SH), picture header, sequence header, slice level, slice group level, or region the initial block contains information on whether and how to apply the interleaving prediction, wherein the information includes a flag that is selectively included based on the condition of the block, When encoding and decoding the block using the sub-block prediction method, the flag is ignored, The sub-block prediction method is an optional temporal motion vector prediction method, an affine prediction method, a frame rate up-conversion method, a bidirectional optical flow method or a space-time motion vector prediction method, wherein the condition of the block is related to the block size and/or codec information of the block. 2. The method of claim 1, wherein the condition is that the block is uni-directionally predicted, and wherein the first intermediate prediction block is generated from the first set of sub-blocks from the same reference picture, And the second intermediate prediction block is generated from the second set of sub-blocks. 3. The method of claim 1, wherein the condition satisfied by the block is that the block uses bidirectional predictive coding. 4. The method of claim 1, wherein the condition comprises encoding and decoding the block using bidirectional prediction instead of unidirectional prediction, wherein the bidirectional prediction is based on a reference list L0 and a reference list L1, and the single The direction prediction is based only on the reference list L0 or the reference list L1. 5. The method of claim 1, wherein the condition is that the block is bi-directionally predicted, and wherein the first inter-prediction block is obtained from the first set of sub-blocks using a first reference list of the blocks block, and the second intermediate prediction block is generated from the second set of sub-blocks using a second reference list of the blocks. 6. The method of claim 1, wherein the block satisfies the condition that the block is predicted using multiple hypothesis prediction, and wherein the interleaved prediction is applied to prediction directions available for multiple reference blocks. 7. The method of claim 1, wherein the condition is that the block is bidirectionally predicted, and wherein the first inter prediction of the block is generated from the first set of sub-blocks using a first reference list block and generating the second intermediate prediction block of the block from the second set of sub-blocks using a second reference list, the method further comprising: generating one or more third intermediate prediction blocks from one or more third set of sub-blocks of the block using the first reference list; generating one or more fourth intermediate prediction blocks from one or more fourth groups of sub-blocks of the block using the second reference list; wherein the prediction block is determined based on the one or more third inter-prediction blocks and/or the one or more fourth inter-prediction blocks. 8. The method of claim 1, wherein the condition is that the block is a multiple hypothesis codec block, the method further comprising: generating one or more additional intermediate prediction blocks from one or more additional groups of sub-blocks of the block using the reference list used by the block; and wherein the prediction block is determined based on the one or more additional intermediate prediction blocks. 9. The method of claim 8, wherein the prediction block is determined as an equal-weighted weighted sum of the first inter-prediction block, the second inter-prediction block, and the one or more additional inter-prediction blocks . 10. The method of claim 1, wherein partitioning the block into a first set of sub-blocks and partitioning the block into a second set of sub-blocks is performed for motion prediction of the block in the first direction. 11. The method of any one of claims 1 to 10, further comprising: Information of the first division mode and the second division mode is included in the codec representation. 12. The method of claim 11, wherein the information of the first partition mode and the second partition mode is included at: (1) sequence level, (2) picture level, (3) view level, (4) Slice level, (5) Codec tree unit, (6) LCU level, (7) Codec unit level, (8) Prediction unit level, (10) Tree unit level, or (11) Region one of the grades. 13. The method of claim 1, wherein the codec representation is at (1) sequence level, (2) picture level, (3) view level, (4) slice level, (5) codec tree unit ( CTU), (6) largest codec unit (LCU) level, (7) codec unit (CU) level, (8) prediction unit (PU) level, (9) tree unit (TU) level, (10) slice level, (11) slice group level, or (12) region level, which may include multiple CU/PU/TU/LCU levels, contains relevant information for the interleaving prediction. 14. The method of claim 1, comprising signaling a flag indicating whether to use the interleaved prediction if the block is affine coded. 15. The method of claim 1, wherein a flag indicates whether to use the interleaved prediction, and the flag is not signaled. 16. The method of claim 1, inheriting a flag indicating whether to use the interleaved prediction for the block from previous codec information in the codec representation. 17. The method of claim 16, comprising inheriting the flag from an adjacent block from which the affine model is inherited. 18. The method of claim 17, comprising inheriting the flags of predefined neighboring blocks, the predefined neighboring blocks including the neighboring block to the left or the neighboring block above. 19. The method of claim 16, comprising inheriting the flag from an initially encountered affine coded neighboring block. 20. The method of claim 16, comprising inferring that there is no interleaving prediction when no adjacent blocks are affine coded. 21. The method of claim 16, comprising inheriting the flag when the block applies unidirectional prediction. 22. The method of claim 16, comprising inheriting the flag when the block and an adjacent block from which the flag is inherited are located in the same CTU. 23. The method of claim 16, comprising inheriting the flag when the block and neighboring blocks inheriting therefrom are in the same CTU row. 24. The method of claim 1, wherein the conditions include width and height of the block. 25. The method of claim 1, wherein the condition comprises encoding and decoding another block of a video frame without using the block. 26. A video processing apparatus comprising a processor configured to implement the method of any of claims 1 to 25. 27. A non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 25.

Images