CN118921468A - Method of decoding and encoding video and apparatus for transmitting compressed video data - Google Patents

Abstract

The present invention provides a method of decoding and encoding video and an apparatus for transmitting compressed video data. The method of decoding video according to the present invention may include: deriving a merge candidate from a neighboring block neighboring the current block; generating a merge candidate list including merge candidates; designating one of a plurality of merging candidates included in a merging candidate list; deriving an angular motion vector of the current block based on the specified merge candidate; deriving a motion vector of a sub-block in the current block based on the angular motion vector of the current block, the current block being divided into a plurality of sub-blocks having a predefined size of 4×4; and performing motion compensation of the sub-block based on the motion vector, wherein when the neighboring block is not included in the same coding tree unit CTU as the current block, an angular motion vector of the merging candidate is derived based on the motion vectors at the lower left and lower right corners of the neighboring block.

Description

Method for decoding and encoding video and device for transmitting compressed video data

This application is a divisional application of invention patent application No. 201980035029.4, whose application date is May 23, 2019, international application number is PCT/KR2019/006221, and which entered the Chinese national phase on November 24, 2020, and whose invention name is “Method and device for processing video signals”.

Technical Field

The invention relates to a method and an apparatus for processing a video signal.

Background Art

Recently, the demand for high-resolution and high-quality images, such as high-definition (HD) images and ultra-high-definition (UHD) images, has increased in various application fields. However, compared with conventional image data, higher resolution and quality image data has an increased data volume. Therefore, when the image data is transmitted by using a medium such as a conventional wired and wireless broadband network, or when the image data is stored by using a conventional storage medium, the cost of transmission and storage increases. In order to solve these problems arising as the resolution and quality of image data increase, efficient image encoding/decoding technology can be used.

The image compression technology includes various technologies, including: an inter-frame prediction technology that predicts a pixel value included in a current picture based on a previous picture or a subsequent picture of the current picture; an intra-frame prediction technology that predicts a pixel value included in the current picture by using pixel information in the current picture; an entropy coding technology that assigns a short code to a value with a high frequency of occurrence and a long code to a value with a low frequency of occurrence, etc. Image data can be effectively compressed by using such an image compression technology and can be transmitted or stored.

Meanwhile, along with the demand for high-resolution images, the demand for stereoscopic image content as a new image service is also increasing. Video compression technology for efficiently providing stereoscopic image content with high resolution and ultra-high resolution is being studied.

Summary of the invention

Technical issues

The present invention provides a method and apparatus for efficiently performing inter-frame prediction on an encoding/decoding target block when encoding/decoding a video signal.

The present invention provides a method and apparatus for performing motion compensation by using a plurality of merge candidate lists when encoding/decoding a video signal.

The present invention provides a method and apparatus for performing motion compensation based on an affine model when decoding/decoding a video signal.

The present invention provides a method and apparatus for deriving an angular motion vector from neighboring blocks or non-neighboring blocks of a current block when decoding/decoding a video signal.

Technical problems that can be obtained from the present invention are not limited to the above-mentioned technical tasks, and a person of ordinary skill in the technical field to which the present invention belongs can clearly understand other unmentioned technical tasks according to the following description.

Technical Solution

According to the video signal decoding method and device of the present invention, a merge candidate can be derived based on a candidate block, a first merge candidate list including the merge candidate can be generated, one of multiple merge candidates included in the first merge candidate list can be specified, an affine vector of a current block can be derived based on motion information of the specified merge candidate, a motion vector of a sub-block in the current block can be derived based on the affine vector, and motion compensation of the sub-block can be performed based on the motion vector.

According to the video signal encoding method and device of the present invention, a merge candidate can be derived based on a candidate block, a first merge candidate list including the merge candidate can be generated, one of multiple merge candidates included in the first merge candidate list can be specified, an affine vector of a current block can be derived based on motion information of the specified merge candidate, a motion vector of a sub-block in the current block can be derived based on the affine vector, and motion compensation of the sub-block can be performed based on the motion vector.

With the video signal encoding/decoding method and apparatus according to the present invention, when the candidate block is encoded by affine inter prediction, the affine vector of the merge candidate may be derived based on the affine vector of the candidate block.

For the video signal encoding/decoding method and apparatus according to the present invention, the position of the affine vector of the candidate block may be different based on whether the current block and the candidate block are included in the same CTU (Coding Tree Unit).

For the video signal encoding/decoding method and apparatus according to the present invention, an affine vector of a merge candidate can be derived by combining translation motion vectors of a plurality of candidate blocks.

With the video signal encoding/decoding method and apparatus according to the present invention, when the number of merge candidates included in the first merge candidate list is less than the maximum number, the merge candidates included in the second merge candidate list may be added to the first merge candidate list.

With the video signal encoding/decoding method and apparatus according to the present invention, the number of affine parameters of the current block may be determined based on at least one of the size or shape of the current block.

For the video signal encoding/decoding method and apparatus according to the present invention, the candidate blocks may include at least one of a neighboring block of the current block and a non-neighboring block on the same line as the neighboring block.

The present invention also provides a method for decoding a video, the method comprising: deriving a merge candidate from a neighboring block adjacent to a current block; generating a merge candidate list including the merge candidate; specifying one merge candidate from multiple merge candidates included in the merge candidate list; deriving an angular motion vector of the current block based on the specified merge candidate; deriving a motion vector of a sub-block in the current block based on the angular motion vector of the current block, the current block being divided into multiple sub-blocks with a predefined size of 4×4; and performing motion compensation of the sub-block based on the motion vector, wherein, when the neighboring block is not included in the same coding tree unit CTU as the current block, the angular motion vector of the merge candidate is derived based on the motion vectors at the lower left and lower right corners of the neighboring block.

The present invention also provides a method for encoding a video, the method comprising: deriving a merge candidate from a neighboring block adjacent to a current block; generating a merge candidate list including the merge candidate; specifying one merge candidate from multiple merge candidates included in the merge candidate list; deriving an angular motion vector of the current block based on the specified merge candidate; deriving a motion vector of a sub-block in the current block based on the angular motion vector of the current block, the current block being divided into multiple sub-blocks with a predefined size of 4×4; and performing motion compensation of the sub-block based on the motion vector, wherein, when the neighboring block is not included in the same coding tree unit CTU as the current block, the angular motion vector of the merge candidate is derived based on the motion vectors at the lower left and lower right corners of the neighboring block.

The present invention also provides a device for sending compressed video data, the device comprising: a processor configured to obtain the compressed video data; and a sending unit configured to send the compressed video data, wherein obtaining the compressed video data comprises: deriving a merge candidate from a neighboring block adjacent to a current block; generating a merge candidate list including the merge candidate; specifying one of multiple merge candidates included in the merge candidate list; deriving an angular motion vector of the current block based on the specified merge candidate; deriving a motion vector of a sub-block in the current block based on the angular motion vector of the current block, the current block being divided into multiple sub-blocks of a predefined size of 4×4; and performing motion compensation of the sub-block based on the motion vector, wherein, when the neighboring block is not included in the same coding tree unit CTU as the current block, the angular motion vector of the merge candidate is derived based on the motion vectors at the lower left and lower right corners of the neighboring block.

It should be understood that the above-summarized features are exemplary aspects of the following detailed description of the invention and do not limit the scope of the invention.

Beneficial Effects

According to the present invention, the efficiency of inter prediction can be improved by performing motion compensation by using a plurality of merge candidate lists.

According to the present invention, the efficiency of inter-frame prediction can be improved by obtaining motion information based on a plurality of merge candidates.

According to the present invention, the efficiency of inter-frame prediction can be improved by performing motion compensation based on an affine model.

According to the present invention, the efficiency of inter-frame prediction can be improved by deriving an angular motion vector from a neighboring block or a non-neighboring block of a current block.

Effects that can be obtained from the present invention may not be limited to the above-mentioned effects, and other unmentioned effects may be clearly understood by a person of ordinary skill in the technical field to which the present invention pertains from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an apparatus for encoding a video according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an apparatus for decoding a video according to an embodiment of the present invention.

FIG. 3 is a diagram showing partition mode candidates that can be applied to a coding block when the coding block is encoded through inter prediction.

FIG. 4 shows an example of hierarchical division of coding blocks based on a tree structure as an embodiment to which the present invention is applied.

FIG. 5 is a diagram showing a partition shape that allows binary tree-based partitioning as an embodiment to which the present invention is applied.

FIG6 shows a ternary tree partition shape.

FIG. 7 is a diagram showing an example of binary tree-based partitioning that allows only specific shapes.

FIG. 8 is a diagram for describing an example in which information related to the number of times binary tree division is allowed is encoded/decoded according to an embodiment to which the present invention is applied.

FIG. 9 is a flowchart showing an inter-frame prediction method as an embodiment to which the present invention is applied.

FIG. 10 is a diagram illustrating a process of deriving motion information of a current block when a merge mode is applied to the current block.

FIG. 11 is a diagram showing an example of spatially adjacent blocks.

FIG. 12 is a diagram showing an example of deriving a motion vector of a temporal merging candidate.

FIG. 13 is a diagram showing positions of candidate blocks that can be used as co-located blocks.

FIG. 14 is a diagram illustrating a process of deriving motion information of a current block when the AMVP mode is applied to the current block.

FIG. 15 is a diagram showing an example of deriving a merge candidate from a second merge candidate block when a first merge candidate block is unavailable.

FIG. 16 is a diagram showing an example of deriving a merge candidate from a second merge candidate block located on the same line as a first merge candidate block.

17 to 20 are diagrams showing an order of searching for merge candidate blocks.

FIG. 21 is a diagram showing an example of deriving merging candidates of non-square blocks based on square blocks.

FIG. 22 is a diagram showing an example of deriving merge candidates based on high-level node blocks.

FIG. 23 is a diagram showing an example of determining the availability of spatially neighboring blocks based on a merged estimated region.

FIG. 24 is a diagram showing an example of deriving a merge candidate based on a merge estimation region.

FIG. 25 is a diagram showing a motion model.

FIG. 26 is a diagram showing an affine motion model using angular motion vectors.

FIG. 27 is a diagram showing an example of determining a motion vector in units of sub-blocks.

FIG. 28 is a diagram showing an example of determining the position of a corner according to the shape of a current block.

FIG. 29 is a flowchart showing the motion compensation process in the affine inter mode.

30 and 31 are diagrams showing examples of deriving an affine vector of a current block based on motion vectors of neighboring blocks.

FIG. 32 is a diagram showing candidate blocks for deriving spatial merging candidates.

FIG. 33 shows an example of deriving an affine vector of a current block from affine vectors of candidate blocks.

DETAILED DESCRIPTION

Various modifications can be made to the present invention, and there are various embodiments of the present invention, examples of which will now be provided with reference to the accompanying drawings, and examples of which will be described in detail. However, the present invention is not limited thereto, and the exemplary embodiments may be interpreted as including all modifications, equivalents or alternatives within the technical concept and technical scope of the present invention. In the accompanying drawings described, similar reference numerals refer to similar elements.

The terms 'first', 'second', etc. used in the specification may be used to describe various components, but these components should not be interpreted as being limited to these terms. These terms are only used to distinguish one component from other components. For example, without departing from the scope of the present invention, a 'first' component may be referred to as a 'second' component, and a 'second' component may also be similarly referred to as a 'first' component. The term 'and/or' includes a combination of a plurality of items or any one of a plurality of items.

In the present disclosure, when an element is referred to as being "connected" or "coupled" to another element, it should be understood that it includes not only that the element is directly connected or coupled to the other element, but also that another element may exist between them. When an element is referred to as being "directly connected" or "directly coupled" to another element, it should be understood that there are no other elements between them.

The terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. Expressions used in the singular include expressions in the plural form unless the expression has a significantly different meaning in the context. In this specification, it should be understood that terms such as "including", "having" etc. are intended to indicate the existence of features, numbers, steps, actions, elements, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, actions, elements, parts or combinations thereof may exist or may be added.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, the same constituent elements in the drawings are denoted by the same reference numerals, and repeated description of the same elements will be omitted.

FIG. 1 is a block diagram showing an apparatus for encoding a video according to an embodiment of the present invention.

1 , a device 100 for encoding a video may include a picture division module 110 , prediction modules 120 and 125 , a transform module 130 , a quantization module 135 , a rearrangement module 160 , an entropy encoding module 165 , an inverse quantization module 140 , an inverse transform module 145 , a filter module 150 , and a memory 155 .

The components shown in FIG. 1 are shown independently to represent the characteristic functions that are different from each other in the device for encoding a video. Therefore, this does not mean that each component is composed of a separate hardware or software component unit. In other words, for convenience, each component includes each of the listed components. Therefore, at least two components in each component can be combined to form a component, or a component can be divided into multiple components to perform each function. Without departing from the essence of the present invention, the embodiment of combining each component and the embodiment of segmenting a component are also included in the scope of the present invention.

In addition, some of the components may not be indispensable components for performing the basic functions of the present invention, but selective components that only improve the performance of the present invention. The present invention can be implemented by excluding components for improving performance and only including indispensable components for implementing the essence of the present invention. Structures that exclude selective components that only improve performance and only include indispensable components are also included in the scope of the present invention.

The picture partitioning module 110 may partition an input picture into one or more processing units. Here, a processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). The picture partitioning module 110 may partition a picture into a combination of a plurality of coding units, prediction units, and transform units, and may encode a picture by selecting a combination of coding units, prediction units, and transform units using a predetermined criterion (e.g., a cost function).

For example, a picture may be divided into a plurality of coding units. A recursive tree structure such as a quadtree structure may be used to divide a picture into coding units. A coding unit divided into other coding units with a picture or a maximum coding unit as a root may be divided into child nodes corresponding to the number of divided coding units. Coding units that cannot be further divided according to predetermined restrictions are used as leaf nodes. That is, when it is assumed that only square division is feasible for a coding unit, a coding unit may be divided into up to four other coding units.

Hereinafter, in an embodiment of the present invention, a coding unit may mean a unit that performs encoding or a unit that performs decoding.

The prediction unit may be one of partitions split into a square shape or a rectangular shape having the same size in a single coding unit, or the prediction unit may be one of partitions split so as to have different shapes/sizes in a single coding unit.

When a prediction unit subject to intra prediction is generated based on a coding unit and the coding unit is not a minimum coding unit, intra prediction may be performed without splitting the coding unit into a plurality of prediction units NxN.

The prediction modules 120 and 125 may include an inter prediction module 120 that performs inter prediction and an intra prediction module 125 that performs intra prediction. It may be determined whether inter prediction or intra prediction is performed for a prediction unit, and detailed information (e.g., intra prediction mode, motion vector, reference picture, etc.) according to each prediction method may be determined. Here, the processing unit subjected to prediction may be different from the processing unit for which the prediction method and detailed content are determined. For example, the prediction method, prediction mode, etc. may be determined by the prediction unit, and the prediction may be performed by the transform unit. The residual value (residual block) between the generated prediction block and the original block may be input to the transform module 130. In addition, the prediction mode information, motion vector information, etc. used for prediction may be encoded together with the residual value by the entropy encoding module 165, and may be transmitted to a device for decoding a video. When a specific encoding mode is used, it may be transmitted to a device for decoding a video by encoding the original block as it is without generating a prediction block through the prediction modules 120 and 125.

The inter-frame prediction module 120 may predict a prediction unit based on information of at least one of a previous picture or a subsequent picture of the current picture, or in some cases, may predict a prediction unit based on information of some coding regions in the current picture. The inter-frame prediction module 120 may include a reference picture interpolation module, a motion prediction module, and a motion compensation module.

The reference picture interpolation module may receive reference picture information from the memory 155, and may generate pixel information of integer pixels or less than integer pixels according to the reference picture. In the case of luma pixels, an 8-tap interpolation filter based on DCT with different filter coefficients may be used to generate pixel information of integer pixels or less than integer pixels in units of 1/4 pixels. In the case of chrominance signals, a 4-tap interpolation filter based on DCT with different filter coefficients may be used to generate pixel information of integer pixels or less than integer pixels in units of 1/8 pixels.

The motion prediction module can perform motion prediction based on the reference picture interpolated by the reference picture interpolation module. Various methods such as full search based block matching algorithm (FBMA), three-step search (TSS), new three-step search algorithm (NTS), etc. can be used as methods for calculating motion vectors. Based on the interpolated pixels, the motion vector can have a motion vector value in units of 1/2 pixel or 1/4 pixel. The motion prediction module can predict the current prediction unit by changing the motion prediction method. Various methods such as skip method, merge method, AMVP (Advanced Motion Vector Prediction) method, intra-frame block copy method, etc. can be used as motion prediction methods.

The intra prediction module 125 may generate a prediction unit based on reference pixel information adjacent to the current block, which is pixel information in the current picture. When the adjacent block of the current prediction unit is a block subject to inter-frame prediction and thus the reference pixel is a pixel subject to inter-frame prediction, the reference pixel information of the adjacent block subject to intra-frame prediction may be used to replace the reference pixel included in the block subject to inter-frame prediction. That is, when the reference pixel is unavailable, at least one reference pixel among the available reference pixels may be used to replace the unavailable reference pixel information.

The prediction mode of the intra prediction may include a directional prediction mode using reference pixel information according to a prediction direction and a non-directional prediction mode that does not use directional information when performing prediction. The mode for predicting luminance information may be different from the mode for predicting chrominance information, and in order to predict chrominance information, the intra prediction mode information for predicting luminance information or the predicted luminance signal information may be used.

When performing intra prediction, when the size of the prediction unit is the same as the size of the transform unit, intra prediction can be performed on the prediction unit based on pixels located on the left, upper left, and upper sides of the prediction unit. However, when performing intra prediction, when the size of the prediction unit is different from the size of the transform unit, intra prediction can be performed using reference pixels based on the transform unit. In addition, intra prediction using NxN partitioning can be used only for the minimum coding unit.

In the intra prediction method, according to the prediction mode, a prediction block can be generated after applying an AIS (Adaptive Intra Smoothing) filter to a reference pixel. The type of the AIS filter applied to the reference pixel may be different. In order to perform the intra prediction method, the intra prediction mode of the current prediction unit may be predicted according to the intra prediction mode of a prediction unit adjacent to the current prediction unit. When the prediction mode of the current prediction unit is predicted by using the mode information predicted according to the adjacent prediction unit, when the intra prediction mode of the current prediction unit is the same as the intra prediction mode of the adjacent prediction unit, predetermined flag information may be used to transmit information indicating that the prediction mode of the current prediction unit and the prediction mode of the adjacent prediction unit are the same as each other. When the prediction mode of the current prediction unit is different from the prediction mode of the adjacent prediction unit, entropy coding may be performed to encode the prediction mode information of the current block.

In addition, a residual block including information on a residual value, which is a difference between the prediction unit subjected to prediction and the original block of the prediction unit, may be generated based on the prediction unit generated by the prediction modules 120 and 125. The generated residual block may be input to the transform module 130.

The transform module 130 may transform the residual block including information on the residual value between the original block and the prediction unit generated by the prediction modules 120 and 125 by using a transform method such as discrete cosine transform (DCT), discrete sine transform (DST), and KLT. Whether to apply DCT, DST, or KLT in order to transform the residual block may be determined based on intra prediction mode information of the prediction unit used to generate the residual block.

The quantization module 135 may quantize the value transformed to the frequency domain by the transform module 130. The quantization coefficient may be different according to the block or importance of the picture. The value calculated by the quantization module 135 may be provided to the inverse quantization module 140 and the rearrangement module 160.

The rearrangement module 160 may rearrange coefficients of the quantized residual value.

The rearrangement module 160 can change the coefficient of the two-dimensional block form into the coefficient of the one-dimensional vector form by the coefficient scanning method. For example, the rearrangement module 160 can use a zigzag scanning method to scan from the DC coefficient to the coefficient of the high frequency domain so that the coefficient is changed into a one-dimensional vector form. According to the size of the transform unit and the intra-frame prediction mode, the zigzag scanning can be replaced by the vertical scanning of the coefficient of the two-dimensional block form in the column direction or the horizontal scanning of the coefficient of the two-dimensional block form in the row direction. That is to say, it can be determined according to the size of the transform unit and the intra-frame prediction mode which scanning method to use among the zigzag scanning, the vertical scanning and the horizontal scanning.

The entropy encoding module 165 may perform entropy encoding based on the value calculated by the rearrangement module 160. The entropy encoding may use various encoding methods such as exponential Golomb coding, context adaptive variable length coding (CAVLC), and context adaptive binary arithmetic coding (CABAC).

The entropy coding module 165 can encode various information from the rearrangement module 160 and the prediction modules 120 and 125, such as residual value coefficient information and block type information of the coding unit, prediction mode information, partition unit information, prediction unit information, transformation unit information, motion vector information, reference frame information, block interpolation information, filtering information, etc.

The entropy encoding module 165 may entropy encode coefficients of the coding unit input from the rearrangement module 160 .

The inverse quantization module 140 may inversely quantize the value quantized by the quantization module 135, and the inverse transform module 145 may inversely transform the value transformed by the transform module 130. The residual values generated by the inverse quantization module 140 and the inverse transform module 145 may be combined with the prediction units predicted by the motion estimation module, the motion compensation module, and the intra prediction module of the prediction modules 120 and 125, so that a reconstructed block may be generated.

The filter module 150 may include at least one of a deblocking filter, an offset correction unit, and an adaptive loop filter (ALF).

The deblocking filter can remove block distortion that occurs due to the boundaries between blocks in the reconstructed picture. In order to determine whether to perform deblocking, the pixels in several rows or columns included in the block can be the basis for determining whether to apply the deblocking filter to the current block. When the deblocking filter is applied to the block, a strong filter or a weak filter can be applied according to the required deblocking filter strength. In addition, when applying the deblocking filter, horizontal filtering and vertical filtering can be processed in parallel.

The offset correction module can correct the offset from the original picture in units of pixels in the picture subjected to deblocking. In order to perform offset correction on a specific picture, a method of applying an offset in consideration of edge information of each pixel or a method of dividing the pixels of the picture into a predetermined number of regions, determining the region to be subjected to performing the offset, and applying the offset to the determined region can be used.

Adaptive loop filtering (ALF) can be performed based on a value obtained by comparing a filtered reconstructed picture with an original picture. The pixels included in the picture can be divided into predetermined groups, a filter to be applied to each group in the group can be determined, and filtering can be performed separately for each group. Information on whether ALF is applied and a luminance signal can be transmitted by a coding unit (CU). The shape and filter coefficients of the filter used for ALF can be different for each block. In addition, regardless of the characteristics of the application target block, a filter for ALF of the same shape (fixed shape) can be applied.

The memory 155 may store the reconstructed block or the reconstructed picture calculated by the filter module 150. The stored reconstructed block or the reconstructed picture may be provided to the prediction modules 120 and 125 when performing inter prediction.

FIG. 2 is a block diagram showing an apparatus for decoding a video according to an embodiment of the present invention.

2 , the apparatus 200 for decoding a video may include an entropy decoding module 210 , a rearrangement module 215 , an inverse quantization module 220 , an inverse transform module 225 , prediction modules 230 and 235 , a filter module 240 , and a memory 245 .

When a video bitstream is input from the apparatus for encoding a video, the input bitstream may be decoded according to an inverse process of the apparatus for encoding a video.

The entropy decoding module 210 may perform entropy decoding according to an inverse process of entropy encoding performed by an entropy encoding module of a device for encoding a video. For example, corresponding to a method performed by a device for encoding a video, various methods such as exponential Golomb coding, context adaptive variable length coding (CAVLC), and context adaptive binary arithmetic coding (CABAC) may be applied.

The entropy decoding module 210 may decode information about intra prediction and inter prediction performed by the apparatus for encoding a video.

The rearrangement module 215 may perform rearrangement on the bitstream entropy-decoded by the entropy decoding module 210 based on the rearrangement method used in the apparatus for encoding a video. The rearrangement module may reconstruct and rearrange coefficients in the form of a one-dimensional vector into coefficients in the form of a two-dimensional block. The rearrangement module 215 may receive information related to coefficient scanning performed in the apparatus for encoding a video, and may perform rearrangement via a method of inversely scanning the coefficients based on the scanning order performed in the apparatus for encoding a video.

The inverse quantization module 220 may perform inverse quantization based on a quantization parameter received from the apparatus for encoding a video and coefficients of the rearranged block.

The inverse transform module 225 may perform an inverse transform, i.e., inverse DCT, inverse DST, and inverse KLT, on the quantization result of the apparatus for encoding the video, which is an inverse process of the transform, i.e., DCT, DST, and KLT, performed by the transform module. The inverse transform may be performed based on a transmission unit determined by the apparatus for encoding the video. The inverse transform module 225 of the apparatus for decoding the video may selectively perform a transform scheme (e.g., DCT, DST, and KLT) according to multiple pieces of information such as a prediction method, a size of a current block, a prediction direction, and the like.

The prediction modules 230 and 235 may generate a prediction block based on the information on prediction block generation received from the entropy decoding module 210 and previously decoded block or picture information received from the memory 245 .

As described above, similar to the operation of the apparatus for encoding a video, when performing intra prediction, when the size of the prediction unit is the same as the size of the transform unit, intra prediction can be performed on the prediction unit based on pixels located on the left, upper left, and upper sides of the prediction unit. When performing intra prediction, when the size of the prediction unit is different from the size of the transform unit, intra prediction can be performed using reference pixels based on the transform unit. In addition, intra prediction using NxN partitioning can be used only for the minimum coding unit.

The prediction modules 230 and 235 may include a prediction unit determination module, an inter-frame prediction module, and an intra-frame prediction module. The prediction unit determination module may receive various information such as prediction unit information, prediction mode information of an intra-frame prediction method, information about motion prediction of an inter-frame prediction method, etc. from the entropy decoding module 210, may divide the current coding unit into prediction units, and may determine whether to perform inter-frame prediction or intra-frame prediction on the prediction unit. By using the information required in the inter-frame prediction of the current prediction unit received from the device for encoding a video, the inter-frame prediction module 230 may perform inter-frame prediction on the current prediction unit based on information of at least one of a previous picture or a subsequent picture of the current picture including the current prediction unit. Alternatively, inter-frame prediction may be performed based on information of some pre-reconstructed areas in the current picture including the current prediction unit.

In order to perform inter prediction, which mode of the skip mode, the merge mode, the AMVP mode, and the inter block copy mode to use as a motion prediction method of a prediction unit included in the coding unit may be determined for the coding unit.

The intra prediction module 235 can generate a prediction block based on pixel information in the current picture. When the prediction unit is a prediction unit subjected to intra prediction, intra prediction can be performed based on intra prediction mode information of the prediction unit received from the device for encoding the video. The intra prediction module 235 may include an adaptive intra smoothing (AIS) filter, a reference pixel interpolation module, and a DC filter. The AIS filter performs filtering on the reference pixels of the current block, and it can be determined whether to apply the filter according to the prediction mode of the current prediction unit. AIS filtering can be performed on the reference pixels of the current block by using the AIS filter information received from the device for encoding the video and the prediction mode of the prediction unit. When the prediction mode of the current block is a mode in which AIS filtering is not performed, the AIS filter may not be applied.

When the prediction mode of the prediction unit is a prediction mode in which intra prediction is performed based on pixel values obtained by interpolating reference pixels, the reference pixel interpolation module may interpolate the reference pixels to generate reference pixels that are integer pixels or smaller than integer pixels. When the prediction mode of the current prediction unit is a prediction mode in which a prediction block is generated without interpolating reference pixels, the reference pixels may not be interpolated. When the prediction mode of the current block is a DC mode, a DC filter may generate a prediction block by filtering.

The reconstructed block or picture may be provided to the filter module 240. The filter module 240 may include a deblocking filter, an offset correction module, and an ALF.

Information about whether to apply a deblocking filter to a corresponding block or picture and information about which filter of a strong filter and a weak filter to apply when applying the deblocking filter can be received from a device for encoding a video. The deblocking filter of a device for decoding a video can receive information about the deblocking filter from a device for encoding a video, and can perform deblocking filtering on the corresponding block.

The offset correction module may perform offset correction on the reconstructed picture based on the type of offset correction applied to the picture when encoding is performed and offset value information.

The ALF may be applied to the coding unit based on information on whether the ALF is applied, ALF coefficient information, etc. received from the apparatus for encoding a video. The ALF information may be provided to be included in a specific parameter set.

The memory 245 may store the reconstructed picture or block to be used as a reference picture or block, and may provide the reconstructed picture to the output module.

As described above, in the embodiments of the present invention, for convenience of explanation, a coding unit is used as a term indicating a unit for encoding, but the coding unit may be used as a unit that performs decoding as well as encoding.

In addition, the current block may represent a target block to be encoded/decoded. And, according to the encoding/decoding step, the current block may represent a coding tree block (or coding tree unit), a coding block (or coding unit), a transform block (or transform unit), a prediction block (or prediction unit), etc. In the present specification, a 'unit' may represent a basic unit for performing a specific encoding/decoding process, and a 'block' may represent a sample array of a predetermined size. Unless otherwise specified, 'block' and 'unit' may be used as the same meaning. For example, in the examples mentioned later, it can be understood that the coding block and the coding unit have the same meaning as each other.

A picture can be encoded/decoded by being divided into basic blocks having a square shape or a non-square shape. At this time, the basic block can be referred to as a coding tree unit. The coding tree unit can be defined as a coding unit of the maximum size allowed within a sequence or slice. Information indicating whether the coding tree unit has a square shape or a non-square shape or information related to the size of the coding tree unit can be signaled through a sequence parameter set, a picture parameter set, or a slice header. The coding tree unit can be divided into smaller-sized partitions. At this time, if it is assumed that the depth of the partition generated by splitting the coding tree unit is 1, the depth of the partition generated by splitting the partition with a depth of 1 can be defined as 2. That is, the partition generated by splitting the partition with a depth of k in the coding tree unit can be defined as a depth of k+1.

A partition of any size generated by splitting a coding tree unit may be defined as a coding unit. A coding unit may be recursively split or divided into a basic unit for performing prediction, quantization, transformation, or loop filtering, etc. For example, a partition of any size generated by splitting a coding unit may be defined as a coding unit, or may be defined as a transformation unit or a prediction unit, which is a basic unit for performing prediction, quantization, transformation, or loop filtering, etc.

Alternatively, a prediction block having the same size as or smaller than the coding block can be determined by predictive partitioning of the coding block. For the predictive partitioning of the coding block, any one of the partitioning mode (Part_mode) candidates representing the partitioning shape of the coding block can be specified. Information for determining a partitioning index indicating any one of the partitioning mode candidates can be sent by a bitstream signal. Alternatively, the partitioning index of the coding block can be determined based on at least one of the size, shape, or coding mode of the coding block. The size or shape of the prediction block can be determined based on the partitioning mode specified by the partitioning index. The partitioning mode candidates may include asymmetric partitioning shapes (e.g., nLx2N, nRx2N, 2NxnU, 2NxnD). The number or type of asymmetric partitioning mode candidates available for the coding block can be determined based on at least one of the size, shape, or coding mode of the coding block.

FIG. 3 is a diagram showing partition mode candidates that can be applied to a coding block when the coding block is encoded through inter prediction.

When a coding block is encoded by inter prediction, any one of the eight partition mode candidates shown in FIG. 3 may be applied to the coding block.

On the other hand, when the coding block is encoded by intra prediction, only the square partitioning mode may be applied to the coding block. In other words, when the coding block is encoded by intra prediction, the partitioning mode, PART_2Nx2N or PART_NxN, may be applied to the coding block.

When the coding block has a minimum size, PART_NxN can be applied. In this article, the minimum size of the coding block can be predefined in the encoder and decoder. Alternatively, information related to the minimum size of the coding block can be signaled through the bitstream. In an example, the minimum size of the coding block can be signaled through the slice header. Therefore, the minimum size of the coding block can be determined differently for each slice.

In another example, the partition mode candidates available for the coding block may be determined differently according to at least one of the size or shape of the coding block. In an example, the number or type of partition mode candidates available for the coding block may be determined differently according to at least one of the size or shape of the coding block.

Alternatively, the type or number of asymmetric partitioning pattern candidates available for the coding block may be determined based on the size or shape of the coding block. The number or type of asymmetric partitioning pattern candidates available for the coding block may be determined differently according to at least one of the size or shape of the coding block. In the example, when the coding block has a non-square shape with a width greater than the height, at least one of PART_2NxN, PART_2NxnU, or PART_2NxnD may not be used as a partitioning pattern candidate for the coding block. When the coding block has a non-square shape with a height greater than the width, at least one of PART_Nx2N, PART_nLx2N, PART_nRx2N may not be used as a partitioning pattern candidate for the coding block.

Generally, the prediction block may have a size of 4x4 to 64x64. However, when the coding block is encoded by inter-frame prediction, the prediction block may be restricted to not have a size of 4x4 to reduce the memory bandwidth when performing motion compensation.

Based on the partition mode, the coding block may be recursively partitioned. In other words, based on the partition mode determined by the partition index, the coding block may be partitioned and each partition generated by partitioning the coding block may be defined as a coding block.

Hereinafter, the method of dividing the coding unit will be described in more detail. In the examples mentioned later, the coding unit may refer to a coding tree unit or a coding unit included in a coding tree unit. In addition, "division" generated by dividing a coding block may refer to a "coding block". The division method mentioned later may be applied when dividing a coding block into a plurality of prediction blocks or transform blocks.

The coding unit may be divided by at least one line. In this case, the angle of the line dividing the coding unit may be a value in the range of 0 to 360 degrees. For example, the angle of the horizontal line may be 0 degrees, the angle of the vertical line may be 90 degrees, the angle of the diagonal line in the upper right direction may be 45 degrees, and the angle of the upper left diagonal line may be 135 degrees.

When the coding unit is divided by multiple lines, all the lines in the multiple lines may have the same angle. Alternatively, at least one of the multiple lines may have an angle different from the other lines. Alternatively, the multiple lines that divide the coding tree unit or the coding unit may have a predefined angle difference (e.g., 90 degrees).

Information about the lines dividing the coding unit may be determined by a division pattern. Alternatively, information about at least one of the number, direction, angle, or position of the lines in the block may be encoded.

For convenience of description, in an example mentioned later, it is assumed that a coding unit is divided into a plurality of coding units by using at least one of a vertical line or a horizontal line.

The number of vertical lines or horizontal lines that divide the coding unit may be at least one or more. In the example, the coding unit may be divided into 2 partitions by using one vertical line or one horizontal line. Alternatively, the coding unit may be divided into 3 partitions by using two vertical lines or two horizontal lines. Alternatively, the coding unit may be divided into 4 partitions by using one vertical line or one horizontal line, and the width and height of the 4 partitions are halved relative to the coding unit.

When a coding unit is divided into multiple partitions by using at least one vertical line or at least one horizontal line, the partitions may have a uniform size. Alternatively, one partition may have a different size from the other partitions, or each partition may have a different size. In the example, when the coding unit is divided by two horizontal lines or two vertical lines, the coding unit may be divided into 3 partitions. In this case, the width ratio or height ratio of the 3 partitions may be n:2n:n, 2n:n:n, or n:n:2n.

In the examples mentioned later, dividing the coding block into 4 partitions is called quadtree-based partitioning. And, dividing the coding block into 2 partitions is called binary tree-based partitioning. In addition, dividing the coding block into 3 partitions is called ternary tree-based partitioning.

In the figures mentioned later, it will be shown that the coding unit is divided using one vertical line and/or one horizontal line, but it will be described that dividing the coding unit into more divisions than shown by using more vertical lines and/or more horizontal lines than shown or dividing the coding unit into fewer divisions than shown is also included in the scope of the present invention.

FIG. 4 shows an example of hierarchical division of coding blocks based on a tree structure as an embodiment to which the present invention is applied.

The input video signal is decoded in predetermined block units, and the basic unit for decoding the input video signal is called a coding block. A coding block may be a unit that performs intra/inter prediction, transformation, and quantization. In addition, a prediction mode (e.g., an intra prediction mode or an inter prediction mode) may be determined in units of coding blocks, and the prediction blocks included in the coding block may share the determined prediction mode. A coding block may be a square block or a non-square block of any size in the range of 8x8 to 64x64, or may be a square block or a non-square block having a size of 128x128, 256x256, or larger.

Specifically, the coding block may be hierarchically divided based on at least one of a quadtree partitioning method, a binary tree partitioning method, or a ternary tree partitioning method. Quadtree-based partitioning may mean a method of dividing a 2Nx2N coding block into four NxN coding blocks. Binary tree-based partitioning may mean a method of dividing one coding block into two coding blocks. Ternary tree-based partitioning may mean a method of dividing one coding block into three coding blocks. Even when ternary tree-based or binary tree-based partitioning is performed, square coding blocks may exist at a lower depth.

The partitions generated by the binary tree-based partitioning may be symmetric or asymmetric. In addition, the coding blocks based on the binary tree partitioning may be square blocks or non-square blocks (eg, rectangular).

5 is a diagram showing the division shape of a coding block based on binary tree division. The division shape of the coding block based on binary tree division may include a symmetric type such as 2NxN (non-square coding unit in the horizontal direction) or Nx2N (non-square coding unit in the vertical direction), or an asymmetric type such as nLx2N, nRx2N, 2NxnU, or 2NxnD. Only one of the symmetric type or the asymmetric type may be allowed as the division shape of the coding block.

The ternary tree partition shape may include at least one of a shape of dividing the coding block into 2 vertical lines or a shape of dividing the coding block into 2 horizontal lines. Three non-square partitions may be generated by the ternary tree partition.

FIG6 shows a ternary tree partition shape.

The ternary tree partition shape may include a shape of dividing the coding block into 2 horizontal lines or a shape of dividing the coding block into 2 vertical lines. The width ratio or height ratio of the partition generated by dividing the coding block may be n:2n:n, 2n:n:n, or n:n:2n.

The position of the partition having the largest width or height among the three partitions may be predefined in the encoder and the decoder. Alternatively, information indicating the partition having the largest width or height among the three partitions may be signaled through a bitstream.

For coding units, only square-shaped or non-square-shaped symmetrical partitions may be allowed. In this case, the partitioning of the coding unit into square partitions may correspond to a quadtree CU partition, and the partitioning of the coding unit into a symmetrically shaped non-square partition may correspond to a binary tree partition. The partitioning of the coding tree unit into a square partition and a symmetrically shaped non-square partition may correspond to a quadtree and binary tree CU partition (QTBT).

Binary tree or ternary tree based partitioning may be performed on a coding block for which quadtree based partitioning is no longer performed. A coding block generated by binary tree or ternary tree based partitioning may be divided into smaller coding blocks. In this case, at least one of quadtree partitioning, ternary tree partitioning, or binary tree partitioning may be set not to be applied to the coding block. Alternatively, for a coding block, binary tree partitioning in a predetermined direction or ternary tree partitioning in a predetermined direction may not be allowed. In an example, for a coding block generated by binary tree or ternary tree based partitioning, quadtree partitioning and ternary tree partitioning may be set to be not allowed. For the coding block, only binary tree partitioning may be allowed.

Alternatively, only the largest coding block among the three coding blocks generated by the ternary tree-based partitioning may be partitioned into smaller coding blocks. Alternatively, only the largest coding block among the three coding blocks generated by the ternary tree-based partitioning may be allowed to be partitioned based on a binary tree or partitioned based on a ternary tree.

The division shape of the lower depth division can be determined accordingly based on the division shape of the higher depth division. In the example, when the higher division and the lower division are divided based on the binary tree, for the lower depth division, only the binary tree-based division of the same shape as the binary tree division shape of the higher depth division can be allowed. For example, when the binary tree division shape of the higher depth division is 2NxN, the binary tree division shape of the lower depth division can also be set to 2NxN. Alternatively, when the binary tree division shape of the higher depth division is Nx2N, the division shape of the lower depth division can also be set to Nx2N.

Alternatively, for the maximum partition among the partitions generated by ternary-tree based partitioning, binary tree partitioning in the same partition direction as a partition at a higher depth or ternary tree partitioning in the same partition direction as a partition at a higher depth may be set to be disallowed.

Alternatively, the partition shape of the partition of the lower depth can be determined by considering the partition shape of the partition of the higher depth and the partition shape of the adjacent partition of the lower depth. Specifically, if the partition of the higher depth is partitioned based on a binary tree, the partition shape of the partition of the lower depth can be determined so that the same result as the result of partitioning the partition of the higher depth based on a quadtree will not occur. In the example, when the partition shape of the partition of the higher depth is 2NxN and the partition shape of the adjacent partition of the lower depth is Nx2N, the partition shape of the current partition of the lower depth may not be set to Nx2N. This is because when the partition shape of the current partition of the lower depth is Nx2N, it leads to the same result as the result of partitioning the partition of the higher depth based on a quadtree of an NxN shape. When the partition shape of the partition of the higher depth is Nx2N and the partition shape of the adjacent partition of the lower depth is 2NxN, the partition shape of the current partition of the lower depth may not be set to 2NxN. In other words, when the binary tree partition shape of a higher depth partition is different from the binary tree partition shape of an adjacent lower depth partition, the binary tree partition shape of the current lower depth partition may be set to be the same as that of the higher depth partition.

Alternatively, the binary tree division shape of the division at a lower depth may be set to be different from the binary tree division shape of the division at a higher depth.

The allowed binary tree partition shape can be determined in units of sequence, slice or coding unit. In an example, the binary tree partition shape allowed by the coding tree unit can be limited to 2NxN or Nx2N shape. The allowed partition shape can be predefined in the encoder or decoder. Alternatively, information about the allowed partition shape or the disallowed partition shape can be encoded and sent by signaling through the bitstream.

FIG. 7 is a diagram showing an example of binary tree-based partitioning that allows only specific shapes.

FIG. 7( a ) shows an example in which only partitioning based on a binary tree in an N×2N shape is allowed, and FIG. 7( b ) shows an example in which only partitioning based on a binary tree in a 2N×N shape is allowed.

In order to represent various partition shapes, information about quadtree partition, information about binary tree partition, or information about ternary tree partition may be used. The information about quadtree partition may include at least one of information indicating whether quadtree-based partition is performed or information about the size/depth of the coding block that allows quadtree-based partition. The information about binary tree partition may include at least one of the following: information indicating whether binary tree-based partition is performed, information about whether binary tree-based partition is vertical or horizontal, information about the size/depth of the coding block that allows binary tree-based partition, or information about the size/depth of the coding block that does not allow binary tree-based partition. The information about ternary tree partition may include at least one of the following: information indicating whether ternary tree-based partition is performed, information about whether ternary tree-based partition is vertical or horizontal, information about the size/depth of the coding block that allows ternary tree-based partition, or information about the size/depth of the coding block that does not allow ternary tree-based partition. The information about the size of the coding block may represent at least one minimum or maximum value of the width, height, product of width and height, or ratio of width to height of the coding block.

In an example, when the width or height of a coding block is smaller than the minimum size of allowed binary tree partitioning, or when the partition depth of the coding block is greater than the maximum depth of allowed binary tree partitioning, binary tree-based partitioning may not be allowed for the coding block.

In an example, when the width or height of a coding block is smaller than the minimum size allowed for ternary tree partitioning, or when the partition depth of the coding block is greater than the maximum depth allowed for ternary tree partitioning, ternary tree-based partitioning may not be allowed for the coding block.

Information about the conditions for allowing binary tree-based or ternary tree-based partitioning may be signaled through a bitstream. The information may be encoded in units of a sequence, a picture, or a partial image. A partial image may mean at least one of a slice, a tile group, a tile, a brick, a coding block, a prediction block, or a transform block.

In an example, the syntax 'max_mtt_depth_idx_minus1' may be encoded/decoded by the bitstream, indicating the maximum depth allowed for binary/ternary tree partitioning. In this case, max_mtt_depth_idx_minus1+1 may indicate the maximum depth allowed for binary/ternary tree partitioning.

In an example, at least one of the number of allowed binary/ternary tree partitions, the maximum depth of allowed binary/ternary tree partitions, or the number of depths of allowed binary/ternary tree partitions may be signaled at a sequence or slice level. Thus, at least one of the number of allowed binary/ternary tree partitions, the maximum depth of allowed binary/ternary tree partitions, or the number of depths of allowed binary/ternary tree partitions may be different for a first slice and a second slice. In an example, while for a first slice, binary/ternary tree partitions may be allowed at only one depth, for a second slice, binary/ternary tree partitions may be allowed at two depths.

In the example shown in FIG. 8 , FIG. 8 shows that binary tree partitioning is performed on a coding unit having a depth of 2 and a coding unit having a depth of 3. Therefore, at least one of the following may be encoded/decoded through a bitstream: information indicating the number of times binary tree partitioning is performed in a coding tree unit (2 times), information indicating the maximum depth (depth 3) of partitions generated by binary tree partitioning in a coding tree unit, or information indicating the number of partition depths (2 depths, depth 2 and depth 3) to which binary tree partitioning is applied in a coding tree unit.

Alternatively, the number of times binary/ternary tree divisions are allowed, the depth of binary/ternary tree divisions allowed, or the number of depths of binary/ternary tree divisions allowed may be predefined in the encoder and decoder. Alternatively, the number of times binary/ternary tree divisions are allowed, the depth of binary/ternary tree divisions allowed, or the number of depths of binary/ternary tree divisions allowed may be determined based on at least one of the index of the sequence or slice or the size/shape of the coding unit. In the example, for the first slice, binary/ternary tree division may be allowed at one depth, and for the second slice, binary/ternary tree division may be allowed at two depths.

In another example, at least one of the number of times binary tree partitioning is allowed, the depth of binary tree partitioning is allowed, or the number of depths of binary tree partitioning is allowed can be set differently according to a temporal level identifier (TemporalID) of a slice or picture. In this article, the temporal level identifier (TemporalID) is used to identify each of a plurality of layers of scalability having at least one or more perspective, space, time, or quality in an image.

As shown in Figure 4, the first coding block 300 with a division depth (split depth) of k can be divided into a plurality of second coding blocks based on a quadtree. For example, the second coding blocks 310 to 340 may be square blocks with a width and a height halved relative to the first coding block, and the division depth of the second coding block may be increased to k+1.

The second coding block 310 with a division depth of k+1 may be divided into a plurality of third coding blocks with a division depth of k+2. The division of the second coding block 310 may be performed by selectively using one of a quadtree or a binary tree according to a division method. In this case, the division method may be determined based on at least one of information indicating a quadtree-based division or information indicating a binary tree-based division.

When the second coding block 310 is divided based on a quadtree, the second coding block 310 may be divided into four third coding blocks 310a having half the width and height of the second coding block, and the division depth of the third coding block 310a may be increased to k+2. In other words, when the second coding block 310 is divided based on a binary tree, the second coding block 310 may be divided into two third coding blocks. In this case, each of the two third coding blocks may be a non-square block with one of the width and height halved relative to the second coding block, and the division depth may be increased to k+2. The second coding block may be determined as a non-square block in the horizontal direction or in the vertical direction according to the division direction, and the division direction may be determined based on information about whether the binary tree-based division is performed in the vertical direction or in the horizontal direction.

Meanwhile, the second coding block 310 may be determined as a leaf coding block that is no longer divided based on the quadtree or the binary tree, and in this case, the corresponding coding block may be used as a prediction block or a transform block.

Similar to the division of the second coding block 310, the third coding block 310a may be determined as a leaf coding block, or may be further divided based on a quadtree or a binary tree.

On the other hand, the third coding block 310b divided based on the binary tree can be further divided into a coding block 310b-2 in the vertical direction or a coding block 310b-3 in the horizontal direction based on the binary tree, and the division depth of the corresponding coding block can be increased to k+3. Alternatively, the third coding block 310b can be determined as a leaf coding block 310b-1 that is no longer divided based on the binary tree, and in this case, the corresponding coding block 310b-1 can be used as a prediction block or a transform block. However, the above-mentioned division process can be restrictively performed based on at least one of the following: information about the size/depth of the coding block that allows quadtree-based division, information about the size/depth of the coding block that allows binary tree-based division, or information about the size/depth of the coding block that does not allow binary tree-based division.

The number of candidates representing the size of the coding block may be limited to a predetermined number, or the size of the coding block in a predetermined unit may have a fixed value. In an example, the size of the coding block in a sequence or in a picture may be limited to any one of 256x256, 128x128, or 32x32. Information representing the size of the coding block in the sequence or in the picture may be signaled in a sequence header or a picture header.

As a result of quadtree and binary tree based partitioning, the coding unit can be represented as a square or rectangular shape of any size.

As shown in Figure 4, the first coding block 300 with a division depth (split depth) of k can be divided into a plurality of second coding blocks based on a quadtree. For example, the second coding blocks 310 to 340 may be square blocks having a width and height half of the first coding block, and the division depth of the second coding block may be increased to k+1.

The second coding block 310 with a division depth of k+1 may be divided into a plurality of third coding blocks with a division depth of k+2. The division of the second coding block 310 may be performed by selectively using one of a quadtree or a binary tree according to a division method. In this case, the division method may be determined based on at least one of information indicating a quadtree-based division or information indicating a binary tree-based division.

When the second coding block 310 is divided based on a quadtree, the second coding block 310 may be divided into four third coding blocks 310a whose width and height are halved relative to the second coding block, and the division depth of the third coding block 310a may be increased to k+2. In other words, when the second coding block 310 is divided based on a binary tree, the second coding block 310 may be divided into two third coding blocks. In this case, each of the two third coding blocks may be a non-square block whose width and height are halved relative to the second coding block, and the division depth may be increased to k+2. The second coding block may be determined as a non-square block in the horizontal direction or in the vertical direction according to the division direction, and the division direction may be determined based on information about whether the binary tree-based division is performed in the vertical direction or in the horizontal direction.

Meanwhile, the second coding block 310 may be determined as a leaf coding block that is no longer divided based on the quadtree or the binary tree, and in this case, the corresponding coding block may be used as a prediction block or a transform block.

Similar to the division of the second coding block 310, the third coding block 310a may be determined as a leaf coding block, or may be further divided based on a quadtree or a binary tree.

On the other hand, the third coding block 310b divided based on the binary tree can be further divided into a coding block 310b-2 in the vertical direction or a coding block 310b-3 in the horizontal direction based on the binary tree, and the division depth of the corresponding coding block can be increased to k+3. Alternatively, the third coding block 310b can be determined as a leaf coding block 310b-1 that is no longer divided based on the binary tree, and in this case, the corresponding coding block 310b-1 can be used as a prediction block or a transform block. However, the above-mentioned division process can be restrictively performed based on at least one of the following: information about the size/depth of the coding block that allows quadtree-based division, information about the size/depth of the coding block that allows binary tree-based division, or information about the size/depth of the coding block that does not allow binary tree-based division.

The number of candidates representing the size of the coding block may be limited to a predetermined number, or the size of the coding block in a predetermined unit may have a fixed value. In an example, the size of the coding block in a sequence or in a picture may be limited to any one of 256x256, 128x128, or 32x32. Information representing the size of the coding block in the sequence or in the picture may be signaled in a sequence header or a picture header.

As a result of quadtree and binary tree based partitioning, the coding unit can be represented as a square or rectangular shape of any size.

The transform skip may be set not to be used for a coding unit generated by binary tree-based partitioning or ternary tree-based partitioning. Alternatively, the transform skip may be set to be applied to at least one of a vertical direction or a horizontal direction in a non-square coding unit. In the example, when the transform skip is applied to the horizontal direction, it means that scaling is performed only in the horizontal direction without performing transform/inverse transform, and transform/inverse transform using DCT or DST is performed in the vertical direction. When the transform skip is applied to the vertical direction, it means that scaling is performed only in the vertical direction without performing transform/inverse transform, and transform/inverse transform using DCT or DST is performed in the horizontal direction.

Information about whether to skip the inverse transform in the horizontal direction or information about whether to skip the inverse transform in the vertical direction may be transmitted by a bitstream signal. In an example, the information about whether to skip the inverse transform in the horizontal direction may be a 1-bit flag 'hor_transform_skip_flag', and the information about whether to skip the inverse transform in the vertical direction may be a 1-bit flag 'ver_transform_skip_flag'.

The encoder may determine whether to encode 'hor_transform_skip_flag' or 'ver_transform_skip_flag' according to the size and/or shape of the current block. In an example, when the current block has an Nx2N shape, hor_transform_skip_flag may be encoded and the encoding of ver_transform_skip_flag may be omitted. When the current block has a 2NxN shape, ver_transform_skip_flag may be encoded and hor_transform_skip_flag may be omitted.

Alternatively, based on the size and/or shape of the current block, it may be determined whether to perform transform skipping in the horizontal direction or whether to perform transform skipping in the vertical direction. In an example, when the current block has an Nx2N shape, transform skipping may be applied to the horizontal direction, and transform/inverse transform may be performed in the vertical direction. When the current block has a 2NxN shape, transform skipping may be applied to the vertical direction, and transform/inverse transform may be performed in the horizontal direction. Transform/inverse transform may be performed based on at least one of DCT or DST.

As a result of partitioning based on a quadtree, a binary tree, or a ternary tree, a coding block that is no longer divided can be used as a prediction block or a transform block. In other words, a coding block generated by quadtree partitioning or binary tree partitioning can be used as a prediction block or a transform block. In the example, a predicted image can be generated in units of coding blocks, and the difference between the residual signal, the original image, and the predicted image can be transformed in units of coding blocks. In order to generate a predicted image in units of coding blocks, motion information can be determined based on the coding blocks, or an intra-frame prediction mode can be determined based on the coding blocks. Therefore, the coding block can be encoded by using at least one of a skip mode, an intra-frame prediction, or an inter-frame prediction.

Alternatively, multiple coding blocks generated by dividing the coding block may be set to share at least one of motion information, merge candidates, reference samples, reference sample rows, or intra-frame prediction modes. In an example, when the coding block is divided by a ternary tree, the divisions generated by dividing the coding block may share at least one of motion information, merge candidates, reference samples, reference sample rows, or intra-frame prediction modes according to the size or shape of the coding block. Alternatively, only a portion of the multiple coding blocks may be set to share information, and the remaining coding blocks may be set not to share information.

In another example, a prediction block or a transform block smaller than the coding block may be used by partitioning the coding block.

Hereinafter, a method of performing inter prediction on a coding block or a prediction block generated by splitting the coding block will be described in detail.

FIG. 9 is a flowchart showing an inter-frame prediction method as an embodiment to which the present invention is applied.

9, the motion information of the current block may be determined S910. The motion information of the current block may include at least one of a motion vector of the current block, a reference picture index of the current block, an inter-prediction direction of the current block, or a weight of weighted prediction. The weight of weighted prediction may represent a weight applied to the L0 reference block and a weight applied to the L1 reference block.

The motion vector of the current block may be determined based on information signaled by the bitstream. The precision of the motion vector represents the basic unit used to express the motion vector of the current block. For example, the precision of the motion vector of the current block may be determined as one of integer pixels, 1/2 pixels, 1/4 pixels, or 1/8 pixels. The precision of the motion vector may be determined on a per-picture basis, on a per-slice basis, on a per-tile group basis, on a per-tile basis, or on a per-block basis. A block may represent a coding tree unit, a coding unit, a prediction unit, or a transform unit.

The motion information of the current block may be obtained based on at least one of information signaled through a bitstream or motion information of a neighboring block adjacent to the current block.

FIG. 10 is a diagram illustrating a process of deriving motion information of a current block when a merge mode is applied to the current block.

The merge mode indicates a method of deriving motion information of a current block from neighboring blocks.

When the merge mode is applied to the current block, a spatial merge candidate may be derived according to the spatial neighboring blocks of the current block S1010. The spatial neighboring blocks may include at least one of the blocks adjacent to the upper boundary, left boundary, or corner (e.g., at least one of the upper left corner, upper right corner, or lower left corner) of the current block.

FIG. 11 is a diagram showing an example of spatially adjacent blocks.

As an example shown in FIG. 11, the spatial neighboring blocks may include at least one of the following: a neighboring block _A1 adjacent to the left side of the current block, a neighboring block B1 adjacent to the upper side of the current block, a neighboring block _A0 adjacent to the lower left corner of the current block, a neighboring block _B0 adjacent to the upper right corner of the current block, and a neighboring block _B2 adjacent to the upper left corner of the current block. For example, assuming that the position of the upper left corner sample of the current block is (0, 0), the width of the current block is W, and the height of the current block is H. Block _A1 may include a sample at a position (-1, H-1). Block _B1 may include a sample at a position (W-1, -1). Block _A0 may include a sample at a position (-1, H). Block _B0 may include a sample at a position (W, -1). Block _B2 may include a sample at a position (-1, -1).

Further extending the example of FIG. 11 , the spatial merge candidate may be derived based on a block adjacent to the upper left sample of the current block or a block adjacent to the upper center sample of the current block. For example, the block adjacent to the upper left sample of the current block may include at least one of a block including a sample at position (0, -1) or a block including a sample at position (-1, 0). Alternatively, the spatial merge candidate may be derived based on at least one of a block adjacent to the upper center sample of the current block or a block adjacent to the left center sample of the current block. For example, a block adjacent to the upper center sample of the current block may include a sample at position (W/2, -1). A block adjacent to the left center sample of the current block may include a sample at position (-1, H/2).

Based on the size and/or shape of the current block, the position of the upper adjacent block and/or the left adjacent block for deriving the spatial merge candidate can be determined. In the example, when the size of the current block is greater than a threshold, the spatial merge candidate can be derived based on the block adjacent to the upper center sample of the current block and the block adjacent to the left center sample of the current block. On the other hand, when the size of the current block is less than the threshold, the spatial merge candidate can be derived based on the block adjacent to the upper right sample of the current block and the block adjacent to the lower left sample of the current block. In this article, the size of the current block can be expressed based on at least one of the width, height, the sum of the width and the height, the product of the width and the height, or the ratio of the width to the height. The threshold can be an integer, such as 2, 4, 8, 16, 32, or 128.

According to the shape of the current block, the availability of the extended spatial neighboring blocks can be determined. In the example, when the current block is a non-square block with a width greater than a height, it can be determined that the block adjacent to the upper left sample of the current block, the block adjacent to the left center sample, or the block adjacent to the lower left sample of the current block is unavailable. At the same time, when the current block is a block with a height greater than a width, it can be determined that the block adjacent to the upper left sample of the current block, the block adjacent to the upper center sample, or the block adjacent to the upper right sample of the current block is unavailable.

The motion information of the spatial merging candidate may be set to be the same as the motion information of the spatial neighboring blocks.

The spatial merging candidate may be determined by searching the adjacent blocks in a predetermined order. In an example, in the example shown in FIG. 11 , the search may be performed in the order of blocks A ₁ , B ₁ , B ₀ , A ₀ , and B ₂ for determining the spatial merging candidate. Herein, when at least one of the remaining blocks (i.e., A ₁ , B ₁ , B ₀ , and A ₀ ) does not exist or at least one is encoded by an intra prediction mode, block B ₂ may be used.

The order of searching for spatial merging candidates may be predefined in the encoder/decoder. Alternatively, the order of searching for spatial merging candidates may be adaptively determined according to the size or shape of the current block. Alternatively, the order of searching for spatial merging candidates may be determined based on information signaled through the bitstream.

A temporal merge candidate may be derived from the temporal neighboring blocks of the current block S1020. The temporal neighboring blocks may refer to the co-located blocks included in the co-located picture. The co-located picture has a different POC from the current picture including the current block. The co-located picture may be determined as a picture with a predefined index in a reference picture list or a picture with the smallest POC difference from the current picture. Alternatively, the co-located picture may be determined by information signaled by a bitstream. The information signaled by the bitstream may include at least one of: information indicating a reference picture list (e.g., an L0 reference picture list or an L1 reference picture list) including the co-located picture and an index indicating the co-located picture in the reference picture list. Information for determining the co-located picture may be signaled in at least one of a picture parameter set, a slice header, and a block level.

The motion information about the temporal merge candidate may be determined based on the motion information of the co-located block. In an example, the motion vector of the temporal merge candidate may be determined based on the motion vector of the co-located block. For example, the motion vector of the temporal merge candidate may be set to be the same as the motion vector of the co-located block. Alternatively, the motion vector of the temporal merge candidate may be derived by scaling the motion vector of the co-located block based on at least one of the POC difference between the current picture of the current block and the reference picture and the POC difference between the co-located co-located picture and the reference picture.

FIG. 12 is a diagram showing an example of deriving a motion vector of a temporal merging candidate.

In the example shown in Figure 12, tb represents the POC difference between the current picture curr_pic and the reference picture curr_ref of the current picture, and td represents the POC difference between the co-located picture col_pic and the reference picture col_ref of the co-located block. The motion vector of the temporal merge candidate can be derived by scaling the motion vector of the co-located block col_PU based on tb and/or td.

Alternatively, considering whether the co-located block is available, the motion vector of the co-located block and the motion vector obtained by scaling the motion vector of the co-located block can be used as the motion vector of the temporal merge candidate. In an example, the motion vector of the co-located block is set as the motion vector of the first temporal merge candidate, and the value obtained by scaling the motion vector of the co-located block can be set as the motion vector of the second temporal merge candidate.

The inter-frame prediction direction of the temporal merge candidate may be set to be the same as the inter-frame prediction direction of the temporal neighboring block. However, the reference picture index of the temporal merge candidate may have a fixed value. In the example, the reference picture index of the temporal merge candidate may be set to "0". Alternatively, the reference picture index of the temporal merge candidate may be adaptively determined based on at least one of the reference picture index of the spatial merge candidate and the reference picture index of the current picture.

A specific block having the same position and size as the current block within the co-located picture, or a block adjacent to a block adjacent to a block having the same position and size as the current block may be determined as the co-located block.

FIG. 13 is a diagram showing positions of candidate blocks that can be used as co-located blocks.

The candidate blocks may include at least one of: a block adjacent to the upper left corner of the current block in the same picture, a block adjacent to the center sample of the current block in the same picture, and a block adjacent to the lower left corner of the current block in the same picture.

In an example, the candidate blocks may include at least one of: a block TL within the co-located picture including the position of the upper left sample of the current block, a block BR within the co-located picture including the position of the lower right sample of the current block, a block H within the co-located picture adjacent to the lower right corner of the current block, a block C3 within the co-located picture including the position of the center sample of the current block, and a block C0 within the co-located picture adjacent to the center sample of the current block (e.g., a block including the position of a sample separated from the center sample of the current block by (-1, -1)).

In addition to the example shown in FIG. 13 , a block within the co-located picture including a position of a neighboring block adjacent to a predetermined boundary of the current block may be selected as the co-located block.

The number of temporal merging candidates may be 1 or more. In an example, at least one temporal merging candidate may be derived based on at least one co-located block.

Information about the maximum number of temporal merge candidates may be encoded by an encoder and sent by a signal. Alternatively, the maximum number of temporal merge candidates may be derived based on the maximum number of merge candidates that may be included in the merge candidate list and/or the maximum number of spatial merge candidates. Alternatively, the maximum number of temporal merge candidates may be determined based on the number of available co-located blocks.

Whether the candidate block is available can be determined according to a predetermined priority, and at least one co-located block can be determined based on the above determination and the maximum number of candidates for temporal merging. In the example, when block C3 including the position of the center sample of the current block and block H adjacent to the lower right corner of the current block are candidate blocks, any one of block C3 and block H can be determined as a co-located block. When block H is available, block H can be determined as a co-located block. However, when block H is not available (for example, when block H is encoded by intra-frame prediction, when block H is not available, or when block H is located outside the maximum coding unit (LCU), etc.), block C3 can be determined as a co-located block.

In another example, when at least one of the plurality of blocks adjacent to the lower right corner position of the current block in the co-located picture is unavailable (e.g., block H and/or block BR), the unavailable block may be replaced with another available block. The other available block replacing the unavailable block may include at least one block adjacent to the center sample position of the current block in the co-located picture (e.g., C0 and/or C3) and a block adjacent to the lower left corner of the current block in the co-located picture (e.g., TL).

When at least one of the multiple blocks adjacent to the center sample position of the current block in the same picture is unavailable, or when at least one of the multiple blocks adjacent to the upper left corner position of the current block in the same picture is unavailable, the unavailable block can be replaced by another available block.

Subsequently, a merge candidate list including a spatial merge candidate and a temporal merge candidate may be generated (S1030). When configuring the merge candidate list, a merge candidate having the same motion information as an existing merge candidate may be removed from the merge candidate list.

Information about the maximum number of merge candidates may be signaled via the bitstream. In an example, information indicating the maximum number of merge candidates may be signaled via a sequence parameter or a picture parameter. In an example, when the maximum number of merge candidates is six, a total of six may be selected from the spatial merge candidates and the temporal merge candidates. For example, five spatial merge candidates may be selected from five merge candidates, and one temporal merge candidate may be selected from two temporal merge candidates.

Alternatively, the maximum number of merge candidates may be predefined in the encoder and the decoder. For example, the maximum number of merge candidates may be 2, 3, 4, 5, or 6. Alternatively, the maximum number of merge candidates may be determined based on at least one of whether merge with MVD (MMVD) is performed, whether combined prediction is performed, or whether triangulation is performed.

If the number of merge candidates included in the merge candidate list is less than the maximum number of merge candidates, the merge candidates included in the second merge candidate list may be added to the merge candidate list.

The second merge candidate list may include merge candidates derived based on motion information of a block encoded/decoded by inter-frame prediction before the current block. In an example, if motion compensation of a block whose coding mode is inter-frame prediction is performed, the merge candidate derived based on the motion information of the block may be added to the second merge candidate list. If encoding/decoding of the current block is completed, the motion information of the current block may be added to the second merge candidate list for inter-frame prediction of subsequent blocks.

The second merge candidate list may be initialized in units of CTU, tile, or slice. The maximum number of merge candidates that may be included in the second merge candidate list may be predefined in the encoder and decoder. Alternatively, information indicating the maximum number of merge candidates that may be included in the second merge candidate list may be signaled via a bitstream.

The indexes of the merge candidates included in the second merge candidate list may be determined based on the order of addition to the second merge candidate list. In an example, the index assigned to the Nth merge candidate added to the second merge candidate list may have a smaller value than the index assigned to the N+1th merge candidate added to the second merge candidate list. For example, the index of the N+1th merge candidate may be set to a value increased by 1 relative to the index of the Nth merge candidate. Alternatively, the index of the Nth merge candidate may be set to the index of the N+1th merge candidate, and the value of the index of the Nth merge candidate may be subtracted by 1.

Alternatively, the index assigned to the Nth merge candidate added to the second merge candidate list may have a greater value than the index assigned to the N+1th merge candidate added to the second merge candidate list. For example, the index of the Nth merge candidate may be set to the index of the N+1th merge candidate, and the value of the index of the Nth merge candidate may be increased by 1.

Based on whether the motion information of the block on which motion compensation is performed is the same as the motion information of the merge candidate included in the second merge candidate list, it can be determined whether to add the merge candidate derived from the block to the second merge candidate list. In the example, when a merge candidate having the same motion information as the block is included in the second merge candidate list, the merge candidate derived based on the motion information of the block may not be added to the second merge candidate list. Alternatively, when a merge candidate having the same motion information as the block is included in the second merge candidate list, the merge candidate may be deleted from the second merge candidate list, and the merge candidate derived based on the motion information of the block may be added to the second merge candidate list.

When the number of merge candidates included in the second merge candidate list is the same as the maximum number of merge candidates, the merge candidate with the lowest index or the merge candidate with the highest index may be deleted from the second merge candidate list, and the merge candidate derived based on the motion information of the block may be added to the second merge candidate list. In other words, after deleting the oldest merge candidate among the merge candidates included in the second merge candidate list, the merge candidate derived based on the motion information of the block may be added to the second merge candidate list.

When the number of merge candidates included in the merge candidate list has not reached the maximum number of merge candidates, a combined merge candidate obtained by combining two or more merge candidates or a merge candidate having a (0, 0) motion vector (zero motion vector) may be included in the merge candidate list.

Alternatively, an average merge candidate in which the average value of the motion vectors of two or more merge candidates is found may be added to the merge candidate list. The average merge candidate may be derived by finding the average value of the motion vectors of two or more merge candidates included in the merge candidate list. In the example, when the first merge candidate and the second merge candidate are added to the merge candidate list, the average value of the motion vector of the first merge candidate and the motion vector of the second merge candidate may be calculated so as to obtain the average merge candidate. In detail, the L0 motion vector of the average merge candidate may be derived by calculating the average value of the L0 motion vector of the first merge candidate and the L0 motion vector of the second merge candidate, and the L1 motion vector of the average merge candidate may be derived by calculating the average value of the L1 motion vector of the first merge candidate and the L1 motion vector of the second merge candidate. When bidirectional prediction is applied to any one of the first merge candidate and the second merge candidate, and unidirectional prediction is performed on the other merge candidate, the motion vector of the bidirectional merge candidate may be set as is to the L0 motion vector or the L1 motion vector of the average merge candidate. In the example, when the L0 direction prediction and the L1 direction prediction are performed on the first merge candidate, but the L0 direction prediction is performed on the second merge candidate, the L0 motion vector of the average merge candidate can be derived by calculating the average of the L0 motion vector of the first merge candidate and the L0 motion vector of the second merge candidate. At the same time, the L1 motion vector of the average merge candidate can be derived as the L1 motion vector of the first merge candidate.

When the reference picture of the first merge candidate is different from the second merge candidate, the motion vector of the first merge candidate or the second merge candidate may be scaled according to the distance between the reference picture of each merge candidate and the current picture (i.e., POC difference). For example, after scaling the motion vector of the second merge candidate, the average merge candidate may be derived by calculating the average of the motion vector of the first merge candidate and the scaled motion vector of the second merge candidate. In this article, priority may be set based on the value of the reference picture index of each merge candidate, the distance between the reference picture of each merge candidate and the current block, or whether bidirectional prediction is applied, and scaling may be applied to the motion vector of a merge candidate with a high (or low) priority.

The reference picture index of the average merge candidate may be set to indicate a reference picture at a specific position within the reference picture list. In an example, the reference picture index of the average merge candidate may indicate the first reference picture or the last reference picture within the reference picture list. Alternatively, the reference picture index of the average merge candidate may be set to be the same as the reference picture index of the first merge candidate or the second merge candidate. In an example, when the reference picture index of the first merge candidate is the same as the second merge candidate, the reference picture index of the average merge candidate may be set to be the same as the reference picture index of the first merge candidate and the second merge candidate. When the reference picture index of the first merge candidate is different from the second merge candidate, the priority may be set based on the value of the reference picture index of each merge candidate, the distance between the reference picture of each merge candidate and the current block, or whether bidirectional prediction is applied, and the reference picture index of the merge candidate with high (or low) priority may be set to the reference picture index of the average merge candidate. In an example, when bidirectional prediction is applied to the first merge candidate and unidirectional prediction is applied to the second merge candidate, the reference picture index of the first merge candidate to which bidirectional prediction is applied may be determined as the reference picture index of the average merge candidate.

Based on the priority between the combinations of merge candidates, a sequence of combinations for generating average merge candidates can be determined. The priorities can be predefined in the encoder and the decoder. Alternatively, the sequence of combinations can be determined based on whether bidirectional prediction of the merge candidates is performed. For example, a combination of merge candidates encoded using bidirectional prediction can be set to have a higher priority than a combination of merge candidates encoded using unidirectional prediction. Alternatively, the sequence of combinations can be determined based on the reference pictures of the merge candidates. For example, a combination of merge candidates with the same reference picture can have a higher priority than a combination of merge candidates with different reference pictures.

The merge candidates may be included in the merge candidate list according to a predefined priority. A merge candidate with a high priority may be assigned a small index value. In the example, the spatial merge candidate may be added to the merge candidate list before the temporal merge candidate. In addition, the spatial merge candidates may be added to the merge candidate list in the order of the spatial merge candidates of the left adjacent block, the spatial merge candidates of the upper adjacent block, the spatial merge candidates of the block adjacent to the upper right corner, the spatial merge candidates of the block adjacent to the lower left corner, and the spatial merge candidates of the block adjacent to the upper left corner. Alternatively, it may be arranged so that the spatial merge candidate derived from the adjacent block adjacent to the upper left corner of the current block (B2 of FIG. 11) is added to the merge candidate list later than the temporal merge candidate.

In another example, the priority between the merge candidates may be determined according to the size or shape of the current block. In the example, when the current block has a rectangular shape with a width greater than a height, the spatial merge candidate of the left neighboring block may be added to the merge candidate list before the spatial merge candidate of the upper neighboring block. In other words, when the current block has a rectangular shape with a height greater than a width, the spatial merge candidate of the upper neighboring block may be added to the merge candidate list before the spatial merge candidate of the left neighboring block.

In another example, the priority between the merge candidates can be determined according to the motion information of each merge candidate. In the example, the merge candidate with bidirectional motion information can have a higher priority than the merge candidate with unidirectional motion information. Therefore, the merge candidate with bidirectional motion information can be added to the merge candidate list before the merge candidate with unidirectional motion information.

In another example, a merge candidate list may be generated according to a predefined priority, and then the merge candidates may be rearranged. The rearrangement may be performed based on the motion information of the merge candidate. In the example, the rearrangement may be performed based on whether the merge candidate has bidirectional motion information, the size of the motion vector, the precision of the motion vector, or the POC difference between the reference picture of the merge candidate and the current picture. In detail, a merge candidate with bidirectional motion information may be rearranged to have a higher priority than a merge candidate with unidirectional motion information. Alternatively, a merge candidate with a motion vector having a precision value of fractional pixels may be rearranged to have a higher priority than a merge candidate with a motion vector having a precision of integer pixels.

When generating a merge candidate list, at least one of the merge candidates included in the merge candidate list may be specified based on the merge candidate index ( S1040 ).

The motion information of the current block may be set to be the same as the motion information of the merge candidate specified by the merge candidate index (S1050). In the example, when a spatial merge candidate is selected by the merge candidate index, the motion information of the current block may be set to be the same as the motion information of the spatial neighboring block. Alternatively, when a temporal merge candidate is selected by the merge candidate index, the motion information of the current block may be set to be the same as the motion information of the temporal neighboring block.

FIG. 14 is a diagram illustrating a process of deriving motion information of a current block when the AMVP mode is applied to the current block.

When the AMVP mode is applied to the current block, at least one of the inter prediction direction and the reference picture index of the current block may be decoded from the bitstream S1410. In other words, when the AMVP mode is applied, at least one of the inter prediction direction and the reference picture index of the current block may be determined based on information encoded by the bitstream.

A spatial motion vector candidate may be determined based on the motion vector of the spatial neighboring block of the current block S1420. The spatial motion vector candidate may include at least one of the following: a first spatial motion vector candidate derived from an upper neighboring block of the current block and a second spatial motion vector candidate derived from a left neighboring block of the current block. In this article, the upper neighboring block may include at least one of the blocks adjacent to the upper side and upper right corner of the current block, and the left neighboring block of the current block includes at least one of the blocks adjacent to the left side and lower left corner of the current block. The block adjacent to the upper left corner of the current block may be used as an upper neighboring block or may be used as a left neighboring block.

Alternatively, a spatial motion vector candidate may be derived from a spatial non-adjacent block that is not adjacent to the current block. In an example, a spatial motion vector candidate for the current block may be derived by using at least one of: a block located on the same vertical line as a block adjacent to the upper side, upper right corner, or upper left corner of the current block; a block located on the same horizontal line as a block adjacent to the left side, lower left corner, or upper left corner of the current block; and a block located on the same diagonal line as a block adjacent to a corner of the current block. When spatially adjacent blocks are not available, spatial motion vector candidates may be derived by using spatially non-adjacent blocks.

In another example, at least two spatial motion vector candidates may be derived by using spatial adjacent blocks and spatial non-adjacent blocks. In an example, a first spatial motion vector candidate and a second spatial motion vector candidate may be derived by using adjacent blocks adjacent to the current block. Meanwhile, a third spatial motion vector candidate and/or a fourth spatial motion vector candidate may be derived based on blocks that are not adjacent to the current block but are adjacent to the above adjacent blocks.

When the current block is different from the spatial neighboring blocks on the reference picture, the spatial motion vector may be obtained by scaling the motion vectors of the spatial neighboring blocks. A temporal motion vector candidate may be determined based on the motion vectors of the temporal neighboring blocks of the current block S1430. When the current block is different from the temporal neighboring blocks on the reference picture, the temporal motion vector may be obtained by scaling the motion vectors of the temporal neighboring blocks. In this article, when the number of spatial motion vector candidates is equal to or less than a predetermined number, the temporal motion vector candidate may be derived.

A motion vector candidate list including spatial motion vector candidates and temporal motion vector candidates may be generated S1440.

When generating the motion vector candidate list, at least one of the motion vector candidates included in the motion vector candidate list may be specified based on information specifying at least one of the motion vector candidate list S1450.

The motion vector candidate specified by the information may be set as a prediction value of the motion vector of the current block, and the motion vector of the current block may be obtained by adding the residual value of the motion vector and the prediction value of the motion vector S1460. Herein, the residual value of the motion vector may be parsed through the bitstream.

When the motion information of the current block is obtained, motion compensation of the current block may be performed based on the obtained motion information S920. In detail, motion compensation of the current block may be performed based on the inter prediction direction, the reference picture index, and the motion vector of the current block. The inter prediction direction indicates whether L0 prediction, L1 prediction, or bi-prediction is performed. When the current block is encoded by bi-prediction, a prediction block of the current block may be obtained based on a weighted sum operation or an average operation of the L0 reference block and the L1 reference block.

When the prediction sample is obtained by performing motion compensation, the current block can be reconstructed based on the generated prediction sample. In detail, the reconstructed sample can be obtained by adding the prediction sample of the current block to the residual sample.

As in the above example, based on the motion information of the block encoded/decoded using inter-frame prediction before the current block, the merge candidate of the current block can be derived. For example, based on the motion information of the neighboring block at the predefined position adjacent to the current block, the merge candidate of the current block can be derived. Examples of the neighboring blocks may include at least one of the following: a block adjacent to the left side of the current block, a block adjacent to the upper side of the current block, a block adjacent to the upper left corner of the current block, a block adjacent to the upper right corner of the current block, and a block adjacent to the lower left corner of the current block.

The merging candidate of the current block can be derived based on the motion information of the blocks other than the neighboring blocks. For ease of description, the neighboring block at a predefined position adjacent to the current block is referred to as a first merging candidate block, and the block at a position different from the first merging candidate block is referred to as a second merging candidate block.

The second merge candidate block may include at least one of the following: a block encoded/decoded using inter-frame prediction before the current block, a block adjacent to the first merge candidate block, or a block located on the same line as the first merge candidate block. FIG. 15 shows a second merge candidate block adjacent to the first merge candidate block, and FIG. 16 shows a second merge candidate block located on the same line as the first merge candidate block.

When the first merge candidate block is not available, the merge candidate derived based on the motion information of the second merge candidate block is added to the merge candidate list. Alternatively, even if at least one of the spatial merge candidate and the temporal merge candidate is added to the merge candidate list, when the number of merge candidates included in the merge candidate list is less than the maximum number of merge candidates, the merge candidate derived based on the motion information of the second merge candidate block is added to the merge candidate list.

FIG. 15 is a diagram showing an example of deriving a merge candidate from a second merge candidate block when a first merge candidate block is unavailable.

When the first merge candidate block AN (herein, N ranges from 0 to 4) is unavailable, a merge candidate of the current block is derived based on motion information of a second merge candidate block BM (herein, M ranges from 0 to 6). That is, the merge candidate of the current block can be derived by replacing the unavailable first merge candidate block with the second merge candidate block.

Among the blocks adjacent to the first merge candidate block, a block placed in a predefined direction from the first merge candidate block may be set as a second merge candidate block. The predefined direction may be a left direction, a right direction, an upward direction, a downward direction, or a diagonal direction. The predefined direction may be set for each first merge candidate block. For example, the predefined direction of the first merge candidate block adjacent to the left side of the current block may be a left direction. The predefined direction of the first merge candidate block adjacent to the upper side of the current block may be an upward direction. The predefined direction of the first merge candidate block adjacent to the corner of the current block may include at least one of a left direction, an upward direction, or a diagonal direction.

For example, when A0 adjacent to the left side of the current block is not available, a merge candidate for the current block is derived based on B0 adjacent to A1. When A1 adjacent to the upper side of the current block is not available, a merge candidate for the current block is derived based on B1 adjacent to A1. When A2 adjacent to the upper right corner of the current block is not available, a merge candidate for the current block is derived based on B2 adjacent to A2. When A3 adjacent to the lower left corner of the current block is not available, a merge candidate for the current block is derived based on B3 adjacent to A3. When A4 adjacent to the upper left corner of the current block is not available, a merge candidate for the current block is derived based on at least one of B4 to B6 adjacent to A4.

The example shown in FIG15 is only used to describe the embodiments of the present invention, and does not limit the present invention. The position of the second merge candidate block may be set to be different from the sample shown in FIG15. For example, the second merge candidate block adjacent to the first merge candidate block may be positioned in the upward direction or the downward direction of the first merge candidate block, and the first merge candidate block is adjacent to the left side of the current block. Alternatively, the second merge candidate block adjacent to the first merge candidate block may be positioned in the left direction or the right direction of the first merge candidate block, and the first merge candidate block is adjacent to the upper side of the current block.

FIG. 16 is a diagram showing an example of deriving a merge candidate from a second merge candidate block located on the same line as a first merge candidate block.

The blocks located on the same line as the first merge candidate block may include at least one of the following: a block located on the same horizontal line as the first merge candidate block, a block located on the same vertical line as the first merge candidate block, or a block located on the same diagonal line as the first merge candidate block. The y coordinate positions of the blocks located on the same horizontal line are the same. The x coordinate positions of the blocks located on the same vertical line are the same. The difference between the x coordinate positions of the blocks located on the same diagonal line is the same as the difference between the y coordinate positions.

Assume that the upper left sample of the current block is located at (0, 0), and the width and height of the current block are W and H, respectively. In FIG. 18, it is shown that the position of the second merge candidate block (e.g., B4, C6) located on the same vertical line as the first merge candidate block is determined based on the rightmost block on the upper side of the coding block (e.g., block A1 including coordinates (W-1, -1)). In addition, in FIG. 18, it is shown that the position of the second merge candidate block (e.g., B1, C1) located on the same horizontal line as the first merge candidate block is determined based on the lowest block on the left side of the coding block (e.g., block A0 including coordinates (-1, H-1)).

In another example, the position of the second merge candidate block may be determined based on the leftmost block on the upper side of the coding block (e.g., a block including coordinates (0, -1)) or a block located at the upper center of the coding block (e.g., a block including coordinates (W/2, -1)). In addition, the position of the second merge candidate block may be determined based on the uppermost block on the left side of the coding block (e.g., a block including coordinates (-1, 0)) or a block located at the left center of the coding block (e.g., a block including coordinates (-1, H/2)).

In another example, when there are multiple upper adjacent blocks adjacent to the upper side of the current block, the second merge candidate block can be determined by using all or part of the multiple upper adjacent blocks. In the example, the second merge candidate block can be determined by using a block at a specific position among the multiple upper adjacent blocks (for example, at least one of the upper adjacent block located at the leftmost side, the upper adjacent block located at the rightmost side, or the upper adjacent block located at the center). The number of upper adjacent blocks used to determine the second merge candidate block among the multiple upper adjacent blocks can be 1, 2, 3 or more. In addition, when there are multiple left adjacent blocks adjacent to the left side of the current block, the second merge candidate block can be determined by using all or part of the multiple left adjacent blocks. In the example, the second merge candidate block can be determined by using a block at a specific position among the multiple left adjacent blocks (for example, at least one of the left adjacent block located at the bottom, the left adjacent block located at the top, or the left adjacent block located at the center). The number of left adjacent blocks used to determine the second merge candidate block among the multiple left adjacent blocks can be 1, 2, 3 or more.

Depending on the size and/or shape of the current block, the position and/or number of the upper adjacent blocks and/or the left adjacent blocks used to determine the second merge candidate block can be determined differently. In the example, when the size of the current block is greater than the threshold, the second merge candidate block can be determined based on the upper center block and/or the left center block. On the other hand, when the size of the current block is less than the threshold, the second merge candidate block can be determined based on the upper right block and/or the lower left block. The threshold can be an integer, such as 8, 16, 32, 64, or 128.

A first merge candidate list and a second merge candidate list may be constructed, and motion compensation of the current block may be performed based on at least one of the first merge candidate list or the second merge candidate list.

The first merge candidate list may include at least one of: a spatial merge candidate derived based on motion information of a neighboring block at a predefined position adjacent to the current block, or a temporal merge candidate derived based on motion information of a co-located block.

The second merge candidate list may include merge candidates derived based on the motion information of the second merge candidate block.

As an embodiment of the present invention, a first merge candidate list may be constructed to include merge candidates derived from a first merge candidate block, and a second merge candidate list may be constructed to include merge candidates derived from a second merge candidate block. In an example, in the example shown in FIG. 15 , merge candidates derived from blocks A0 to A4 may be added to the first merge candidate list, and merge candidates derived from blocks B0 to B6 may be added to the second merge candidate list. In an example, in the example shown in FIG. 16 , merge candidates derived from blocks A0 to A4 may be added to the first merge candidate list, and merge candidates derived from blocks B0 to B5, C0 to C7 may be added to the second merge candidate list.

Alternatively, the second merge candidate list may include merge candidates derived based on motion information of a block encoded/decoded using inter-frame prediction before the current block. For example, when motion compensation of a block whose coding mode is inter-frame prediction is performed, a merge candidate derived based on the motion information of the block is added to the second merge candidate list. When encoding/decoding of the current block is completed, the motion information of the current block is added to the second merge candidate list for inter-frame prediction of subsequent blocks.

The indexes of the merge candidates included in the second merge candidate list may be determined based on the order in which the merge candidates are added to the second merge candidate list. For example, the index assigned to the Nth merge candidate added to the second merge candidate list may have a lower value than the index assigned to the N+1th merge candidate added to the second merge candidate list. For example, the index of the N+1th merge candidate may be set to have a value that is 1 higher than the index of the Nth merge candidate. Alternatively, the index of the Nth merge candidate may be set to the index of the N+1th merge candidate, and the value of the index of the Nth merge candidate is reduced by 1.

Alternatively, the index assigned to the Nth merge candidate added to the second merge candidate list may have a higher value than the index assigned to the N+1th merge candidate added to the second merge candidate list. For example, the index of the Nth merge candidate may be set to the index of the N+1th merge candidate, and the value of the index of the Nth merge candidate is increased by 1.

Based on whether the motion information of the block subjected to motion compensation is the same as the motion information of the merge candidate included in the second merge candidate list, it can be determined whether to add the merge candidate derived from the block to the second merge candidate list. For example, when a merge candidate having the same motion information as the block is included in the second merge candidate list, the merge candidate derived based on the motion information of the block is not added to the second merge candidate list. Alternatively, when a merge candidate having the same motion information as the block is included in the second merge candidate list, the merge candidate is deleted from the second merge candidate list, and the merge candidate derived based on the motion information of the block is added to the second merge candidate list.

When the number of merge candidates included in the second merge candidate list is the same as the maximum number of merge candidates, a merge candidate with a lowest index or a merge candidate with a highest index is detected from the second merge candidate list, and a merge candidate derived based on the motion information of the block is added to the second merge candidate list. That is, after deleting the oldest merge candidate among the merge candidates included in the second merge candidate list, a merge candidate derived based on the motion information of the block may be added to the second merge candidate list.

The second merge candidate list may be initialized in units of CTU, tile, or slice. In other words, a block different from the current block included in the CTU, tile, or slice may be set to be unavailable as a second merge candidate block. The maximum number of merge candidates that may be included in the second merge candidate list may be predefined in the encoder and the decoder. Alternatively, information indicating the maximum number of merge candidates that may be included in the second merge candidate list may be signaled via a bitstream.

The first merge candidate list or the second merge candidate list may be selected, and the selected merge candidate list may be used to perform inter prediction of the current block. Specifically, based on the index information, any one of the merge candidates included in the merge candidate list may be selected, and motion information of the current block may be obtained from the merge candidate.

Information specifying the first merge candidate list or the second merge candidate list may be signaled through the bitstream, and the decoder may select the first merge candidate list or the second merge candidate list based on the information.

Alternatively, among the first merge candidate list and the second merge candidate list, a merge candidate list including a larger number of available merge candidates may be selected.

Alternatively, the first merge candidate list or the second merge candidate list may be selected based on at least one of the size, shape, and split depth of the current block.

Alternatively, the merge candidate list is configured by adding (or appending) the other one of the first merge candidate list and the second merge candidate list to any one of the first merge candidate list and the second merge candidate list.

For example, inter prediction may be performed based on a merge candidate list having at least one merge candidate included in a first merge candidate list and at least one merge candidate included in a second merge candidate list.

For example, the merge candidates included in the second merge candidate list may be added to the first merge candidate list. Alternatively, the merge candidates included in the first merge candidate list may be added to the second merge candidate list.

When the number of merge candidates included in the first merge candidate list is less than the maximum number, or when the first merge candidate block is unavailable, merge candidates included in the second merge candidate list are added to the first merge candidate list.

Alternatively, when the first merge candidate block is not available, a merge candidate derived from a block adjacent to the first merge candidate block among the merge candidates included in the second merge candidate list is added to the first merge candidate list. Referring to FIG. 15 , when A0 is not available, a merge candidate derived based on motion information of B0 among the merge candidates included in the second merge candidate list is added to the first merge candidate list. When A1 is not available, a merge candidate derived based on motion information of B1 among the merge candidates included in the second merge candidate list is added to the first merge candidate list. When A2 is not available, a merge candidate derived based on motion information of B2 among the merge candidates included in the second merge candidate list is added to the first merge candidate list. When A3 is not available, a merge candidate derived based on motion information of B3 among the merge candidates included in the second merge candidate list is added to the first merge candidate list. When A4 is not available, a merge candidate derived based on motion information of B4, B5 or B6 among the merge candidates included in the second merge candidate list is added to the first merge candidate list.

Alternatively, the merge candidate to be added to the first merge candidate list may be determined according to the priority of the merge candidates included in the second merge candidate list. The priority may be determined based on the index value assigned to each merge candidate. For example, when the number of merge candidates included in the first merge candidate list is less than the maximum number, or when the first merge candidate block is unavailable, the merge candidate with the smallest index value or the merge candidate with the largest index value among the merge candidates included in the second merge candidate list is added to the first merge candidate list.

When a merge candidate having the same motion information as a merge candidate with the highest priority among the merge candidates included in the second merge candidate list is included in the first merge candidate list, the merge candidate with the highest priority may not be added to the first merge candidate list. In addition, it may be determined whether a merge candidate with the next priority (for example, a merge candidate to which an index value greater than the index value assigned to the merge candidate with the highest priority is assigned by 1 or a merge candidate to which an index value less than the index value assigned to the merge candidate with the highest priority is assigned by 1) can be added to the first merge candidate list.

Alternatively, a merge candidate list including merge candidates derived based on motion information of the first merge candidate block and merge candidates derived based on motion information of the second merge candidate block may be generated. The merge candidate list may be a combination of the first merge candidate list and the second merge candidate list.

For example, according to a predetermined search order, a merge candidate list may be generated by searching the first merge candidate block and the second merge candidate block.

17 to 20 are diagrams showing an order of searching for merge candidate blocks.

17 to 20 illustrate the order of searching for merging candidates as follows.

A0→A1→A2→A3→A4→B0→B1→B2→B3→B4→(B5)→(B6).

Only when block B4 is unavailable or when the number of merge candidates included in the merge candidate list is equal to or less than a preset number, the search for blocks B5 and B6 is performed.

A search order different from the examples shown in FIGS. 17 to 20 may be set.

A merge candidate list may be generated that includes a combination of at least one merge candidate included in a first merge candidate list and at least one merge candidate included in a second merge candidate list. For example, the combined merge candidate list may include N merge candidates among the merge candidates included in the first merge candidate list and M merge candidates among the merge candidates included in the second merge candidate list. The letters N and M may represent the same number or different numbers. Alternatively, at least one of N and M may be determined based on at least one of the number of merge candidates included in the first merge candidate list and the number of merge candidates included in the second merge candidate list. Alternatively, information for determining at least one of N and M may be signaled via a bitstream. Any one of N and M may be derived by subtracting the other of N and M from the maximum number of merge candidates in the combined merge candidate list.

The merge candidates to be added to the combined merge candidate list may be determined according to a predefined priority. The predefined priority may be determined based on an index assigned to the merge candidate.

Alternatively, the merge candidates to be added to the combined merge candidate list may be determined based on the association between the merge candidates. For example, when A0 included in the first merge candidate list is added to the combined merge candidate list, a merge candidate (e.g., B0) at a position adjacent to A0 is not added to the combined merge list.

When the number of merge candidates included in the first merge candidate list is less than N, more than M merge candidates among the merge candidates included in the second merge candidate list are added to the combined merge candidate list. For example, when N is 4 and M is 2, 4 merge candidates among the merge candidates included in the first merge candidate list are added to the combined merge candidate list, and 2 merge candidates among the merge candidates included in the second merge candidate list are added to the combined merge candidate list. When the number of merge candidates included in the first merge candidate list is less than 4, two or more merge candidates among the merge candidates included in the second merge candidate list are added to the combined merge candidate list. When the number of merge candidates included in the second merge candidate list is less than 2, four or more merge candidates among the merge candidates included in the first merge candidate list are added to the combined merge candidate list.

That is, the value of N or M may be adjusted according to the number of merge candidates included in each merge candidate list. By adjusting the value of N or M, the total number of merge candidates included in the combined merge candidate list may be fixed. When the total number of merge candidates included in the combined merge candidate list is less than the maximum number of merge candidates, a combined merge candidate, an average merge candidate, or a zero motion vector candidate is added.

Motion compensation of the current block may be performed by using at least one of the merge candidates included in the first merge candidate list and the second merge candidate list. The encoder may encode index information for specifying any one of a plurality of merge candidates. In an example, 'merge_idx' may specify any one of a plurality of merge candidates. In an example, Table 1 represents a merge index of each merge candidate derived from the first merge candidate block and the second merge candidate block shown in FIG. 16.

【Table 1】

However, as the number of merge candidates included in the merge candidate list increases, the codeword used to encode the merge index becomes longer. Therefore, there is a problem of reducing encoding/decoding efficiency. In order to reduce the length of the codeword, the merge index can be determined by using a prefix and a suffix. In the example, the merge index can be determined by using merge_idx_prefix representing the prefix of the merge index and merge_idx_suffix representing the suffix of the merge index.

Table 2 shows the merged index prefix value and the merged index suffix value of each merged index, and Table 3 shows the process of determining the merged index based on the merged index prefix value and the merged index suffix value.

【Table 2】

【Table 3】

As shown in Tables 2 and 3, when the merge index prefix value is less than the threshold value, the merge index may be set to be the same as the merge index prefix value. On the other hand, when the merge index prefix value is greater than the threshold value, the merge index may be determined by subtracting a reference value from the merge index prefix and adding a merge index suffix to a value that shifts the result. The reference value may be a threshold value, or may be a value obtained by subtracting 1 from the threshold value.

Tables 2 and 3 show that the threshold is 4. The threshold may be determined based on at least one of the number of merge candidates included in the merge candidate list, the number of second merge candidate blocks, or the number of lines including the second merge candidate blocks. Alternatively, the threshold may be predefined in the encoder and the decoder.

Whether to use the prefix and suffix to determine the merge index may be determined according to the number of merge candidates included in the merge candidate list or the maximum number of merge candidates that can be included in the merge candidate list. In an example, when the maximum number of merge candidates that can be included in the merge candidate list is greater than a threshold, a merge index prefix and a merge index suffix used to determine the merge index may be sent with a signal. On the other hand, when the maximum number of merge candidates is less than the threshold, the merge index may be sent with a signal.

The rectangular block may be divided into a plurality of triangular blocks. Merging candidates of the triangular blocks may be derived based on the rectangular block including the triangular blocks. The triangular blocks may share the same merging candidate.

A merge index may be signaled for each triangle block. In this case, the triangle blocks may be set to not use the same merge candidate. In an example, a merge candidate for a first triangle block may not be used as a merge candidate for a second triangle block. Thus, the merge index for the second triangle block may specify any one of the remaining merge candidates other than the merge candidate selected for the first triangle block.

The merge candidate may be derived based on a block having a predetermined shape or a predetermined size or larger. When the current block is not in the predetermined shape, or when the size of the current block is smaller than the predetermined size, a merge candidate of the current block is derived based on a block including the current block and in the predetermined shape or in the predetermined size or larger. The predetermined shape may be a square shape or a non-square shape.

When the predetermined shape is a square shape, a merge candidate of the non-square shaped coding unit is derived based on the square shaped coding units including the non-square shaped coding units.

FIG. 21 is a diagram showing an example of deriving merging candidates of non-square blocks based on square blocks.

Merge candidates for non-square blocks can be derived based on square blocks including non-square blocks. For example, merge candidates for non-square shaped coding block 0 and non-square shaped coding block 1 can be derived based on square shaped blocks including coding block 0 and coding block 1. That is, the position of the spatially adjacent blocks can be determined based on the position, width/height or size of the square shaped blocks. Merge candidates for coding block 0 and coding block 1 can be derived based on at least one of the spatially adjacent blocks A0, A1, A2, A3 and A4 adjacent to the square shaped blocks.

Temporal merge candidates may be determined based on square-shaped blocks. That is, temporal neighboring blocks may be determined based on the position, width/height, or size of the square-shaped blocks. For example, merge candidates for coding block 0 and coding block 1 may be derived based on temporal neighboring blocks determined based on square-shaped blocks.

Alternatively, any one of the spatial merge candidate and the temporal merge candidate may be derived based on a square block, and the other merge candidate may be derived based on a non-square block. For example, a spatial merge candidate for coding block 0 may be derived based on a square block, and a temporal merge candidate for coding block 0 may be derived based on coding block 0.

A plurality of blocks included in a block of a predetermined shape or a predetermined size or larger may share a merge candidate. For example, in the example shown in FIG. 21 , at least one of a spatial merge candidate and a temporal merge candidate of coding block 0 and coding block 1 may be the same.

The predetermined shape may be a non-square shape, such as 2NxN, Nx2N, etc. When the predetermined shape is a non-square shape, a merge candidate of the current block may be derived based on a non-square block including the current block. For example, when the current block is in a 2Nxn shape (herein, n is 1/2N), a merge candidate of the current block is derived based on a non-square block of a 2NxN shape. Alternatively, when the current block is in an nx2N shape, a merge candidate of the current block is derived based on a non-square block of an Nx2N shape.

Information indicating a predetermined shape or a predetermined size may be signaled through a bitstream. For example, information indicating any one of a non-square shape or a square shape may be signaled through a bitstream.

Alternatively, the predetermined shape or the predetermined size may be determined according to a rule predefined in an encoder and a decoder.

When the child node does not meet the predetermined condition, a merge candidate of the child node is derived based on the parent node that meets the predetermined condition. In this article, the predetermined condition may include at least one of the following: whether the block is a block generated due to quadtree partitioning, whether the size of the block is exceeded, the shape of the block and the picture boundary, and whether the depth difference between the child node and the parent node is equal to or greater than a predetermined value.

For example, the predetermined condition may include whether the block is a block generated due to quadtree partitioning, and whether the block is a coded block of a square shape of a predetermined size or larger. When the current block is generated by binary tree partitioning or ternary tree partitioning, a merge candidate of the current block is derived based on a high-level node block that includes the current block and satisfies the predetermined condition. When there is no high-level node block that satisfies the predetermined condition, a merge candidate of the current block is derived based on the current block, a block that includes the current block and is of a predetermined size or larger, or a high-level node block that includes the current block and has a depth difference of one with the current block.

FIG. 22 is a diagram showing an example of deriving merge candidates based on high-level node blocks.

Block 0 and block 1 are generated by partitioning a square block based on a binary tree. Merge candidates of block 0 and block 1 may be derived based on neighboring blocks (i.e., at least one of A0, A1, A2, A3, and A4) determined from high-level node blocks including block 0 and block 1. Therefore, block 0 and block 1 may use the same spatial merge candidate.

A high-level node block including block 2, block 3, and block 4 may be generated by partitioning a square block based on a binary tree. In addition, block 2 and block 3 may be generated by partitioning a non-square shaped block based on a binary tree. Merge candidates of non-square shaped blocks 2, block 3, and block 4 may be derived based on a high-level node block including non-square shaped blocks 2, block 3, and block 4. That is, a merge candidate may be derived based on an adjacent block (e.g., at least one of B0, B1, B2, B3, and B4) determined according to the position, width/height, or size of the square block including blocks 2, block 3, and block 4. Therefore, blocks 2, block 3, and block 4 may use the same spatial merge candidate.

Temporal merge candidates for non-square shaped blocks may be derived based on high-level node blocks. For example, temporal merge candidates for blocks 0 and 1 may be derived based on a square block including blocks 0 and 1. Temporal merge candidates for blocks 2, 3, and 4 may be derived based on a square block including blocks 2, 3, and 4. In addition, the same temporal merge candidates derived from temporal neighboring blocks determined based on each quadtree block may be used.

The lower-level node blocks included in the higher-level node block may share at least one of the spatial merge candidate and the temporal merge candidate. For example, the lower-level node blocks included in the higher-level node block may use the same merge candidate list.

Alternatively, at least one of the spatial merge candidate and the temporal merge candidate may be derived based on the low-level node block, and the other may be derived based on the high-level node block. For example, the spatial merge candidate of block 0 and block 1 may be derived based on the high-level node block. However, the temporal merge candidate of block 0 may be derived based on block 0, and the temporal merge candidate of block 1 may be derived based on block 1.

Alternatively, when the number of samples included in the low-level node block is less than a predefined number, a merge candidate is derived based on a high-level node block including a predefined number or more of samples. For example, when at least one of the following conditions is met: at least one of the low-level node blocks generated based on at least one of quadtree partitioning, binary tree partitioning, and ternary tree partitioning is less than a preset size; at least one of the low-level node blocks is a non-square block; the high-level node block does not exceed the picture boundary; and the width or height of the high-level node block is equal to or greater than a predefined value, a merge candidate is derived based on a square or non-square high-level node block including a predefined number or more of samples (e.g., 64, 128, or 256 samples). The low-level node blocks included in the high-level node block can share the merge candidates derived based on the high-level node block.

A merge candidate can be derived based on any one of the low-level node blocks, and other low-level node blocks can be set to use the merge candidate. The low-level node blocks may be included in blocks of a predetermined shape or a predetermined size or larger. For example, the low-level node blocks may share a merge candidate list derived based on any one of the low-level node blocks. Information of the low-level node blocks serving as the basis for derivation of the merge candidate may be signaled via a bitstream. The information may be index information indicating any one of the low-level node blocks. Alternatively, the low-level node blocks serving as the basis for derivation of the merge candidate may be determined based on at least one of the position, size, shape, and scanning order of the low-level node blocks.

Information indicating whether a low-level node block shares a merge candidate list derived based on a high-level node block may be signaled through a bitstream. Based on this information, it may be determined whether a merge candidate for a block that is not in a predetermined shape or a block that is less than a predetermined size is derived based on a high-level node block including the block. Alternatively, it may be determined whether to derive a merge candidate based on a high-level node block according to rules predefined in an encoder and a decoder.

When there is a neighboring block adjacent to the current block within a predefined area, it is determined that the neighboring block is not available as a spatial merging candidate. The predefined area may be a parallel processing area defined for parallel processing between blocks. The parallel processing area may be referred to as a merge estimation region (MER). For example, when a neighboring block adjacent to the current block is included in the same merge estimation region as the current block, it is determined that the neighboring block is not available. A shift operation may be performed to determine whether the current block and the neighboring block are included in the same merge estimation region. Specifically, based on whether the value obtained by shifting the position of the upper left reference sample of the current block is the same as the value obtained by shifting the position of the upper left reference sample of the adjacent block, it may be determined whether the current block and the adjacent block are included in the same merge estimation region.

FIG. 23 is a diagram showing an example of determining the availability of spatially neighboring blocks based on a merged estimated region.

In FIG. 23 , it is shown that the merged estimation area is in an Nx2N shape.

A merge candidate for block 1 may be derived based on a spatial neighboring block adjacent to block 1. The spatial neighboring blocks may include B0, B1, B2, B3, and B4. Herein, it may be determined that the spatial neighboring blocks B0 and B3 included in the same merge estimation region as block 1 are not available as merge candidates. Therefore, a merge candidate for block 1 may be derived from at least one of the spatial neighboring blocks B1, B2, and B4 other than the spatial neighboring blocks B0 and B3.

A merge candidate for block 3 may be derived based on a spatial neighboring block adjacent to block 3. The spatial neighboring blocks may include C0, C1, C2, C3, and C4. Herein, it may be determined that the spatial neighboring block C0 included in the same merge estimation region as block 3 is not available as a merge candidate. Therefore, a merge candidate for block 3 may be derived from at least one of the spatial neighboring blocks C1, C2, C3, and C4 other than the spatial neighboring block C0.

Based on at least one of the position, size, width, and height of the merge estimation region, a merge candidate of a block included in the merge estimation region may be derived. For example, a merge candidate of a plurality of blocks included in the merge estimation region may be derived from at least one of a spatial neighboring block and a temporal neighboring block determined based on at least one of the position, size, width, and height of the merge estimation region. The blocks included in the merge estimation region may share the same merge candidate.

FIG. 24 is a diagram showing an example of deriving a merge candidate based on a merge estimation region.

When a plurality of coding units are included in a merge estimation region, merge candidates of the plurality of coding units may be derived based on the merge estimation region. That is, by using the merge estimation region as a coding unit, merge candidates may be derived based on the position, size, or width/height of the merge estimation region.

For example, merge candidates of both coding unit 0 (CU0) and coding unit 1 (CU1), which are both in (n/2)xN (herein, n is N/2) size and included in the merge estimation region in the (N/2)xN size, may be derived based on the merge estimation region. That is, merge candidates of coding unit 0 and coding unit 1 may be derived based on at least one of neighboring blocks C0, C1, C2, C3, and C4 adjacent to the merge estimation region.

For example, merging candidates of coding unit 2 (CU2), coding unit 3 (CU3), coding unit 4 (CU4), and coding unit 5 (CU5) in an nxn size included in a merging estimation region in an NxN size may be derived based on the merging estimation region. That is, merging candidates of coding unit 2, coding unit 3, coding unit 4, and coding unit 5 may be derived based on at least one of neighboring blocks C0, C1, C2, C3, and C4 adjacent to the merging estimation region.

The shape of the merged estimation area may be a square shape or a non-square shape. For example, it may be determined that a square-shaped coding unit (or prediction unit) or a non-square-shaped coding unit (or prediction unit) is a merged estimation area. The ratio between the width and height of the merged estimation area may be limited to not exceed a predetermined range. For example, the merged estimation area cannot have a non-square shape with a ratio between width and height exceeding 2, or a non-square shape with a ratio between width and height less than 1/2. That is, the non-square merged estimation area may be in the shape of 2NxN or Nx2N. Information about the limitation on the ratio between width and height may be signaled via a bitstream. Alternatively, the limitation on the ratio between width and height may be predefined in the encoder and decoder.

At least one of information indicating the shape of the merged estimation region and information indicating the size of the merged estimation region may be signaled through the bitstream. For example, at least one of information indicating the shape of the merged estimation region and information indicating the size of the merged estimation region may be signaled through a slice header, a tile group header, a picture parameter, or a sequence parameter.

The shape of the merged estimation area or the size of the merged estimation area may be updated on a per-sequence basis, on a per-picture basis, on a per-slice basis, on a per-tile group basis, on a per-tile basis, or on a per-block (CTU) basis. When the shape of the merged estimation area or the size of the merged estimation area is different from the shape or size of the previous unit, information indicating the new shape of the merged estimation area or the new size of the merged estimation area is signaled through the bitstream.

At least one block may be included in the merged estimation area. The blocks included in the merged estimation area may be in a square shape or a non-square shape. The maximum number or minimum number of blocks that the merged estimation area can include may be determined. For example, three, four or more CUs may be included in the merged estimation area. The determination may be based on information signaled by the bitstream. Alternatively, the maximum number or minimum number of blocks that the merged estimation area can include may be predefined in the encoder and the decoder.

In at least one of a case where the number of blocks included in the merge estimation region is less than the maximum number and a case where the number is greater than the minimum number, parallel processing of the blocks may be allowed. For example, when the number of blocks included in the merge estimation region is equal to or less than the maximum number, or when the number of blocks included in the merge estimation region is equal to or greater than the minimum number, a merge candidate of the block is derived based on the merge estimation region. When the number of blocks included in the merge estimation region is greater than the maximum number, or when the number of blocks included in the merge estimation region is less than the minimum number, a merge candidate of each block in the block is derived based on the size, position, width, or height of each block in the block.

The information indicating the shape of the merged estimation region may include a 1-bit flag. For example, the syntax "isrectagular_mer_flag" may indicate a square-shaped or non-square-shaped merged candidate region. An isrectagular_mer_flag value of 1 may indicate a non-square-shaped merged estimation region, while an isrectagular_mer_flag value of 0 may indicate a square-shaped merged estimation region.

When the information indicates a non-square shaped merged estimation region, information indicating at least one of a width, a height, and a ratio between the width and the height of the merged estimation region is signaled through the bitstream. Based on this, the size and/or shape of the merged estimation region can be determined.

Next, according to an embodiment of the present disclosure, an inter-frame prediction method based on an affine motion model will be described in detail.

The motion of an object can be categorized into translational motion, rotational motion, and affine motion. Translational motion refers to the linear movement of an object. Rotational motion refers to the rotational movement of an object. Affine motion means that the movement, rotation, or displacement is different for each part of the object.

FIG. 25 is a diagram showing a motion model.

Figure 25(a) shows a translational motion model. Figure 25(b) shows a rotational motion model. Figure 25(c) shows an affine motion model.

In the translational motion model, the linear motion of an object can be represented by 2D coordinates and/or motion vectors (MVx, MVy).

In the rotational motion model, the motion of an object can be represented by a predetermined rotation angle.

On the other hand, in the affine motion model, since the motion of the object is nonlinear, at least one of the movement, rotation or displacement may be different for each part of the object. In the affine motion model, the motion of the object may be represented based on the adjustment of the movement, rotation or displacement of each part of the object. For example, the affine motion model may be represented by using predetermined parameters (a to f), as in the following formula 1.

[Formula 1]

x′＝ax+by+e

y′＝cx+dy+f

In Formula 1, x and y represent positions of samples included in the current block, and x' and y' represent positions of samples included in the reference block. An affine motion vector ( _Vx , _Vy ) that is a position difference between samples in the current block and samples of the reference block may be represented as in Formula 2.

[Formula 2]

Vx＝x-x′

Vy＝y-y′

The affine motion vector (V _x , V _y ) may be derived by using Formula 1 and Formula 2 as in Formula 3 below.

[Formula 3]

Vx＝(1-a)x-by-e

y＝(1-d)y-cx-f

The complex motion of an object can be expressed by an affine motion model. However, compared with a translation motion model or a rotation motion model, in an affine motion model, the encoding/decoding efficiency may be reduced because more parameters should be encoded to express the motion of an object. In order to solve this problem, when motion prediction and/or motion compensation are performed based on an affine motion model, a motion vector of a current block angle may be used. Specifically, after the current block is divided into a plurality of sub-blocks, a motion vector of each divided sub-block may be determined by using a plurality of angular motion vectors. Hereinafter, an affine inter-frame prediction method using an angular motion vector will be described in detail.

FIG. 26 is a diagram showing an affine motion model using angular motion vectors.

In the affine motion model, a motion vector of at least one of the upper left corner, the upper right corner, the lower left corner, or the lower right corner of the current block may be used. In an example, motion prediction and/or motion compensation of the current block may be performed by using a motion vector v0 at the upper left corner of the current block and a motion vector v1 at the upper right corner of the current block. Alternatively, motion prediction and/or motion compensation of the current block may be performed by using a motion vector v0 at the upper left corner of the current block, a motion vector v1 at the upper right corner of the current block, and a motion vector v2 at the lower left corner of the current block.

When three angular motion vectors are used, it can be expressed that the current block is changed into a rectangle whose two sides are formed by connecting pixels indicated by the three angular motion vectors of the current block in the reference picture. In other words, a rectangle whose two sides are formed by connecting pixels indicated by the three angular motion vectors in the reference picture can be estimated as the reference block of the current block.

When three angular motion vectors are used, the motion vector of each subblock in the current block may be determined based on Formula 4 below.

[Formula 4]

In Formula 4, w and h represent the width and height of the current block, respectively. x and y represent the position of the sub-block. The position of the sub-block may be the position of a predefined sample in the sub-block. The predefined sample may be an upper left sample, a lower left sample, an upper right sample, or a center sample.

When three angular motion vectors are used, the motion vector of the subblock is derived based on a total of 6 motion vector parameters. Therefore, affine inter prediction using three angular motion vectors can be referred to as 6-parameter mode.

When two angular motion vectors are used, it can be expressed that the current block is changed into a rectangle whose sides are formed by connecting pixels indicated by two angular motion vectors of the current block in the reference picture. In other words, a rectangle whose sides are formed by connecting pixels indicated by two angular motion vectors in the reference picture can be estimated as a reference block of the current block.

As in the example shown in FIG. 26 , the relationship of the following Formula 5 is established among the motion vector of the lower left corner, the motion vector of the upper left corner, and the motion vector of the upper right corner.

[Formula 5]

v _2x = v _0x + v _0y - v _1y

v _2y = -v _0x +v _0y +v _1x

Based on Formula 5, Formula 4 may be changed as in the following Formula 6.

[Formula 6]

When two angular motion vectors are used, the motion vector of each subblock in the current block may be determined based on Formula 6.

When two angular motion vectors are used, the motion vector of the sub-block is derived based on a total of 4 motion vector parameters. Therefore, affine inter prediction using two angular motion vectors can be called 4-parameter mode.

FIG. 27 is a diagram showing an example of determining a motion vector in units of sub-blocks.

The current block may be divided into a plurality of sub-blocks, and a motion vector of each sub-block may be obtained based on the angular motion vector. Based on the motion vector of the sub-block, motion compensation of the sub-block may be performed.

The number of sub-blocks in the current block may be determined based on the size and/or shape of the current block. In this case, the size of the current block may be represented by width, height, the sum of the width and the height, or the product of the width and the height. The shape of the current block may represent at least one of whether the current block is square, whether the current block is non-square, whether the width is greater than the height, or the ratio of the width to the height.

In the example, when the size of the current block is less than a threshold value, the current block may be divided into sub-blocks having a first size. On the other hand, when the size of the current block is greater than a threshold value, the current block may be divided into sub-blocks having a second size. In this case, the threshold value may be an integer such as 8, 16, 32, or 64. The first size may be 4x4, 8x8, or 16x16, and the second size may be a value greater than the first size. For a detailed example, when the size of the current block is 16x16, the current block may be divided into four sub-blocks having a size of 8x8. On the other hand, when the size of the current block is greater than 16x16 (e.g., 32x32 or 64x64), the current block may be divided into sub-blocks having a size of 16x16.

Alternatively, a sub-block of a fixed size may be used regardless of the size of the current block. In this case, the number of sub-blocks included in the current block may be different depending on the size of the current block.

Alternatively, information indicating the size of the sub-block may be signaled in the bitstream. This information may be signaled at a picture level, a slice level, or a block level.

Information for determining at least one of the position of a corner or the number of motion vector parameters for affine inter prediction may be signaled in a bitstream. In an example, information indicating whether the number of motion vector parameters is 4 or 6 may be signaled in a bitstream. When the information indicates that the number of motion vector parameters is 4, motion compensation may be performed by using a motion vector of an upper left corner and a motion vector of an upper right corner. When the information indicates that the number of motion vector parameters is 6, motion compensation may be performed by using a motion vector of an upper left corner, a motion vector of an upper right corner, and a motion vector of a lower left corner.

Alternatively, information for specifying the positions of three corners among the four corners of the current block or information for specifying the positions of two corners among the four corners of the current block may be signaled in a bitstream.

In another example, the number of motion vector parameters may be determined based on at least one of the size, shape, or inter prediction mode of the current block. In this case, the size of the current block may be represented by width, height, the sum of the width and the height, or the product of the width and the height. The shape of the current block may represent at least one of whether the current block is square, whether the current block is non-square, whether the width is greater than the height, or the ratio of the width to the height. The inter prediction mode includes a merge mode and/or an AMVP mode.

In an example, the number of motion vector parameters may be determined differently depending on whether the current block is a square. In an example, when the current block is a square (e.g., NxN), affine prediction may be performed by using three angular motion vectors. When the current block is a non-square (e.g., Nx2N or 2NxN), affine prediction may be performed by using two angular motion vectors.

Alternatively, the number of motion vector parameters may be determined based on the result of comparing the size of the current block with a threshold. In an example, when the size of the current block is greater than the threshold, affine prediction may be performed by using three angular motion vectors. When the size of the current block is less than the threshold, affine prediction may be performed by using two angular motion vectors. The threshold may be a predefined value in an encoder/decoder. Alternatively, information representing the threshold may be signaled in a bitstream. The information may be signaled at a sequence level, a picture level, a slice level, or a block level.

Alternatively, the motion vector of the upper left corner may be used as a default value, and whether to use the motion vector of the upper right corner and/or whether to use the motion vector of the lower left corner may be determined based on the width and/or height of the current block. In an example, whether to use the motion vector of the upper right corner may be determined based on whether the width of the current block is equal to or greater than a threshold. In addition, whether to use the motion vector of the lower left corner may be determined based on whether the width of the current block is equal to or greater than a threshold.

In another example, the position of the corner may be determined based on at least one of the size, shape, or inter-prediction mode of the current block.

FIG. 28 is a diagram showing an example of determining the position of a corner according to the shape of a current block.

When the current block has a shape (Nx2N) with a height longer than a width, affine prediction can be performed by using a motion vector of the upper left corner and a motion vector of the lower left corner. (Refer to FIG. 28(a))

When the current block has a shape (2NxN) with a width longer than a height, affine prediction can be performed by using a motion vector of the upper left corner and a motion vector of the upper right corner. (Refer to FIG. 28(b))

In the embodiments mentioned later, the angular motion vector is referred to as an affine vector. In addition, each affine vector at multiple angles can be identified by prefixing an ordinal number to the affine vector. In the example, the first affine vector represents the affine vector of the first angle, the second affine vector represents the affine vector of the second angle, and the third affine vector represents the affine vector of the third angle. In addition, the motion vector of each sub-block derived based on the affine vector is referred to as an affine motion vector.

When motion compensation of a current block (eg, a coding block or a prediction block) is performed based on an affine motion vector, it may be defined as an affine inter mode.

Based on the size and/or shape of the current block, it may be determined whether the affine inter mode is allowed. In an example, the affine inter mode may not be allowed when the size of the current block is less than a threshold. Alternatively, the affine inter mode may not be allowed when the current block is non-square. In a detailed example, the affine inter mode may be allowed only when the current block is square and its size is equal to or greater than 16x16.

Information indicating whether the affine inter mode is applied to the current block may be signaled in a bitstream. The information may be signaled when the coding mode of the current block is determined to be the inter mode. When the affine inter mode is not allowed for the current block, encoding of the information may be omitted. The information may be a 1-bit flag. In an example, when the flag is 1, it indicates that the affine inter mode is applied, and when the flag is 0, it indicates that the affine inter mode is not applied. Alternatively, the information may be an index indicating any one of a plurality of motion models.

Hereinafter, the motion compensation process based on the affine inter mode will be described in detail.

FIG. 29 is a flowchart showing the motion compensation process in the affine inter mode.

At least one of the position of the corner or the number of angular motion vector parameters for the affine inter mode may be determined S2910. Information for determining at least one of the position of the corner or the number of angular motion vectors may be signaled in a bitstream. Alternatively, at least one of the position of the corner or the number of angular motion vectors may be determined based on at least one of the size, shape, or inter prediction mode of the current block.

Based on the motion vector of the neighboring block, an affine vector at each corner may be derived S2920. Specifically, the affine vector at each corner may be derived from the affine vector of the neighboring block or the translation motion vector of the neighboring block.

30 and 31 are diagrams showing examples of deriving an affine vector of a current block based on motion vectors of neighboring blocks.

Fig. 30 shows a 4-parameter mode, and Fig. 31 shows a 6-parameter mode. Fig. 30(a) shows a 4-parameter mode using an affine vector at the upper left corner and an affine vector at the upper right corner, and Fig. 30(b) shows a 4-parameter mode using an affine vector at the upper left corner and an affine vector at the lower left corner.

The affine vector of the corner can be obtained based on the motion vector of the neighboring block adjacent to the corner. In this case, the neighboring block may include at least one of the neighboring block adjacent to the upper side of the corner, the neighboring block adjacent to the left side of the corner, or the neighboring block adjacent to the diagonal direction of the corner.

In the examples shown in FIG. 30( a), FIG. 30( b), and FIG. 31 , the affine vector of the upper left corner may be derived based on at least one of the neighboring blocks A, B, or C adjacent to the upper left corner. The affine vector of the upper right corner may be derived based on at least one of the neighboring blocks D or E adjacent to the upper right corner. The affine vector of the lower left corner may be derived based on at least one of the neighboring blocks F or G adjacent to the lower left corner.

After searching the adjacent blocks in a predefined order, the affine vector of the corner can be derived based on the motion vector of the adjacent block that is first determined to be available. In the examples shown in Figures 30 (a), 30 (b) and 31, the affine vector of the upper left corner can be derived based on the motion vector of the adjacent block that is first determined to be available when the adjacent blocks A, B and C adjacent to the upper left corner are sequentially searched. The affine vector of the upper right corner can be derived based on the motion vector of the adjacent block that is first determined to be available when the adjacent blocks D and E adjacent to the upper right corner are sequentially searched. The affine vector of the lower left corner can be derived based on the motion vector of the adjacent block that is first determined to be available when the adjacent blocks F and G adjacent to the lower left corner are sequentially searched.

Alternatively, information for specifying any one of the neighboring blocks close to each corner may be signaled in the bitstream. In an example, information for identifying one of the neighboring blocks adjacent to the upper left corner, information for identifying one of the neighboring blocks adjacent to the upper right corner, or information for identifying one of the neighboring blocks adjacent to the lower left corner may be signaled in the bitstream. The affine vector of the corner may be determined based on the motion vector of the neighboring block specified by the information.

The affine vector may be derived based on a non-adjacent block that is not adjacent to the corner. The non-adjacent block may include at least one of a block that is on the same line as the block adjacent to the corner or a block that is not adjacent to the corner among blocks adjacent to the current block. In an example, in the examples shown in FIGS. 30 and 31 , the affine vector of the upper left corner may be derived based on a block other than adjacent blocks A, B, and C (e.g., adjacent blocks D, E, F, or G) or a block on the same line as adjacent blocks A, B, or C.

The affine vector of the angle may be set to be the same as the motion vector of the neighboring block. Alternatively, the motion vector of the neighboring block may be set to a motion vector predictor, and the affine vector of the angle may be derived by adding a motion vector difference to the motion vector predictor. A motion vector difference representing the difference between the affine vector and the motion vector predictor may be signaled in a bitstream.

In another example, the second affine vector can be derived based on the difference encoding of the first affine vector and the second affine vector. To this end, the difference vector representing the difference between the second affine vector and the first affine vector can be encoded in the bitstream. The decoder can derive the second affine vector by adding the difference vector decoded in the bitstream to the first affine vector. When using the affine vectors of three angles, the third affine vector can be derived based on the difference encoding between the third affine vector and the first affine vector or the difference encoding between the third affine vector and the second affine vector.

In another example, when the value of the affine vector of the first corner is the same as the value of the affine vector of the second corner, or when the adjacent blocks used to derive the affine vector of the first corner and the affine vector of the second corner are the same, the affine vector of the second corner can be derived from a block that is not adjacent to the second corner. In the example shown in FIG. 30( a), when the affine vector of the upper left corner and the affine vector of the upper right corner are the same, the affine vector of the upper right corner can be derived based on the motion vector of the block F or G that is not adjacent to the upper right corner.

Alternatively, when the value of the affine vector of the first angle is the same as the value of the affine vector of the second angle, or when the adjacent block used to derive the affine vector of the first angle and the adjacent block used to derive the affine vector of the second angle are the same, the affine vector of the second angle can be derived based on the affine vector of the first angle. Specifically, the second affine vector can be derived by adding an offset to the first affine vector or subtracting an offset from the first affine vector or by scaling the first affine vector based on a predefined scaling factor.

Information indicating whether the affine vectors are identical may be signaled in a bitstream. The information may include at least one of a flag indicating whether there are identical affine vectors or an index for identifying a position of an angle at which the affine vectors are identical to each other.

Depending on the shape of the current block, the number and/or position of the candidate adjacent blocks for deriving the affine vector may be different. In the example, depending on the shape of the prediction unit (whether it is a square shape or a non-square shape), the candidate blocks for deriving the angular affine vector may be constructed differently. In the example, when the current block is a square as in the example shown in FIG. 30 (a), the affine vector of the upper left corner may be derived based on any one of the adjacent blocks A, B or C. On the other hand, when the current block is a non-square as in the example shown in FIG. 30 (b), the affine vector of the upper left corner may be derived based on any one of the adjacent blocks A, B or F.

According to an embodiment of the present disclosure, an affine merge mode for deriving an affine vector of a current block may be defined. The affine merge mode represents a method of deriving motion information of a merge candidate as motion information of the current block. In this case, the motion information may include at least one of a motion vector, a reference picture index, a reference picture list, or a prediction direction. The merge candidate may include at least one of a spatial merge candidate or a temporal merge candidate. The spatial merge candidate represents a merge candidate derived from a block included in the same picture as the current block, and the temporal merge candidate represents a merge candidate derived from a block included in a different picture from the current block.

FIG. 32 is a diagram showing candidate blocks for deriving spatial merging candidates.

The spatial merging candidate may be derived based on at least one of a neighboring block adjacent to the current block or a block on the same line as the neighboring block. The block on the same line as the neighboring block may include at least one of a block on the same horizontal line, a block on the same vertical line, or a block on the same diagonal line as the neighboring block adjacent to the current block.

A merge candidate derived from a block with affine motion information may be referred to as an inheritance merge candidate. The inheritance merge candidate may be derived from a spatial neighboring block or a temporal neighboring block of the current block, and the derived inheritance merge candidate may be added to the merge candidate list. Alternatively, an inheritance merge candidate that is first searched in a predetermined order only among the left neighboring blocks of the current block and an inheritance merge candidate that is first searched in a predetermined order only among the upper neighboring blocks of the current block may be added to the merge candidate list. In this case, the left neighboring blocks may include A0 and A3, and the upper neighboring blocks may include A1, A2, and A4.

The motion information of the inherited merge candidate can be obtained based on the motion information of the candidate block. In the example, the prediction direction and reference picture index of the inherited merge candidate can be set to be the same as the candidate block. The affine vector of the inherited merge candidate can be derived based on the affine vector of the candidate block. In this case, according to the number of parameters, the affine vector of the inherited merge candidate may include at least one of the affine vector of the upper left corner, the affine vector of the upper right corner, or the affine vector of the lower left corner. In this case, the angular affine vector of the candidate block used to derive the affine vector of the inherited merge candidate may be different depending on whether the candidate block is adjacent to the boundary of the CTU. In the example, when the current block and the candidate block belong to the same CTU, the affine vector of the merge candidate can be derived based on the affine vector of the upper left corner of the candidate block and the affine vector of the upper right corner of the candidate block. On the other hand, when the current block and the candidate block belong to different CTUs, the affine vector of the merge candidate can be derived based on the affine vector of the lower left corner of the candidate block and the affine vector of the lower right corner of the candidate block.

Next, the constructed merge candidate can be added to the merge candidate list. The constructed merge candidate can be generated by combining the translation motion information of multiple blocks. In the example, the constructed merge candidate can be generated by combining two or more candidate blocks among the candidate blocks shown in Figure 32. When the constructed merge candidate is constructed, the candidate blocks can be combined so that the affine vector of the corner is derived based on the translation motion vector of the adjacent block adjacent to the corner. In the example, in the case of 6-parameter mode, the constructed merge candidate can be generated by combining a block adjacent to the upper left corner of the current block, a block adjacent to the upper right corner of the current block, and a block adjacent to the lower left corner of the current block.

The combination priority for generating the constructed merge candidate may be predefined in the encoder and the decoder. The constructed merge candidate may be generated by combining a plurality of blocks according to the combination priority. In this case, a combination of blocks with different reference pictures may be determined to be unusable as the constructed merge candidate.

When a constructed merge candidate is selected, a translation motion vector of each block constructing the constructed merge candidate may be set to an affine vector of a corner adjacent thereto.

On the other hand, an affine merge candidate may be derived by using one or more temporal neighboring blocks belonging to a picture different from the current picture. Specifically, when temporal neighboring blocks are encoded by affine inter prediction, an inheritance merge candidate may be derived from the temporal neighboring blocks.

Based on the neighboring blocks adjacent to the current block, a first merge candidate list may be constructed, and when the number of merge candidates included in the first merge candidate list is less than the maximum number, merge candidates included in the second merge candidate list may be added to the first merge candidate list. Specifically, merge candidates included in the second merge candidate list that are not included in the first merge candidate list may be added to the first merge candidate list.

In this case, the first merge candidate list may include constructed merge candidates and/or inherited merge candidates derived from neighboring blocks adjacent to the current block. The second merge candidate list may include constructed merge candidates and/or inherited merge candidates derived from blocks not adjacent to the current block. Alternatively, the second merge candidate list may include merge candidates derived based on affine motion information of a block encoded/decoded by affine inter-frame prediction before the current block. Constructing the second merge candidate list and adding the merge candidates included in the second merge candidate list to the first merge candidate list may follow the above-mentioned embodiment.

The encoder selects a merge candidate with the highest coding efficiency among the merge candidates (e.g., spatial merge candidates and temporal merge candidates) included in the merge candidate list, and signals information specifying the selected merge candidate. The decoder may select the merge candidate based on the information signaled in the bitstream, and may obtain the motion information of the current block based on the motion information of the selected merge candidate. Specifically, the motion information of the current block may be set to be the same as the motion information of the merge candidate.

FIG. 33 shows an example of deriving an affine vector of a current block from affine vectors of candidate blocks.

As described above, the affine vector of the merge candidate can be derived based on the affine vector of the block encoded by affine inter-frame prediction. Therefore, the affine vector of the current block can be derived from the affine vector of the candidate block according to the affine vector of the candidate block.

In the example shown in FIG. 33 , when A4 is used in the affine merge mode, it can be understood that at least one of the affine vectors v'0, v'1 or v'2 of the current block is derived based on at least one of the affine vectors v0, v1 or v2 of the block corresponding to A4.

Alternatively, it can be understood that at least one of the affine vectors v'0, v'1, or v'2 of the current block is derived based on at least one of the affine vectors v0, v1, or v2 of the partial area corresponding to A4. In this case, the partial area may be any one of a slice, a tile, a CTB, a CTB column, a CU, or a PU including the position of A4 (e.g., (-1, -1)).

Depending on the size and/or shape of the current block, it may be determined whether the affine merge mode is allowed. In an example, the affine merge mode may be allowed only when the size of the current block is greater than a threshold. The threshold may be a fixed value preset in the encoder and decoder. Alternatively, information for determining the threshold may be signaled in a bitstream.

Whether to use the affine merge mode for the current block may be determined based on a flag indicating whether the affine merge mode is used. The flag may be derived based on at least one of the size, shape, partition depth, partition type, or component type of the current block. Alternatively, the flag may be encoded and signaled in a bitstream.

When the flag indicates that the affine merge mode is not used, the affine vector of the current block can be derived by adding the affine vector difference value to the affine vector prediction value.

The affine vector prediction value may be set to be the same as the affine vector of the motion vector candidate. The motion vector candidate may be derived from at least one of the non-adjacent blocks or adjacent blocks encoded by affine inter-frame prediction. In an example, the motion vector candidate may be derived from a block encoded by affine inter-frame prediction, which is searched first when searching the left adjacent blocks of the current block in a predetermined order. In addition, the motion vector candidate may be derived from a block encoded by affine inter-frame prediction, which is searched first when searching the upper adjacent blocks of the current block in a predetermined order. In this case, the left adjacent blocks may include A0 and A3, and the upper adjacent blocks may include A1, A2, and A4.

The affine vector of the motion vector candidate can be derived based on the affine vector of the candidate block. In this case, according to the number of parameters, the affine vector of the motion vector candidate may include at least one of the affine vector of the upper left corner, the affine vector of the upper right corner, or the affine vector of the lower left corner. The angular affine vector of the candidate block used to derive the affine vector of the motion vector candidate can be determined differently depending on whether the candidate block is adjacent to the boundary of the CTU. In the example, when the current block and the candidate block belong to the same CTU, the affine vector of the motion vector candidate can be derived based on the affine vector of the upper left corner of the candidate block and the affine vector of the upper right corner of the candidate block. On the other hand, when the current block and the candidate block belong to different CTUs, the affine vector of the motion vector candidate can be derived based on the affine vector of the lower left corner of the candidate block and the affine vector of the lower right corner of the candidate block.

Information for specifying any one of a plurality of motion vector candidates may be signaled in a bitstream. When a motion vector candidate is selected by the information, an affine vector of the selected motion vector candidate may be set as an affine vector prediction value. The affine vector of the current block may be derived by adding an affine vector difference value decoded in the bitstream to the affine vector prediction value.

When the affine vector of the current block is derived, the affine motion vector of the sub-block may be derived based on the affine vector S2930. Based on the obtained affine motion vector, motion compensation of each sub-block may be performed S2940.

The above-mentioned examples describe that the affine merge mode is separate from the regular merge mode. According to an embodiment of the present disclosure, the regular merge mode and the affine merge mode can be used by combining the regular merge mode and the affine merge mode. In the example, a merge candidate can be derived based on at least one of the adjacent blocks or non-adjacent blocks adjacent to the current block, and a merge candidate list including a combination of merge candidates can be generated. In this case, when the candidate block is encoded by a translational motion model, the merge candidate derived from the candidate block may include translational motion information. In addition, when the candidate block is encoded by an affine motion model, the merge candidate derived from the candidate block may include affine motion information.

According to the merge candidate selected from the merge candidate list, the motion model of the current block may be determined. In an example, when the selected merge candidate is derived from a block encoded/decoded by an affine mode, affine inter prediction may be performed on the current block.

Alternatively, when the prediction mode of the neighboring block corresponding to the selected merge candidate is different from the prediction mode of the current block, the motion information of the current block may be derived by using at least only a portion of the motion information of the selected merge candidate. In this case, the prediction mode may include an affine merge mode, an affine inter mode, a merge mode, an AMVP mode, or at least one of a mode predefined in an encoder and a decoder.

In an example, when a neighboring block corresponding to a selected merge candidate is encoded by an affine inter mode and the current block is encoded by a normal merge mode, a motion vector of the current block can be derived by using all (e.g., v0, v1, and v2) or part (e.g., v0 and v1) of the affine vectors of the neighboring blocks.

Alternatively, when a neighboring block corresponding to a selected merge candidate is encoded by a first prediction mode and a current block is encoded by a second prediction mode, a motion vector in motion information of the merge candidate may be referenced, but a reference picture index may not be referenced.

It is within the scope of the present invention to apply the embodiments described with emphasis on the decoding process or the encoding process to the encoding process or the decoding process. It is also within the scope of the present invention to change the embodiments described in a predetermined order to a different order.

Although the above embodiments have been described based on a series of steps or flow charts, the above embodiments are not intended to limit the time series order of the present invention, and can be executed simultaneously or in different orders. In addition, each of the components (e.g., units, modules, etc.) constituting the block diagram in the above embodiments can be implemented as a hardware device or software, and multiple components can be combined into one hardware device or software. The above embodiments can be implemented in the form of program instructions, which can be executed by various computer components and recorded in a computer-readable recording medium. Computer-readable storage media can include program instructions, data files, data structures, etc., alone or in combination. Examples of computer-readable storage media include: magnetic recording media such as hard disks, floppy disks, and tapes; optical data storage media such as CD-ROMs or DVD-ROMs; magneto-optical media such as optical disks; and hardware devices such as read-only memories (ROMs), random access memories (RAMs), and flash memories, which are particularly structured to store and implement program instructions. Hardware devices can be configured to be operated by one or more software modules or software modules can be configured to be operated by one or more hardware devices to perform processing according to the present invention.

Industrial Applicability

The present invention can be applied to electronic devices capable of encoding/decoding images.

The solution of the present invention also includes:

(1) A method for decoding a video, the method comprising:

Deriving merge candidates according to the candidate blocks;

generating a first merge candidate list including the merge candidate;

specifying one of the plurality of merge candidates included in the first merge candidate list;

deriving an affine vector of the current block based on motion information of the specified merge candidate;

deriving a motion vector of a sub-block in the current block based on the affine vector; and

Motion compensation of the sub-block is performed based on the motion vector.

(2) The method according to (1), wherein, when the candidate block is encoded by affine inter-frame prediction, the affine vector of the merge candidate is derived based on the affine vector of the candidate block.

(3) The method according to (2), wherein a position of the affine vector of the candidate block is different based on whether the current block and the candidate block are included in the same CTU (coding tree unit).

(4) The method according to (1), wherein the affine vector of the merge candidate is derived by combining translation motion vectors of multiple candidate blocks.

(5) The method according to (1), wherein, when the number of merge candidates included in the first merge candidate list is less than a maximum number, the merge candidates included in the second merge candidate list are added to the first merge candidate list.

(6) The method according to (1), wherein the method further comprises determining the number of affine parameters of the current block, and

The number of the affine parameters is determined based on at least one of a size or a shape of the current block.

(7) The method according to (1), wherein the candidate block includes at least one of a neighboring block of the current block and a non-neighboring block on the same line as the neighboring block.

(8) A method for encoding a video, the method comprising:

Deriving merge candidates according to the candidate blocks;

generating a first merge candidate list including the merge candidate;

specifying one of the plurality of merge candidates included in the first merge candidate list;

deriving an affine vector of the current block based on motion information of the specified merge candidate;

deriving a motion vector of a sub-block in the current block based on the affine vector; and

Motion compensation of the sub-block is performed based on the motion vector.

(9) The method according to (8), wherein, when the candidate block is encoded by affine inter-frame prediction, the affine vector of the merge candidate is derived based on the affine vector of the candidate block.

(10) The method according to (9), wherein a position of the affine vector of the candidate block is different based on whether the current block and the candidate block are included in the same CTU (coding tree unit).

(11). The method according to (8), wherein the affine vector of the merge candidate is derived by combining translation motion vectors of multiple candidate blocks.

(12) The method according to (8), wherein, when the number of merge candidates included in the first merge candidate list is less than a maximum number, the merge candidates included in the second merge candidate list are added to the first merge candidate list.

(13) The method according to (8), further comprising determining the number of affine parameters of the current block, and

The number of the affine parameters is determined based on at least one of a size or a shape of the current block.

(14) The method according to (8), wherein the candidate block includes at least one of a neighboring block of the current block and a non-neighboring block on the same line as the neighboring block.

(15) A device for decoding a video, the device comprising:

The inter-frame prediction unit is used to perform the following operations:

Deriving merge candidates according to the candidate blocks;

generating a first merge candidate list including the merge candidate;

specifying one of the plurality of merge candidates included in the first merge candidate list;

deriving an affine vector of the current block based on motion information of the specified merge candidate;

deriving a motion vector of a sub-block in the current block based on the affine vector; and

Motion compensation of the sub-block is performed based on the motion vector.

Claims (7)

1. A method for decoding a video, the method comprising: deriving merge candidates from neighboring blocks adjacent to the current block; generating a merge candidate list including the merge candidates; specifying a merge candidate from among a plurality of merge candidates included in the merge candidate list; deriving an angular motion vector of the current block based on the specified merge candidate; deriving a motion vector of a sub-block in the current block based on the angular motion vector of the current block, the current block being divided into a plurality of sub-blocks having a predefined size of 4×4; and performing motion compensation for the sub-block based on the motion vector, Wherein, when the neighboring block is not included in the same coding tree unit CTU as the current block, the angular motion vector of the merge candidate is derived based on motion vectors at the lower left corner and the lower right corner of the neighboring block. 2. The method according to claim 1, wherein, when the neighboring block is included in the same coding tree unit (CTU) as the current block, the angular motion vector of the merge candidate is derived based on a top-left corner motion vector and a top-right corner motion vector of the neighboring block. 3 . The method of claim 1 , wherein the merge candidate list includes constructed merge candidates generated by combining translation motion vectors of a plurality of neighboring blocks. 4. A method for encoding a video, the method comprising: deriving merge candidates from neighboring blocks adjacent to the current block; generating a merge candidate list including the merge candidates; specifying a merge candidate from among a plurality of merge candidates included in the merge candidate list; deriving an angular motion vector of the current block based on the specified merge candidate; deriving a motion vector of a sub-block in the current block based on the angular motion vector of the current block, the current block being divided into a plurality of sub-blocks having a predefined size of 4×4; and performing motion compensation for the sub-block based on the motion vector, Wherein, when the neighboring block is not included in the same coding tree unit CTU as the current block, the angular motion vector of the merge candidate is derived based on motion vectors at the lower left corner and the lower right corner of the neighboring block. 5. The method according to claim 4, wherein, when the neighboring block is included in the same coding tree unit (CTU) as the current block, the angular motion vector of the merge candidate is derived based on a top-left corner motion vector and a top-right corner motion vector of the neighboring block. The method of claim 4 , wherein the merge candidate list includes constructed merge candidates generated by combining translation motion vectors of a plurality of adjacent blocks. 7. A device for sending compressed video data, the device comprising: a processor configured to obtain the compressed video data; and a sending unit, configured to send the compressed video data, Wherein, obtaining the compressed video data comprises: deriving merge candidates from neighboring blocks adjacent to the current block; generating a merge candidate list including the merge candidates; specifying a merge candidate among a plurality of merge candidates included in the merge candidate list; deriving an angular motion vector of the current block based on the specified merge candidate; deriving a motion vector of a sub-block in the current block based on the angular motion vector of the current block, the current block being divided into a plurality of sub-blocks having a predefined size of 4×4; and performing motion compensation for the sub-block based on the motion vector, Wherein, when the neighboring block is not included in the same coding tree unit CTU as the current block, the angular motion vector of the merge candidate is derived based on motion vectors at the lower left corner and the lower right corner of the neighboring block.