WO2024258146A1 - Method and apparatus for processing video signal - Google Patents

Abstract

An image decoding method and apparatus according to the present disclosure may generate at least two prediction blocks for a current block, and generate a final prediction block of the current block by blending the at least two prediction blocks on the basis of a prescribed mask.

Description

Video signal processing method and device

The present invention relates to a video signal processing method and device, and more specifically, to a video encoding and decoding method and device.

The market demand for high-resolution video is increasing, and technology that can efficiently compress high-resolution images is required. In response to this market demand, the ISO/IEC's MPEG (Moving Picture Expert Group) and the ITU-T's VCEG (Video Coding Expert Group) jointly formed the JCT-VC (Joint Collaborative Team on Video Coding), completed the development of the HEVC (High Efficiency Video Coding) video compression standard in January 2013, and are actively conducting research and development on next-generation compression standards.

Video compression is largely composed of intra-prediction, inter-prediction, transformation, quantization, entropy coding, and in-loop filtering. Meanwhile, as the demand for high-resolution images increases, the demand for stereoscopic image content as a new image service is also increasing. Discussions are underway on video compression technology to effectively provide high-resolution and ultra-high-resolution stereoscopic image content.

The purpose of the present invention is to provide a method and device for processing a video signal.

The video decoding method and device according to the present disclosure can generate at least two prediction blocks for a current block, and generate a final prediction block of the current block by weighting the at least two prediction blocks based on a predetermined mask. Here, each of the at least two prediction blocks is a unidirectional or bidirectional prediction block, and the bidirectional prediction block can be generated based on a weighted sum of an L0 prediction block and an L1 prediction block.

In the image decoding method and device according to the present disclosure, the mask can be determined based on geometric segmentation information.

In the image decoding method and device according to the present disclosure, the geometric segmentation information can specify any one of a plurality of mask candidates pre-defined in the image decoding device.

In the image decoding method and device according to the present disclosure, the geometric segmentation information may include a flag indicating whether an inverted mask is used.

In the image decoding method and device according to the present disclosure, the mask can be generated based on a differential block between the at least two prediction blocks.

In the image decoding method and device according to the present disclosure, the mask may be generated based on a ratio block between the at least two prediction blocks. Here, the ratio block may be derived by dividing a sample value belonging to one of the at least two prediction blocks by a sample value at the same position belonging to the other one.

In the video decoding method and device according to the present disclosure, each of the at least two prediction blocks can be generated based on any one of an inter mode, an intra mode, or an intra block copy mode.

In the video decoding method and device according to the present disclosure, the weight for the weighted sum can be determined based on at least one of the ratio of surrounding blocks to which the intra mode is applied or the POC difference between the current picture to which the current block belongs and the reference picture.

The video encoding method and device according to the present disclosure can generate at least two prediction blocks for a current block, and generate a final prediction block of the current block by weighting and combining the at least two prediction blocks based on a predetermined mask. Here, each of the at least two prediction blocks is a unidirectional or bidirectional prediction block, and the bidirectional prediction block can be generated based on a weighted sum of an L0 prediction block and an L1 prediction block.

A computer-readable digital storage medium is provided having stored thereon video/image information causing a method of decrypting a message according to the present disclosure to be performed.

A computer-readable digital storage medium storing video/image information generated by a method for encoding a message according to the present disclosure is provided.

A method and device for transmitting video/image information generated by a mesh encoding method according to the present disclosure are provided.

According to the present disclosure, the prediction performance of a final prediction block can be improved by using an adaptive mask on the texture of a current block.

According to the present disclosure, prediction performance can be improved by utilizing parameters based on the correlation between a current template and a reference template.

Figure 1 is a block diagram showing an image encoding device according to the present invention.

Figure 2 is a block diagram showing an image decoding device according to the present invention.

FIG. 3 illustrates a prediction method performed by an image encoding/decoding device as an embodiment according to the present disclosure.

FIG. 4 illustrates a mask-based blending method for multiple prediction blocks as an embodiment according to the present disclosure.

Figures 5 to 7 illustrate a mask generating method according to the present disclosure.

FIG. 8 illustrates a parameter-based prediction method as an example according to the present disclosure.

FIG. 9 illustrates a parameter-based prediction method in color inter-prediction of intra prediction as an embodiment according to the present disclosure.

FIG. 10 illustrates a parameter-based prediction method in intra block copy (IBC) mode as an embodiment according to the present disclosure.

Figure 11 illustrates a parameter-based prediction method in inter prediction mode.

FIG. 12 is an example according to the present disclosure, illustrating a relationship between two different areas.

FIG. 13 is an example according to the present disclosure, illustrating multiple associations for two different areas.

FIG. 14 is an example according to the present disclosure, showing nonlinear correlations for two different domains.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings attached to this specification so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are assigned similar drawing reference numerals throughout the specification.

Throughout this specification, when a part is said to be 'connected' to another part, this includes not only cases where they are directly connected, but also cases where they are electrically connected with another element in between.

Additionally, whenever it is said throughout this specification that a part "includes" a component, this does not exclude other components, unless otherwise specifically stated, but rather means that other components may be included.

Additionally, while the terms first, second, etc. may be used to describe various components, the components should not be limited by the terms. The terms are used only to distinguish one component from another.

In addition, in the embodiments of the device and method described herein, some of the components of the device or some of the steps of the method may be omitted. Also, the order of some of the components of the device or some of the steps of the method may be changed. Also, other components or other steps may be inserted into some of the components of the device or some of the steps of the method.

Additionally, some components or some steps of the first embodiment of the present invention may be added to the second embodiment of the present invention, or some components or some steps of the second embodiment may be replaced.

In addition, the components shown in the embodiments of the present invention are independently depicted to represent different characteristic functions, and do not mean that each component is formed as a separate hardware or software configuration unit. That is, each component is described by listing each component for convenience of explanation, and at least two components among each component may be combined to form a single component, or one component may be divided into multiple components to perform a function. Such integrated embodiments and separate embodiments of each component are also included in the scope of the rights of the present invention as long as they do not deviate from the essence of the present invention.

First, the terms used in this application are briefly explained as follows.

The video decoding apparatus described below may be a device included in a server terminal such as a civilian security camera, a civilian security system, a military security camera, a military security system, a personal computer (PC), a laptop computer, a portable multimedia player (PMP), a wireless communication terminal, a smart phone, a TV application server and a service server, and may refer to various devices including user terminals such as various devices, communication devices such as a communication modem for communicating with a wired or wireless communication network, a memory for storing various programs and data for decoding a video or for inter- or intra-predicting for decoding, and a microprocessor for executing a program and performing calculations and controls.

In addition, an image encoded into a bitstream by an image encoding device may be transmitted to an image decoding device through wired or wireless communication networks such as the Internet, a local area network, a wireless LAN, a WiBro network, a mobile communication network, etc. in real time or non-real time, or through various communication interfaces such as a cable, a universal serial bus (USB), etc., and may be decoded, restored into an image, and played back. Alternatively, the bitstream generated by the image encoding device may be stored in memory. The memory may include both volatile memory and non-volatile memory. In this specification, the memory may be expressed as a recording medium storing a bitstream.

Typically, a video may be composed of a series of pictures, and each picture may be divided into coding units such as blocks. In addition, those skilled in the art to which this embodiment pertains will understand that the term picture described below may be replaced with other terms having equivalent meanings, such as image, frame, etc. In addition, those skilled in the art to which this embodiment pertains will understand that the term coding unit may be replaced with other terms having equivalent meanings, such as unit block, block, etc.

Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described in more detail. In describing the present invention, duplicate descriptions of identical components will be omitted.

Figure 1 is a block diagram showing an image encoding device according to the present invention.

Referring to FIG. 1, a conventional image encoding device (100) may include a picture segmentation unit (110), a prediction unit (120, 125), a transformation unit (130), a quantization unit (135), a reordering unit (160), an entropy encoding unit (165), an inverse quantization unit (140), an inverse transformation unit (145), a filter unit (150), and a memory (155).

The picture splitting unit (110) can split the input picture into at least one processing unit. At this time, the processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). Hereinafter, in the embodiment of the present invention, the coding unit may be used to mean a unit that performs encoding, or may be used to mean a unit that performs decoding.

A prediction unit may be divided into at least one square or rectangular shape of the same size within one coding unit, or may be divided such that one prediction unit among the prediction units divided within one coding unit has a different shape and/or size from another prediction unit. When generating a prediction unit that performs intra prediction based on a coding unit, if it is not the minimum coding unit, intra prediction can be performed without being divided into a plurality of NxN prediction units.

The prediction unit (120, 125) may include an inter prediction unit (120) that performs inter prediction or inter-screen prediction, and an intra prediction unit (125) that performs intra prediction or intra-screen prediction. It may be determined whether to use inter prediction or intra prediction for a prediction unit, and specific information (e.g., intra prediction mode, motion vector, reference picture, etc.) according to each prediction method may be determined. A residual value (residual block) between the generated prediction block and the original block may be input to the transformation unit (130). In addition, prediction mode information, motion vector information, etc. used for prediction may be encoded together with the residual value by an entropy encoding unit (165) and transmitted to an image decoding device.

The inter prediction unit (120) may predict a prediction unit based on information of at least one picture among the previous picture or the subsequent picture of the current picture, and in some cases, may predict a prediction unit based on information of a part of an encoded region within the current picture. The inter prediction unit (120) may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.

The reference picture interpolation unit can receive reference picture information from the memory (155) and generate pixel information below an integer pixel from the reference picture. In the case of luminance pixels, a DCT-based 8-tap interpolation filter (DCT-based Interpolation Filter) with different filter coefficients can be used to generate pixel information below an integer pixel in units of 1/4 pixels. In the case of a chrominance signal, a DCT-based 4-tap interpolation filter (DCT-based Interpolation Filter) with different filter coefficients can be used to generate pixel information below an integer pixel in units of 1/8 pixels.

The motion prediction unit can perform motion prediction based on a reference picture interpolated by a reference picture interpolation unit. Various methods such as FBMA (Full search-based Block Matching Algorithm), TSS (Three Step Search), and NTS (New Three-Step Search Algorithm) can be used to derive a motion vector. The motion vector can have a motion vector value of 1/2 or 1/4 pixel unit based on the interpolated pixel. The motion prediction unit can predict the current prediction unit by using different motion prediction methods. Various methods such as Skip Mode, Merge Mode, AMVP Mode, Intra Block Copy Mode, and Affine Mode can be used as the motion prediction method.

The intra prediction unit (125) can generate a prediction unit based on reference pixel information surrounding the current block, which is pixel information within the current picture. If the surrounding block of the current prediction unit is a block on which inter prediction is performed and the reference pixel is a pixel on which inter prediction is performed, the reference pixel included in the block on which inter prediction is performed can be used as a replacement for the reference pixel information of the surrounding block on which intra prediction is performed. That is, if the reference pixel is not available, the unavailable reference pixel information can be used as a replacement for at least one reference pixel among the available reference pixels.

In addition, a residual block including residual value information, which is a difference value between the prediction unit that performed the prediction and the original block of the prediction unit based on the prediction unit generated in the prediction unit (120, 125), can be generated. The generated residual block can be input to the transformation unit (130).

In the transformation unit (130), the residual block including the residual value information of the prediction unit generated through the original block and the prediction unit (120, 125) can be transformed using a transformation method such as DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), or KLT. Whether to apply DCT, DST, or KLT to transform the residual block can be determined based on the intra prediction mode information of the prediction unit used to generate the residual block.

The quantization unit (135) can quantize the values converted to the frequency domain in the transformation unit (130). The quantization coefficients can vary depending on the block or the importance of the image. The values produced by the quantization unit (135) can be provided to the dequantization unit (140) and the reordering unit (160).

The rearrangement unit (160) can perform rearrangement of coefficient values for quantized residual values.

The rearrangement unit (160) can change a two-dimensional block-shaped coefficient into a one-dimensional vector shape by using a coefficient scanning method. For example, the rearrangement unit (160) can change the two-dimensional block-shaped coefficient into a one-dimensional vector shape by scanning from the DC coefficient to the coefficient of the high-frequency region by using a zig-zag scan method. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional block-shaped coefficient in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficient in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, it is possible to determine which scan method among the zig-zag scan, the vertical scan, and the horizontal scan is to be used.

The entropy encoding unit (165) can perform entropy encoding based on the values produced by the rearrangement unit (160) to generate a bitstream. The entropy encoding can use various encoding methods such as, for example, Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding). In this regard, the entropy encoding unit (165) can encode residual value coefficient information of an encoding unit from the rearrangement unit (160) and the prediction unit (120, 125). In addition, according to the present invention, it is possible to signal and transmit information indicating that motion information is derived and used on the decoder side and information on a technique used to derive motion information.

In the inverse quantization unit (140) and the inverse transformation unit (145), the values quantized in the quantization unit (135) are inversely quantized and the values transformed in the transformation unit (130) are inversely transformed. The residual values generated in the inverse quantization unit (140) and the inverse transformation unit (145) can be combined with the predicted prediction units through the motion estimation unit, motion compensation unit, and intra prediction unit included in the prediction unit (120, 125) to generate a reconstructed block.

The filter unit (150) may include at least one of a deblocking filter, an offset correction unit, and an ALF (Adaptive Loop Filter). The deblocking filter may remove block distortion caused by boundaries between blocks in a restored picture. The offset correction unit may correct an offset from an original image on a pixel basis for an image on which deblocking has been performed. In order to perform offset correction for a specific picture, a method may be used in which pixels included in the image are divided into a certain number of regions, and then regions to be offset-performed are determined and offsets are applied to the regions, or a method may be used in which an offset is applied by considering edge information of each pixel. ALF (Adaptive Loop Filtering) may be performed based on a value obtained by comparing a filtered restored image with an original image. After dividing pixels included in the image into a predetermined group, one filter to be applied to the group may be determined, and filtering may be performed differentially for each group.

The memory (155) can store a restored block or picture produced through the filter unit (150), and the stored restored block or picture can be provided to the prediction unit (120, 125) when performing inter prediction.

Figure 2 is a block diagram showing an image decoding device according to the present invention.

Referring to FIG. 2, the video image decoding device (200) may include an entropy decoding unit (210), a reordering unit (215), an inverse quantization unit (220), an inverse transformation unit (225), a prediction unit (230, 235), a filter unit (240), and a memory (245).

When a video bitstream is input into a video encoding device, the input bitstream can be decoded in the opposite procedure to that of the video encoding device.

The entropy decoding unit (210) can perform entropy decoding in a procedure opposite to that of performing entropy encoding in the entropy encoding unit of the video image encoding device. For example, various methods such as Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding) can be applied in response to the method performed in the video image encoding device.

The entropy decoding unit (210) can decode information related to intra prediction and inter prediction performed in the image encoding device.

The reordering unit (215) can perform reordering based on the method in which the bitstream that has been entropy decoded by the entropy decoding unit (210) is reordered by the encoding unit. The coefficients expressed in the form of a one-dimensional vector can be reordered by restoring them back to coefficients in the form of a two-dimensional block.

The inverse quantization unit (220) can perform inverse quantization based on the quantization parameters provided from the image encoding device and the coefficient values of the rearranged block.

The inverse transform unit (225) can perform inverse transform, i.e., inverse DCT, inverse DST, and inverse KLT, on the transforms performed by the transform unit, i.e., DCT, DST, and KLT, on the quantization result performed by the image image encoding device. The inverse transform can be performed based on the transmission unit determined by the image image encoding device. In the inverse transform unit (225) of the image image decoding device, a transform technique (e.g., DCT, DST, KLT) can be selectively performed according to a plurality of pieces of information, such as a prediction method, a size of a current block, and a prediction direction.

The prediction unit (230, 235) can generate a prediction block based on prediction block generation related information provided from the entropy decoding unit (210) and previously decoded block or picture information provided from the memory (245).

As described above, when performing intra prediction or intra-screen prediction in the same manner as the operation in a video image encoding device, if the size of a prediction unit and the size of a transformation unit are the same, intra prediction for the prediction unit is performed based on the pixels on the left side of the prediction unit, the pixels on the upper left side, and the pixels on the upper side. However, if the sizes of the prediction unit and the transformation unit are different when performing intra prediction, intra prediction can be performed using reference pixels based on the transformation unit. In addition, intra prediction using NxN division only for the minimum coding unit can be used.

The prediction unit (230, 235) may include a prediction unit determination unit, an inter prediction unit, and an intra prediction unit. The prediction unit determination unit may receive various information such as prediction unit information input from the entropy decoding unit (210), prediction mode information of an intra prediction method, and motion prediction-related information of an inter prediction method, and may distinguish a prediction unit from a current encoding unit and determine whether the prediction unit performs inter prediction or intra prediction. On the other hand, if the video encoding device (100) does not transmit motion prediction-related information for the inter prediction, but instead transmits information instructing that motion information be derived and used on the decoder side and information on a technique used to derive motion information, the prediction unit determination unit determines whether the inter prediction unit (230) performs prediction based on the information transmitted from the video encoding device (100).

The inter prediction unit (230) can perform inter prediction on the current prediction unit based on information included in at least one picture among the previous picture or the subsequent picture of the current picture including the current prediction unit, using information required for inter prediction of the current prediction unit provided by the video image encoding device. In order to perform inter prediction, it can be determined based on the encoding unit whether the motion prediction method of the prediction unit included in the corresponding encoding unit is one of Skip Mode, Merge Mode, AMVP Mode, Intra block copy mode, and Affine mode.

The intra prediction unit (235) can generate a prediction block based on pixel information in the current picture. If the prediction unit is a prediction unit that has performed intra prediction, intra prediction can be performed based on intra prediction mode information of the prediction unit provided by the video image encoding device.

The intra prediction unit (235) may include an AIS (Adaptive Intra Smoothing) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a unit that performs filtering on the reference pixels of the current block and may determine whether to apply the filter according to the prediction mode of the current prediction unit and apply it. The AIS filter may be performed on the reference pixels of the current block using the prediction mode and AIS filter information of the prediction unit provided by the video image encoding device. If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied.

The reference pixel interpolation unit can interpolate the reference pixel to generate a reference pixel of a pixel unit less than an integer value when the prediction mode of the prediction unit is a prediction unit that performs intra prediction based on the pixel value interpolated by the reference pixel. When the prediction mode of the current prediction unit is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is the DC mode.

The restored block or picture may be provided to a filter unit (240). The filter unit (240) may include a deblocking filter, an offset correction unit, and an ALF.

Information on whether a deblocking filter has been applied to a corresponding block or picture from a video encoding device and, if a deblocking filter has been applied, information on whether a strong filter or a weak filter has been applied can be provided. A deblocking filter of a video decoding device can receive information related to a deblocking filter provided from a video encoding device and perform deblocking filtering on a corresponding block in a video decoding device.

The offset correction unit can perform offset correction on the restored image based on the type of offset correction applied to the image during encoding, information on the offset value, etc. ALF can be applied to the encoding unit based on information on whether ALF is applied, ALF coefficient information, etc. provided from the image encoding device. Such ALF information can be provided by being included in a specific parameter set.

The memory (245) can store a restored picture or block so that it can be used as a reference picture or reference block, and can also provide the restored picture to an output unit.

FIG. 3 illustrates a prediction method performed by an image encoding/decoding device as an embodiment according to the present disclosure.

Referring to FIG. 3, at least two prediction blocks can be generated for the current block (S300).

At least two prediction blocks can be generated for a current block. For example, the current block can be divided into at least two partitions. For each of the at least two partitions, a prediction block can be generated from a reference picture of the corresponding partition. The division can be performed based on geometric division. The prediction based on geometric division according to the present disclosure can be any one of the inter modes pre-defined in the image encoding/decoding apparatus. Here, the prediction block can be a unidirectional prediction block or a bidirectional prediction block. The unidirectional prediction block can mean a prediction block in the L0 or L1 direction. The bidirectional prediction block can be generated through a weighted sum of the L0 and L1 prediction blocks. Through this, at least two prediction blocks can be generated for the current block.

Referring to FIG. 3, blending may be performed on at least two prediction blocks based on a predetermined mask to generate a final prediction block of the current block (S310).

The above mask has the same size as the current block and can define weights for each sample location within the prediction block. The above mask can be defined for each of the at least two prediction blocks. The method of generating/determining the above mask will be described in detail with reference to FIGS. 4 to 7.

FIG. 4 illustrates a mask-based blending method for multiple prediction blocks as an embodiment according to the present disclosure.

For at least two prediction blocks for the current block, blending can be performed based on a predetermined mask to generate a final prediction block of the current block.

The mask may be determined based on geometric segmentation information. The geometric segmentation information may be information related to one or more segmentation lines for dividing the current block into at least two partitions. For example, the geometric segmentation information may include at least one of a distance between the segmentation line and the center of the current block or an angle of a line segment orthogonal to the segmentation line. The geometric segmentation information may be signaled through a bitstream.

The weight of the mask can be calculated based on at least one of the angles or distances mentioned above. In this case, the angles and/or distances can be quantized and defined in the form of indices. For example, for the use of functions such as cos() and sin(), pre-determined values according to the indices can be used based on a table.

Alternatively, the geometric segmentation information may be an index. Based on the index, an identically pre-defined mask may be used in the image encoding/decoding device. The image encoding/decoding device may define a plurality of mask candidates, and any one of the plurality of mask candidates may be selectively used based on the index.

The above geometric segmentation information may include a flag indicating whether an inverted mask is used. For example, if the flag is a first value, this may indicate that an inverted mask is used. In this case, a mask derived by inverting a mask selected by the index (i.e., an inverted mask) may be used. On the other hand, if the flag is a second value, this may indicate that an inverted mask is not used. In this case, a mask selected by the index may be used. The flag may be encoded in a bitstream based on a determination of whether a current block uses an inverted mask, and the encoded flag may be signaled to an image decoding device through the bitstream.

Referring to Fig. 4, there may be a curved edge inside the block, such as the original block. In this case, a large amount of error may occur in the blending area of the final prediction block. Therefore, by using an adaptive mask for the texture of the current block, the prediction performance of the final prediction block can be improved. To use the adaptive mask, an optimal mask is determined/generated in an image encoding device, and information specifying this is transmitted to an image decoding device, so that the same mask can be used in the image encoding/decoding device.

Alternatively, the mask can be generated based on at least two prediction blocks.

For example, a mask may be generated based on a differential block between at least two prediction blocks. At this time, at least two weights for blending may be derived based on the differential sample value(s) in the differential block. Here, the at least two weights may be different values or the same values. To this end, a table defining weights corresponding to each differential sample value may be used. That is, weights corresponding to differential sample values in the differential block may be derived from the table. Here, the weights may include at least two weights applied to at least two prediction blocks. Alternatively, the at least two weights may be derived based on a mapping function according to the differential sample values.

Alternatively, the mask can be generated based on a ratio block between at least two prediction blocks.

A ratio block can be derived by dividing a prediction sample value in one of at least two prediction blocks by a prediction sample value at the same location in the other prediction block. At least two weights for blending can be derived based on the sample value(s) in the ratio block.

FIG. 5 illustrates a method for generating a mask according to the present disclosure.

A plurality of prediction blocks may be used to generate a final prediction block for a current block. Each of the prediction blocks may be generated based on one of the inter mode, the intra mode, or the intra block copy (IBC) mode. For example, the plurality of prediction blocks may be generated through the same prediction method (i.e., one of the inter mode, the intra mode, or the intra block copy mode). Alternatively, one of the plurality of prediction blocks may be generated based on the inter mode, and the other may be generated based on one of the intra mode or the intra block copy mode. Alternatively, one of the plurality of prediction blocks may be generated based on the intra mode, and the other may be generated based on one of the inter mode or the intra block copy mode.

Multiple prediction blocks for the current block can be weighted to generate a final prediction block.

The weights for the above weighted sum may be determined based on at least one of a ratio of neighboring blocks to which the intra mode is applied among neighboring blocks adjacent to the current block or a difference between the POC (picture order count) of the current picture to which the current block belongs and the POC of the reference picture.

A final prediction block can be generated using multiple prediction block-specific masks.

The above mask can be derived based on information signaled in predetermined units as described above. Here, the predetermined unit can be a picture, a slice, a coding tree unit row (CTU row), a CTU, or a coding block.

Alternatively, the mask may be derived based on the differential block between the at least two prediction blocks described above. For example, if the differential sample value in the differential block is 0, the weight of the corresponding sample position may be derived as the average weight. On the other hand, a larger weight may be used as the differential sample value in the differential block is larger. In this case, one of the at least two prediction blocks may be selected according to the sign of the differential sample value. A flag for the selection may be separately signaled. The differential sample value may be replaced with the absolute value of the differential sample value, and this may be equally applied in the embodiments described below.

Alternatively, a value mapped to the differential sample value may be derived based on a linear function, and a mask may be derived based on the derived value. The linear function according to the present disclosure may be defined as at least one of a weight multiplied by the differential sample value or an offset added/subtracted to/from the differential sample value. That is, the derivation of the linear function may mean the derivation of at least one of the weight or the offset. This may be applied to the embodiments described below in the same sense. At this time, when the differential sample value is greater than a predetermined threshold, the differential sample value may be clipped based on a value based on the bit depth of the mask. On the other hand, when the differential sample value is less than or equal to the threshold, the mapping may be performed based on a linear function different from the previous linear function. At this time, each linear function may be identically pre-defined in the image encoding/decoding device.

The image encoding device can transmit linear function information to the image decoding device in predetermined units. Here, the predetermined units are as described above.

Alternatively, a linear function may be derived based on the prediction block(s) for the surrounding block(s) of the current block, and a mask may be derived based on this. For example, if the surrounding block uses two prediction blocks, one or more linear functions may be obtained for the current block based on the reconstructed surrounding block and the two prediction blocks. A value mapped to a differential sample value for the current block may be derived based on the obtained linear function, and a mask may be derived based on this.

Alternatively, a predetermined representative value may be derived from the prediction block of the current block, and a linear function may be derived based on the derived representative value. For example, a linear function may be derived based on an average value of any one of at least two prediction blocks for the current block. Based on an average value of any one of at least two prediction blocks for the current block, any one of a plurality of linear function candidates may be selected. The average value may be defined as an average of all or part of the prediction sample values belonging to the prediction block.

Alternatively, a linear function can be derived based on the representative values of the pre-restored sample values around the current block. One of a plurality of linear function candidates can be selected based on the representative values of the pre-restored sample values around the current block.

Figure 6 illustrates a mask generation method according to the present disclosure.

At least two prediction blocks according to the present disclosure may be generated based on an intra mode. However, the present invention is not limited thereto, and as described above, the at least two prediction blocks may be generated based on different prediction methods.

Each prediction block may be generated based on a non-directional mode (e.g., planar mode, DC mode), a directional mode, IBC, or Intra-Template Matching Prediction (IntraTMP). Alternatively, the prediction samples of each prediction block may have position-based intra prediction sample filtering (PDPC) applied.

A mask can be derived based on differential sample value(s) within a differential block for a current block, and the final predicted block can be generated by applying the mask to at least two predicted blocks for the current block.

At this time, a linear function (i.e., a weight and/or an offset of the linear function) can be derived based on at least one of the maximum or minimum values among the differential sample values and the bit depth of the mask. The sign of the weight and/or an offset of the linear function can be signaled in a predetermined unit.

The sign of the weight of a linear function can be predicted based on the previously restored reference sample value(s) for the current block, and the predicted sign can be used as is.

Alternatively, the sign of the gradient of the linear function can be derived based on the reconstructed sample value(s) around the current block and the predicted block of the current block. For example, the sign of the gradient of the linear function can be derived based on the first derivative in the horizontal or vertical direction.

Alternatively, the sign of the gradient of the linear function can be derived based on the continuity between the pre-reconstructed sample value(s) around the current block and the prediction block of the current block. Here, the continuity can be judged based on the difference between the prediction sample located at the boundary of the current block and the pre-reconstructed surrounding samples. That is, according to the sign of the gradient, more prediction candidates can be generated than the existing intra prediction, and high encoding efficiency can be provided by adaptively using a more appropriate prediction block.

Figure 7 illustrates a mask generation method according to the present disclosure.

According to FIG. 7, differential sample value(s) can be calculated for some areas within at least two prediction blocks for a current block, and a mask can be generated based on the differential sample value(s). A final prediction block can be generated by blending at least two prediction blocks based on the generated mask.

Referring to Fig. 7, differential sample values can be calculated only for some areas (Difference areas) within the two prediction blocks (Predictor 0, Predictor 1). The above-mentioned some areas (Difference areas) can be determined based on one or more straight lines. Hereinafter, some areas where differential sample values are calculated will be called difference areas.

Here, the information about the straight line can be signaled in a given unit. The given unit can be a picture, a slice, a coding tree unit row (CTU row), a CTU, or a coding block. The information about the straight line can include at least one of a weight (or a slope) or an offset (or an intercept) representing a linear function. Alternatively, the information about the straight line can include at least one of a slope or a distance from the center of the current block. Alternatively, the information about the straight line can be an index indicating one straight line parameter from a table defining a plurality of straight line parameters.

When the difference area is a straight-line-based area, a first area (1 ^st area) and a second area (2 ^nd area) can be distinguished as shown in Fig. 7. A flag can be additionally signaled to induce masks for the two areas.

For example, when the value of the flag is 0, a mask may be derived that causes the first region to use the first prediction block (Predictor 0), and a mask may be derived that causes the second region to use the second prediction block (Predictor 1). Conversely, when the value of the flag is 1, a mask may be derived that causes the first region to use the second prediction block, and a mask may be derived that causes the second region to use the first prediction block.

Alternatively, the same prediction block can be used for both regions, and blending can be performed only on the difference region to generate the final prediction block. In this case, a specific flag can be used to determine which prediction block to use for the two regions.

For example, when the value of the flag is 0, a mask may be derived that uses the first prediction block in the first region and the second region, and a mask based on the differential sample value may be derived in the difference region. Conversely, when the value of the flag is 1, a mask may be derived that uses the second prediction block in the first region and the second region, and a mask based on the differential sample value may be derived for the difference region.

In the mask for the difference region, the slope of the linear function can be determined based on at least one of the features of the first region and the difference region or the features of the second region and the difference region. Here, the features of the first region and the difference region can mean continuity between the first region and the difference region. Similarly, the features of the second region and the difference region can be continuity between the second region and the difference region. The meaning of continuity is as described above.

FIG. 8 illustrates a parameter-based prediction method as an example according to the present disclosure.

Referring to Fig. 8, the prediction mode of the current block can be determined (S800).

The prediction mode of the current block can be determined as one of a plurality of prediction modes pre-defined in the video encoding device and the video decoding device. The plurality of prediction modes can include at least one of an intra mode, an inter-color prediction mode, an inter mode, or an IBC mode.

Referring to Fig. 8, parameters can be derived based on the prediction mode of the current block (S810).

The parameters according to the present disclosure may mean parameters of a mapping function, offset parameters, parameters of a filter, or parameters for illumination compensation. A method of deriving parameters according to a prediction mode will be described in detail with reference to FIGS. 9 to 11.

Referring to FIG. 8, a prediction block of the current block can be generated based on the reference block and parameters of the current block (S820).

The reference block according to the present disclosure may be a block of a different component type from the current block. For example, if the current block is a chrominance component block, the reference block may be a luminance component block corresponding to the current block. If the current block is a Cr component block, the reference block may be a Cb component block corresponding to the current block.

Alternatively, the reference block may belong to a previously restored region within the current picture to which the current block belongs. Alternatively, the reference block may belong to a different picture from the current block.

The above parameters can be applied to samples of the above reference block to derive prediction samples of the current block. The above parameters can be applied to all samples belonging to the reference block, or can be applied to only some samples within the reference block. The above reference block can be divided into at least two regions, and different parameters can be applied to each region.

FIG. 9 illustrates a parameter-based prediction method in color inter-prediction of intra prediction as an embodiment according to the present disclosure.

The color inter-prediction according to the present disclosure can predict at least one of a second block or a third block based on a pre-reconstructed or pre-predicted first block, as illustrated in FIG. 9. Alternatively, the third block can be predicted based on a pre-reconstructed or pre-predicted second block.

A mapping function having color correlation may be used to generate a prediction block for the second block and/or the third block. The parameters of the mapping function used for the current block may be transmitted from the image encoding device to the image decoding device. Here, the parameters of the mapping function may include at least one of a weight multiplied by the sample value of the first block or one or more offsets added/subtracted from the sample of the first block. It may be interpreted in the same sense in the embodiment described below.

Alternatively, any one of a plurality of mapping functions stored in the form of a pre-defined table or list in the image encoding device and the image decoding device may be selectively used.

Alternatively, the parameters of the mapping function may be derived based on the correlation between the pre-restored sample value(s) around the first block and the pre-restored sample value(s) around the second block, and prediction may be performed based on the derived parameters of the mapping function. In addition, the parameters of the mapping function may be derived based on the correlation between the pre-restored sample value(s) around the first block and the pre-restored sample value(s) around the third block, and prediction may be performed based on the derived parameters. The correlation will be examined in detail with reference to FIGS. 12 to 14.

FIG. 10 illustrates a parameter-based prediction method in intra block copy (IBC) mode as an embodiment according to the present disclosure.

The IBC mode can derive a prediction block of the current block based on a reference block at a position moved by a predetermined block vector (BV) from the current block, as illustrated in Fig. 10.

A reference block specified by a block vector may be derived as a prediction block of the current block, or a corrected reference block may be derived as a prediction block of the current block.

For example, if there is an identical pattern inside the reference block and the current block, the IBC mode can be used. In this case, the illumination may be different depending on the location when the image is acquired, and in this case, the prediction performance can be improved by using the reference block by correcting it through a mapping function, illumination offset, or filtering.

Here, parameters of the mapping function, illumination offset, or filter can be derived from the image encoding device and transmitted to the image decoding device.

Alternatively, an index specifying one or more parameters from a table defining a plurality of pre-defined parameters may be encoded in the image encoding device and transmitted to the image decoding device.

Alternatively, parameters can be derived based on the correlation between the pre-reconstructed sample(s) around the current block and the pre-reconstructed sample(s) around the reference block. This correlation will be examined in more detail with reference to FIGS. 12 to 14.

Figure 11 illustrates a parameter-based prediction method in inter mode.

Inter mode is a mode that derives a prediction block of a current block based on a reference block at a position that is moved by a motion vector (MV) from a block at the same position as the current block within a reference picture, as illustrated in Fig. 11. At this time, if the accuracy (or precision) of the MV is smaller than an integer unit, an interpolation filter can be applied to samples at integer positions within the reference block to derive a prediction block. Here, the interpolation filter can be determined based on a position according to the accuracy of the MV.

In a video encoding device, a reference block with the smallest error is selected, but errors due to illumination changes between pictures may still remain. Therefore, by additionally performing illumination compensation on a prediction block derived through interpolation, prediction performance can be improved.

For the above lighting compensation, a mapping function, an offset, a filter, etc. may be used. The parameters of the mapping function, the offset, or the filter are calculated in the image encoding device and can be transmitted to the image decoding device.

Alternatively, an index specifying one or more parameters from a table defining a plurality of pre-defined parameters may be encoded in the image encoding device and transmitted to the image decoding device.

Alternatively, parameters can be derived based on the correlation between the pre-reconstructed sample(s) around the current block and the pre-reconstructed sample(s) around the reference block. This correlation will be examined in more detail with reference to FIGS. 12 to 14.

FIG. 12 is an example according to the present disclosure, illustrating a relationship between two different areas.

The correlation between the pre-restored sample(s) around the current block and the pre-restored sample(s) around the reference block can be expressed as a two-dimensional dot graph as shown in Fig. 12, and this can be approximated by a linear straight line.

For example, a set of pre-reconstructed samples around a current block is called a current template, and a set of pre-reconstructed samples around a reference block is called a reference template. A dot graph can be derived using sample values belonging to each of the current template and the reference template.

In a specific block, the relationship between the current template and the reference template can be expressed as a linear line as in Fig. 12. In Fig. 12, the first component can mean the reference template, and the second component can mean the current template.

In this case, a linear mapping function can be derived to express the association using the sample values of the current and reference templates. Here, the linear mapping function can be expressed by parameters such as a slope and an intercept, and the slope and intercept can be derived using the sample values of the two templates.

At least one of the slope or intercept of the linear mapping function can be derived via ordinary regression analysis.

Alternatively, parameters can be derived based on the minimum value of the first component and the corresponding sample value of the second component, and the maximum value of the first component and the corresponding sample value of the second component.

Alternatively, the parameters may be derived based on the average of relatively small sample values of the first component and the corresponding average of sample values of the second component, and the average of relatively large sample values of the first component and the corresponding average of sample values of the second component.

FIG. 13 is an example according to the present disclosure, illustrating multiple associations for two different areas.

The correlation between the pre-restored sample(s) around the current block and the pre-restored sample(s) around the reference block can be expressed as a two-dimensional dot graph as shown in Fig. 13, and can be classified into multiple groups based on the characteristics of the samples.

For example, when samples of the current template and the reference template are classified into two groups as illustrated in FIG. 13, a mapping function, offset, or filter that represents a separate association for each group may be used.

The K-mean algorithm can be used to classify the above samples. Alternatively, the samples can be classified into two groups based on the average value of the first component. For example, samples smaller than the average value of the first component can be classified into the first group, and samples larger than or equal to the average value of the first component can be classified into the second group. Alternatively, the samples can be classified into two groups based on the average value of the second component. Alternatively, the samples can be classified into two groups based on the weighted average value of the first component or the second component.

Parameters can be derived for each group using a linear regression algorithm, and predictions can be made based on the parameters.

FIG. 14 is an example according to the present disclosure, showing nonlinear correlations for two different domains.

The correlation between the pre-restored sample(s) around the current block and the pre-restored sample(s) around the reference block can be expressed as a two-dimensional dot graph as shown in Fig. 14, and this can be approximated as a curve.

The above curve can be expressed as a polynomial with the term "Cf(x)" added to the existing linear function. In this case, parameters can be derived through polynomial regression analysis, and prediction can be performed based on the parameters. Here, C is a parameter derived through polynomial regression analysis, and f(x) can be a predefined function according to x.

For example, f(x) could be a function that squares x.

Alternatively, f(x) can be a function that squares x and divides it by the maximum value of the current bit depth.

Alternatively, f(x) may mean a new value derived based on the value of x and the surrounding sample(s). For example, it may mean the first derivative in the horizontal direction. That is, since the curve changes according to the first derivative in the horizontal direction, it may be approximated by various curves, and adaptively mapped according to the texture of the image. Alternatively, f(x) may mean the first derivative in the vertical direction. Alternatively, f(x) may mean the first derivative in the diagonal direction. Alternatively, f(x) may mean the sum of the first derivatives in the horizontal and vertical directions. Alternatively, f(x) may mean activity such as the sum of the first derivatives in the horizontal, vertical, and diagonal directions. Here, the derivative may use a method such as the Sobel operation or the difference. Alternatively, f(x) may mean directionality derived based on the first derivatives in the horizontal and vertical directions. Alternatively, f(x) can mean the sum of directionality and activity derived based on the various differential values mentioned above.

The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combinations of two or more.

Additionally, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

The scope of the present disclosure includes software or machine-executable instructions (e.g., an operating system, an application, firmware, a program, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and being executable on the device or the computer.

Claims (10)

A step of generating at least two prediction blocks for the current block; and A step of generating a final prediction block of the current block by weighting at least two prediction blocks based on a predetermined mask, Each of the above at least two prediction blocks is a unidirectional or bidirectional prediction block, A method for decoding an image, wherein the above bidirectional prediction block is generated based on a weighted sum of an L0 prediction block and an L1 prediction block. In the first paragraph, An image decoding method, wherein the above mask is determined based on geometric segmentation information. In the second paragraph, An image decoding method, wherein the above geometric segmentation information specifies one of a plurality of mask candidates pre-defined in an image decoding device. In the second paragraph, An image decoding method, wherein the geometric segmentation information includes a flag indicating whether an inverted mask is used. In the first paragraph, A method for decoding an image, wherein the mask is generated based on a differential block between at least two prediction blocks. In the first paragraph, The above mask is generated based on a ratio block between the at least two prediction blocks, A method for decoding an image, wherein the ratio block is derived by dividing a sample value belonging to one of the at least two prediction blocks by a sample value at the same position belonging to the other one. In the first paragraph, A method for decoding an image, wherein each of the at least two prediction blocks is generated based on one of inter mode, intra mode, or intra block copy mode. In Article 7, A method for decoding an image, wherein the weights for the above weighted sum are determined based on at least one of the ratio of surrounding blocks to which the intra mode is applied or the POC difference between the current picture to which the current block belongs and the reference picture. A step of generating at least two prediction blocks for the current block; and A step of generating a final prediction block of the current block by weighting at least two prediction blocks based on a predetermined mask, Each of the above at least two prediction blocks is a unidirectional or bidirectional prediction block, A method for encoding an image, wherein the above bidirectional prediction block is generated based on a weighted sum of an L0 prediction block and an L1 prediction block. A step of generating at least two prediction blocks for the current block; A step of generating a final prediction block of the current block by weighting the at least two prediction blocks based on a predetermined mask; A step of deriving a residual block of the current block based on the final prediction block; A step of generating a bitstream by encoding the residual block; and Including the step of transmitting the above bitstream, Each of the above at least two prediction blocks is a unidirectional or bidirectional prediction block, A bitstream transmission method, wherein the above bidirectional prediction block is generated based on a weighted sum of an L0 prediction block and an L1 prediction block.

Images