JP7583869B2 - Image decoding device, image decoding method and program - Google Patents

Description

The present invention relates to an image decoding device, an image decoding method, and a program.

In Non-Patent Document 1, for a technology called motion compensation prediction (hereinafter, MC prediction), there are provided technologies called merge coding (hereinafter, merge) and adaptive vector coding (hereinafter, AMVP) for deriving motion vectors (hereinafter, mv), and two types of interpolation filters are provided for generating an MC prediction image signal using the derived mv.

The first is the same interpolation filter as in Non-Patent Document 2 (hereafter, HEVC filter), and the second is the interpolation filter newly introduced in Non-Patent Document 1 (hereafter, smoothing filter).

Both are applied when the reference of mv is at a sub-pel accuracy position, but the smoothing filter described above is limited to cases where the reference of mv is at a 1/2-pel accuracy position, and is applied only when the half-pel index (hpelIfIdx), which indicates whether the smoothing filter described above is valid, indicates that it is valid. In all other cases, the HEVC filter is applied.

When the block to be coded (hereinafter, the target block) is a merge, the above-mentioned hpelIfIdx is inherited from an adjacent block for which processing has already been completed, and when the target block is an AMVP, the value of hpelIfIdx is determined from the derived mv.

Versatile Video Coding (Draft 6), JVET-N1001 ITU-T H.265 High Efficiency Video Coding

However, in the above-mentioned non-patent document 1, when the target block is a merge and pair-wise average merge, MMVD (Merge Motion Vector Difference), or DMVR (Decoder-side Motion Vector Refinement) is enabled, even if the hpelIfIdx inherited from the adjacent block indicates that the smoothing filter is enabled, the reference destination of mv may not be the 1/2 pixel accuracy position due to pair-wise average merge, MMVD, or DMVR.

At this time, the following two problems arise in the target block and in future blocks that inherit hpelIfIdx and mv from the target block.
1. Mixing different filters in MC prediction of a target block.
2. Propagation of inconsistencies in the validity conditions of the smoothing filter for future blocks (inconsistencies between hpelIfIdx and mv).

The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image decoding device, an image decoding method, and a program that can control the reference pixel precision position of a motion vector derived by pairwise average merging, MMVD, or DMVR according to a half-pel index, or control the half-pel index according to a derived motion vector, thereby eliminating deficiencies in the effective conditions of the smoothing filter of the target block, and further avoiding the propagation of deficiencies in the effective conditions of the smoothing filter to future blocks.

The first feature of the present invention is an image decoding device including: a merge unit configured to decode, from a merge index, a motion vector and a half-pel index indicating whether the motion vector refers to a half-pel accuracy position; a motion vector refinement unit configured to refine the motion vector by MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); a filter determination unit configured to determine whether an interpolation filter is used and the type of the interpolation filter based on the refined motion vector and the half-pel index; and a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, and the merge unit is configured to generate a merge list by a predetermined merge list construction method and decode the motion vector and the half-pel index from the merge list and the merge index.

The second feature of the present invention is an image decoding method comprising: a step A of decoding a motion vector and a half-pel index indicating whether the motion vector refers to a half-pel accuracy position from a merge index; a step B of refining the motion vector by MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); a step C of determining whether an interpolation filter is used and the type of the interpolation filter based on the refined motion vector and the half-pel index; and a step D of generating a motion compensation prediction pixel signal using the interpolation filter, and the gist of the step A is to generate a merge list by a predetermined merge list construction method, and to decode the motion vector and the half-pel index from the merge list and the merge index.

The third feature of the present invention is a program for causing a computer to function as an image decoding device, the image decoding device including: a merge unit configured to decode a motion vector and a half-pel index indicating whether the motion vector refers to a half-pel accuracy position from a merge index; a motion vector refinement unit configured to refine the motion vector by MMVD (Merge Motion Vector Difference) or DMVR (Decoder-side Motion Vector Refinement); a filter determination unit configured to determine whether an interpolation filter is used and the type of the interpolation filter based on the refined motion vector and the half-pel index; and a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter, and the merge unit is configured to generate a merge list by a predetermined merge list construction method and decode the motion vector and the half-pel index from the merge list and the merge index.

According to the present invention, it is possible to control the reference pixel precision position of a motion vector derived by pairwise average merging, MMVD, or DMVR according to a half-pel index, or to control the half-pel index according to a derived motion vector, thereby providing an image decoding device, an image decoding method, and a program that can eliminate deficiencies in the effective conditions of a smoothing filter for a target block and further prevent the propagation of deficiencies in the effective conditions of a smoothing filter to future blocks.

1 is a diagram illustrating an example of a configuration of an image processing system 1 according to an embodiment. FIG. 2 is a diagram illustrating an example of functional blocks of an image encoding device 100 according to an embodiment. 2 is a diagram showing an example of functional blocks of an inter prediction unit 111 of an image encoding device 100 according to an embodiment. FIG. FIG. 2 is a diagram illustrating an example of functional blocks of an image decoding device 200 according to an embodiment. 2 is a diagram illustrating an example of functional blocks of an inter prediction unit 241 of an image decoding device 200 according to an embodiment. FIG. 11 is a flowchart showing an example of a process of determining whether an interpolation filter is applied and the type of the filter in a filter determination unit 111D1/241C1 according to an embodiment. 11 is a flowchart illustrating an example of a process for determining a half-pel index (hpelIfIdx) according to an embodiment. FIG. 1 is a diagram for explaining the determination process of Non-Patent Document 1. A figure showing an example of functional blocks of an mv refinement unit 241B of an image decoding device 200 according to an embodiment. 11 is a flowchart illustrating an example of a merge list construction process according to an embodiment. FIG. 11 is a diagram illustrating an example of a merge list generated in a merge list construction process according to an embodiment. FIG. 13 illustrates an example of spatial merging according to an embodiment. FIG. 13 illustrates an example of a temporal merge according to one embodiment. 11 is a diagram illustrating an example of a motion vector scaling process in a temporal merge according to an embodiment. FIG. FIG. 13 is a diagram illustrating an example of history merging according to an embodiment. FIG. 11 is a diagram illustrating an example of a set of new motion vectors and half-pel indexes calculated by pairwise average merging according to an embodiment. 11 is a flowchart illustrating an example of a process for setting a new half-pel index generated by pairwise average merging according to an embodiment. 11A and 11B are diagrams for explaining an example of a method for resolving deficiencies in the setting conditions of half-pel indexes in pairwise average merging according to an embodiment. 11A and 11B are diagrams for explaining an example of a method for resolving deficiencies in the setting conditions of half-pel indexes in pairwise average merging according to an embodiment. 11A and 11B are diagrams for explaining an example of a method for resolving deficiencies in the setting conditions of half-pel indexes in pairwise average merging according to an embodiment. 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in MMVD according to an embodiment. FIG. 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in MMVD according to an embodiment. FIG. 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in MMVD according to an embodiment. FIG. 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in MMVD according to an embodiment. FIG. 11 is a diagram for explaining an example of a method for resolving deficiencies in the setting conditions of a half-pel index in MMVD according to an embodiment. FIG. FIG. 13 is a diagram for explaining a modification example in which a motion vector derived by pairwise average merging according to an embodiment is modified by MMVD or DMVR. FIG. 13 is a diagram for explaining an example of derivation of a motion vector by pairwise average merging according to an embodiment. FIG. 13 is a diagram for explaining an example of derivation of a motion vector by pairwise average merging according to an embodiment.

The following describes embodiments of the present invention with reference to the drawings. Note that the components in the following embodiments can be replaced with existing components as appropriate, and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

First Embodiment
An image processing system 10 according to a first embodiment of the present invention will now be described with reference to Figures 1 to 28. Figure 1 is a diagram showing an image processing system 10 according to this embodiment.

As shown in FIG. 1, the image processing system 10 according to this embodiment has an image encoding device 100 and an image decoding device 200.

The image encoding device 100 is configured to generate encoded data by encoding an input image signal. The image decoding device 200 is configured to generate an output image signal by decoding the encoded data.

The encoded data may be transmitted from the image encoding device 100 to the image decoding device 200 via a transmission path. The encoded data may be stored in a storage medium and then provided from the image encoding device 100 to the image decoding device 200.

(Image encoding device 100)
The image encoding device 100 according to this embodiment will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to this embodiment.

As shown in FIG. 2, the image encoding device 100 includes an inter prediction unit 111, an intra prediction unit 112, a subtractor 121, an adder 122, a transform/quantization unit 131, an inverse transform/inverse quantization unit 132, an encoding unit 140, an in-loop filter processing unit 150, and a frame buffer 160.

The inter prediction unit 111 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

Specifically, the inter prediction unit 111 is configured to identify a reference block included in a reference frame by comparing a frame to be coded (hereinafter, the target frame) with a reference frame stored in the frame buffer 160, and to determine a motion vector (mv) for the identified reference block.

The inter prediction unit 111 is also configured to generate a prediction signal included in a block to be coded (hereinafter, a target block) for each target block based on the reference block and the motion vector. The inter prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (intra-frame prediction).

Specifically, the intra prediction unit 112 is configured to identify a reference block included in the target frame and generate a prediction signal for each target block based on the identified reference block. The intra prediction unit 112 is also configured to output the prediction signal to the subtractor 121 and the adder 122.

Here, the reference block is a block that is referenced for the target block. For example, the reference block is a block adjacent to the target block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the transform/quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal that is the difference between the prediction signal generated by intra prediction or inter prediction and the input image signal.

The adder 122 is configured to add the prediction signal to the prediction residual signal output from the inverse transform/inverse quantization unit 132 to generate a pre-filter decoded signal, and output the pre-filter decoded signal to the intra prediction unit 112 and the in-loop filter processing unit 150.

Here, the unfiltered decoded signal constitutes a reference block used by the intra prediction unit 112.

The transform/quantization unit 131 is configured to perform a transform process on the prediction residual signal and to obtain coefficient level values. Furthermore, the transform/quantization unit 131 may be configured to quantize the coefficient level values.

The conversion process is a process of converting a prediction residual signal into a frequency component signal. In such a conversion process, a basis pattern (transformation matrix) corresponding to a discrete cosine transform (DCT) may be used, or a basis pattern (transformation matrix) corresponding to a discrete sine transform (DST) may be used.

The inverse transform and inverse quantization unit 132 is configured to perform inverse transform processing of the coefficient level values output from the transform and quantization unit 131. Here, the inverse transform and inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level values prior to the inverse transform processing.

Here, the inverse transformation process and inverse quantization are performed in the reverse order to the transformation process and quantization performed by the transformation/quantization unit 131.

The encoding unit 140 is configured to encode the coefficient level values output from the transform/quantization unit 131 and output the encoded data.

Here, for example, the coding is entropy coding, which assigns codes of different lengths based on the probability of occurrence of coefficient level values.

The encoding unit 140 is also configured to encode control data used in the decoding process in addition to the coefficient level values.

Here, the control data may include size data such as coding block (CU: Coding Unit) size, prediction block (PU: Prediction Unit) size, and transform block (TU: Transform Unit) size.

The control data may also include header information such as a sequence parameter set (SPS), a picture parameter set (PPS), and a slice header, as described below.

The in-loop filter processing unit 150 is configured to perform filtering on the unfiltered decoded signal output from the adder 122, and to output the filtered decoded signal to the frame buffer 160.

Here, for example, the filtering process is a deblocking filtering process that reduces distortion that occurs at the boundaries of blocks (encoding blocks, prediction blocks, or transformation blocks).

The frame buffer 160 is configured to store reference frames used by the inter prediction unit 111.

Here, the filtered decoded signal constitutes a reference frame used by the inter prediction unit 111.

(Inter prediction unit 111)
Hereinafter, the inter prediction unit 111 of the image encoding device 100 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding device 100 according to this embodiment.

As shown in FIG. 3, the inter prediction unit 111 has an mv derivation unit 111A, an AMVR unit 111B, an mv refinement unit 111B, and a prediction signal generation unit 111D.

The inter prediction unit 111 is an example of a prediction unit configured to generate a prediction signal to be included in a target block based on a motion vector.

As shown in FIG. 3, the mv derivation unit 111A has an AMVP (Adaptive Motion Vector Prediction) unit 111A1 and a merge unit 111A2, and is configured to receive the target frame and reference frame from the frame buffer 160 as input and obtain a motion vector.

The AMVP unit 111A1 is configured to identify a reference block contained in the reference frame by comparing the target frame with the reference frame, and to search for a motion vector for the identified reference block.

The above-mentioned search process is also performed on multiple reference frame candidates to determine the reference frame and motion vector to be used for prediction of the target block, and output to the downstream prediction signal generation unit 111D.

A maximum of two reference frames and two motion vectors can be used for one block. When only one set of reference frames and motion vectors is used for one block, it is called "uni-prediction," and when two sets of reference frames and motion vectors are used, it is called "bi-prediction." In the following, the first set is called "L0" and the second set is called "L1."

Furthermore, when the AMVP unit 111A finally transmits the above-mentioned determined motion vector to the decoding device, in order to reduce the amount of coding, it selects, from among the motion vector predictor (mvp) candidates derived from adjacent coded motion vectors, an mvp that has a small difference from the motion vector of the target block, i.e., a motion vector difference (mvd).

The index indicating the selected mvp and mvd and the index indicating the reference frame (hereinafter, Refidx) are coded by the coding unit 140 and transmitted to the image decoding device 200. This process is generally called adaptive motion vector predictive coding (AMVP: Adaptive: Motion Vector Prediction).

Note that the above-mentioned motion vector search method, reference frame and motion vector determination method, mvp selection method, and mvd calculation method can use known techniques, so details are omitted.

The AMVR unit 111B has an AMVR (Adaptive Motion Vector Resolution) function that changes the transmission accuracy of the mvd calculated by the AMVP unit 111A.

Since mvd is derived from the sum of the motion vector of the target block and mvp as described above, changing the transmission accuracy of mvd means changing the accuracy of the motion vector of the target block itself.

In Non-Patent Document 1, three variations in the transmission accuracy of mvd using AMVR are provided. The normal transmission accuracy of mvd is 1/16 pixel accuracy, and in this case, the motion vector ultimately refers to a position with 1/16 pixel accuracy, but when AMVR is enabled, the transmission accuracy of mvd can be selected from 1/4 pixel accuracy, 1/2 pixel accuracy, and 1 pixel accuracy (i.e., integer pixel accuracy).

However, if the target block is affine, the transmission accuracy of mvd is selected to be 1/4 pixel accuracy, 1/16 pixel accuracy, or 1 pixel accuracy. Here, affine refers to Affine in Non-Patent Document 1.

Furthermore, in Non-Patent Document 1, only when 1/2 pixel precision is selected as the transmission precision of the above-mentioned mvd, an interpolation filter other than the interpolation filter applied when a precision other than 1/2 pixel precision is selected is selected in the downstream prediction signal generation unit 111D. Details will be described later.

In addition, if the AMVR unit 111B determines that AMVR processing is enabled, it transmits to the image decoding device 200 a flag indicating that AMVR is enabled and an index indicating the accuracy to which mvd is corrected by AMVR.

The merge unit 111A2 does not search and derive motion information of the target block as in the AMVP unit 111A1, nor transmits it as mvd, which is the difference from an adjacent block. Instead, it takes the target frame and the reference frame as input, and inherits and uses the motion information of the reference block as is, using an adjacent block in the same frame as the target block or a block at the same position in a frame other than the target frame as the reference block. This process is generally called merge coding (hereinafter, merge).

When the target block is a merge block, a merge list for the target block is first created. The merge list is a list that lists multiple combinations of reference frames and motion vectors. An index (hereinafter, a merge index) is assigned to each combination, and instead of encoding the Refidx and motion vector information separately, only the above-mentioned merge index is encoded and transmitted to the image decoding device 200.

Here, by standardizing the method of creating a merge list between the image encoding device 100 and the image decoding device 200, the image decoding device 200 can decode the Refidx and motion vector information from only the merge index information. The method of creating a merge list will be described in detail later.

The mv refinement unit 111C is configured to perform a refinement process to modify the motion vectors output from the merge unit 111A2. Details will be described later.

The prediction signal generation unit 111D is configured to receive a motion vector as input and output an MC prediction image signal, and has a filter determination unit 111D1 and a filter application unit 111D2.

The filter determination unit 111D1 determines whether an interpolation filter is applied and the filter type based on the motion vector. Details will be described later.

The filter application unit 111D2 is configured to generate a prediction signal from the selected interpolation filter, motion vector, and reference frame when the filter determination unit 111D1 determines that the interpolation filter is valid.

If the filter determination unit 111D1 determines that the interpolation filter is invalid, the interpolation filter is not used and a prediction signal is generated using the motion vector and reference frame.

(Image Decoding Device 200)
The image decoding device 200 according to this embodiment will be described below with reference to Fig. 4. Fig. 4 is a diagram showing an example of functional blocks of the image decoding device 200 according to this embodiment.

As shown in FIG. 4, the image decoding device 200 includes a decoding unit 210, an inverse transform/inverse quantization unit 220, an adder 230, an inter prediction unit 241, an intra prediction unit 242, an in-loop filter processing unit 250, and a frame buffer 260.

The decoding unit 210 is configured to decode the encoded data generated by the image encoding device 100 and to decode the coefficient level values.

Here, the decoding is, for example, entropy decoding, which is the reverse procedure of the entropy encoding performed by the encoding unit 140.

The decoding unit 210 may also be configured to obtain control data by decoding the encoded data.

As mentioned above, the control data may include size data such as the coding block size, the prediction block size, and the transform block size.

The inverse transform/inverse quantization unit 220 is configured to perform inverse transform processing of the coefficient level values output from the decoding unit 210. Here, the inverse transform/inverse quantization unit 220 may be configured to perform inverse quantization of the coefficient level values prior to the inverse transform processing.

Here, the inverse transformation process and inverse quantization are performed in the reverse order to the transformation process and quantization performed by the transformation/quantization unit 131.

The adder 230 is configured to add the prediction signal to the prediction residual signal output from the inverse transform/inverse quantization unit 220 to generate a pre-filter decoded signal, and output the pre-filter decoded signal to the intra prediction unit 242 and the in-loop filter processing unit 250.

Here, the unfiltered decoded signal constitutes a reference block used by the intra prediction unit 242.

The inter prediction unit 241, like the inter prediction unit 111, is configured to generate a prediction signal by inter prediction (inter-frame prediction).

Specifically, the inter prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from the encoded data and a reference signal included in a reference frame. The inter prediction unit 241 is configured to output the prediction signal to the adder 230.

The intra prediction unit 242, like the intra prediction unit 112, is configured to generate a prediction signal by intra prediction (intra-frame prediction).

Specifically, the intra prediction unit 242 is configured to identify a reference block included in the target frame and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230.

Similar to the in-loop filter processing unit 150, the in-loop filter processing unit 250 is configured to perform filtering on the unfiltered decoded signal output from the adder 230, and to output the filtered decoded signal to the frame buffer 260.

Here, for example, the filtering process is a deblocking filtering process that reduces distortion that occurs at the boundaries of blocks (encoded blocks, predicted blocks, transformed blocks, or subblocks obtained by dividing them).

Like frame buffer 160, frame buffer 260 is configured to store reference frames used by inter prediction unit 241.

Here, the filtered decoded signal constitutes a reference frame used by the inter prediction unit 241.

(Inter prediction unit 241)
The inter prediction unit 241 according to this embodiment will be described below with reference to Fig. 5. Fig. 5 is a diagram showing an example of functional blocks of the inter prediction unit 241 according to this embodiment.

As shown in FIG. 5, the inter prediction unit 241 has an mv decoding unit 241A, an mv refinement unit 241B, and a prediction signal generation unit 111C.

The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The mv decoding unit 241A has an AMVP unit 241A1 and a merge unit 241A2, and is configured to obtain motion vectors by decoding the target frame and reference frame input from the frame buffer 260, and the control data received from the image encoding device 100.

The AMVP unit 241A1 is configured to receive a target frame, a reference frame, and indices indicating mvp and mvd from the image encoding device 100, Refidx, and an index indicating the transmission accuracy of mvd from the AMVR unit 111B, and to decode the motion vector. A known method can be used to decode the motion vector, so details thereof will be omitted.

The merge unit 241A2 is configured to receive a merge index from the image encoding device 100 and decode the motion vector.

Specifically, the merge unit 241A2 is configured to construct a merge list in the same manner as the image encoding device 100, and to obtain a motion vector corresponding to the received merge index from the constructed merge list. The method of constructing the merge list will be described in detail later.

The mv refinement unit 241B is configured to perform a refinement process to modify motion vectors, similar to the refinement unit 111C.

The prediction signal generation unit 241C has a filter determination unit 241C1 and a filter application unit 241C2, and is configured to generate a prediction signal based on a motion vector, similar to the prediction signal generation unit 111C.

Note that the filter determination unit 241C1 of the image decoding device 200 has exactly the same configuration as the filter determination unit 111D1 of the image encoding device 100, so in the following, in this embodiment, the operation of the filter determination unit 111D1 will be described as a representative example.

(Interpolation filter determination process)
Hereinafter, the process of determining whether or not an interpolation filter is applied and the filter type in the filter determination unit 111D1/241C1 according to this embodiment will be described with reference to FIG.

Figure 6 is a flowchart showing an example of the process order for determining whether or not to apply an interpolation filter and the filter type in the filter determination unit 111D1 according to this embodiment.

As shown in FIG. 6, in step S6-1, the filter determination unit 111D1 determines whether the reference position of mv is a sub-pel position. If the reference position of mv is a sub-pel position, the process proceeds to step S6-2. If the reference position of mv is not a sub-pel position, i.e., if the reference position of mv is an integer pixel value position, the process proceeds to step S6-3.

In step S6-2, the filter determination unit 111D1 determines that an interpolation filter is to be applied, and the process proceeds to step S6-4.

In step S6-3, the filter determination unit 111D1 determines that an interpolation filter is not to be applied and ends this process.

In step S6-4, the filter determination unit 111D1 determines whether the half-pel index (hpelIfIdx), which will be described later, is invalid, i.e., whether it is "0". If hpelIfIdx is "0", the process proceeds to step S6-5, and if hpelIfIdx is valid, i.e., not "0", the process proceeds to step S6-6.

In step S6-5, the filter determination unit 111D1 determines that the HEVC filter will be used as the interpolation filter, and ends this process.

Here, the HEVC filter is an 8-tap linear interpolation filter similar to that described in Non-Patent Document 2.

In step S6-6, the filter determination unit 111D1 determines that a smoothing filter is to be used as the interpolation filter.

Here, the smoothing filter may be, for example, a 6-tap Gaussian filter as used in Non-Patent Document 1. In addition, the value indicated by hpelIfIdx may be used to set the use of multiple filters as the interpolation filter.

In this way, when the reference position of mv is a decimal pixel position, it is possible to adaptively switch between two or more types of interpolation filters based on hpelIfIdx, and it is possible to select the interpolation filter to be used according to the image characteristics, which is expected to result in improved coding performance.

(Half-pel index (hpelIfIdx) setting process)
Hereinafter, the process of determining the half-pel index (hpelIfIdx) according to this embodiment will be described with reference to FIG.

Figure 7 is a flowchart showing an example of the determination process order of hpelIfIdx in the filter determination unit 111D1 according to this embodiment.

As described above, hpelIfIdx is related to determining the type of interpolation filter, and if hpelIfIdx is "0", i.e., invalid, the same interpolation filter as in Non-Patent Document 2 is used. On the other hand, if hpelIfIdx is not "0", i.e., valid, a smoothing filter is used.

When the current block is a merge block, hpelIfIdx is obtained from the reference block by combining the motion vector and RefIdx, and is associated with the merge index.

On the other hand, if the target block is not in merge mode and the motion vector is decoded by AMVP, the value of hpelIfIdx is determined based on whether or not the above-mentioned AMVR is applied and on certain conditions.

As shown in FIG. 7, in step S7-1, the filter determination unit 111D1 determines whether AMVR is enabled. For example, the determination may be made by decoding a flag (amvr_flag) indicating whether AMVR is applied, as in Non-Patent Document 1.

If it is determined in step S7-1 that the AMVR is valid, the process proceeds to step S7-2; if it is determined that the AMVR is invalid, the process proceeds to step S7-3.

In step S7-2, the filter determination unit 111D1 determines whether the value of hpelIfIdx is "0" based on predetermined condition 1.

If it is determined in step S7-2 that the specified condition 1 is satisfied, the process proceeds to step S7-4; if it is determined that the specified condition 1 is not satisfied, the process proceeds to step S7-5.

Here, in the predetermined condition 1 of step S7-2, the index (amvr#precision#idx) indicating the transmission precision of AMVR may be decoded and judged, and if the transmission precision of AMVR is 1/2 pixel precision, it is judged that the predetermined condition 1 is satisfied, and otherwise it is judged that the predetermined condition 1 is not satisfied.

In addition, in the predetermined condition 1 of step S7-2, it may be determined whether the target block is affine. If the target block is affine, it is determined that the predetermined condition 1 is not satisfied, and if the target block is not affine, it is determined that the predetermined condition 1 is satisfied if amvr#precision#idx indicates that the transmission precision of mvd is 1/2 pixel precision.

Here, affine refers to the Affine method used in Non-Patent Document 1, and since the present invention can use known methods, a detailed explanation is omitted.

In addition, in the predetermined condition 1 of step S7-2, it may be determined whether the target block is an IBC. If the target block is an IBC, it is determined that the predetermined condition 1 is not satisfied, and if the target block is not an IBC, it is determined that the predetermined condition 1 is satisfied if amvr#precision#idx indicates that the transmission precision of mvd is 1/2 pixel precision.

Here, IBC refers to the Intra Block Copy (IBC) adopted in Non-Patent Document 1, and since the present invention can adopt known methods, the explanation is omitted.

In step S7-3, the filter determination unit 111D1 determines that hpelIfIdx is "0" and terminates this process.

In step S7-4, the filter determination unit 111D1 determines that hpelIfIdx is "1" and ends this process.

In step S7-5, the filter determination unit 111D1 determines that hpelIfIdx is "0" and terminates this process.

Up to this point, the determination process of hpelIfIdx has been explained using a flowchart, but as in the determination process of Non-Patent Document 1 shown in Figure 8, the filter determination unit 111D1 may determine the Avmrshift value based on the decoded results of amvr#flag, amvr#precision#idx, and inter#affine#flag and whether the target block is an IBC, and determine the value of hpelIfidx based on the determined Amvr#shift value.

(MV Refinement Department)
Hereinafter, the mv refinement unit 241B according to this embodiment will be described with reference to Fig. 9. Note that since the mv refinement unit 111C of the encoding device has exactly the same configuration as the mv refinement unit 241B of the decoding device, in this embodiment, the mv refinement unit 241B will be representatively described.

Figure 9 shows an example of a functional block of the mv refinement unit 241B in this embodiment.

As shown in FIG. 9, the mv refinement unit 241B has an MMVD unit 241B1 and a DMVR unit 241B2.

In the MMVD section 241B1 and the DMVR section 241B2, specifically, the motion vectors are refined by MMVD (Merge mode with MVD) and DMVR (Decoder side Motion Vector Refinement) adopted in Non-Patent Document 1.

Specifically, the MMVD unit 241B1 is configured to set the correction range of the reference position by distance and direction (vertical and horizontal) based on the reference position of the motion vector output from the merge unit 111A2, as shown in FIG. 9.

The MMVD unit 241B1 is configured to identify the correction position with the smallest specified cost within the correction range for the correction distance, and to perform a refinement process to correct the motion vector based on the correction reference position.

The DMVR unit 241B2 is also configured to set a search range based on a reference position identified by the motion vector output from the merge unit 111A2, identify a modified reference position with the smallest predetermined cost from within the search range, and perform a refinement process to modify the motion vector based on the modified reference position.

Here, the mv refinement unit 241B also determines whether MMVD and DMVR are applied to the target block, but since the present invention can use the known method described in Non-Patent Document 1 for this determination process, a description thereof will be omitted.

Note that if both MMVD and DMVR are invalid, the motion vector output from the merge unit 111A2 is not modified and is output as is to the downstream prediction signal generation unit 241C.

(How to build a merge list)
The merge list construction process according to this embodiment will be described below with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the merge list construction process according to this embodiment.

As shown in FIG. 10, the merge list construction process according to the present invention is composed of a total of five merge list construction processes, similar to that of Non-Patent Document 1.

Specifically, it consists of a spatial merge in step S10-1, a time merge in step S10-2, a history merge in step S10-3, a pairwise average merge in step S10-4, and a merge list construction process using a zero merge in step S10-5. Each merge list construction process will be described later.

Figure 11 shows an example of a merge list generated by the merge list construction process according to this embodiment.

As described above, the merge list is a list in which the motion vector, Refidx, and hepelIfIdx corresponding to the merge index are registered.

The maximum number of merge indexes is set to "5" in Non-Patent Document 1, but may be freely set according to the designer's intention.

In addition, mvL0, mvL1, RefIdxL0, and RefIdxL1 in Figure 11 indicate the motion vector and reference image index of the reference image lists L0 and L1, respectively.

Here, reference image lists L0 and L1 refer to lists in which reference frames are registered, and the reference frames are identified by RefIdx.

Note that the merge list shown in FIG. 11 shows both L0 and L1 motion vectors and reference image indices, but some reference blocks may use one-way prediction. In such cases, the motion vectors and reference image indices for one direction are registered in the merge list. Also, if there is no motion vector in the reference block, the process of constructing a merge list for that reference block is skipped.

(Spatial Merge)
12 is a diagram showing spatial merging. Spatial merging is a technique for inheriting mv, RefIdx, and hpelIfIdx from an adjacent block that exists in the same frame as the current block.

Specifically, the above-mentioned mv, Refidx, and hpelfIdx are inherited from adjacent blocks that are in the positional relationship shown in FIG. 12, and the processing order may be the order shown in FIG. 12, as in Non-Patent Document 1.

In the merge list process, parameters are registered in the merge list in the processing order described above, but there is a check mechanism to ensure that the same motion vector and reference image index are not registered in the merge list twice. This is called pruning processing.

The reason for the existence of the pruning process is to increase the variety of motion vectors and reference image indexes registered in the merge list. From the viewpoint of the image encoding device 100, this means that a motion vector that minimizes a specified cost according to the image characteristics can be selected. In the image decoding device 200, a prediction signal with high prediction accuracy can be generated based on the selected motion vector and reference image index, which can ultimately be expected to have an effect of improving encoding performance.

In spatial merging, for example, when adjacent block _A1 is registered, the identity of the motion vector and reference image index registered in adjacent block _A1 is confirmed in adjacent block _B1 , which is the next block to be processed, and if identity is confirmed, the motion vector and reference image index of adjacent block _B1 are not registered in the merge list.

In addition, in Non-Patent Document 1, for adjacent blocks _B0 , _A0 , and _B2 , in addition to checking the identity of the motion vectors and reference image indexes already registered in the merge list, there is also a confirmation as to whether the target block is a triangle merge.

In this embodiment, a triangle merge judgment may be added to the pruning process, and a known method can be used for triangle merging, so a description is omitted.

In addition, in Non-Patent Document 1, the maximum number of merge indexes that can be registered by spatial merging is set to "4", and as for adjacent block _B2 , if four merge indexes have already been registered by the previous spatial merging process, the process of _B2 is skipped.

In this embodiment, similarly to Non-Patent Document 1, the process for the adjacent block _B2 may be determined based on the number of existing merge indexes registered.

(Time Merge)
Fig. 13 is a diagram showing temporal merging. Temporal merging is a technique for identifying a block ( _C1 in Fig. 13) that exists in a different frame from the target block but is located at the same position as the target block, as a reference block, and inheriting a motion _vector and a reference image index.

In non-patent document 1, the maximum number of merge indexes that can be registered in a merge list using such a time merge list is "1", and in this embodiment, the maximum number of registrations can be set in a similar manner.

Motion vectors inherited in temporal merging are characterized by being scaled, and Figure 14 shows this scaling process.

Specifically, as shown in FIG. 14, mv of the reference block is scaled as follows based on the distance tb between the reference frame of the target block and the frame in which the target frame exists, and the distance td between the reference frame of the reference block and the reference frame of the reference block:

mv'=(td/tb)×mv
In the temporal merge, this scaled mv' is registered in the merge index as the motion vector corresponding to the merge index.

(History Merge)
15 is a diagram showing history merge. History merge is a technique in which a motion vector, a reference image index, and a half-pel index of an inter-prediction block that has been coded before a target block are separately stored in a storage area called a history merge table, and when the merge list is not filled by the previous spatial merge process or the time merge process, the merge index registered in the history merge table is registered in the merge list.

Figure 15 shows an example of the history merge table construction process, in which the motion vector, reference image index, and half-pel index corresponding to a previously coded block are registered in the history merge table using the history merge index.

In Non-Patent Document 1, the maximum number of history merge indexes that can be registered in this history merge table is set to a maximum of "6", but this may be freely set according to the designer's intention.

The process of registering history merge indexes in this history merge table is a FIFO process, and when the history merge table becomes full, the last registered history merge index is deleted each time a new history merge table is added.

In addition, the history merge index registered in the history merge table may be initialized when the target block crosses a coding tree block (CTU), as in Non-Patent Document 1.

(Pairwise average merging)
FIG. 16 is a diagram showing a set of a new motion vector and a half-pel index calculated by pairwise average merging.

Pairwise average merging is a technique that generates new motion vectors, reference image indexes, and half-pel indexes using motion vectors, reference image indexes, and half-pel indexes corresponding to two pairs of merge indexes already registered in the merge list.

The two sets of merge indexes used for pairwise average merging may be the 0th and 1st merge indexes registered in the merge list, as in Non-Patent Document 1, or the designer may freely set a different combination of the two sets.

The method for generating a new motion vector in pairwise average merging is to average the motion vectors corresponding to two pairs of merge indexes in the merge list, as shown in Figure 16.

Specifically, for example, when there are two motion vectors corresponding to two sets of merge indexes (i.e., in the case of bi-prediction), the pair-wise average merge motion vectors mvL0Avg and mvL1Avg are calculated independently using the motion vectors _mvL0P0 / _mvL1P0 and _mvL0P1 /mvL1P1 in the L0 and _L1 directions as follows:

mvL0Avg=(mvL0P ₀ +mvL0P ₁ )/2
mvL1Avg=(mvL1P ₀ +mvL1P ₁ )/2
Here, if either mvL0P ₀ /mvL1P ₀ or mvL0P ₁ /mvL1P ₁ does not exist, the above calculation is performed excluding the non-existent motion vector.

At this time, Non-Patent Document 1 specifies that the reference image indexes _RefIdxL0P0 and _RefIdxL1P0 associated with the merge index _P0 should always be used as the reference image indexes associated with the pairwise average merge index.

In addition, the new half-pel index hpelIfIdxAvg generated by the pair-wise average merge is set based on two pairs of half-pel indexes hpelIfIdxP0 and hpelIfIdxP1 corresponding to two pairs of merge indexes already registered in the merge list, as shown in Figures 17( _a) and 17(b ₎ .

Specifically, in step S17-1 it is determined whether hpelIfIdxP ₀ and hpelIfIdxP ₁ are the same. If they are the same, the process proceeds to step S17-2, and if they are not the same, the process proceeds to step S17-3.

At step S17-2, hpelIfIdxAvg is set to _hpelIfIdxP0 .

In step S17-3, hpelIfIdxAvg is set to invalid, i.e., "0".

Here, if the possible values of _hpelIfIdxP0 and _hpelIfIdxP1 are "0" or "1", the processes of steps S17-1 and S17-3 shown in FIG. 17A may be replaced with processes such as steps S17-4 to S17-6 in FIG. 17B.

(Fixed the half-pel index setting conditions for pairwise average merging)
Hereinafter, an example of a method for resolving deficiencies in the setting conditions of the half-pel index in pairwise average merging according to this embodiment will be described with reference to FIGS.

Figure 18 shows an example in which the pairwise average merging according to this embodiment results in an inadequacy in the setting conditions for the half-pel index.

Specifically, as described above, the new motion vector and half-pel index generated by pairwise average merging are derived by averaging the motion vectors corresponding to two different sets of merge indexes already registered in the merge list and determining the half-pel index value according to predetermined conditions.

In the example shown in Figure 16, the hpelIfIdxAvg generated by pairwise average merging is "1", and both mvL0Avg and mvL1Avg are at 1/2 pixel accuracy positions, so the target block satisfies the validity conditions for the smoothing filter described above.

On the other hand, even if hpelIfIdxAvg generated by pairwise average merging is valid, i.e., "1", and mvL0P ₀ , mvL1P ₀ , mvL0P ₁ and mvL1P ₁ are all at 1/2 pixel accuracy positions as shown in Fig. 18, if one of mvL0P ₀ and mvL0P ₁ to be averaged is at a 1/2 pixel position and the other is at an integer pixel position, mvL0Avg will refer to a 1/4 pixel accuracy position. This is the same for mvL1Avg generated by averaging mvL1P ₀ and mvL1P ₁ as shown in Fig. 18.

Considering that the target block generates a prediction signal using a motion vector obtained by pairwise average merging, in this case, even though the half-pel index inherited from the surrounding reference blocks is "1", the motion vector references a quarter-pixel accuracy position, and therefore does not satisfy the effective condition for the smoothing filter. This causes the use of different interpolation filters between blocks, since an HEVC filter is used as the interpolation filter for the target block, while a smoothing filter is used as the interpolation filter for the surrounding reference blocks.

Furthermore, future blocks that reference the target block inherit half-pel index = 1 and mv whose reference is at a quarter-pixel accuracy position from the target block, which leads to the propagation of a mismatch with the effective conditions of this smoothing filter to future blocks.

This issue has not been resolved by known methods, so the following two methods are proposed in this embodiment. Figures 19 and 20 show these two solutions.

First, the first method is shown in FIG. 19. Specifically, in FIG. 19, in the case described above, that is, as shown in step S19-4, even if the hpelIfIdxAvg generated by inheritance is "1", when mvL0Avg or mvL1Avg refers to a position other than a 1/2 pixel accuracy position, the process proceeds to step S19-5, where the inherited hpelIfIdxAvg is corrected from "1" to "0". This makes it possible to avoid improper setting of the validity conditions of the smoothing filter for the target block and future blocks as described above, and the associated propagation.

The second method is shown in FIG. 20. The process shown in FIG. 20 is the same as the process shown in FIG. 19 except for step S20-5 in FIG. 20 and step S19-5 in FIG. 19, and in this embodiment, the solution method shown in step S20-5 is proposed as another solution method equivalent to step 19-5.

Specifically, in step S20-5, if the reference destination of mvL0Avg and mvL1Avg is not a 1/2 pixel accuracy position even though hpelIfIdxAvg was "1" in the previous step, the reference destination of mvL0Avg and mvL1Avg is rounded to a 1/2 pixel accuracy position. This makes it possible to avoid improper setting of the effective conditions of the smoothing filter and the associated propagation for the target block and future blocks.

The former solution is a technique for modifying hpelIfIdx according to mv, and the latter solution is a technique for modifying mv according to hpelIfIdx. Designers are free to decide which solution to adopt.

(Resolving deficiencies in MMVD)
The above-mentioned deficiencies in the setting conditions of the smoothing filter based on the half-pel index and the motion vector in the pair-wise average merging may also occur in a target block for which MMVD is valid, because even if the reference destination of the motion vector output from the merging unit 241A2 is a 1/2 pixel accuracy position and the half-pel index is "1", the motion vector may be corrected by MMVD in the MMVD unit 241B1, and the reference destination of the motion vector may shift from the 1/2 pixel accuracy position.

Therefore, in this invention, we propose a solution in which hpelIfIdx is modified according to mv, as in pairwise average merging, and a solution in which mv is modified according to hpelIfIdx.

Below, with reference to Figures 21 to 23, an example of a method for resolving deficiencies in the setting conditions of the half-pel index in MMVD according to this embodiment will be described.

Figure 21 shows the MMVD. As described above, the MMVD is configured to use the motion vector output by the merge unit 241A2 as a reference and to correct the reference position of a new motion vector based on the values of mmvd#distance#idx and mmvd#direction#idx shown in Figure 21.

In Non-Patent Document 1, mvd is calculated based on the two indexes mentioned above and added to the original motion vector. When such mvd has 1/4 pixel accuracy, the motion vector after applying MMVD will be shifted from the 1/2 pixel accuracy position.

Note that even if the mvd of MMVD has integer precision, the pixel precision of the original motion vector does not change, unlike the case of AMVR described above. For example, if the original motion vector is at a 1/2 pixel precision position and mvd is "1", the new motion vector will still refer to the 1/2 precision position.

Therefore, to resolve the above-mentioned cases, two solutions are shown below using Figures 22 and 23.

Figure 22 shows a solution that modifies hpelIfIdx depending on mv.

Specifically, in step S22-1, if hpelIfIdx is "1", the process proceeds to step S22-2, and if hpelIfIdx is not "1", the process ends.

If the specified condition 2 is met in step S22-2, this process ends, and if the specified condition 2 is not met, this process proceeds to step S22-3.

Here, the predetermined condition 2 may be supplemented with a determination as to whether the mv after application of MMVD refers to a 1/2 pixel precision position. Also, a determination may be supplemented as to whether the mv input to the MMVD unit 241B1 is at a 1/2 pixel position and the mvd added by MMVD is at 1/2 pixel precision.

In step S22-3, since mv is shifted from the 1/2 pixel accuracy position due to MMVD in step S22-2, the half-pel index is corrected from "1" to "0" and this process ends.

Figure 23 shows a solution where mv is modified according to hpelIfIdx.

Specifically, another solution equivalent to the solution shown in step S22-3 in FIG. 22, that is, a process of rounding mv after applying MMVD to a 1/2 pixel precision position, is implemented in step S23-3.

[Modification: Variation of rounding process related to MMVD]
The following describes variations in the rounding process of mv when mv is modified by MMVD as shown in step S23-3 above.

The above method involves directly rounding mv after applying MMVD to a 1/2 pixel precision position, but as a first alternative method, mvd when applying MMVD may be rounded to a 1/2 pixel precision position, thereby indirectly rounding mv after applying MMVD.

For example, one possible method is to change the table in Figure 24, which shows the offset value of mvd corresponding to mmvd#distance#idx. Specifically, the following two methods are possible.

The first method is to add the hpelIfIdx check to the table in Figure 24. Figure 25 shows the table with the hpelIfIdx check added.

In the conventional table shown in Figure 24, the offset value of mvd by MMVD was determined only by slice#fpel#mmvd#enabled#flag, which is a slice-level determination flag indicating whether the correction accuracy of MMVD is integer pixel accuracy or not. However, in Figure 25, the table for the case where slice#fpel#mmvd#enabled#flag is "0", i.e., when the correction accuracy of the motion vector by MMVD is allowed up to decimal pixel accuracy, is divided into two according to the value of hpelIfIdx.

If hpelIfIdx is "0", mmvd#distance#idx starts from "1", and if hpelIfIdx is "1", mmvd#distance#idx starts from "2".

The actual MVD offset value of the MMVD in Non-Patent Document 1 is multiplied by the value shown in the table by 4, and then multiplied by the 1/16 pixel precision, which is the calculation precision of the motion vector in Non-Patent Document 1. Therefore, when MmvdDistance shown in FIG. 25 is "1", the motion vector after application of MMVD will have 1/4 pixel precision, and when MmvdDistance is "2", the motion vector after application of MMVD will have 1/2 pixel precision. As a result, it is possible to eliminate the deficiencies in mv after application of the half-pel index and MMVD described above.

(Resolving DMVR deficiencies)
The above-mentioned pairwise average merging and improper setting conditions of the smoothing filter based on the half-pel index and motion vector in MMVD may occur even in a target block for which DMVR is enabled, because even if the reference destination of the motion vector output from the merging unit 241A2 is a 1/2 pixel accuracy position and the half-pel index is "1", the motion vector may be corrected by the DMVR unit 241B2, and the reference destination of the motion vector may shift from the 1/2 pixel accuracy position.

Therefore, in this modified example, we propose a solution in which hpelIfIdx is modified according to mv, as in pairwise average merging, and a solution in which mv is modified according to hpelIfIdx. However, the configuration example is exactly the same as the solution in MMVD described above, i.e., the configuration example shown in Figures 22 and 23, so we will not explain it here.

[Modification example: variation of rounding process for DMVR]
In the following, similar to the variations of rounding process for mv after application of MMVD described above, rounding process for mv after application of DMVR will be described.

As examples of rounding variations for MMVD, we have introduced a technique in which mv after MMVD is applied is directly rounded, or mvd when MMVD is applied is rounded, and mv after MMVD is indirectly rounded. However, the variations for DMVR are basically the same, so we will not explain the overlapping parts. On the other hand, we will explain the differences between the rounding processes of DMVR and MMVD.

Specifically, in DMVR, mvd is calculated independently for the integer part and the decimal part. Therefore, when selecting mvd as the target for rounding, the sum of mvd may be directly rounded as in the case of MMVD described above, but in DMVR, only the decimal part of mvd may be rounded. This is because the cases in which mv deviates from the 1/2 pixel accuracy position are limited to cases in which the decimal part of mvd calculated by DMVR becomes other than 1/2 pixel accuracy.

In addition, in DMVR, mvd is derived by first identifying the position of the motion vector to be corrected by searching as described above, and then calculating it using a parabolic fitting process. By performing a rounding process in this parabolic fitting process, mv after application of DMVR can be rounded to a 1/2 pixel accuracy position.

[Examples of changes: Other examples of changes related to DMVR]
Other modification examples regarding the DMVR will be described below.

DMVR is known to be a relatively heavy process for the image decoding device 200, since it involves searching for a motion vector to be modified. Therefore, in order to facilitate the realization of parallelization of block processing, the motion vector after application of DMVR is not used, for example, as a decision criterion for the deblocking filter, and the motion vector before application of DMVR is used.

The same is true for half-pel indexes. In Non-Patent Document 1, the motion vector used to generate a prediction signal for the target block is the one after DMVR is applied, but for example, the motion vector inherited by surrounding future blocks in the merge process is the one before DMVR is applied.

The solutions shown in Figures 22 and 23 both involve checking the reference position of mv after applying DMVR, and then modifying hpelIfIdx or mv.

However, with this solution, correction processing must be performed after DMVR is applied, which may reduce the feasibility of parallelizing block processing.

Therefore, the following three measures can be taken as alternatives:

The first method is that when DMVR is enabled in the target block, as described above, the mv is modified only in the target block, and the mv inherited by future blocks that reference the target block inherits the mv before DMVR was applied.

The second method is to always set hpelIfIdx to "0" when it is determined that DMVR may be enabled in the target block. The timing at which it can be determined that DMVR may be enabled in the target block is when control data is decoded by CABAC, and since a known method can be used to determine this timing in this modified example, a description thereof will be omitted.

The third method is to disable DMVR for target blocks where hpelIfIdx is "1".

By adopting any of the three alternative methods described above, it is expected that the decrease in feasibility of parallelizing the processing of the blocks mentioned above can be avoided.

[Example of change: Resolving problems when pairwise averaging and MMVD/DMVR are in series]
Hereinafter, with reference to FIG. 26, a modification example in which the motion vectors derived by the pairwise average merging according to this embodiment are corrected by MMVD or DMVR will be described.

Figure 26 shows an example of a modification in which motion vectors derived by pairwise average merging are modified by MMVD or DMVR.

In the above example, we described a solution in which the half-pel index is corrected to "0" when the half-pel index is "1" and the motion vector deviates from the 1/2-pel accuracy position due to pairwise average merging.

However, after correcting this half-pel index to "0", it is possible that the motion vector that has shifted from the 1/2 pixel accuracy position due to MMVD or DMVR may return to the 1/2 pixel accuracy position again.

In such a case, it would be ideal to correct the half-pel index that was once corrected from "1" to "0" back to "1". An example of this method is explained using Figure 26.

Note that the target of MMVD in Non-Patent Document 1 is limited to the merge index "0" and the first motion vector, and in this case, the motion vector derived by pairwise average merging is not subject to NNVD. However, assuming that the valid targets of MMVD will be expanded in the future to include motion vectors with a merge index of "2" or more, an example of such a method will be described below.

First, up to step S26-4, the flow chart is the same as the flow chart for pairwise average merging described above, i.e., the flow chart shown in FIG. 19, but the processing from step S26-4 onwards has been changed to take into account the above case.

Specifically, in step S26-4, if the motion vector deviates from the 1/2 pixel accuracy position, hpelIfIdxAvg is corrected to "0" and hpelIfIdxAvg' is set to "1".

Here, hpelIfIdxAvg' is an index that holds the uncorrected value of the half-pel index for pairwise average merging.

In step S26-4, if the motion vector does not deviate from the 1/2 pixel accuracy position, hpelIfIdx remains at "1" without being modified, and hpelIfIdxAvg' is set to "1".

Next, in step S26-7, a determination is made regarding the specified condition 3. If it is determined that the specified condition 3 is satisfied, the process proceeds to step S26-8, and if it is determined that the specified condition 3 is not satisfied, the process ends.

Here, the predetermined condition 3 may include a determination as to whether the motion vector after application of MMVD or DMVR refers to a half-pixel precision position. Also, a determination as to whether the mvd added to the motion vector by MMVD or DMVR is half-pixel precision may be added.

Next, in step S26-8, the predetermined condition 4 is judged. If it is determined that the predetermined condition 4 is satisfied, the process proceeds to step S26-9. If it is determined that the predetermined condition 4 is not satisfied, the process ends.

Here, the predetermined condition 4 may include determining whether hpelIfIdxAvg and hpelIfIdxAvg' are both "1".

In step S26-9, hpelIfIdx, which was once corrected from "1" to "0" in the previous step, is re-corrected to "1" because MMVD or DMVR has determined that mv refers to a 1/2 pixel accuracy position.

(Scaling of Motion Vectors in Pairwise Average Merging)
Hereinafter, the scaling process of motion vectors in pairwise average merging according to this embodiment will be described with reference to Fig. 27 and Fig. 28. Fig. 27 is a diagram showing an example of derivation of motion vectors by pairwise average merging according to this embodiment.

As shown in FIG. 27, the pairwise average merge is derived using the motion vectors and reference image indexes associated with two different merge indexes already registered in the merge list.

Here, in Non-Patent Document 1, the merge indexes used in the pairwise average merge are specified as 0th and 1st. If only one of the two merge indexes used in the pairwise average merge can be used, the one available merge index is registered in the merge list as a merge index associated with the pairwise average merge index.

In other words, a single available set of motion vector and reference image index is registered as the motion vector and reference image index associated with the pairwise average merge index.

Also, if neither of the two different merge indexes are available, the pairwise average merge is not registered in the merge list.

For example, when there are motion vectors mvL0P ₀ /mvL1P ₀ and mvL0P ₁ /mvL1P ₁ associated with two different pairs of merge indexes used for the derivation, mvL0Avg and mvL1Avg, which are newly derived vectors by pairwise average merging, are derived as follows:

mvL0Avg=(mvL0P ₀ +mvL0P ₁ )/2
mvL1Avg=(mvL1P ₀ +mvL1P ₁ )/2
Here, if either mvL0P ₀ /mvL1P ₀ or mvL0P ₁ /mvL1P ₁ does not exist, the above calculation is performed excluding the non-existent motion vector.

At this time, Non-Patent Document 1 specifies that the reference image indexes _RefIdxL0P0 and _RefIdxL1P0 associated with the merge index _P0 should always be used as the reference image indexes associated with the pairwise average merge index.

However, the reference frames of mvL0P ₁ and mvL1P ₁ averaged in the above formula should correspond to RefIdxL0P ₁ and RefIdxL1P ₁ , which are reference image indexes associated with merge index "1".

Therefore, in this embodiment, _mvL0P1 and _mvL1P1 are replaced with _mvL0P1 ' and mvL0P1 _' scaled by _RefIdxL0P0 , _{RefIdxL1P0 ,} RefIdxL0P1 and _RefIdxL1P1 , and new motion vectors are derived by pair-wise average merging.

Here, the scaling process performed on the above-mentioned motion vectors is explained using Figure 28.

For example, as shown in FIG. 28, there are motion vectors and reference image indices that correspond to two different merge indices used to derive the pairwise average merge.

At this time, this means that the scaled mvL0P ₁ ' and mvL0P ₁ ' are corrected based on the distance between the target frame and the reference frames indicated by RefIdxL0P ₀ , RefIdxL1P ₀ , RefIdxL0P ₁ and RefIdxL1P ₁ which are the reference image indices of the target frame and each merge index.

Specifically, as shown in FIG. 28, when the distances between the target frame and each reference frame are expressed as tL0P0, tL0P1, tL1P0, and tL1P1, mvL0P ₁ ' and mvL0P ₁ ' are calculated as follows.

mvL0P ₁ '=(tL0P1/tL0P0)×mvL0P ₁
mvL1P ₁ '=(tL1P1/tL1P0)×mvL1P ₁
The merge index 1 used for pairwise average merging corresponds to the motion vector and reference image index associated with a temporal merge (i.e., when the reference frame is not the same as the target frame, as in spatial and history merges). When performing temporal merging, the distance is calculated and scaled starting from a reference frame that includes a reference block that is at a common position between the target block and a different frame.

As an effect of introducing the above-mentioned scaling process, when the pairwise average merge is calculated from motion vectors associated with two different merge indexes, one of the motion vectors is scaled to one reference image index, making it possible to average two pairs of motion vectors corresponding to reference frames indicated by a common reference image index. This improves the accuracy of the motion vectors, and as a result, is expected to improve coding performance.

[Example of change: Changing the scaling target]
In the above example, the motion vectors associated with the merge index _P1 are scaled, but the motion vectors that refer to reference frames farther from the target frame may be scaled. This allows the reference frame for pairwise average merging to be closer to the target frame, which is expected to improve prediction accuracy.

(Half-pel index control by block size)
Hereinafter, the control of the half-pel index based on the target block size according to this embodiment will be described.

In the above example, it was explained that when the half-pel index is enabled and the motion vector is in a 1/2 pixel position, a smoothing filter is applied.

On the other hand, if the block size of the target block is small, it is better not to apply a smoothing filter to the target block.

This is because it is generally known that there is a correlation between block size and image characteristics; small block sizes tend to have edges and complex patterns, while large block sizes tend to be flat and simple.

In addition, since the reference block of the target block is assumed to have the same characteristics, it is possible to take these characteristics into account and prohibit the application of a smoothing filter when the block size of the target block is small, i.e., to always set the value of the half-pel index to "0".

For example, the block size threshold for always setting the half-pel index to 0 may be set to 32 pixels or less.

The expected effects are as follows:

First, in a certain block size where the half-pel index is always set to "0", the application of a smoothing filter is prohibited, making RDO processing unnecessary on the image encoding device 100 side. This is expected to result in an increase in the speed of the image encoding device 100.

Secondly, when the target block shown in Figure 8 is both AMVP and AMVR, the bit length of the flag used to determine the half-pel index, specifically, amvr_precision_idx, can be shortened.

In Figure 8, since there are variations in which the target block indicates a 1/2 pixel precision position, the bit length of amvr_precision_idx needs to be 3 bits, but this can be shortened to 2 bits for the above-mentioned specified block size. As a result, the amount of code that needs to be transmitted from the image encoding device 100 to the image decoding device 200 is reduced, which is expected to improve encoding performance.

According to the present invention, it is possible to control the reference pixel precision position of a motion vector derived by pairwise average merging, MMVD, or DMVR according to a half-pel index, or to control the half-pel index according to a derived motion vector, thereby providing an image decoding device, an image decoding method, and a program that can eliminate deficiencies in the effective conditions of a smoothing filter for a target block and further prevent the propagation of deficiencies in the effective conditions of a smoothing filter to future blocks.

The image encoding device 100 and image decoding device 200 described above may be realized as a program that causes a computer to execute each function (each process).

In each of the above-described embodiments, the present invention has been described by taking as an example the application to the image encoding device 100 and the image decoding device 200, but the present invention is not limited to this and can be similarly applied to an image encoding system and an image decoding system having the functions of the image encoding device 100 and the image decoding device 200.

10... Image processing system 100... Image encoding device 111, 241... Inter prediction unit 111A... mv derivation unit 111A1, 241A1... AMVP unit 111A2, 241A2... Merge unit 111B... AMVR unit 111C, 241B... mv refinement unit 111D, 241C... Prediction signal generation unit 111D1, 241C1... Filter determination unit 111D2, 241C2... Filter Data application unit 112, 242... Intra prediction unit 121... Subtractor 122, 230... Adder 131... Transformation and quantization unit 132, 220... Inverse transformation and inverse quantization unit 140... Encoding unit 150, 250... In-loop filter processing unit 160, 260... Frame buffer 200... Image decoding device 210... Decoding unit 241A... mv decoding unit 241B1... MMVD unit 241B2... DMVR unit

Claims (7)

An image decoding device,
a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal;
a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the motion vector and the half-pel index;
a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter,
the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index;
The target block is a merge.
The image decoding device, wherein the merging unit is configured to control a value of the half-pel index to be derived depending on a reference position of the motion vector derived by pair-wise average merging. An image decoding device,
a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal;
a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the motion vector and the half-pel index;
a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter,
the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index;
The target block is a merge.
The image decoding device, wherein the merging unit is configured to control a reference position of the motion vector to be derived in accordance with the half-pel index derived by pair-wise average merging. An image decoding device,
a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal;
a motion vector refinement unit configured to refine the motion vector by MMVD (Merge Motion Vector Difference);
a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the refined motion vector and the half-pel index;
a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter,
the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index;
The target block is a merge.
The image decoding device, characterized in that, when MMVD is applied, the motion vector refinement unit is configured to control a reference position of a motion vector after application of MMVD in accordance with the half-pel index. An image decoding device,
a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal;
a motion vector refinement unit configured to refine the motion vector by a decoder-side motion vector refinement (DMVR);
a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the refined motion vector and the half-pel index;
a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter,
the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index;
The target block is a merge.
An image decoding device characterized in that, when DMVR is applied, the motion vector refinement unit is configured to control the half-pel index depending on the reference position of the motion vector after application of DMVR. An image decoding device,
a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal;
a motion vector refinement unit configured to refine the motion vector by a decoder-side motion vector refinement (DMVR);
a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the refined motion vector and the half-pel index;
a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter,
the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index;
The target block is a merge.
An image decoding device characterized in that, when DMVR is applied, the motion vector refinement unit is configured to control the reference position of the motion vector after application of DMVR in accordance with the half-pel index. A step A of decoding a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal from the merge index;
a step B of determining whether an interpolation filter is used and a type of the interpolation filter based on the motion vector and the half-pel index;
and C. generating a motion compensation prediction pixel signal using the interpolation filter,
In the step A, a merge list is generated by a predetermined merge list construction method, and the motion vector and the half-pel index are decoded from the merge list and the merge index;
The target block is a merge.
An image decoding method, comprising the steps of: controlling a value of a half-pel index to be derived in the step A according to a reference position of the motion vector derived by pair-wise average merging. A program for causing a computer to function as an image decoding device,
The image decoding device comprises:
a merge unit configured to decode, from the merge index, a half-pel index indicating a type of an interpolation filter for generating a motion vector and a motion compensation prediction pixel signal;
a filter determination unit configured to determine whether an interpolation filter is used and a type of the interpolation filter based on the motion vector and the half-pel index;
a filter application unit configured to generate a motion compensation prediction pixel signal using the interpolation filter,
the merging unit is configured to generate a merge list by a predetermined merge list construction method, and decode the motion vector and the half-pel index from the merge list and the merge index;
The target block is a merge.
The program, wherein the merging unit is configured to control a value of a half-pel index to be derived depending on a reference position of the motion vector derived by pair-wise average merging.

Images