JP7200074B2 - Image decoding device, image decoding method and program - Google Patents

Description

The present invention relates to an image decoding device, an image decoding method and a program.

Conventionally, a technique called PROF (Prediction Refinement with Optical Flow) and a technique called deblocking filter are known (see, for example, Non-Patent Document 1).

PROF is a technology that generates a prediction signal with high accuracy using a motion model that corresponds to translation and rotation in units of 4x4 pixel blocks. Improvement can be expected.

The deblocking filter is a low-pass filter applied to block boundaries, and is expected to have the effect of suppressing block noise and improving subjective image quality.

Versatile Video Coding (Draft 6), JVET-N1001

In the prior art, a deblocking filter may also be applied to block boundaries where PROF processing is applied. However, there is a problem that the subjective image quality of a block whose subjective image quality has been sufficiently improved by the PROF process may deteriorate due to the application of the deblocking filter.

Therefore, the present invention has been made in view of the above problems, and by not applying a deblocking filter to sub-block boundaries to which PROF processing has been applied, an image that can prevent subjective image quality from deteriorating. An object is to provide a decoding device, an image decoding method, and a program.

A first feature of the present invention is an image decoding device configured to determine the propriety of PROF processing for each block, and to perform the PROF processing for each sub-block obtained by dividing the block when the PROF processing is applied. and an affine prediction signal generation unit configured to determine suitability of a deblocking filter for each block boundary, and determine not to apply a deblocking filter to a sub-block boundary of a block to which the PROF processing is applied. and an in-loop filtering unit.

A second feature of the present invention is an image decoding method, comprising a step of determining whether or not a PROF process is appropriate for each block, and performing the PROF process on each sub-block obtained by dividing the block when the PROF process is applied; determining whether the deblocking filter is appropriate for each block boundary, and determining that the deblocking filter is not applied to the sub-block boundary of the block to which the PROF processing has been applied.

A third feature of the present invention is a program for use in an image decoding apparatus, wherein a computer determines whether or not a PROF process is appropriate for each block, and when the PROF process is applied, each sub-block obtained by dividing the block is subjected to the PROF process. and determining whether a deblocking filter is appropriate for each block boundary, and determining that the deblocking filter is not applied to a sub-block boundary of a block to which the PROF processing has been applied. and

According to the present invention, it is possible to provide an image decoding device, an image decoding method, and a program that can prevent deterioration of subjective image quality by not applying a deblocking filter to sub-block boundaries to which PROF processing has been applied. can be done.

It is a figure showing an example of composition of image processing system 10 concerning one embodiment. 1 is a diagram showing an example of functional blocks of an image encoding device 100 according to an embodiment; FIG. FIG. 3 is a diagram showing an example of the structure of encoded data (bitstream) output from the encoding unit 140 of the image encoding device 100 according to one embodiment. It is a figure which shows an example of the control data contained in SPS141 shown in FIG. It is an example of control data included in the slice header 143A shown in FIG. 3 is a diagram showing an example of functional blocks of an inter prediction unit 111 of the image encoding device 100 according to one embodiment; FIG. 3 is a flowchart showing an example of a processing procedure of a refinement unit 111C of an inter prediction unit 111 of the image encoding device 100 according to one embodiment; 4 is a flowchart showing an example of a processing procedure of a prediction signal generation unit 111D of an inter prediction unit 111 of the image encoding device 100 according to one embodiment; 11 is a flow chart showing an example of a processing procedure of an affine motion vector calculation unit 111E of the inter prediction unit 111 of the image encoding device 100 according to one embodiment. FIG. 4 is a diagram for explaining an example of a processing procedure of an affine motion vector calculation unit 111E of an inter prediction unit 111 of an image encoding device 100 according to an embodiment; 11 is a flow chart showing an example of a processing procedure of an affine prediction signal generation unit 111F of an inter prediction unit 111 of an image encoding device 100 according to an embodiment; 4 is a diagram showing an example of functional blocks of an in-loop filtering unit 150 of the image encoding device 100 according to one embodiment; FIG. FIG. 4 is a diagram for explaining a processing procedure of a boundary strength determination unit 153 (153A/153B) of an in-loop filter processing unit 150 of the image encoding device 100 according to one embodiment; 2 is a diagram showing an example of functional blocks of an image decoding device 200 according to one embodiment; FIG. 3 is a diagram showing an example of functional blocks of an inter prediction unit 241 of the image decoding device 200 according to one embodiment; FIG. 3 is a diagram showing an example of functional blocks of an in-loop filtering unit 250 of the image decoding device 200 according to one embodiment; FIG. FIG. 11 is a diagram for explaining Modification 1; FIG. 11 is a diagram for explaining Modification 1; FIG. 11 is a diagram for explaining Modification 2; FIG. 11 is a diagram for explaining Modification 2;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the constituent elements in the following embodiments can be appropriately replaced with existing constituent elements and the like, and various variations including combinations with other existing constituent elements are possible. Therefore, the following description of the embodiments is not intended to limit the scope of the invention described in the claims.
(First embodiment)
An image processing system 10 according to a first embodiment of the present invention will be described below with reference to FIGS. 1 to 16. FIG. FIG. 1 is a diagram showing an image processing system 10 according to an embodiment according to this embodiment.

As shown in FIG. 1 , the image processing system 10 has an image encoding device 100 and an image decoding device 200 .

The image encoding device 100 is configured to generate encoded data by encoding an input image signal. The image decoding device 200 is configured to generate an output image signal by decoding encoded data.

Here, such encoded data may be transmitted from the image encoding device 100 to the image decoding device 200 via a transmission line. Also, the encoded data may be stored in a storage medium and then provided from the image encoding device 100 to the image decoding device 200 .
(Image encoding device 100)
The image coding apparatus 100 according to this embodiment will be described below with reference to FIG. FIG. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to this embodiment.

As shown in FIG. 2 , the image coding apparatus 100 includes an inter prediction unit 111, an intra prediction unit 112, a subtractor 121, an adder 122, a transform/quantization unit 131, an inverse transform/inverse quantization It has a section 132 , an encoding section 140 , an in-loop filtering section 150 and a frame buffer 160 .

The inter prediction unit 111 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

Specifically, the inter prediction unit 111 identifies a reference block included in the reference frame by comparing a frame to be encoded (hereinafter referred to as a target frame) and a reference frame stored in the frame buffer 160, and identifies a reference block included in the reference frame. It is configured to determine a motion vector for the reference block.

Also, the inter prediction unit 111 is configured to generate, for each prediction block, a prediction signal included in the prediction block based on the reference block and motion vector. The inter prediction section 111 is configured to output the prediction signal to the subtractor 121 and the adder 122 . Here, the reference frame is a frame different from the target frame.

The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (intra-frame prediction).

Specifically, the intra prediction unit 112 is configured to identify a reference block included in the target frame and generate a prediction signal for each prediction block based on the identified reference block. Also, the intra prediction unit 112 is configured to output the prediction signal to the subtractor 121 and the adder 122 .

Here, the reference block is a block that is referred to for a prediction target block (hereinafter referred to as target block). For example, the reference block is a block adjacent to the target block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the transform/quantization section 131 . Here, the subtractor 121 is configured to generate a prediction residual signal that is a difference between a prediction signal generated by intra prediction or inter prediction and an input image signal.

The adder 122 adds the prediction signal to the prediction residual signal output from the inverse transform/inverse quantization unit 132 to generate a pre-filtering decoded signal. It is configured to output to the loop filter processing unit 150 .

Here, the unfiltered decoded signal constitutes a reference block used in intra prediction section 112 .

The transform/quantization unit 131 is configured to perform transform processing on the prediction residual signal and to obtain coefficient level values. Further, the transform/quantization unit 131 may be configured to quantize the coefficient level values.

Here, transform processing is processing for transforming a prediction residual signal into a frequency component signal. In such transformation processing, a basis pattern (transformation matrix) corresponding to discrete cosine transform (DCT) may be used, and a basis pattern (transformation matrix) corresponding to discrete sine transform (DST) may be used. may be used.

The inverse transform/inverse quantization unit 132 is configured to perform inverse transform processing on the coefficient level values output from the transform/quantization unit 131 . Here, the inverse transform/inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level values prior to the inverse transform processing.

Here, the inverse transform processing and inverse quantization are performed in the reverse order of the transform processing and quantization performed by the transform/quantization unit 131 .

The encoding unit 140 is configured to encode the coefficient level values output from the transform/quantization unit 131 and output encoded data.

Here, for example, the coding is entropy coding that assigns codes of different lengths based on the probability of occurrence of coefficient level values.

In addition to the coefficient level values, the encoding unit 140 is also configured to encode control data used in the decoding process.

Here, the control data may include size data such as a coding block (CU: Coding Unit) size, a prediction block (PU: Prediction Unit) size, a transform block (TU: Transform Unit) size, and the like.

The in-loop filtering section 150 is configured to perform filtering on the unfiltered decoded signal output from the adder 122 and output the filtered decoded signal to the frame buffer 160 .

Here, for example, the filter processing is deblocking filter processing that reduces distortion that occurs at boundaries of blocks (encoding blocks, prediction blocks, or transform blocks).

The frame buffer 160 is configured to accumulate reference frames used by the inter prediction section 111 .

Here, the filtered decoded signal constitutes a reference frame used in inter prediction section 111 .
(Encoder 140)
The control data encoded by the encoding unit 140 will be described below with reference to FIGS. 3 to 5. FIG. FIG. 3 is a configuration example of encoded data (bitstream) output from the encoding unit 140 .

First, such a bitstream may include an SPS (Sequence Parameter Set) 141 at the beginning. The SPS 141 is a set of control data for each sequence (set of pictures). A specific example will be described later. Each SPS 141 includes at least id information for individual identification when multiple SPSs 141 exist.

Such a bitstream may include a PPS (Picture Parameter Set) 142 next to the SPS 141 . The PPS 142 is a collection of control data for each picture (set of slices). Each PPS 142 includes at least id information for individual identification when multiple SPSs 141 exist. Also, each PPS 142 includes at least SPS id information for designating the SPS 141 corresponding to each PPS 142 .

Such a bitstream may include a PPS 142 followed by a slice header 143A. The slice header 143A is a set of control data for each slice. A specific example will be described later. Each slice header 143A includes at least PPS id information for designating the PPS 142 corresponding to each slice header 143A.

Such a bitstream may include slice data 144A next to slice header 143A. The slice data 144A may include the above-described coefficient level values, size data, and the like.

As described above, slice headers 143A/143B, PPS 142, and SPS 141 correspond to slice data 144A/144B one by one. As described above, in the slice header 143A/143B, which PPS 142 is to be referred to is specified by PPS id information, and further, in that PPS 142, which SPS 141 is to be referred to is specified by SPS id information. A common SPS 141 and PPS 142 can be used for the slice data 144A/144B.

In other words, the SPS 141 and PPS 142 do not necessarily need to be transmitted for each slice data 144A/144B. For example, as shown in FIG. 3, it is also possible to adopt a stream configuration in which the SPS 141 and PPS 142 are not encoded immediately before the slice header 143B and slice data 144B.

Note that the configuration in FIG. 3 is merely an example. As long as each slice data 144A/144B corresponds to control data specified by the slice headers 144A/144B, PPS 142, and SPS 141, elements other than those described above may be added as constituent elements of the stream. Also, in the same way, it may be shaped into a configuration different from that in FIG. 3 at the time of transmission.

FIG. 4 is a diagram showing an example of control data contained in the SPS 141. As shown in FIG.

The SPS 141 includes at least id information (sps_seq_parameter_set_id) for identifying each SPS 141 as described above.

The SPS 141 may include an “sps_bdof_enabled_flag” that is a flag for controlling availability (valid/invalid) of BDOF (Bi-Directional Optical Flow) processing, which will be described later. When the flag value is "0", it means that the BDOF processing is not used in the slice corresponding to the SPS141. On the other hand, when the value of this flag is "1", it means that BDOF processing can be used within the slice corresponding to this SPS141. Whether or not the BDOF process is actually used in each block in the slice is determined for each block in the process described later.

That is, “sps_bdof_enabled_flag” is a flag (fourth flag) indicating whether or not the BDOF processing included in the SPS 141 can be used for each sequence.

The SPS 141 may include an “sps_dmvr_enabled_flag” that is a flag for controlling availability (enable/disable) of DMVR (Decoder-side Motion Vector Refinement) processing, which will be described later. When the value of this flag is '0', it means that the DMVR processing is not used in the slice corresponding to this SPS141. On the other hand, when the value of this flag is "1", it means that DMVR processing can be used within the slice corresponding to this SPS141. Whether or not DMVR processing is actually used in each block in the slice is determined for each block in the processing described later.

That is, “sps_dmvr_enabled_flag” is a flag (third flag) indicating whether or not the DMVR process included in the SPS 141 can be used for each sequence.

The SPS 141 may include an "sps_affine_enabled_flag" which is a flag for controlling availability (validity/invalidity) of affine motion compensation, which will be described later. When the flag value is '0', it means that affine motion compensation is not used in the slice corresponding to SPS141. On the other hand, when the value of this flag is "1", it means that affine motion compensation can be used in the slice corresponding to this SPS141. Whether or not affine motion compensation is actually used in each block in a slice is determined for each block in a process described later.

That is, “sps_affine_prof_enabled_flag” is a flag (fifth flag) indicating whether or not the PROF process included in the SPS 141 can be used for each sequence.

When 'sps_affine_enabled_flag' is '1', the SPS 141 may additionally include a flag 'sps_affine_type_flag'. When the flag value is "0", it means that the number of parameters for affine motion compensation, which will be described later, is always "4" in the slice corresponding to SPS141. On the other hand, when the value of the flag is "1", the number of parameters can be selected for each block from "4" or "6" when performing affine motion compensation within the slice corresponding to the SPS141. means that

When the value of 'sps_affine_enabled_flag' is '1', the SPS 141 may additionally include a flag 'sps_affine_prof_enabled_flag'. When the value of the flag is "0", it means that the later-described PROF (Prediction Refinement with Optical Flow) processing is not used in the slice corresponding to the SPS141. On the other hand, when the flag value is "1", it means that the PROF process can be used in the slice corresponding to the SPS141. Whether or not the PROF process is actually used in each block in the slice is determined for each block in the process described later.

When the value of at least one of 'sps_bdof_enabled_flag', 'sps_dmvr_enabled_flag' and 'sps_affine_prof_enabled_flag' is '1', the SPS 141 may additionally include a flag 'sps_bdof_dmvr_prof_slice_present_flag'. When the flag is "1", it means that the slice header 143A/143B corresponding to the SPS 141 includes a flag "slice_disable_bdof_dmvr_prof_flag", which will be described later. When the flag value is '0', it means that the slice header 143A/143B corresponding to the SPS 141 does not include 'slice_disable_bdof_dmvr_prof_flag'.

That is, "sps_bdof_dmvr_prof_slice_present_flag" is a flag (second flag) indicating whether or not "slice_disable_bdof_dmvr_prof_flag" is included in the slice header 143A/143B. Note that if such a flag does not exist, it can be implicitly assumed that the value of such flag is "0".

FIG. 5 is a diagram showing an example of control data included in the slice headers 143A/143B.

As described above, the slice header 143A/143B includes at least "slice_pic_parameter_set_id" which is PPS id information for designating the PPS 142 corresponding to the slice. As described above, the SPS 141 referenced in the PPS designated by the PPS id information becomes the SPS 141 corresponding to the slice header 143A/143B.

When 'sps_bdof_dmvr_prof_slice_present_flag' is included in the SPS 141 corresponding to the slice header 143A/143B and the value of the flag is '1', the slice header 143A/143B includes 'slice_disable_bdof_dmvr_prof_flag'. good too. When the value of this flag is "1", each block included in this slice may be controlled not to use BDOF processing, DMVR processing, and PROF processing, as will be described later. When the value of this flag is "0", each block included in this slice may be controlled so that BDOF processing, DMVR processing, and PROF processing can be used.

That is, "slice_disable_bdof_dmvr_prof_flag" is a flag (first flag) that collectively controls whether or not the DMVR processing, BDOF processing, and PROF processing included in the slice header 143A/143B can be used.

In other words, "slice_disable_bdof_dmvr_prof_flag" is a flag for controlling whether the PROF process included in the slice header 143A/143B is enabled or disabled, and is a flag for controlling whether the DMVR process included in the slice header 143A/143B is enabled or disabled. 143A/143B.

The flag values described above are merely examples. When the meanings given to the flag values (“0” and “1”) are reversed, equivalent processing can be realized by reversing the corresponding processing accordingly.

Further, as described above, by providing a flag for controlling not to use BDOF processing, DMVR processing, and PROF processing in units of slices, the image coding apparatus 100 explicitly controls not to use these functions. Therefore, the processing load and power consumption of the image encoding device 100 and the corresponding image decoding device 200 can be reduced by not using such functions.

(Inter prediction unit 111)
The inter prediction unit 111 of the image coding apparatus 100 according to this embodiment will be described below with reference to FIG. FIG. 6 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding device 100 according to this embodiment.

As illustrated in FIG. 6 , the inter prediction unit 111 includes a motion vector search unit 111A, a motion vector encoding unit 111B, a refinement unit 111C, a prediction signal generation unit 111D, an affine motion vector calculation unit 111E, and an affine motion vector calculation unit 111E. and a prediction signal generator 111F.

The inter prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on motion vectors.

The motion vector search unit 111A is configured to identify a reference block included in the reference frame by comparing the target frame and the reference frame, and search for a motion vector for the identified reference block.

Also, the search described above is performed for a plurality of reference frame candidates, and the reference frame and motion vector used for prediction in the prediction block are determined. A maximum of two reference frames and motion vectors can be used for one block. The case of using only one set of reference frames and motion vectors for one block is called uni-prediction, and the case of using two sets of reference frames and motion vectors is called bi-prediction. Hereinafter, the first set will be called L0, and the second set will be called L1.

Furthermore, the motion vector search unit 111A is configured to determine the encoding method of the reference frame and motion vector. Coding methods include a merge mode, affine motion compensation, etc., which will be described later, in addition to the normal method of transmitting reference frame and motion vector information.

It should be noted that the method of searching for motion vectors, the method of determining reference frames, and the method of determining encoding methods of reference frames and motion vectors can employ known methods, and thus the details thereof will be omitted.

The motion vector encoding unit 111B is configured to encode the reference frame and motion vector information determined by the motion vector search unit 111A using the encoding method similarly determined by the motion vector search unit 111A. Since a known method can be adopted for a specific method of encoding the reference frame and motion vector information, the details thereof will be omitted.

When the encoding method for the block is the merge mode, the image encoding device 100 first creates a merge list for the block. Here, the merge list is a list listing a plurality of combinations of reference frames and motion vectors.

Each combination is assigned an index, and the image encoding device 100 encodes only the index and transmits it to the image decoding device 200 instead of encoding the reference frame and motion vector information individually. The image encoding device 100 side and the image decoding device 200 side share the merge list creation method, so that the image decoding device 200 side decodes the reference frame and motion vector information only from the information related to the index. can do.

As for the method of creating the merge list, since it is possible to adopt a known method, the details thereof will be omitted.

Affine motion compensation is a method of transmitting a small number of parameters for each block and deriving motion vectors for each sub-block into which the block is divided based on the parameters and a predetermined model. The details of the technique for deriving such motion vectors will be described later.

In the case of affine motion compensation, the motion vector encoding unit 111B is configured to encode control point motion vector information and reference frame information, which will be described later. Control point motion vectors carry two or three motion vector information per block, as described below.

Also, if such a block is a block that performs bi-prediction, two or three pieces of control point motion vector information are transmitted for each of L0 and L1. As for a specific encoding method, a known method can be adopted, so details thereof will be omitted.

If affine motion compensation is used in the block, the process proceeds to the processing of the affine motion vector calculation unit 111E shown in FIG. 6; otherwise, the processing proceeds to the refinement unit 111C.

The refinement unit 111C is configured to perform a refinement process (for example, DMVR) for correcting the motion vector encoded by the motion vector encoder 111B.

Specifically, the refining unit 111C sets a search range based on the reference position specified by the motion vector encoded by the motion vector encoding unit 111B, and selects a modification having the lowest predetermined cost from the search range. A refinement process is configured to identify the reference position and modify the motion vector based on the modified reference position.

FIG. 7 is a flowchart illustrating an example of a processing procedure of the refining unit 111C.

As shown in FIG. 7, in step S71, the refinement unit 111C determines whether a predetermined condition for applying the refinement process is satisfied. If all such predetermined conditions are satisfied, the process proceeds to step S72. On the other hand, if any one of these predetermined conditions is not satisfied, the processing procedure proceeds to step S75 and the refinement processing ends.

Here, the predetermined condition includes a condition that the block is a block for bi-prediction.

Further, the predetermined condition is that one of the reference frame on the L0 side and the reference frame on the L1 side is a temporally past frame with respect to the frame, and the other reference frame is a temporally future frame with respect to the frame. It may be a frame, and may include a condition that the temporal distance from the frame (for example, the absolute value of the difference in POC (Picture Order Count)) is equal between the L0 side and the L1 side.

The predetermined condition may also include a condition that motion vectors are encoded in merge mode.

Also, the predetermined condition is that the value of "sps_dmvr_enabled_flag" in the SPS 141 of the sequence to which the block belongs indicates that DMVR can be used in the sequence (for example, if the value of the flag is "1"). It may include the condition that

Furthermore, the predetermined condition is that the value of "slice_disable_bdof_dmvr_prof_flag" in the slice header 143A/143B of the slice to which the block belongs indicates that DMVR can be used in the slice (for example, the value of the flag is " 0”).

In step S72, the refining unit 111C generates a search image based on the motion vector encoded by the motion vector encoding unit 111B and the reference frame information.

Here, if the motion vector points to a non-integer pixel position, the refining unit 111C applies a filter to the pixel values of the reference frame and interpolates the pixels at the non-integer pixel position. At this time, the refining unit 111C can reduce the amount of calculation by using an interpolation filter with a smaller number of taps than an interpolation filter used in the prediction signal generation unit 111D, which will be described later. For example, the refinement unit 111C can interpolate pixel values at non-integer pixel positions by bilinear interpolation.

In step S73, the refining unit 111C uses the search image generated in step S72 to perform a search with integer pixel accuracy. Here, integer pixel precision means searching only for points that are at intervals of integer pixels based on the motion vector encoded by the motion vector encoding unit 111B.
The refining unit 111C determines the motion vector after correction at the integer-pixel interval position through the search in step S72. A known method can be used as a search method.

For example, the refining unit 111C can also perform a search by a method of searching only for points where the differential motion vectors on the L0 side and the L1 side are combinations in which only the signs are inverted. During the search, the refining unit 111C may, for example, calculate the search cost at each search point and determine to correct the motion vector to the search point with the lowest search cost.

Here, the refining unit 111C may calculate so as to reduce the search cost only at the search point corresponding to the motion vector encoded by the motion vector encoding unit 111B. For example, for this search point, the refining unit 111C sets a value obtained by reducing the search cost calculated by the same method as for other search points to 3/4 as the final search cost for this search point. good too.

As a result of the search in step S73, the motion vector may have the same value as before the search.

In step S74, the refining unit 111C performs motion vector search with non-integer pixel precision using the motion vector after modification with integer pixel precision determined in step S73 as an initial value. A known method can be used as a motion vector search method.

The refining unit 111C can also use the result of step S73 as an input and determine a vector with non-integer pixel precision using a parametric model such as parabolic fitting without actually performing a search.

In step S74, the refining unit 111C determines the motion vector after modification with non-integer pixel precision, and then proceeds to step S45 to end the refining process. Here, for the sake of convenience, the expression "corrected motion vector with non-integer pixel precision" is used, but the result of the search in step S74 may result in the same value as the motion vector with integer pixel precision obtained in step S73. There is also

The refining unit 111C may divide a block larger than a predetermined threshold into smaller sub-blocks and perform refinement processing on each sub-block. For example, the refining unit 111C sets the execution unit of the refining process to 16×16 pixels, and if the horizontal or vertical size of the block is larger than 16 pixels, it is divided into 16 pixels or less. can do. At this time, the motion vectors of all sub-blocks in the same block encoded by the motion vector encoding unit 111B are used as the reference motion vectors for the refinement process.

When performing processing for each subblock, the refining unit 111C may perform all the procedures in FIG. 7 for each subblock. Also, the refining unit 111C may process only a part of the processing in FIG. 7 for each sub-block. Specifically, the refining unit 111C may process steps S71 and S72 of FIG. 7 for each block, and may process steps S73 and S74 for each sub-block.

The predicted signal generator 111D is configured to generate a predicted signal based on the modified motion vector output from the refiner 111C.

Here, as will be described later, the prediction signal generation unit 111D determines whether or not to perform BDOF processing for each block based on information (for example, search cost) calculated in the course of the refinement processing described above. is configured as

Specifically, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector encoded by the motion vector encoding unit 111B when the motion vector is not corrected. On the other hand, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector corrected by the refinement unit 111C when the motion vector is corrected.

FIG. 8 is a flow chart showing an example of a processing procedure of the prediction signal generator 111D. Here, when refinement processing is performed in units of subblocks by the refinement unit 111C, processing in the prediction signal generation unit 111D is also performed in units of subblocks. In that case, the word "block" in the following description can be appropriately read as "sub-block".

As shown in FIG. 8, in step S81, the prediction signal generator 111D generates a prediction signal.

Specifically, the prediction signal generation unit 111D receives the motion vector encoded by the motion vector encoding unit 111B or the motion vector encoded by the refinement unit 111C, and the position indicated by the motion vector is a non-integer. For pixel locations, a filter is applied to the pixel values of the reference frame to interpolate pixels at non-integer pixel locations. Here, as a specific filter, a horizontal/vertical separable filter with a maximum of 8 taps disclosed in Non-Patent Document 1 can be applied.

When the block is a block to be bi-predicted, the prediction signal generation unit 111D generates a prediction signal based on the first (hereinafter referred to as L0) reference frame and motion vector and the second (hereinafter referred to as L1) generates both the reference frame and the motion vector prediction signal.

In step S82, the prediction signal generation unit 111D checks whether or not the conditions for applying the BDOF process, which will be described later, are satisfied for each block.

Such application conditions include at least the condition that the block in question is a block for bi-prediction.

The application condition is that the search cost of the search point with the minimum search cost in the integer pixel position search in step S73 of the refining unit 111C is greater than or equal to a predetermined threshold. It may include the condition that

Also, the predetermined condition is that the value of "sps_bdof_enabled_flag" in the SPS 141 of the sequence to which the block belongs indicates that BDOF can be used in the sequence (for example, if the value of the flag is "1"). It may include the condition that

Also, the applicable condition is that the value of "slice_disable_bdof_dmvr_prof_flag" in the slice header 143A/143B of the slice to which the block belongs indicates that BDOF processing can be used in the slice (for example, the value of the flag is be “0”).

In addition, the application condition is that one of the reference frame on the L0 side and the reference frame on the L1 side is a temporally past frame with respect to the current frame, and the other reference frame is a temporally future frame with respect to the current frame. and may include a condition that the ratio between the L0 side and the L1 side of the temporal distance (for example, the absolute value of the POC difference) from the frame is a predetermined ratio.

Here, the method of determining the ratio may be, for example, proportional to the reciprocal of the weighting factor when calculating using the gradient between the L0 side and the L1 side in the later-described BDOF process and the luminance value.

For example, when the weighting factor on the L0 side is set to "2/3" and the weighting factor on the L1 side is set to "1/3", the distance between L0 and the relevant frame is set to "1" as the ratio of the distance to the reference frame. Then, the BDOF process may be applied only when the distance between L1 and the frame is "2".

Similarly, for example, when the weighting factor on the L0 side is set to "1/2" and the weighting factor on the L1 side is set to "1/2", the distance between L0 and the frame is set to "1/2" as the ratio of the distance to the reference frame. 1", only when the distance between L1 and the frame is "1", that is, when the distances between the frame and L0 and L1 are equal. BDOF processing may be applied.

That is, the prediction signal generation unit 111D performs calculation using pixel values of two reference frames (L0/L1) or values calculated from pixel values of two reference frames (L0/L1) as the applicable condition. A condition regarding the temporal distance between the two reference frames (L0/L1) and the relevant frames may be provided so as to be proportional to the reciprocal of the weighting factor of the case.

In addition, the prediction signal generation unit 111D, as an applicable condition, sets the temporal distance between each of the two reference frames (L0/L1) and the corresponding frame as a condition regarding the temporal distance when the weighting factors are equal. are equal to each other.

Here, as an example, a case where the weighting factor is "1/2" has been described. The BDOF process may be applied only when the distances between L0 and L1 are equal.

If it is determined that the application condition is satisfied, the process proceeds to step S83, and if the application condition is not satisfied, the process proceeds to step S84.

In step S83, the prediction signal generator 111D performs BDOF processing to generate a prediction signal. Since a known method can be used for the BDOF processing itself, only the outline will be described and the detailed description will be omitted.

First, the prediction signal generator 111D calculates gradients of luminance values in the vertical and horizontal directions for each pixel in the reference block of the block. A specific calculation method can be calculated by the method described in Non-Patent Document 1, for example. Here, the horizontal gradient of L0 is called "gradientHL0", the vertical gradient of L0 is called "gradientVL0", the horizontal gradient of L1 is called "gradientHL1", and the vertical gradient of L1 is called " gradientVL1".

Second, the prediction signal generator 111D calculates the correction MV (vx, vy) of the block. A specific calculation method can be calculated by the method described in Non-Patent Document 1, for example. Here, the sum of the gradients (tempH, tempV) and the luminance value difference (diff) between the reference blocks of L0 and L1 are calculated as follows, and as described in Non-Patent Document 1, the above ( vx, vy) may be used.

diff=predSampleL0×weightL0−predSampleL1×weightL1
tempH=gradientHL0×weightL0+gradientHL1×weightL1
tempV=gradientVL0×weightL0+gradientVL1×weightL1
Here, "predSampleL0" and "predSampleL1" are the luminance values of the reference blocks of L0 and L1, respectively, and "weightL0" and "weightL1" are weight coefficients for calculating the gradient and luminance value difference described above, respectively. be.

Here, an example of using a common weight for the three parameters "diff", "tempH", and "tempV" has been described. Similar weights can be set.

In addition, as described above, an application condition regarding the distance from the reference frame for BDOF processing may be set in proportion to the inverse of the weight. In other words, as described above, weights for addition, subtraction, multiplication, and division may be set so as to be proportional to the reciprocal of the ratio of the distance between the current frame and the reference frames of L0 and L1.

By setting in this way, when it is assumed that the gradient, luminance value, etc., between L0 and L1 change linearly, an appropriate weighting factor can be set for the temporal position of the frame. Also, by setting the application condition of the BDOF process corresponding to the weighting factor, it is possible to ensure that the appropriate weighting factor is always used.

Third, the prediction signal generation unit 111D uses the gradient and the correction MV obtained above to calculate the brightness value correction amount “bdofOffset” for each pixel, for example, as follows.

bdofOffset = vx x (weightL0 x gradientHL0 - weightL1 x gradientHL1) + vy x (weightL0 x gradientVL0 - weightL1 x gradientVL1)
Fourthly, the prediction signal generation unit 111D can calculate the prediction value of the corrected luminance value of each pixel using "bdofOffset" as described in Non-Patent Document 1, for example.

That is, in the BDOF processing, the prediction signal generation unit 111D performs calculation using pixel values of two reference frames (L0/L1) or values calculated from pixel values of two reference frames (L0/L1). Based on the weighting factor of , the application condition is provided with a condition regarding the temporal distance between the two reference frames and the frame in question.

After the BDOF process is performed, the process proceeds to step S85 and ends.

In step S84, if the block is a block for bi-prediction, the prediction signal generator 111D combines the L0 and L1 prediction signals generated in step S81 to generate a final prediction signal. Since a known method can be used for a specific synthesizing method, the details are omitted. If the block is a block for which uni-prediction is to be performed, the prediction signal generator 111D uses the prediction signal generated in step S81 as it is as the final prediction signal.

After generating the final prediction signal by the above procedure, the prediction signal generation unit 111D proceeds to step S85 and ends the processing.

When the motion vector search unit 111A determines to perform affine motion compensation on the block, the affine motion vector calculation unit 111E divides the block based on the control point motion vector encoded by the motion vector encoding unit 111B. It is configured to calculate a motion vector for each sub-block.

FIG. 9 is a flowchart showing an example of the processing procedure of the affine motion vector calculation unit 111E.

As shown in FIG. 9, in step S91, the affine motion vector calculator 111E calculates values of a total of four parameters "dHorX", "dHorY", "dVerX" and "dVerY".

Here, the affine motion vector calculation unit 111E uses the value of the control point motion vector encoded by the motion vector encoding unit 111B for parameter calculation.

There are two or three control point motion vectors per block, as described above. FIG. 10 shows an example in which there are three control point motion vectors (“CPMV0”, “CPMV1”, and “CPMV2”).

Here, in FIG. 10, the solid line indicates the boundary of the block, the dotted line indicates the boundary of sub-blocks into which the block is divided, the solid arrow indicates the control point motion vector, and the dotted arrow indicates each It shows the motion vectors of the sub-blocks.

When there are two control point motion vectors, only CPMV0 and CPMV1 in FIG. 10 exist. A specific method for calculating the values of the four parameters "dHorX", "dHorY", "dVerX" and "dVerY" from the value of the control point motion vector can use a known technique. Description of is omitted.

In step S92, the affine motion vector calculator 111E determines whether to apply the "fallbackMode" to the block.

Here, "fallbackMode" is a mode in which the same motion vector is used in all sub-blocks into which the block is divided. Since a known method can be used for the determination method of "fallbackMode", the detailed description is omitted. If the "fallbackMode" is applied to the block, the process proceeds to step S93, and if the "fallbackMode" is not applied to the block, the process proceeds to step S94.

In step S93, the affine motion vector calculation unit 111E calculates the motion vector of each sub-block when "fallbackMode" is used.

As described above, in the case of "fallbackMode", all sub-blocks in the same block have the same motion vector value. Since a known method can be used for a specific calculation method, a detailed description is omitted.

For example, in Non-Patent Document 1, the values of "dHorX", "dHorY", "dVerX", and "dVerY" described above, and the coordinate value of the center of the block when the upper left coordinate of the block is set as the origin ( A motion vector is calculated by substituting "1/2" of the height and width) into a predetermined model. In this processing procedure, after step S93 is completed, the process proceeds to step S97 and ends.

In step S94, the affine motion vector calculator 111E calculates the motion vector of each sub-block when the "fallbackMode" is not applied. Since a known method can be used for a specific calculation method, a detailed description is omitted.

For example, in Non-Patent Document 1, the values of “dHorX”, “dHorY”, “dVerX” and “dVerY” and the coordinate value of the center of each sub-block when the origin is the upper left coordinate of the block are predetermined. The motion vector is calculated by substituting it into the model. After step S94 ends, the processing procedure proceeds to step S95.

In step S95, the affine motion vector calculator 111E determines whether or not the block satisfies the conditions for applying the PROF process.

The applicable condition may include a condition that the value of 'sps_affine_prof_enabled_flag' is '1'.

Also, the application condition may include a condition that the value of "slice_disable_bdof_dmvr_prof_flag" indicates that the PROF process can be used in the slice (for example, the value of the flag is "0"). .

Furthermore, the application condition may include a condition that "fallbackMode" is not applied to the block.

When determining that all application conditions are satisfied, the affine motion vector calculation unit 111E determines to apply the PROF processing to the block, and the processing procedure proceeds to step S96. Otherwise, it is decided not to apply the PROF process to the block, and the process proceeds to step S97 and ends.

In the above example, the affine motion vector calculation unit 111E determines whether or not all application conditions are satisfied. may

Specifically, all of the application conditions described above are inverted and defined as conditions for not applying PROF processing. and proceed to step S97, and if none of the above apply, it may be determined to apply the PROF processing to the block and proceed to step S96.

Conditions for not applying the PROF process corresponding to the above example can be defined as follows.

Such non-applicable conditions may include the condition that the value of 'sps_affine_prof_enabled_flag' is '0'.

Also, the non-applicable condition includes the condition that the value of "slice_disable_bdof_dmvr_prof_flag" indicates that the use of PROF processing is prohibited in the slice (for example, the value of the flag is "1"). It's okay.

Further, such non-applicable conditions may include a condition that "fallbackMode" is applied to the block.

In step S96, the affine motion vector calculator 111E calculates the value of "diffMV" used in PROF processing. The value of "diffMV" is calculated for each pixel position when the upper left coordinate of each sub-block is set as the origin, for example. Note that if each sub-block in the block has the same size (for example, 4×4 pixels), calculating the “diffMV” for each pixel position for one sub-block allows the other sub-blocks to have the same size. values can be used.

"diffMV" can be calculated, for example, as follows.

diffMV[x][y][0]=x*(dHorX<<2)+y*(dVerX<<2)-posOffsetX
diffMV[x][y][1]=x*(dHorY<<2)+y*(dVerY<<2)-posOffsetY
Here, x, y means each pixel position (x, y) when the upper left coordinate of each sub-block is taken as the origin, [0] means the x-direction component of "diffMV", and [ 1] means the y-direction component of "diffMV".

In addition, the following relationship is established.

posOffsetX=6*dHorX+6*dVerX
posOffsetY=6*dHorY+6*dVerY
After that, the affine motion vector calculation unit 111E performs right shift and clipping processing as described in Non-Patent Document 1.

After "diffMV" is calculated, the process proceeds to step S97 and ends.

The above description is the processing for one reference frame of the block. For example, in the case of bi-prediction with two reference frames, the processes of steps S91 to S97 described above are executed for L0 and L1, respectively. Therefore, there is a possibility that the determination result of whether or not the PROF process is applied and the calculation result of "diffMV" are different for L0 and L1.

FIG. 11 is a flow chart showing an example of processing of the affine prediction signal generator 111F.

The affine prediction signal generation unit 111F performs PROF processing when “slice_disable_bdof_dmvr_prof_flag” indicates that PROF processing can be used in the slice corresponding to the slice header (when the value of the flag is “0”). , the prediction signal is generated without performing the PROF process when it indicates that the PROF process cannot be used (when the value of the flag is "0").

Note that when the slice header does not include "slice_disable_bdof_dmvr_prof_flag", the affine prediction signal generation unit 111F implicitly interprets and processes the flag as indicating that the flag is usable (the value of the flag is "0"). may be configured to perform

In step S111, the affine prediction signal generator 111F generates a prediction signal.

Specifically, the affine prediction signal generation unit 111F receives as input the motion vector for each subblock calculated by the affine motion vector calculation unit 111E, and if the position indicated by the motion vector is a non-integer pixel position, the reference frame A filter is applied to the pixel values of to interpolate pixels at non-integer pixel positions. Here, as a specific filter, a horizontally and vertically separable filter with a maximum of 6 taps disclosed in Non-Patent Document 1 can be applied.

If the block is a block for bi-prediction, the affine prediction signal generator 111F generates both L0 and L1 prediction signals.

In step S112, the affine prediction signal generator 111F determines whether or not to apply the PROF process in the prediction direction (L0 or L1) of the block. For such determination, the determination result in step S95 described above is used as it is.

If it is determined that the PROF process is applied to the prediction direction of the block, the process proceeds to step S113. On the other hand, if it is determined not to apply the PROF process to the prediction direction of the block, the process proceeds to step S114.

In step S113, the affine prediction signal generator 111F executes PROF processing. Since a known method can be used for specific processing of the PROF processing, details thereof are omitted. The affine prediction signal generator 111F corrects the prediction signal by PROF processing.

In step S114, if the block is a block for bi-prediction, the affine prediction signal generation unit 111F generates the prediction signals of L0 and L1 generated in step S111, or the prediction signals obtained by correcting these prediction signals in step S113. to produce the final prediction signal. Since a known method can be used for a specific synthesizing method, the details are omitted. If the block is a block for which uni-prediction is performed, the affine prediction signal generation unit 111F uses the prediction signal generated in step S111 or the prediction signal obtained by correcting the prediction signal in step S113 as the final prediction. signal.

After the generation of the final prediction signal is completed, the process proceeds to step S115 and ends.

In the above example, the prediction signal generation unit 111D and the affine prediction signal generation unit 111F are explained as different processing blocks for convenience. can also

For example, steps S81 and S111, and steps S84 and S114 can be regarded as the same process.

In this case, if steps S112, S113 and S114 in FIG. 10 are sequentially executed after step S82 in FIG. 8 is determined as No, the flow charts in FIGS. can be regarded as At this time, the BDOF processing and the PROF processing are exclusively applied in units of blocks.
(In-loop filtering unit 150)
The in-loop filtering unit 150 according to this embodiment will be described below. FIG. 12 is a diagram showing the in-loop filtering unit 150 according to this embodiment.

As shown in FIG. 12 , the in-loop filtering section 150 has a block boundary detection section 151 , a boundary strength determination section 153 , a filter determination section 154 and a filter processing section 155 .

Here, the configuration with "A" at the end is the configuration for deblocking filter processing for vertical block boundaries, and the configuration with "B" at the end is for horizontal block boundaries. It is a configuration related to deblocking filter processing.

A case in which deblocking filtering is performed on horizontal block boundaries after deblocking filtering is performed on vertical block boundaries will be exemplified below.

Deblocking filtering may be applied to encoded blocks, predicted blocks, and transformed blocks, as described above. Also, deblocking filtering may be applied to sub-blocks obtained by dividing each of the above blocks. That is, the target block and adjacent blocks may be encoded blocks, predicted blocks, transformed blocks, or sub-blocks obtained by dividing these blocks.

The definition of the sub-block includes the sub-block described as the processing unit of the refining section 111C and the prediction signal generating section 111D. Also, the definition of sub-block includes the sub-block described as the processing unit of the affine motion vector calculation unit 111E and the affine prediction signal generation unit 111E. When applying the deblocking filter to sub-blocks, the blocks described below can be read as sub-blocks as appropriate.

Since the deblocking filtering process for vertical block boundaries and the deblocking filtering process for horizontal block boundaries are similar processes, the deblocking filtering process for vertical block boundaries will be described below.

151 A of block boundary detection parts are comprised so that the boundary of a block may be detected based on the control data which show block size. Here, the blocks are coding blocks (CU), prediction blocks (PU), and transform blocks (TU). Since a known method can be applied to a specific detection method, detailed description is omitted.

153 A of boundary strength determination parts are comprised so that the boundary strength of the block boundary of a target block and an adjacent block may be determined.

Also, the boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether the target block and the adjacent block are intra-prediction blocks.

For example, as shown in FIG. 13, the boundary strength determination unit 153A determines whether at least one of the target block and the adjacent block is an intra-prediction block (that is, at least one of the blocks on both sides of the block boundary is an intra-prediction block). If it is a prediction block), it may be determined that the boundary strength of the block boundary is "2".

In addition, the boundary strength determination unit 153A determines whether or not the target block and adjacent blocks include non-zero (zero) orthogonal transform coefficients, and whether or not the block boundary is a boundary between transform blocks, based on control data. , may be configured to determine the boundary strength of a block boundary.

For example, as shown in FIG. 13, when at least one of the target block and the adjacent block contains a non-zero orthogonal transform coefficient and the block boundary is a transform block boundary, (i.e., when at least one of the blocks on both sides of the block boundary has a non-zero transform coefficient and is a TU boundary), the boundary strength of the block boundary is determined to be "1". may be

In addition, the boundary strength determination unit 153A determines whether or not the absolute value of the motion vector difference between the target block and the adjacent block is equal to or greater than a threshold value (for example, 1/2 pixel). It may be configured to determine strength.

For example, as shown in FIG. 13, the boundary strength determination unit 153A determines when the absolute value of the difference between the motion vectors of the target block and the adjacent block is equal to or greater than a threshold value (for example, 1/2 pixel) (that is, when both sides of the block boundary The boundary strength of the block boundary may be determined to be "1" when the absolute value of the motion vector difference between the blocks is equal to or greater than a threshold value (eg, 1/2 pixel).

Further, the boundary strength determination unit 153A is configured to determine the boundary strength of the block boundary based on control data indicating whether or not the reference blocks referred to in the motion vector prediction of the target block and the adjacent block are different. may be

For example, as shown in FIG. 13, the boundary strength determination unit 153A determines whether the reference blocks referred to in the motion vector prediction of the current block and the adjacent block are different (that is, when the reference images are different in the blocks on both sides of the block boundary). ), the boundary strength of the block boundary may be determined to be “1”.

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether or not the numbers of motion vectors of the target block and the adjacent blocks are different.

For example, as shown in FIG. 13, the boundary strength determination unit 153A determines whether the block It may be configured to determine that the boundary strength of the boundary is "1".

For example, as shown in FIG. 13, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "0" when none of the above conditions are satisfied.

It should be noted that the larger the value of the boundary strength, the higher the possibility that the block distortion occurring at the block boundary is large.

The boundary strength determination method described above may be determined using a common method for the luminance signal and the color difference signal, or may be determined using partially different conditions.

The filter determination unit 154A is configured to determine the type of filtering (for example, deblocking filtering) to be applied to block boundaries.

For example, the filter determination unit 154A determines whether or not to apply filter processing to block boundaries, weak filter processing and It may be configured to determine which filtering of the strong filtering is to be applied.

The filter determining unit 154A may be configured to determine not to apply filtering when the boundary strength of the block boundary is "0".

The filter processing unit 155A is configured to process the pre-deblocking image based on the determination of the filter determination unit 154A. The processing for the pre-deblocking image includes no filtering, weak filtering, strong filtering, and the like.
(Image decoding device 200)
The image decoding device 200 according to this embodiment will be described below with reference to FIG. FIG. 14 is a diagram showing an example of functional blocks of the image decoding device 200 according to this embodiment.

As shown in FIG. 14, the image decoding device 200 includes a decoding unit 210, an inverse transform/inverse quantization unit 220, an adder 230, an inter prediction unit 241, an intra prediction unit 242, and an in-loop filtering unit. 250 and a frame buffer 260 .

The decoding unit 210 is configured to decode the encoded data generated by the image encoding device 100 and decode the coefficient level values.

Here, for example, the decoding is entropy decoding in a procedure opposite to the entropy encoding performed by the encoding unit 140 .

Further, the decoding unit 210 may be configured to acquire the control data by decoding the encoded data.

Note that, as described above, the control data may include size data such as the encoding block size, prediction block size, transform block size, and the like.

Also, as described above, the control data may include header information such as sequence parameter sets (SPS), picture parameter sets (PPS), slice headers, and the like.

For example, the decoding unit 210 is configured to decode a flag for controlling whether or not to use the PROF process, or a flag for collectively controlling whether or not to use the DMVR process, the BDOF process, and the PROF process, from the slice header. may

The inverse transform/inverse quantization unit 220 is configured to perform inverse transform processing on the coefficient level values output from the decoding unit 210 . Here, the inverse transform/inverse quantization unit 220 may be configured to perform inverse quantization of the coefficient level values prior to the inverse transform processing.

Here, the inverse transform processing and inverse quantization are performed in the reverse order of the transform processing and quantization performed by the transform/quantization unit 131 .

The adder 230 adds the prediction signal to the prediction residual signal output from the inverse transform/inverse quantization unit 220 to generate a pre-filtering decoded signal. It is configured to output to the filter processing unit 250 .

Here, the unfiltered decoded signal constitutes a reference block used in intra prediction section 242 .

Like the inter prediction section 111, the inter prediction section 241 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

Specifically, the inter prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from encoded data and a reference signal included in a reference frame. The inter prediction section 241 is configured to output a prediction signal to the adder 230 .

The intra prediction section 242, like the intra prediction section 112, is configured to generate a prediction signal by intra prediction (intra-frame prediction).

Specifically, the intra prediction unit 242 is configured to identify a reference block included in the target frame and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction section 242 is configured to output the prediction signal to the adder 230 .

Similar to in-loop filtering section 150 , in-loop filtering section 250 performs filtering on the unfiltered decoded signal output from adder 230 and outputs the filtered decoded signal to frame buffer 260 . is configured to

Here, for example, the filter processing is deblocking filter processing that reduces distortion occurring at boundaries of blocks (encoding blocks, prediction blocks, transform blocks, or sub-blocks obtained by dividing them).

The frame buffer 260, like the frame buffer 160, is configured to accumulate reference frames used by the inter prediction section 241. FIG.

Here, the decoded signal after filtering constitutes a reference frame used in the inter prediction section 241 .
(Inter prediction unit 241)
The inter prediction unit 241 according to this embodiment will be described below with reference to FIG. 15 . FIG. 15 is a diagram showing an example of functional blocks of the inter prediction unit 241 according to this embodiment.

As shown in FIG. 15, the inter prediction unit 241 includes a motion vector decoding unit 241B, a refinement unit 241C, a prediction signal generation unit 241D, an affine motion vector calculation unit 241E, and an affine prediction signal generation unit 241F. .

The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on motion vectors.

The motion vector decoding unit 241B is configured to acquire motion vectors by decoding control data received from the image encoding device 100 .

The motion vector decoding unit 241B is also configured to decode information as to whether or not affine motion compensation is used for the block.

Here, when the affine motion compensation is not used for the block, the processing of the refinement processing unit 241C and the prediction signal generation unit 241D is performed. The processing of the generation unit 241F is performed.

The refinement unit 241C is configured to perform a refinement process for correcting motion vectors, similar to the refinement unit 111C.

The prediction signal generator 241D is configured to generate a prediction signal based on the motion vector, like the prediction signal generator 111D.

The affine motion vector calculator 241E, like the affine motion vector calculator 111E, is configured to calculate the motion vector of each sub-block using the control point motion vectors decoded by the motion vector decoder 241B. .

Similar to the affine prediction signal generation unit 111F, the affine prediction signal generation unit 241F generates a prediction signal using the motion vector of each sub-block calculated by the affine motion vector calculation unit 241E. As with the prediction signal generator 111D and the affine prediction signal generator 111F, they can be integrated into the same processing block.
(In-loop filtering unit 250)
The in-loop filtering unit 250 according to this embodiment will be described below. FIG. 16 is a diagram showing the in-loop filtering unit 250 according to this embodiment.

As shown in FIG. 16 , the in-loop filtering section 250 has a block boundary detection section 251 , a boundary strength determination section 253 , a filter determination section 254 and a filter processing section 255 .

Here, the configuration with "A" at the end is the configuration for deblocking filter processing for vertical block boundaries, and the configuration with "B" at the end is for horizontal block boundaries. It is a configuration related to deblocking filter processing.

Here, a case is illustrated in which deblocking filtering is performed on horizontal block boundaries after deblocking filtering is performed on vertical block boundaries.

Deblocking filtering may be applied to encoded blocks, predicted blocks, and transformed blocks, as described above. Also, deblocking filtering may be applied to sub-blocks obtained by dividing each block described above. That is, the target block and adjacent blocks may be encoded blocks, predicted blocks, transformed blocks, or sub-blocks obtained by dividing these blocks.

Since the deblocking filtering process for vertical block boundaries and the deblocking filtering process for horizontal block boundaries are similar processes, the deblocking filtering process for vertical block boundaries will be described below.

The block boundary detection section 251A, like the block boundary detection section 151A, is configured to detect block boundaries based on control data indicating the block size.

The boundary strength determination unit 253A is configured to determine the boundary strength of the block boundary between the target block and the adjacent block, similarly to the boundary strength determination unit 153A. The method for determining the boundary strength of block boundaries is as described above.

The filter determination unit 254A is configured to determine the type of deblocking filter processing to be applied to block boundaries, similar to the filter determination unit 154A. The method for determining the type of deblocking filtering is as described above.

The filter processing unit 255A is configured to process the pre-deblocking image based on the determination of the filter determination unit 254A, similarly to the filter processing unit 155A. The processing for the pre-deblocking image includes no filtering, weak filtering, strong filtering, and the like.
(Modification 1)
Modification 1 of the above-described embodiment will be described below with reference to FIGS. 17 and 18. FIG. In the following, differences with respect to the above-described embodiments will be mainly described. In the above-described embodiment, detailed descriptions of the BDOF processing and the PROF processing are omitted, but in this modified example, an example of specific processing will be described.

First, a specific example of the BDOF processing in step S83 will be described with reference to FIG.

As shown in FIG. 17, in step S171, the prediction signal generation unit 111D/241D performs horizontal and vertical directions for each pixel in the reference blocks of L0 and L1 corresponding to the block or sub-block to which the BDOF process is applied. Luminance value gradients are calculated respectively.

Specifically, the prediction signal generator 111D/241D calculates the above-described gradient for each pixel position (x, y) in the block using the following formula.

gradientHL0[x][y]=(predSamplesL0[hx+1][vy]>>shift1)-(predSamplesL0[hx−1][vy]>>shift1)
gradientV0[x][y]=(predSamplesL0[hx][vy+1>shift1)-(predSamplesL0[hx][vy-1]>>shift1)
gradientHL1[x][y]=(predSamplesL1hx+1][vy]>>shift1)−(predSamplesL1[hx−1][vy]>>shift1)
gradientV1[x][y]=(predSamplesL1[hx][vy+1]>shift1)-(predSamplesL0[hx][vy-1]>>shift1)
Here, "predSamplesL0" and "predSamplesL1" are the prediction signals of L0 and L1 calculated in step S81. [hx] and [vy] are defined by the following formulas.

In the following formula, 'nCbW' is the width of the block, 'nCbH' is the height of the block, and 'Clip3 (min, max, input) is the value of 'input' to 'min≤ input≦max”.

hx=Clip3(1, nCbW, x)
hy=Clip3(1, nCbH, y)
Note that "shift1" is defined by the following formula.

shift1=Max(6, bitDepth-6)
Here, "bitDepth" is the bit depth when processing pixel values.

In step S172, the prediction signal generator 111D/241D calculates the corrected MV.

Here, first, the prediction signal generator 111D/241D calculates the values of the following three parameters for each pixel of the block or sub-block.

diff[x][y]=(predSamplesL0[hx][vy]>>shift2)-(predSamplesL1[hx][vy]>>shift2)
tempH[x][y]=(gradientHL0[x][y]+gradientHL1[x][y])>>shift3
tempV[x][y]=(gradientVL0[x][y]+gradientVL1[x][y])>>shift3
Here, "shift2" and "shift3" are defined by the following equations.

shift2=Max(4, bitDepth-8)
shift3=Max(3,15-bitDepth)
Second, the prediction signal generator 111D/241D divides the block or sub-block into 4×4 pixel blocks, and calculates parameters for each 4×4 pixel block as follows.

Here, “xSb” and “yXb” are the coordinates of the upper left pixel of each 4×4 pixel block when the upper left pixel of the block or sub-block is taken as the origin. Also, "Sign(input)" returns "1" if the value of "input" is positive, returns "-1" if the value of "input" is negative, and returns "-1" if the value of "input" is " 0", it is a function that returns "0".

The prediction signal generator 111D/241D calculates the correction MV (vx, vy) as follows using the values calculated as described above.

vx=sGx2>0? Clip3 (-mvRefineThres, mvRefineThres-1, -(sGxdI<<3)>>Floor(Log2(sGx2))): 0
vy=sGx2>0? Clip3 (-mvRefineThres, mvRefineThres-1, ((sGydI<<3)-((vx×sGxGy _m ) <<12+vx×sGxGy _s )>>1)>>Floor(Log2(sGx2))): 0
Here, "mvRefineThres" is defined by the following formula.

mvRefineThres=1<<Max(5, bitDepth-7)
With the above configuration, the bit width of each component of the corrected MV (vx, vy) can be suppressed within "1<<Max(5, bitDepth-7)+1" bits including the sign.

In step S173, the prediction signal generator 111D/241D corrects the prediction signal of each pixel as follows.

bdofOffset=(vx×(gradientHL0[x+1][y+1]−gradientHL1[x+1][y+1]))>>1+(vy×(gradientVL0[x+1][y+1]−gradientVL1[x+1][y+1]))>>1
pbSamples[x][y]=Clip3(0, (2 ^bitDepth )−1, (predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1]+bdofOffset)>>shift4)
Here, "offset4" and "shift4" are defined by the following equations.

offset4=1<<(shift4-1)
shift4=Max(3,15-bitDepth)
After performing all of the above processes, the prediction signal generator 111D/241D proceeds to step S174 and ends the process.

Next, a specific example of the PROF processing in step S113 will be described with reference to FIG.

As shown in FIG. 18, in step S181, the affine prediction signal generator 111F/241F calculates horizontal and vertical luminance value gradients for each pixel in the L0 and L1 reference blocks to which the PROF process is applied. do.

Specifically, the affine prediction signal generator 111F/241F calculates the gradient for each pixel position (x, y) in the block using the following formula.

gradientH[x][y]=(predSamples[x+2][y]>>shift1)−(predSamples[x][y]>>shift1)
gradientV[x][y]=(predSamples[x][y+2]>>shift1)−(predSamples[x][y]>>shift1)
Here, "predSamples" is the L0 or L1 prediction signal generated in step S111. here,
shift1=Max(6, bitDepth-6)
and

When applying the PROF processing to both L0 and L1, the affine prediction signal generator 111F/241F calculates the above-described "gradientH" and "gradientV" for L0 and L1, respectively. In this case, the calculation content in step S181 is completely the same as the calculation content in step S171 when viewed in units of pixels.

In step S182, the affine prediction signal generator 111F/241F calculates the corrected MV. This processing is actually performed inside the affine motion vector calculation unit 111E/241E as step S96, but for the sake of convenience, it will be described as being performed in step S182. Actually, the affine prediction signal generator 111F/241F may use the value calculated in step S96 as it is.

First, the affine prediction signal generation unit 111F/241F calculates the value of "diffMv" in the same manner as described in step S96 above.

diffMv[x][y][0]=x*(dHorX<<2)+y*(dVerX<<2)-posOffsetX
diffMv[x][y][1]=x*(dHorY<<2)+y*(dVerY<<2)-posOffsetY
Second, the affine prediction signal generators 111F/241F shift the above-described "diffMv" values to the right by 7 bits.

Third, the affine prediction signal generator 111F/241F performs clipping processing as follows.

diffMv[x][y][i]=Clip3(-dmvLimit, dmvLimit-1, diffMv[x][y][i])
here,
dmvLimit=1<<Max(5, bitDepth-7)
is.

With the above configuration, the bit width of each component of "diffMv" can be suppressed to within "1<<Max(5, bitDepth-7)+1" bits with encoding. This is the same bit width as the corrected MV calculated in step S172.

In step S183, the affine prediction signal generator 111F/241F corrects the prediction signal.

As described below, the affine prediction signal generation unit 111F/241F corrects the value of the prediction signal “pbSamples” for the block to which the PROF processing is applied and the prediction direction.

dI=gradientH[x][y]*diffMv[x][y][0]+gradientV[x][y]*diffMv[x][y][1]
pbSamples[x][y]=predSamples[x+1][y+1]+(dI>>1)
When applying the PROF processing to both L0 and L1, the affine prediction signal generator 111F/241F performs the above-described processing for each of L0 and L1.

After completing the above-described correction, the affine prediction signal generator 111F/241F moves to step S184 and ends the PROF process.

After completing the PROF process, the affine prediction signal generator 111F/241F executes the prediction value combining process described in step S114. Synthesis of predicted values can be performed, for example, by the following formula.

predSamples[x][y]=Clip3(0, (1<<bitDepth)−1, (predSamplesL0[x][y]+predSamplesL1[x][y]+offset2)>>shift2)
here,
offset2=1<<(shift2-1)
shift2=Max(3,15-bitDepth)
is.

Further, when both the L0-side prediction signal "predSamplesL0" and the L1-side prediction signal "predSamplesL1" have been corrected by the PROF process, substituting the calculation in step S183 for "predSamplesL0" and "predSamplesL1" yields the following: can be transformed as

dI_Bi=(dI_L0>>1)+(dI_L1>>1)
pbSamples[x][y]=Clips(0, (1<<bitDepth)−1, (predSamplesL0_orig[x][y]+offset2+predSamplesL1_orig[x][y]+dI_Bi)>>shift2)
Here, “dI_L0” and “dI_L1” are dI values of L0 and L1, respectively, and “predSamplesL0_orig” and “predSamplesL1_orig” are prediction signals before correction by PROF processing.

Since the gradient and correction MV values calculated by the BDOF processing and PROF processing described above are clipped so that the bit widths are the same, the bit widths of "bdofOffset" and "dI_Bi" calculated therefrom are are also the same.

That is, the offset amount calculated during the BDOF processing performed by the prediction signal generation unit prediction signal generation unit 111D/241D and the two reference frames (L0/L1) during the PROF processing performed by the affine prediction signal generation unit 111F/241F The calculation may be performed so that the bit width of the combined value of the calculated offset amounts is the same.

Also, the values of “offset2” and “shift2” when synthesizing the predicted signal after PROF processing are the same as the values of “offset4” and “shift4” of BDOF processing. Therefore, the prediction value generation processing when the BDOF processing and the PROF processing are applied to both L0 and L1 are completely the same formula.

By configuring the BDOF processing and the PROF processing as described above, the following is obtained.

First, the gradient calculation processing in steps S171 and S181 can be exactly the same.

That is, the gradient calculation processing during the BDOF processing performed by the prediction signal generation units 111D/241D and the gradient calculation processing during the PROF processing performed by the affine prediction signal generation units 111F/241F are performed by the two reference frames (L0/ L1) can be treated identically when applied to both.

Secondly, the bit widths of the corrected MVs output from steps S172 and S182 can be exactly the same.

That is, the corrected MV amount used in the BDOF processing performed by the prediction signal generation units 111D/241D and the corrected MV amount used in the PROF processing performed by the affine prediction signal generation units 111F/241F are set to have the same bit width. Clipping processing can be performed for each.

Thirdly, the prediction value correction processing in step S173 and the prediction value correction and prediction value combination processing in steps S183 and S114 can be exactly the same processing.

That is, the final prediction signal generation processing using the offset amount performed by the prediction signal generation unit 111D/241D and the final prediction using the combined value of the offset amount performed by the affine prediction signal generation unit 111F/241F The signal generation process can be the same process.

Since the BDOF processing and the PROF processing are applied exclusively block by block, as described above, the same processing portion described above, when implemented on hardware, is the BDOF processing and the PROF processing. can be shared. As a result, a reduction in circuit scale can be expected.
(Modification 2)
Modification 2 of the above-described embodiment will be described below with reference to FIGS. 19 and 20. FIG. In the following, differences with respect to the above-described embodiments will be mainly described. In this modified example, an example different from the above embodiment will be described for the block boundary detection method in the block boundary detection units 151/251 and the boundary strength determination method in the boundary strength determination units 153/253.

Although the block boundary detection section 151A will be described below, block boundaries can also be detected by similar processing in the block boundary detection sections 151B/251A/251B.

FIG. 19 is a flow chart showing an example of processing of the block boundary detection unit 151A. A case where processing is performed in units of CUs will be described below as an example.

In step S191, the block boundary detection unit 151A detects CU boundaries. Since the block boundary detection unit 151A is a processing block that detects a vertical boundary, it detects the left vertical boundary of the CU as a block boundary to be filtered.

Note that "detecting as a block boundary" means, for example, preparing a two-dimensional flag array (for example, "edgeFalgs[x][y]") indicating whether the pixel position in the screen is a boundary or not. This corresponds to setting the flag value to "1" at pixel positions detected as boundaries and setting the flag value to "0" at other pixel positions.

Here, when the left vertical boundary of the CU is positioned at a coding unit boundary such as a picture boundary, a slice boundary, or a tile boundary, the block boundary detection unit 151A does not detect the boundary as a block boundary. You can do it.

After the block boundary detection unit 151A detects the CU boundary in the above procedure, the process proceeds to step S192.

In step S192, the block boundary detection unit 151A detects TU boundaries (transformation block boundaries).

The block boundary detection unit 151A detects whether each pixel position is a TU boundary while increasing the horizontal coordinate by 4 pixels and the vertical pixel position by 1 pixel, starting from the vertical boundary on the left side of the CU. to confirm. If the pixel position is a TU boundary, the block boundary detection unit 151A detects the pixel position as a block boundary.

After the block boundary detection unit 151A detects the TU boundary in the above procedure, the process proceeds to step S193.

In step S193, the block boundary detection unit 151A detects sub-block boundaries.

The sub-block boundary to be detected in step S193 may be, for example, the sub-block boundary of a CU subjected to inter prediction.

At this time, the above-mentioned sub-block boundaries include boundaries of sub-blocks (eg, 16×16 pixel size) in DMVR processing performed by refinement units 111C/241C, and affine motion vector calculation units 111E/241E and affine motion vector calculation units 111E/241E. Both boundaries of sub-blocks (for example, 4×4 pixel size) in affine motion compensation performed by the prediction signal generator 111F/241F may be included.

The sub-block boundary to be detected in step S193 may be, for example, the sub-block boundary of a CU to which affine motion compensation or "merge_subblock" is applied.

Note that sub-blocks for affine motion compensation match sub-blocks for PROF processing when PROF processing is applied to the CU.

Here, when the sub-block boundary for affine motion compensation is to be detected in step S193, the block boundary detection unit 151A detects the sub-block boundary for affine motion compensation and the PROF process when the PROF process is applied to the CU. may be excluded from detection targets.

That is, the block boundary detection unit 151A may detect sub-block boundaries for affine motion compensation only when affine motion compensation is applied to the CU and PROF processing is not applied to the CU. Of course, the block boundary detection unit 151A may detect sub-block boundaries for affine motion compensation and sub-block boundaries for PROF processing for CUs to which PROF processing is applied.

Note that the block boundary detection unit 151A regards a sub-block boundary of a block on which bi-prediction is performed as a "CU to which PROF processing is applied" when PROF processing is applied to at least one of L0 or L1. may Also, the block boundary detection unit 151A may regard a CU to which the PROF process is applied only when the PROF process is applied to both L0 and L1. The following description can be similarly interpreted.

Starting from the vertical boundary on the left side of the CU, the block boundary detection unit 151A increases the horizontal coordinates by the sub-block size calculated by the following method and increases the vertical pixel position by one pixel. Check if each pixel location is a sub-block boundary.

If the pixel position is a sub-block boundary, the block boundary detection unit 151A detects the pixel position as a block boundary. Here, even if the position is a sub-block boundary, the block boundary detection unit 151A may perform processing not to detect the position as a sub-block boundary when the PROF processing is applied to the CU.

The above sub-block size can be calculated, for example, as follows.

Sub-block size = Max(8, (nCbW/NumSbX))
Here, "nCbW" is the width of the CU (the number of pixels in the horizontal direction), and "NumSbX" is the number of sub-blocks included in the CU in the horizontal direction.

When the above formula is used, the sub-block boundary is determined whether it is a sub-block boundary at intervals of every 8 pixels in the horizontal direction, starting from the vertical boundary on the left side of the CU. Become.

Also, the sub-block size described above may be calculated as follows, for example.

Sub-block size = Max(4, (nCbW/NumSbX))
In this case, the sub-block boundary is judged whether or not it is a sub-block boundary at intervals of every 4 pixels in the horizontal direction, starting from the vertical boundary on the left side of the CU in question.

That is, the block boundary detection unit 151A is configured to determine whether or not the target pixel position is a sub-block boundary at intervals of at least every four pixels with reference to the coordinates of the left end or top end of the target block. there is

After the block boundary detection unit 151A detects the sub-block boundary in the above procedure, the process proceeds to step S194 and ends.

In the above example, the CU boundary, the TU boundary, and the sub-block boundary are simply detected as block boundaries without being distinguished from each other, but they may be stored separately.

For example, they may be stored separately as CU boundaries, TU boundaries, and sub-block boundaries, or as TU boundaries and motion vector boundaries. At this time, CU boundaries are detected as both TU boundaries and motion vector boundaries. Also, TU boundaries are detected as TU boundaries, and sub-block boundaries are detected as motion vector boundaries. This distinction can be used, for example, in the boundary strength determination section 153A described below.

Although the boundary strength determination unit 153A will be described below, exactly the same determination can be performed by the boundary strength determination units 153B/253A/253B.

The boundary strength determination unit 153A may be configured to determine the boundary strength to be "0" when the target pixel position is not a block boundary.

That is, the block boundary detection unit 151 is configured not to detect sub-blocks in blocks to which the PROF processing is applied as block boundaries, and the boundary strength determination unit 153A detects pixel positions not detected as block boundaries. The boundary strength is determined to be "0", and the filter determination unit 154 is configured to determine not to apply the deblocking filter to the block boundary when the boundary strength is "0". may

153 A of boundary strength determination parts may be comprised so that the boundary strength of a block boundary may be determined based on the control data which show whether a target block and an adjacent block are intra prediction blocks.

For example, as shown in FIG. 20, the boundary strength determination unit 153A determines whether at least one of the target block and the adjacent block is an intra-prediction block (that is, at least one of the blocks on both sides of the block boundary is an intra-prediction block). If it is a prediction block), it may be determined that the boundary strength of the block boundary is "2".

In addition, the boundary strength determination unit 153A determines based on control data indicating whether or not the target block and adjacent blocks include non-zero (zero) orthogonal transform coefficients and whether or not the block boundary is a boundary between transform blocks. It may be arranged to determine the boundary strength of the block boundary.

For example, as shown in FIG. 20, when at least one of the target block and the adjacent block contains a non-zero orthogonal transform coefficient, and the block boundary is a boundary between transform blocks, the boundary strength determination unit 153A (i.e., when at least one of the blocks on both sides of the block boundary has a non-zero transform coefficient and is a TU boundary), the boundary strength of the block boundary is determined to be "1". may be

In addition, the boundary strength determination unit 153A determines the boundary strength of the block boundary based on control data indicating whether the absolute value of the difference between the motion vectors of the target block and the adjacent block is equal to or greater than a threshold value (eg, ½ pixel). may be configured to determine. At that time, the boundary strength of the block boundary may be determined based on whether the block boundary is a sub-block boundary within the block to which the PROF processing is applied.

For example, as shown in FIG. 20, the boundary strength determination unit 153A determines when the absolute value of the difference between the motion vectors of the target block and the adjacent block is equal to or greater than a threshold value (for example, 1/2 pixel) (that is, when both sides of the block boundary If the absolute value of the motion vector difference between the blocks is equal to or greater than a threshold value (for example, 1/2 pixel), and the block boundary is the sub-block (4 It may be configured to determine that the boundary strength of the block boundary is "1" when it is not a x4 pixel block) boundary.

That is, the boundary strength determination unit 153A is configured to determine the boundary strength for each block boundary, the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is 1/2 pixel or more, and , the boundary strength may be determined to be "1" when the block boundary is not a sub-block boundary to which PROF processing is applied.

Here, the filter determining unit 154 may be configured to determine not to apply the deblocking filter to the block boundary when the boundary strength is "0".

For example, when the absolute value of the difference between the motion vectors of the target block and the adjacent block is equal to or greater than a threshold value (for example, 1/2 pixel), the boundary strength determination unit 153A determines the difference between the motion vectors of the blocks on both sides of the block boundary. is equal to or greater than a threshold value (for example, 1/2 pixel)), and the blocks on both sides of the block boundary are not blocks determined to apply PROF processing in step S95. It may be configured to determine that the boundary strength is "1".

For example, when the absolute value of the difference between the motion vectors of the target block and the adjacent block is equal to or greater than a threshold value (for example, 1/2 pixel), the boundary strength determination unit 153A determines the difference between the motion vectors of the blocks on both sides of the block boundary. is equal to or greater than a threshold (e.g., 1/2 pixel)), and at least one of the block boundaries is not the block determined to apply PROF processing in step S95, It may be configured to determine that the boundary strength of the block boundary is "1".

Note that sub-block boundaries of blocks to be bi-predicted may be regarded as “blocks determined to apply PROF processing” when PROF processing is applied to at least one of L0 and L1. That is, in this case, if the PROF process is not applied to both L0 and L1, the boundary strength is determined to be "1" if the motion vector difference is equal to or greater than the threshold.

That is, when the target block is bi-prediction and when the PROF process is applied to at least one of the two reference frames, the boundary strength determination unit 153A determines that the PROF process has been applied to the target block. It may be configured to assume

Also, only when the PROF process is applied to both L0 and L1, it may be regarded as "a block determined to apply the PROF process". That is, in this case, if the PROF process is applied only to L0 or L1 and if the motion vector difference is equal to or greater than the threshold, the boundary strength is determined to be "1".

As another example, at block boundaries to which PROF processing is applied, it is also possible to determine whether the motion vector difference is not determined or the motion vector difference is regarded as "0" in determining the boundary strength. . As a specific example, it can be determined as follows.

If the block is a block that performs uni-prediction and PROF is applied to the block, the boundary strength determination at the sub-block boundary does not determine the motion vector difference or considers the motion vector difference to be "0". .

If the block is a block that performs bi-prediction and the reference frame on the L0 side and the reference frame on the L1 side are different, the prediction direction to which PROF is applied (L0 only, L1 only, or both of the L0 side and L1 side) ), the motion vector difference is not determined or the motion vector difference is regarded as "0".

If the block is a block that performs bi-prediction, and the reference frame on the L0 side and the reference frame on the L1 side are the same frame, and when PROF is applied on the L0 side, the movement of L0 on both sides of the sub-block boundary Do not judge the difference between the vectors or consider the difference between the L0 motion vectors on both sides of the sub-block boundary to be "0".

Similarly, when PROF is applied on the L1 side, the difference between the motion vectors of L1 on both sides of the sub-block boundary is not determined, or the difference between the motion vectors of L1 on both sides of the sub-block boundary is set to "0". Consider.

In this case, the motion vector difference between L0 and L1 is conventionally used to determine the boundary strength. Specifically, for example, the difference between the L0-side motion vector of the sub-block on the left side of the sub-block boundary and the L1-side motion vector of the sub-block on the right side of the sub-block boundary is used to determine the boundary strength. .

That is, when determining the difference between the motion vectors in the same prediction directions of the blocks on both sides of the target boundary, the boundary strength determination unit 153A determines that the motion vectors in the prediction directions are , or the difference between the motion vectors in the prediction directions may be regarded as "0".

Alternatively, if the block is a block that performs bi-prediction and the reference frame on the L0 side and the reference frame on the L1 side are the same frame, motion vectors on both sides of the sub-block boundary apply PROF. It is also possible to not determine the vector difference or to consider the motion vector difference to be "0".

In this case, for example, when PROF is applied on both the L0 side and the L1 side, the difference in motion vector between L0 and L1 is not determined, or the difference in motion vector between L0 and L1 is regarded as "0". is different from the example above.

In addition, the boundary strength determination unit 153A is configured to determine the boundary strength of the block boundary based on control data indicating whether or not the reference blocks referred to in the motion vector prediction of the target block and the adjacent block are different. may be

For example, as shown in FIG. 20, the boundary strength determination unit 153A determines whether the reference blocks referred to in the motion vector prediction of the target block and the adjacent block are different (that is, when the reference images are different in the blocks on both sides of the block boundary). ), the boundary strength of the block boundary may be determined to be “1”.

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether or not the numbers of motion vectors of the target block and the adjacent blocks are different.

For example, as shown in FIG. 20, the boundary strength determination unit 153A determines whether the block It may be configured to determine that the boundary strength of the boundary is "1".

An example of a method of determining boundary strength using motion vectors and reference blocks has been described above. Only the block boundaries detected as .

In other words, determination using motion vectors and reference blocks may be skipped at block boundaries that are TU boundaries but not sub-block boundaries or motion vector boundaries. At this time, at the TU boundary, it is only confirmed whether or not at least one of the blocks on both sides of the boundary has a non-zero orthogonal transform coefficient.

For example, as shown in FIG. 20, the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is "0" when none of the above conditions are satisfied.

That is, the affine prediction signal generator 111F/241F is configured to determine whether the PROF process is appropriate for each block, and to perform the PROF process for each sub-block obtained by dividing the block when applying the PROF process.

Further, the in-loop filtering unit 150/250 is configured to determine whether the deblocking filter is appropriate for each block boundary, and to determine not to apply the deblocking filter to sub-block boundaries of blocks to which PROF processing has been applied. It is

With the configuration as described above, the boundary strength of the block boundary or sub-block boundary to which the PROF processing is applied becomes "0", and the deblocking filter is not applied. Since the PROF process has the effect of improving the subjective image quality, it is possible to prevent the subjective image quality from being impaired by correcting the pixel values at the block boundary using the deblocking filter with such a configuration.

Also, the image encoding device 100 and the image decoding device 200 described above may be implemented as a program that causes a computer to execute each function (each process).

In each of the above-described embodiments, the present invention is applied to the image encoding device 100 and the image decoding device 200 as examples, but the present invention is not limited to such examples. The same can be applied to an image encoding/decoding system having the functions of the encoding device 100 and the image decoding device 200 .

According to the present invention, degradation of subjective image quality can be prevented by not applying a deblocking filter to sub-block boundaries to which PROF processing has been applied.

10... Image processing system 100... Image encoding devices 111, 241... Inter prediction unit 111A... Motion vector search unit 111B... Motion vector encoding units 111C, 241C... Refinement units 111D, 241D... Prediction signal generation units 111E, 241E... Affine motion vector calculation units 111F, 241F... Affine prediction signal generation units 112, 242... Intra prediction unit 121... Subtractors 122, 230... Adder 131... Transform/quantization units 132, 220... Inverse transform/inverse quantization unit 140 Encoding units 150, 250 In-loop filtering units 160, 260 Frame buffer 200 Image decoding device 210 Decoding unit 241B Motion vector decoding unit

Claims (9)

An image decoding device,
an affine prediction signal generator configured to determine whether or not PROF (Prediction Refinement with Optical Flow) processing is appropriate for each block, and to perform the PROF processing for each sub-block obtained by dividing the block when the PROF processing is applied;
an in-loop filtering unit configured to determine whether the deblocking filter is appropriate for each block boundary, and to determine not to apply the deblocking filter to sub-block boundaries of blocks to which the PROF processing has been applied. An image decoding device characterized by: The in-loop filtering unit is configured to determine whether the target pixel position is the sub-block boundary at intervals of at least every four pixels with reference to the coordinates of the left end or the upper end of the target block. 2. The image decoding device according to claim 1, further comprising a block boundary detection unit. The in-loop filtering unit is
The method is configured to determine a boundary strength for each of the block boundaries, wherein an absolute value of a difference between motion vectors of blocks on both sides of the block boundary is 1/2 pixel or more, and PROF processing is applied to the block boundary. a boundary strength determination unit configured to determine the boundary strength as "1" if the boundary is not a sub-block boundary;
3. The filter determination unit according to claim 1, further comprising a filter determination unit configured to determine not to apply a deblocking filter to the block boundary when the boundary strength is '0'. Image decoding device. The in-loop filtering unit is
a block boundary detection unit configured not to detect sub-blocks in a block to which the PROF processing is applied as the block boundary;
a boundary strength determination unit configured to determine a boundary strength of "0" at a pixel position not detected as the block boundary;
3. The filter determination unit according to claim 1, further comprising a filter determination unit configured to determine not to apply a deblocking filter to the block boundary when the boundary strength is '0'. Image decoding device. The boundary strength determination unit applies the PROF processing to the target block when the target block is bi-predictive and when the PROF processing is applied to at least one of two reference frames. 5. The image decoding device according to claim 3, wherein the image decoding device is configured to be regarded as When the target block is bi-predictive and the PROF processing is applied to both of two reference frames, the boundary strength determination unit considers that the PROF processing has been applied to the target block. 5. The image decoding device according to claim 3, wherein: When determining a difference between motion vectors in the same prediction direction of blocks on both sides of the target boundary, the boundary strength determination unit determines that, when the PROF process is applied to the prediction direction, motion vectors in the prediction directions 5. The image decoding apparatus according to claim 3 , wherein the apparatus is configured not to determine the difference or to regard the difference between the motion vectors in the prediction directions as zero. a step of determining whether or not PROF (Prediction Refinement with Optical Flow) processing is appropriate for each block, and performing the PROF processing on each sub-block obtained by dividing the block when applying the PROF processing;
determining suitability of a deblocking filter for each block boundary, and determining not to apply the deblocking filter to a sub-block boundary of a block to which the PROF processing has been applied. A program for use in an image decoding device, the computer comprising:
a step of determining whether or not PROF (Prediction Refinement with Optical Flow) processing is appropriate for each block, and performing the PROF processing on each sub-block obtained by dividing the block when applying the PROF processing;
determining whether a deblocking filter is appropriate for each block boundary, and determining that the deblocking filter is not applied to a sub-block boundary of a block to which the PROF processing has been applied.

Images