CN114270867B - Image decoding device, image decoding method, and program - Google Patents

Abstract

The image decoding device 200 of the present invention includes: an affine prediction signal generating unit 241F, which is configured to determine whether PROF processing is suitable for each block, and when PROF processing is applied, PROF processing is performed on each sub-block obtained by dividing the block; and a loop filter processing unit 250, which is configured to determine whether deblocking filtering is suitable for each block boundary, and determine not to apply deblocking filtering to the sub-block boundary of the block after PROF processing is applied.

Description

Image decoding device, image decoding method and program

[Technical field]

The present invention relates to an image decoding device, an image decoding method and a program.

[Background Technology]

Conventionally, PROF (Prediction Refinement with Optical Flow) technology and deblocking filtering technology are known (for example, refer to Non-Patent Document 1).

PROF is a technology that uses a motion model corresponding to parallel translation and rotation in units of 4×4 pixel blocks to generate a prediction signal with high accuracy. Compared with when PROF processing is not applied, improvements in both subjective and objective image quality can be expected.

The deblocking filter is a low-pass filter applied to block boundaries, and can be expected to suppress block noise and improve subjective image quality.

[Prior art literature]

Non-patent literature

Non-patent document 1: Versatile Video Coding (Draft 6), JVET-N1001

[Summary of the invention]

Problems to be solved by the invention

In the prior art, it is possible to apply a deblocking filter to the block boundary where PROF processing is applied. However, there is a problem that the subjective image quality of a block that has been sufficiently improved by PROF processing may be degraded by applying a deblocking filter.

Therefore, the present invention is completed in view of the above-mentioned problems, and its purpose is to provide an image decoding device, an image decoding method and a program, which can prevent subjective image quality degradation by not applying deblocking filtering to sub-block boundaries after PROF processing.

Means for solving problems

The first feature of the present invention is a picture decoding device, comprising: an affine prediction signal generating unit, configured to determine whether PROF processing is suitable for each block, and when the PROF processing is applied, the PROF processing is performed on each sub-block obtained by dividing the block; and a loop filter processing unit, configured to determine whether deblocking filtering is suitable for each block boundary, and determine not to apply deblocking filtering to the sub-block boundary of the block after the PROF processing is applied.

The second feature of the present invention is a method for decoding an image, comprising the following steps: determining whether PROF processing is suitable for each block, and when applying the PROF processing, performing the PROF processing on each sub-block obtained by dividing the block; and determining whether deblocking filtering is suitable for each block boundary, and determining not to apply deblocking filtering to the sub-block boundary of the block after the PROF processing is applied.

The third feature of the present invention is a program used in an image decoding device, which enables a computer to execute the following steps: determining whether PROF processing is suitable for each block, and when applying the PROF processing, performing the PROF processing on each sub-block obtained by dividing the block; and determining whether deblocking filtering is suitable for each block boundary, and determining not to apply deblocking filtering to the sub-block boundary of the block after the PROF processing is applied.

Effects of the Invention

According to the present invention, it is possible to provide an image decoding device, an image decoding method, and a program capable of preventing subjective image quality degradation by not applying a deblocking filter to a sub-block boundary after PROF processing is applied.

[Brief Description of the Drawings]

FIG. 1 is a diagram showing an example of a configuration of an image processing system 10 according to an embodiment.

FIG. 2 is a diagram showing an example of a functional block of the image encoding device 100 according to an embodiment.

FIG. 3 is a diagram showing an example of the structure of coded data (bit stream) output from the coding unit 140 of the image coding device 100 according to one embodiment.

FIG. 4 is a diagram showing an example of control data included in the SPS 141 shown in FIG. 3 .

FIG. 5 shows an example of control data included in the slice header 143A shown in FIG. 3 .

FIG. 6 is a diagram showing an example of a functional block of the inter prediction unit 111 of the image encoding device 100 according to an embodiment.

FIG. 7 is a flowchart showing an example of a processing procedure of the refinement unit 111C of the inter prediction unit 111 of the image encoding device 100 according to one embodiment.

FIG. 8 is a flowchart showing an example of a processing procedure of the prediction signal generating unit 111D of the inter prediction unit 111 of the image encoding device 100 according to one embodiment.

FIG. 9 is a flowchart showing an example of a processing procedure of the affine motion vector calculation unit 111E of the inter prediction unit 111 of the image encoding device 100 according to one embodiment.

FIG. 10 is a diagram for explaining an example of a processing procedure of the affine motion vector calculation unit 111E of the inter prediction unit 111 of the image encoding device 100 according to one embodiment.

FIG. 11 is a flowchart showing an example of a processing procedure of the affine prediction signal generating unit 111F of the inter prediction unit 111 of the image encoding device 100 according to one embodiment.

FIG. 12 is a diagram showing an example of a functional block of the loop filter processing unit 150 of the image encoding device 100 according to an embodiment.

FIG. 13 is a diagram for explaining the processing procedure of the boundary strength determination unit 153 ( 153A/ 153B) of the loop filter processing unit 150 of the image encoding device 100 according to one embodiment.

FIG. 14 is a diagram showing an example of a functional block of an image decoding device 200 according to an embodiment.

FIG. 15 is a diagram showing an example of a functional block of the inter prediction unit 241 of the image decoding device 200 according to an embodiment.

FIG. 16 is a diagram showing an example of a functional block of the loop filter processing unit 250 of the image decoding device 200 according to an embodiment.

FIG. 17 is a diagram for explaining Modification Example 1. FIG.

FIG. 18 is a diagram for explaining Modification Example 1. FIG.

FIG. 19 is a diagram for explaining Modification Example 2. FIG.

FIG. 20 is a diagram for explaining Modification Example 2. FIG.

[Specific implementation method]

The following is an explanation of the embodiments of the present invention with reference to the accompanying drawings. In addition, the constituent elements in the following embodiments may be appropriately replaced with existing constituent elements, etc., and various modifications including combinations with other existing constituent elements may be performed. Therefore, the content of the invention recorded in the claims should not be limited by the description of the following embodiments.

(First Embodiment)

Next, an image processing system 10 according to a first embodiment of the present invention will be described with reference to Fig. 1 to Fig. 16. Fig. 1 is a diagram showing an image processing system 10 according to the present embodiment.

As shown in FIG. 1 , the image processing system 10 includes an image encoding device 100 and an image decoding device 200 .

The image encoding device 100 is configured to generate encoded data by encoding an input image signal, and the image decoding device 200 is configured to generate an output image signal by decoding the encoded data.

Here, the coded data may be transmitted from the image coding apparatus 100 to the image decoding apparatus 200 via a transmission path. Alternatively, the coded data may be provided from the image coding apparatus 100 to the image decoding apparatus 200 after being stored in a storage medium.

(Image Coding Device 100)

Next, the image encoding device 100 according to the present embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to the present embodiment.

As shown in FIG. 2 , the image encoding device 100 includes an inter-frame prediction unit 111 , an intra-frame prediction unit 112 , a subtractor 121 , an adder 122 , a transform and quantization unit 131 , an inverse transform and inverse quantization unit 132 , an encoding unit 140 , a loop filter processing unit 150 , and a frame buffer 160 .

The inter-frame prediction unit 111 is configured to generate a prediction signal by inter-frame prediction.

Specifically, the inter-frame prediction unit 111 is configured to specify a reference block included in the reference frame by comparing a frame to be encoded (hereinafter referred to as a target frame) with a reference frame stored in the frame buffer 160 , and determine a motion vector of the specified reference block.

The inter prediction unit 111 is configured to generate a prediction signal included in each prediction block based on the reference block and the motion vector. The inter prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

The intra prediction unit 112 is configured to generate a prediction signal by intraframe prediction.

Specifically, the intra prediction unit 112 is configured to specify a reference block included in the target frame, and to generate a prediction signal for each prediction block based on the specified reference block. Furthermore, the intra prediction unit 112 is configured to output the prediction signal to the subtractor 121 and the adder 122 .

Here, the reference block is a block that is referenced by a block to be predicted (hereinafter referred to as a target block). For example, the reference block is a block adjacent to the target block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output a prediction residual signal to the transform and quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal which is a difference between a prediction signal generated by intra prediction or inter prediction and the input image signal.

The adder 122 is configured to add the prediction signal and the prediction residual signal output from the inverse transform and inverse quantization unit 132 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 112 and the loop filter processing unit 150 .

Here, the decoded signal before filtering constitutes a reference block used in the intra prediction unit 112 .

The transform and quantization unit 131 is configured to perform a transform process on the prediction residual signal and obtain a coefficient level value. In addition, the transform and quantization unit 131 may be configured to perform quantization on the coefficient level value.

Here, the transformation process is a process of transforming the prediction residual signal into a frequency component signal. In this transformation process, a base model (transformation matrix) corresponding to a discrete cosine transform (DCT) or a base model (transformation matrix) corresponding to a discrete sine transform (DST) can be used.

The inverse transform and inverse quantization unit 132 is configured to perform an inverse transform process on the coefficient level value output from the transform and quantization unit 131. Here, the inverse transform and inverse quantization unit 132 may be configured to perform an inverse quantization on the coefficient level value before the inverse transform process.

Here, the inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131 .

The encoding unit 140 is configured to encode the coefficient level value output from the transform and quantization unit 131 and output the encoded data.

Here, for example, the encoding is entropy encoding that assigns codes of different lengths based on the probability of occurrence of coefficient level values.

Furthermore, the encoding unit 140 is configured to encode not only the coefficient level value but also the control data used in the decoding process.

Here, the control data may include size data such as the coding block (CU, Coding Unit) size, the prediction block (PU, Prediction Unit) size, and the transform block (TU, Transform Unit) size.

The loop filter processing unit 150 is configured to perform filtering on the pre-filtering decoded signal output from the adder 122 , and output the post-filtering decoded signal to the frame buffer 160 .

Here, the filtering process is, for example, a deblocking filtering process that reduces distortion generated at a boundary portion of a block (coding block, prediction block, or transform block).

The frame buffer 160 is configured to store reference frames used by the inter-frame prediction unit 111 .

Here, the decoded signal after the filtering process constitutes a reference frame used in the inter-frame prediction unit 111.

(Encoding unit 140)

Next, the control data encoded by the encoding unit 140 will be described using Fig. 3 to Fig. 5. Fig. 3 shows an example of the structure of the encoded data (bit stream) output from the encoding unit 140.

First, the bitstream may include an SPS (Sequence Parameter Set) 141 at the beginning. The SPS 141 is a set of control data in units of a sequence (a set of pictures). A specific example will be described below. Each SPS 141 includes at least id information for identifying each SPS 141 when there are multiple SPSs 141.

The bitstream may also include a PPS (Picture Parameter Set) 142 after the SPS 141. The PPS 142 is a set of control data in units of pictures (a set of slices). Each PPS 142 includes at least id information for identifying each SPS 141 when there are multiple SPSs 141. In addition, each PPS 142 includes at least SPS id information for specifying the SPS 141 corresponding to each PPS 142.

The bit stream may also include a slice header 143A after the PPS 142. The slice header 143A is a collection of control data in units of slices. A specific example will be described below. Each slice header 143A includes at least PPS id information for specifying the PPS 142 corresponding to each slice header 143A.

The bitstream may also include slice data 144A after the slice header 143A. The slice data 144A may include the coefficient level value and size data mentioned above.

As described above, the structure is as follows: each of the slice headers 143A/143B, PPS 142, and SPS 141 corresponds to each slice data 144A/144B. As described above, in the slice header 143A/143B, the PPS id information specifies which PPS 142 is referenced, and in the PPS 142, the SPS id information further specifies which SPS 141 is referenced, so that the common SPS 141 and PPS 142 can be used for a plurality of slice data 144A/144B.

In other words, it is not necessary to transmit the SPS 141 and the PPS 142 for each slice data 144A/144B. For example, as shown in FIG3 , the stream structure may be such that the SPS 141 and the PPS 142 are not encoded before the slice header 143B and the slice data 144B.

In addition, the structure of FIG3 is only an example. If the structure is a structure in which the control data specified by the slice header 143A/143B, PPS 142, and SPS 141 correspond to each slice data 144A/144B, elements other than the above may be added as stream components. Also, similarly, the structure may be shaped into a different structure than that of FIG3 during transmission.

FIG. 4 is a diagram showing an example of control data included in the SPS 141 .

As described above, the SPS 141 includes at least id information (sps_seq_parameter_set_id) for identifying each SPS 141 .

SPS 141 may also include a flag "sps_bdof_enabled_flag" for controlling whether the following BDOF (Bi-Directional Optical Flow) processing can be used (valid or invalid). When the value of this flag is "0", it indicates that the BDOF processing is not used in the slice corresponding to the SPS 141. On the other hand, when the value of this flag is "1", it indicates that the BDOF processing can be used in the slice corresponding to the SPS 141. In the following processing, it is determined for each block whether the BDOF processing is actually used in each block in the slice.

That is, "sps_bdof_enabled_flag" is a flag (fourth flag) included in the SPS 141 and indicating whether or not the BDOF process can be used in units of sequences.

SPS 141 may also include a flag "sps_dmvr_enabled_flag" for controlling whether the following DMVR (Decoder-side Motion Vector Refinement) processing can be used (valid or invalid). When the value of this flag is "0", it indicates that the DMVR processing is not used in the slice corresponding to the SPS 141. On the other hand, when the value of this flag is "1", it indicates that the DMVR processing can be used in the slice corresponding to the SPS 141. In the following processing, it is determined for each block whether the DMVR processing is actually used in each block in the slice.

That is, "sps_dmvr_enabled_flag" is a flag (third flag) included in the SPS 141 and indicating whether or not the DMVR process can be used in units of sequences.

The SPS 141 may further include a flag "sps_affine_enabled_flag" for controlling whether the affine motion compensation described below can be used (valid or invalid). When the value of this flag is "0", it indicates that affine motion compensation is not used in the slice corresponding to the SPS 141. On the other hand, when the value of this flag is "1", it indicates that affine motion compensation can be used in the slice corresponding to the SPS 141. In the following processing, it is determined for each block whether affine motion compensation is actually used in each block in the slice.

That is, "sps_affine_prof_enabled_flag" is a flag (fifth flag) included in the SPS 141 and indicating whether or not PROF processing can be used in units of sequences.

When "sps_affine_enabled_flag" is "1", SPS 141 may further include a flag "sps_affine_type_flag". When the value of this flag is "0", it indicates that the number of parameters for the affine motion compensation described below is always set to "4" in the slice corresponding to this SPS 141. On the other hand, when the value of this flag is "1", it indicates that when affine motion compensation is performed in the slice corresponding to this SPS 141, either "4" or "6" can be selected as the number of parameters for each block.

When the value of "sps_affine_enabled_flag" is "1", the SPS 141 may further include a flag "sps_affine_prof_enabled_flag". When the value of this flag is "0", it indicates that the PROF (Prediction Refinement with Optical Flow) processing described below is not used in the slice corresponding to the SPS 141. On the other hand, when the value of this flag is "1", it indicates that the PROF processing can be used in the slice corresponding to the SPS 141. In the following processing, it is determined for each block whether the PROF processing is actually used in each block in the slice.

When the value of at least any one of the flags "sps_bdof_enabled_flag", "sps_dmvr_enabled_flag" and "sps_affine_prof_enabled_flag" is "1", the SPS 141 may further include a flag "sps_bdof_dmvr_prof_slice_present_flag". When the flag is "1", it indicates that the following flag "slice_disable_bdof_dmvr_prof_flag" is included in the slice header 143A/143B corresponding to the SPS 141. When the value of the flag is "0", it indicates that the slice header 143A/143B corresponding to the SPS 141 does not include "slice_disable_bdof_dmvr_prof_flag".

That is, "sps_bdof_dmvr_prof_slice_present_flag" is a flag (second flag) indicating whether "slice_disable_bdof_dmvr_prof_flag" is included in the slice header 143A/143B. When this flag does not exist, it can be implicitly regarded that the value of this flag is "0".

FIG. 5 is a diagram showing an example of control data included in the slice header 143A/ 143B.

As described above, the slice header 143A/143B includes at least PPS id information "slice_pic_parameter_set_id" for specifying the PPS 142 corresponding to the slice. As described above, the SPS 141 referenced in the PPS specified by the PPS id information is the SPS 141 corresponding to the slice header 143A/143B.

When the SPS 141 corresponding to the slice header 143A/143B includes "sps_bdof_dmvr_prof_slice_present_flag" and the value of the flag is "1", the slice header 143A/143B may also include "slice_disable_bdof_dmvr_prof_flag". When the value of the flag is "1", as described below, it can be controlled so that BDOF processing, DMVR processing, and PROF processing are not used in each block included in the slice. When the value of the flag is "0", it can be controlled so that BDOF processing, DMVR processing, and PROF processing can be used in each block included in the slice.

That is, "slice_disable_bdof_dmvr_prof_flag" is a flag (first flag) included in the slice header 143A/143B for collectively controlling whether or not the DMVR process, the BDOF process, and the PROF process can be used.

In other words, "slice_disable_bdof_dmvr_prof_flag" is a flag included in the slice header 143A/143B to control whether PROF processing can be used, a flag included in the slice header 143A/143B to control whether DMVR processing can be used, and a flag included in the slice header 143A/143B to control whether BDOF processing can be used.

The flag values described above are merely examples, and when the meanings of the values ("0" and "1") assigned to the flags are reversed, equivalent processing can be achieved by reversing the corresponding processing.

Furthermore, as described above, by setting a flag to control not to use BDOF processing, DMVR processing, and PROF processing in units of slices, the image encoding device 100 side can explicitly control not to use this function. Therefore, by not using this function, the processing load and power consumption of the image encoding device 100 and the corresponding image decoding device 200 can be reduced.

(Inter-frame prediction unit 111)

Next, the inter prediction unit 111 of the image encoding device 100 according to the present embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding device 100 according to the present embodiment.

As shown in FIG. 6 , the inter prediction unit 111 includes a motion vector search unit 111A, a motion vector encoding unit 111B, a refinement unit 111C, a prediction signal generation unit 111D, an affine motion vector calculation unit 111E, and an affine prediction signal generation unit 111F.

The inter prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector search unit 111A is configured to specify a reference block included in the reference frame by comparing the target frame with the reference frame, and to search for a motion vector of the specified reference block.

Furthermore, the above search is performed on multiple candidate reference frames to determine the reference frame and motion vector used for prediction in the prediction block. Up to two reference frames and motion vectors can be used for one block. The case where only one set of reference frames and motion vectors is used for one block is called unidirectional prediction, and the case where two sets of reference frames and motion vectors are used is called bidirectional prediction. In the following, the first set is called L0 and the second set is called L1.

The motion vector search unit 111A is configured to determine a coding method for a reference frame and a motion vector. In addition to a conventional method for transmitting information of a reference frame and a motion vector separately, there are the following merge mode and affine motion compensation.

In addition, regarding the method of searching for a motion vector, the method of determining a reference frame, and the method of determining the encoding method of a reference frame and a motion vector, known methods can be used, and therefore, detailed descriptions thereof are omitted.

The motion vector coding unit 111B is configured to code the reference frame and motion vector information determined by the motion vector search unit 111A using the coding method determined by the motion vector search unit 111A. A known method can be used for coding the reference frame and motion vector information, and thus a detailed description thereof will be omitted.

When the encoding method of the block is the merge mode, a merge list for the block is first created on the side of the image encoding device 100. Here, the merge list is a list that lists a plurality of combinations of reference frames and motion vectors.

An index is assigned to each combination, and the image encoding device 100 encodes only the index instead of encoding the reference frame and motion vector information separately, and transmits it to the image decoding device 200. By sharing the creation method of the merge list in advance on the image encoding device 100 side and the image decoding device 200 side, the image decoding device 200 side can decode the reference frame and motion vector information based on only the index information.

In addition, regarding the method of creating the merge list, a known method can be used, and therefore, a detailed description thereof is omitted.

Affine motion compensation is a method that transmits a small amount of parameters for each block and derives a motion vector for each sub-block obtained by dividing the block based on the parameters and a predetermined model. The details of the method for deriving the motion vector will be described below.

In the case of affine motion compensation, the motion vector encoding unit 111B is configured to encode the control point motion vector information and the reference frame information described below. As described below, the control point motion vector transmits two or three pieces of motion vector information for each block.

Furthermore, when the block is a block for bidirectional prediction, two or three pieces of information on control point motion vectors are transmitted for L0 and L1, respectively. As for the specific encoding method, a known method can be used, and therefore, its detailed description is omitted.

When affine motion compensation is used for the block, the process proceeds to the process of the affine motion vector calculation unit 111E shown in FIG. 6 . Otherwise, the process proceeds to the process of the thinning unit 111C.

The thinning unit 111C is configured to perform a thinning process (for example, DMVR) for correcting the motion vector encoded by the motion vector encoding unit 111B.

Specifically, the refinement unit 111C is configured to perform the following refinement processing: set a search range based on the reference position specified by the motion vector encoded by the motion vector encoding unit 111B, specify a correction reference position with the smallest specific cost from the search range, and correct the motion vector based on the correction reference position.

FIG. 7 is a flowchart showing an example of the processing procedure of the refinement unit 111C.

As shown in Fig. 7, in step S71, the refinement unit 111C determines whether specific conditions for applying refinement processing are satisfied. When all the specific conditions are satisfied, the process proceeds to step S72. On the other hand, when any of the specific conditions is not satisfied, the process proceeds to step S75 and the refinement processing ends.

Here, the specific condition includes the condition that the block is a block to be bidirectionally predicted.

Furthermore, the specific conditions may also include the following conditions: either the reference frame on the L0 side or the reference frame on the L1 side is a past frame in time relative to the frame, the other reference frame is a subsequent frame in time relative to the frame, and the time distances of the L0 side and the L1 side from the frame (for example, the absolute value of the difference in POC (Picture Order Count)) are equal.

Furthermore, the specific condition may also include a condition that the motion vector is encoded in the merge mode.

Furthermore, the specific condition may also include the following condition: in the SPS 141 of the sequence to which the block belongs, the value of "sps_dmvr_enabled_flag" indicates that DMVR can be used in the sequence (for example, the value of the flag is "1").

Furthermore, the specific condition may include the following condition: in the slice header 143A/143B of the slice to which the block belongs, the value of "slice_disable_bdof_dmvr_prof_flag" indicates that DMVR can be used in the slice (for example, the value of the flag is "0").

In step S72 , the thinning unit 111C generates a search image based on the motion vector encoded by the motion vector encoding unit 111B and the information of the reference frame.

Here, when the motion vector points to a non-integer pixel position, the thinning section 111C applies a filter to interpolate the pixel at the non-integer pixel position to the pixel value of the reference frame. At this time, the thinning section 111C can reduce the amount of calculation by using an interpolation filter having a smaller number of taps than the interpolation filter used in the prediction signal generation section 111D described below. For example, the thinning section 111C can interpolate the pixel value at the non-integer pixel position by bilinear interpolation.

In step S73, the thinning unit 111C searches with integer pixel accuracy using the search image generated in step S72. Here, integer pixel accuracy means searching only points that are integer pixel apart based on the motion vector encoded by the motion vector encoding unit 111B.

The thinning section 111C determines the corrected motion vector at the integer pixel interval position by searching in step S72. As a search method, a known method can be used.

For example, the refinement unit 111C may perform a search by searching only points corresponding to a combination obtained by inverting the signs of the differential motion vectors on the L0 side and the L1 side. During the search, the refinement unit 111C may calculate a search cost at each search point and correct the motion vector at the search point with the minimum search cost.

Here, the refinement unit 111C may calculate the search cost so as to reduce the search cost only at the search point corresponding to the motion vector encoded by the motion vector encoding unit 111B. For example, the refinement unit 111C may set a value obtained by reducing the search cost calculated by the same method as other search points to 3/4 as the final search cost of the search point.

The search result in step S73 may also be the same value as the motion vector before the search.

In step S74, the thinning unit 111C uses the corrected motion vector with integer pixel accuracy determined in step S73 as an initial value and searches for a motion vector with non-integer pixel accuracy. As a method for searching a motion vector, a known method can be used.

Furthermore, the refinement unit 111C may use the result of step S73 as input and determine a vector with non-integer pixel accuracy using a parametric model such as parabola fitting without actually performing a search.

After determining the corrected motion vector of non-integer pixel accuracy in step S74, the refinement unit 111C moves to step S45 to end the refinement process. Here, for convenience, the expression of the corrected motion vector of non-integer pixel accuracy is used, but depending on the search result of step S74, the result may be the same value as the motion vector of integer pixel accuracy obtained in step S73.

The thinning unit 111C may also divide a block larger than a predetermined threshold into smaller sub-blocks and perform thinning processing on each sub-block. For example, the thinning unit 111C may set the execution unit of the thinning processing to 16×16 pixels in advance, and when the horizontal or vertical size of the block is larger than 16 pixels, it is divided into smaller blocks of 16 pixels. In this case, for all sub-blocks in the same block, the motion vector of the block encoded by the motion vector encoding unit 111B is used as the reference motion vector for the thinning processing.

When processing each sub-block, the refinement unit 111C may perform all the steps of FIG7 on each sub-block. In addition, the refinement unit 111C may perform only part of the processing of FIG7 on each sub-block. Specifically, the refinement unit 111C may perform the processing of steps S71 and S72 of FIG7 on each block, and perform the processing of steps S73 and S74 on each sub-block.

The prediction signal generation unit 111D is configured to generate a prediction signal based on the corrected motion vector output from the refinement unit 111C.

Here, as described below, the prediction signal generation unit 111D is configured to determine whether to perform the BDOF process on each block based on the information (for example, the search cost) calculated in the above-mentioned thinning process.

Specifically, when the motion vector is not corrected, the prediction signal generator 111D is configured to generate a prediction signal based on the motion vector encoded by the motion vector encoder 111B. On the other hand, when the motion vector is corrected, the prediction signal generator 111D is configured to generate a prediction signal based on the motion vector corrected by the refinement unit 111C.

8 is a flowchart showing an example of the processing sequence of the prediction signal generating unit 111D. Here, when the refinement unit 111C performs refinement processing in units of sub-blocks, the processing of the prediction signal generating unit 111D is also performed in units of sub-blocks. In this case, the term "block" in the following description can be appropriately replaced with "sub-block".

As shown in FIG. 8 , in step S81 , the prediction signal generation unit 111D generates a prediction signal.

Specifically, the prediction signal generation unit 111D receives the motion vector encoded by the motion vector encoding unit 111B or the motion vector encoded by the thinning unit 111C as input, and when the position indicated by the motion vector is a non-integer pixel position, applies a filter to interpolate the pixel at the non-integer pixel position to the pixel value of the reference frame. Here, as for the specific filter, a horizontally and vertically separable filter with a maximum of 8 taps disclosed in Non-Patent Document 1 can be applied.

When the block is a bidirectionally predicted block, the prediction signal generator 111D generates both a prediction signal based on a first set (hereinafter referred to as L0) of reference frames and motion vectors and a prediction signal based on a second set (hereinafter referred to as L1) of reference frames and motion vectors.

In step S82 , the prediction signal generation unit 111D checks for each block whether or not the application conditions of the BDOF process described below are satisfied.

The application condition includes at least the condition that the block is a block to be bidirectionally predicted.

Furthermore, the application condition may also include the following condition: in the integer pixel position search performed by the refinement unit 111C in step S73 , the search cost of the search point with the minimum search cost is greater than or equal to a predetermined threshold.

Furthermore, the specific condition may also include the following condition: in the SPS 141 of the sequence to which the block belongs, the value of "sps_bdof_enabled_flag" indicates that BDOF can be used in the sequence (for example, the value of the flag is "1").

Furthermore, the application condition may also include the following condition: in the slice header 143A/143B of the slice to which the block belongs, the value of "slice_disable_bdof_dmvr_prof_flag" indicates that BDOF processing can be used in the slice (for example, the value of the flag is "0").

Furthermore, the application conditions may also include the following conditions: either the reference frame on the L0 side or the reference frame on the L1 side is a past frame in time relative to the frame, the other reference frame is a subsequent frame in time relative to the frame, and the ratio of the time distances of the L0 side and the L1 side from the frame (for example, the absolute value of the difference in POC) is a predetermined ratio.

Here, as a method of determining the ratio, for example, the ratio may be determined in a manner proportional to the inverse of a weight coefficient when performing calculations using gradients or brightness values on the L0 side and the L1 side in the BDOF process described below.

For example, when the weight coefficient on the L0 side is set to "2/3" and the weight coefficient on the L1 side is set to "1/3", as the ratio of the distance to the reference frame, if the distance between L0 and the frame is set to "1", then the distance between L1 and the frame is "2". Only in this case is BDOF processing applied.

Similarly, for example, when the weight coefficient on the L0 side is set to "1/2" and the weight coefficient on the L1 side is set to "1/2", as a ratio of the distance between L0 and the frame, the distance between L1 and the frame is "1", that is, the distance between the frame and L0 is equal to the distance between the frame and L1. Only in this case is BDOF processing applied.

That is, the prediction signal generating unit 111D may also include the following condition as the application condition: the two reference frames (L0/L1) are related to the temporal distance between the frame and the frame in a manner that is proportional to the inverse of the weight coefficient when performing calculations using the respective pixel values of the two reference frames (L0/L1) or the respective values calculated based on the pixel values of the two reference frames (L0/L1).

Furthermore, the prediction signal generating unit 111D may include the following condition as the application condition: when the weight coefficients are equal, as a condition related to the temporal distance, the temporal distances between the two reference frames (L0/L1) and the frame are equal.

In addition, here, the case where the weight coefficient is "1/2" is used as an example, but when the same weight is used on the L0 side and the L1 side (for example, it can also be "1"), BDOF processing is applied only when the distance between the frame and L0 is equal to the distance between the frame and L1.

When it is determined that the application condition is met, the processing sequence moves to step S83, and when it is determined that the application condition is not met, the processing sequence moves to step S84.

In step S83, the prediction signal generation unit 111D performs BDOF processing and generates a prediction signal. The BDOF processing itself can use a known method, and therefore, only an overview is described, and a detailed description thereof is omitted.

First, the prediction signal generation unit 111D calculates the gradient of the brightness value in the vertical direction and the horizontal direction for each pixel in the reference block of the block. As a specific calculation method, for example, the method described in non-patent document 1 can be used for calculation. Here, the horizontal gradient of L0 is called "gradientHL0", the vertical gradient of L0 is called "gradientVL0", the horizontal gradient of L1 is called "gradientHL1", and the vertical gradient of L1 is called "gradientVL1".

Second, the prediction signal generation unit 111D calculates the correction MV (vx, vy) of the block. As a specific calculation method, for example, the method described in non-patent document 1 can be used for calculation. Here, the above-mentioned gradient sum (tempH, tempV) and the brightness value difference (diff) between the reference blocks L0 and L1 are calculated as follows, as described in non-patent document 1, and can also be used to calculate the above-mentioned (vx, vy).

diff＝predSampleL0×weightL0－predSampleL1×weightL1

tempH＝gradientHL0×weightL0+gradientHL1×weightL1

tempV＝gradientVL0×weightL0+gradientVL1×weightL1

Here, "predSampleL0" and "predSampleL1" are the brightness values of the reference blocks L0 and L1, respectively, and "weightL0" and "weightL1" are weight coefficients when calculating the above-mentioned gradient and brightness value difference, respectively.

Here, an example of using common weights is described using the three parameters "diff", "tempH" and "tempV" as examples. The same weights can be set when adding, subtracting, multiplying and dividing information on the L0 side and information on the L1 side.

Furthermore, the application conditions of the BDOF processing related to the distance from the reference frame may be set as described above in a manner proportional to the inverse of the weight. In other words, the weights for addition, subtraction, multiplication, and division may be set as described above in a manner proportional to the inverse of the ratio of the distance between the frame and each reference frame of L0 and L1.

By setting in this way, when assuming that the gradient or brightness value changes linearly between L0 and L1, an appropriate weight coefficient can be set for the time position of the frame. In addition, by setting the application conditions of the BDOF processing corresponding to the weight coefficient, it is possible to ensure that an appropriate weight coefficient is always used.

Thirdly, the prediction signal generation unit 111D uses the gradient and correction MV obtained as described above to calculate the brightness value correction amount "bdofOffset" of each pixel, for example, as follows.

bdofOffset＝vx×(weightL0×gradientHL0―weightL1×gradientHL1)+vy×(weightL0×gradientVL0―weightL1×gradientVL1)

Fourthly, the prediction signal generation unit 111D may calculate the prediction value of the corrected luminance value of each pixel using “bdofOffset” as described in Non-Patent Document 1, for example.

That is, the prediction signal generating unit 111D is configured to, in BDOF processing, use weight coefficients when performing calculations based on respective pixel values of two reference frames (L0/L1) or respective values calculated based on the pixel values of the two reference frames (L0/L1), wherein the above-mentioned application conditions include conditions related to the temporal distance between the two reference frames and the frame.

After the BDOF process is performed, the process sequence moves to step S85 and the process ends.

In step S84, when the block is a block for bidirectional prediction, the prediction signal generator 111D synthesizes the prediction signals of L0 and L1 generated in step S81 to generate a final prediction signal. As for the specific synthesis method, a known method can be used, so its detailed description is omitted. When the block is a block for unidirectional prediction, the prediction signal generator 111D directly uses the prediction signal generated in step S81 as the final prediction signal.

After generating the final prediction signal according to the above procedure, the prediction signal generating unit 111D moves to step S85 and ends the processing.

When the motion vector search unit 111A determines that affine motion compensation is to be performed on the block, the affine motion vector calculation unit 111E is configured to calculate a motion vector for each sub-block obtained by dividing the block based on the control point motion vector encoded by the motion vector encoding unit 111B.

FIG. 9 is a flowchart showing an example of the processing procedure of the affine motion vector calculation unit 111E.

As shown in FIG. 9 , in step S91 , the affine motion vector calculation unit 111E calculates values of four parameters, namely “dHorX”, “dhorY”, “dVerX”, and “dVerY”.

Here, the affine motion vector calculation unit 111E uses the value of the control point motion vector encoded by the motion vector encoding unit 111B when calculating the parameter.

As described above, each block has two or three control point motion vectors. Fig. 10 shows an example in which there are three control point motion vectors ("CPMV0", "CPMV1", "CPMV2").

Here, in FIG. 10 , the solid line indicates the boundary of the block, the dotted line indicates the boundary of the sub-blocks obtained by dividing the block, the solid arrow indicates the control point motion vector, and the dotted arrow indicates the motion vector of each sub-block.

When there are two control point motion vectors, there are only CPMV0 and CPMV1 in Figure 10. The specific method of calculating the values of the four parameters "dHorX", "dhorY", "dVerX" and "dVerY" according to the values of the control point motion vectors can use a known method, so its detailed description is omitted.

In step S92 , the affine motion vector calculation unit 111E determines whether to apply “fallbackMode” to the block.

Here, "fallbackMode" is a mode in which the same motion vector is used in all sub-blocks obtained by dividing the block. A known method can be used for determining "fallbackMode", so its detailed description is omitted. When "fallbackMode" is applied to the block, the processing sequence proceeds to step S93, and when "fallbackMode" is not applied to the block, the processing sequence proceeds to step S94.

In step S93 , the affine motion vector calculation unit 111E calculates the motion vector of each sub-block when “fallbackMode” is used.

As described above, when the mode is "fallbackMode", the motion vector values of all sub-blocks in the same block are the same. A known method can be used as a specific calculation method, and therefore, a detailed description thereof is omitted.

For example, in Non-Patent Document 1, the values of the above-mentioned "dHorX", "dHorY", "dVerX", and "dVerY" and the coordinate value of the center of the block when the coordinates of the upper left corner of the block are taken as the origin (values obtained by setting the height and width of the block to "1/2" respectively) are substituted into a predetermined model to calculate a motion vector. After step S93 is completed, this processing sequence moves to step S97 and ends the processing.

In step S94, the affine motion vector calculation unit 111E calculates the motion vector of each sub-block when the "fallbackMode" is not applied. A known method can be used as a specific calculation method, and therefore, a detailed description thereof is omitted.

For example, in non-patent document 1, the values of "dHorX", "dHorY", "dVerX", and "dVerY" and the coordinate values of the center of each sub-block when the coordinates of the upper left corner of the block are taken as the origin are substituted into a predetermined model to calculate a motion vector. After step S94, this processing sequence moves to step S95.

In step S95 , the affine motion vector calculation unit 111E determines whether or not the conditions for applying the PROF process to the block are satisfied.

The application condition may include a condition that the value of "sps_affine_prof_enabled_flag" is "1".

Furthermore, the application condition may include a condition that the value of "slice_disable_bdof_dmvr_prof_flag" indicates that PROF processing can be used in the slice (for example, the value of the flag is "0").

Furthermore, the application condition may include a condition that "fallbackMode" is not applied to the block.

When the affine motion vector calculation unit 111E determines that all application conditions are satisfied, it determines that the PROF process is applied to the block, and the process proceeds to step S96. Otherwise, it determines that the PROF process is not applied to the block, and the process proceeds to step S97 to end the process.

The above shows an example in which the affine motion vector calculation unit 111E determines whether all the application conditions are satisfied. However, the affine motion vector calculation unit 111E may determine by other logically equivalent methods.

Specifically, all the above-mentioned application conditions can also be reversed and defined as conditions for not applying PROF processing. When any situation meets the non-application conditions, the affine motion vector calculation unit 111E determines not to apply PROF processing to the block and proceeds to step S97. When none of the non-application conditions are met, it is determined to apply PROF processing to the block and proceeds to step S96.

The conditions for not applying PROF processing corresponding to the above example can be defined as follows.

The non-application condition may include a condition that the value of "sps_affine_prof_enabled_flag" is "0".

Furthermore, the non-application condition may include a condition that the value of "slice_disable_bdof_dmvr_prof_flag" indicates that the use of PROF processing in the slice is prohibited (for example, the value of the flag is "1").

Furthermore, the non-application condition may also include a condition that "fallbackMode" is applied to the block.

In step S96, the affine motion vector calculation unit 111E calculates the value of "diffMV" used in the PROF process. For example, the value of "diffMV" is calculated for each pixel position when the coordinates of the upper left corner of each sub-block are taken as the origin. In addition, when the size of each sub-block in the block is the same (for example, 4×4 pixels), the value of "diffMV" calculated for each pixel position of one sub-block can be used in other sub-blocks.

For example, "diffMV" can be calculated as follows.

diffMV[x][y][0]＝x×(dHorX＜＜2)+y×(dVerX＜＜2)-posOffsetX

diffMV[x][y][1]＝x×(dHorY＜＜2)+y×(dVerY＜＜2)－posOffsetYHere, x and y represent the positions of each pixel (x, y) with the coordinates of the upper left corner of each sub-block as the origin, [0] represents the x-direction component of "diffMV", and [1] represents the y-direction component of "diffMV".

In addition, the following relationship holds.

posOffsetX＝6×dHorX+6×dVerX

posOffsetY＝6×dHorY+6×dVerY

Thereafter, as described in Non-Patent Document 1, the affine motion vector calculation unit 111E performs right shift and clipping processing.

After calculating "diffMV", the processing sequence moves to step S97 and ends the processing.

The above description is about the processing of one reference frame in the block. For example, when it is a bidirectional prediction with two reference frames, the processing of steps S91 to S97 is performed on L0 and L1 respectively. Therefore, the judgment result of whether to apply PROF processing or the calculation result of "diffMV" may be different for L0 and L1 respectively.

FIG. 11 is a flowchart showing an example of processing of the affine prediction signal generating unit 111F.

The affine prediction signal generating unit 111F is configured to generate a prediction signal by performing PROF processing when "slice_disable_bdof_dmvr_prof_flag" indicates that PROF processing can be used in the slice corresponding to the slice header (when the value of the flag is "0"), and not performing PROF processing when it indicates that PROF processing cannot be used (when the value of the flag is "1").

Furthermore, the affine prediction signal generator 111F may be configured to implicitly interpret that, when the slice header does not include “slice_disable_bdof_dmvr_prof_flag”, the flag indicates that the flag can be used (the value of the flag is “0”) and perform processing.

In step S111 , the affine prediction signal generation unit 111F generates a prediction signal.

Specifically, the affine prediction signal generating unit 111F takes the motion vector of each sub-block calculated by the affine motion vector calculating unit 111E as input, and when the position indicated by the motion vector is a non-integer pixel position, applies a filter to interpolate the pixel at the non-integer pixel position to the pixel value of the reference frame. Here, as for the specific filter, a horizontally and vertically separable filter with a maximum of 6 taps disclosed in Non-Patent Document 1 can be applied.

When the block is a block to be bidirectionally predicted, the affine prediction signal generator 111F generates both prediction signals of L0 and L1.

In step S112, the affine prediction signal generation unit 111F determines whether PROF processing is applied to the prediction direction (L0 or L1) of the block. In this determination, the determination result in step S95 can be used directly.

When it is determined that the PROF process is applied to the prediction direction of the block, the process sequence proceeds to step S113. On the other hand, when it is determined that the PROF process is not applied to the prediction direction of the block, the process sequence proceeds to step S114.

In step S113, the affine prediction signal generation unit 111F performs PROF processing. The specific processing of the PROF processing can use a known method, so its detailed description is omitted. The affine prediction signal generation unit 111F corrects the prediction signal through the PROF processing.

In step S114, when the block is a block for bidirectional prediction, the affine prediction signal generator 111F synthesizes the prediction signals of L0 and L1 generated in step S111 or the prediction signal obtained by correcting the prediction signal in step S113 to generate a final prediction signal. The specific synthesis method can use a known method, so its detailed description is omitted. When the block is a block for unidirectional prediction, the affine prediction signal generator 111F directly uses the prediction signal generated in step S111 or the prediction signal obtained by correcting the prediction signal in step S113 as the final prediction signal.

After the generation of the final prediction signal is completed, the processing sequence moves to step S115 and ends the processing.

In the above example, for convenience, the prediction signal generation unit 111D and the affine prediction signal generation unit 111F are described as different processing blocks, but part of their processing may be shared or may be set as a single processing block.

For example, step S81 and step S111 , and step S84 and step S114 may be regarded as the same process.

In this case, if the determination in step S82 of FIG8 is "No", steps S112, S113 and S114 of FIG10 are executed in sequence, the flowcharts of FIG8 and FIG10 can be integrated and regarded as a single processing block. In this case, the BDOF process and the PROF process are exclusively applied in units of blocks.

(Loop Filter Processing Unit 150)

Next, the loop filter processing unit 150 of the present embodiment will be described. Fig. 12 is a diagram showing the loop filter processing unit 150 of the present embodiment.

As shown in FIG. 12 , the loop filter processing section 150 includes a block boundary detection section 151 , a boundary strength determination section 153 , a filter determination section 154 , and a filter processing section 155 .

Here, the configuration with "A" at the end is a configuration related to the deblocking filtering process at the block boundary in the vertical direction, and the configuration with "B" at the end is a configuration related to the deblocking filtering process at the block boundary in the horizontal direction.

The following describes an example of a case where a deblocking filtering process is performed on a block boundary in the horizontal direction after a deblocking filtering process is performed on a block boundary in the vertical direction.

As described above, the deblocking filter process can be applied to a coding block, a prediction block, or a transform block. Furthermore, the deblocking filter process can also be applied to sub-blocks obtained by dividing the above blocks. That is, the target block and the adjacent block can be a coding block, a prediction block, a transform block, or sub-blocks obtained by dividing them.

The definition of a sub-block includes the sub-block described as the processing unit of the refinement unit 111C and the prediction signal generation unit 111D. In addition, the definition of a sub-block includes the sub-block described as the processing unit of the affine motion vector calculation unit 111E and the affine prediction signal generation unit 111F. When deblocking filtering is applied to a sub-block, the block in the following description can be appropriately replaced with a sub-block.

Since the deblocking filtering process for the block boundary in the vertical direction is the same as the deblocking filtering process for the block boundary in the horizontal direction, the deblocking filtering process for the block boundary in the vertical direction will be described below.

The block boundary detection unit 151A is configured to detect the block boundary based on the control data indicating the block size. Here, the block is a coding block (CU), a prediction block (PU), or a transform block (TU). A known method can be applied to the specific detection method, so its detailed description is omitted.

The boundary strength determination unit 153A is configured to determine the boundary strength of the block boundary between the target block and the adjacent block.

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether the target block and the adjacent block are intra-prediction blocks.

For example, as shown in Figure 13, the boundary strength judgment unit 153A can also be configured to judge that the boundary strength of the block boundary is "2" when at least one of the object block and the adjacent blocks is an intra-frame prediction block (that is, when at least one of the blocks on both sides of the block boundary is an intra-frame prediction block).

The boundary strength determination unit 153A may also be configured to determine the boundary strength of a block boundary based on control data indicating whether a non-zero orthogonal transform coefficient is included in the object block and the adjacent blocks, and whether the block boundary is a boundary of a transform block.

For example, as shown in Figure 13, the boundary strength judgment unit 153A can also be configured to judge that the boundary strength of the block boundary is "1" when at least any one of the object block and the adjacent blocks contains a non-zero orthogonal transform coefficient and the block boundary is the boundary of the transform block (that is, at least any one of the blocks on both sides of the block boundary has a non-zero transform coefficient and it is the boundary of the TU).

The boundary strength determination unit 153A may also be configured to determine the boundary strength of the block boundary based on control data indicating whether the absolute value of the difference between the motion vectors of the target block and the adjacent block is greater than a threshold value (eg, 1/2 pixel).

For example, as shown in FIG. 13 , the boundary strength judgment unit 153A may be configured to judge that the boundary strength of the block boundary is “1” when the absolute value of the difference between the motion vectors of the object block and the adjacent block is greater than a threshold value (e.g., 1/2 pixel) (i.e., when the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is greater than a threshold value (e.g., 1/2 pixel)).

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether the reference blocks used for prediction of the motion vectors of the target block and the adjacent blocks are different.

For example, as shown in FIG. 13 , the boundary strength judgment unit 153A may be configured to judge the boundary strength of the block boundary as “1” when the reference blocks used for predicting the motion vectors of the object block and the adjacent block are different (i.e., the reference images in the blocks on both sides of the block boundary are different).

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether the number of motion vectors of the target block and the number of motion vectors of the adjacent blocks are different.

For example, as shown in FIG. 13 , the boundary strength judgment unit 153A may be configured to judge that the boundary strength of the block boundary is “1” when the number of motion vectors of the object block and the adjacent block is different (that is, when the number of motion vectors in the blocks on both sides of the block boundary is different).

For example, as shown in FIG. 13 , the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is “0” when none of the above conditions are satisfied.

In addition, it is likely that the larger the value of the boundary strength, the larger the block distortion generated at the block boundary.

For brightness signals and color difference signals, the above-mentioned boundary strength judgment method can be judged by a common method or by using partially different conditions.

The filter determination unit 154A is configured to determine the type of filter processing (for example, deblocking filter processing) to be applied to the block boundary.

For example, the filter determination unit 154A may be configured to determine whether to apply filtering to the block boundary, or whether to apply either weak filtering or strong filtering, based on the boundary strength of the block boundary, or quantization parameters contained in the target block and the adjacent blocks.

The filter determination unit 154A may be configured to determine that the filter process is not to be applied when the boundary strength of the block boundary is “0”.

The filter processing unit 155A is configured to process the image before deblocking based on the determination result of the filter determination unit 154A. The process performed on the image before deblocking is no filter processing, weak filter processing, strong filter processing, or the like.

(Image Decoding Device 200)

Next, the image decoding apparatus 200 according to the present embodiment will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of functional blocks of the image decoding apparatus 200 according to the present embodiment.

As shown in FIG. 14 , the image decoding apparatus 200 includes a decoding unit 210 , an inverse transform and inverse quantization unit 220 , an adder 230 , an inter prediction unit 241 , an intra prediction unit 242 , a loop filter processing unit 250 , and a frame buffer 260 .

The decoding unit 210 is configured to decode the encoded data generated by the image encoding device 100 and decode the coefficient level values.

Here, for example, decoding is entropy decoding in the reverse order to the entropy encoding performed by the encoding unit 140 .

Furthermore, the decoding unit 210 may be configured to acquire control data by decoding the encoded data.

In addition, as described above, the control data may include size data such as coding block size, prediction block size, and transform block size.

Furthermore, as described above, the control data may also include header information such as a sequence parameter set (SPS), a picture parameter set (PPS), and a slice header.

For example, the decoding unit 21 may be configured to decode, from the slice header, a flag for controlling whether the PROF process can be used, or a flag for collectively controlling whether the DMVR process, the BDOF process, and the PROF process can be used.

The inverse transform and inverse quantization unit 220 is configured to perform an inverse transform process on the coefficient level value output from the decoding unit 210. Here, the inverse transform and inverse quantization unit 220 may be configured to perform an inverse quantization on the coefficient level value before the inverse transform process.

Here, the inverse transform process and the inverse quantization are performed in the reverse order of the transform process and the quantization performed by the transform and quantization unit 131 .

The adder 230 is configured to add the prediction signal and the prediction residual signal output from the inverse transform and inverse quantization unit 220 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 242 and the loop filter processing unit 250 .

Here, the decoded signal before filtering constitutes a reference block used in the intra prediction unit 242 .

Similar to the inter-frame prediction unit 111 , the inter-frame prediction unit 241 is configured to generate a prediction signal by inter-frame prediction.

Specifically, the inter-frame prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from the coded data and a reference signal included in a reference frame. The inter-frame prediction unit 241 is configured to output the prediction signal to the adder 230 .

Similar to the intra prediction unit 112 , the intra prediction unit 242 is configured to generate a prediction signal by intra prediction (intraframe prediction).

Specifically, the intra prediction unit 242 is configured to designate a reference block included in the target frame, and to generate a prediction signal for each prediction block based on the designated reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230 .

Similar to the loop filter processing unit 150 , the loop filter processing unit 250 is configured to perform filtering on the pre-filtering decoded signal output from the adder 230 and output the post-filtering decoded signal to the frame buffer 260 .

Here, the filtering process is, for example, a deblocking filtering process for reducing distortion generated at a boundary portion of a block (a coding block, a prediction block, a transform block, or a sub-block obtained by dividing these blocks).

Similar to the frame buffer 160 , the frame buffer 260 is configured to store reference frames used by the inter-frame prediction unit 241 .

Here, the filtered decoded signal constitutes a reference frame used in the inter-frame prediction unit 241 .

(Inter-frame prediction unit 241)

Next, the inter-frame prediction unit 241 of the present embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of functional blocks of the inter-frame prediction unit 241 of the present embodiment.

As shown in FIG. 15 , the inter prediction unit 241 includes a motion vector decoding unit 241B, a refinement unit 241C, a prediction signal generation unit 241D, an affine motion vector calculation unit 241E, and an affine prediction signal generation unit 241F.

The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on a motion vector.

The motion vector decoding unit 241B is configured to obtain a motion vector by decoding the control data received from the image encoding device 100 .

Furthermore, the motion vector decoding unit 241B is configured to also decode information on whether affine motion compensation is used for the block.

Here, when affine motion compensation is not used for the block, the processing of the refinement processing unit 241C and the prediction signal generation unit 241D is performed, and when affine motion compensation is used for the block, the processing of the affine motion vector calculation unit 241E and the affine prediction signal generation unit 241F is performed.

Similar to the thinning unit 111C, the thinning unit 241C is configured to execute thinning processing for correcting the motion vector.

Similar to the prediction signal generating unit 111D, the prediction signal generating unit 241D is configured to generate a prediction signal based on a motion vector.

Similar to the affine motion vector calculation unit 111E, the affine motion vector calculation unit 241E is configured to calculate the motion vector of each sub-block using the control point motion vector decoded by the motion vector decoding unit 241B.

Similar to the affine prediction signal generating unit 111F, the affine prediction signal generating unit 241F generates a prediction signal using the motion vector of each sub-block calculated by the affine motion vector calculating unit 241E, and similarly to the prediction signal generating unit 111D and the affine prediction signal generating unit 111F, the prediction signal generating unit 241D and the affine prediction signal generating unit 241F can also be integrated into the same processing block.

(Loop Filter Processing Unit 250)

Next, the loop filter processing unit 250 of the present embodiment will be described. Fig. 16 is a diagram showing the loop filter processing unit 250 of the present embodiment.

As shown in FIG. 16 , the loop filter processing section 250 includes a block boundary detection section 251 , a boundary strength determination section 253 , a filter determination section 254 , and a filter processing section 255 .

Here, the configuration with "A" at the end is a configuration related to the deblocking filtering process at the block boundary in the vertical direction, and the configuration with "B" at the end is a configuration related to the deblocking filtering process at the block boundary in the horizontal direction.

Here, a case where the deblocking filtering process is performed on the block boundary in the horizontal direction after the deblocking filtering process is performed on the block boundary in the vertical direction is exemplified.

As described above, the deblocking filter process can be applied to a coding block, a prediction block, or a transform block. Furthermore, the deblocking filter process can also be applied to sub-blocks obtained by dividing the above blocks. That is, the target block and the adjacent block can be a coding block, a prediction block, a transform block, or sub-blocks obtained by dividing them.

Since the deblocking filtering process for the block boundary in the vertical direction is the same as the deblocking filtering process for the block boundary in the horizontal direction, the deblocking filtering process for the block boundary in the vertical direction will be described below.

Similar to the block boundary detection unit 151A, the block boundary detection unit 251A is configured to detect a block boundary based on control data indicating a block size.

The boundary strength determination unit 253A is configured to determine the boundary strength of the block boundary between the target block and the adjacent block, similarly to the boundary strength determination unit 153A. The method of determining the boundary strength of the block boundary is as described above.

Similar to the filter determination unit 154A, the filter determination unit 254A is configured to determine the type of deblocking filter processing to be applied to the block boundary. The method of determining the type of deblocking filter processing is as described above.

Similar to the filter processing unit 155A, the filter processing unit 255A is configured to process the image before deblocking based on the determination result of the filter determination unit 254A. The process performed on the image before deblocking is no filter processing, weak filter processing, strong filter processing, etc.

(Change Example 1)

Next, a modification example 1 of the above embodiment will be described with reference to Figures 17 and 18. Next, the differences from the above embodiment will be mainly described. In the above embodiment, detailed descriptions of BDOF processing and PROF processing were omitted, but in this modification example, an example of specific processing will be described.

First, a specific example of the BDOF process in step S83 will be described using FIG. 17 .

As shown in FIG. 17 , in step S171 , the prediction signal generator 111D/ 241D calculates the luminance value gradients in the horizontal and vertical directions for each pixel in the reference blocks L0 and L1 corresponding to the block or subblock to which the BDOF process is applied.

Specifically, the prediction signal generation unit 111D/241D calculates the above-mentioned gradient for each pixel position (x, y) in the block using the following equation.

gradientHL0[x][y]＝(predSamplesL0[hx+1][vy]>>shift1)-(predSamplesL0[hx-1][vy]>>shift1)

gradientV0[x][y]＝(predSamplesL0[hx][vy+1]＞shift1)-(predSamplesL0[hx][vy-1]＞＞shift1)

gradientHL1[x][y]＝(predSamplesL1[hx+1][vy]>>shift1)-(predSamplesL1[hx-1][vy]>>shift1)

gradientV1[x][y]＝(predSamplesL1[hx][vy+1]＞shift1)-(predSamplesL0[hx][vy-1]＞＞shift1)

Here, "predSamplesL0" and "predSamplesL1" are prediction signals of L0 and L1 calculated in step S81. Furthermore, "hx" and "vy" are defined by the following equations.

In the following formula, "nCbW" is the width of the block, "nCbH" is the height of the block, and "Clip3(min, max, input)" is a function that limits the value of "input" to "min≦input≦max".

hx＝Clip3(1，nCbW，x)

hy＝Clip3(1，nCbH，y)

In addition, "shift1" is defined by the following formula.

shift1=Max(6, bitDepth-6)

Here, "bitDepth" is the bit depth when processing pixel values.

In step S172, the prediction signal generation unit 111D/241D calculates the corrected MV.

Here, first, the prediction signal generation unit 111D/241D calculates the values of the following three parameters for each pixel of the block or sub-block.

diff[x][y]＝(predSamplesL0[hx][vy]>>shift2)-(predSamplesL1[hx][vy]>>shift2)

tempH[x][y]＝(gradientHL0[x][y]+gradientHL1[x][y])>>shift3

tempV[x][y]=(gradientVL0[x][y]+gradientVL1[x][y])>>shift3

Here, "shift2" and "shift3" are defined by the following formulas.

shift2Max(4, bitDepth-8)

shift3 = Max(3, 15-bitDepth)

Secondly, the prediction signal generation unit 111D/241D divides the block or sub-block into 4×4 pixel blocks, and calculates parameters for each 4×4 pixel block as described below.

[Mathematical formula 1]

sGx2＝∑ _i ∑ _j Abs(tempH[xSb+i][ySb+j])with i，j＝-1.4

sGy2＝∑ _i ∑ _j Abs(tempV[xSb+i][ySb+j])with i，j＝-1..4

sGxGy＝∑ _i ∑ _j (Sign(tempV[xSb+i][ySb+j])*tempH[xSb+i][ySb+j])with i，j＝-1..4

sGxGy _m = sGxGy>>12

sGxGy _s =sGxGy&((1＜＜12)-1)

sGxdI＝∑ _i ∑ _j (-Sign(tempH[xSb+i][ySb+j])*diff[xSb+i][ySb+j])with ij＝-1..4

sGydI＝∑ _i ∑ _j (-Sign(tempV[xSb+i][ySb+j])*diff[xSb+i][ySb+j])with ij＝-1..4

Here, "xSb" and "yXb" are the coordinates of the upper left pixel of each 4×4 pixel block when the upper left pixel of the block or sub-block is the origin. And, "Sign(input)" is a function that returns "1" when the value of "input" is positive, returns "-1" when the value of "input" is negative, and returns "0" when the value of "input" is "0".

As described below, the prediction signal generation unit 111D/241D calculates the corrected MV (vx, vy) using the values calculated in the above manner.

vx=sGx2>0? Clip3(-mvRefineThres, mvRefineThres-1, -(sGxdI＜＜3)＞＞Floor(Log2(sGx2))): 0

vy=sGx2>0? Clip3(-mvRefineThres, mvRefineThres-1, ((sGydI＜＜3)-((vx×sGxGy _m )＜＜12+vx×sGxGy _s )＞＞1)＞＞Floor(Log2(sGx2))): 0

Here, "mvRefineThres" is defined by the following formula.

mvRefineThres＝1＜＜Max(5, bitDepth－7)

By setting it as said structure, the bit width of each component of correction MV (vx, vy) can be suppressed to "1<<Max(5, bitDepth-7)+1" bits (including the sign).

In step S173 , the prediction signal generation unit 111D/ 241D corrects the prediction signal of each pixel as described below.

bdofOffset＝(vx×(gradientHL0[x+1][y+1]-gradientHL1[x+1][y+1]))>>1+(vy×(gradientVL0[x+1][y+1] -gradientVL1[x+1][y+1]))>>1

pbSamples[x][y]＝Clip3(0, (2 ^bitDepth )-1, (predSamplesL0[x+1][y+1]+offset4+predSamplesL1[x+1][y+1]+bdofOffset)>> shift4)

Here, "offset4" and "shift4" are defined as follows.

offset4＝1＜＜(shift4－1)

shift4＝Max(3,15-bitDepth)

After performing all the above processing, the prediction signal generation unit 111D/241D proceeds to step S174 and ends the processing.

Next, a specific example of the PROF process in step S113 will be described using FIG. 18 .

As shown in FIG. 18 , in step S181 , the affine prediction signal generation unit 111F/ 241F calculates the luminance value gradients in the horizontal and vertical directions for each pixel in the reference blocks L0 and L1 to which the PROF process is applied.

Specifically, the affine prediction signal generation unit 111F/241F calculates the gradient for each pixel position (x, y) within the block using the following formula.

gradientH[x][y]＝(predSamples[x+2][y]>>shift1)-(predSamples[x][y]>>shift1)

gradientV[x][y]＝(predSamples[x][y+2]>>shift1)-(predSamples[x][y]>>shift1)

Here, "predSamples" is the prediction signal of L0 or L1 generated in step S111. Here, it is assumed that shift1 = Max(6, bitDepth-6).

When the PROF process is applied to both L0 and L1, the affine prediction signal generation unit 111F/241F calculates the above-mentioned "gradientH" and "gradientV" for L0 and L1, respectively. At this time, the calculation content in step S181 is exactly the same as the calculation content in step S171 in terms of pixels.

In step S182, the affine prediction signal generating unit 111F/241F calculates the correction MV. This process is actually performed as step S96 inside the affine motion vector calculating unit 111E/241E, but for convenience, it is described as the process performed in step S182. In fact, the affine prediction signal generating unit 111F/241F can directly use the value calculated in step S96.

First, similarly to the description of step S96 above, the affine prediction signal generation unit 111F/241F calculates the value of "diffMv".

diffMv[x][y][0]＝x×(dHorX＜＜2)+y×(dVerX＜＜2)-posOffsetX

diffMv[x][y][1]＝x×(dHorY＜＜2)+y×(dVerY＜＜2)-posOffsetY

Second, the affine prediction signal generation unit 111F/241F right-shifts the above-mentioned value of "diffMv" by 7 bits.

Third, as described below, the affine prediction signal generation unit 111F/241F performs clipping processing.

diffMv[x][y][i]=Clip3(-dmvLimit, dmvLimit-1, diffMv[x][y][i])

Here,

dmvLimit＝1＜＜Max(5, bitDepth-7).

By adopting the above configuration, the bit width of each component of "diffMv" can be suppressed to within "1<<Max(5, bitDepth-7)+1" bits (including the sign). This bit width is the same as the correction MV calculated in step S172.

In step S183, the affine prediction signal generation unit 111F/241F corrects the prediction signal.

As described below, the affine prediction signal generation unit 111F/241F corrects the value of the prediction signal "pbSamples" for the block and prediction direction to which the PROF process is applied.

dI＝gradientH[x][y]×diffMv[x][y][0]+gradientV[x][y]×diffMv[x][y][1]

pbSamples[x][y]＝predSamples[x+1][y+1]+(dI>>1)

When the PROF process is applied to both L0 and L1, the affine prediction signal generation unit 111F/241F executes the above-described process on L0 and L1, respectively.

After completing the above correction, the affine prediction signal generation unit 111F/241F moves to step S184 and ends the PROF processing.

After the PROF process is completed, the affine prediction signal generation unit 111F/241F performs the prediction value synthesis process described in step S114. The prediction value synthesis can be performed, for example, by the following equation.

predSamples[x][y]＝Clip3(0, (1＜＜bitDepth)-1, (predSamplesL0[x][y]+predSamplesL1[x][y]+offset2)>>shift2)

Here,

offset2＝1＜＜(shift2－1)

shift2=Max(3,15-bitDepth).

In addition, when both the prediction signal "predSamplesL0" on the L0 side and the prediction signal "predSamplesL1" on the L1 side are corrected by further PROF processing, if the calculation of step S183 is substituted into "predSamplesL0" and "predSamplesL1", the following transformation can be performed.

dI_Bi＝(dI_L0>>1)+(dI_L1>>1)

pbSamples[x][y]＝Clips(0, (1＜＜bitDepth)-1, (predSamplesL0_orig[x][y]+offset2+predSamplesL1_orig[x][y]+dI_Bi)>>shift2)

Here, “dI_L0” and “dI_L1” are the values of dI of L0 and L1, respectively, and “predSamplesL0_orig” and “predSamplesL1_orig” are prediction signals before correction by PROF processing.

The values of the gradient and correction MV calculated in the BDOF process and PROF process described above are respectively clipped so as to have the same bit width, and therefore the bit widths of “bdofOffset” and “dI_Bi” calculated thereby are also the same.

That is, the calculation may be performed as follows: the offset calculated in the BDOF processing performed by the prediction signal generating unit 111D/241D and the offset calculated for the two reference frames (L0/L1) in the PROF processing performed by the affine prediction signal generating unit 111F/241F have the same bit width as the value synthesized.

Furthermore, the values of "offset2" and "shift2" when synthesizing the predicted signal after PROF processing are the same as the values of "offset4" and "shift4" in BDOF processing. Therefore, the prediction value generation process when applying BDOF processing and PROF processing to both L0 and L1 is exactly the same formula.

By configuring the BDOF process and the PROF process as described above, the following effects are achieved.

First, the gradient calculation processing of step S171 and step S181 can be made completely the same processing.

That is, when the PROF processing is applied to two reference frames (L0/L1), the gradient calculation processing in the BDOF processing performed by the prediction signal generating unit 111D/241D and the gradient calculation processing in the PROF processing performed by the affine prediction signal generating unit 111F/241F can be made the same processing.

Second, the bit widths of the corrected MVs, which are outputs of step S172 and step S182, can be made completely the same.

That is, it is possible to perform clipping processing so that the correction MV amount in the BDOF processing performed by the prediction signal generating unit 111D/241D and the correction MV amount used in the PROF processing performed by the affine prediction signal generating unit 111F/241F have the same bit width.

Third, the predicted value correction process of step S173, the predicted value correction process of step S183, and the predicted value synthesis process of step S114 can be made completely the same process.

That is, the final prediction signal generation process performed by the prediction signal generation unit 111D/241D using the offset and the final prediction signal generation process performed by the affine prediction signal generation unit 111F/241F using the value synthesized by the offset can be made the same process.

As described above, since the BDOF process and the PROF process are applied exclusively to each block, when the same processing portion is implemented in hardware, the processing circuit can be shared between the BDOF process and the PROF process, thereby reducing the circuit scale.

(Change Example 2)

Next, a second modification of the above embodiment will be described with reference to FIGS. 19 and 20. The following mainly describes the differences from the above embodiment. In this modification, a block boundary detection method in the block boundary detection unit 151/251 and a boundary strength determination method in the boundary strength determination unit 153/253 are described using examples different from those in the above embodiment.

The block boundary detection unit 151A will be described below, but the block boundary detection units 151B/251A/251B can also detect block boundaries through the same processing.

Fig. 19 is a flowchart showing an example of the processing of the block boundary detection unit 151A. Hereinafter, a case where the processing is performed in units of CUs will be described as an example.

In step S191, the block boundary detection unit 151A detects a CU boundary. Since the block boundary detection unit 151A is a processing block for detecting a vertical boundary, the vertical boundary on the left side of the CU is detected as a block boundary to be filtered.

In addition, "detected as a block boundary" is equivalent to, for example, preparing in advance a two-dimensional flag sequence (for example, "edgeFlags[x][y]") indicating whether the pixel position in the picture is a boundary, setting the flag value of the pixel position detected as the boundary to "1", and setting the flag value of the pixel position not detected as the boundary to "0".

Here, when the vertical boundary on the left side of the CU is located at a boundary of a coding unit such as a picture boundary, a slice boundary, or a tile boundary, the block boundary detection unit 151A may not detect the boundary as a block boundary.

After detecting the CU boundary in the above-described procedure, the block boundary detection unit 151A moves to step S192 .

In step S192, the block boundary detection unit 151A detects a TU boundary (transform block boundary).

The block boundary detection unit 151A starts from the vertical boundary on the left side of the CU, increases the horizontal coordinate by 4 pixels each time, and increases the vertical pixel position by 1 pixel each time, while checking whether each pixel position is a TU boundary. If the pixel position is a TU boundary, the block boundary detection unit 151A detects the pixel position as a block boundary.

After detecting the TU boundary in the above-described procedure, the block boundary detection unit 151A moves to step S193.

In step S193 , the block boundary detection unit 151A detects sub-block boundaries.

In step S193 , the sub-block boundary to be detected may be, for example, a sub-block boundary of a CU after inter-frame prediction.

At this time, the above-mentioned sub-block boundaries may include both the boundaries of sub-blocks (for example, 16×16 pixel size) in the DMVR processing performed by the refinement unit 111C/241C, and the boundaries of sub-blocks (for example, 4×4 pixel size) in the affine motion compensation performed by the affine motion vector calculation unit 111E/241E and the affine prediction signal generation unit 111F/241F.

In step S193 , the sub-block boundary to be detected may be, for example, a sub-block boundary of a CU after affine motion compensation or “merge_subblock” is applied.

In addition, when the PROF process is applied to the CU, the sub-blocks subjected to the affine motion compensation are consistent with the sub-blocks subjected to the PROF process.

Here, when the sub-block boundary of affine motion compensation is used as the detection target in step S193, when PROF processing is applied to the CU, the block boundary detection unit 151A may not use the sub-block boundary of affine motion compensation and the sub-block boundary of PROF processing as the detection target.

That is, the block boundary detection unit 151A may detect the sub-block boundary of affine motion compensation only when affine motion compensation is applied to the CU and PROF processing is not applied to the CU. Of course, for a CU to which PROF processing is applied, the block boundary detection unit 151A may detect the sub-block boundary of affine motion compensation and the sub-block boundary of PROF processing.

In addition, for the sub-block boundary of the block to be bidirectionally predicted, when the PROF process is applied to at least one of L0 and L1, the block boundary detection unit 151A may regard the block as a "CU to which the PROF process is applied". Furthermore, the block boundary detection unit 151A may regard the block as a "CU to which the PROF process is applied" only when the PROF process is applied to both L0 and L1. The same explanation can be applied in the following description.

The block boundary detection unit 151A takes the vertical boundary on the left side of the CU as the starting point, increases the horizontal coordinate by the sub-block size calculated by the following method, and increases the vertical pixel position by 1 pixel, while checking whether each pixel position is a sub-block boundary.

When the pixel position is a sub-block boundary, the block boundary detection unit 151A detects the pixel position as a block boundary. Here, even if the position is a sub-block boundary, when the PROF process is applied to the CU, the block boundary detection unit 151A may not detect it as a sub-block boundary.

For example, the above sub-block size can be calculated as follows.

Subblock size = Max(8, (nCbW/NumSbX))

Here, "nCbW" is the width of the CU (the number of pixels in the horizontal direction), and "NumSbX" is the number of sub-blocks included in the CU in the horizontal direction.

When the above formula is used, it is determined whether the sub-block boundary is a sub-block boundary starting from the vertical boundary on the left side of the CU and located at a minimum of every 8 pixels in the horizontal direction.

For example, the above sub-block size may also be calculated as follows.

Subblock size = Max(4, (nCbW/NumSbX))

At this time, it is determined whether the sub-block boundary is a sub-block boundary starting from the vertical boundary on the left side of the CU and located at a minimum of every 4 pixels in the horizontal direction.

That is, the block boundary detection unit 151A is configured to determine whether the target pixel position is a sub-block boundary at a minimum interval of 4 pixels, based on the coordinates of the left end or the top end of the target block.

After detecting the sub-block boundary in accordance with the above-described procedure, the block boundary detection unit 151A moves to step S194 and ends the processing.

The above description takes the example of simply detecting the block boundary without distinguishing the CU boundary, TU boundary, and sub-block boundary respectively, but they may also be distinguished and stored.

For example, the data may be stored as CU boundaries, TU boundaries, and sub-block boundaries, or as TU boundaries and motion vector boundaries. In this case, the CU boundary is detected as both the TU boundary and the motion vector boundary. Furthermore, the TU boundary is detected as the TU boundary, and the sub-block boundary is detected as the motion vector boundary. This distinction may be used, for example, in the boundary strength determination unit 153A described below.

The boundary strength judgment unit 153A is described below. Exactly the same judgment can also be performed in the boundary strength judgment unit 153B/253A/253B.

The boundary strength determination unit 153A may be configured to determine the boundary strength as “0” when the target pixel position is not a block boundary.

That is, the block boundary detection unit 151 can be configured to not detect the sub-blocks within the block to which PROF processing is applied as block boundaries, the boundary strength judgment unit 153A can be configured to judge the boundary strength of pixel positions that are not detected as block boundaries as "0", and the filter determination unit 154 can be configured to determine that deblocking filtering is not applied to the block boundary when the boundary strength is "0".

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether the target block and the adjacent block are intra-prediction blocks.

For example, as shown in Figure 20, the boundary strength judgment unit 153A can also be configured to judge that the boundary strength of the block boundary is "2" when at least one of the object block and the adjacent blocks is an intra-frame prediction block (that is, when at least one of the blocks on both sides of the block boundary is an intra-frame prediction block).

The boundary strength determination unit 153A may also be configured to determine the boundary strength of a block boundary based on control data indicating whether a non-zero orthogonal transform coefficient is included in the object block and the adjacent blocks, and whether the block boundary is a boundary of a transform block.

For example, as shown in Figure 20, the boundary strength judgment unit 153A can also be configured to judge that the boundary strength of the block boundary is "1" when at least any one of the object block and the adjacent blocks contains a non-zero orthogonal transform coefficient and the block boundary is the boundary of the transform block (that is, at least any one of the blocks on both sides of the block boundary has a non-zero transform coefficient and it is the boundary of the TU).

The boundary strength determination unit 153A may also be configured to determine the boundary strength of the block boundary based on control data indicating whether the absolute value of the difference between the motion vectors of the target block and the adjacent block is greater than a threshold value (e.g., 1/2 pixel). In this case, the boundary strength of the boundary may also be determined based on whether the block boundary is a sub-block boundary within the block to which the PROF process is applied.

For example, as shown in FIG. 20 , the boundary strength judgment unit 153A may be configured to judge the boundary strength of a block boundary as “1” when the absolute value of the difference between the motion vectors of the object block and the adjacent block is greater than a threshold value (e.g., 1/2 pixel) (i.e., the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is greater than a threshold value (e.g., 1/2 pixel)), and the block boundary is not a sub-block (4×4 pixel block) boundary of the block to which PROF processing is determined to be applied in step S95.

That is, the boundary strength judgment unit 153A may be configured to judge the boundary strength for each block boundary, and to judge the boundary strength as "1" when the absolute value of the difference in motion vectors of blocks on both sides of the block boundary is greater than 1/2 pixel and the block boundary is not a sub-block boundary after PROF processing is applied.

Here, the filter determination unit 154 may be configured to determine not to apply the deblocking filter to the block boundary when the boundary strength is “0”.

For example, the boundary strength judgment unit 153A may also be configured to judge the boundary strength of the block boundary as "1" when the absolute value of the difference between the motion vectors of the object block and the adjacent block is greater than a threshold value (e.g., 1/2 pixel) (i.e., the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is greater than a threshold value (e.g., 1/2 pixel)), and the blocks on both sides of the block boundary are not the blocks to which PROF processing is determined in step S95.

For example, the boundary strength judgment unit 153A may also be configured to judge the boundary strength of the block boundary as "1" when the absolute value of the difference between the motion vectors of the object block and the adjacent block is greater than a threshold value (e.g., 1/2 pixel) (i.e., the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is greater than a threshold value (e.g., 1/2 pixel)), and when at least one block at the block boundary is not a block determined in step S95 to be subject to PROF processing.

In addition, for a sub-block boundary of a block subjected to bidirectional prediction, when PROF processing is applied in at least one of L0 and L1, it can also be regarded as "a block determined to be subjected to PROF processing". That is, in this case, when PROF processing is not applied in both L0 and L1, the boundary strength is determined to be "1" when the difference in motion vectors is greater than the threshold.

That is, the boundary strength determination unit 153A may be configured to regard the PROF process as being applied to the target block when the target block is bidirectionally predicted and the PROF process is applied to at least one of the two reference frames.

Alternatively, only when PROF processing is applied to both L0 and L1, it may be considered as "a block determined to be applied PROF processing". That is, in this case, when PROF processing is applied only to L0 or L1 and the difference in motion vector is greater than the threshold, the boundary strength is determined to be "1".

As another example, the following judgment may be made: for a block boundary to which PROF processing is applied, when judging the boundary strength, the difference in motion vectors is not judged, or the difference in motion vectors is regarded as “0”. As a specific example, the judgment may be made as follows.

When the block is a block for unidirectional prediction and PROF is applied to the block, when judging the boundary strength of the sub-block boundary, the difference of the motion vector is not judged or is regarded as "0".

When the block is a block for bidirectional prediction and the reference frame on the L0 side is different from the reference frame on the L1 side, the difference in motion vectors is not determined or is regarded as "0" in the prediction directions (only L0, only L1, or both) of PROF applied in the L0 side and the L1 side.

When the block is a block for bidirectional prediction and the reference frame on the L0 side and the reference frame on the L1 side are the same frame, when PROF is applied on the L0 side, the difference between the motion vectors of L0 on both sides of the sub-block boundary is not determined, or the difference between the motion vectors of L0 on both sides of the sub-block boundary is regarded as "0".

Likewise, when PROF is applied on the L1 side, the difference between the motion vectors of L1 on both sides of the sub-block boundary is not determined, or is regarded as "0".

In this case, as in the past, the difference in motion vectors between L0 and L1 is used to determine the boundary strength. Specifically, for example, the difference in motion vectors between the L0 side of the sub-block on the left side of the sub-block boundary and the L1 side of the sub-block on the right side of the sub-block boundary is used to determine the boundary strength.

That is, the boundary strength judgment unit 153A may also be configured to, when judging the difference in motion vectors of blocks on both sides of the object boundary in the same prediction direction, not judge the difference in motion vectors in the prediction direction when PROF processing is applied in the prediction direction, or regard the difference in motion vectors in the prediction direction as "0".

Alternatively, when the block is a block for bidirectional prediction and the reference frame on the L0 side and the reference frame on the L1 side are the same frame, when PROF is applied to the motion vectors on both sides of the sub-block boundary respectively, the difference in the motion vectors may not be determined, or the difference in the motion vectors may be regarded as "0".

In this case, for example, when PROF is applied to both the L0 side and the L1 side, the difference in motion vector between L0 and L1 is not determined, or is regarded as "0", which is different from the above example.

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether the reference blocks used for prediction of the motion vectors of the target block and the adjacent blocks are different.

For example, as shown in FIG. 20 , the boundary strength judgment unit 153A may be configured to judge the boundary strength of the block boundary as “1” when the reference blocks used for predicting the motion vectors of the object block and the adjacent block are different (i.e., the reference images in the blocks on both sides of the block boundary are different).

The boundary strength determination unit 153A may be configured to determine the boundary strength of the block boundary based on control data indicating whether the number of motion vectors of the target block and the number of motion vectors of the adjacent blocks are different.

For example, as shown in FIG. 20 , the boundary strength judgment unit 153A may be configured to judge that the boundary strength of the block boundary is “1” when the number of motion vectors of the object block and the adjacent block is different (that is, when the number of motion vectors in the blocks on both sides of the block boundary is different).

The above describes an example of a method of using a motion vector or a reference block to determine the boundary strength, but the determination using a motion vector or a reference block may be performed only on a block boundary detected as the above-mentioned CU boundary, sub-block boundary or motion vector boundary.

In other words, it is also possible to skip the determination of using a motion vector or a reference block at a block boundary that is a TU boundary but not a subblock boundary or a motion vector boundary. In this case, at the TU boundary, only whether there is a non-zero orthogonal transform coefficient in at least one of the blocks on both sides of the boundary.

For example, as shown in FIG. 20 , the boundary strength determination unit 153A may be configured to determine that the boundary strength of the block boundary is “0” when none of the above conditions are satisfied.

That is, the affine prediction signal generation unit 111F/241F is configured to determine whether PROF is appropriate for each block, and when the PROF process is applied, the PROF process is performed on each sub-block obtained by dividing the block.

Furthermore, the loop filter processing unit 150 / 250 is configured to determine whether the deblocking filter is appropriate for each block boundary, and determine not to apply the deblocking filter to the sub-block boundary of the block to which the PROF process is applied.

By setting the above configuration, the boundary strength of the block boundary or sub-block boundary after the PROF processing is applied is "0", and the deblocking filter is no longer applied. Since the PROF processing has the effect of improving the subjective image quality, by setting the configuration as such, it is possible to prevent the following situation: due to the correction of the pixel value of the block boundary by the deblocking filter, the subjective image quality is damaged.

Furthermore, the above-described image encoding apparatus 100 and image decoding apparatus 200 may be realized by a program that causes a computer to execute each function (each step).

In addition, in the above-mentioned embodiments, the application of the present invention to the image encoding device 100 and the image decoding device 200 is described as an example, but the present invention is not limited to this example, and can also be applied to an image encoding/decoding system having the functions of the image encoding device 100 and the image decoding device 200.

According to the present invention, it is possible to prevent subjective image quality from being deteriorated by not applying a deblocking filter to a sub-block boundary after PROF processing is applied.

[Explanation of symbols]

10: image processing system; 100: image encoding device; 111, 241: inter-frame prediction unit; 111A: motion vector search unit; 111B: motion vector encoding unit; 111C, 241C: refinement unit; 111D, 241D: prediction signal generation unit; 111E, 241E: affine motion vector calculation unit; 111F, 241F: affine prediction signal generation unit; 112, 242: intra-frame prediction unit; 121: subtractor; 122, 230: adder; 131: transformation and quantization unit; 132, 220: inverse transformation and inverse quantization unit; 140: encoding unit; 150, 250: loop filter processing unit; 160, 260: frame buffer; 200: image decoding device; 210: decoding unit; 241B: motion vector decoding unit.

Claims (8)

1. An image decoding device, comprising: an affine prediction signal generating unit configured to determine whether a PROF (Prediction Refinement with Optical Flow) process is suitable for each block, and when the PROF process is applied, perform the PROF process on each sub-block obtained by dividing the block; and The loop filter processing unit is configured to determine whether deblocking filtering is appropriate for each block boundary, and determine not to apply deblocking filtering to the sub-block boundary of the block to which the PROF processing is applied, The loop filter processing unit further includes a block boundary detection unit configured to determine whether the target pixel position is the sub-block boundary at a minimum interval of 4 pixels based on the coordinates of the left end or the upper end of the target block. 2. The image decoding device according to claim 1, wherein the loop filtering processing unit comprises: a boundary strength determination unit configured to determine the boundary strength for each block boundary, and configured to determine the boundary strength to be "1" when the absolute value of the difference between the motion vectors of the blocks on both sides of the block boundary is greater than 1/2 pixel and the boundary is not a sub-block boundary after the PROF process is applied; and The filter determination unit is configured to determine not to apply the deblocking filter to the block boundary when the boundary strength is "0". 3. The image decoding device according to claim 1, wherein the loop filtering processing unit comprises: a block boundary detection unit configured not to detect a sub-block within a block to which the PROF process is applied as the block boundary; a boundary strength determination unit configured to determine the boundary strength of a pixel position not detected as the block boundary as "0"; and The filter determination unit is configured to determine not to apply the deblocking filter to the block boundary when the boundary strength is "0". 4. The image decoding device according to claim 2 or 3 is characterized in that the boundary strength judgment unit is configured to regard the PROF processing as being applied to the object block when the object block is bidirectionally predicted and the PROF processing is applied to at least one of the two reference frames. 5. The image decoding device according to claim 2 or 3 is characterized in that the boundary strength judgment unit is configured to regard the PROF processing as being applied to the object block when the object block is bidirectionally predicted and the PROF processing is applied to two reference frames. 6. The image decoding device according to claim 2 or 3 is characterized in that the boundary strength judgment unit is configured to, when judging the difference between the motion vectors of blocks on both sides of the object boundary in the same prediction direction, not judge the difference between the motion vectors in the prediction direction, or regard the difference between the motion vectors in the prediction direction as "0" when the PROF processing is applied in the prediction direction. 7. An image decoding method, characterized in that it comprises the following steps: Determining whether a PROF (Prediction Refinement with Optical Flow) process is suitable for each block, and when the PROF process is applied, performing the PROF process on each sub-block obtained by dividing the block; and Determine whether deblocking filtering is suitable for each block boundary, and determine not to apply deblocking filtering to the sub-block boundary of the block after applying the PROF processing, Based on the coordinates of the left end or the top end of the object block, it is determined whether the object pixel position is the sub-block boundary at a minimum interval of every 4 pixels. 8. A computer-readable storage medium storing a program, wherein the program is used in an image decoding device and causes a computer to execute the following steps: Determining whether a PROF (Prediction Refinement with Optical Flow) process is suitable for each block, and when the PROF process is applied, performing the PROF process on each sub-block obtained by dividing the block; and Determine whether deblocking filtering is suitable for each block boundary, and determine not to apply deblocking filtering to the sub-block boundary of the block after applying the PROF processing, Based on the coordinates of the left end or the top end of the object block, it is determined whether the object pixel position is the sub-block boundary at a minimum interval of every 4 pixels.